3.5_from_keras_to_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# From Keras to Pytorch\n",
    "## And build \"Coord Conv\" with 4 lines of code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keras\n",
    "\n",
    "Yes yes, keras is friendlier for beginners.\n",
    "\n",
    "#### Work well with google's tensorflow\n",
    "Even the official [\"get started\" tutorial page of tensorflow](https://www.tensorflow.org/tutorials/) starts with keras api now (2018-07-12).\n",
    "\n",
    "#### Simplicity\n",
    "Keras doesn't complicate the DL when it's not necessary, like a dense layer should cost you a line of code, no more.\n",
    "\n",
    "And choosing optimizer, loss function, extra metrics, all in one line of code.\n",
    "\n",
    "#### Keras codes:\n",
    "For using keras. See the previous notebooks\n",
    "\n",
    "## Pytorch\n",
    "Pytorch is another level of awesomeness, if tensorflow isn't from google and bear its reputation, pytorch should have taken the industry by now.\n",
    "\n",
    "If you try reading paper, that has some interesting structure, method about DL, like different strategy between train/inference, several optimizer, a wierd shape of model and wierder strategy of updating them partially, or some time just a very unique loss function work with a very tailor made data generator.\n",
    "\n",
    "I used to do those edgy models with keras. It's hard, it's just hard. Like modify a luxury hotel for a school or a hospital or a jail or all kinds of other buildings. The overhaul on engineering alone is nightmare \n",
    "\n",
    "![search trends on deep learning lib](docs/dl_lib_trends.png)\n",
    "\n",
    "Among other convenient features, pytorch is trending, becoming the world's most used deep learning framework in AI labs.(Not yet for business)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Requirements\n",
    "\n",
    "```\n",
    "python 3.6\n",
    "\n",
    "numpy == '1.14.3'\n",
    "pandas == '0.23.0'\n",
    "tensorflow == '1.8.0'\n",
    "keras == '2.2.0'\n",
    "tqdm == '4.23.4'\n",
    "```\n",
    "Other versions of above library will probably work\n",
    "\n",
    "If you don't have any of these, try the following format in the command line:\n",
    "```\n",
    "~/anaconda3/bin/pip install keras==2.2.0\n",
    "```\n",
    "assuming your anaconda3 is at ```~/anaconda3/```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.datasets import FashionMNIST\n",
    "from torchvision import transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using the [fashion-mnist](https://github.com/zalandoresearch/fashion-mnist) dataset as the problem we are going to tackle today."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform = transforms.Compose([transforms.Resize((28,28)),\n",
    "                                transforms.Grayscale(),\n",
    "                                transforms.ToTensor(),\n",
    "                                transforms.Normalize([.5,.5,.5],\n",
    "                                                     [.5,.5,.5])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz\n",
      "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz\n",
      "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz\n",
      "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz\n",
      "Processing...\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "ds = FashionMNIST(\"/data/fm\",train = True, \n",
    "              transform=transform,\n",
    "              download=True, # Set this to False second time you ran it\n",
    "             )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_val = FashionMNIST(\"/data/fm\",train = False, \n",
    "              transform=transform,\n",
    "              download=False, # Set this to False second time you ran it\n",
    "             )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### alternative downloading url:\n",
    "In case it's too slow\n",
    "Click this [tar.gz file address](http://45.76.223.58/fm.tar.gz) to download, you can uncomment the following and run "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %mkdir -p data\n",
    "# %mkdir -p data/fm\n",
    "# !!cd data; wget http://45.76.223.58/fm.tar.gz\n",
    "# !!tar -xzvf fm.tar.gz; cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 308,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 309,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = DataLoader(ds,batch_size=64,shuffle=True,)\n",
    "gen = iter(dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 310,
   "metadata": {},
   "outputs": [],
   "source": [
    "x,y = next(gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 1, 28, 28]), torch.Size([64]))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.size(),y.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 322,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x32ab57ba8>"
      ]
     },
     "execution_count": 322,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAENxJREFUeJzt3VuMnPV5x/Hfs7OzZ5/A+IAxNjEuhSCVJCuCQtsQISg5SCYXQfFF5FZRHKlBaqRUKuImXLQqqhJSLqqoTnFiJEJAAgIXtA2hUZ1UCcKmKMY4LY5ZYPHi9dnr9e7sYZ5e7JgusO/zLnO2/9+PhHZ2nnnnfRjvb9+Z/b/v/2/uLgDp6Wh1AwBag/ADiSL8QKIIP5Aowg8kivADiSL8QKIIP5Aowg8kqrOZO+uybu9RfzN3eUGw3p6w3r2xFNZPT/ZmP7fFZ3D6jIX1np7psD5ZKoZ1dQT7L8f77uqaCeuzp+J9dx4dD+sXo0mNa8pL8QtbUVP4zewOSQ9KKkj6F3e/P3p8j/r1Sbu1ll1elDqu/sOwvumHr4f1Zw98NLNW7I4DNHU8/sVz7bXDYf3A7y8P6x09s5m18kT847dhw9GwPvbE2rC+8p9/HdYvRi/484t+bNVv+82sIOmfJH1W0nWStprZddU+H4DmquUz/42SDrr7IXefkvQTSVvq0xaARqsl/OskvTXv++HKfe9hZtvNbI+Z7ZlW/NkVQPPUEv6F/qjwgb/uuPsOdx9098GiumvYHYB6qiX8w5LWz/v+CkmHa2sHQLPUEv4XJW02s6vMrEvSlyU9U5+2ADRa1UN97j5jZndL+nfNDfXtdPf9dessIeNXLw3rn1l2IKzvX7sms7a6byzcdmptIax3dpTD+kc2job1S3uyx9qPT8bnfGwcOBHWd1+VM9QXVlHTOL+7Pyvp2Tr1AqCJOL0XSBThBxJF+IFEEX4gUYQfSBThBxLV1Ov5sbCRm+Kx9l+cvjasj54ZyKydOpd9rb8kLeudDOt5Rk7G5yiMdmX3lufURHy58czy7MuFkY8jP5Aowg8kivADiSL8QKIIP5Aowg8kiqG+NrD0o8fD+v5T2ZfsSlK5HP0Oj4fDxqe6wvrkdPwjkjc1eFSP+5amZ+Mh0MLSqbCOGEd+IFGEH0gU4QcSRfiBRBF+IFGEH0gU4QcSxTh/G9h8ybGw/saZFWF9zfIzmbXJmfifuJQzjn9p/7mwnndJ77nx7Mtye3rjcfrpnN4KhXhaccQ48gOJIvxAogg/kCjCDySK8AOJIvxAogg/kKiaxvnNbEjSmOYuGp9x98F6NHWxKVy7OawvKw6H9bGJeCnqY5PFD93Ted0902F9VV98DsIVA6fC+vLiRGbt0NlLw21/91Y8j0Fn10xYR6weJ/l8xt3jnxAAbYe3/UCiag2/S/qZme01s+31aAhAc9T6tv9mdz9sZqskPWdmv3P33fMfUPmlsF2SetRX4+4A1EtNR353P1z5OirpKUk3LvCYHe4+6O6DRXXXsjsAdVR1+M2s38yWnL8t6XZJr9SrMQCNVcvb/tWSnjKz88/zY3f/t7p0BaDhqg6/ux+S9Ed17OWiVR6Il5o+ORX/LWRmJn6Dtm5l9lj7qr6xcNublr8e1gcK8RLer02sDusvHtuQWRsrxWsGDCzNPkdAksaO94d1xBjqAxJF+IFEEX4gUYQfSBThBxJF+IFEMXV3E0wti4e0+jvjKaw7O+MpqmeCpa5fHY0vi13fezKs37Zsf1h/YN+tYf0PVh/NrJ2eiIdA+7vj12Xi6PKwjhhHfiBRhB9IFOEHEkX4gUQRfiBRhB9IFOEHEsU4fxPMDBRq2n5yIj5PIJo8e+JcvO3bk/FY+VOzHw/rn954MKxPzGZPKz5sy8JtL+2NlwcvDYVl5ODIDySK8AOJIvxAogg/kCjCDySK8AOJIvxAohjnb4Jyp4X16XJ8HkAxZynqUrBE9zXrjoTbvno0nnpbu1eE5b//y51h/Tuv/1lmrdDh4bZ9OfMcLH2DJbprwZEfSBThBxJF+IFEEX4gUYQfSBThBxJF+IFE5Y7zm9lOSV+QNOru11fuu0TSY5I2ShqSdJe7xxPAJ2y6t7bfsTPT8XkAs8ES3revejXc9rEdd4T1JUPjYf3zfx0v4f23M9k/YoWOeD2Cld3xvs++diysz4ZVLOan8keS3v8Tco+k5919s6TnK98DuIDkht/dd0s68b67t0jaVbm9S9Kdde4LQINV+350tbuPSFLl66r6tQSgGRp+br+ZbZe0XZJ61Nfo3QFYpGqP/EfMbK0kVb6OZj3Q3Xe4+6C7DxbVXeXuANRbteF/RtK2yu1tkp6uTzsAmiU3/Gb2qKRfS7rGzIbN7KuS7pd0m5m9Jum2yvcALiC5n/ndfWtGKV6YHe86e0V8Pf/kbPzPYDnj4R3B5ks64nH4pY/+Jqzn2VuKr7lf0TORWTtd6gm3Xd4Zz9s/e/D1sI4YZ/gBiSL8QKIIP5Aowg8kivADiSL8QKKYursJzq2Pp5g+Ptkf1suz8SW9K1eMZdZ2vvGpcNsBHQrreX547E/CejT9dt5QX6mc9+MZD4EixpEfSBThBxJF+IFEEX4gUYQfSBThBxJF+IFEMc7fBDYQj/OfLdU2w9HNq7PH6v/1qZvCbWsd5z813VvT9pGJclfOI+LLlRHjyA8kivADiSL8QKIIP5Aowg8kivADiSL8QKIY52+CvoFSWJ8tx1N79/TG02NPe/b1/ut+mT11dj3kTTveYZ5ZK3v8/13KeW7UhiM/kCjCDySK8AOJIvxAogg/kCjCDySK8AOJyh1INbOdkr4gadTdr6/cd5+kr0k6WnnYve7+bKOavNB1F6fDerEQzz9/bjK+rn18Jns+gI7//O9w21q9dOjKsP656/Zn1kbPLQm3PVoayNn72Zw6Ios58v9I0h0L3P89d7+h8h/BBy4wueF3992STjShFwBNVMtn/rvN7LdmttPMVtStIwBNUW34vy9pk6QbJI1I+m7WA81su5ntMbM904rPcQfQPFWF392PuPusu5cl/UDSjcFjd7j7oLsPFlXbRJUA6qeq8JvZ2nnfflHSK/VpB0CzLGao71FJt0haaWbDkr4t6RYzu0GSSxqS9PUG9gigAXLD7+5bF7j7oQb0ctEqTRfD+oZlo2G9N+c8gd5CXG+kjmPxOQhruk9n1kb74nH8k6W+eN9hFXl4/YBEEX4gUYQfSBThBxJF+IFEEX4gUcyN3ATjo/3xA1bG5c6O+JLfa/reyaz9vm9duG353Ll453myZ+aWJB0uLa/6qdf2ZQ8TStKRqp8ZEkd+IFmEH0gU4QcSRfiBRBF+IFGEH0gU4QcSxTh/EyzfF7/Ml3/iTFh/4Uw8ReKsspe6rnkcP0fvaHz86LTZzNrxyfj8h09eOhTWj3DsqgmvHpAowg8kivADiSL8QKIIP5Aowg8kivADiWKcvwmWvD0T1jf0HgvrL2hDWD85kz1e3tEXT39d63kAXafiC/rLnn186bB424Pjl+Xs/XhOHRGO/ECiCD+QKMIPJIrwA4ki/ECiCD+QKMIPJCp3nN/M1kt6WNIaSWVJO9z9QTO7RNJjkjZKGpJ0l7ufbFyrF66+tyfCeo/F5wEUcubtPzC2JrNW+tTGcNviz/eG9TwDI9nX60tSd0f28uHdhfj/u6sjfm7UZjFH/hlJ33L3ayXdJOkbZnadpHskPe/umyU9X/kewAUiN/zuPuLuL1Vuj0k6IGmdpC2SdlUetkvSnY1qEkD9fajP/Ga2UdLHJL0gabW7j0hzvyAkrap3cwAaZ9HhN7MBSU9I+qa7x5POvXe77Wa2x8z2TKtUTY8AGmBR4TezouaC/4i7P1m5+4iZra3U10oaXWhbd9/h7oPuPlhUdz16BlAHueE3M5P0kKQD7v7AvNIzkrZVbm+T9HT92wPQKIu5pPdmSV+RtM/MXq7cd6+k+yU9bmZflfSmpC81psULX2HkRFjv64g/DuUt0R2ZXhr/ExerfuY5ve/Ew5ilcvYeZsrxsae/M35djoZV5MkNv7v/SsqcGP7W+rYDoFk4ww9IFOEHEkX4gUQRfiBRhB9IFOEHEsXU3U0wM/x2WB+euiSsL+2eDOvRFNgTK+Lf7/HE3vk6ht4J6+OzXZm1rkJ8ye7EbN5ZCPE5Bohx5AcSRfiBRBF+IFGEH0gU4QcSRfiBRBF+IFGM87eBvaeuDOs9hezpr/NMrM66Grs+Zo/GV9VPzGafw3BZ99lw23LmleSoB478QKIIP5Aowg8kivADiSL8QKIIP5Aowg8kinH+NvDq8NqwPrjxjaqfO5g2vyneOLMis7Zp2fFw2wPH4+UfV2qsqp4whyM/kCjCDySK8AOJIvxAogg/kCjCDySK8AOJyh3nN7P1kh6WtEZSWdIOd3/QzO6T9DX9/zLp97r7s41q9GK2/D96wnr5L+Lr2qPr/XuOVdVS3bhn9768GM+7f/pMf1hfWVVHOG8xJ/nMSPqWu79kZksk7TWz5yq177n7dxrXHoBGyQ2/u49IGqncHjOzA5LWNboxAI31oT7zm9lGSR+T9ELlrrvN7LdmttPMFjyP08y2m9keM9szrVJNzQKon0WH38wGJD0h6ZvufkbS9yVtknSD5t4ZfHeh7dx9h7sPuvtgUd11aBlAPSwq/GZW1FzwH3H3JyXJ3Y+4+6y7lyX9QNKNjWsTQL3lht/MTNJDkg64+wPz7p9/KdoXJb1S//YANMpi/tp/s6SvSNpnZi9X7rtX0lYzu0GSSxqS9PWGdJiAZa9PhfX1vSfDerRE939dXQ63jS+arV1vMXsY8vLuU+G2n970WlgfrqojnLeYv/b/SlpwAnXG9IELGGf4AYki/ECiCD+QKMIPJIrwA4ki/ECimLq7DXTvfyusP7n3E2G9cKaQWbv68fGqeqqX009enll7ZEV8fdiSN+NzFJbpN1X1hDkc+YFEEX4gUYQfSBThBxJF+IFEEX4gUYQfSJS5Z18LXvedmR2VNH+96ZWSWjy5dKZ27a1d+5LorVr17G2Du1+2mAc2Nfwf2LnZHncfbFkDgXbtrV37kuitWq3qjbf9QKIIP5CoVod/R4v3H2nX3tq1L4neqtWS3lr6mR9A67T6yA+gRVoSfjO7w8z+x8wOmtk9reghi5kNmdk+M3vZzPa0uJedZjZqZq/Mu+8SM3vOzF6rfF1wmbQW9Xafmb1dee1eNrPPtai39Wb2CzM7YGb7zeyvKve39LUL+mrJ69b0t/1mVpD0v5Ju09zU6y9K2ururza1kQxmNiRp0N1bPiZsZn8q6aykh939+sp9/yDphLvfX/nFucLd/6ZNertP0tlWr9xcWVBm7fyVpSXdKenP1cLXLujrLrXgdWvFkf9GSQfd/ZC7T0n6iaQtLeij7bn7bkkn3nf3Fkm7Krd3ae6Hp+kyemsL7j7i7i9Vbo9JOr+ydEtfu6CvlmhF+NdJmj91zbDaa8lvl/QzM9trZttb3cwCVleWTT+/fHqjF935sHJXbm6m960s3TavXTUrXtdbK8K/0Oo/7TTkcLO7f1zSZyV9o/L2FouzqJWbm2WBlaXbQrUrXtdbK8I/LGn9vO+vkHS4BX0syN0PV76OSnpK7bf68JHzi6RWvo62uJ93tdPKzQutLK02eO3aacXrVoT/RUmbzewqM+uS9GVJz7Sgjw8ws/7KH2JkZv2Sblf7rT78jKRtldvbJD3dwl7eo11Wbs5aWVotfu3abcXrlpzkUxnK+EdJBUk73f3vmt7EAszsI5o72ktzMxv/uJW9mdmjkm7R3FVfRyR9W9JPJT0u6UpJb0r6krs3/Q9vGb3dorm3ru+u3Hz+M3aTe/tjSb+UtE/S+SmA79Xc5+uWvXZBX1vVgteNM/yARHGGH5Aowg8kivADiSL8QKIIP5Aowg8kivADiSL8QKL+D3XKsk02V2ZJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x.numpy()[18].reshape(28,28))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x127823f28>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFE1JREFUeJzt3W2MXOV1B/D/mdnZF6/XNmvjV+wsGBtwTArRxpCSpLQuCNxIEKmgIDVyqghHapCImrZBfAh8aCVKCylNUyonWDFKAkmbEGhEU5DblEQCYxsQxjUYY4xt/LI2y9rr3Z3deTn9sEOzmH3OGc+dmTvk+f8ka3fnzHPvs3fm7N31eV5EVUFE8cmk3QEiSgeTnyhSTH6iSDH5iSLF5CeKFJOfKFJMfqJIMfmJIsXkJ4pUWzNP1i4d2onuZp6SKCp5jGBCx6Wa5yZKfhG5DsADALIAvquq91jP70Q3rpC1SU5JRIatuqXq59b8a7+IZAF8G8D1AFYBuEVEVtV6PCJqriR/868BsFdV96nqBIBHAdxQn24RUaMlSf4lAA5O+fpQ5bH3EZENIrJdRLYXMJ7gdERUT0mSf7r/VPjA/GBV3aiq/aran0NHgtMRUT0lSf5DAJZO+fo8AIeTdYeImiVJ8m8DsEJEzheRdgCfB/BEfbpFRI1Wc6lPVYsichuA/8RkqW+Tqu6qW89iIk5ZNsFqSyf/5Eoz/s6l9rlzw3Z8Yk7ZjM97KRyb/f3nzLbUWInq/Kr6JIAn69QXImoiDu8lihSTnyhSTH6iSDH5iSLF5CeKFJOfKFJNnc8fLaeOn+mwhz2X83kzvu9vPxk+tV2GxwVff9Z+QkJ77w+PMxhaGe43AHzkG07fGjg+Iga88xNFislPFCkmP1GkmPxEkWLyE0WKyU8UKZb6WoBXyhv54yvMeLGnFIyt/LPna+pTvVz45+Fpu3s29Ztt5ROXmnHdttNun2sPty1MmG1jwDs/UaSY/ESRYvITRYrJTxQpJj9RpJj8RJFi8hNFinX+JpBs1oxrsWjG3/4De2rqqnuPBmP2ke1aOODXwzOdnWbcGsNw0T/b4xve+qNZZnzZNjMMLRbsJ0SOd36iSDH5iSLF5CeKFJOfKFJMfqJIMfmJIsXkJ4pUojq/iOwHMAygBKCoqvYE7TR5yzwnOXTCOn5mxgwznh21f0YX9x8w4xZ3Xrtz3crj4zWfOzt42oxPzJ5Z87EB+3XRUngNhKZogWXF6zHI5/dV9UQdjkNETcRf+4kilTT5FcBTIrJDRDbUo0NE1BxJf+2/SlUPi8h8AE+LyKuq+szUJ1R+KGwAgE7Yf9sSUfMkuvOr6uHKxwEAjwFYM81zNqpqv6r252DvSUdEzVNz8otIt4j0vPc5gGsBvFKvjhFRYyX5tX8BgMdkshTUBuCHqvqLuvSKiBqu5uRX1X0AfqeOfWmsBtZVvTq+p3T5SjN+7e+9ZMbfSHR2RwOvW6nXruPr3GRr6yd9XX7bsdRHFCkmP1GkmPxEkWLyE0WKyU8UKSY/UaS4dHcLGLqwy4z/YtdHzfgK7Khnd1rG7DmjZjy7crkZL+1paBHU5k0hb4EpvbzzE0WKyU8UKSY/UaSY/ESRYvITRYrJTxQpJj9RpFqrzp+gNpqdM9tsuvtee9osSt65wyFx2uoMe5loGSvb8cGcGX/z0Y8FY4VBewtteCuatzn16IwTN44/b/4ps+nQgL1F99Bf2p2X3JxgTAvOfS+brA6fydmv+YVfeDHR8euBd36iSDH5iSLF5CeKFJOfKFJMfqJIMfmJIsXkJ4pUa9X5E8xxLg2dNOPd++xauUeNkrLaO3QDsJ8wdoG9RPX5Fw+Y8YPPLwnGukbsWnipK1k92xvjIEa5e3Co12w7b+WgGR/ca7eftSf8mpeTvR2QcXb4bhtJf76+h3d+okgx+YkixeQnihSTnyhSTH6iSDH5iSLF5CeKlFvnF5FNAD4LYEBVV1ce6wXwIwB9APYDuFlV33XPJoC0hU/pbal8+uYrg7HDV9tz4pf9h33szITdvtATrtVbtWzAHwfQ/pxdE86902PGFy4Ld6Dc5qw1kPUm9NsyBfu6WbLONW//N3s/g7ljw2Y8v3hGMKYZb3yC/ZqI820PXWAPJDjwr5cGY8tu2mkfvE6qufN/D8B1Zzx2B4AtqroCwJbK10T0IeImv6o+A+DMoVY3ANhc+XwzgBvr3C8iarBa/+ZfoKpHAKDycX79ukREzdDwsf0isgHABgDoRPhvMCJqrlrv/MdEZBEAVD4GZ56o6kZV7VfV/px01Hg6Iqq3WpP/CQDrK5+vB/B4fbpDRM3iJr+IPALgWQAXicghEfkSgHsAXCMirwO4pvI1EX2IuH/zq+otgdDasz6b+rV8y9HfNYI5uy47eq7zrTrFeCucKdiHzjg14/Gl7Xb7xU68ED6+N+/cWqcAAMSZlu6NYbDr6fa9Z7jPrvN74yusvpfsS4qyM/7Be009xX0zgzFvDwpv7YpqcYQfUaSY/ESRYvITRYrJTxQpJj9RpJj8RJFqqaW7B//0k2Y8u2A0GMvt6Tbblp1SoLUFt8dbBrqcc8pGTqnQY5Xbyk4pr+T0zSv1eX1vPx2e+5o/x64Tlt0l0W3m6+Jcl2KnN9XZjhedkeydJ8LtpfccuzFLfUSUBJOfKFJMfqJIMfmJIsXkJ4oUk58oUkx+oki1VJ3/4bvuM+Prnr49GGvrtAvSxRnOEtZJprY6yzh7yzyLM73U49XiLdm8HS90JVv6e3hZ+P7ijRFwt9F2XjNznIDXtt2+qOrcNr2+z9oXjhUWzTHbitH2bPDOTxQpJj9RpJj8RJFi8hNFislPFCkmP1GkmPxEkWpqnb94bjeO3xSesz+s28z23XvDxdORiybMtqV37WJ6krnj3vLYjaZizA13xgCU59rxYrd9gMxE7Vt8e7V079jeGg1WLd5bctxbFtxT7LEPMNETTr38PPu9ai9oXj3e+YkixeQnihSTnyhSTH6iSDH5iSLF5CeKFJOfKFJunV9ENgH4LIABVV1deexuALcCOF552p2q+qR3rFnzRrD21ueC8b/Yc7PZvmMoXNcdzdqT5svOd+rVnKUUrjk70/Wh3rmderW3HoC1GIH3fRXm2VumX/ytYTP+2q32dtKZ8XDfcqfse09xhjNIwRliYM65dxdBqH38AgD3tjreGz7/2Fx7EELPeUuCMTnqLYLwG9Xc+b8H4LppHv+mql5W+ecmPhG1Fjf5VfUZAINN6AsRNVGSv/lvE5GXRWSTiDj7CxFRq6k1+R8EsBzAZQCOAAguviciG0Rku4hsHx0ar/F0RFRvNSW/qh5T1ZKqlgF8B8Aa47kbVbVfVftnzOmotZ9EVGc1Jb+ILJry5ecAvFKf7hBRs1RT6nsEwNUA5onIIQB3AbhaRC7D5MbW+wF8uYF9JKIGcJNfVW+Z5uGHajlZe6aIvs4TwfjjO64022cWh2uvmQHnTwqnrOutwy5GObyccN19b966N/fcGidQmGvX8bMn7YOXX37VjLe/G16fAQDKK0bDwVPdZluXNwzAGh/hbNSgbcnWGsjk7TdU1hj/MDbfbIrRjy4Kxson61vnJ6LfQkx+okgx+YkixeQnihSTnyhSTH6iSDV16e7xchteH1sQjGdXnLbbH5sRjIlTemkbs+NSdko3RsUs6TLPyRkl0IJd+hlfbO+TnZkRvuaAP+0282a4fftJs6lZDqtKgublNrtxYaYzDXu+vfe5nrIW4LbPPbgqXFsuvlj9N807P1GkmPxEkWLyE0WKyU8UKSY/UaSY/ESRYvITRaqpdX4BkDOK4k+uedBsv+6hvwrGFn76bbPt2j98zYxf2HHMjO8Y6QvGZmbt5ck6M3YtvcOJJ9GTsevNebXHAfS9fNyMH5x4q+bjr+g4arZtpE6xr/lQyZ5ufLRoL1m+ayS8vDYA/Nfey4Ox3CmzKcp1WhCLd36iSDH5iSLF5CeKFJOfKFJMfqJIMfmJIsXkJ4pUU+v8w8e78ct/Ci/P/Xd//aLZvnTJSDB24Giv2fahg58y4509dq1+dvdYMNaetSf0l51loktOvLPNXn67UAovv93htO3I2vHBMXs+f1fOrpfni+G32NwuY1lv+NfNk3G34Q5rc/ZFHyva4yOKzlrw1tbl+XlmUyzcOhGMtY1V/z3zzk8UKSY/UaSY/ESRYvITRYrJTxQpJj9RpJj8RJFy6/wishTAwwAWAigD2KiqD4hIL4AfAegDsB/Azar6rnWs7IkR9G56Nhjf/Q277vvp898Ixp79+cfMtmPL7Hr0RN6+FCcmesJBp56sJbteneuwa+3For2NtnX8csHZ39shWbve7W11bV2bo6VzzKaZXOM2RFBvnwZni+4Z3fY6CXO7nTEMHeHje9vFd73xTjCWGbffS+97bhXPKQL4mqpeAuBKAF8RkVUA7gCwRVVXANhS+ZqIPiTc5FfVI6r6QuXzYQC7ASwBcAOAzZWnbQZwY6M6SUT1d1Z/84tIH4DLAWwFsEBVjwCTPyAAzK9354iocapOfhGZCeAnAL6qqs4qY+9rt0FEtovI9gLs8fNE1DxVJb+I5DCZ+D9Q1Z9WHj4mIosq8UUABqZrq6obVbVfVftzqNPKg0SUmJv8IiIAHgKwW1XvnxJ6AsD6yufrATxe/+4RUaNUM6X3KgBfALBTRF6qPHYngHsA/FhEvgTgAICbknZm3VO3m/Fd674djF3au9ps2/m2PQUzb6+0DGkPl7zccpijXLJ/Bpcn7HKddf5cl13iLBWTDfWQjFPmNC6NZOxSnleO81jlumybfe6M85rm8/b7qdBpv2YTs8N96xh03g/7DwZjWgxP9z2Tm/yq+muENwxfW/WZiKilcIQfUaSY/ESRYvITRYrJTxQpJj9RpJj8RJFq6tLdnkvuM2cE44urrg/GPnHFHrPt1l3LzXimy54KadWcs212Tbij0661e1N2s2127TaTCZ+/XHZ+vnvLWydcPrts3V+cc2edKb3e92ZNdS45bQun28245OzX/J1he4tvGOMj2sIr1AMAtGi8V89itXLe+YkixeQnihSTnyhSTH6iSDH5iSLF5CeKFJOfKFItVefv+/4hMz44Ed4uujtr18Ifv/ZbZtzbDnqgNDMYe31iodk2qV0j9mIDLxw/LxgbOt1lts1m7cJwwVlLoC1nj4+wavHW+ITJuFe0dtZREGNJc+f1znbbYzPKBfu+mT/lrFqVC39vva9Vv/x2ErzzE0WKyU8UKSY/UaSY/ESRYvITRYrJTxQpJj9RpFqqzv/Ln33cjK++/rVgbLxsfyv3Hr7OjF8886gZt5wohMcAAMD+03PN+IGTc8z4Z5aEtyYHgH+85JFg7H9GLjbbvnp6kRn3rutE2R4HMFoMz4svlJy17Z1jtzvr/meNcQTefH6rbTVmtdtbeD+/ty8Y6/z5i4nOXS3e+YkixeQnihSTnyhSTH6iSDH5iSLF5CeKFJOfKFJunV9ElgJ4GMBCTE6g3qiqD4jI3QBuBXC88tQ7VfXJJJ05f9M+M/7irJXBWGmBPZ+/q8epuxaWmXFr/XpvXf7Fs06Z8dlddt/+/Vf9Znzn6sXB2GjB3kc+P2HH25x96pPWwy2jTt/yeTtePBFey6DjuLNOgbN2/shS57rk7fUCLnnw7WCsqGex+H4C1QzyKQL4mqq+ICI9AHaIyNOV2DdV9e8b1z0iahQ3+VX1CIAjlc+HRWQ3AHtpGSJqeWf1N7+I9AG4HMDWykO3icjLIrJJRM4JtNkgIttFZHsB44k6S0T1U3Xyi8hMAD8B8FVVPQXgQQDLAVyGyd8M7puunapuVNV+Ve3PwVnXjIiapqrkF5EcJhP/B6r6UwBQ1WOqWlLVMoDvAFjTuG4SUb25yS8iAuAhALtV9f4pj0+dDvY5AK/Uv3tE1CiiTllBRD4F4FcAduI3ayXfCeAWTP7KrwD2A/hy5T8Hg2ZJr14haxN2OdhRM5xd3me3b3fKRnPCZaOR8+zlsZ2ZqcjPtX8GL/iX5824uWUzffg472UYObtVt+CUDla1r3o1/9v/awDTHSxRTZ+I0sURfkSRYvITRYrJTxQpJj9RpJj8RJFi8hNFqqWW7k7EGa9Q2vtmosNbhVN74W7fLCfenAme1DKaNKWXd36iSDH5iSLF5CeKFJOfKFJMfqJIMfmJIsXkJ4qUO5+/ricTOQ7grSkPzQNwomkdODut2rdW7RfAvtWqnn37iKqeW80Tm5r8Hzi5yHZVtRelT0mr9q1V+wWwb7VKq2/8tZ8oUkx+okilnfwbUz6/pVX71qr9Ati3WqXSt1T/5iei9KR95yeilKSS/CJynYi8JiJ7ReSONPoQIiL7RWSniLwkIttT7ssmERkQkVemPNYrIk+LyOuVj9Nuk5ZS3+4Wkbcr1+4lEVmXUt+Wish/i8huEdklIrdXHk/12hn9SuW6Nf3XfhHJAtgD4BoAhwBsA3CLqv5vUzsSICL7AfSrauo1YRH5DIDTAB5W1dWVx+4FMKiq91R+cJ6jql9vkb7dDeB02js3VzaUWTR1Z2kANwL4IlK8dka/bkYK1y2NO/8aAHtVdZ+qTgB4FMANKfSj5anqMwAGz3j4BgCbK59vxuSbp+kCfWsJqnpEVV+ofD4M4L2dpVO9dka/UpFG8i8BcHDK14fQWlt+K4CnRGSHiGxIuzPTWPDezkiVj/NT7s+Z3J2bm+mMnaVb5trVsuN1vaWR/NOtiNVKJYerVPXjAK4H8JXKr7dUnap2bm6WaXaWbgm17nhdb2kk/yEAS6d8fR6Awyn0Y1qqerjycQDAY2i93YePvbdJauXjQMr9+X+ttHPzdDtLowWuXSvteJ1G8m8DsEJEzheRdgCfB/BECv34ABHprvxHDESkG8C1aL3dh58AsL7y+XoAj6fYl/dplZ2bQztLI+Vr12o7XqcyyKdSyvgHAFkAm1T1b5reiWmIyAWYvNsDkysb/zDNvonIIwCuxuSsr2MA7gLwMwA/BrAMwAEAN6lq0//jLdC3q3GWOzc3qG+hnaW3IsVrV88dr+vSH47wI4oTR/gRRYrJTxQpJj9RpJj8RJFi8hNFislPFCkmP1GkmPxEkfo/8if8ttV6CyUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x.numpy()[1].reshape(28,28))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x127970f28>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFadJREFUeJzt3X1w1dWZB/Dvc29uEvJCIAYCSBDkRUVtUSNqdSzWl2rXXWy7MnXcLnYccde6rVt3Xct0Rt3a0dm19WXbaYsLW1zf0FYru+taKbNWXZUaLAUVBYqRVxMRQhJCkvvy7B+5dKPkPCfc93C+nxmG5D733Hvyy/3ml5vzO+eIqoKIwhMpdgeIqDgYfqJAMfxEgWL4iQLF8BMFiuEnChTDTxQohp8oUAw/UaDKCvlk5VKhlagu5FMOm4iYda0Z5az119htpSppP7ba7cu3HjTrI1Wy3n4t1DYeMOv7eqrMerTHfVxj+3rNtpqwv2elqhcH0K999gsqLavwi8ilAO4HEAXwr6p6t3X/SlTjLLkwm6fMm0hlpVmPnznbWdt1nt02ctp+s55M2r+ATblyg1kfqTovO9usn/d3a8z6z984w6yPXRtz1iY8uclsm9zzkVkvVWt09bDvm/Gv/SISBfAjAJcBmA3gKhFxJ4SISko27/nnAtiiqltVtR/A4wDm56ZbRJRv2YT/WADbB32+I33bx4jIIhFpEZGWOPqyeDoiyqVswj/UHxUOmx+sqktUtVlVm2OoyOLpiCiXsgn/DgBNgz6fDGBXdt0hokLJJvyvA5gpItNEpBzAVwCszE23iCjfMh7qU9WEiNwI4FcYGOpbpqpv5axnR6hsQqNZ3/yN4816fFzCrFfUuceFYzF7zDi5doxZf/ja+8z6p3ZGzfrS/VOctaqI/XeWTb0TzfrE8g6zHkXKrH+trtVZq5B1Ztsz7vhrsx45yV6Fqvv8HmftnZNn2I9d32TWx6+0h3drH3/NrMO6rqRAq2tlNc6vqs8CeDZHfSGiAuLlvUSBYviJAsXwEwWK4ScKFMNPFCiGnyhQBZ3Pn1eVnkuHPTOcT7pli1lP7tt3hB0avsV3zjXrm35i13940UPO2r+3fcZs+8GB0Wa9YVS3Wb+s4U2zfuJz7rH6Wde22M+NVz31LHjWbzg4/0y7vdrXN3iVwE5ZPPMTBYrhJwoUw08UKIafKFAMP1GgGH6iQI2ooT6pcA/nbf/SZLPttG+/YtY/+stzzHr/l91DfSeP+8Bse/5Ye6XYPfFas/7+O/bS3Q/u/KyzdkXj78y2/fX2S+DlDnvq652vXm7WI53ux5/+uj0tdk7NNrO+rts9lRkAVm+d5awlt9nLhp/wo51mPdlgD5F2XG2vTFz3iGfKbwHwzE8UKIafKFAMP1GgGH6iQDH8RIFi+IkCxfATBUq0gFMLR0u95muX3uiMaWa9+5RxZn3HBfbPwe9e+qSzdsdTC8y2ZcZW0QAgnt2gU+V2XeLuWsTz2P119vc/MdqeuqpRu730uY9rrMs+LlHPTtPxas9zT3Vv8f3wWUvNtm/3Hbbz3Mc8+fmzzLrG7OsnUtvd+9toX+bb2q3R1ejUvcPaoptnfqJAMfxEgWL4iQLF8BMFiuEnChTDTxQohp8oUFmN84tIK4AuAEkACVVttu6f7Tj//mfdc8v3/t4ex5/2bXsZ6L4v2Es1V7/hnlue+KDNbEvFEW04JuO2UmmvNZDYYc/3961V8NKK0521SffYa09YjmScPxeLeVygqnty8DhEVED8tZ8oUNmGXwE8LyJrRWRRLjpERIWR7a/956rqLhEZD2CViLyjqi8OvkP6h8IiAKhEVZZPR0S5ktWZX1V3pf9vB/A0gMM2lVPVJararKrNMXj20yOigsk4/CJSLSK1hz4GcAkAe9dGIioZ2fza3wjgaRnY7bQMwKOq+lxOekVEeZdx+FV1K4BP57AvXu1b3eO2yxb81Gx7Q89fmfWm73rGVo+d5Cz1/qm9hfa2y+1rKSrG9pr1WMwzKd/Q329/ixPxqFmPRu35/DXVdt/HVrn3HOhN2H1r32uvjV/1W/tvSJMf/4OzpseMMdsm3rb3WvCN48dT9nGd8uR293ObLXOHQ31EgWL4iQLF8BMFiuEnChTDTxQohp8oUEfN0t1b7rW3RF7yZw+a9W89cL1Zv/8bP3HW5o2yh8Me6bKnlq47YG81vae/xqyXR9yDQ2Ni9vbetVF7qK4naa8bvql7vFnfus/9tavaM0/PnfSeWf/exBfMel1klLN26n03mG2TczvN+ueO22zWN5+Z+fLb2eDS3UTkxfATBYrhJwoUw08UKIafKFAMP1GgGH6iQI2ocf7EhWc4a2Wr15ptNz1oL839nxc/YNaveOJbztqM239vtk319Jh1qbBXOEqvmeCkCfc4vyY904F933/Pc0c8fbcb2+ce7e+368bXDQDv3XWOs/abq//ZbPudXZea9R1nd5v17gX2dSd1z73trCU77WsMLBznJyIvhp8oUAw/UaAYfqJAMfxEgWL4iQLF8BMFakSN80dnTHPWNl83wWw74w57LL5ulb0Uc9efu+e1a689d1v77HrK094rlfnS3iNZdJy9LXuqY7+ztvdq9zUjADD2nQNmfd+J1Wa97KCdq9oVr5n1THGcn4i8GH6iQDH8RIFi+IkCxfATBYrhJwoUw08UKO8W3SKyDMDlANpV9ZT0bfUAVgCYCqAVwAJV3Ze/bg5IbnGv4z7zIXt9+c232buJz7zuI7MuUfe4b++Z0822e26wx4yvmbHGrNeX2XPH+9X9bdyfdK9dDwDxlP0SaIh1mfXn98w261Vl7jn5MbH3O2issOe1//fSmWa97r24s9bwv21m2/bPNpr1mt32WgIV//W6WS8Fwznz/wzAJ1c2uBXAalWdCWB1+nMiGkG84VfVFwHs/cTN8wEsT3+8HMAVOe4XEeVZpu/5G1V1NwCk/7f3bCKikuN9z58tEVkEYBEAVKIq309HRMOU6Zm/TUQmAkD6/3bXHVV1iao2q2pzDFks9khEOZVp+FcCWJj+eCGAZ3LTHSIqFG/4ReQxAK8COEFEdojItQDuBnCxiGwGcHH6cyIaQUbUfP5s+OZ+Jz/80Ky33/AZZ+3RW+4x2z7deZpZ/3nrHLM+qtw9Xg0A3b3ut1OJpGdtfLWnfifiUbOe9Dw+jJdXWbm9DsG4Ovv6hh+e8JhZ//Iz33TWZtyUn/n0h/j2YvCt8ZApzucnIi+GnyhQDD9RoBh+okAx/ESBYviJAjWyhvo820Wbsvw6Ny1xb/FdObbXbDvlyg1ZPbdXxB6Os0jU0zZiH3Pf9uHWNtzeJcs9S5JHPn2SWb/zl8udte+cPM9+as+26lm9FoGsX48uHOojIi+GnyhQDD9RoBh+okAx/ESBYviJAsXwEwVqZI3zW/I97mo8fuKC082mB8fHzHqs217CurzDvfw1AJR1ucfLI532eLV22ktze/mOu3Vc68eYTQ8eX2/WE1X2NQq1b7uXY0++u8Vs68VxfiIaqRh+okAx/ESBYviJAsXwEwWK4ScKFMNPFKi8b9dVMHm+XiE63r30t/ba2zXXPr7WrJcdP9Wsd51qb4W47wT3Ntx9Y8eabeO1Zhka8RxX38rdRr3mfbttw3r7GoXyX7WY9d03updbH5/tOL9PAa+fyRTP/ESBYviJAsXwEwWK4ScKFMNPFCiGnyhQDD9RoLzj/CKyDMDlANpV9ZT0bbcDuA7AoX2tF6vqs/nqZE741rb3rBGfbGt31jbfMdVs++vHXjbrFWLXfavyWxt493q24O5V+9Hj1kA9gKi1BzeAmLjXKqiN2Me8yjNnfmmHvbX5by5411mznzkHslnnoECGc+b/GYBLh7j9XlWdk/5X2sEnosN4w6+qLwLYW4C+EFEBZfOe/0YRWS8iy0TEvoaUiEpOpuH/MYDpAOYA2A3g+647isgiEWkRkZY4PHuzEVHBZBR+VW1T1aSqpgA8CGCucd8lqtqsqs0xVGTaTyLKsYzCLyITB336RQBv5qY7RFQowxnqewzAPAANIrIDwG0A5onIHAAKoBXA9XnsIxHlgTf8qnrVEDcvzUNf8ipSab/l8O3HvvNW99zwez73sNn24v+42axPeNkeE65ttftWtn2Ps5byrcufzHLEO2pfJyDWeHe5vZ9BYtZks976N2YZ9762wll7YMaJdmPfdSFq77VQCuP4PrzCjyhQDD9RoBh+okAx/ESBYviJAsXwEwXq6NmiO8/kzFOdtch+zzbY23aa9ci4BrOebKgz66lR7hHbSJ+9rHikx97+GxH7/JAqt0eLNeZur1H7scv2HzTriNtf29a/mOCsHXfbK/Zj+5TolF1u0U1EXgw/UaAYfqJAMfxEgWL4iQLF8BMFiuEnCtTRs0V3nkVaP3DWUlPd48mAfzpxYvsO+8k9dTGmn4pn2izKy82y7zoQ8UwJtgac1TNOn0xYi5IDkQr7uB5321aznlcleh3AYDzzEwWK4ScKFMNPFCiGnyhQDD9RoBh+okAx/ESBOnrG+fM8rprq7HTWOmZNN9u23TDDrDdNsvdBnVK7z6xPr/7QWZsYc1+fAABjovZaBEnP+SHp2QLc0paw1yl4/6C9zsH6vRPN+vZd9c7arK+tNdtmrQTG8X145icKFMNPFCiGnyhQDD9RoBh+okAx/ESBYviJAuUd5xeRJgAPAZgAIAVgiareLyL1AFYAmAqgFcACVbUHpPNJPD/HNLutqKtWjXbWrml0bwUNACv3zDHrG9rt8ep9PaPMesv62c7a6Pfs8ebYAXur6VSZPY4fr/bUa9z1/jFmU/SNtfsm9X1m/Uuf+p2z9sL155htG376qv3cvnUQ+uy+lYLhnPkTAG5W1ZMAnA3g6yIyG8CtAFar6kwAq9OfE9EI4Q2/qu5W1TfSH3cB2AjgWADzASxP3205gCvy1Ukiyr0jes8vIlMBnAZgDYBGVd0NDPyAADA+150jovwZdvhFpAbALwDcpKruC90Pb7dIRFpEpCWO0n8fRBSKYYVfRGIYCP4jqvpU+uY2EZmYrk8E0D5UW1VdoqrNqtocg73gIhEVjjf8IiIAlgLYqKo/GFRaCWBh+uOFAJ7JffeIKF+GM6X3XABfBbBBRNalb1sM4G4AT4jItQC2AbgyP10sjEhVlVnf+OuZztq//aN7Si0ARGrtJaqbovYW3kh5poeWtzlLiZmT7YeucC/7DQCRfnuItGx/r1mXvfvNej69NflkZ62x0/6e+QaGvUN5I2Dpbm/4VfVluJdfvzC33SGiQuEVfkSBYviJAsXwEwWK4ScKFMNPFCiGnyhQR8/S3WpP//SRpklmfcJv3dtFR2fPsh/7oD0mrKPsKx+ly15e29riW/Z8ZLaNZjkenc1E6UhlpecO9rkpUueeZg0ASLlfE1qR55d+CYzj+/DMTxQohp8oUAw/UaAYfqJAMfxEgWL4iQLF8BMF6iga589uXDW56Q9mvWyCe/ntZI09Tt9+gb3VdHeT3XdtqjbrTePdc/Kbj9lmtq2J2tcg9KTsJao74vay4vuN+o4ue+3unv6YXe+1+9bX6f6+1K232zauN8tHBZ75iQLF8BMFiuEnChTDTxQohp8oUAw/UaAYfqJAjaxxfmvueZbj/IlVTWb94Vn/4qz1ep467lz5fEBPyv42bIrb2yC+sP8kZ63loylmW5+IeLb4jtgz+sdUHHTWzhrXarY9JnbArDfEusx6ubj3S/j8JVvNthdV32LWJ9/1iln3rVWQ6rX3OygEnvmJAsXwEwWK4ScKFMNPFCiGnyhQDD9RoBh+okB5x/lFpAnAQwAmAEgBWKKq94vI7QCuA3Boo/PFqvpsvjqab70J+1D8yV1/76w1vrTPbBvZY9fVM+ab7PDtce9uX1m7126a9Ky8b6x9DwCatOsdRq1zVI391H32fH6N2+2Rcn9tK+ZcZDadUG+vc3A0GM5FPgkAN6vqGyJSC2CtiKxK1+5V1Xvy1z0iyhdv+FV1N4Dd6Y+7RGQjgGPz3TEiyq8jes8vIlMBnAZgTfqmG0VkvYgsE5GxjjaLRKRFRFriOPp/lSIaKYYdfhGpAfALADepaieAHwOYDmAOBn4z+P5Q7VR1iao2q2pzDPZad0RUOMMKv4jEMBD8R1T1KQBQ1TZVTapqCsCDAObmr5tElGve8IuIAFgKYKOq/mDQ7RMH3e2LAN7MffeIKF+G89f+cwF8FcAGEVmXvm0xgKtEZA4ABdAK4Pq89HAQibqXqNaEe/omAESPqTfrdX/rfmwAqNvrXtpbJ9pLc/fMsafVSsqeNlt2wP7aNOKeMtxfbX+LkxX2dGPxjARGEp5lx43Dqp7twaP99jCiJD3bh1e6z21lPfYXVr7Hnk7s2xDe93osBcP5a//LwJAT0kfsmD4R8Qo/omAx/ESBYviJAsXwEwWK4ScKFMNPFKgRtXS3+qafWuL2uGvfpNFmPTF9yKkLAIDyjn6zbUVbj/3YdfZlz/1j7KmtiVGZ/wwv77CPSypmP3Yq5rtOwD0WP6rdvaw3AEhf3K7H7deDlrtf3l0z7e/3/qnu7zcA1Hu28B4J4/w88xMFiuEnChTDTxQohp8oUAw/UaAYfqJAMfxEgRLNcmvrI3oykQ8BvD/opgYAewrWgSNTqn0r1X4B7Fumctm341R13HDuWNDwH/bkIi2q2ly0DhhKtW+l2i+AfctUsfrGX/uJAsXwEwWq2OFfUuTnt5Rq30q1XwD7lqmi9K2o7/mJqHiKfeYnoiIpSvhF5FIReVdEtojIrcXog4uItIrIBhFZJyItRe7LMhFpF5E3B91WLyKrRGRz+n977mlh+3a7iOxMH7t1IvKFIvWtSUT+R0Q2ishbIvLN9O1FPXZGv4py3Ar+a7+IRAFsAnAxgB0AXgdwlaq+XdCOOIhIK4BmVS36mLCInA+gG8BDqnpK+rZ/ArBXVe9O/+Acq6r/UCJ9ux1Ad7F3bk5vKDNx8M7SAK4AcA2KeOyMfi1AEY5bMc78cwFsUdWtqtoP4HEA84vQj5Knqi8C2PuJm+cDWJ7+eDkGXjwF5+hbSVDV3ar6RvrjLgCHdpYu6rEz+lUUxQj/sQC2D/p8B0pry28F8LyIrBWRRcXuzBAa09umH9o+fXyR+/NJ3p2bC+kTO0uXzLHLZMfrXCtG+Ida96mUhhzOVdXTAVwG4OvpX29peIa1c3OhDLGzdEnIdMfrXCtG+HcAaBr0+WQAu4rQjyGp6q70/+0Ankbp7T7cdmiT1PT/7UXuzx+V0s7NQ+0sjRI4dqW043Uxwv86gJkiMk1EygF8BcDKIvTjMCJSnf5DDESkGsAlKL3dh1cCWJj+eCGAZ4rYl48plZ2bXTtLo8jHrtR2vC7KRT7poYz7AEQBLFPV7xW8E0MQkeMxcLYHBlY2frSYfRORxwDMw8CsrzYAtwH4JYAnAEwBsA3Alapa8D+8Ofo2DwO/uv5x5+ZD77EL3LfzALwEYAP+f0PdxRh4f120Y2f06yoU4bjxCj+iQPEKP6JAMfxEgWL4iQLF8BMFiuEnChTDTxQohp8oUAw/UaD+D2IFhytnYRodAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x.numpy()[2].reshape(28,28))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1279e2eb8>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEHtJREFUeJzt3X9snfV1x/HP8bWdXyQQSAgmZIOElJZlgzI3UNimMEQLHVLo2qKmqMqmaqmqIhWNTUNoUtkfSGxaYWjqmNKSNWz8qlYYaGIFFKgoDAImY/wYtDCWQkhIAiE/SGI7ts/+8E1lgp/zXHx/0vN+SZHte+7j5/g6H19fn+d5vubuApBPV7sbANAehB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFLdrdxZr03z6ZrVyl0CqQxqv4Z9yGq5b13hN7OLJN0kqSLp++5+fXT/6Zqls+2CenYJILDRN9R83yn/2m9mFUnflXSxpNMlrTKz06f6+QC0Vj2v+ZdLetXdX3P3YUl3SlrZmLYANFs94V8o6Y0JH2+p3vY+ZrbGzAbMbOCQhurYHYBGqif8k/1R4QPnB7v7Wnfvd/f+Hk2rY3cAGqme8G+RtGjCxydJ2lpfOwBapZ7wPy1pqZmdYma9kr4s6b7GtAWg2aY86nP3ETO7QtIDGh/1rXP3FxvWGYCmqmvO7+73S7q/Qb0AaCEO7wWSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiCpulbpNbPNkvZJGpU04u79jWgKQPPVFf6q89397QZ8HgAtxK/9QFL1ht8lPWhmz5jZmkY0BKA16v21/zx332pmx0t6yMxedvdHJ96h+kNhjSRN18w6dwegUep65nf3rdW3OyTdI2n5JPdZ6+797t7fo2n17A5AA005/GY2y8xmH35f0mckvdCoxgA0Vz2/9i+QdI+ZHf48t7v7jxvSFYCmm3L43f01SWc0sBcALcSoD0iK8ANJEX4gKcIPJEX4gaQIP5BUI87qQ5t1zSw+bHrswIGm7nv0/LPCeuWRTU3df9OMH79SzL01fTQRz/xAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBRz/l8BzZzlD382vhr7I//0/bB+6m3fKKwt+fMn4p13VeL62Ghcr0edc/zKaafGd9i+s7A0untPXfuuFc/8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AUc/5fAduuOrew9tnL41n63T85O6wfd9o7Yf3xwbGwPue0XWE91Mw5fp0O/GH8uK294caw/rn/uLKw9rFvPDWlnj4snvmBpAg/kBThB5Ii/EBShB9IivADSRF+IKnSOb+ZrZN0iaQd7r6setuxku6SdLKkzZIuc/d3m9fmR5t1xw+zj4zU9fmfu+ofCmtPDR0Kt919XvE1/yVpfu++sP70wcVhfd/LxxZ/7nDL9upefHJYv+SvHg7rBzz+ni9aXHw+f6vU8sz/A0kXHXHb1ZI2uPtSSRuqHwP4CCkNv7s/KunIw7RWSlpffX+9pEsb3BeAJpvqa/4F7r5Nkqpvj29cSwBaoenH9pvZGklrJGm64teXAFpnqs/8282sT5Kqb3cU3dHd17p7v7v392jaFHcHoNGmGv77JK2uvr9a0r2NaQdAq5SG38zukPSEpNPMbIuZfU3S9ZIuNLNXJF1Y/RjAR0jpa353X1VQuqDBvZRfpz1Sdu53yXrrVon37aPB5y+5xnu9c/yhiz8V1h8f3FRYe2z/6eG2p84sfMUmSXpjsHhOL0mvH4zrA1+5obD2wKUnhNv+5aaVYX10W/w3JA+e2uy4oXDbB3/378P6K4eOC+tPHlwS1td//J8La19Z9WfhtnPueDKs14oj/ICkCD+QFOEHkiL8QFKEH0iK8ANJtf7S3dHIrY1LLtc7jotUPrE0rP9sTTw2+uKKeLRzKDh9dGbXcLjtKwfj0zJmVOLtT5v5Vlj/x3d/q7C2bMYb4bYPn1t8qrIk9VXiUd+Yir/n743Fo76NQ/H3ZNjj0fBvTHszrJ/UPaOw9vFvvRhuu/WOsFwznvmBpAg/kBThB5Ii/EBShB9IivADSRF+IKnWz/lL5u1Ns/w3w/LBE4vnrpL01tnFc90vXvR4uO3lc28N62+MHBPWf7wn7v26zX9QWPtCX/HpvpI0rSs+vqGvd09YL3NUZbCw9s7IUeG2/zoUn/J7YHTqV4aa1xNfkvy3p28O6/NLjn8Y9PgU8ieGZhXWtu4/OtxWinuvFc/8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5BUS+f8I/Nm6e0vfLqw3nNpvGzxSbN3F9Zm98TnZ8/rfSGs/+m8n4b17aM9hbV79p4VbvvdneeH9W6Lr2OwdEZ8ee23ZxTPyxf1vhNuO70rXsK7TJfGwvqe0eJ5dsXiYz7md8fz7Ep3fAzC7MrBwtrCSrxtj8Vf167g/4Mk/fu+M8L6z/cXX0dhy+54zn+StoT1WvHMDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJlc75zWydpEsk7XD3ZdXbrpX0J5IOD+avcff7yz5X73FDWnT5a4X1tYvvDre/eVfxUtV7R6aH25bV/3rnirB+MJjrlp0Tf2rJnH7zYHyN+MHguvyStGRW8fERD++Jl+heOmN7WN9Vcs79gp54Xr5tuHhmPb97b7jtMZUDYX12V/EcX5J6So6fiByK1veWtN/jOX/Zvs+a83ph7ZLj/jvc9hadEtZrVcsz/w8kXTTJ7Te6+5nVf6XBB9BZSsPv7o9K2tWCXgC0UD2v+a8ws+fMbJ2ZzW1YRwBaYqrhv1nSEklnStom6TtFdzSzNWY2YGYDw7vj12gAWmdK4Xf37e4+6u5jkr4naXlw37Xu3u/u/b3HxBfJBNA6Uwq/mfVN+PDzkuJT5gB0nFpGfXdIWiFpnpltkfRtSSvM7ExJLmmzpK83sUcATWDewuvoHz1tgZ974uWF9Zevmxduf96S/y2s/f7cl8Nty2bKZdeQ3zVaXH93pPicdUkaHItnwrsPxS+Hhsfin9Hdwbnn58wpfswk6VDJOvOfmvF/YX1xd3yMw9zKzLDeqZ4cjOf0t+86J6yXfU8ff3VJYW3+A/F6BEf/y5OFtY2+QXt9V7xoQBVH+AFJEX4gKcIPJEX4gaQIP5AU4QeSaumob44d62fbBU353Na/LKy/ef6csO6fjk9NvemMOwtrK6bHl7+uWPwz9qmhePvekstjTwtOHz2mK962rzsecT4zFC9Fvfq//jis9z5U/Lgf/1Q8fvVnXgzr6orHlNZTPCK17nh8OrZ/f7zvNuruK166/D933qU9wzsY9QEoRviBpAg/kBThB5Ii/EBShB9IivADSX205vwWjC9b+HUcqTInPoZg8JyPhfW3l/WG9QMnxrP63neLf4afsDFeurz3J/Flon0kPmW3qUrm+PL4camHVeJ9Vxb2hXV1xc+rfiC4pN28kktiDhcfF/LEL9Zrz+BbzPkBFCP8QFKEH0iK8ANJEX4gKcIPJEX4gaRKr9vfUaJZfslMuGtGvES39cazdh8uPq99dG98XnrPgwNhve/BsNxUXSXHKNjM+BLUYyUz6bGZxZct7zoQX8fAgnn2+B3icbYdmvoxCt5VMiovuQaDSq7BoKNnf6h+JrL3gqXLx2o/9oFnfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IqnTOb2aLJN0q6QSNDy/XuvtNZnaspLsknSxps6TL3P3d5rWqeK47Fi+pXHod9jqu027T4iWVu2bH18ZXybnjmh0vAR7Ns/3AYLxtpeTnf0lvtmVbXD8Y7L9kTq+jSr7unnjp8/C4kLLjPoZKjhEovX5Eyde2c1dhaey9+P+iHyo+5sS99mMbannmH5F0lbt/QtI5kr5pZqdLulrSBndfKmlD9WMAHxGl4Xf3be6+qfr+PkkvSVooaaWk9dW7rZd0abOaBNB4H+o1v5mdLOmTkjZKWuDu26TxHxCSjm90cwCap+bwm9lRkn4k6Up3jw9mf/92a8xswMwGDim+nhyA1qkp/GbWo/Hg3+bud1dv3m5mfdV6n6Qdk23r7mvdvd/d+3sU/2EMQOuUht/MTNItkl5y9xsmlO6TtLr6/mpJ9za+PQDNUsspvedJ+qqk583s2ept10i6XtIPzexrkl6X9KXmtDhBGy/PHfGh+OXMaEm91Pb6Nu9Upd/NwXhMifqUht/dH1Px0LKOi/ADaCeO8AOSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kVRp+M1tkZo+Y2Utm9qKZfat6+7Vm9qaZPVv997nmtwugUbpruM+IpKvcfZOZzZb0jJk9VK3d6O5/27z2ADRLafjdfZukbdX395nZS5IWNrsxAM31oV7zm9nJkj4paWP1pivM7DkzW2dmcwu2WWNmA2Y2cEhDdTULoHFqDr+ZHSXpR5KudPe9km6WtETSmRr/zeA7k23n7mvdvd/d+3s0rQEtA2iEmsJvZj0aD/5t7n63JLn7dncfdfcxSd+TtLx5bQJotFr+2m+SbpH0krvfMOH2vgl3+7ykFxrfHoBmqeWv/edJ+qqk583s2ept10haZWZnSnJJmyV9vSkdAmiKWv7a/5gkm6R0f+PbAdAqHOEHJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9Iyty9dTsz2ynpFxNumifp7ZY18OF0am+d2pdEb1PVyN5+3d3n13LHlob/Azs3G3D3/rY1EOjU3jq1L4nepqpdvfFrP5AU4QeSanf417Z5/5FO7a1T+5Lobara0ltbX/MDaJ92P/MDaJO2hN/MLjKzn5nZq2Z2dTt6KGJmm83s+erKwwNt7mWdme0wsxcm3HasmT1kZq9U3066TFqbeuuIlZuDlaXb+th12orXLf+138wqkn4u6UJJWyQ9LWmVu/9PSxspYGabJfW7e9tnwmb2e5Lek3Sruy+r3vY3kna5+/XVH5xz3f0vOqS3ayW91+6Vm6sLyvRNXFla0qWS/khtfOyCvi5TGx63djzzL5f0qru/5u7Dku6UtLINfXQ8d39U0q4jbl4paX31/fUa/8/TcgW9dQR33+bum6rv75N0eGXptj52QV9t0Y7wL5T0xoSPt6izlvx2SQ+a2TNmtqbdzUxiQXXZ9MPLpx/f5n6OVLpycysdsbJ0xzx2U1nxutHaEf7JVv/ppJHDee5+lqSLJX2z+ustalPTys2tMsnK0h1hqiteN1o7wr9F0qIJH58kaWsb+piUu2+tvt0h6R513urD2w8vklp9u6PN/fxSJ63cPNnK0uqAx66TVrxuR/iflrTUzE4xs15JX5Z0Xxv6+AAzm1X9Q4zMbJakz6jzVh++T9Lq6vurJd3bxl7ep1NWbi5aWVptfuw6bcXrthzkUx1l/J2kiqR17n5dy5uYhJkt1vizvTS+iOnt7ezNzO6QtELjZ31tl/RtSf8m6YeSfk3S65K+5O4t/8NbQW8rNP6r6y9Xbj78GrvFvf2OpJ9Kel7SWPXmazT++rptj13Q1yq14XHjCD8gKY7wA5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+Q1P8D+2KwIGyherAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x.numpy()[3].reshape(28,28))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLP (from previews session)\n",
    "\n",
    "Starting what you are familiared with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "from torch.nn import functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp_01 = nn.Sequential(*[\n",
    "                        nn.Linear(28*28,512,bias=True),\n",
    "                        nn.ReLU(),\n",
    "                        nn.Linear(512,512,bias=True),\n",
    "                        nn.ReLU(),\n",
    "                        nn.Linear(512,10,bias=True),\n",
    "                        nn.Softmax(dim=-1),\n",
    "                        ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.optim import Adam\n",
    "from torch import argmax\n",
    "optimizer = Adam(mlp_01.parameters(),lr = 0.001)\n",
    "loss_func = nn.CrossEntropyLoss()\n",
    "def accuracy(y_pred, y_true):\n",
    "    \"\"\"\n",
    "    :param y_pred: predition of y (will be argmaxed)\n",
    "    :param y_true: true label of y (index)\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    return (argmax(y_pred,dim=-1) == y_true).float().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tnrange,tqdm_notebook,tqdm\n",
    "import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e39c5797d35c43a68469c6a32b7ccc30",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=5), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d4996c0a45c4beca71181d41a8bdd38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "93b3d7691521438ba15f560d7974ce52",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1ee923cf35b047f99761933fd8bcf2aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dbab5a9ba1684199ac46d3f6821d17b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3eac5da0c7ce44a5be400202726072c4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dl = DataLoader(ds,batch_size=64,shuffle=True)\n",
    "for j in tnrange(5):\n",
    "    # make a generator out of dataloader\n",
    "    gen = iter(dl)\n",
    "    t =tqdm_notebook(range(len(dl)))\n",
    "    loss_ = 0\n",
    "    acc = 0\n",
    "    for i in t:\n",
    "        # get input data:x, supervising label:y\n",
    "        x,y = next(gen)\n",
    "        \n",
    "        # flatten x, just like keras's x = Flatten()(x)\n",
    "        bs = x.size()[0]\n",
    "        x = x.resize_(bs,28*28)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # forward path\n",
    "        y_ = mlp_01(x)\n",
    "        # calculate loss\n",
    "        loss = loss_func(y_,y)\n",
    "        \n",
    "        # back propagation (from loss to weights)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        # Calculate metric: accuracy just to see how the model is in a percentage sense\n",
    "        loss_ += loss.item()\n",
    "        acc += accuracy(y_,y).item()\n",
    "        \n",
    "        if i%10==9:\n",
    "            acc /= 10; loss_ /= 10\n",
    "            t.set_postfix({\"😎accuracy\":acc,\"😎loss\":loss_,})\n",
    "            acc = 0; loss_ = 0\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 325,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(4)"
      ]
     },
     "execution_count": 325,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred.argmax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steps with in a learning step\n",
    "\n",
    "Assume the model is $\\large y=f(x)$, This represent the entire model, whether it is LR, DNN, CNN, RNN. The parameters of the data is $\\large \\theta$. We want to feed $m$ rows of data, $\\large {x}_{m}, {y}_{m}$, pair by pair into the model, to change/optimize $\\large \\theta$.\n",
    "\n",
    "* Get the x, y data from generator, batch_size is number of rows of data/batch\n",
    "\n",
    "* Clean the gradients\n",
    "\n",
    "* Forward pass, predict a $\\large \\hat{y}$ using $\\large \\hat{y_{m}}=f(x_{m})$, it's telling the model to take a guess given $\\large x_{m}$, $\\large m$ means the current batch index\n",
    "\n",
    "* Calculate loss, comparing the predict label, with supervising/guide lining/correct label, $\\large L = CrossEntropy(\\hat{y},y)$\n",
    "\n",
    "* Using backward() function on loss to calculate gradients $\\large \\frac{\\partial L}{\\partial \\theta}$, to measure in which way $\\large \\theta$'s change can effect $\\large L$. \n",
    "\n",
    "* optimizer.step() to change $\\large \\theta$ in the direction for a tiny amount (learning rate $\\large \\alpha$)\n",
    "\n",
    "Machine learning is to repeatedly do the above step. It's basicly what pytorch do during a batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next Improvement\n",
    "\n",
    "* Make use of valid dataset\n",
    "* Wrapper for DL\n",
    "* Simple Convolution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the following and install the ray library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import os\n",
    "# os.system(\"cd %s/lib/python3*/site-packages/;git clone https://github.com/raynardj/ray\"%(os.environ[\"_\"].split(\"/bin\")[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ray import matchbox"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = matchbox.Trainer(dataset=ds,val_dataset=ds_val,batch_size=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv_block(in_,out_,ks=3):\n",
    "    \"\"\"\n",
    "    return one convolutional block\n",
    "    \"\"\"\n",
    "    return nn.Sequential(*[\n",
    "        nn.Conv2d(in_channels=in_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=True),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=out_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=True),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=out_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=True),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "    ])\n",
    "\n",
    "class Flatten(nn.Module):\n",
    "    def forward(self, input):\n",
    "        \"\"\"\n",
    "        a pytorch version of Flatten layer\n",
    "        \"\"\"\n",
    "        return input.view(input.size(0), -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A sequential CNN(Convolution Neural Network) model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnn_01 = nn.Sequential(*[\n",
    "                        # The shape is (batch size, number of channels, height, width)\n",
    "                        conv_block(1,16), # shape in: (bs,1,28,28), out:(bs,16,14,14), grid size /2 because of max pooling\n",
    "                        conv_block(16,32), # shape in: (bs,16,3,3), out:(bs,32,7,7),\n",
    "                        conv_block(32,64), # shape in: (bs,32,7,7), out:(bs,64,3,3),\n",
    "                        Flatten(), # shape in:(bs,64,3,3), out: (bs,64*3*3), dimension reduced\n",
    "                        # The following are stardard MLP\n",
    "                        nn.Linear(64*3*3,256,bias=False),\n",
    "                        nn.BatchNorm1d(256),\n",
    "                        nn.ReLU(),\n",
    "                        nn.Dropout(.3),\n",
    "                        nn.Linear(256,10),\n",
    "                        nn.BatchNorm1d(10),\n",
    "                        nn.Softmax(dim=-1),\n",
    "                        ])\n",
    "optimizer = Adam(cnn_01.parameters(),lr = 0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [],
   "source": [
    "def action(*args,**kwargs):\n",
    "    \"\"\"\n",
    "    Define each training step\n",
    "    \"\"\"\n",
    "    x,y = args[0]\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    y_ = cnn_01(x)\n",
    "    \n",
    "    loss = loss_func(y_,y)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    return {\n",
    "        \"loss\":loss.item(),\n",
    "        \"acc\":accuracy(y_,y).item(),\n",
    "           }\n",
    "def val_action(*args,**kwargs):\n",
    "    \"\"\"\n",
    "    Define each validation step\n",
    "    \"\"\"\n",
    "    x,y = args[0]\n",
    "    y_ = cnn_01(x)\n",
    "    \n",
    "    loss = loss_func(y_,y)\n",
    "    return {\n",
    "        \"loss\":loss.item(),\n",
    "        \"acc\":accuracy(y_,y).item(),\n",
    "           }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4668e7a923074183b334b1ec8fcd546d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4759b16c1b3e4c1f9611a2e90f9ace5f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=157), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3852b664e814ec88d88ad435fb768cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "65d2755054b54d0eb8d09fd92297c988",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=157), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer.action = action\n",
    "trainer.val_action = val_action\n",
    "trainer.train(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excersice No 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a model using conv block with Batch Normalization inside, here is the code to retuan a batchnormalzied conv block. you can use it for building model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv_block_bn(in_,out_,ks=3):\n",
    "    \"\"\"\n",
    "    return one convolutional block\n",
    "    \"\"\"\n",
    "    return nn.Sequential(*[\n",
    "        nn.Conv2d(in_channels=in_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False), \n",
    "        # changed bias to False, Batchnorm has param size just like previous layer's bias, sort of perform alike also\n",
    "        nn.BatchNorm2d(out_),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=out_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False),\n",
    "        nn.BatchNorm2d(out_),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=out_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False),\n",
    "        nn.BatchNorm2d(out_),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coord Conv Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use objective oriented programming to for the coord conv, it's a technique to blend in the positional information(coordinations) in the convolution layer. Check the [coord conv paper](http://arxiv.org/abs/1807.03247).\n",
    "\n",
    "Or check their [awesome blog post](https://eng.uber.com/coordconv/)\n",
    "\n",
    "![coord_conv pic](docs/coord_conv.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [],
   "source": [
    "class add_coord(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(add_coord,self).__init__()\n",
    "        \n",
    "    def forward(self,x):\n",
    "        bs,ch,h,w = x.size()\n",
    "        h_coord = torch.range(start = 0,end = h-1).unsqueeze(0).unsqueeze(0).unsqueeze(-1).repeat([bs,1,1,w])/(h/2)-1\n",
    "        w_coord = torch.range(start = 0,end = w-1).unsqueeze(0).unsqueeze(0).unsqueeze(0).repeat([bs,1,h,1])/(w/2)-1\n",
    "        return torch.cat([x,h_coord,w_coord],dim=1)\n",
    "    \n",
    "class Coord2d(nn.Module):\n",
    "    def __init__(self,in_channels,out_channels,kernel_size,stride=1,padding=0,dilation=1,groups=1,bias=True):\n",
    "        \"\"\"\n",
    "        Coord Convolution Module\n",
    "        Coord2d(in_channels,out_channels,kernel_size,stride=1,padding=0,dilation=1,groups=1,bias=True)\n",
    "        Use it just like using a normal pytorch nn.Module\n",
    "        \"\"\"\n",
    "        super(Coord2d,self).__init__()\n",
    "        self.add_coord = add_coord()\n",
    "        self.conv = nn.Conv2d(in_channels=in_channels+2,\n",
    "                              out_channels=out_channels,\n",
    "                              kernel_size=kernel_size,stride=stride,padding=padding,dilation=dilation,groups=groups,bias=bias)\n",
    "        \n",
    "    def forward(self,x):\n",
    "        x = self.add_coord(x)\n",
    "        for i in range(5):\n",
    "            x = self.ln1(x)\n",
    "        x = self.conv(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have wrapped up the above Coord2d code into my personal tool library: [ray](https://github.com/raynardj/ray)\n",
    "\n",
    "From now on, you can from ray.matchbox_lego import Coord2d to use Coord2d module without writing the structure again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coord Conv Explained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 328,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.rand(64,3,28,28)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs,ch,h,w = x.size()\n",
    "h_coord = torch.range(start = 0,end = h-1).unsqueeze(0).unsqueeze(0).unsqueeze(-1).repeat([bs,1,1,w])/(h/2)-1\n",
    "w_coord = torch.range(start = 0,end = w-1).unsqueeze(0).unsqueeze(0).unsqueeze(0).repeat([bs,1,h,1])/(w/2)-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([64, 1, 28, 28])"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h_coord.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 345,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([64, 5, 28, 28])"
      ]
     },
     "execution_count": 345,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cat([x,h_coord,w_coord],dim=1).size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [],
   "source": [
    "def coord_block(in_,out_,ks=3):\n",
    "    \"\"\"\n",
    "    return one convolutional block\n",
    "    \"\"\"\n",
    "    return nn.Sequential(*[\n",
    "#         nn.Conv2d(in_channels=in_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False),\n",
    "        Coord2d(in_channels=in_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False), \n",
    "        nn.BatchNorm2d(out_),\n",
    "        nn.ReLU(),\n",
    "#         nn.Conv2d(in_channels=out_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False),\n",
    "        Coord2d(in_channels=out_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False),\n",
    "        nn.BatchNorm2d(out_),\n",
    "        nn.ReLU(),\n",
    "#         nn.Conv2d(in_channels=out_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False),\n",
    "        Coord2d(in_channels=out_,out_channels=out_,kernel_size=ks,padding=ks//2,bias=False),\n",
    "        nn.BatchNorm2d(out_),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new model with coord conv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnn_coord = nn.Sequential(*[\n",
    "                        # The shape is (batch size, number of channels, height, width)\n",
    "                        coord_block(1,16), # shape in: (bs,1,28,28), out:(bs,16,14,14), grid size /2 because of max pooling\n",
    "                        coord_block(16,32), # shape in: (bs,16,3,3), out:(bs,32,7,7),\n",
    "                        coord_block(32,64), # shape in: (bs,32,7,7), out:(bs,64,3,3),\n",
    "                        Flatten(), # shape in:(bs,64,3,3), out: (bs,64*3*3), dimension reduced\n",
    "                        # The following are stardard MLP\n",
    "                        nn.Linear(64*3*3,256,bias=False),\n",
    "                        nn.BatchNorm1d(256),\n",
    "                        nn.ReLU(),\n",
    "                        nn.Dropout(.3),\n",
    "                        nn.Linear(256,10),\n",
    "                        nn.BatchNorm1d(10),\n",
    "                        nn.Softmax(dim=-1),\n",
    "                        ])\n",
    "optimizer = Adam(cnn_coord.parameters(),lr = 0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer_coord = matchbox.Trainer(dataset=ds,val_dataset=ds_val,batch_size=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [],
   "source": [
    "def action(*args,**kwargs):\n",
    "    \"\"\"\n",
    "    Define each training step\n",
    "    \"\"\"\n",
    "    x,y = args[0]\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    y_ = cnn_coord(x)\n",
    "    \n",
    "    loss = loss_func(y_,y)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    return {\n",
    "        \"loss\":loss.item(),\n",
    "        \"acc\":accuracy(y_,y).item(),\n",
    "           }\n",
    "def val_action(*args,**kwargs):\n",
    "    \"\"\"\n",
    "    Define each validation step\n",
    "    \"\"\"\n",
    "    x,y = args[0]\n",
    "    y_ = cnn_coord(x)\n",
    "    \n",
    "    loss = loss_func(y_,y)\n",
    "    return {\n",
    "        \"loss\":loss.item(),\n",
    "        \"acc\":accuracy(y_,y).item(),\n",
    "           }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d93bb21816e470ab4eab6dd9d7a2bd0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b23bc4f07044babbfedac9d0b89df4c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=157), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "718a43ed2fa948f49c691a2c10c141fa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=938), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dfc385b7be214bf489a3f14efa8649d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=157), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer_coord.action = action\n",
    "trainer_coord.val_action = val_action\n",
    "trainer_coord.train(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excercise\n",
    "\n",
    "* Redo the course with your own code.\n",
    "* Try change the filter numbers, batch size, model structure, to achieve better result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}