{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional neural networks\n",
    "\n",
    "In the previous unit we have learned how to define a multi-layered neural network using class definition, but those networks were generic, and not specialized for computer vision tasks. In this unit we will learn about **Convolutional Neural Networks (CNNs)**, which are specifically designed for computer vision.\n",
    "\n",
    "Computer vision is different from generic classification, because when we are trying to find a certain object in the picture, we are scanning the image looking for some specific **patterns** and their combinations. For example, when looking for a cat, we first may look for horizontal lines, which can form whiskers, and then certain combination of whiskers can tell us that it is actually a picture of a cat. Relative position and presence of certain patterns is important, and not their exact position on the image. \n",
    "\n",
    "To extract patterns, we will use the notion of **convolutional filters**. But first, let us load all dependencies and functions that we have defined in the previous units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "import matplotlib.pyplot as plt\n",
    "from torchinfo import summary\n",
    "import numpy as np\n",
    "\n",
    "from pytorchcv import load_mnist, train, plot_results, plot_convolution, display_dataset\n",
    "load_mnist(batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional filters\n",
    "\n",
    "Convolutional filters are small windows that run over each pixel of the image and compute weighted average of the neighboring pixels.\n",
    "\n",
    "\n",
    "\n",
    "They are defined by matrices of weight coefficients. Let's see the examples of applying two different convolutional filters over our MNIST handwritten digits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43332 characters]\n",
      "text/plain": [
       "<Figure size 576x216 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 44688 characters]\n",
      "text/plain": [
       "<Figure size 576x216 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_convolution(torch.tensor([[-1.,0.,1.],[-1.,0.,1.],[-1.,0.,1.]]),'Vertical edge filter')\n",
    "plot_convolution(torch.tensor([[-1.,-1.,-1.],[0.,0.,0.],[1.,1.,1.]]),'Horizontal edge filter')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First filter is called a **vertical edge filter**, and it is defined by the following matrix:\n",
    "$$\n",
    "\\left(\n",
    "    \\begin{matrix}\n",
    "     -1 & 0 & 1 \\cr\n",
    "     -1 & 0 & 1 \\cr\n",
    "     -1 & 0 & 1 \\cr\n",
    "    \\end{matrix}\n",
    "\\right)\n",
    "$$\n",
    "When this filter goes over relatively uniform pixel field, all values add up to 0. However, when it encounters a vertical edge in the image, high spike value is generated. That's why in the images above you can see vertical edges represented by high and low values, while horizontal edges are averaged out.\n",
    "\n",
    "An opposite thing happens when we apply horizontal edge filter - horizontal lines are amplified, and vertical are averaged out.\n",
    "\n",
    "In classical computer vision, multiple filters were applied to the image to generate features, which then were used by machine learning algorithm to build a classifier. However, in deep learning we construct networks that **learn** best convolutional filters to solve classification problem.\n",
    "\n",
    "To do that, we introduce **convolutional layers**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Covolutional layers\n",
    "\n",
    "Convolutional layers are defined using `nn.Conv2d` construction. We need to specify the following:\n",
    "* `in_channels` - number of input channels. In our case we are dealing with a grayscale image, thus number of input channels is 1.\n",
    "* `out_channels` - number of filters to use. We will use 9 different filters, which will give the network plenty of opportunities to explore which filters work best for our scenario.\n",
    "* `kernel_size` is the size of the sliding window. Usually 3x3 or 5x5 filters are used.\n",
    "\n",
    "Simplest CNN will contain one convolutional layer. Given the input size 28x28, after applying nine 5x5 filters we will end up with a tensor of 9x24x24 (the spatial size is smaller, because there are only 24 positions where a sliding interval of length 5 can fit into 28 pixels).\n",
    "\n",
    "After convolution, we flatten 9x24x24 tensor into one vector of size 5184, and then add linear layer, to produce 10 classes. We also use `relu` activation function in between layers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "==========================================================================================\n",
       "Layer (type:depth-idx)                   Output Shape              Param #\n",
       "==========================================================================================\n",
       "├─Conv2d: 1-1                            [1, 9, 24, 24]            234\n",
       "├─Flatten: 1-2                           [1, 5184]                 --\n",
       "├─Linear: 1-3                            [1, 10]                   51,850\n",
       "==========================================================================================\n",
       "Total params: 52,084\n",
       "Trainable params: 52,084\n",
       "Non-trainable params: 0\n",
       "Total mult-adds (M): 0.18\n",
       "==========================================================================================\n",
       "Input size (MB): 0.00\n",
       "Forward/backward pass size (MB): 0.04\n",
       "Params size (MB): 0.21\n",
       "Estimated Total Size (MB): 0.25\n",
       "=========================================================================================="
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class OneConv(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(OneConv, self).__init__()\n",
    "        self.conv = nn.Conv2d(in_channels=1,out_channels=9,kernel_size=(5,5))\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.fc = nn.Linear(5184,10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = nn.functional.relu(self.conv(x))\n",
    "        x = self.flatten(x)\n",
    "        x = nn.functional.log_softmax(self.fc(x),dim=1)\n",
    "        return x\n",
    "\n",
    "net = OneConv()\n",
    "\n",
    "summary(net,input_size=(1,1,28,28))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that this network contains around 50k trainable parameters, compared to around 80k in fully-connected multi-layered networks. This allows us to achieve good results even on smaller datasets, because convolutional networks generalize much better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  0, Train acc=0.947, Val acc=0.969, Train loss=0.001, Val loss=0.001\n",
      "Epoch  1, Train acc=0.979, Val acc=0.975, Train loss=0.001, Val loss=0.001\n",
      "Epoch  2, Train acc=0.985, Val acc=0.977, Train loss=0.000, Val loss=0.001\n",
      "Epoch  3, Train acc=0.988, Val acc=0.975, Train loss=0.000, Val loss=0.001\n",
      "Epoch  4, Train acc=0.988, Val acc=0.976, Train loss=0.000, Val loss=0.001\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 39324 characters]\n",
      "text/plain": [
       "<Figure size 1080x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "hist = train(net,train_loader,test_loader,epochs=5)\n",
    "plot_results(hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we are able to achieve higher accuracy, and much faster, compared to the fully-connected networks from previous unit.\n",
    "\n",
    "We can also visualize the weights of our trained convolutional layers, to try and make some more sense of what is going on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAAtCAYAAAAN3bjCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAF+ElEQVR4nO3dX0zVZRgH8OccOXnQg6QgRoAczsHjzEGpmJn/pqQsHcEqK1MalY6i5bSiP6hbtelaKXrjRWJb86JlwLQ/pomLdMJUdKwCSQkQFTBBICAQDofTVVd8H8avi7eLvp/L74+H33vOgWe/7Tx7X1swGBQiIjLD/l8vgIjo/4RNl4jIIDZdIiKD2HSJiAxi0yUiMihktIudLbFwtCHj8jq1pvVCNMyHJuhTEtHl+FpF0Vs2lGeezYUF7f0u9R43GqfCPLRZfwsGfP0wb1yfD9eVunwXXNe1HP21ly/ZD/MFpzarNU8mV8F875zDcF2+4g/hAtyRHeo9rtbGwjwq4Y5a82BkM8wLUw7Bda2a/z5c11/T9c8x7GwDzIM9PWqNzTke5ic6DsJ19bbEw3V91YvfExGRxgH891X5lE+tCfzeCPPS4SK4rryf18J1+Zy31HsUtc6D+dVq/bXM2Hze0ro8ewrgugJhAfUe25d9A/Od59eoNa5f8OdYvXvriHWttOP36mrhfPX3L5pdB/OuwVC1pqYhBuZN2e/A90qET7pEREax6RIRGcSmS0RkEJsuEZFBbLpERAaNOr3w9JVnYJ4Vd06t+cL+MMxtOyPVmi4v/lZSM31iJ8x7dujfyIYuxi/1xXU/qDVh9rvKlXyYOmquw9xZNVO9R1pFHr73PWqJ1HvwN+WaqPBemH/sKVFrcgbWW7qHiMhrUWWWft5+sw3mQzPD1Job2TNgPnivPiES8au1/UWSD+PJEd+neL0iIq0rp8E85iCeUBARkScmWVrX13VJMI+LiFNr9niLYN7ndqg12Z2vW1pX4pfdMG9Ya+31iYjM9Tap124f9Yz599zZtBDmuQtPqjWrXdUw31i7Qa2x9Y8b85r+wSddIiKD2HSJiAxi0yUiMohNl4jIIDZdIiKD2HSJiAwadWQsL/4EzFdN8Ks1U0NOwXx/mb7xhyTi8Q7NvuiLME87PaTWjFvwKMydNr2m/m6UpXUF2vFmMHFHWtWa/kQ8Shf+Hh4/ExF5YJL++5DQEPx5bazJUmsuzMGjRvs63WpNRikeNWp6Gf/891V4fOeFpqXqPWKdXTA/WrxYrel2q3uPQKEePAJV++YUtcZVj/OChGK1ZktMtpVlyWDrRHwhAr8nIiIe5T/8sn9YrRkeb23ELuDC840xZXqfuP54BMz7hvRZyd77xj6eVbT9E5gnOPTNlLw/5sI8+oi+JsdD1p9b+aRLRGQQmy4RkUFsukREBrHpEhEZxKZLRGTQqNMLcSF/wjzv1iK1pnJbCswnxuCjXEREOpL1b1KRzLo0mPc8m6DWeNfgr5cru+PVmtxp1jZwaXsFT2H0uvWayg0FMA+360eEPN+43MqyJLgDf1Ms2/rUmpJevFnJ4ev4+BcRkVn51/AFZXoh4dgmmPsKtY2GRM5lzoa5Uy+R6D0V+MKurTBOisLTIR/N+1a9R3F3Msx9DmXiQEQmHMAbN2nuP4OnCppv6xverLiLJ1S06RQRkdz048qVN2BqP42Pj2rPwxNDIiIfTK2BuafqEbVmin76zwh/BPD/T/olfSOnYT9+Bm1O0/vTbJ8+ZaThky4RkUFsukREBrHpEhEZxKZLRGQQmy4RkUFsukREBo06MpZeooyI+PUNRGL9gzBvyXCrNSlzroy2jBF++8kL82CSvlFHf9dkmHd16OeqPTb5Msy1QZgz2/fC3GV3qvfIu4V/28V8PHonItI9XfnYlH2DHE34bK/ILfq6DtTh86hCV+H3UUTkZhY+J0wzqwBv1DLs0tcV0of/9pY+d0mtOZaIz+3TtOW7YZ66Gp9nJyLiVc4JO+5apta8+/khS+sKr8Dnh7VmRqs1jlJ8np6nIUeteTv1O0vrqt+Nx7wCEQNqjTaS+NLccrXmM/+SMa9p3clXYR5VoW+aMxyH/7b6Y/RNsVq6rZ8DxyddIiKD2HSJiAxi0yUiMohNl4jIIDZdIiKDbMGgtaM5iIjo3+OTLhGRQWy6REQGsekSERnEpktEZBCbLhGRQWy6REQG/Q1lDl4iOxY0zAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig,ax = plt.subplots(1,9)\n",
    "with torch.no_grad():\n",
    "    p = next(net.conv.parameters())\n",
    "    for i,x in enumerate(p):\n",
    "        ax[i].imshow(x.detach().cpu()[0,...])\n",
    "        ax[i].axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that some of those filters look like they can recognize some oblique strokes, while others look pretty random. \n",
    "\n",
    "## Multi-layered CNNs and pooling layers\n",
    "\n",
    "First convolutional layers looks for primitive patterns, such as horizontal or vertical lines, but we can apply further convolutional layers on top of them to look for higher-level patterns, such as primitive shapes. Then more convolutional layers can combine those shapes into some parts of the picture, up to the final object that we are trying to classify. \n",
    "\n",
    "When doing so, we may also apply one trick: reducing the spatial size of the image. Once we have detected there is a horizontal stoke within sliding 3x3 window, it is not so important at which exact pixel it occurred. Thus we can \"scale down\" the size of the image, which is done using one of the **pooling layers**:\n",
    "\n",
    " * **Average Pooling** takes a sliding window (for example, 2x2 pixels) and computes an average of values within the window\n",
    " * **Max Pooling** replaces the window with the maximum value. The idea behind max pooling is to detect a presence of a certain pattern within the sliding window.\n",
    "\n",
    "Thus, in a typical CNN there would be several convolutional layers, with pooling layers in between them to decrease dimensions of the image. We would also increase the number of filters, because as patterns become more advanced - there are more possible interesting combinations that we need to be looking for.\n",
    "\n",
    "![An image showing several convolutional layers with pooling layers.](./images/cnn-pyramid.png)\n",
    "\n",
    "Because of decreasing spatial dimensions and increasing feature/filters dimensions, this architecture is also called **pyramid architecture**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "==========================================================================================\n",
       "Layer (type:depth-idx)                   Output Shape              Param #\n",
       "==========================================================================================\n",
       "├─Conv2d: 1-1                            [1, 10, 24, 24]           260\n",
       "├─MaxPool2d: 1-2                         [1, 10, 12, 12]           --\n",
       "├─Conv2d: 1-3                            [1, 20, 8, 8]             5,020\n",
       "├─MaxPool2d: 1-4                         [1, 20, 4, 4]             --\n",
       "├─Linear: 1-5                            [1, 10]                   3,210\n",
       "==========================================================================================\n",
       "Total params: 8,490\n",
       "Trainable params: 8,490\n",
       "Non-trainable params: 0\n",
       "Total mult-adds (M): 0.47\n",
       "==========================================================================================\n",
       "Input size (MB): 0.00\n",
       "Forward/backward pass size (MB): 0.06\n",
       "Params size (MB): 0.03\n",
       "Estimated Total Size (MB): 0.09\n",
       "=========================================================================================="
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class MultiLayerCNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MultiLayerCNN, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 10, 5)\n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(10, 20, 5)\n",
    "        self.fc = nn.Linear(320,10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pool(nn.functional.relu(self.conv1(x)))\n",
    "        x = self.pool(nn.functional.relu(self.conv2(x)))\n",
    "        x = x.view(-1, 320)\n",
    "        x = nn.functional.log_softmax(self.fc(x),dim=1)\n",
    "        return x\n",
    "\n",
    "net = MultiLayerCNN()\n",
    "summary(net,input_size=(1,1,28,28))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note a few things about this definition:\n",
    "* Instead of using `Flatten` layer, we are flattening the tensor inside `forward` function using `view` function. Since flattening layer does not have trainable weights, it is not essential that we create a separate layer instance within our class\n",
    "* We use just one instance of pooling layer in our model, also because it does not contain any trainable parameters, and this one instance can be effectively reused\n",
    "* The number of trainable parameters (~8.5K) is dramatically smaller than in previous cases. This happens because convolutional layers in general have few parameters, and dimensionality of the image before applying final dense layer is significantly reduced. Small number of parameters have positive impact on our models, because it helps to prevent overfitting even on smaller dataset sizes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  0, Train acc=0.952, Val acc=0.977, Train loss=0.001, Val loss=0.001\n",
      "Epoch  1, Train acc=0.982, Val acc=0.983, Train loss=0.000, Val loss=0.000\n",
      "Epoch  2, Train acc=0.986, Val acc=0.983, Train loss=0.000, Val loss=0.000\n",
      "Epoch  3, Train acc=0.986, Val acc=0.978, Train loss=0.000, Val loss=0.001\n",
      "Epoch  4, Train acc=0.987, Val acc=0.981, Train loss=0.000, Val loss=0.000\n"
     ]
    }
   ],
   "source": [
    "hist = train(net,train_loader,test_loader,epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What you should probably observe is that we are able to achieve higher accuracy than with just one layer, and much faster - just with 1 or 2 epochs. It means that sophisticated network architecture needs much fewer data to figure out what is going on, and to extract generic patterns from our images.\n",
    "\n",
    "## Playing with real images from the CIFAR-10 dataset\n",
    "\n",
    "While our handwritten digit recognition problem may seem like a toy problem, we are now ready to do something more serious. Let's explore more advanced dataset of pictures of different objects, called [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html). It contains 60k 32x32 images, divided into 10 classes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data/cifar-10-python.tar.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77b339f50a6b48e98e5db22e0ddd7eb6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./data/cifar-10-python.tar.gz to ./data\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "transform = torchvision.transforms.Compose(\n",
    "    [torchvision.transforms.ToTensor(),\n",
    "     torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
    "\n",
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=14, shuffle=True)\n",
    "testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=14, shuffle=False)\n",
    "classes = ('plane', 'car', 'bird', 'cat',\n",
    "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 123956 characters]\n",
      "text/plain": [
       "<Figure size 1080x216 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_dataset(trainset,classes=classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A well-known architecture for CIFAR-10 is called [LeNet](https://en.wikipedia.org/wiki/LeNet), and has been proposed by *Yann LeCun*. It follows the same principles as we have outlined above, the main difference being 3 input color channels instead of 1. \n",
    "\n",
    "We also do one more simplification to this model - we do not use `log_softmax` as output activation function, and just return the output of last fully-connected layer. In this case we can just use `CrossEntropyLoss` loss function to optimize the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "==========================================================================================\n",
       "Layer (type:depth-idx)                   Output Shape              Param #\n",
       "==========================================================================================\n",
       "├─Conv2d: 1-1                            [1, 6, 28, 28]            456\n",
       "├─MaxPool2d: 1-2                         [1, 6, 14, 14]            --\n",
       "├─Conv2d: 1-3                            [1, 16, 10, 10]           2,416\n",
       "├─MaxPool2d: 1-4                         [1, 16, 5, 5]             --\n",
       "├─Conv2d: 1-5                            [1, 120, 1, 1]            48,120\n",
       "├─Flatten: 1-6                           [1, 120]                  --\n",
       "├─Linear: 1-7                            [1, 64]                   7,744\n",
       "├─Linear: 1-8                            [1, 10]                   650\n",
       "==========================================================================================\n",
       "Total params: 59,386\n",
       "Trainable params: 59,386\n",
       "Non-trainable params: 0\n",
       "Total mult-adds (M): 0.65\n",
       "==========================================================================================\n",
       "Input size (MB): 0.01\n",
       "Forward/backward pass size (MB): 0.05\n",
       "Params size (MB): 0.24\n",
       "Estimated Total Size (MB): 0.30\n",
       "=========================================================================================="
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class LeNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LeNet, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "        self.pool = nn.MaxPool2d(2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.conv3 = nn.Conv2d(16,120,5)\n",
    "        self.flat = nn.Flatten()\n",
    "        self.fc1 = nn.Linear(120,64)\n",
    "        self.fc2 = nn.Linear(64,10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pool(nn.functional.relu(self.conv1(x)))\n",
    "        x = self.pool(nn.functional.relu(self.conv2(x)))\n",
    "        x = nn.functional.relu(self.conv3(x))\n",
    "        x = self.flat(x)\n",
    "        x = nn.functional.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "net = LeNet()\n",
    "\n",
    "summary(net,input_size=(1,3,32,32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training this network properly will take significant amount of time, and should preferably be done on GPU-enabled compute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch  0, Train acc=0.261, Val acc=0.388, Train loss=0.143, Val loss=0.121\n",
      "Epoch  1, Train acc=0.437, Val acc=0.491, Train loss=0.110, Val loss=0.101\n",
      "Epoch  2, Train acc=0.508, Val acc=0.522, Train loss=0.097, Val loss=0.094\n"
     ]
    }
   ],
   "source": [
    "opt = torch.optim.SGD(net.parameters(),lr=0.001,momentum=0.9)\n",
    "hist = train(net, trainloader, testloader, epochs=3, optimizer=opt, loss_fn=nn.CrossEntropyLoss())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy that we have been able to achieve with 3 epochs of training does not seem great. However, remember that blind guessing would only give us 10% accuracy, and that our problem is actually significantly more difficult than MNIST digit classification. Getting above 50% accuracy in such a short training time seems like a good accomplishment.\n",
    "\n",
    "## Takeaways\n",
    "\n",
    "In this unit, we have learned the main concept behind computer vision neural networks - convolutional networks. Real-life architectures that power image classification, object detection, and even image generation networks are all based on CNNs, just with more layers and some additional training tricks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py37_pytorch",
   "language": "python",
   "name": "conda-env-py37_pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}