{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification of Pet's Real-Life Images\n",
    "\n",
    "Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).\n",
    "\n",
    "Now it's time to deal with more challenging task - classification of the original [Oxford-IIIT Dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/). Let's start by loading and visualizing the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz\n",
    "!tar xfz images.tar.gz\n",
    "!rm images.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define generic function to display a series of images from a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "def display_images(l,titles=None,fontsize=12):\n",
    "    n=len(l)\n",
    "    fig,ax = plt.subplots(1,n)\n",
    "    for i,im in enumerate(l):\n",
    "        ax[i].imshow(im)\n",
    "        ax[i].axis('off')\n",
    "        if titles is not None:\n",
    "            ax[i].set_title(titles[i],fontsize=fontsize)\n",
    "    fig.set_size_inches(fig.get_size_inches()*n)\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that all images are located in one directory called `images`, and their name contains the name of the class (breed):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fnames = os.listdir('images')[:5]\n",
    "display_images([Image.open(os.path.join('images',x)) for x in fnames],titles=fnames,fontsize=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify classification and use the same approach to loading images as in the previous part, let's sort all images into corresponding directories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for fn in os.listdir('images'):\n",
    "    cls = fn[:fn.rfind('_')].lower()\n",
    "    os.makedirs(os.path.join('images',cls),exist_ok=True)\n",
    "    os.replace(os.path.join('images',fn),os.path.join('images',cls,fn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also define the number of classes in our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "37"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_classes = len(os.listdir('images'))\n",
    "num_classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing dataset for Deep Learning\n",
    "\n",
    "To start training our neural network, we need to convert all images to tensors, and also create tensors corresponding to labels (class numbers). Most neural network frameworks contain simple tools for dealing with images:\n",
    "* In Tensorflow, use `tf.keras.preprocessing.image_dataset_from_directory`\n",
    "* In PyTorch, use `torchvision.datasets.ImageFolder`\n",
    "\n",
    "As you have seen from the pictures above, all of them are close to square image ratio, so we need to resize all images to square size. Also, we can organize images in minibatches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PREPARE THE DATASET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to separate dataset into train and test portions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SPLIT INTO TRAIN-TEST DATASETS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now define data loaders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DEFINE DATA LOADERS if needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 1479796 characters]",
      "text/plain": [
       "<Figure size 3024x2016 with 7 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 2260232 characters]",
      "text/plain": [
       "<Figure size 3024x2016 with 7 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 2370764 characters]",
      "text/plain": [
       "<Figure size 3024x2016 with 7 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# [OPTIONAL] Plot the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a neural network\n",
    "\n",
    "For image classification, you should probably define a convolutional neural network with several layers. What to keep an eye for:\n",
    "* Keep in mind the pyramid architecture, i.e. number of filters should increase as you go deeper\n",
    "* Do not forget activation functions between layers (ReLU) and Max Pooling\n",
    "* Final classifier can be with or without hidden layers, but the number of output neurons should be equal to number of classes.\n",
    "\n",
    "An important thing is to get the activation function on the last layer + loss function right:\n",
    "* In Tensorflow, you can use `softmax` as the activation, and `sparse_categorical_crossentropy` as loss. The difference between sparse categorical cross-entropy and non-sparse one is that the former expects output as the number of class, and not as one-hot vector.\n",
    "* In PyTorch, you can have the final layer without activation function, and use `CrossEntropyLoss` loss function. This function applies softmax automatically. \n",
    "\n",
    "> **Hint:** In PyTorch, you can use `LazyLinear` layer instead of `Linear`, in order to avoid computing the number of inputs. It only requires one `n_out` parameter, which is number of neurons in the layer, and the dimension of input data is picked up automatically upon first `forward` pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DEFINE NEURAL NETWORK ARCHITECTURE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Neural Network\n",
    "\n",
    "Now we are ready to train the neural network. During training, please collect accuracy on train and test data on each epoch, and then plot the accuracy to see if there is overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TRAIN THE NEURAL NETWORK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 17580 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# PLOT THE RESULT: Train and Test Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if you have done everything correctly, you will probably see that the accuracy is quite low.\n",
    "\n",
    "## Transfer Learning\n",
    "\n",
    "To improve the accuracy, let's use pre-trained neural network as feature extractor. Feel free to experiment with VGG-16/VGG-19 models, ResNet50, etc.\n",
    "\n",
    "> Since this training is slower, you may start with training the model for the small number of epochs, eg. 3. You can always resume training to further improve accuracy if needed.\n",
    "\n",
    "We need to normalize our data differently for transfer learning, thus we will reload the dataset again using different set of transforms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LOAD THE DATASET\n",
    "# Perform standard transformations for VGG-16/VGG-19 if needed "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the pre-trained network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vgg = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now define the classification model for your problem:\n",
    "* In PyTorch, there is a slot called `classifier`, which you can replace with your own classifier for the desired number of classes.\n",
    "* In TensorFlow, use VGG network as feature extractor, and build a `Sequential` model with VGG as first layer, and your own classifier on top"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# BUILD MODEL for your problem with your own linear layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to set all parameters of VGG feature extractor not to be trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MAKE VGG Layers not trainable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start the training. Be very patient, as training takes a long time, and our train function is not designed to print anything before the end of the epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TRAIN THE MODEL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems much better now!\n",
    "\n",
    "## Optional: Calculate Top 3 Accuracy\n",
    "\n",
    "We can also computer Top 3 accuracy using the same code as in the previous exercise.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CALCULATE TOP-3 Accuracy of the model"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
  },
  "kernelspec": {
   "display_name": "py37_pytorch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
