{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "En2vX4FuwHlu"
      },
      "source": [
        "## Simplest Introduction to Neural Networks with Keras\n",
        "\n",
        "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n",
        "\n",
        "### Neural Frameworks\n",
        "\n",
        "There are several frameworks for training neural networks. However, if you want to get started fast and not go into much detail on how things work internally - you should consider using [Keras](https://keras.io/). This short tutorial will get you started, and if you want to get deeper into understanding how things work - look into [Introduction to Tensorflow and Keras](IntroKerasTF.ipynb) notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cACQoFMwHl3"
      },
      "source": [
        "### Getting things ready\n",
        "\n",
        "Keras is a part of Tensorflow 2.x framework. Let's make sure we have version 2.x.x of Tensorflow installed:\n",
        "```\n",
        "pip install tensorflow\n",
        "```\n",
        "or\n",
        "```\n",
        "conda install tensorflow\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xwqVx9-bwHl3",
        "outputId": "2aa591b4-b647-441f-9c8e-4e0da2d517a0",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Tensorflow version = 2.7.0\n",
            "Keras version = 2.7.0\n"
          ]
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "import numpy as np\n",
        "from sklearn.datasets import make_classification\n",
        "import matplotlib.pyplot as plt\n",
        "print(f'Tensorflow version = {tf.__version__}')\n",
        "print(f'Keras version = {keras.__version__}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6tp2xGV7wHl4"
      },
      "source": [
        "## Basic Concepts: Tensor\n",
        "\n",
        "**Tensor** is a multi-dimensional array. It is very convenient to use tensors to represent different types of data:\n",
        "* 400x400 - black-and-white picture\n",
        "* 400x400x3 - color picture \n",
        "* 16x400x400x3 - minibatch of 16 color pictures\n",
        "* 25x400x400x3 - one second of 25-fps video\n",
        "* 8x25x400x400x3 - minibatch of 8 1-second videos\n",
        "\n",
        "Tensors give us a convenient way to represent input/output data, as well we weights inside the neural network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A10prCPowHl7"
      },
      "source": [
        "## Sample Problem\n",
        "\n",
        "Let's consider binary classification problem. A good example of such a problem would be a tumour classification between malignant and benign based on it's size and age. Let's start by generating some sample data:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "j0OTPkGpwHl7",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "np.random.seed(0) # pick the seed for reproducibility - change it to explore the effects of random variations\n",
        "\n",
        "n = 100\n",
        "X, Y = make_classification(n_samples = n, n_features=2,\n",
        "                           n_redundant=0, n_informative=2, flip_y=0.05,class_sep=1.5)\n",
        "X = X.astype(np.float32)\n",
        "Y = Y.astype(np.int32)\n",
        "\n",
        "split = [ 70*n//100 ]\n",
        "train_x, test_x = np.split(X, split)\n",
        "train_labels, test_labels = np.split(Y, split)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "c-_BjSHPwHl8",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def plot_dataset(features, labels, W=None, b=None):\n",
        "    # prepare the plot\n",
        "    fig, ax = plt.subplots(1, 1)\n",
        "    ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
        "    ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
        "    colors = ['r' if l else 'b' for l in labels]\n",
        "    ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
        "    if W is not None:\n",
        "        min_x = min(features[:,0])\n",
        "        max_x = max(features[:,1])\n",
        "        min_y = min(features[:,1])*(1-.1)\n",
        "        max_y = max(features[:,1])*(1+.1)\n",
        "        cx = np.array([min_x,max_x],dtype=np.float32)\n",
        "        cy = (0.5-W[0]*cx-b)/W[1]\n",
        "        ax.plot(cx,cy,'g')\n",
        "        ax.set_ylim(min_y,max_y)\n",
        "    fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "tq0vFchQwHl8",
        "outputId": "9a5aa6a0-c92f-4d72-9e78-c0f615804bff",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_103052/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 27056 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x, train_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Normalizing Data\n",
        "\n",
        "Before training, it is common to bring our input features to the standard range of [0,1] (or [-1,1]). The exact reasons for that we will discuss later in the course, but in short the reason is the following. We want to avoid values that flow through our network getting too big or too small, and we normally agree to keep all values in the small range close to 0. Thus we initialize the weights with small random numbers, and we keep signals in the same range.\n",
        "\n",
        "When normalizing data, we need to subtract min value and divide by range. We compute min value and range using training data, and then normalize test/validation dataset using the same min/range values from the training set. This is because in real life we will only know the training set, and not all incoming new values that the network would be asked to predict. Occasionally, the new value may fall out of the [0,1] range, but that's not crucial.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [],
      "source": [
        "train_x_norm = (train_x-np.min(train_x,axis=0)) / (np.max(train_x,axis=0)-np.min(train_x,axis=0))\n",
        "test_x_norm = (test_x-np.min(train_x,axis=0)) / (np.max(train_x,axis=0)-np.min(train_x,axis=0))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjPlpf2-wHl8"
      },
      "source": [
        "## Training One-Layer Network (Perceptron)\n",
        "\n",
        "In many cases, a neural network would be a sequence of layers. It can be defined in Keras using `Sequential` model in the following manner:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense_2 (Dense)             (None, 1)                 3         \n",
            "                                                                 \n",
            " activation_1 (Activation)   (None, 1)                 0         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 3\n",
            "Trainable params: 3\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.Sequential()\n",
        "model.add(keras.Input(shape=(2,)))\n",
        "model.add(keras.layers.Dense(1))\n",
        "model.add(keras.layers.Activation(keras.activations.sigmoid))\n",
        "\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here, we first create the model, and then add layers to it:\n",
        "* First `Input` layer (which is not strictly speaking a layer) contains the specification of network's input size\n",
        "* `Dense` layer is the actual perceptron that contains trainable weights\n",
        "* Finally, there is a layer with *sigmoid* `Activation` function to bring the result of the network into 0-1 range (to make it a probability).\n",
        "\n",
        "Input size, as well as activation function, can also be specified directly in the `Dense` layer for brevity: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_9\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense_9 (Dense)             (None, 1)                 3         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 3\n",
            "Trainable params: 3\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Dense(1,input_shape=(2,),activation='sigmoid'))\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Before training the model, we need to **compile it**, which essentially mean specifying:\n",
        "* **Loss function**, which defines how loss is calculated. Because we have two-class classification problem, we will use *binary cross-entropy loss*.\n",
        "* **Optimizer** to use. The simplest option would be to use `sgd` for *stochastic gradient descent*, or you can use more sophisticated optimizers such as `adam`.\n",
        "* **Metrics** that we want to use to measure success of our training. Since it is classification task, a good metrics would be `Accuracy` (or `acc` for short)\n",
        "\n",
        "We can specify loss, metrics and optimizer either as strings, or by providing some objects from Keras framework. In our example, we need to specify `learning_rate` parameter to fine-tune learning speed of our model, and thus we provide full name of Keras SGD optimizer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {},
      "outputs": [],
      "source": [
        "model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.2),loss='binary_crossentropy',metrics=['acc'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "After compiling the model, we can do the actual training by calling `fit` method. The most important parameters are:\n",
        "* `x` and `y` specify training data, features and labels respectively\n",
        "* If we want validation to be performed on each epoch, we can specify `validation_data` parameter, which would be a tuple of features and labels\n",
        "* `epochs` specified the number of epochs\n",
        "* If we want training to happen in minibatches, we can speficu `batch_size` parameter. You can also pre-batch the data manually before passing it to `x`/`y`/`validation_data`, in which case you do not need `batch_size`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "70/70 [==============================] - 0s 4ms/step - loss: 0.3379 - acc: 0.9000 - val_loss: 0.3282 - val_acc: 0.9000\n",
            "Epoch 2/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.3270 - acc: 0.9429 - val_loss: 0.3336 - val_acc: 0.9000\n",
            "Epoch 3/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.3195 - acc: 0.9143 - val_loss: 0.3137 - val_acc: 0.9000\n",
            "Epoch 4/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.3087 - acc: 0.9286 - val_loss: 0.2970 - val_acc: 0.9333\n",
            "Epoch 5/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.3006 - acc: 0.9429 - val_loss: 0.3210 - val_acc: 0.9000\n",
            "Epoch 6/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.3003 - acc: 0.9000 - val_loss: 0.2985 - val_acc: 0.9000\n",
            "Epoch 7/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2956 - acc: 0.9286 - val_loss: 0.3037 - val_acc: 0.9000\n",
            "Epoch 8/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2891 - acc: 0.9429 - val_loss: 0.3035 - val_acc: 0.9000\n",
            "Epoch 9/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2809 - acc: 0.9000 - val_loss: 0.2815 - val_acc: 0.9000\n",
            "Epoch 10/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2809 - acc: 0.9286 - val_loss: 0.2907 - val_acc: 0.9000\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x2c420b89910>"
            ]
          },
          "execution_count": 44,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.fit(x=train_x_norm,y=train_labels,validation_data=(test_x_norm,test_labels),epochs=10,batch_size=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4_Atvn5K4K9"
      },
      "source": [
        "You can try to experiment with different training parameters to see how they affect the training:\n",
        "* Setting `batch_size` to be too large (or not specifying it at all) may result in less stable training, because with low-dimensional data small batch sizes provide more precise direction of the gradient for each specific case\n",
        "* Too high `learning_rate` may result in overfitting, or in less stable results, while too low learning rate means it will take more epochs to achieve the result\n",
        "\n",
        "> Note that you can call `fit` function several times in a row to further train the network. If you want to start training from scratch - you need to re-run the cell with the model definition. \n",
        "\n",
        "To make sure our training worked, let's plot the line that separates two classes. Separation line is defined by the equation $W\\times x + b = 0.5$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "PgRTHttLwHl9",
        "outputId": "e4407e1b-edf5-48e5-fdc2-da28120a3c6b",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_103052/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 27928 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x,train_labels,model.layers[0].weights[0],model.layers[0].weights[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvAiaj_JndyP"
      },
      "source": [
        "## Plotting the training graphs\n",
        "\n",
        "`fit` function returns `history` object as a result, which can be used to observe loss and metrics on each epoch. In the example below, we will re-start the training with small learning rate, and will observe how the loss and accuracy behave.\n",
        "\n",
        "> **Note** that we are using slightly different syntax for defining `Sequential` model. Instead of `add`-ing layers one by one, we can also specify the list of layers right when creating the model in the first place - this is a bit shorter syntax, and you may prefer to use it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "70/70 [==============================] - 1s 5ms/step - loss: 0.6600 - acc: 0.6143 - val_loss: 0.6351 - val_acc: 0.8000\n",
            "Epoch 2/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.6384 - acc: 0.7143 - val_loss: 0.6187 - val_acc: 0.8333\n",
            "Epoch 3/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.6188 - acc: 0.7571 - val_loss: 0.6001 - val_acc: 0.8667\n",
            "Epoch 4/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.6022 - acc: 0.7714 - val_loss: 0.5837 - val_acc: 0.9000\n",
            "Epoch 5/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5860 - acc: 0.8571 - val_loss: 0.5673 - val_acc: 0.9000\n",
            "Epoch 6/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5702 - acc: 0.8571 - val_loss: 0.5597 - val_acc: 0.8667\n",
            "Epoch 7/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5568 - acc: 0.8286 - val_loss: 0.5458 - val_acc: 0.9000\n",
            "Epoch 8/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5430 - acc: 0.8714 - val_loss: 0.5325 - val_acc: 0.9000\n",
            "Epoch 9/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5308 - acc: 0.8714 - val_loss: 0.5234 - val_acc: 0.9000\n",
            "Epoch 10/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.5175 - acc: 0.9143 - val_loss: 0.5170 - val_acc: 0.8667\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(1,input_shape=(2,),activation='sigmoid')])\n",
        "model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.05),loss='binary_crossentropy',metrics=['acc'])\n",
        "hist = model.fit(x=train_x_norm,y=train_labels,validation_data=(test_x_norm,test_labels),epochs=10,batch_size=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x2c41a32fe80>]"
            ]
          },
          "execution_count": 50,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 15336 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.plot(hist.history['acc'])\n",
        "plt.plot(hist.history['val_acc'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Multi-Class Classification\n",
        "\n",
        "If you need to solve a problem of multi-class classification, your network would have more that one output - corresponding to the number of classes $C$. Each output will contain the probability of a given class.\n",
        "\n",
        "> Note that you can also use a network with two outputs to perform binary classification in the same manner. That is exactly what we will demonstrate now.\n",
        "\n",
        "When you expect a network to output a set of probabilities $p_1,\\dots, p_C$, we need all of them to add up to 1. To ensure this, we use `softmax` as a final activation function on the last layer. **Softmax** takes a vector input, and makes sure that all components of that vector are transformed into probabilities.\n",
        "\n",
        "Also, since the output of the network is a $C$-dimensional vector, we need labels to have the same form. This can be achieved by using **one-hot encoding**, when the number of a class $i$ is converted to a vector of zeroes, with 1 at the $i$-th position.\n",
        "\n",
        "To compare the probability output of the neural network with expected one-hot-encoded label, we use **cross-entropy loss** function. It takes two probability distributions, and outputs a value of how different they are.\n",
        "\n",
        "So, to summarize what we need to do for multi-class classification with $C$ classes:\n",
        "* The network should have $C$ neurons in the last layer\n",
        "* Last activation function should be **softmax**\n",
        "* Loss should be **cross-entropy loss**\n",
        "* Labels should be converted to **one-hot encoding** (this can be done using `numpy`, or using Keras utils `to_categorical`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "70/70 [==============================] - 1s 6ms/step - loss: 0.6524 - acc: 0.7000 - val_loss: 0.5936 - val_acc: 0.9000\n",
            "Epoch 2/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5715 - acc: 0.8286 - val_loss: 0.5255 - val_acc: 0.8333\n",
            "Epoch 3/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.4820 - acc: 0.8714 - val_loss: 0.4213 - val_acc: 0.9000\n",
            "Epoch 4/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.4426 - acc: 0.9000 - val_loss: 0.3694 - val_acc: 0.9333\n",
            "Epoch 5/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.3602 - acc: 0.9000 - val_loss: 0.3454 - val_acc: 0.9000\n",
            "Epoch 6/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.3209 - acc: 0.8857 - val_loss: 0.2862 - val_acc: 0.9333\n",
            "Epoch 7/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2905 - acc: 0.9286 - val_loss: 0.2787 - val_acc: 0.9000\n",
            "Epoch 8/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2698 - acc: 0.9000 - val_loss: 0.2381 - val_acc: 0.9333\n",
            "Epoch 9/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2639 - acc: 0.8857 - val_loss: 0.2217 - val_acc: 0.9667\n",
            "Epoch 10/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2592 - acc: 0.9286 - val_loss: 0.2391 - val_acc: 0.9000\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(5,input_shape=(2,),activation='relu'),\n",
        "    keras.layers.Dense(2,activation='softmax')\n",
        "])\n",
        "model.compile(keras.optimizers.Adam(0.01),'categorical_crossentropy',['acc'])\n",
        "\n",
        "# Two ways to convert to one-hot encoding\n",
        "train_labels_onehot = keras.utils.to_categorical(train_labels)\n",
        "test_labels_onehot = np.eye(2)[test_labels]\n",
        "\n",
        "hist = model.fit(x=train_x_norm,y=train_labels_onehot,\n",
        "                 validation_data=[test_x_norm,test_labels_onehot],batch_size=1,epochs=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Sparse Categorical Cross-Entropy\n",
        "\n",
        "Often labels in multi-class classification are represented by class numbers. Keras also supports another kind of loss function called **sparse categorical crossentropy**, which expects class number to be integers, and not one-hot vectors. Using this kind of loss function, we can simplify our training code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "70/70 [==============================] - 1s 6ms/step - loss: 0.2353 - acc: 0.9143 - val_loss: 0.2190 - val_acc: 0.9000\n",
            "Epoch 2/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2243 - acc: 0.9286 - val_loss: 0.1886 - val_acc: 0.9333\n",
            "Epoch 3/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2366 - acc: 0.9143 - val_loss: 0.2262 - val_acc: 0.9000\n",
            "Epoch 4/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2259 - acc: 0.9429 - val_loss: 0.2124 - val_acc: 0.9000\n",
            "Epoch 5/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2061 - acc: 0.9429 - val_loss: 0.2691 - val_acc: 0.9000\n",
            "Epoch 6/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2200 - acc: 0.9286 - val_loss: 0.2344 - val_acc: 0.9000\n",
            "Epoch 7/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2133 - acc: 0.9286 - val_loss: 0.1973 - val_acc: 0.9000\n",
            "Epoch 8/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2062 - acc: 0.9429 - val_loss: 0.1893 - val_acc: 0.9000\n",
            "Epoch 9/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2060 - acc: 0.9571 - val_loss: 0.2719 - val_acc: 0.9000\n",
            "Epoch 10/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2021 - acc: 0.9571 - val_loss: 0.2293 - val_acc: 0.9000\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x2c42267de80>"
            ]
          },
          "execution_count": 55,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.compile(keras.optimizers.Adam(0.01),'sparse_categorical_crossentropy',['acc'])\n",
        "model.fit(x=train_x_norm,y=train_labels,validation_data=[test_x_norm,test_labels],batch_size=1,epochs=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Multi-Label Classification\n",
        "\n",
        "Sometime we have cases when our objects can belong to two classes at once. As an example, suppose we want to develop a classifier for cats and dogs on the picture, but we also want to allow cases when both cats and dogs are present.\n",
        "\n",
        "With multi-label classification, instead of one-hot encoded vector, we will have a vector that has 1 in position corresponding to all classes relevant to the input sample. Thus, output of the network should not have normalized probabilities for all classes, but rather for each class individually - which corresponds to using **sigmoid** activation function. Cross-entropy loss can still be used as a loss function.\n",
        "\n",
        "> **Note** that this is very similar to using **different neural networks** to do binary classification for each particular class - only the initial part of the network (up to final classification layer) is shared for all classes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BmHNhUU8bqEX"
      },
      "source": [
        "## Summary of Classification Loss Functions\n",
        "\n",
        "We have seen that binary, multi-class and multi-label classification differ by the type of loss function and activation function on the last layer of the network. It may all be a little bit confusing if you are just starting to learn, but here are a few rules to keep in mind:\n",
        "* If the network has one output (**binary classification**), we use **sigmoid** activation function, for **multiclass classification** - **softmax**\n",
        "* If the output class is represented as one-hot-encoding, the loss function will be **cross entropy loss** (categorical cross-entropy), if the output contains class number - **sparse categorical cross-entropy**.  For **binary classification** - use **binary cross-entropy** (same as **log loss**)\n",
        "* **Multi-label classification** is when we can have an object belonging to several classes at the same time. In this case, we need to encode labels using one-hot encoding, and use **sigmoid** as activation function, so that each class probability is between 0 and 1.\n",
        "\n",
        "| Classification | Label Format | Activation Function | Loss |\n",
        "|---------------|-----------------------|-----------------|----------|\n",
        "| Binary      | Probability of 1st class | sigmoid | binary crossentropy |\n",
        "| Binary      | One-hot encoding (2 outputs) | softmax | categorical crossentropy |\n",
        "| Multiclass |  One-hot encoding | softmax | categorical crossentropy |\n",
        "| Multiclass | Class Number | softmax | sparse categorical crossentropy |\n",
        "| Multilabel | One-hot encoding | sigmoid | categorical crossentropy |\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gZ-kWx84bMDH"
      },
      "source": [
        "**Task**: \n",
        "Use Keras to train a classifier for MNIST handwritten digits:\n",
        "* Notice that Keras contains some standard datasets, including MNIST. To use MNIST from Keras, you only need a couple of lines of code (more information [here](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist))\n",
        "* Try several network configuration, with different number of layers/neurons, activation functions.\n",
        "\n",
        "What is the best accuracy you were able to achieve?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yX6hqiafwHl9"
      },
      "source": [
        "## Takeaways\n",
        "\n",
        "* **Keras** is really recommended for beginners, because it allows to construct networks from layers quite easily, and then train it with just a couple of lines of code\n",
        "* If non-standard architecture is needed, you would need to learn a bit deeper into Tensorflow. Or you can ask someone to implement custom logic as a Keras layer, and then use it in Keras models\n",
        "* It is a good idea to look at PyTorch as well and compare approaches. \n",
        "\n",
        "A good sample notebook from the creator of Keras on Keras and Tensorflow 2.0 can be found [here](https://t.co/k694J95PI8)."
      ]
    }
  ],
  "metadata": {
    "celltoolbar": "Slideshow",
    "colab": {
      "collapsed_sections": [],
      "name": "IntroKerasTF.ipynb",
      "provenance": []
    },
    "interpreter": {
      "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
    },
    "kernelspec": {
      "display_name": "Python 3.8.12 64-bit (conda)",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "livereveal": {
      "start_slideshow_at": "selected"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
