{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "En2vX4FuwHlu"
      },
      "source": [
        "## Introduction to Tensorflow and Keras\n",
        "\n",
        "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n",
        "\n",
        "### Neural Frameworks\n",
        "\n",
        "We have learnt that to train neural networks you need:\n",
        "* Quickly multiply matrices (tensors)\n",
        "* Compute gradients to perform gradient descent optimization\n",
        "\n",
        "What neural network frameworks allow you to do:\n",
        "* Operate with tensors on whatever compute is available, CPU or GPU, or even TPU\n",
        "* Automatically compute gradients (they are explicitly programmed for all built-in tensor functions)\n",
        "\n",
        "Optionally:\n",
        "* Neural Network constructor / higher level API (describe network as a sequence of layers)\n",
        "* Simple training functions (`fit`, as in Scikit Learn)\n",
        "* A number of optimization algorithms in addition to gradient descent\n",
        "* Data handling abstractions (that will ideally work on GPU, too)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cACQoFMwHl3"
      },
      "source": [
        "### Most Popular Frameworks\n",
        "\n",
        "* Tensorflow 1.x - first widely available framework (Google). Allowed to define static computation graph, push it to GPU, and explicitly evaluate it\n",
        "* PyTorch - a framework from Facebook that is growing in popularity\n",
        "* Keras - higher level API on top of Tensorflow/PyTorch to unify and simplify using neural networks (Francois Chollet)\n",
        "* Tensorflow 2.x + Keras - new version of Tensorflow with integrated Keras functionality, which supports **dynamic computation graph**, allowing to perform tensor operations very similar to numpy (and PyTorch)\n",
        "\n",
        "We will consider Tensorflow 2.x and Keras. Make sure you have version 2.x.x of Tensorflow installed:\n",
        "```\n",
        "pip install tensorflow\n",
        "```\n",
        "or\n",
        "```\n",
        "conda install tensorflow\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xwqVx9-bwHl3",
        "outputId": "2aa591b4-b647-441f-9c8e-4e0da2d517a0",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2.7.0\n"
          ]
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "print(tf.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6tp2xGV7wHl4"
      },
      "source": [
        "## Basic Concepts: Tensor\n",
        "\n",
        "**Tensor** is a multi-dimensional array. It is very convenient to use tensors to represent different types of data:\n",
        "* 400x400 - black-and-white picture\n",
        "* 400x400x3 - color picture \n",
        "* 16x400x400x3 - minibatch of 16 color pictures\n",
        "* 25x400x400x3 - one second of 25-fps video\n",
        "* 8x25x400x400x3 - minibatch of 8 1-second videos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qG2bsaR7wHl4"
      },
      "source": [
        "### Simple Tensors\n",
        "\n",
        "You can easily create simple tensors from lists of np-arrays, or generate random ones:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ybpnk08HwHl4",
        "outputId": "fad9ed4a-df82-44a0-84ea-324bc71ea46f",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[1 2]\n",
            " [3 4]], shape=(2, 2), dtype=int32)\n",
            "tf.Tensor(\n",
            "[[-0.33552304 -1.8252622  -1.8532339 ]\n",
            " [ 1.0871267  -1.2779568   0.5240014 ]\n",
            " [-0.12793781 -1.8618349  -0.9020286 ]\n",
            " [ 0.5948797   0.11144501 -2.0396452 ]\n",
            " [ 0.47620854  1.1726047  -0.4405675 ]\n",
            " [-0.27211484 -0.08985762 -0.03376012]\n",
            " [ 0.64274263  0.53368104 -0.9006528 ]\n",
            " [-0.43745974 -1.0081122  -0.13442488]\n",
            " [ 0.36497566  1.3221073  -1.8739727 ]\n",
            " [ 0.94821155 -0.02817811  1.3563292 ]], shape=(10, 3), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "a = tf.constant([[1,2],[3,4]])\n",
        "print(a)\n",
        "a = tf.random.normal(shape=(10,3))\n",
        "print(a)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXFMsV3r09Ux"
      },
      "source": [
        "You can use arithmetic operations on tensors, which are performed element-wise, as in numpy. Tensors are automatically expanded to required dimension, if needed. To extract numpy-array from tensor, use `.numpy()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e5Nu5Xgj1DnQ",
        "outputId": "0dfc8758-4ffd-4968-c7bf-6ba8d435df2e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[ 0.          0.          0.        ]\n",
            " [ 1.4226497   0.54730535  2.3772354 ]\n",
            " [ 0.20758523 -0.03657269  0.9512053 ]\n",
            " [ 0.93040276  1.9367073  -0.18641126]\n",
            " [ 0.8117316   2.9978669   1.4126664 ]\n",
            " [ 0.0634082   1.7354046   1.8194739 ]\n",
            " [ 0.97826564  2.3589432   0.9525811 ]\n",
            " [-0.1019367   0.81715     1.718809  ]\n",
            " [ 0.7004987   3.1473694  -0.02073872]\n",
            " [ 1.2837346   1.7970841   3.2095633 ]], shape=(10, 3), dtype=float32)\n",
            "[0.71496403 0.16117539 0.15672949]\n"
          ]
        }
      ],
      "source": [
        "print(a-a[0])\n",
        "print(tf.exp(a)[0].numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uQ5zN6cVyrG7"
      },
      "source": [
        "## Variables\n",
        "\n",
        "Variables are useful to represent tensor values that can be modified using `assign` and `assign_add`. They are often used to represent neural network weights.\n",
        "\n",
        "As an example, here is a silly way to get a sum of all rows of tensor `a`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7pu0UZ-_yqfB",
        "outputId": "6708c83e-02e6-4442-8757-45918eb1fbc2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([ 2.9411097, -2.9513645, -6.2979555], dtype=float32)>\n"
          ]
        }
      ],
      "source": [
        "s = tf.Variable(tf.zeros_like(a[0]))\n",
        "for i in a:\n",
        "  s.assign_add(i)\n",
        "\n",
        "print(s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rIh1EHcezlNo"
      },
      "source": [
        "Much better way to do it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aQIdWZ1kzn6P",
        "outputId": "1c123d9a-ecd2-4f2e-828e-5ade85ac8f63"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(3,), dtype=float32, numpy=array([ 2.9411097, -2.9513645, -6.2979555], dtype=float32)>"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tf.reduce_sum(a,axis=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-auwezDwHl6"
      },
      "source": [
        "## Computing Gradients\n",
        "\n",
        "For back propagation, you need to compute gradients. This is done using `tf.GradientTape()` idiom:\n",
        " * Add `with tf.GradientTape` block around our computations\n",
        " * Mark those tensors with respect to which we need to compute gradients by calling `tape.watch` (all variables are watched automatically)\n",
        " * Compute whatever we need (build computational graph)\n",
        " * Obtain gradients using `tape.gradient` "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m8vFOXr7wHl6",
        "outputId": "860ac72e-50c7-4ff2-f258-747f27194f90",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[ 0.40935674 -0.3495818 ]\n",
            " [ 0.94165146 -0.33209163]], shape=(2, 2), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "a = tf.random.normal(shape=(2, 2))\n",
        "b = tf.random.normal(shape=(2, 2))\n",
        "\n",
        "with tf.GradientTape() as tape:\n",
        "  tape.watch(a)  # Start recording the history of operations applied to `a`\n",
        "  c = tf.sqrt(tf.square(a) + tf.square(b))  # Do some math using `a`\n",
        "  # What's the gradient of `c` with respect to `a`?\n",
        "  dc_da = tape.gradient(c, a)\n",
        "  print(dc_da)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8sfjBMBu59B5"
      },
      "source": [
        "## Example 1: Linear Regression\n",
        "\n",
        "Now we know enough to solve the classical problem of **Linear regression**. Let's generate small synthetic dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "j723455WwHl7",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.datasets import make_classification, make_regression\n",
        "from sklearn.model_selection import train_test_split\n",
        "import random"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "WJNK_J6v6I-Z",
        "outputId": "eb4a66a6-6b9a-4c8a-bc24-d81eeb2d3f27"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.collections.PathCollection at 0x12892776880>"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 9528 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "np.random.seed(13) # pick the seed for reproducability - change it to explore the effects of random variations\n",
        "\n",
        "train_x = np.linspace(0, 3, 120)\n",
        "train_labels = 2 * train_x + 0.9 + np.random.randn(*train_x.shape) * 0.5\n",
        "\n",
        "plt.scatter(train_x,train_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ng4rZmGc6oxk"
      },
      "source": [
        "Linear regression is defined by a straight line $f_{W,b}(x) = Wx+b$, where $W, b$ are model parameters that we need to find. An error on our dataset $\\{x_i,y_u\\}_{i=1}^N$ (also called **loss function**) can be defined as mean square error:\n",
        "$$\n",
        "\\mathcal{L}(W,b) = {1\\over N}\\sum_{i=1}^N (f_{W,b}(x_i)-y_i)^2\n",
        "$$\n",
        "\n",
        "Let's define our model and loss function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "QxhI4GlB6aiH"
      },
      "outputs": [],
      "source": [
        "input_dim = 1\n",
        "output_dim = 1\n",
        "learning_rate = 0.1\n",
        "\n",
        "# This is our weight matrix\n",
        "w = tf.Variable([[100.0]])\n",
        "# This is our bias vector\n",
        "b = tf.Variable(tf.zeros(shape=(output_dim,)))\n",
        "\n",
        "def f(x):\n",
        "  return tf.matmul(x,w) + b\n",
        "\n",
        "def compute_loss(labels, predictions):\n",
        "  return tf.reduce_mean(tf.square(labels - predictions))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JUxwj3367gD2"
      },
      "source": [
        "We will train the model on a series of minibatches. We will use gradient descent, adjusting model parameters using the following formulae:\n",
        "$$\n",
        "\\begin{array}{l}\n",
        "W^{(n+1)}=W^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial W} \\\\\n",
        "b^{(n+1)}=b^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial b} \\\\\n",
        "\\end{array}\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "-991PErM7fJU"
      },
      "outputs": [],
      "source": [
        "def train_on_batch(x, y):\n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions = f(x)\n",
        "    loss = compute_loss(y, predictions)\n",
        "    # Note that `tape.gradient` works with a list as well (w, b).\n",
        "    dloss_dw, dloss_db = tape.gradient(loss, [w, b])\n",
        "  w.assign_sub(learning_rate * dloss_dw)\n",
        "  b.assign_sub(learning_rate * dloss_db)\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idr2VEWb9rr0"
      },
      "source": [
        "Let's do the training. We will do several passes through the dataset (so-called **epochs**), divide it into minibatches and call the function defined above:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "nOuu0qpx-wAp"
      },
      "outputs": [],
      "source": [
        "# Shuffle the data.\n",
        "indices = np.random.permutation(len(train_x))\n",
        "features = tf.constant(train_x[indices],dtype=tf.float32)\n",
        "labels = tf.constant(train_labels[indices],dtype=tf.float32)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3zdIf6c_85Ht",
        "outputId": "43b04684-8b90-4c65-d5ff-20ebac61c73c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 94.5247\n",
            "Epoch 1: last batch loss = 9.3428\n",
            "Epoch 2: last batch loss = 1.4166\n",
            "Epoch 3: last batch loss = 0.5224\n",
            "Epoch 4: last batch loss = 0.3807\n",
            "Epoch 5: last batch loss = 0.3495\n",
            "Epoch 6: last batch loss = 0.3413\n",
            "Epoch 7: last batch loss = 0.3390\n",
            "Epoch 8: last batch loss = 0.3384\n",
            "Epoch 9: last batch loss = 0.3382\n"
          ]
        }
      ],
      "source": [
        "batch_size = 4\n",
        "for epoch in range(10):\n",
        "  for i in range(0,len(features),batch_size):\n",
        "    loss = train_on_batch(tf.reshape(features[i:i+batch_size],(-1,1)),tf.reshape(labels[i:i+batch_size],(-1,1)))\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We now have obtained optimized parameters $W$ and $b$. Note that their values are similar to the original values used when generating the dataset ($W=2, b=1$)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "US6q0nCBD-LL",
        "outputId": "65a79620-a3eb-445b-aafb-60a60575ab0e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(<tf.Variable 'Variable:0' shape=(1, 1) dtype=float32, numpy=array([[1.8616779]], dtype=float32)>,\n",
              " <tf.Variable 'Variable:0' shape=(1,) dtype=float32, numpy=array([1.0710956], dtype=float32)>)"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "w,b"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "_e6xRMZFDnyI",
        "outputId": "d202b7fe-4383-4d82-b98e-a20f3180093e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x12892ae5eb0>]"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 14912 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.scatter(train_x,train_labels)\n",
        "x = np.array([min(train_x),max(train_x)])\n",
        "y = w.numpy()[0,0]*x+b.numpy()[0]\n",
        "plt.plot(x,y,color='red')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0giuwC9GHzi8"
      },
      "source": [
        "## Computational Graph and GPU Computations\n",
        "\n",
        "Whenever we compute tensor expression, Tensorflow builds a computational graph that can be computed on the available computing device, e.g. CPU or GPU. Since we were using arbitrary Python function in our code, they cannot be included as part of computational graph, and thus when running our code on GPU we would need to pass the data between CPU and GPU back and forth, and compute custom function on CPU.\n",
        "\n",
        "Tensorflow allows us to mark our Python function using `@tf.function` decorator, which will make this function a part of the same computational graph. This decorator can be applied to functions that use standard Tensorflow tensor operations. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "HK7HPLz3Hyrl"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def train_on_batch(x, y):\n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions = f(x)\n",
        "    loss = compute_loss(y, predictions)\n",
        "    # Note that `tape.gradient` works with a list as well (w, b).\n",
        "    dloss_dw, dloss_db = tape.gradient(loss, [w, b])\n",
        "  w.assign_sub(learning_rate * dloss_dw)\n",
        "  b.assign_sub(learning_rate * dloss_db)\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J7HusxWkGjLX"
      },
      "source": [
        "The code has not changed, but if you were running this code on GPU and on larger dataset - you would have noticed the difference in speed. \n",
        "\n",
        "## Dataset API\n",
        "\n",
        "Tensorflow contains a convenient API to work with data. Let's try to use it. We will also train our model from scratch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oYro9Lbr8q0M",
        "outputId": "78c0a6de-71bd-4eef-8819-439495b28672"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 173.4585\n",
            "Epoch 1: last batch loss = 13.8459\n",
            "Epoch 2: last batch loss = 4.5407\n",
            "Epoch 3: last batch loss = 3.7364\n",
            "Epoch 4: last batch loss = 3.4334\n",
            "Epoch 5: last batch loss = 3.1790\n",
            "Epoch 6: last batch loss = 2.9458\n",
            "Epoch 7: last batch loss = 2.7311\n",
            "Epoch 8: last batch loss = 2.5332\n",
            "Epoch 9: last batch loss = 2.3508\n"
          ]
        }
      ],
      "source": [
        "w.assign([[10.0]])\n",
        "b.assign([0.0])\n",
        "\n",
        "# Create a tf.data.Dataset object for easy batched iteration\n",
        "dataset = tf.data.Dataset.from_tensor_slices((train_x.astype(np.float32), train_labels.astype(np.float32)))\n",
        "dataset = dataset.shuffle(buffer_size=1024).batch(256)\n",
        "\n",
        "for epoch in range(10):\n",
        "  for step, (x, y) in enumerate(dataset):\n",
        "    loss = train_on_batch(tf.reshape(x,(-1,1)), tf.reshape(y,(-1,1)))\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A10prCPowHl7"
      },
      "source": [
        "## Example 2: Classification\n",
        "\n",
        "Now we will consider binary classification problem. A good example of such a problem would be a tumour classification between malignant and benign based on it's size and age.\n",
        "\n",
        "The core model is similar to regression, but we need to use different loss function. Let's start by generating sample data:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "j0OTPkGpwHl7",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "np.random.seed(0) # pick the seed for reproducibility - change it to explore the effects of random variations\n",
        "\n",
        "n = 100\n",
        "X, Y = make_classification(n_samples = n, n_features=2,\n",
        "                           n_redundant=0, n_informative=2, flip_y=0.05,class_sep=1.5)\n",
        "X = X.astype(np.float32)\n",
        "Y = Y.astype(np.int32)\n",
        "\n",
        "split = [ 70*n//100, (15+70)*n//100 ]\n",
        "train_x, valid_x, test_x = np.split(X, split)\n",
        "train_labels, valid_labels, test_labels = np.split(Y, split)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "id": "c-_BjSHPwHl8",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def plot_dataset(features, labels, W=None, b=None):\n",
        "    # prepare the plot\n",
        "    fig, ax = plt.subplots(1, 1)\n",
        "    ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
        "    ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
        "    colors = ['r' if l else 'b' for l in labels]\n",
        "    ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
        "    if W is not None:\n",
        "        min_x = min(features[:,0])\n",
        "        max_x = max(features[:,1])\n",
        "        min_y = min(features[:,1])*(1-.1)\n",
        "        max_y = max(features[:,1])*(1+.1)\n",
        "        cx = np.array([min_x,max_x],dtype=np.float32)\n",
        "        cy = (0.5-W[0]*cx-b)/W[1]\n",
        "        ax.plot(cx,cy,'g')\n",
        "        ax.set_ylim(min_y,max_y)\n",
        "    fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "tq0vFchQwHl8",
        "outputId": "9a5aa6a0-c92f-4d72-9e78-c0f615804bff",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_66184/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 27056 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x, train_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Normalizing Data\n",
        "\n",
        "Before training, it is common to bring our input features to the standard range of [0,1] (or [-1,1]). The exact reasons for that we will discuss later in the course, but in short the reason is the following. We want to avoid values that flow through our network getting too big or too small, and we normally agree to keep all values in the small range close to 0. Thus we initialize the weights with small random numbers, and we keep signals in the same range.\n",
        "\n",
        "When normalizing data, we need to subtract min value and divide by range. We compute min value and range using training data, and then normalize test/validation dataset using the same min/range values from the training set. This is because in real life we will only know the training set, and not all incoming new values that the network would be asked to predict. Occasionally, the new value may fall out of the [0,1] range, but that's not crucial.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {},
      "outputs": [],
      "source": [
        "train_x_norm = (train_x-np.min(train_x)) / (np.max(train_x)-np.min(train_x))\n",
        "valid_x_norm = (valid_x-np.min(train_x)) / (np.max(train_x)-np.min(train_x))\n",
        "test_x_norm = (test_x-np.min(train_x)) / (np.max(train_x)-np.min(train_x))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjPlpf2-wHl8"
      },
      "source": [
        "## Training One-Layer Perceptron\n",
        "\n",
        "Let's use Tensorflow gradient computing machinery to train one-layer perceptron.\n",
        "\n",
        "Our neural network will have 2 inputs and 1 output. The weight matrix $W$ will have size $2\\times1$, and bias vector $b$ -- $1$.\n",
        "\n",
        "Core model will be the same as in previous example, but loss function will be a logistic loss. To apply logistic loss, we need to get the value of **probability** as the output of our network, i.e. we need to bring the output $z$ to the range [0,1] using `sigmoid` activation function: $p=\\sigma(z)$.\n",
        "\n",
        "If we get the probability $p_i$ for the i-th input value corresponding to the actual class $y_i\\in\\{0,1\\}$, we compute the loss as $\\mathcal{L_i}=-(y_i\\log p_i + (1-y_i)log(1-p_i))$. \n",
        "\n",
        "In Tensorflow, both those steps (applying sigmoid and then logistic loss) can be done using one call to `sigmoid_cross_entropy_with_logits` function. Since we are training our network in minibatches, we need to average out the loss across all elements of a minibatch using `reduce_mean`: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "id": "kdDxWeCqwHl8",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "W = tf.Variable(tf.random.normal(shape=(2,1)),dtype=tf.float32)\n",
        "b = tf.Variable(tf.zeros(shape=(1,),dtype=tf.float32))\n",
        "\n",
        "learning_rate = 0.1\n",
        "\n",
        "@tf.function\n",
        "def train_on_batch(x, y):\n",
        "  with tf.GradientTape() as tape:\n",
        "    z = tf.matmul(x, W) + b\n",
        "    loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=y,logits=z))\n",
        "  dloss_dw, dloss_db = tape.gradient(loss, [W, b])\n",
        "  W.assign_sub(learning_rate * dloss_dw)\n",
        "  b.assign_sub(learning_rate * dloss_db)\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zAAgw0h6KzUd"
      },
      "source": [
        "We will use minibatches of 16 elements, and do a few epochs of training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PfyqjVb2wHl8",
        "outputId": "308850b8-fe17-4cda-ac27-8bcda210f113",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.3823\n",
            "Epoch 1: last batch loss = 0.5243\n",
            "Epoch 2: last batch loss = 0.4510\n",
            "Epoch 3: last batch loss = 0.3261\n",
            "Epoch 4: last batch loss = 0.4177\n",
            "Epoch 5: last batch loss = 0.3323\n",
            "Epoch 6: last batch loss = 0.6294\n",
            "Epoch 7: last batch loss = 0.6334\n",
            "Epoch 8: last batch loss = 0.2571\n",
            "Epoch 9: last batch loss = 0.3425\n"
          ]
        }
      ],
      "source": [
        "# Create a tf.data.Dataset object for easy batched iteration\n",
        "dataset = tf.data.Dataset.from_tensor_slices((train_x_norm.astype(np.float32), train_labels.astype(np.float32)))\n",
        "dataset = dataset.shuffle(128).batch(2)\n",
        "\n",
        "for epoch in range(10):\n",
        "  for step, (x, y) in enumerate(dataset):\n",
        "    loss = train_on_batch(x, tf.expand_dims(y,1))\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4_Atvn5K4K9"
      },
      "source": [
        "To make sure our training worked, let's plot the line that separates two classes. Separation line is defined by the equation $W\\times x + b = 0.5$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "PgRTHttLwHl9",
        "outputId": "e4407e1b-edf5-48e5-fdc2-da28120a3c6b",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_66184/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 28308 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x,train_labels,W.numpy(),b.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's see how our model behaves on the validation data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "oEQswfCGrmHw",
        "outputId": "3cf61882-60e1-4baa-8e51-0c31ea80875c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.collections.PathCollection at 0x12892a01460>"
            ]
          },
          "execution_count": 61,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 10584 characters]",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "pred = tf.matmul(test_x,W)+b\n",
        "fig,ax = plt.subplots(1,2)\n",
        "ax[0].scatter(test_x[:,0],test_x[:,1],c=pred[:,0]>0.5)\n",
        "ax[1].scatter(test_x[:,0],test_x[:,1],c=valid_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To compute the accuracy on the validation data, we can cast boolean type to float, and compute the mean:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HUjdeIefsIsg",
        "outputId": "f267f505-8ba4-43ef-9ebe-df124c3c05a1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(), dtype=float32, numpy=0.46666667>"
            ]
          },
          "execution_count": 62,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tf.reduce_mean(tf.cast(((pred[0]>0.5)==test_labels),tf.float32))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's explain what goes on here:\n",
        "* `pred` is the values predicted by the network. They are not quite probabilities, because we have not used an activation function, but values greater than 0.5 correspond to class 1, and smaller - to class 0.\n",
        "*  `pred[0]>0.5` creates a boolean tensor of results, where `True` corresponds to class 1, and `False` - to class 0\n",
        "* We compare that tensor to expected labels `valid_labels`, getting the boolean vector or correct predictions, where `True` corresponds to the correct prediction, and `False` - to incorrect one.\n",
        "* We convert that tensor to floating point using `tf.cast`\n",
        "* We then compute the mean value using `tf.reduce_mean` - that is exactly our desired accuracy  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_95qF9lY2kHp"
      },
      "source": [
        "## Using TensorFlow/Keras Optimizers\n",
        "\n",
        "Tensorflow is closely integrated with Keras, which contains a lot of useful functionality. For example, we can use different **optimization algorithms**. Let's do that, and also print obtained accuracy during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ups7nlV22ofp",
        "outputId": "aa4dff06-82b9-4b2f-ca00-33970ea2b989"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 4.7787, acc = 1.0000\n",
            "Epoch 1: last batch loss = 8.4343, acc = 0.5000\n",
            "Epoch 2: last batch loss = 8.3255, acc = 0.5000\n",
            "Epoch 3: last batch loss = 7.5579, acc = 0.5000\n",
            "Epoch 4: last batch loss = 6.5254, acc = 0.5000\n",
            "Epoch 5: last batch loss = 7.3800, acc = 0.5000\n",
            "Epoch 6: last batch loss = 7.7586, acc = 0.5000\n",
            "Epoch 7: last batch loss = 10.4724, acc = 0.0000\n",
            "Epoch 8: last batch loss = 9.4423, acc = 0.5000\n",
            "Epoch 9: last batch loss = 4.1888, acc = 1.0000\n",
            "Epoch 10: last batch loss = 11.2127, acc = 0.0000\n",
            "Epoch 11: last batch loss = 9.0417, acc = 0.5000\n",
            "Epoch 12: last batch loss = 7.9847, acc = 0.5000\n",
            "Epoch 13: last batch loss = 3.7879, acc = 1.0000\n",
            "Epoch 14: last batch loss = 6.8455, acc = 0.5000\n",
            "Epoch 15: last batch loss = 6.5204, acc = 0.5000\n",
            "Epoch 16: last batch loss = 9.2386, acc = 0.5000\n",
            "Epoch 17: last batch loss = 6.2447, acc = 0.5000\n",
            "Epoch 18: last batch loss = 3.9107, acc = 1.0000\n",
            "Epoch 19: last batch loss = 5.7645, acc = 1.0000\n"
          ]
        }
      ],
      "source": [
        "optimizer = tf.keras.optimizers.Adam(0.01)\n",
        "\n",
        "W = tf.Variable(tf.random.normal(shape=(2,1)))\n",
        "b = tf.Variable(tf.zeros(shape=(1,),dtype=tf.float32))\n",
        "\n",
        "@tf.function\n",
        "def train_on_batch(x, y):\n",
        "  vars = [W, b]\n",
        "  with tf.GradientTape() as tape:\n",
        "    z = tf.sigmoid(tf.matmul(x, W) + b)\n",
        "    loss = tf.reduce_mean(tf.keras.losses.binary_crossentropy(z,y))\n",
        "    correct_prediction = tf.equal(tf.round(y), tf.round(z))\n",
        "    acc = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
        "    grads = tape.gradient(loss, vars)\n",
        "    optimizer.apply_gradients(zip(grads,vars))\n",
        "  return loss,acc\n",
        "\n",
        "for epoch in range(20):\n",
        "  for step, (x, y) in enumerate(dataset):\n",
        "    loss,acc = train_on_batch(tf.reshape(x,(-1,2)), tf.reshape(y,(-1,1)))\n",
        "  print('Epoch %d: last batch loss = %.4f, acc = %.4f' % (epoch, float(loss),acc))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvAiaj_JndyP"
      },
      "source": [
        "**Task 1**: Plot the graphs of loss function and accuracy on training and validation data during training\n",
        "\n",
        "**Task 2**: Try to solve MNIST classificiation problem using this code. Hint: use `softmax_crossentropy_with_logits` or `sparse_softmax_cross_entropy_with_logits` as loss function. In the first case you need to feed expected output values in *one hot encoding*, and in the second case - as integer class number."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "995iCprDrgYQ"
      },
      "source": [
        "## Keras\n",
        "### Deep Learning for Humans\n",
        "\n",
        "* Keras is a library originally developed by Francois Chollet to work on top of Tensorflow, CNTK and Theano, to unify all lower-level frameworks. You can still install Keras as a separate library, but it is not advised to do so. \n",
        "* Now Keras is included as part of Tensorflow library\n",
        "* You can easily construct neural networks from layers\n",
        "* Contains `fit` function to do all training, plus a lot of functions to work with typical data (pictures, text, etc.)\n",
        "* A lot of samples\n",
        "* Functional API vs. Sequential API\n",
        "\n",
        "Keras provides higher level abstractions for neural networks, allowing us to operate in terms of layers, models and optimizers, and not in terms of tensors and gradients.  \n",
        "\n",
        "Classical Deep Learning book from the creator of Keras: [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python)\n",
        "\n",
        "### Functional API\n",
        "\n",
        "When using functional API, we define the **input** to the network as `keras.Input`, and then compute the **output** by passing it through a series of computations. Finally, we define **model** as an object that transforms input into output.\n",
        "\n",
        "Once we obtained **model** object, we need to:\n",
        "* **Compile it**, by specifying loss function and the optimizer that we want to use with our model\n",
        "* **Train it** by calling `fit` function with the training (and possibly validation) data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QJWplVfy34Eo",
        "outputId": "9be976f2-4f9a-495c-bddc-a7f9ec30989a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 2)]               0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 1)                 3         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 3\n",
            "Trainable params: 3\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/15\n",
            "9/9 [==============================] - 1s 2ms/step - loss: 0.7812 - accuracy: 0.2857\n",
            "Epoch 2/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.7142 - accuracy: 0.4000\n",
            "Epoch 3/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.6683 - accuracy: 0.6143\n",
            "Epoch 4/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.6221 - accuracy: 0.8429\n",
            "Epoch 5/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.5843 - accuracy: 0.8857\n",
            "Epoch 6/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.5447 - accuracy: 0.9429\n",
            "Epoch 7/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.5135 - accuracy: 0.9286\n",
            "Epoch 8/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4878 - accuracy: 0.9429\n",
            "Epoch 9/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4679 - accuracy: 0.9429\n",
            "Epoch 10/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4446 - accuracy: 0.9429\n",
            "Epoch 11/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4349 - accuracy: 0.8714\n",
            "Epoch 12/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4156 - accuracy: 0.9286\n",
            "Epoch 13/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4019 - accuracy: 0.9429\n",
            "Epoch 14/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.3908 - accuracy: 0.9286\n",
            "Epoch 15/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.3777 - accuracy: 0.9286\n"
          ]
        }
      ],
      "source": [
        "inputs = tf.keras.Input(shape=(2,))\n",
        "z = tf.keras.layers.Dense(1,kernel_initializer='glorot_uniform',activation='sigmoid')(inputs)\n",
        "model = tf.keras.models.Model(inputs,z)\n",
        "\n",
        "model.compile(tf.keras.optimizers.Adam(0.1),'binary_crossentropy',['accuracy'])\n",
        "model.summary()\n",
        "h = model.fit(train_x_norm,train_labels,batch_size=8,epochs=15)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "K2Kf60IrZcqs",
        "outputId": "b60b868d-3562-4715-f5d5-1f9764e45f09"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x12894b95250>]"
            ]
          },
          "execution_count": 65,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 11884 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.plot(h.history['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iJruFXmb_dur"
      },
      "source": [
        "### Sequential API\n",
        "\n",
        "Alternatively, we can start thinking of a model as of a **sequence of layers**, and just specify those layers by adding them to the `model` object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iWc_kSr8_YXt",
        "outputId": "345dbe65-629d-468f-ed75-1d412c966340"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense_1 (Dense)             (None, 5)                 15        \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 1)                 6         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 21\n",
            "Trainable params: 21\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/15\n",
            "9/9 [==============================] - 1s 64ms/step - loss: 0.6994 - accuracy: 0.5000 - val_loss: 0.6719 - val_accuracy: 0.4667\n",
            "Epoch 2/15\n",
            "9/9 [==============================] - 0s 6ms/step - loss: 0.6635 - accuracy: 0.5429 - val_loss: 0.6531 - val_accuracy: 0.4667\n",
            "Epoch 3/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.6469 - accuracy: 0.5857 - val_loss: 0.5775 - val_accuracy: 1.0000\n",
            "Epoch 4/15\n",
            "9/9 [==============================] - 0s 4ms/step - loss: 0.5639 - accuracy: 0.9143 - val_loss: 0.5395 - val_accuracy: 0.7333\n",
            "Epoch 5/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.5236 - accuracy: 0.7143 - val_loss: 0.4498 - val_accuracy: 0.9333\n",
            "Epoch 6/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.4573 - accuracy: 0.8714 - val_loss: 0.3584 - val_accuracy: 1.0000\n",
            "Epoch 7/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.3867 - accuracy: 0.8714 - val_loss: 0.2989 - val_accuracy: 0.9333\n",
            "Epoch 8/15\n",
            "9/9 [==============================] - 0s 7ms/step - loss: 0.3388 - accuracy: 0.8857 - val_loss: 0.2204 - val_accuracy: 1.0000\n",
            "Epoch 9/15\n",
            "9/9 [==============================] - 0s 6ms/step - loss: 0.2815 - accuracy: 0.9429 - val_loss: 0.1957 - val_accuracy: 1.0000\n",
            "Epoch 10/15\n",
            "9/9 [==============================] - 0s 6ms/step - loss: 0.2692 - accuracy: 0.8857 - val_loss: 0.1323 - val_accuracy: 1.0000\n",
            "Epoch 11/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.2591 - accuracy: 0.9429 - val_loss: 0.1105 - val_accuracy: 1.0000\n",
            "Epoch 12/15\n",
            "9/9 [==============================] - 0s 6ms/step - loss: 0.2229 - accuracy: 0.9286 - val_loss: 0.1051 - val_accuracy: 1.0000\n",
            "Epoch 13/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.2146 - accuracy: 0.9143 - val_loss: 0.0919 - val_accuracy: 1.0000\n",
            "Epoch 14/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.2031 - accuracy: 0.9429 - val_loss: 0.0859 - val_accuracy: 1.0000\n",
            "Epoch 15/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.1997 - accuracy: 0.9429 - val_loss: 0.0829 - val_accuracy: 1.0000\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x12894cfba30>"
            ]
          },
          "execution_count": 66,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Dense(5,activation='sigmoid',input_shape=(2,)))\n",
        "model.add(tf.keras.layers.Dense(1,activation='sigmoid'))\n",
        "\n",
        "model.compile(tf.keras.optimizers.Adam(0.1),'binary_crossentropy',['accuracy'])\n",
        "model.summary()\n",
        "model.fit(train_x_norm,train_labels,validation_data=(test_x_norm,test_labels),batch_size=8,epochs=15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BmHNhUU8bqEX"
      },
      "source": [
        "## Classification Loss Functions\n",
        "\n",
        "It is important to correctly specify loss function and activation function on the last layer of the network. The main rules are the following:\n",
        "* If the network has one output (**binary classification**), we use **sigmoid** activation function, for **multiclass classification** - **softmax**\n",
        "* If the output class is represented as one-hot-encoding, the loss function will be **cross entropy loss** (categorical cross-entropy), if the output contains class number - **sparse categorical cross-entropy**.  For **binary classification** - use **binary cross-entropy** (same as **log loss**)\n",
        "* **Multi-label classification** is when we can have an object belonging to several classes at the same time. In this case, we need to encode labels using one-hot encoding, and use **sigmoid** as activation function, so that each class probability is between 0 and 1.\n",
        "\n",
        "| Classification | Label Format | Activation Function | Loss |\n",
        "|---------------|-----------------------|-----------------|----------|\n",
        "| Binary      | Probability of 1st class | sigmoid | binary crossentropy |\n",
        "| Binary      | One-hot encoding (2 outputs) | softmax | categorical crossentropy |\n",
        "| Multiclass |  One-hot encoding | softmax | categorical crossentropy |\n",
        "| Multiclass | Class Number | softmax | sparse categorical crossentropy |\n",
        "| Multilabel | One-hot encoding | sigmoid | categorical crossentropy |\n",
        "\n",
        "> Binary classification can also be handled as a special case of multi-class classification with two outputs. In this case, we need to use **softmax**.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gZ-kWx84bMDH"
      },
      "source": [
        "**Task 3**: \n",
        "Use Keras to train MNIST classifier:\n",
        "* Notice that Keras contains some standard datasets, including MNIST. To use MNIST from Keras, you only need a couple of lines of code (more information [here](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist))\n",
        "* Try several network configuration, with different number of layers/neurons, activation functions.\n",
        "\n",
        "What is the best accuracy you were able to achieve?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yX6hqiafwHl9"
      },
      "source": [
        "## Takeaways\n",
        "\n",
        "* Tensorflow allows you to operate on tensors at low level, you have most flexibility.\n",
        "* There are convenient tools to work with data (`td.Data`) and layers (`tf.layers`)\n",
        "* For beginners/typical tasks, it is recommended to use **Keras**, which allows to construct networks from layers\n",
        "* If non-standard architecture is needed, you can implement your own Keras layer, and then use it in Keras models\n",
        "* It is a good idea to look at PyTorch as well and compare approaches. \n",
        "\n",
        "A good sample notebook from the creator of Keras on Keras and Tensorflow 2.0 can be found [here](https://t.co/k694J95PI8)."
      ]
    }
  ],
  "metadata": {
    "celltoolbar": "Slideshow",
    "colab": {
      "collapsed_sections": [],
      "name": "IntroKerasTF.ipynb",
      "provenance": []
    },
    "interpreter": {
      "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
    },
    "kernelspec": {
      "display_name": "Python 3.8.12 64-bit (conda)",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "livereveal": {
      "start_slideshow_at": "selected"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
