{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "En2vX4FuwHlu"
      },
      "source": [
        "## Introduction to PyTorch\n",
        "\n",
        "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n",
        "\n",
        "### Neural Frameworks\n",
        "\n",
        "We have learnt that to train neural networks you need:\n",
        "* Quickly multiply matrices (tensors)\n",
        "* Compute gradients to perform gradient descent optimization\n",
        "\n",
        "What neural network frameworks allow you to do:\n",
        "* Operate with tensors on whatever compute is available, CPU or GPU, or even TPU\n",
        "* Automatically compute gradients (they are explicitly programmed for all built-in tensor functions)\n",
        "\n",
        "Optionally:\n",
        "* Neural Network constructor / higher level API (describe network as a sequence of layers)\n",
        "* Simple training functions (`fit`, as in Scikit Learn)\n",
        "* A number of optimization algorithms in addition to gradient descent\n",
        "* Data handling abstractions (that will ideally work on GPU, too)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cACQoFMwHl3"
      },
      "source": [
        "### Most Popular Frameworks\n",
        "\n",
        "* Tensorflow 1.x - first widely available framework (Google). Allowed to define static computation graph, push it to GPU, and explicitly evaluate it\n",
        "* PyTorch - a framework from Facebook that is growing in popularity\n",
        "* Keras - higher level API on top of Tensorflow/PyTorch to unify and simplify using neural networks (Francois Chollet)\n",
        "* Tensorflow 2.x + Keras - new version of Tensorflow with integrated Keras functionality, which supports **dynamic computation graph**, allowing to perform tensor operations very similar to numpy (and PyTorch)\n",
        "\n",
        "In this Notebook, we will learn to use PyTorch. You need to make sure that you have recent version of PyTorch installed - to do it, follow the [instructions on their site](https://pytorch.org/get-started/locally/). It is normally as simple as doing\n",
        "```\n",
        "pip install torch torchvision\n",
        "```\n",
        "or\n",
        "```\n",
        "conda install pytorch -c pytorch\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "xwqVx9-bwHl3",
        "outputId": "38564a63-0567-4406-ee1a-1d3618f27351",
        "tags": []
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'1.8.2'"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import torch\n",
        "torch.__version__"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6tp2xGV7wHl4"
      },
      "source": [
        "## Basic Concepts: Tensor\n",
        "\n",
        "**Tensor** is a multi-dimensional array. It is very convenient to use tensors to represent different types of data:\n",
        "* 400x400 - black-and-white picture\n",
        "* 400x400x3 - color picture \n",
        "* 16x400x400x3 - minibatch of 16 color pictures\n",
        "* 25x400x400x3 - one second of 25-fps video\n",
        "* 8x25x400x400x3 - minibatch of 8 1-second videos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qG2bsaR7wHl4"
      },
      "source": [
        "### Simple Tensors\n",
        "\n",
        "You can easily create simple tensors from lists of np-arrays, or generate random ones:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ybpnk08HwHl4",
        "outputId": "54e2c89b-b373-4389-b285-49b0510be931",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[1, 2],\n",
            "        [3, 4]])\n",
            "tensor([[ 1.3577,  0.7550, -1.7503],\n",
            "        [-0.7006,  0.1918, -0.2571],\n",
            "        [ 0.2964, -0.3188,  0.4575],\n",
            "        [-0.1524,  1.3446,  0.7218],\n",
            "        [-1.4642, -1.3296, -0.5098],\n",
            "        [ 2.2282,  0.5065,  0.6176],\n",
            "        [-0.3013,  0.9485,  0.1195],\n",
            "        [ 1.0261,  1.5614, -0.1013],\n",
            "        [-0.2211, -0.4294, -2.2319],\n",
            "        [ 1.4257, -0.6976,  0.0656]])\n"
          ]
        }
      ],
      "source": [
        "a = torch.tensor([[1,2],[3,4]])\n",
        "print(a)\n",
        "a = torch.randn(size=(10,3))\n",
        "print(a)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXFMsV3r09Ux"
      },
      "source": [
        "You can use arithmetic operations on tensors, which are performed element-wise, as in numpy. Tensors are automatically expanded to required dimension, if needed. To extract numpy-array from tensor, use `.numpy()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e5Nu5Xgj1DnQ",
        "outputId": "c1fbcd86-dde6-40b6-8edf-7a37f9d60901"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[ 0.0000,  0.0000,  0.0000],\n",
            "        [-2.0583, -0.5631,  1.4932],\n",
            "        [-1.0613, -1.0738,  2.2078],\n",
            "        [-1.5101,  0.5896,  2.4722],\n",
            "        [-2.8219, -2.0846,  1.2405],\n",
            "        [ 0.8706, -0.2485,  2.3679],\n",
            "        [-1.6590,  0.1935,  1.8698],\n",
            "        [-0.3316,  0.8065,  1.6490],\n",
            "        [-1.5788, -1.1844, -0.4816],\n",
            "        [ 0.0680, -1.4526,  1.8159]])\n",
            "[3.887189   2.1276016  0.17371987]\n"
          ]
        }
      ],
      "source": [
        "print(a-a[0])\n",
        "print(torch.exp(a)[0].numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uQ5zN6cVyrG7"
      },
      "source": [
        "## In-place and out-of-place Operations\n",
        "\n",
        "Tensor operations such as `+`/`add` return new tensors. However, sometimes you need to modify the existing tensor in-place. Most of the operations have their in-place counterparts, which end with `_`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mjkbcw3-ACKS",
        "outputId": "ca021008-9ab6-4b09-c5a5-bbe854cd1493"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Result when adding out-of-place: tensor(8)\n",
            "Result after adding in-place: tensor(8)\n"
          ]
        }
      ],
      "source": [
        "u = torch.tensor(5)\n",
        "print(\"Result when adding out-of-place:\",u.add(torch.tensor(3)))\n",
        "u.add_(torch.tensor(3))\n",
        "print(\"Result after adding in-place:\", u)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLPUcVsXACKT"
      },
      "source": [
        "This is how we can compute the sum or all rows in a matrix in a naive way:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7pu0UZ-_yqfB",
        "outputId": "bd2e8c6a-39e1-4f29-990b-9591e866936c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([ 3.4945,  2.5325, -2.8684])\n"
          ]
        }
      ],
      "source": [
        "s = torch.zeros_like(a[0])\n",
        "for i in a:\n",
        "  s.add_(i)\n",
        "\n",
        "print(s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rIh1EHcezlNo"
      },
      "source": [
        "But it is much better to use"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aQIdWZ1kzn6P",
        "outputId": "89000bb4-f45e-493b-a7b0-39fa4e7d92c1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "tensor([ 3.4945,  2.5325, -2.8684])"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "torch.sum(a,axis=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5UzUmEZhACKT"
      },
      "source": [
        "You can read more on PyTorch tensors in the [official documentation](https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-auwezDwHl6"
      },
      "source": [
        "## Computing Gradients\n",
        "\n",
        "For back propagation, you need to compute gradients. We can set any PyTorch Tensor's attribute `requires_grad` to `True`, which will result in all operations with this tensor being tracked for gradient calculations. To compute the gradients, you need to call `backward()` method, after which the gradient will become available using `grad` attribute:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m8vFOXr7wHl6",
        "outputId": "7054c2b1-0b61-4938-937d-813f75f0b195",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[-0.1728,  0.0913],\n",
            "        [-0.1666, -0.1942]])\n"
          ]
        }
      ],
      "source": [
        "a = torch.randn(size=(2, 2), requires_grad=True)\n",
        "b = torch.randn(size=(2, 2))\n",
        "\n",
        "c = torch.mean(torch.sqrt(torch.square(a) + torch.square(b)))  # Do some math using `a`\n",
        "c.backward() # call backward() to compute all gradients\n",
        "# What's the gradient of `c` with respect to `a`?\n",
        "print(a.grad)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nPj3rtrtACKU"
      },
      "source": [
        "To be more precise, PyTorch automatically **accumulates** gradients. If you specify `retain_graph=True` when calling `backward`, computational graph will be preserved, and new gradient is added to the `grad` field. In order to restart computing gradients from scratch, we need to reset `grad` field to 0 explicitly by calling `zero_()`:  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z_VIw8MoACKU",
        "outputId": "36a28b11-6919-47ab-c3f9-c7f1d8500423"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[-0.5185,  0.2739],\n",
            "        [-0.4998, -0.5826]])\n",
            "tensor([[-0.1728,  0.0913],\n",
            "        [-0.1666, -0.1942]])\n"
          ]
        }
      ],
      "source": [
        "c = torch.mean(torch.sqrt(torch.square(a) + torch.square(b)))\n",
        "c.backward(retain_graph=True)\n",
        "c.backward(retain_graph=True)\n",
        "print(a.grad)\n",
        "a.grad.zero_()\n",
        "c.backward()\n",
        "print(a.grad)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HM9sUkVgCiG9"
      },
      "source": [
        "To compute gradients, PyTorch creates and maintains **compute graph**. For each tensor that has the `requires_grad` flag set to `True`, PyTorch maintains a special function called `grad_fn`, which computes the derivative of the expression according to chain differentiation rule:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PcxHb-7jC7Vv",
        "outputId": "3b3fa138-6d09-4636-8a71-f4a4051c7827"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor(0.9143, grad_fn=<MeanBackward0>)\n"
          ]
        }
      ],
      "source": [
        "print(c)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rvLfNiblACKV"
      },
      "source": [
        "Here `c` is computed using `mean` function, thus `grad_fn` point to a function called `MeanBackward`.\n",
        "\n",
        "In most of the cases, we want PyTorch to compute gradient of a scalar function (such as loss function). However, if we want to compute the gradient of a tensor with respect to another tensor, PyTorch allows us to compute the product of a Jacobian matrix and a given vector.\n",
        "\n",
        "Suppose we have a vector function $\\vec{y}=f(\\vec{x})$, where\n",
        "$\\vec{x}=\\langle x_1,\\dots,x_n\\rangle$ and\n",
        "$\\vec{y}=\\langle y_1,\\dots,y_m\\rangle$, then a gradient of $\\vec{y}$ with respect to $\\vec{x}$ is defined by a **Jacobian**:\n",
        "\n",
        "$$\n",
        "\\begin{align}J=\\left(\\begin{array}{ccc}\n",
        "   \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{1}}{\\partial x_{n}}\\\\\n",
        "   \\vdots & \\ddots & \\vdots\\\\\n",
        "   \\frac{\\partial y_{m}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
        "\\end{array}\\right)\\end{align}\n",
        "$$\n",
        "\n",
        "Instead of giving us access to the whole Jacobian, PyTorch computes the product $v^T\\cdot J$ of Jacobian with some vector\n",
        "$v=(v_1 \\dots v_m)$. In order to do that, we need to call ``backward`` and pass `v` as an argument. The size of `v` should be the same as the size of the original tensor, with respect to which we compute the gradient.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VUNYiQCOACKV",
        "outputId": "e3127c21-fce6-420d-f347-ec40cc827e7e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[-0.8642,  0.0913],\n",
            "        [-0.1666, -0.9710]])\n"
          ]
        }
      ],
      "source": [
        "c = torch.sqrt(torch.square(a) + torch.square(b))\n",
        "c.backward(torch.eye(2)) # eye(2) means 2x2 identity matrix\n",
        "print(a.grad)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dGHlkVlvACKV"
      },
      "source": [
        "More on computing Jacobians in PyTorch can be found in [official documentation](https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FnVvj4LkD15r"
      },
      "source": [
        "# Example 0: Optimization Using Gradient Descent\n",
        "\n",
        "Let's try to use automatic differentiation to find a minimum of a simple two-variable function $f(x_1,x_2)=(x_1-3)^2+(x_2+2)^2$. Let tensor `x` hold the current coordinates of a point. We start with some starting point $x^{(0)}=(0,0)$, and compute the next point in a sequence using gradient descent formula:\n",
        "$$\n",
        "x^{(n+1)} = x^{(n)} - \\eta\\nabla f\n",
        "$$\n",
        "Here $\\eta$ is so-called **learning rage** (we will denote it by `lr` in the code), and $\\nabla f = (\\frac{\\partial f}{\\partial x_1},\\frac{\\partial f}{\\partial x_2})$ - gradient of $f$.\n",
        "\n",
        "To begin, let's define starting value of `x` and the function `f`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "nDw5mV9KEeOa"
      },
      "outputs": [],
      "source": [
        "x = torch.zeros(2,requires_grad=True)\n",
        "f = lambda x : (x-torch.tensor([3,-2])).pow(2).sum()\n",
        "lr = 0.1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wt815LWdEj77"
      },
      "source": [
        "Now let's do 15 iterations of gradient descent. In each iteration, we will update `x` coordinates and print them, to make sure that we are approaching the minimum point at (3,-2):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KfwMf555EyWJ",
        "outputId": "67e2199c-61ff-4ad1-9c48-b4a646bf8bbd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Step 0: x[0]=0.6000000238418579, x[1]=-0.4000000059604645\n",
            "Step 1: x[0]=1.0800000429153442, x[1]=-0.7200000286102295\n",
            "Step 2: x[0]=1.4639999866485596, x[1]=-0.9760000705718994\n",
            "Step 3: x[0]=1.7711999416351318, x[1]=-1.1808000802993774\n",
            "Step 4: x[0]=2.0169599056243896, x[1]=-1.3446400165557861\n",
            "Step 5: x[0]=2.2135679721832275, x[1]=-1.4757120609283447\n",
            "Step 6: x[0]=2.370854377746582, x[1]=-1.5805696249008179\n",
            "Step 7: x[0]=2.4966835975646973, x[1]=-1.6644556522369385\n",
            "Step 8: x[0]=2.597346782684326, x[1]=-1.7315645217895508\n",
            "Step 9: x[0]=2.677877426147461, x[1]=-1.7852516174316406\n",
            "Step 10: x[0]=2.7423019409179688, x[1]=-1.8282012939453125\n",
            "Step 11: x[0]=2.793841600418091, x[1]=-1.8625609874725342\n",
            "Step 12: x[0]=2.835073232650757, x[1]=-1.8900487422943115\n",
            "Step 13: x[0]=2.868058681488037, x[1]=-1.912039041519165\n",
            "Step 14: x[0]=2.894446849822998, x[1]=-1.929631233215332\n"
          ]
        }
      ],
      "source": [
        "for i in range(15):\n",
        "    y = f(x)\n",
        "    y.backward()\n",
        "    gr = x.grad\n",
        "    x.data.add_(-lr*gr)\n",
        "    x.grad.zero_()\n",
        "    print(\"Step {}: x[0]={}, x[1]={}\".format(i,x[0],x[1]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8sfjBMBu59B5"
      },
      "source": [
        "## Example 1: Linear Regression\n",
        "\n",
        "Now we know enough to solve the classical problem of **Linear regression**. Let's generate small synthetic dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "j723455WwHl7",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.datasets import make_classification, make_regression\n",
        "from sklearn.model_selection import train_test_split\n",
        "import random"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "WJNK_J6v6I-Z",
        "outputId": "09e6386e-a6d4-4b81-c8d2-153f0acf9696"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.collections.PathCollection at 0x20b8e1f1ca0>"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 9528 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "np.random.seed(13) # pick the seed for reproducibility - change it to explore the effects of random variations\n",
        "\n",
        "train_x = np.linspace(0, 3, 120)\n",
        "train_labels = 2 * train_x + 0.9 + np.random.randn(*train_x.shape) * 0.5\n",
        "\n",
        "plt.scatter(train_x,train_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ng4rZmGc6oxk"
      },
      "source": [
        "Linear regression is defined by a straight line $f_{W,b}(x) = Wx+b$, where $W, b$ are model parameters that we need to find. An error on our dataset $\\{x_i,y_u\\}_{i=1}^N$ (also called **loss function**) can be defined as mean square error:\n",
        "$$\n",
        "\\mathcal{L}(W,b) = {1\\over N}\\sum_{i=1}^N (f_{W,b}(x_i)-y_i)^2\n",
        "$$\n",
        "\n",
        "Let's define our model and loss function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "QxhI4GlB6aiH"
      },
      "outputs": [],
      "source": [
        "input_dim = 1\n",
        "output_dim = 1\n",
        "learning_rate = 0.1\n",
        "\n",
        "# This is our weight matrix\n",
        "w = torch.tensor([100.0],requires_grad=True,dtype=torch.float32)\n",
        "# This is our bias vector\n",
        "b = torch.zeros(size=(output_dim,),requires_grad=True)\n",
        "\n",
        "def f(x):\n",
        "  return torch.matmul(x,w) + b\n",
        "\n",
        "def compute_loss(labels, predictions):\n",
        "  return torch.mean(torch.square(labels - predictions))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JUxwj3367gD2"
      },
      "source": [
        "We will train the model on a series of minibatches. We will use gradient descent, adjusting model parameters using the following formulae:\n",
        "$$\n",
        "\\begin{array}{l}\n",
        "W^{(n+1)}=W^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial W} \\\\\n",
        "b^{(n+1)}=b^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial b} \\\\\n",
        "\\end{array}\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "-991PErM7fJU"
      },
      "outputs": [],
      "source": [
        "def train_on_batch(x, y):\n",
        "  predictions = f(x)\n",
        "  loss = compute_loss(y, predictions)\n",
        "  loss.backward()\n",
        "  w.data.sub_(learning_rate * w.grad)\n",
        "  b.data.sub_(learning_rate * b.grad)\n",
        "  w.grad.zero_()\n",
        "  b.grad.zero_()\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idr2VEWb9rr0"
      },
      "source": [
        "Let's do the training. We will do several passes through the dataset (so-called **epochs**), divide it into minibatches and call the function defined above:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "nOuu0qpx-wAp"
      },
      "outputs": [],
      "source": [
        "# Shuffle the data.\n",
        "indices = np.random.permutation(len(train_x))\n",
        "features = torch.tensor(train_x[indices],dtype=torch.float32)\n",
        "labels = torch.tensor(train_labels[indices],dtype=torch.float32)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3zdIf6c_85Ht",
        "outputId": "6520288c-da59-4a9f-c37e-cd99779c3073"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 94.5247\n",
            "Epoch 1: last batch loss = 9.3428\n",
            "Epoch 2: last batch loss = 1.4166\n",
            "Epoch 3: last batch loss = 0.5224\n",
            "Epoch 4: last batch loss = 0.3807\n",
            "Epoch 5: last batch loss = 0.3495\n",
            "Epoch 6: last batch loss = 0.3413\n",
            "Epoch 7: last batch loss = 0.3390\n",
            "Epoch 8: last batch loss = 0.3384\n",
            "Epoch 9: last batch loss = 0.3382\n"
          ]
        }
      ],
      "source": [
        "batch_size = 4\n",
        "for epoch in range(10):\n",
        "  for i in range(0,len(features),batch_size):\n",
        "    loss = train_on_batch(features[i:i+batch_size].view(-1,1),labels[i:i+batch_size])\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We now have obtained optimized parameters $W$ and $b$. Note that their values are similar to the original values used when generating the dataset ($W=2, b=1$)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "US6q0nCBD-LL",
        "outputId": "c804b779-3231-4f6f-c854-032d211b2853"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(tensor([1.8617], requires_grad=True), tensor([1.0711], requires_grad=True))"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "w,b"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "_e6xRMZFDnyI",
        "outputId": "79e6c360-265a-401d-ce39-8f211917a13d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x20b8e30a850>]"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 14912 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.scatter(train_x,train_labels)\n",
        "x = np.array([min(train_x),max(train_x)])\n",
        "with torch.no_grad():\n",
        "  y = w.numpy()*x+b.numpy()\n",
        "plt.plot(x,y,color='red')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0giuwC9GHzi8"
      },
      "source": [
        "## Computations on GPU\n",
        "\n",
        "To use GPU for computations, PyTorch supports moving tensors to GPU and building computational graph for GPU. Traditionally, in the beginning of our code we define available computation device `device` (which is either `cpu` or `cuda`), and then move all tensors to this device using a call `.to(device)`. We can also create tensors on the specified device upfront, by passing the parameter `device=...` to tensor creation code. Such code works without changes both on CPU and GPU: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HK7HPLz3Hyrl",
        "outputId": "7e14cccb-d376-4e59-be66-4ab3f5c3f6f4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Doing computations on cpu\n",
            "Epoch 0: last batch loss = 94.5247\n",
            "Epoch 1: last batch loss = 9.3428\n",
            "Epoch 2: last batch loss = 1.4166\n",
            "Epoch 3: last batch loss = 0.5224\n",
            "Epoch 4: last batch loss = 0.3807\n",
            "Epoch 5: last batch loss = 0.3495\n",
            "Epoch 6: last batch loss = 0.3413\n",
            "Epoch 7: last batch loss = 0.3390\n",
            "Epoch 8: last batch loss = 0.3384\n",
            "Epoch 9: last batch loss = 0.3382\n"
          ]
        }
      ],
      "source": [
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "\n",
        "print('Doing computations on '+device)\n",
        "\n",
        "### Changes here: indicate device\n",
        "w = torch.tensor([100.0],requires_grad=True,dtype=torch.float32,device=device)\n",
        "b = torch.zeros(size=(output_dim,),requires_grad=True,device=device)\n",
        "\n",
        "def f(x):\n",
        "  return torch.matmul(x,w) + b\n",
        "\n",
        "def compute_loss(labels, predictions):\n",
        "  return torch.mean(torch.square(labels - predictions))\n",
        "\n",
        "def train_on_batch(x, y):\n",
        "  predictions = f(x)\n",
        "  loss = compute_loss(y, predictions)\n",
        "  loss.backward()\n",
        "  w.data.sub_(learning_rate * w.grad)\n",
        "  b.data.sub_(learning_rate * b.grad)\n",
        "  w.grad.zero_()\n",
        "  b.grad.zero_()\n",
        "  return loss\n",
        "\n",
        "batch_size = 4\n",
        "for epoch in range(10):\n",
        "  for i in range(0,len(features),batch_size):\n",
        "    ### Changes here: move data to required device\n",
        "    loss = train_on_batch(features[i:i+batch_size].view(-1,1).to(device),labels[i:i+batch_size].to(device))\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A10prCPowHl7"
      },
      "source": [
        "## Example 2: Classification\n",
        "\n",
        "Now we will consider binary classification problem. A good example of such a problem would be a tumour classification between malignant and benign based on it's size and age.\n",
        "\n",
        "The core model is similar to regression, but we need to use different loss function. Let's start by generating sample data:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "j0OTPkGpwHl7",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "np.random.seed(0) # pick the seed for reproducibility - change it to explore the effects of random variations\n",
        "\n",
        "n = 100\n",
        "X, Y = make_classification(n_samples = n, n_features=2,\n",
        "                           n_redundant=0, n_informative=2, flip_y=0.1,class_sep=1.5)\n",
        "X = X.astype(np.float32)\n",
        "Y = Y.astype(np.int32)\n",
        "\n",
        "split = [ 70*n//100, (15+70)*n//100 ]\n",
        "train_x, valid_x, test_x = np.split(X, split)\n",
        "train_labels, valid_labels, test_labels = np.split(Y, split)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "c-_BjSHPwHl8",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def plot_dataset(features, labels, W=None, b=None):\n",
        "    # prepare the plot\n",
        "    fig, ax = plt.subplots(1, 1)\n",
        "    ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
        "    ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
        "    colors = ['r' if l else 'b' for l in labels]\n",
        "    ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
        "    if W is not None:\n",
        "        min_x = min(features[:,0])\n",
        "        max_x = max(features[:,1])\n",
        "        min_y = min(features[:,1])*(1-.1)\n",
        "        max_y = max(features[:,1])*(1+.1)\n",
        "        cx = np.array([min_x,max_x],dtype=np.float32)\n",
        "        cy = (0.5-W[0]*cx-b)/W[1]\n",
        "        ax.plot(cx,cy,'g')\n",
        "        ax.set_ylim(min_y,max_y)\n",
        "    fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "tq0vFchQwHl8",
        "outputId": "919f1922-f789-4779-cbdc-4f9e742c358b",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_89704/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 27200 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x, train_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjPlpf2-wHl8"
      },
      "source": [
        "## Training One-Layer Perceptron\n",
        "\n",
        "Let's use Tensorflow gradient computing machinery to train one-layer perceptron.\n",
        "\n",
        "Our neural network will have 2 inputs and 1 output. The weight matrix $W$ will have size $2\\times1$, and bias vector $b$ -- $1$.\n",
        "\n",
        "To make our code more structured, let's group all parameters into a single class:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "J1KaixW-cMWJ"
      },
      "outputs": [],
      "source": [
        "class Network():\n",
        "  def __init__(self):\n",
        "     self.W = torch.randn(size=(2,1),requires_grad=True)\n",
        "     self.b = torch.zeros(size=(1,),requires_grad=True)\n",
        "\n",
        "  def forward(self,x):\n",
        "    return torch.matmul(x,self.W)+self.b\n",
        "\n",
        "  def zero_grad(self):\n",
        "    self.W.data.zero_()\n",
        "    self.b.data.zero_()\n",
        "\n",
        "  def update(self,lr=0.1):\n",
        "    self.W.data.sub_(lr*self.W.grad)\n",
        "    self.b.data.sub_(lr*self.b)\n",
        "\n",
        "net = Network()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rQ7W6TOacIAI"
      },
      "source": [
        "> Note that we use `W.data.zero_()` instead of `W.zero_()`. We need to do this, because we cannot directly modify a tensor that is being tracked using *Autograd* mechanism.\n",
        "\n",
        "Core model will be the same as in previous example, but loss function will be a logistic loss. To apply logistic loss, we need to get the value of **probability** as the output of our network, i.e. we need to bring the output $z$ to the range [0,1] using `sigmoid` activation function: $p=\\sigma(z)$.\n",
        "\n",
        "If we get the probability $p_i$ for the i-th input value corresponding to the actual class $y_i\\in\\{0,1\\}$, we compute the loss as $\\mathcal{L_i}=-(y_i\\log p_i + (1-y_i)log(1-p_i))$. \n",
        "\n",
        "In PyTorch, both those steps (applying sigmoid and then logistic loss) can be done using one call to `binary_cross_entropy_with_logits` function. Since we are training our network in minibatches, we need to average out the loss across all elements of a minibatch - and that is also done automatically by `binary_cross_entropy_with_logits` function: \n",
        "\n",
        "> The call to `binary_crossentropy_with_logits` is equivalent to a call to `sigmoid`, followed by a call to `binary_crossentropy`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "kdDxWeCqwHl8",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def train_on_batch(net, x, y):\n",
        "  z = net.forward(x).flatten()\n",
        "  loss = torch.nn.functional.binary_cross_entropy_with_logits(input=z,target=y)\n",
        "  net.zero_grad()\n",
        "  loss.backward()\n",
        "  net.update()\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zAAgw0h6KzUd"
      },
      "source": [
        "To loop through our data, we will use built-in PyTorch mechanism for managing datasets. It is based on two concepts:\n",
        "* **Dataset** is the main source of data, it can be either **Iterable** or **Map-style**\n",
        "* **Dataloader** is responsible for loading the data from a dataset and splitting it into minibatches.\n",
        "\n",
        "In our case, we will define a dataset based on a tensor, and split it into minibatches of 16 elements. Each minibatch contains two tensors, input data (size=16x2) and labels (a vector of length 16 of integer type - class number)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PfyqjVb2wHl8",
        "outputId": "b3a685a9-304c-4e7e-adf9-2858cc47c3a5",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[tensor([[ 1.5442,  2.5290],\n",
              "         [-1.6284,  0.0772],\n",
              "         [-1.7141,  2.4770],\n",
              "         [-1.4951,  0.7320],\n",
              "         [-1.6899,  0.9243],\n",
              "         [-0.9474, -0.7681],\n",
              "         [ 3.8597, -2.2951],\n",
              "         [-1.3944,  1.4300],\n",
              "         [ 4.3627,  3.1333],\n",
              "         [-1.0973, -1.7011],\n",
              "         [-2.5532, -0.0777],\n",
              "         [-1.2661, -0.3167],\n",
              "         [ 0.3921,  1.8406],\n",
              "         [ 2.2091, -1.6045],\n",
              "         [ 1.8383, -1.4861],\n",
              "         [ 0.7173, -0.9718]]),\n",
              " tensor([1., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 0., 1., 1., 1., 1.])]"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Create a tf.data.Dataset object for easy batched iteration\n",
        "dataset = torch.utils.data.TensorDataset(torch.tensor(train_x),torch.tensor(train_labels,dtype=torch.float32))\n",
        "dataloader = torch.utils.data.DataLoader(dataset,batch_size=16)\n",
        "\n",
        "list(dataloader)[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xrwgkbQjhkEp"
      },
      "source": [
        "Now we can loop through the whole dataset to train our network for 15 epochs:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 78,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QGchp9D6gVJa",
        "outputId": "b4c4751d-cb56-4104-d5b5-f1ae9d3d858d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.6491\n",
            "Epoch 1: last batch loss = 0.6064\n",
            "Epoch 2: last batch loss = 0.5822\n",
            "Epoch 3: last batch loss = 0.5679\n",
            "Epoch 4: last batch loss = 0.5592\n",
            "Epoch 5: last batch loss = 0.5537\n",
            "Epoch 6: last batch loss = 0.5501\n",
            "Epoch 7: last batch loss = 0.5478\n",
            "Epoch 8: last batch loss = 0.5463\n",
            "Epoch 9: last batch loss = 0.5454\n",
            "Epoch 10: last batch loss = 0.5447\n",
            "Epoch 11: last batch loss = 0.5443\n",
            "Epoch 12: last batch loss = 0.5441\n",
            "Epoch 13: last batch loss = 0.5439\n",
            "Epoch 14: last batch loss = 0.5438\n"
          ]
        }
      ],
      "source": [
        "for epoch in range(15):\n",
        "  for (x, y) in dataloader:\n",
        "    loss = train_on_batch(net,x,y)\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Obtained parameters:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5QaDiCQUkFOT",
        "outputId": "45b4a66b-1222-40f4-c758-d58f1c7daf8c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[ 0.1330],\n",
            "        [-0.2810]], requires_grad=True) tensor([0.], requires_grad=True)\n"
          ]
        }
      ],
      "source": [
        "print(net.W,net.b)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4_Atvn5K4K9"
      },
      "source": [
        "To make sure our training worked, let's plot the line that separates two classes. Separation line is defined by the equation $W\\times x + b = 0.5$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "PgRTHttLwHl9",
        "outputId": "d9abf92f-cb70-4c56-ccd0-5e027239da58",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_89704/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 31300 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x,train_labels,net.W.detach().numpy(),net.b.detach().numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1W4TZfXOmIlS"
      },
      "source": [
        "Not let's compute the accuracy on the validation dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HUjdeIefsIsg",
        "outputId": "a1a363d4-a307-4769-9ccf-fe8a857b62af"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "tensor(0.7333)"
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pred = torch.sigmoid(net.forward(torch.tensor(valid_x)))\n",
        "torch.mean(((pred.view(-1)>0.5)==(torch.tensor(valid_labels)>0.5)).type(torch.float32))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's explain what is going on here:\n",
        "* `pred` is the vector of predicted probabilities for the whole validation dataset. We compute it by running original validation data `valid_x` through our network, and applying `sigmoid` to get probabilities.\n",
        "* `pred.view(-1)` creates a flattened view of the original tensor. `view` is similar to `reshape` function in numpy.\n",
        "* `pred.view(-1)>0.5` returns a boolean tensor or truth value showing the predicted class (False = class 0, True = class 1)\n",
        "* Similarly, `torch.tensor(valid_labels)>0.5)` creates the boolean tensor of truth values for validation labels\n",
        "* We compare those two tensors element-wise, and get another boolean tensor, where `True` corresponds to correct prediction, and `False` - to incorrect.\n",
        "* We convert that tensor to floating point, and take it's mean value using `torch.mean` - that is the desired accuracy "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_95qF9lY2kHp"
      },
      "source": [
        "## Neural Networks and Optimizers\n",
        "\n",
        "In PyTorch, a special module `torch.nn.Module` is defined to represent a neural network. There are two methods to define your own neural network:\n",
        "* **Sequential**, where you just specify a list of layers that comprise your network\n",
        "* As a **class** inherited from `torch.nn.Module`\n",
        "\n",
        "First method allows you to specify standard networks with sequential composition of layers, while the second one is more flexible, and gives an opportunity to express networks of arbitrary complex architectures. \n",
        "\n",
        "Inside modules, you can use standard **layers**, such as:\n",
        "* `Linear` - dense linear layer, equivalent to one-layered perceptron. It has the same architecture as we have defined above for our network\n",
        "* `Softmax`, `Sigmoid`, `ReLU` - layers that correspond to activation functions \n",
        "* There are also other layers for special network types - convolution, recurrent, etc. We will revisit many of them later in the course.\n",
        "\n",
        "> Most of the activation function and loss functions in PyTorch are available in two form: as a **function** (inside `torch.nn.functional` namespace) and **as a layer** (inside `torch.nn` namespace). For activation functions, it is often easier to use functional elements from `torch.nn.functional`, without creating separate layer object.\n",
        "\n",
        "If we want to train one-layer perceptron, we can just use one built-in `Linear` layer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D77pXPR6oFRs",
        "outputId": "efa49e5c-72d4-4781-89d4-4ab6597d2b0e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[Parameter containing:\n",
            "tensor([[-0.0422,  0.1821]], requires_grad=True), Parameter containing:\n",
            "tensor([0.6582], requires_grad=True)]\n"
          ]
        }
      ],
      "source": [
        "net = torch.nn.Linear(2,1) # 2 inputs, 1 output\n",
        "\n",
        "print(list(net.parameters()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0tbe0Et_oiNo"
      },
      "source": [
        "As you can see, `parameters()` method returns all the parameters that need to be adjusted during training. They correspond to weight matrix $W$ and bias $b$. You may note that they have `requires_grad` set to `True`, because we need to compute gradients with respect to parameters.\n",
        "\n",
        "PyTorch also contains built-in **optimizers**, which implement optimization methods such as **gradient descent**. Here is how we can define a **stochastic gradient descent optimizer**:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "B4AxyrFMozh0"
      },
      "outputs": [],
      "source": [
        "optim = torch.optim.SGD(net.parameters(),lr=0.05)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6eB8v58eo9pp"
      },
      "source": [
        "Using the optimizer, our training loop will look like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ups7nlV22ofp",
        "outputId": "503d8ae9-35f3-4ecb-e2ff-4da2ec2914eb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.7596041560173035, val acc = 0.5333333611488342\n",
            "Epoch 1: last batch loss = 0.6602361798286438, val acc = 0.6000000238418579\n",
            "Epoch 2: last batch loss = 0.5847358107566833, val acc = 0.6666666865348816\n",
            "Epoch 3: last batch loss = 0.5263020992279053, val acc = 0.7333333492279053\n",
            "Epoch 4: last batch loss = 0.48015740513801575, val acc = 0.800000011920929\n",
            "Epoch 5: last batch loss = 0.4430023431777954, val acc = 0.8666666746139526\n",
            "Epoch 6: last batch loss = 0.41254672408103943, val acc = 0.8666666746139526\n",
            "Epoch 7: last batch loss = 0.3871781527996063, val acc = 0.800000011920929\n",
            "Epoch 8: last batch loss = 0.3657420873641968, val acc = 0.800000011920929\n",
            "Epoch 9: last batch loss = 0.34739670157432556, val acc = 0.800000011920929\n"
          ]
        }
      ],
      "source": [
        "val_x = torch.tensor(valid_x)\n",
        "val_lab = torch.tensor(valid_labels)\n",
        "\n",
        "for ep in range(10):\n",
        "  for (x,y) in dataloader:\n",
        "    z = net(x).flatten()\n",
        "    loss = torch.nn.functional.binary_cross_entropy_with_logits(z,y)\n",
        "    optim.zero_grad()\n",
        "    loss.backward()\n",
        "    optim.step()\n",
        "  acc = ((torch.sigmoid(net(val_x).flatten())>0.5).float()==val_lab).float().mean()\n",
        "  print(f\"Epoch {ep}: last batch loss = {loss}, val acc = {acc}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vRLXEQ4Qrcvx"
      },
      "source": [
        "> You may notice that to apply our network to input data we can use `net(x)` instead of `net.forward(x)`, because `nn.Module` implements Python `__call__()` function\n",
        "\n",
        "Taking this into account, we can define generic `train` function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5c6WsBhlrlIs",
        "outputId": "54de8404-4170-4a15-abba-039d06d5e946"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.48486900329589844, val acc = 0.7333333492279053\n",
            "Epoch 1: last batch loss = 0.41338109970092773, val acc = 0.800000011920929\n",
            "Epoch 2: last batch loss = 0.35756850242614746, val acc = 0.800000011920929\n",
            "Epoch 3: last batch loss = 0.31495171785354614, val acc = 0.800000011920929\n",
            "Epoch 4: last batch loss = 0.2824164032936096, val acc = 0.800000011920929\n",
            "Epoch 5: last batch loss = 0.2572754919528961, val acc = 0.800000011920929\n",
            "Epoch 6: last batch loss = 0.23751722276210785, val acc = 0.800000011920929\n",
            "Epoch 7: last batch loss = 0.2217157930135727, val acc = 0.800000011920929\n",
            "Epoch 8: last batch loss = 0.2088666558265686, val acc = 0.800000011920929\n",
            "Epoch 9: last batch loss = 0.19824868440628052, val acc = 0.800000011920929\n"
          ]
        }
      ],
      "source": [
        "def train(net, dataloader, val_x, val_lab, epochs=10, lr=0.05):\n",
        "  optim = torch.optim.Adam(net.parameters(),lr=lr)\n",
        "  for ep in range(epochs):\n",
        "    for (x,y) in dataloader:\n",
        "      z = net(x).flatten()\n",
        "      loss = torch.nn.functional.binary_cross_entropy_with_logits(z,y)\n",
        "      optim.zero_grad()\n",
        "      loss.backward()\n",
        "      optim.step()\n",
        "    acc = ((torch.sigmoid(net(val_x).flatten())>0.5).float()==val_lab).float().mean()\n",
        "    print(f\"Epoch {ep}: last batch loss = {loss}, val acc = {acc}\")\n",
        "\n",
        "net = torch.nn.Linear(2,1)\n",
        "\n",
        "train(net,dataloader,val_x,val_lab,lr=0.03)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KzuIDqJ8sFYm"
      },
      "source": [
        "## Defining Network as a Sequence of Layers\n",
        "\n",
        "Now let's train multi-layered perceptron. It can be defined just by specifying a sequence of layers. The resulting object will automatically inherit from `Module`, e.g. it will also have `parameters` method that will return all parameters of the whole network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tBtytmEAsq-O",
        "outputId": "06ad840b-c2b7-409e-e01e-a9170548151d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Sequential(\n",
            "  (0): Linear(in_features=2, out_features=5, bias=True)\n",
            "  (1): Sigmoid()\n",
            "  (2): Linear(in_features=5, out_features=1, bias=True)\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "net = torch.nn.Sequential(torch.nn.Linear(2,5),torch.nn.Sigmoid(),torch.nn.Linear(5,1))\n",
        "print(net)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5r5RbLB1s6YB"
      },
      "source": [
        "We can train this multi-layered network using the function `train` that we have defined above:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ogXKdcfIs_ND",
        "outputId": "957ccd8d-0076-4e9b-89f1-edc1de75f18e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.5835739970207214, val acc = 0.800000011920929\n",
            "Epoch 1: last batch loss = 0.4642275869846344, val acc = 0.800000011920929\n",
            "Epoch 2: last batch loss = 0.35158076882362366, val acc = 0.800000011920929\n",
            "Epoch 3: last batch loss = 0.26132312417030334, val acc = 0.800000011920929\n",
            "Epoch 4: last batch loss = 0.19465585052967072, val acc = 0.800000011920929\n",
            "Epoch 5: last batch loss = 0.14735405147075653, val acc = 0.800000011920929\n",
            "Epoch 6: last batch loss = 0.11454981565475464, val acc = 0.800000011920929\n",
            "Epoch 7: last batch loss = 0.09244414418935776, val acc = 0.800000011920929\n",
            "Epoch 8: last batch loss = 0.07805468142032623, val acc = 0.800000011920929\n",
            "Epoch 9: last batch loss = 0.06894762068986893, val acc = 0.800000011920929\n"
          ]
        }
      ],
      "source": [
        "train(net,dataloader,val_x,val_lab)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jY4R1XEGtEzJ"
      },
      "source": [
        "## Defining a Network as a Class\n",
        "\n",
        "Using a class inherited from `torch.nn.Module` is a more flexible method, because we can define any computations inside it. `Module` automates a lot of things, eg. it automatically understands all internal variables that are PyTorch layers, and gathers their parameters for optimization. You just need to define all layers of the network as members of the class:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SlsJmGu0tMsZ",
        "outputId": "240d5c89-096c-4392-99cd-1ade5ff3e3e1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "MyNet(\n",
            "  (fc1): Linear(in_features=2, out_features=10, bias=True)\n",
            "  (func): ReLU()\n",
            "  (fc2): Linear(in_features=10, out_features=1, bias=True)\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "class MyNet(torch.nn.Module):\n",
        "  def __init__(self,hidden_size=10,func=torch.nn.Sigmoid()):\n",
        "    super().__init__()\n",
        "    self.fc1 = torch.nn.Linear(2,hidden_size)\n",
        "    self.func = func\n",
        "    self.fc2 = torch.nn.Linear(hidden_size,1)\n",
        "\n",
        "  def forward(self,x):\n",
        "    x = self.fc1(x)\n",
        "    x = self.func(x)\n",
        "    x = self.fc2(x)\n",
        "    return x\n",
        "  \n",
        "net = MyNet(func=torch.nn.ReLU())\n",
        "print(net)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HwdapRxft-7M",
        "outputId": "6eb900cf-4902-4a04-c62b-497b68455406"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.7821246981620789, val acc = 0.46666666865348816\n",
            "Epoch 1: last batch loss = 0.7457502484321594, val acc = 0.5333333611488342\n",
            "Epoch 2: last batch loss = 0.7120334506034851, val acc = 0.5333333611488342\n",
            "Epoch 3: last batch loss = 0.6811249256134033, val acc = 0.6666666865348816\n",
            "Epoch 4: last batch loss = 0.6533011794090271, val acc = 0.7333333492279053\n",
            "Epoch 5: last batch loss = 0.627849280834198, val acc = 0.7333333492279053\n",
            "Epoch 6: last batch loss = 0.6030643582344055, val acc = 0.800000011920929\n",
            "Epoch 7: last batch loss = 0.5775002837181091, val acc = 0.800000011920929\n",
            "Epoch 8: last batch loss = 0.5522137880325317, val acc = 0.8666666746139526\n",
            "Epoch 9: last batch loss = 0.5250465869903564, val acc = 0.8666666746139526\n"
          ]
        }
      ],
      "source": [
        "train(net,dataloader,val_x,val_lab,lr=0.005)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvAiaj_JndyP"
      },
      "source": [
        "**Task 1**: Plot the graphs of loss function and accuracy on training and validation data during training\n",
        "\n",
        "**Task 2**: Try to solve MNIST classificiation problem using this code. Hint: use `crossentropy_with_logits` as a loss function."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Takeaways\n",
        "\n",
        "* PyTorch allows you to operate on tensors at low level, you have most flexibility.\n",
        "* There are convenient tools to work with data, such as Datasets and Dataloaders.\n",
        "* You can define neural network architectures using `Sequential` syntax, or inheriting a class from `torch.nn.Module`\n",
        "* For even simpler approach to defining and training a network - look into PyTorch Lightning"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "celltoolbar": "Slideshow",
    "colab": {
      "collapsed_sections": [],
      "name": "IntroPyTorch.ipynb",
      "provenance": []
    },
    "interpreter": {
      "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
    },
    "kernelspec": {
      "display_name": "Python 3.8.12 64-bit (conda)",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "livereveal": {
      "start_slideshow_at": "selected"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
