microsoft/AI-For-Beginners

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "En2vX4FuwHlu"
      },
      "source": [
        "## Simplest Introduction to Neural Networks with Keras\n",
        "\n",
        "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n",
        "\n",
        "### Neural Frameworks\n",
        "\n",
        "There are several frameworks for training neural networks. However, if you want to get started fast and not go into much detail on how things work internally - you should consider using [Keras](https://keras.io/). This short tutorial will get you started, and if you want to get deeper into understanding how things work - look into [Introduction to Tensorflow and Keras](IntroKerasTF.ipynb) notebook."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "8cACQoFMwHl3"
      },
      "source": [
        "### Getting things ready\n",
        "\n",
        "Keras is a part of Tensorflow 2.x framework. Let's make sure we have version 2.x.x of Tensorflow installed:\n",
        "```\n",
        "pip install tensorflow\n",
        "```\n",
        "or\n",
        "```\n",
        "conda install tensorflow\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xwqVx9-bwHl3",
        "outputId": "2aa591b4-b647-441f-9c8e-4e0da2d517a0",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Tensorflow version = 2.7.0\n",
            "Keras version = 2.7.0\n"
          ]
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "import numpy as np\n",
        "from sklearn.datasets import make_classification\n",
        "import matplotlib.pyplot as plt\n",
        "print(f'Tensorflow version = {tf.__version__}')\n",
        "print(f'Keras version = {keras.__version__}')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "6tp2xGV7wHl4"
      },
      "source": [
        "## Basic Concepts: Tensor\n",
        "\n",
        "**Tensor** is a multi-dimensional array. It is very convenient to use tensors to represent different types of data:\n",
        "* 400x400 - black-and-white picture\n",
        "* 400x400x3 - color picture \n",
        "* 16x400x400x3 - minibatch of 16 color pictures\n",
        "* 25x400x400x3 - one second of 25-fps video\n",
        "* 8x25x400x400x3 - minibatch of 8 1-second videos\n",
        "\n",
        "Tensors give us a convenient way to represent input/output data, as well we weights inside the neural network."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "A10prCPowHl7"
      },
      "source": [
        "## Sample Problem\n",
        "\n",
        "Let's consider binary classification problem. A good example of such a problem would be a tumour classification between malignant and benign based on it's size and age. Let's start by generating some sample data:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "j0OTPkGpwHl7",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "np.random.seed(0) # pick the seed for reproducibility - change it to explore the effects of random variations\n",
        "\n",
        "n = 100\n",
        "X, Y = make_classification(n_samples = n, n_features=2,\n",
        "                           n_redundant=0, n_informative=2, flip_y=0.05,class_sep=1.5)\n",
        "X = X.astype(np.float32)\n",
        "Y = Y.astype(np.int32)\n",
        "\n",
        "split = [ 70*n//100 ]\n",
        "train_x, test_x = np.split(X, split)\n",
        "train_labels, test_labels = np.split(Y, split)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "c-_BjSHPwHl8",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def plot_dataset(features, labels, W=None, b=None):\n",
        "    # prepare the plot\n",
        "    fig, ax = plt.subplots(1, 1)\n",
        "    ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
        "    ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
        "    colors = ['r' if l else 'b' for l in labels]\n",
        "    ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
        "    if W is not None:\n",
        "        min_x = min(features[:,0])\n",
        "        max_x = max(features[:,1])\n",
        "        min_y = min(features[:,1])*(1-.1)\n",
        "        max_y = max(features[:,1])*(1+.1)\n",
        "        cx = np.array([min_x,max_x],dtype=np.float32)\n",
        "        cy = (0.5-W[0]*cx-b)/W[1]\n",
        "        ax.plot(cx,cy,'g')\n",
        "        ax.set_ylim(min_y,max_y)\n",
        "    fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "tq0vFchQwHl8",
        "outputId": "9a5aa6a0-c92f-4d72-9e78-c0f615804bff",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_103052/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABOsElEQVR4nO2dd3zddfX/Xyc3u0maNLM7pZRSSmkLoYBlz7ItoCBDv4LwU0EFVETBAYii4EJBRBTEgYLsMovMQoEW6KQFOtLdJk2bZs97fn+87od7k9zZ3J3zfDzuI7n3fu7nnrve5322qCoMwzAMIyPRAhiGYRjJgSkEwzAMA4ApBMMwDMODKQTDMAwDgCkEwzAMw4MpBMMwDAMAkJloAQZDWVmZVldXJ1oMwzCMlOK9997bqarl/W9PaYVQXV2NxYsXJ1oMwzCMlEJENvi73VxGhmEYBgBTCIZhGKlFby/Q2QnEoMtESruMDMMwhgzr1gEvvQS88w7gdgPDhgEnnwwcdRRQUhKVpzCFYBiGkey88grwt78B2dnAqFGAywV0dACPPw7Mnw9cdx0wduygn8ZcRoZhGMnMxx8D999PReAoAwDIzQXGj+f/v/kNFcQgMYVgGIaRzDz7LFBQQOvAH6WlwK5dwJIlg34qUwiGYRjJSmsrsGwZUFYW/LiiIuCNNwb9dKYQDMMwkhXHDZQRYqnOzgaamgb9dEmjEEQkV0TeFZGlIrJSRG5KtEyGYRgJZdgwQISppsHo6KDraJAkjUIA0AngeFWdDmAGgDkicnhiRTIMw0ggubnA4YcDdXXBj2tpAY49dtBPlzQKQUmL52qW52LzPQ3DGNrMmQP09DCe4I/t24ExY4CpUwf9VEmjEABARFwisgRAHYD5qvqOn2OuEJHFIrK4vr4+7jIahmHElbFjgW99C9izB9i4kYqhqwvYvRtYvx4YMQK45hogK2vQTyUag/LnwSIixQAeB/ANVV0R6Liamhq15naGYQwJGhqAhQuZTdTeDlRUACedBMyYAeTkRHQqEXlPVWv6356Ulcqq2igirwKYAyCgQjAMwxgylJYCZ5zBS4xIGpeRiJR7LAOISB6AEwGsTqhQhmEYQ4hkshBGAvibiLhARfWwqs5LsEyGYRhDhqRRCKq6DMDMRMthGFGnrg5YsABYtAjo7mb/mRNOACZP9valMYwkIGkUgmGkJa+9xi6VIswGycwEVq0CFi9mMPCrXwXy8hItpWEAMIVgGLFj2TLgL38BRo/umwWSm8vhJkuXAn/9K3DllYmTEWAV7McfA9u20WIZNw6orqYSSzZ6e6lQP/mEMwHGjwemTYs4y8bwjykEIyBuNyviRbiGJeP6kLSoAo89xsEl/hYrES5mixYBW7ZQaSSClSuplHbt6juBa9w44PLLo9JjP2p88gnwxz8y/z4jg5fubiA/H/i//wNmzUq0hCmPKQRjAO3twFtvsevurl28bfx44LTTgEMOMbd3WGzfDmzYwIU1ECJ8M999F5g7N36yOaxYAdxxB11ZTl99gIqhoQG49VbgxhtZBZto1q0DbruNXT19ZQWAtjbgD38ArrrKlMIgSZq0UyM5aGoCfv5z4MEHvZvYceO4Kfv974F772UVvRGC5mYu9qHMqpwcIBEV9z09wH33Mbe9qKjvfSJstywC/POf8ZetP6rAP/7BWMvw4QPvz88HqqoYq+nqir98aYQpBKMP990HbN0KTJjAmRwA14WSEmCffYC336blYIQgJ4c+t1B0dw9ckOPBqlVAYyNQWBj4mIoKHrdtW/Sfv60NWL4ceP997v6DvVebN/OYYN088/PZ0mHlyujLOoQwl5HxKVu2MA7a3yJ3EOEEv+efB045xeJ4QRkzhlq0pcWrWfujyh3tIYfEVzaAPXFC9dgX4THbtgEjR0bneTs6gCeeAF5+mVaKCJVBZSVw/vnATD+Z53V1lCOUteVy8Uvs7xxGWJiFYHzK8uX8zQX73eXk8De9bt3A+3p6uP6Z1Q4uTmeeCezYEXj3u2MHza59942vbJEQzV5nXV3A737HHUV5OX2RY8dyB9LVxbnA/qZ+ZWSEJ4fbbQGuQWIWgvEpLS1Mkw+FSN953vX1TKh57DHGIFwu4KijgC9+EZgyJXbyJj1HHw1s2gTMn0/f94gRfPPa2rjrraxkIDQR6VsTJoR2abndXIijlQH16qt06UyYMPA1FxVx6tcDDzCNtLjYe58TmHe7g1s1qlSwxl5jFoLxKSNG0KUdCrfb63r+6CNa+nfcwcSa3l5mKT32GHDuucDf/x5bmZOajAzg4ouBb3+bLqQNG3jp7OSb9sMfRmXK1V4xeTKfe8+ewMfU1XFxrqwc/PP19gLPPcdzBVKAubn8cr3Tr+t9aSndasFiGbt20a01adLgZR3CmIVgfMrMmUzm6O0NbHm3tFBx7LMPY3hXXsnf6ahRfX/nhYW8/5ZbWON01FFxeQnJhwgwfTovnZ30q+XlhfbfxxqXC7jiCuCXv+QHXlLi/QBVqQwyM4GLLorO8+3aRfMxVF1DURGwZAmDVL5ceCF7/2/ezIXf+YI6srrdwLXXJv59TXHs3TM+paQEOP54bmL9eRO6u+n2Pvdc/u6efx6orWUyir9N37Bh/N3+4Q8xFz01yMnhm5Isi9bkycD113MRdqwX51JdDdxwA9M5o0E4GVcA3xt/x44YQXkOOYSB440b6Y7buJG7kxtvpMzGoDALwejD+edzZ//WW7Tgi4u5Cdu1i5vbL3wBOOIIHvvf/3KNC+YCLytjZmF9PeOIRpIxaRLNuPXrqe1FuIuPduV0SQknenV2Bk9Pa2oCPvMZ//eNGAF87WvABRdQGajSBRUtpWWYQjD6kpVFT8LxxzMzcM0a7vJPPJEx0lGjvMfW1YVOPXVqs3bsMIWQtIhwlx3LgGx2Nqd7zZsXOK+5t5eXUP7FkhJejKhjCsEYgAg3jqHic0VFVArB6O3l+QKl4htJjir9gmvX0kSsquIw972Z33viiaxs3LaN5/E1Lbu76f4580zb8ScQUwjGXjN3LnDzzcGPaWykazeZeqQZYbJ1K/CnPzGmAHj9+wUFzJ46/PDIzjd8OGMWf/oTu6s6vZx6ehjAPu88KgQjYZhCMPaauXOBO+9kHzR/2ZMdHbx8+ctWL5Ry7NgB/OxntBDGj++7m29rA+6+m8ohkL8/EKWlwPe/zxjAypXMUS4vZ4rbYM3I9nZ+GTMyeM69sWKGOKYQjLDZs4c923JyGCwuLATuugv4f/+PXoCiIt7X28vjurqAz3+eGz8jxXjkEX6AvkEjh/x83v63v3Ehj3TAjwiLzYJ1go2E3bvZYOu11/jlc2Q85RS6qXJzo/M8Q4CkUQgiMhbAgwCqALgB3Kuqv0usVAbANhVPPcU+R47XYOxYWvezZjHb6J57WIi6Zw+tgf32Ay65hMeEU/1sJBG7djE1LFjb69xcZgx98EHkVkI0qatje949e1if4FgF7e3Aww9zCNG119pUujBJpp9qD4Bvq+r7IlII4D0Rma+qHyZasKHMe++xjiAvj+uD01Zmzx62pZk7l5c77uBGrbGRCmDkSFMEKcv27d7GdsHIyWG6aqIUgioH5rS3D7Q28vLYImPNGpbNR6vALs1Jmp+sqm4DsM3zf7OIrAIwGoAphASxaxd3/uXltMAdRFifUFDAxpX77QcceKBlA36KKs2qxYvpOystpSmVqKlokSISXjM51cQW2dXWUiGFas/76qvAZz/LokAjKEmjEHwRkWoAMwG84+e+KwBcAQDjouWDNPzy1lt0yfoqA18yMxlHeP55KgQDNJPuuos708xM5t93dABPPgnU1ACXXZb87ovRo7mYOtk/gejqYrVzoli1KnR73qwsfonXrWNfJiMoSacQRKQAwKMArlbVpv73q+q9AO4FgJqamij25jUAJmm8+y5bxjz9NC2BYE0mS0s5iTFUAeqQoLWVvYEaGgZm5qjS/9bRAVxzTXKnXRUVAbNnA2++GThf2JnzkMidQEdH+H5JG/MXFkmlEEQkC1QG/1TVxxItz1Cit5eu1uee4/XcXKafr1vH2eazZvl3B/nOOh/yCmHhQu+4uf44mTXLl3Nnm+wm1TnnUM4tWxgQcnYEqgwUNTUxWJudnTgZKytDt+dV5cV8mWGRJF22ABERAH8BsEpVf51oeYYa//0vM4lGj+a6VVHBdSAvjxbCm2/SHd6fjg4ek+xekJijSm1aURH4GKdke/78+Mm1txQXAz/4ATBjhreJ3MaN3CUUFrLALNEumBkz6BIKphSamhhHCBRnMPqQTBbCbACXAFguIks8t/1AVW2Cb4ypr2ccoLq6rydjwgTWFxQX00Pw0Ud0g/tSVwecdVZye0DiQmcno/ChFh6ns2gqUFLC/uYNDQzg9vYyw6C6OjFDffozbBiDxf/+N9/3/u6j9nZ+JpdemhzypgBJoxBUdQEA+9QSwNtve7sI+FJezgK0Xbu4sd2yhZ4Op85n505uFo85Jv4yx5yeHrp/uru5MI4YEfx43/78wRaf3t6Qfm+3mzHpJUtYFFxZCRx6KD+LhFBamrhBPqE49VQGt596ymuBqXIHk5VFhZZoSyaFSBqFYCSODRv8Z+RlZACHHQYsWkQroqODHQ3y8rhQVVQAV1+dZu7Znh7gpZfo/mlu5pvQ2wscdBALLgL13M/KYtO32trgK3dDA3DyyQHvrqtjktKGDdQbWVl83//zH3ag/cIXrCNDHzIy+LkcdRTT4pz2vAcdRC1qXRUjwhSC8Wlmnj+ys1l3tHs3i1JHjmSM4YgjgAMOSLPFqaeHhU7vvssX6mg6t5vN2G65BfjOdwIPij7lFOD22/k4fz60ri6e6+ij/T68sRG47TZ6OvrrHbebeqqri5mr5gHpR1kZfZfGoEiaoLKROGbO5I4/ECK0CmbMAG66CfjqVzkRMq2UAQC8/jqVwT779C2+yMig36a4GPj977li++PAA6kUamvpsnBwMnM2bWKX0JEj/T78f//jYf5GGGdkUEksWJA6IQgj9TCFYGD6dFrWTQOqPogqg8tz5qShEnDo7eXwlmBD4AsLqTk/+MD//SL06XzlK7y+YYM3Q6ewEPj2t4ETTvD70K4uJh8Fm2efkcH3//XXI3hdhhEB5jIykJMDfOMbrKlyuhE7aecdHVQG06Zx4FUgVBl03raN6+Lo0QE3wslJfT2356Gq3wsK2JIiUP+ejAxG2Y88ktV9HR18zKhRQf08TnfYULUcRUXs1mAYscAUggGAHQh++EMWpy1f7m1il5cHnHsurYNANUgbNwJ//zvd7BkZXPfcbsYYLrkkRRRDT094fXlcLq7c4RwXQe67836HSlJyu61poBE77KtlfEp1NYtPd+5kEDkzkzv9YMWoGzZwjkpmZt/0dKe/209/Ctxwg/+2+klFcTGFDtW/p7U1JkVOxcWcHNncTCsgEI2NDFMYRiywGIIxgLIyzlOeMCG4MlAF7ruPx5SX993ZinCBc7uBBx+MvcyDpqCAbqAdOwIf43ZTYcyeHfWnFwFOP52eq0CNRtvbaXhEOrnSMMLFFIKx16xbx5hpsJqtigpWOG/ZEj+59pozzmDUdvfugfe53cweOu64mJk7RxzBxX79+r6JTKosDty+nfHq4uKYPL1hmMvI2Hs2buTfYD5v575Nm1JgHEBlJfC973FQdG0tS7IzM5lZ5HZzHOOFF8bs6V0ujiOtrmYrkfp6bzymupr3TZ0as6c3DFMIxkDcbu5Sm5roDpo40f9YWn+uDbeb1bZbtrDrQ2Ghd1FLCcaPZ3XYhx9621WPGcOS7WA5oVEiM5Ouo5NPpsLt6gKGD2dg3orRjFhjCsH4FFXWZT3yCDssOAuQ2801MS+Pt02cyE4B/bOHmprYF6mtjQuby0WXfHMzq2wPPjhF5p1nZbE4Y/r0hIowcWLCnt4YophCMD7lpZeYPlpe7k2k2bCBdVgLFlApTJ8OrF3LVj8nn8yeZ3v2cAFbsICP6e/jHjWKj/nzn4GrrrKdrmEkKxZUNgDQzfPQQ1z0Cwt527ZtwPvv8/ro0QxsNjdziNbYsVQK48czFXLFCrqIfDs+dHTwthkz6AN/7z0qBsMwkhNTCAYA7u5FvGmmqlzk8/Pp/hHh/598wvucuqtly9hsbfNmZmQ2NnovLhcLdkeM4ONzcoDXXkvgi4wmqizY2LaNtQnJQlMTte769dTIhhEB5jIyALD3vq+rp7GR65zvbTk5XG+cFguZmVQCu3cDhxzCGIPTD6m4mO4kX/dQQQGzjVIat5uBknnzqAyc0uzPfIa9+RNVll1fDzz5JMd4OmRmsnfS6af7729uGP0whWAA4Drn27mho2Ogr9+3CtkhM5PKA2DsIdgEyd7exI7gHTRuN/C3vwEvv8wXO24c35SeHi7E77wDXHdd/KPB27axXLy9nQEbp/V2Vxf9esuXM53WZgMYITCXkQGAa9iePd7r/tr5d3dzQfdd1N1uWgLjxvV9vD/27EnxKtuFC6kMJkxgfwnfNCwRmko/+lHf1texRhW45x5q2zFj+n5w2dn0623ZAjz8cPxkMlKWpFIIIvJXEakTkRWJlmWoceyxHAvs1As4s2F276Y3oqGBQeUJE7yu84YGrkMHHkivREND4HqD1lauT4ceGpeXE31Ugaef7tsKtqeHQZTnn2e+7scf00r48pc53CBQD4posm4dU8HKywMfM2oU8OabgfubG4aHiF1GIjIMQIeqBpixNSgeAPAHAKnQ/SatqK6mUnjlFe72N2/mjr6xkSmlPT287NrFXkfZ2d4xmlu2ALNm0V39v//xNidTye2mAmlvB665xnt7ylFfz1SssWN5vbeXSmDHDlaOOUoiO5ta9IEHuADPnRtbuT76iNZJsFzezExvtWECayuM5CekQhCRDAAXALgIwKEAOgHkiEg9gGcB3Kuqn0RDGFV9XUSqo3EuIzJE2Ko6Px+4/35mEw0bxlhCe7s3A0mESsGZuz5tGnD33UxH/eIXOWxs3jxW2YpwkzxtGnD22bwvZenu7rvwbt7M5kIlJX0XYyfIPH48g7yHHkpXTqzo6vLv3+uPSOA5qYbhIRwL4RUALwH4PoAVquoGABEZAeA4ALeJyOOq+o/YiWnEg8xM4LTTgGee4WLvjAjYts07IwHg7Tk5nAOTk8OMooceYr3B0Ucz1XT7dm/bBcf9lNIMH86/vb18IxyN2X9n3tVFMygzk5c33uAUtVgxciQ/kGCoeoM9hhGEcBTCiara3f9GVd0F4FEAj4pI3AYrisgVAK4AgHGhplsZEbN4MTeczm5elZvhSZN4u1OD0NLidZFnZ3ONXLAAOOcc/p/08w8ipaCA/YwWL6bPrKXFqyQcnIXX+V6WlFCTxlIhHHQQtXJnZ+Bxa42NdHXZ78UIQcigsqp2i8j+InKCiPTJWxOROc4xsRLQjzz3qmqNqtaUBwukGXvFunWsJ3Do6uLal5PDWEJ2NhWCCGMMDQ2MEeTksJYhrTn9dC76TU0D/faqXHhHj+6rKGIdWM7LAz7/eWptf5PcWlv5QV14ofUMMUISTgzhmwCuBLAKwF9E5Fuq+qTn7p8BeD6G8hlxJiOjb6aQs575riW9vYwjvPEGlQRAF/vIkSw8c+Kuacfo0cB3vwv87ndcfPfsoSbs6uKbNno0MHOm981qbKT/LNYcfzw/lP/8h3Lk5/ODa2tjN8GrrwamTIm9HEbKE47L6HIAh6hqiyfg+18RqVbV3wGI6pZDRB4CcCyAMhHZDODHqvqXaD6HEZypU/u2l3DqDrq7ufj39jJg3NJCt5BTk7BnD4+55RbWQKVtp8799gPuuAO49152Ahw2jJpw/HhaBo4y6O2lojj22NjLJMJOg4cfDixaBKxZQzNu6lQqqJRoMWskA+EoBJeqtgCAqtaKyLGgUhiPKCsEVY2hs9UIh+nTuca1tNBtnpEB7LsvsHIlg8cNDcwoqqzs2/fI7WZQGQD+8Afg9tvTeBh8Xh6n1TQ308c2ZkzfMu/ubmrNOXPi67cvKmLu7wknxO85jbQinMK07SIyw7niUQ5nACgDMC1GchkJIicH+NrXvIVoqtz8FhXx+o4dVBhlZTze7aZnZMIEbpCLi3l95coEvoh4kJvLwopZs+gnq61lgVhtLWsWPvc5BpPNb2+kEKIhgl4iMgZAj6pu93PfbFV9M1bChaKmpkYXL16cqKdPaz7+GPjXv7i+ZWTQ+7FsGRVCZaW31kmEFsSUKd5N8tatTEm96KKEvoT4oMrisJUr+QaMGUNXjW8fcMNIMkTkPVWt6X97SKNeVTcHuS9hysCILfvtB/z4x0xe2bmTLumuLrrPMzPpFcnLA6qqBrqoMzJ4f1rjjJd7+mmWartc1JDTpzPfP6Wr8IyhSrp6eY0oIOIdhgNQMeTl8XpGEGdjZ6d34lpaosqMnmeeoe/M6XrqdgOrVzP/9lvf8gZVDCNFSKrmdkZyU1bGNhT19YGP6e6msqgZYIymEcuWAc8+ywZQvplFGRn0p5WXA3fd5e0LHglud3ya4hmGH8K2EEREwH5G+6jqzSIyDkCVqr4bM+mMpOO884Cf/pRrXXEx16/6emY61tczO+nooxl7mDo1uCWRsjz7LBVBoB5C+fl8M955BzjllNDna27m0J0XXvBW+X3mM6wvSNuijhiiykaEXV3MhuhfUW4EJGRQ+dMDRf4IwA3geFWdIiIlAF5U1YQ1NLagcmJYu5appQ0NzLqsr+cmOTeXweWSEiqG2bM5XjOt0k/b2oCvf50+sWAZRE1NXIx+8pPg59u+HfjlL9khtayMyqSnhwtaTw9baR99dFRfQtqiSsX69NNswOVysR5kxgzgrLOYCmcAGERQ2YfDVPVgEfkAAFR1t4ik8vwrYy+ZOJF1Br/6FTsq778/rYVRo7ztdMrK2IK/rAw499yEihtduru9vTuCkZnJYEowurqAX/+aLWV9gy5ZWax67uwE/vIXFr5NmjR42dMZVabFPf984LjONdew95MRkEgUQreIuAAoAIhIOWgxGGnExo3Aq68ygaa7m+vSKaew4NV3UlpHBxt+HnOMfwsgI4Pejuef56jhtMnCHDaMb0SwZnIA3UBTpwY/17JltASqq/3fn5PDN+7ZZxmkHiz19ZQrJ4dKJp38ee+/zy9bdXVfV54T12ltZVzn9ttpuRl+iUQh3AngcQAVInIrgPMA3BgTqYyE8OKL3GRlZnLITUYG3UJ3380symuu8f6WPvyQ1ngwd5AzWGfVKuCQQ+LzGmJOZiZw4onMMApUhez0ETr++ODneuON0HOOy8uBpUu9peN7w+rVwGOPUYM7zaoqKoAzz6RfL9WL51SpNEtKAsd1hg1jfObdd/n5GX4Ja4vgCSi/DuA6AD8HsA3AZ1X1kRjKZsSRJUvYmmfUKNZWZWdz7Sspoet10ybgj3/0JsC0tfH/nh5aC8Fa8re1xeUlxI/jjvMuMP1RpZl14IHA5MnBz+M0xwuGM3CnvX3vZF24EPj5z+lTHzeOZtv48TT//vQnDrJI9aymlhYGs0IN3iguZozBCEhYFoKqqog8oaqHAFgdY5mMOKMKPP44MGJEX7eQL6NH0ypYv57Wwp49rGZeupT3i1CRTJw4MKkjbdxFDiNGANdfD/z2t0ynys+nOdTWxoV22jT2/wg1yaykhG6c/nNFe3rokhKhwlDduzdx507gvvvoHupfPVhURKX2/PNUXqnsW+/pCT1GFODnESquM8SJxGX0togcqqqLYiaNkRB27OCmNlgfNmeE5rvvcp155BHvfGWnjcWWLbQkZs3iGuR0SE3LzsujRwM/+xlbVrz9NneplZV0wUyYEJ4b5phjgA8+oFsIoEJZu5ZKxrcH+bHH7l3H0jffpDIJ9FiXi4rhuedSWyEUFPDL2dUVeEcD8DPad9/4yZWCRKIQjgPw/0RkA4BWsNOpqmoKf5MMgPE2xzMRjJwcWubz53M9dALLxcV8fGEhlcCiRVzDGhqY7Zd2FoJDVhZTGve2InnqVProtm/nbn3BAi5qBQVcrLu7WfCxfj3bbV9xRXjzkx3eeYfWTDBKSxlj6OhI3TbZWVl0473wQvC4Tns7jzMCEkmawakAJgI4HsCZYMfTM2MhlBFf8vNDF8i2t7Np3ZIl3rkrU6bQTdTYSKXidnO9am/npMkjjwTOPjteryIFycpipL6wkBH9zk7+r0qfXFsbcMQR7I+0cCHw0kuRnb+rK7QC8Z3fkMqceCK/yA0NA+9z4jrTp1v6bgjCthBUdUMsBTESR1UVd/xNTQP9/x0dwIoVbHLnzFHOz+emdto0Zg+NG0dPhxNjHTmSl8suS6/MxphQXg5ccAFdTy0t1KyZmXRtjB/vzSyqqmImzQknhF/pN3YsO7H6zkTtT3s7nyPYMalAaSnjOr/5jf+4zowZtLDsCxmUSFpX/Mjf7ap6c/TEMRKBCHfyd95Jz4Wz3nR0MDOyvd3b2bmxkW7n1lbeN3s2XeeVld7zud20Juy3FyYffsg3d9SowMfk51Pjbt4cuG6hPyeeyPx81cD+wLo6Vg6mw4c1Zgxw223cwbzzDr+kTlynujr102vjQCQxhFaf/3NBl9Gq6IpjxBpnrsH8+cxEdNrmzJ7NWe3//S/dQRUV3LQ2NlJBDB/OCY2vv86kjvx8ejgWL+b0Rt/1pLXVO0DHCIO2Nu9w6mBE2ld8//1pxq1c6a3c9WXHDsYY0qk1RlYWqyhnzky0JClJJC6jX/leF5E7ADwVdYmMT3FcyT093JUHS6AIh6YmWtTr1nGBLyjguZ9+Gpg3D7j8cuCHPwT+9z8mqKxeTcWw337cvGZlMeV0xQrKkpNDhVFXR4+GQ0MD8MUvDk7WIcXIkUzfCoYq/fzFxfx/3Tq+8S4Xd78VFQMf43Kx79Kf/8xspsxMavvubmrzMWOAb37TKneNTxlM27F8AFGdAiIicwD8DoALwH2qels0z58quN3M1Jk3j6mcABfg44+nFyBU4kigc/7+90wL9e3xlZ3NzWN7OwvPfvhDjgs+8kgOw+k/12DsWHY2bWujleBUMzsKob6e8s2a1fdxe/bQtdvbS+th7Fiz4D/lsMNYCOJ2B3bdNDRwx19fz/5H27b1fQOnTwcuuYQL/vr11PQjRvCN/uY3+cEvXEi3U0EBP6D99ossa8lIeyKJISyHp48RuGCXA7glWoJ4+iTdBeAkAJsBLBKRp1T1w2g9RyrgdgP3389+QqWl3oWzq4s1RAsWMHYWzN3sj08+4SXQ4Jq8PMYPnn6aiS+q/tcmx8W0cCGtA6dSedcutskpLQW+/W1vLLS5GXj4YVocvq9x/HjgwgtDF/MOCSor6bZ55RXu9vu/8a2tDDhPncrOqMXFfbutut30A158MbVtVhbv6+3lcRdcwJSwYIUmhoHILIQzfP7vAbBDVYM0LIiYWQDWqOo6ABCRfwM4G8CQUgivvUZlMGFC33XB2cnv3An87nesiYpkc7dgARfzYLvysjJg+XLu5isruc7427QWFTHZZds2eiKcTqfHHceNqpPO3tLCGN+2bcxicuRVZbfn224Drr2Wbu4hz8UXcwFfsICunaIiXm9q4od/1VXA3/7GD6l/T6PeXloFjjl52GH8q0qt/YtfAFdfbRPcjJBEklrwdVXd4LlsUdUeEflFFGUZDWCTz/XNntv6ICJXiMhiEVlcH2x0VwrS28sdelVVYM9BWRldx6siDOc74y+DkZHBS0sLsyGnTeNz+SMzk8cceihd1Ndfz3XIt7bpqaeYbTRuXF/lJUJvRmkpcM891k0AAHf1l10G3HwzK5hLS6lFL7mEvruMDJpb/hrcrVlDE62qivnAHR28XYTtMcrL+UbvbT8kY8gQiUI4yc9tp0ZLELDyuT8DSqVU9V5VrVHVmnKn5D9N2LqVG7phw4Ifl53N7J5IKCqi2ykYTtzSWdTPP59//dX6tLdzQ3rxxf4LXNvaaOkEc20VFNAb4vRDGvKIUHtecglwww3AddcxcFRYSH+fv6yC3l4GmAsLvbuI5ua+xwwbRiXx3nuxfw1GShNSIYjI1zzxg8kissznsh7A8ijKshmA77zAMQC2RvH8SU9XV3jp4FlZXEgj4TOfCd11dPduNqdzgtajRwPf/z4X7tpaFntu3sz/m5uBr36VwWd/bN3KtSpUNmVuLtPwjRD49jbypaWFgRzfYjV/JefDhtEfaBhBCCeG8C8Az4Ftr6/3ub1ZVXdFUZZFACaJyAQAWwBcAODCKJ4/6Rk+nItosDoigLvzkSMjO/fUqfQo1NX5z1B02uZcdlnf5x4/Hrj1Vm9QureXu/6DDgreudlpjb1xI93gLhc9F6Wlfc/vDLUyQrDPPuzV0x/fL4ujCPp3TwXsjTbCIqRCUNU9APYA+IJnjvIksDANIgJVfT0agnhiElcBeAHMYvqrqq6MxrlThbIyJoNs2OBtgNkfVf6uDz88snNnZjKu+Mtfes/v9DCqq6OSOf98/00vMzKYDRRuRpAqN6Nvv00LwOmG+tFHtDYOPdTbIqOjw0bdhoUTrXfyfR3y871fitZW7hT8BYva2qhUDCMIYccQROQr4JCcFwDc5Pn7k2gKo6rPqup+qjpRVW+N5rlThXPO4e/an3tHle6aI46IPO0UoIXwk5+wIrmzk4ph82YGj2+4ATj99OjUBrzwAtPqJ0ygZVBYSAVQXEy32Btv0OXk9F479NDBP2fak5sLXHopg8a+/sLsbOYmO0VqBxww8LFOdXOkuwhjyCEa5rQkTxzhUABvq+oMEdkfwE2qen4sBQxGTU2NLo40upoCvP8+k0K6u7mQulzeBXTWLLp1Qg3aCoUqz5eZGd3apKYm1jFUVvL8r7/ubYjnKBunXqGqCvjSl5jCaoTJ4sUcbdfU5HURdXQw7XTKlL71CQA/hE2buAs405oTG0RE3lPVmv63R1KH0KGqHSICEclR1dUiYmVFMeDgg5lp+M473mH3Bx7IbMRwZ6+EwhnGFW0WL6b3IjublyOP5G179vD+jAzGITZsYGp9qLHDRj9qalhPsGoVexG5XPxSZGdzF7FxIz9cZ55CVhYrAOfMSbTkRgoQiULYLCLFAJ4AMF9EdmOIZQHFk6Ii4KSTeEkl1q/v68IuKmLB2u7drIVwuxlH6O7mhtbaV0RAczMb1e3ZwzexpoZ+OIebbmIf8lWr6BOsqmKTt1B5zIbhIZLmdnM9//5ERF4BMBzA8zGRykgYTlp7UxMtiIkTI2uV73INTGZxCtF8ezBt3JgeHZfjQnc38NhjbFHrzA92eoscfTTwhS94y9D33dfGRBp7TSS9jATARQD2UdWbRWQcgBkAQrRpNFIBVWYF/fe/LHp1du5ZWbRSzj47vA7NBx7IgrRgdHXxXHsTGB9yuN3AX//KZlDjxvWtN+jtZa+TXbvYwC7cwTmGEYBIvkF3A3CDIzRvBtAM4FEw0GykOPPnA//4B2sUfBvgdXWxncaWLcCVV4Zecw46iN6M5uaB6fBODHTrVuDUU1N/SFfUaWykVl6zhhp56lS6e95803/TO5eLSmLJEjaVsnQtY5BEohAOU9WDReQDAFDV3SIyyA79RjKwYwfw0ENsj9+/O0J2Ntei995jkHv27ODnys6m4rj9drqxi4uZKem022lv5/mqq4N3ex5SqHKm8n/+w/8LCvh38WIOpRg5MvAb5fQreu45UwjGoInk59jtaVGtACAi5aDFYKQ4CxZwvQk0gEeEaaLPPOO/K0J/pkxhXUNZGVt2v/qqN01+6lQmxdx1F9t8p/ps96jw6qs0z6qqaJ6VlvLNGz+eKaWffMKWsYEoLmY03yqRjUESiYVwJ4DHAVSIyK0AzgNwY0ykMuLK0qXcZAajqIjp7C0t/jsj9GfiRMYIJk7kuV0u1lQ4Ssftpvu7qooFcUOWzk5aBqNH+9fIWVm8LF/ON8vSsowYEk5zu797/i0DcB3Y02gbgM+q6iMxlM2IE2536HWmf7ucUOzcySE6kydzHSsv77veZWRwDXzmmb7tr5ubWY29aVNk44NTlhUraAX4axkL8I1zu1m6vitA67Bdu/hGm//NGCThWAiHiMh4AJcCeBDAQ84dIjIiyg3ujASw777AW28FT1dva6OVEG5K+/LlgaeuOeTkcC1cs4YekqeeYkzVyarMywNOOQU4+eTYFNElBXV1wd+kffZhfxHAO+fAF2fw9mWXxUY+Y0gRjkK4B6w32AeAb0N1AeMJQ6ZjVkcHg6I5OX37i6U6xx5LN3awIO+OHSx4DbfNhdPhNBzWrwfuvpsZTaNGeR/X0cE02JUr2Zgv0CY6pcnODu77HzGCSmHZMtYg+NLVxfSvI46wsXNGVAin2+mdAO4UkT+q6tfiIFPSsXkzG7YtXMjfrioLQE89FZg0KdHSDZ7x46kUXn55YKq7KtNEx44NPPvAH04r71CoAk88QWtgdL/5eLm5zEZatQqYNw8477zwnz9l2G8//g3U89xJP21p4f8bNnjvy8oCzjqLRSLRbEhlDFlCKgQRESUBlYFzTHRFSw6WLwd++1v+3qqqvK2cV61iKuaXv8zFNJUR4ZCu/Hwqvt5evk5nNsO0acBXvhJZB4Tp071z3gOtVe3t3o6nY8YElm30aNZJnHFGGloJY8bQ/79hA79g/ti2jc3pvvhF9hBvaeGHNXly+hZzuN3Axx8zoOR2c0cyZYoV38WYcN7dV0TkUQBPqupG50ZPDcKRAL4E4BUAD8REwgSyaxfw+9/TavddDDMy2M2zs5Nzz8ePT/2e/i4X15w5c1jntHMn0+GnTh24cw+HkhKv1VFdPXDz29tLy6O6OvDcZofsbAaYN270bqjTBhHg8suBn/2ML3DkSG9JeGcnlcHYsd72FP4GVqQba9YA997LL4aId7hPcTFbgA+F9yBBhKMQ5oAB5Yc808wawQE5LgAvAviNqi6JlYCJ5K236LYNtDPOyeFi9dJL/E33x+3motfZyVRNf5PKko2iIrbHiQZf+AJjCYsW8fWXlPA9aWhgfOC00/je+JvZ3B+RgS70tKGsDPjhD5ly9dprXr9kdjZbVs+ZM3Qa1K1fD9x2G19vdXXf+5qbgV//Gvj2ty1mEiPCiSF0gG0r7haRLDD9tF1VG2MsW8J54w3WCAWjooKZMZde6nWNuN3sNvD009xpOy2fJ03iAJz994+97MlAdjbw9a9zZvKLL3Ljl5HB+MsJJzC76ZVXODMhGM5AMN/meGlHSQkwdy6/UB9+SNfI4Yez1fVQcZOoAg8+SDeYv8IYpwDmr39lf3iLm0SdiL5pqtoN1iAMCTo6Qm/MnO6ePT38XxX45z+5AFZUMEgL8PZt24Cf/xz42teGzvAql4ubuUAbukMOYZFu/znxvjQ00FUUyMWe8qjSt/bvf9M3lpPjbV1RXMxeIOmQvRCKTZtoIfg20+pPYSHjLatX059pRJWkqGQRkc+JyEoRcYvIgCk+iaK8vO+0Qn90dDC+5xRdLV3KAOiECX0rep32D1VVwH33hecmGQoMH85EmQ0b/LuEWlr4GXz+8/GXLW688gr7eJSV0U0yciTzb53gyy9+weBqurNtmzdmEIotW2IvzxAkKRQCgBUAzgFnNicNJ57onfQViLo6Fk45xVTPPMNFLlA+f14eLYqFC6Mvb6py1ln0lmzZwrhqQwNQX881sK2NLuOJExMtZYxoa6NlMHas/+q74cN5+3/+E3/Z4k24bTlCVTwae81eOSdFpEpVt0dLCFVd5TlvtE4ZFWbO5EZt+3b/7opdu2gdOEHY9nb6yR03USBKStg59Iwzoi9zKpKRwdjKMcdQUdbWMtHmoIP4GaRdqqkv77/vdRMFoqyMec6BvojpwrhxXOwD1WQA3t4pwdxKxl6zt9GqZwEcHE1BkpGcHO5Of/1rLlIFBdzhd3XRciguBr7zHW/8yxlmFUqvuVw8h9GX0tIhqCQ3bw49eciZkbxzZ/gKYds2lp8vWMCdSlkZTdnDD+cXORmpquKEpTVrAr/O3bu5S7OpcDFhbxVCxFt5EXkJgL9P+QZVfTKC81wB4AoAGBdqKx4FSkuBn/yEPcheeYVWQWkpcP753L361gUNG0aLIVivMoDZc5Y1ZwDwVjqGIhI3yTvvAH/6E48vL+cXtq2N0ftnngGuu45ximTkkkuAn/6UCs23u6sqFWJ3N3Dttdb1NUbsrUL4c6QPUNUT9/K5+p/nXgD3AkBNTU1cqqOzsrj4z5wZ/DiXi5uwxx8PbNGq8rd5/PHRl9NIQaZMYV+OYPT0cHEPZwO0fj1wzz1McfPdrRQU8FJfD/zqV8CttyZnx8DKSuDGG6m8VqzgbU6Abt99qTDisBEcquytQngsqlKkEcccQ0t9xw5+t31RZTbNjBlpWHFrDKCnhyUFL71E939uLvvQHXEE3Y0AWJRSVkZXSKChFFu3suw7HFfPCy8w5S1QS4vycvo/ly4FZs2K/EXFg8pK+mp37KBLTdWbeWWWQUzZ21D9s9EUQkTmishmAEcAeEZEXojm+ePJ8OHA9dfTSq+tpeW7cye/1xs2ADU1wFe/akkS6U5zMwtuf/UrYO1armOtrcDDD9Njs2yZ50CXi3UGHR1cAH3dR93d/NKMGsWoeyg6OoB33w1dEl9UxF1LslNZyUKVmhr2TzFlEHPiFkMIhqo+Dk5jSwsqKoCbb2YfsnffZS59ZSXjefa9Tn/cbvbAWr9+YB+nwkIqht/+Fvjxjz2uxQkT6CZ55BF2U3R2CxkZ9C3OnRueddDe7g1AByMnB2hs3LsXZ6Q1cYshDDUyMugenjIl0ZIY8eaTFZ34+M09GN+7HrKxhzvyceP4VwTDhtGCeOYZtvYAwDqEa6+lj7++nl+gsWMj62GUl0f3SrAWswAbSKVCYy0j7uyVQlDVu6MtiGGkBZ98gtevXoTcLSMhw9u5sDc00G80ZgwzE1wulJezM0VLS7/Nf3k5L3tDbi5w2GGsbQiWRdTUxGBXuLS0UEmJMPMnrQtDhjZDpGuWYcSBLVuAX/4S9Z2fRV5JLpDr8RXl5nLnvmkTrx9yCFwugYgfhTBYTjmF3Rbb2/0HluvqqHCmTw99rp07Odf0rbe8BWEuF/sqjR7N1zVhAmsHQtVSGCmBKQTDiBZPP02XUKFg665+LhsRphZt3gxMmgQtGg63OwaZn9XV9EPdcw+vV1RwsW5ro6VSUkLXVKgn3rGDMxpaW72ToerqGBR7+WVqMWcKUkEBJyjNmBHlF2PEm3AmpoXTdNg9FNphG0ZAmpu5WI4ejc/oZnywfSTK+s/dFqELadMm7B41HBMn+qSfRpNDD6V76rXXvJXKpaXAxRfTpRTKJFFlYVtXl3eU3c6dtBTy8njbnj1MlT34YCqN3/yGqaI2vCalCcdC2Oq5BMuNcQGwahFj6LJnz6cZPtOrdmBEXjsa2vJQmt/e97isLPQ0NqMxnzM0YpZxNnIkcMEFvERKbS2wbp23ulKVY/Ryc72WRWEhXWAHHMDAd1kZ8MADwO23Bw5o797NmgqnriDUsBEj7oSjEFapatAaXRH5IEryGEZqkpn5qZ8929WLaw5/G794czY27ilC5bBW5GT2wq3AztZ8tGgJzrkqPDd+Qvj4Y/51tNXu3Qx2DB/uPSYjg693924u7s6cgo8+opLwZedOptS++673caoMsJ9//sAKTiNhhKMQjojSMYaRvlRUcJfc3AwUFmLs8CbcfNyreLW2GvPXTkBnbxYUwLS81ZhzbQmmnJ3E9ShdXX13+c5QkP4COy0lHFRZku2rEOrr2SajtZWuJt+xgitWUPnccEPy9lYaYoQ7QtMvIvJlVb0/2DGG0Z/OTnZOWLKExbjjxrGdQ1lZoiUbBBkZwOmnc/rRsGFARgZG5LXjnCmrcPbk1WjrzkJWy27kSicw95wol3ZGmaoq1jI4+NNczi6/fyZTf3fRgw8yhjF6dN/bMzJYgV1Xx/fsxhuTWEMOHQabZXQTgPujIUi86OmhleskfQyVcbXJwurVwB/+QA9Efj7XhffeAx59FDj1VMZD162joqis5JTERPdga2qit2PhQq5tI0eygHj//futf0ceCXzyCYO5FRWfjsxzaQ8KGzwTvr73vcS/oFBMm0YZOzv513EV+c4p6Ozk63Oi4o6lUF3tPc/27ay8Dja7oLycNRobN9qMgyQgnCyjZYHuApAyzr+2NmbLvfAC/weYbDFnDvuGBeoFZkSP2lrGHIcP77tuAFx0f/5zri/jx3tnVefnMy561FGJ2UCuXMk2FJ2d3g3EqlUsKps8GfjmN32Sdlwu4MtfZufCZ56hT90ZtH344bQgRo2K/4uIlNxc+vbvv5/V0oWFXLh37eL/3d3UjDNnej+U+npgn336diKtrQ09IMS5r7bWFEISEM7+uBLAKQB297tdALwVdYliQHMz8MtfchNSVeVNbmhrAx56CFi0iBlzkXQJSEdaWrjQLV7sdeUcdRTXhGgsxo8+yg1nUVHf29vamNGYkcEWO4cd5lXQHR30KPT0xL9l+KZNHI5UUtJ3XsuwYVzj164F7roL+O53fZoVulwcoXfkkVwke3qoAZNlKI3jr3vrLW+NwVFHsbW0b8fF447jl+Dhh6mZx47ljn/rViqFww6jCed20+3jcrEWwfeLEs6cBwdfF5WRMMJRCPMAFKjqkv53iMir0RYoFjz4ILuOTpjQ9/b8fG5qamupGL7ylYSIlxQsWwbcfbfXE+By0XUzfz7Xti99aXDFqDt3MoY4duzA+z76iHHMoiIqhC1bvAOxcnPpfv7Xv+hO8nhh4sIzz9Ai8LeWizBGumoVB3wNaGeekZF82TMbNrBeoLGRLyori7ukN95g060rr/S+WBFWPR9+OP1la9fSf7d5Mz/Mnh4+1u1mutQFFwwMDFdU9A06B0Ik+d6rIUo4QeXLgtx3YXTFiT719dzx+luIHEaP5obpvPNiVCiU5Kxdy3WirKzvTnj4cP7eX3/d6w3pT2sr3T2ZmXx8IEti1y6ukf3bfnd2cifurENZWTyfLzk53EAuWhQ/K6G1lc8XzMMjQtlefz0F5lvU19NMdrn6+uuGD+ei/dFHDO5897t9AyPDhwMnncSLQ0MDtTZAJRCo99I++/ALtWdP35RVX5x01v33H9TLM6JD2odUP/449PRBx8378cfJOzMkljz+OF00/lxmGRlcP15/nUFfR2Fs3Qo8+ywDrQDfv6oqzkQ+/PCB77dPmn4fnIxG53i323+gPz+fO/F4KYTmZv4N1Ul62DB2eUh6/vc/+t/87YxEePuqVf7rCPpTWhpeUVlGBvDFL3qL1fqbWm1tVFRXXx36jTbiQsgxLSLyfjSOSRSdneEdpzo0B9/v3MnAabAGm87O/p13eH3tWuCmm+hJGDWKa8nYsVxv/vhHuuj6u4/HjKH7p6NfgnL/VPbeXv/eg0hGCkeDnBy+hlAej66uFIg9dXdzIHgwt4wIte7LL0f3uQ84gL2TOjvpm92yhbuJ2lpaB9/4RujZtEbcCMdCmBIk0whgcDmAPZh4Ak0l7I+ThjrUaGzkQhsqaJyby99xRweHu+Tn9/UCiDAGUFDAzeikScDs2d77s7M5b/qxxxjLcZ6voIDP39vLxTUvz79yam+P72yJ4mJ6PHbuDP4damkBPvOZuIm1d7S2UilkZwc/rqCAwbZoM20ao/MrVjAtF+CX4KCDkj8Fd4gRjkIIx7mXtCkCBxzARcZJqfZHezt/C5Mnx1e2ZCA7O7y4X08PlcDSpXSn9E8bdcjI4II+bx4XSl9Fc+qpXA+WL6frOS+PMYNx42ilFBWxQK2/JdDWxs/u4IP3+mVGjAizRH/7Wway/bmxdu+m4kjaFhQOWVlecyeY5u/pGZgCFk0ZZs40ayDJCWmEq+oG3wuAywBcAWAWgGzP7ZtjLejekpMDnHsukyN6egbe393Nne/nPjc0W7qPGsU1wKnNCERXFxfkt98OnUFZWMgMxfr6vrfn5ADf+hbw+c/z+TZu9AaU99mHgdn+WUR79tBH/5WvxL9WZOZM4LOfpZw7d3oVZ1cX5e7pAa65JgU2ucOGMWi7a1fw4xobU8DcMWJJxEFlVf2RiFQCmAngXBGZqKqXD0YIEbkdwJkAugCsBfDlaLbTPuEELkCPP87dp7MJ2rOHfy+6iKnYiaK1lbtNl4uZevGMr2Vmcif84INclP1tIBsamEE0ZQoDyaGqu50uz93dA+/Lzmbg+ZRT6J3o7fXGJx96iHEKX4ulshL4znfodYg3IhxnPGkSX/eqVXxdWVl0f51wQgq12zjtNOCOO2jSOF+wzk5qt8xMfhA5OcztNYYsouH4CwCIyG8BXKPhPiASIUROBvCyqvaIyC8AQFW/F+pxNTU1unjx4rCfp66O7eFXr+b1qVPp507Uj7qujrnub77J6243ldVpp7EuKF4WS28vcO+9TL316biA7m7u9PPygOuvZ2D4H/9gZ4b+rWn6n2/rVuB3v4u8Hmv3btY/OIpiwoT4BpOD0dbmDSKnnDWpygDOk09S023fzi8gwBdWWMh+Qp/9bMxE6O7mU2VnW2eARCMi76lqTf/bI7EQWgA8JSIXqGqrZxH/sarODvXAUKjqiz5X3wZw3mDP6Y+KCuCcc2Jx5sjZvJmtGjo7vQOpAFoL//gH429XXRU6DhgNXC7giisYb3n2WbpInF3+ccdxN+8Eeo88EnjppeDu6B07wpvD4o+SEuCQQ/b+tcSS/HxeUhIRfvlbWhgYaWvjqizCKsDycpaS5+ayn0sU2bmT35lXXqFSUOVm7NRT+Z2znnbJQ9gKQVVvFJELAbwqIp0AWgFcHwOZLgXwn0B3isgVYAwD43z7pqQQvb3AnXdywXUGUjkMG8Zd8dKl7Lt05pnxkcnl4tz1o46iq9npuNB/Jzd+PFBTw6Kt6uqBP2anqOz00+MithEJ9fUsKDnpJH5wvb3ccTi7ju5u4N//po9s4sSoPGVtLevhOjvp/svOpiW8fj3wi1+wGPTMM00pJAthKwQROQHA5aAiGAngMlX9KILHvwSgys9dN6jqk55jbgDQA+Cfgc6jqvcCuBegyyjc508mVq/mbzNQLy8RZuE8/zx35/GwEhwyMoK70ES8LT4WL6brZNgwri3NzbQKvvvd4C4lI0EsWMC/gcycrCzGEV56KSoKoaODFfBZWbTOHZxMtJIS4L//5QYoETEiYyCRuIxuAPBDVV0gItMA/EdErlXVsCpZVPXEYPeLyJcAnAHghFjEKZKJpUtD+6Bzc+ni3bQpapu1qJGby7Y3GzawDc6WLbztsMM4Z938w0nKm2+GrjCuqGBk//LLBx28WbKEiRuBUpQzM2mFzpuX5gqhqYltEJyGXZMnJ20QKhKX0fE+/y8XkVMBPApg0HlqIjIHwPcAHKOqIRIgU5+OjvAyiUT8p8omAyL8oQf6sRtJSGdn4J5CDk7f8Z6eQZumCxaELmsYMYK1KcHaHaUsHR3sFvvaa31L9wsKmAt/zDFJ5yvb615GqrrN40aKBn8AkANgvvANeltVvxqlcycdo0aFbqmhyu/QUKyeNmJEVRV9lcFKr9va+KWLwg62tTX0aZxxCWnXNqari8H71asZKPTN1W5vB/76V/pY4xUkDJNB2YSq2h4NIVR1X1Udq6ozPJe0VQaAt4FesBbwDQ2sJbKuwEbUOPlkFp8Fo76egaso7FwrKkIXPPb28qmSvh9UpLz5Jsvvx48fWLiTl8fmX489xvTfJCJJMryHFiNG8Le5YYN/pdDSwt3VeTFJvjWGLNOnc4FyWlf3p66O1sPsQWeSA6BHpK0teGsUJ0U5ZdN5/eF2s8CosjKwYs3KYozmjTfiK1sITCEkiM99junemzbxsns387U3bKAyuPba5AsmGylOTg6/WGPHMh90+3Y67+vqeL2wELjuuqj1M5o8me1ItmzxrxRaWrh2nnpqVJ4ueWhupokfqhCnuJgFR0lE2s9DSFZcLuDCC9nf/623+HvMzGS/oIMPTrMdk5E8FBcDN9zArJc33uAuZPhwWgVTp0Y1+8XlYnfrO+9k4LiggK6h7m56rnJy2Auqfy1OyhOqiaBD/97vSYAphARTVZU81dPGEMHlYmOqOPQTLypi25NVq9gWfft26qTTT2csLVbNVRNKYSGVbGtr8OBIY2PSleWbQjAMI6ZkZrLOIK1rDXxxuegH++c/Bw5yd+jtZWrv0UfHV7YQWAzBMAwj2hx1FJXBxo0Dxwd2ddFHfNppwYd2JwCzEAzDMKJNXh77tj/4IBt/ud3euEJODnDBBbQi0qUwzTAMwwhCQQHw9a8zcL96NSuXi4sZvE/S/i6mEAzDMGJJWRn7xqcAFkMwDMMwAJhCMAzDMDyYQjAMwzAAWAzBSFHcbmDtWtb2ZGayzUdaFjkZRhwxhWCkHO+/z0mP9fW87rRQPvJI4Pzzre2HYewtphCMlOLNN4E//YkjGH1HkPb0cFzwpk0c4ZmkWX2GkdRYDMFIGZqagPvvZ3FnYWHf+zIzqSDWrmXPHMMwIscsBCNlWLSILWBycwMfM3Ik8PzznPGSpGNrU5OmJmDZMmDXLvrkDjyQnRmNtMIUghE2W7YA69dzUS4rY7/7/sOgYsmyZQMtg/7k5jK2UF+fdG1iUpPeXuDRR4EXXuD/Lpd3qtPMmcCll4b+UIyUISkUgojcAuBsAG4AdQD+T1W3JlYqw6G+niNgV63idVUOeyoqAi65BKipiY8cKdxmPjVRBf7+d+Dll4Fx4/pqf1Vq6DvuYH9rC9qkBckSQ7hdVQ9S1RkA5gH4UYLlMTw0NAC33gqsW0cffXU1mziOH0+XzJ13AgsXxkeWSZM4ZSsYXV3cxJaWxkemtGb9euDVV/mh9zcFRbyT1956KwHCGbEgKRSCqjb5XB0GwPZ3ScITT3ARHjly4O68oIC3P/AA0N4ee1mOOIIb0+7uwMds28YpdMHiDEaYvPoqkJ1NczAQFRWcH9y/xbORkiSFQgAAEblVRDYBuAhBLAQRuUJEFovI4nonEd2ICU1N3PwFix3m5QGdnawNiDVlZcDcuWwx39nZ9z5VKoPSUs6qNqLAxx+zO2cwhg1jdWBbWzwkMmJM3BSCiLwkIiv8XM4GAFW9QVXHAvgngKsCnUdV71XVGlWtKS8vj5f4Q5Lt2/k3VOA4L48zc+PBGWcAF1/MZJfaWiqH2lpgwwa6sr7/fU4vNKKAyxU6GKMafnDHSHriFlRW1RPDPPRfAJ4B8OMYimNEkXgGcEWYUnr00cDy5Qx45+QA++8PjB5t61JUOeggYP784HOBm5r4xlt5eFqQLFlGk1TV2WOeBWB1IuUxyMiR/NvTE9xK6OgA9tsvPjI55OVxSLsRQ44+GnjuOQZt/BV1qHL4y+c+Z5o4TUiWGMJtHvfRMgAnA/hWogUymF4+ezZ984Foa2MA9+CD4yeXESdGjuRiv3Ej0Nra9z5nLnBNjWnmNCIpLARVPTfRMhj+mTsXWLkS2LqVwWXfhJPmZrpsvvENy+pJW047jQUnjz3GQI1T5JGdDZx5JnD22fGtTjRiimgKV/DU1NTo4sWLEy1G2tPQwFnhS5d6bxMBRoxgYdqMGQkTzYgXvb3AmjXcBeTkAPvua8VoKYyIvKeqA0pKTbUbISktBa65BtixgwVqTuuKSZOYiGIMAVwu9iox0hpTCEbYVFbyYhhGepIsQWXDMAwjwZhCMAzDMACYQjAMwzA8mEIwDMMwAJhCMAzDMDyYQjAMwzAAmEIwDMMwPJhCMAzDMACYQjAMwzA8mEIwDMMwAJhCMAzDMDyYQjAMwzAAmEIwDMMwPJhCMAzDMACYQjAMwzA8JJVCEJHviIiKSFmiZUl2VDnWcskSYNkyoLEx0RIZhpHqJM2AHBEZC+AkABsTLUuyU1sL/OtfwCefcMaxMwX1sMOA888HiosTKZ1hGKlK0igEAL8BcB2AJxMtSDKzZg1w220caztuHGcbAxxruWgRsHYt8IMfmFIwDCNyksJlJCJnAdiiqktDHjyE6e0F/vhHoKAAKC/3KgOAI2/HjAEaGoBHHkmcjIZhpC5xsxBE5CUAVX7uugHADwCcHOZ5rgBwBQCMGzcuavKlAqtWccGvrg58zKhRwMKFwOc/DwwfHjfRDMNIA+KmEFT1RH+3i8g0ABMALBVueccAeF9EZqnqdj/nuRfAvQBQU1OjsZM4+fj4YyAzxCfmctFy2LTJFIJhGJGR8BiCqi4HUOFcF5FaADWqujNhQiUpvb193USBUAXc7tjLYxhGepEUMQQjPMaNA7q7gx/jKIPKyvjIZBhG+pBwC6E/qlqdaBmSlenTgdxcoL0dyMvzf0x9PTB1qikEwzAixyyEFCI3F7jkEhaktbcPvH/3bqCnh7UIhmEYkZJ0FoIRnNmz+ffvfwd27GCQWZWKoLISuPpqupYMwzAixRRCCjJ7NlBTw5YVmzZRKUyaBEyezMplwzCMvcEUQoqSkwMceigvhmEY0cD2k4ZhGAYAUwiGYRiGB1MIhmEYBgBAVFO3+4OI1APY4HNTGYBkq3BORpkAkytSklGuZJQJMLkiJRFyjVfV8v43prRC6I+ILFbVmkTL4UsyygSYXJGSjHIlo0yAyRUpySSXuYwMwzAMAKYQDMMwDA/pphDuTbQAfkhGmQCTK1KSUa5klAkwuSIlaeRKqxiCYRiGsfekm4VgGIZh7CWmEAzDMAwAaaoQROQ7IqIiUpZoWQBARG4RkWUiskREXhSRUYmWCQBE5HYRWe2R7XERKU60TAAgIp8TkZUi4haRhKbjicgcEflIRNaIyPWJlMVBRP4qInUisiLRsvgiImNF5BURWeX5/L6VBDLlisi7IrLUI9NNiZbJFxFxicgHIjIv0bIAaagQRGQsgJMAbEy0LD7crqoHqeoMAPMA/CjB8jjMB3Cgqh4E4GMA30+wPA4rAJwD4PVECiEiLgB3ATgVwAEAviAiByRSJg8PAJiTaCH80APg26o6BcDhAK5MgverE8DxqjodwAwAc0Tk8MSK1IdvAViVaCEc0k4hAPgNgOsAJE20XFWbfK4OQ5LIpqovqmqP5+rbAMYkUh4HVV2lqh8lWg4AswCsUdV1qtoF4N8Azk6wTFDV1wHsSrQc/VHVbar6vuf/ZnChG51gmVRVWzxXszyXpPj9icgYAKcDuC/RsjiklUIQkbMAbFHVpYmWpT8icquIbAJwEZLHQvDlUgDPJVqIJGM0gE0+1zcjwQtcqiAi1QBmAngnwaI4bpklAOoAzFfVhMvk4bfg5tWdYDk+JeXmIYjISwCq/Nx1A4AfADg5vhKRYHKp6pOqegOAG0Tk+wCuAvDjZJDLc8wNoLn/z3jIFK5cSYD4uS0pdpfJjIgUAHgUwNX9rOOEoKq9AGZ4YmSPi8iBqprQ+IuInAGgTlXfE5FjEymLLymnEFT1RH+3i8g0ABMALBURgO6P90VklqpuT5RcfvgXgGcQJ4UQSi4R+RKAMwCcoHEsSong/UokmwGM9bk+BsDWBMmSEohIFqgM/qmqjyVaHl9UtVFEXgXjL4kOyM8GcJaInAYgF0CRiPxDVS9OpFBp4zJS1eWqWqGq1apaDf6YD46HMgiFiEzyuXoWgNWJksUXEZkD4HsAzlLVtkTLk4QsAjBJRCaISDaACwA8lWCZkhbhTuwvAFap6q8TLQ8AiEi5kz0nInkATkQS/P5U9fuqOsazVl0A4OVEKwMgjRRCknObiKwQkWWgSyvh6Xge/gCgEMB8T0rsPYkWCABEZK6IbAZwBIBnROSFRMjhCbhfBeAFMED6sKquTIQsvojIQwAWApgsIptF5LJEy+RhNoBLABzv+T4t8eyAE8lIAK94fnuLwBhCUqR4JiPWusIwDMMAYBaCYRiG4cEUgmEYhgHAFIJhGIbhwRSCYRiGAcAUgmEYhuHBFIJhGIYBwBSCkSaISLWItHt61ji3DWhdLSJ5nvz4rsG2R/ec6zVPV1SIyDc9rZ8jagEiIsUi8vXByBLGcwxomS0i2SLyuoikXMcCIzaYQjDSibWeFuMBW1erarvnmGi0oLgUwGOeXjkA8HUAp6nqRRGep9jz2IgQEu5v+AH0a5nt6eD6PwDnR/rcRnpiCsFIGTzDV07y/P9TEbkzyOHxaF19EQCnQeA9APYB8JSIXCMiF3sGsywRkT/5WBFPiMh7nmEtV3jOcxuAiZ5jb/dYO747+e+IyE88/1d7rJC7AbwPYGyg5/IlSMvsJzyvwzBMIRgpxY/BjrEXga2VrwlybExbV3t6G+2jqrUAoKpfBa2O4wA8D+66Z3uskV54F91LVfUQADUAvikipQCuh8e6UdXvhvH0kwE8qKozAeQHea5wWAHg0AiON9IY8x0aKYOqvu5poHYtgGMdV42I3AI2VfMl1q2rywA0BrjvBACHAFjk6bybB/biB6gE5nr+HwtgEoBIGzBuUNW3w3iukKhqryeeUugZamMMYUwhGCmDp8X5SAA7ncVLRKrg/3sccetqEbkSwOWeq6cBmOt7XVV9H98Oti32eyoAf1PVPiNJPX3vTwRwhKq2eVox+ztHD/pa7/2PaQ31XBGSA6BjEI830gRzGRkpgYiMBAf4nA2gVURO8dw1E8ASPw+JuHW1qt7lcdvMUNWt/a/3O3Y3AJeI+FvQ/wfgPBGp8Mg+QkTGAxgOYLdHGewPzh0GgGaw66zDDgAVIlIqIjngvIpABHqusPC4rOpVtTvcxxjpiykEI+kRkXwAj4ED3FcBuAXATzx3z4AfhRCn1tUvAjjSz3N/COBGAC962i7PBy2b5wFkem67BZxjDVVtAPCmp0X67Z7F+WZw/OQ8BOnfH+S5+hCkZfZxAJ7dmxdvpB/W/tpIaUTkL6BbZxyAeap6YJiPqwVQo6o7B/HcMwFcq6qX7O05Eo2IPAbg+6r6UaJlMRKPWQhGSqOql6mqG8yuGe5bmOYPpzANQBYGOdxcVT8Ah68MSPNMBTyutCdMGRgOZiEYhmEYAMxCMAzDMDyYQjAMwzAAmEIwDMMwPJhCMAzDMACYQjAMwzA8mEIwDMMwAJhCMAzDMDyYQjAMwzAAAP8fjkYFTXjbEJ0AAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x, train_labels)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Normalizing Data\n",
        "\n",
        "Before training, it is common to bring our input features to the standard range of [0,1] (or [-1,1]). The exact reasons for that we will discuss later in the course, but in short the reason is the following. We want to avoid values that flow through our network getting too big or too small, and we normally agree to keep all values in the small range close to 0. Thus we initialize the weights with small random numbers, and we keep signals in the same range.\n",
        "\n",
        "When normalizing data, we need to subtract min value and divide by range. We compute min value and range using training data, and then normalize test/validation dataset using the same min/range values from the training set. This is because in real life we will only know the training set, and not all incoming new values that the network would be asked to predict. Occasionally, the new value may fall out of the [0,1] range, but that's not crucial.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [],
      "source": [
        "train_x_norm = (train_x-np.min(train_x,axis=0)) / (np.max(train_x,axis=0)-np.min(train_x,axis=0))\n",
        "test_x_norm = (test_x-np.min(train_x,axis=0)) / (np.max(train_x,axis=0)-np.min(train_x,axis=0))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "SjPlpf2-wHl8"
      },
      "source": [
        "## Training One-Layer Network (Perceptron)\n",
        "\n",
        "In many cases, a neural network would be a sequence of layers. It can be defined in Keras using `Sequential` model in the following manner:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense_2 (Dense)             (None, 1)                 3         \n",
            "                                                                 \n",
            " activation_1 (Activation)   (None, 1)                 0         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 3\n",
            "Trainable params: 3\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.Sequential()\n",
        "model.add(keras.Input(shape=(2,)))\n",
        "model.add(keras.layers.Dense(1))\n",
        "model.add(keras.layers.Activation(keras.activations.sigmoid))\n",
        "\n",
        "model.summary()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here, we first create the model, and then add layers to it:\n",
        "* First `Input` layer (which is not strictly speaking a layer) contains the specification of network's input size\n",
        "* `Dense` layer is the actual perceptron that contains trainable weights\n",
        "* Finally, there is a layer with *sigmoid* `Activation` function to bring the result of the network into 0-1 range (to make it a probability).\n",
        "\n",
        "Input size, as well as activation function, can also be specified directly in the `Dense` layer for brevity: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_9\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense_9 (Dense)             (None, 1)                 3         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 3\n",
            "Trainable params: 3\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Dense(1,input_shape=(2,),activation='sigmoid'))\n",
        "model.summary()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Before training the model, we need to **compile it**, which essentially mean specifying:\n",
        "* **Loss function**, which defines how loss is calculated. Because we have two-class classification problem, we will use *binary cross-entropy loss*.\n",
        "* **Optimizer** to use. The simplest option would be to use `sgd` for *stochastic gradient descent*, or you can use more sophisticated optimizers such as `adam`.\n",
        "* **Metrics** that we want to use to measure success of our training. Since it is classification task, a good metrics would be `Accuracy` (or `acc` for short)\n",
        "\n",
        "We can specify loss, metrics and optimizer either as strings, or by providing some objects from Keras framework. In our example, we need to specify `learning_rate` parameter to fine-tune learning speed of our model, and thus we provide full name of Keras SGD optimizer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {},
      "outputs": [],
      "source": [
        "model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.2),loss='binary_crossentropy',metrics=['acc'])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "After compiling the model, we can do the actual training by calling `fit` method. The most important parameters are:\n",
        "* `x` and `y` specify training data, features and labels respectively\n",
        "* If we want validation to be performed on each epoch, we can specify `validation_data` parameter, which would be a tuple of features and labels\n",
        "* `epochs` specified the number of epochs\n",
        "* If we want training to happen in minibatches, we can specify `batch_size` parameter. You can also pre-batch the data manually before passing it to `x`/`y`/`validation_data`, in which case you do not need `batch_size`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "70/70 [==============================] - 0s 4ms/step - loss: 0.3379 - acc: 0.9000 - val_loss: 0.3282 - val_acc: 0.9000\n",
            "Epoch 2/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.3270 - acc: 0.9429 - val_loss: 0.3336 - val_acc: 0.9000\n",
            "Epoch 3/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.3195 - acc: 0.9143 - val_loss: 0.3137 - val_acc: 0.9000\n",
            "Epoch 4/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.3087 - acc: 0.9286 - val_loss: 0.2970 - val_acc: 0.9333\n",
            "Epoch 5/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.3006 - acc: 0.9429 - val_loss: 0.3210 - val_acc: 0.9000\n",
            "Epoch 6/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.3003 - acc: 0.9000 - val_loss: 0.2985 - val_acc: 0.9000\n",
            "Epoch 7/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2956 - acc: 0.9286 - val_loss: 0.3037 - val_acc: 0.9000\n",
            "Epoch 8/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2891 - acc: 0.9429 - val_loss: 0.3035 - val_acc: 0.9000\n",
            "Epoch 9/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2809 - acc: 0.9000 - val_loss: 0.2815 - val_acc: 0.9000\n",
            "Epoch 10/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2809 - acc: 0.9286 - val_loss: 0.2907 - val_acc: 0.9000\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x2c420b89910>"
            ]
          },
          "execution_count": 44,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.fit(x=train_x_norm,y=train_labels,validation_data=(test_x_norm,test_labels),epochs=10,batch_size=1)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "s4_Atvn5K4K9"
      },
      "source": [
        "You can try to experiment with different training parameters to see how they affect the training:\n",
        "* Setting `batch_size` to be too large (or not specifying it at all) may result in less stable training, because with low-dimensional data small batch sizes provide more precise direction of the gradient for each specific case\n",
        "* Too high `learning_rate` may result in overfitting, or in less stable results, while too low learning rate means it will take more epochs to achieve the result\n",
        "\n",
        "> Note that you can call `fit` function several times in a row to further train the network. If you want to start training from scratch - you need to re-run the cell with the model definition. \n",
        "\n",
        "To make sure our training worked, let's plot the line that separates two classes. Separation line is defined by the equation $W\\times x + b = 0.5$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "PgRTHttLwHl9",
        "outputId": "e4407e1b-edf5-48e5-fdc2-da28120a3c6b",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_103052/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABRPklEQVR4nO2dd3hb5fXHv0feM86wnT1IAgECmayEPQNl71GgQEtp6a8UuqC7pZMWaGmhLYUWKBRoGaWBEGYCCTODQDYBEmdvJ95LOr8/vrqRbGvasnQln8/z6EkkXV8dSVfvec8WVYVhGIZheFItgGEYhuEOTCEYhmEYAEwhGIZhGH5MIRiGYRgATCEYhmEYfkwhGIZhGACA7FQL0B0GDBigI0eOTLUYhhE3H+/6GD71YdyAcakWxeiFLFq0aKeqlnd8PK0VwsiRI7Fw4cJUi2EYcXPSIyehua0Z86+dn2pRjF6IiFSFetxcRoaRAnzqQ5YnK9ViGEY70tpCMIx0xevzmkIw4qepCfjsM6C5GejTBxg5EvAkbl9vCsEwUoBPfciV3FSLYaQLra3AzJnASy/x/wCgClRUABddBEydmpCXMZeRYaQAr3rhEfv5GTHQ1gbcey/w3HNA//7A8OG8jRhB5fCHPwBz5iTkpeyKNIwUYDEEI2befhtYvJjuodwOVmVpKTB0KPDoo8DOnd1+KVMIhpECvD6zEIwYUAVmzQIGDABEQh+Tl8fj3n672y9nV6RhpACf+pAlZiEYUdi7F9i2jZZAJMrKgCVLuv1yphAMIwVYDMGICdXwlkEwHg/g9Xb75VxzRYpIvoi8LyIfishyEflpqmUyjJ7C0k6NmCgpAYqKgMbGyMft2QOMHt3tl3ONQgDQDOBEVZ0AYCKAGSJyZGpFMoyewVxGRkxkZwOnnUa3UTh8PmYiHXdct1/ONQpBSZ3/bo7/ZvM9jYzEXEZGzBx/PDBwILB5M11IwbS1AevWASecwFTUbuKqK1JEskRkCYDtAF5R1fdCHHO9iCwUkYU7duxIuoyGkQgs7dSImeJi4LvfBfbbD6iqAtavBzZtoiLYvBn43OeAK66ILdYQBVdVKquqF8BEESkD8KyIjFfVZR2OuR/A/QAwdepUsyCMtMTSTo246NuXSmHjRuCjj4CGBqaiTp7MFhYJwlUKwUFV94jIXAAzACyLcrhhpB0WQzDiRgQYNoy3HsI1WxQRKfdbBhCRAgAnA1iVUqEMo4ewGILhRtxkIQwC8LCIZIGK6t+q+nyKZTKMHsEsBMONuEYhqOpHACalWg7DSAYWQzDciGsUgmFkLKrAJ58Aq1ezO+WgQZZlZLgSUwiG0ZNs2gT85S/Ahg0MCmZlAW1t8I6ogWfT5thbExhGEjCFYBg9xbZtwK9+xUV/xIh2C78XQNbKVcCrrwKnnJI6GdMVn4+fa5ZZWYnEFIJh9BT//S9HHQ4Z0ukpHxRZxaXAk08CRx7JnjWpwOcDVq4EXnmFbi2PBzjkEODEE1kI5SbrxecDli0DZs+mzI6iPf105uN3nBVgxI0pBCMs27fz5vFwBke0DrxGEHv3Au+9F1IZAIAXCk9WNjtULljABTjZtLQAf/sb5SwuZgtlVWDhQmD+fFbAXnSRO5SC1wv84x/Am2/yQnTaNOzdC9x3H3DwwcDXvw4UFKRWzjTHFILRiY0bgX//G1i6tP387unTgQsuSGhhZOayfXsgZhACHxRZEC5ga9cmWTg/TzwBvP8+MGpU+0V/0CAuwDNnAv36ASefnBr5gpk1i8qg41D5sjJekKtWAY88Anz5y6mSMCOwvDejHevWAbffDqxZEyiKHDaMa8Rbb9ElvndvqqVMAzyRf1pe+OCB0A0S5dgeYc8eYO5c7rRDWQBZWcDgwZzj6wx1TxXNzVQIQ4aE/qycCt533wWsv1m3MIVg7MPnA/78Z07kq6xs/9vLzuZvbudOWg9GFAYP5qIaZjGlheABmpro7kg2H37ILzxSULagAKirY2wh0TQ3Ax9/DCxfTpO0YxfPYNas4fF5eeGP8Xh4jqVLEy9rL8JcRsY+Pv6YiTEjR4Y/ZtAg4J13gIsvNtdRRAoK2JL45ZcZ+OyAVxSe5lYOP5kwIfny7d0bW4aOCJVComhp4W7/5ZepDEXonho5kvGKUMqxqSm2c2dlAbW1iZO1F2IWgrGPjz+OvkY4z1dV9bw8ac9ZZ9HNsX49+9b7UfUBALIamoAbboi88+0pSkpiH7mYqEBtaytw773As89yNzF8OM3OESOA6mrgjjsYYO9IUVFs5/d62RXU6DJmIRj7aGuLLaFE/K5vB5+PVv3MmVz7iouZCXjEEb08E9DpY/+f/zAA43eLeH1twCjAc+yxTPFMBePHB77IcDGM5mZ+gWPHJuY1580DFi/unM4qwoU8Px+4/37ggAPap7SNGRMYIxlOOXm9gZRZo8uYQjD2MXRou41sSFT52ysv5/26OuAnP2F9VWMjYw0+H/DUU1xH7r4b2H//HhfdvZSUANdeC1x4ISP2Xi98ZSXAI/9EVnll6uQqL6fGfu+90IFln49V1hdemBgLxuejq6iyMvyuo6CAF+D777fPbMrJAc45B3j4YbqWOpqxqtyJnHSSWQjdxBSCsY9DD+UmramJ/4Zi1y4u9IMHUzHceivw0kv8HfbpE/ite73AZ58BV10FPPMMj+/VlJbyAwbgbeXA9JQ3t7v6amYbrVjB9FInKLRrF1BTAxxzDGsREsGuXcDu3dHHPJaWAh980DnV9aSTeI5Zs3hx9u/Pi626mruSww8HLr00MbL2YkwhGPvIz+ckvvvvp+u7o1LYu5fK4vLL+VtcvJixwQEDOm8is7KAigpuMu+5B/j1r5P3PtyOV+m7T3n764IC4JZb+EW+8AIDQyLAuHHAjBl0vyQqJdbni80f6fGEzswSYSbDlCnA668zm8jn4+7klFOAAw+0NhYJwBSC0Y6jj6YF/thjTAjJyeH9tjbWAH3nO6xjAoB//pO/33AeBREqi+efB378YysidfA5QWU3dDvNzWXrjCOPpFkn0jN1EWVl9Ce2tEQOLNXW0pUVChHGE8aMSbx8BgBTCEYHRIBjjwUOO4yWe1UVN17jxgEHHcTftMPKldEX+fx8Wvrr1nETZ3AWAuACl1FHenKHnZcXSMMN5zbyernzOOaYnpPDiIgpBCMkBQXAtGm8hSOW9cPp7mzWfIB9FkKqXUbJ5rTTGMTeto3+xGAXktfL3cepp4bt/2T0PC7bohjpxFFHRa9Zqq+nt8B+4wGcGILrLIRwbN4MvP0200ZXrYq9fqEjffsyC6Gigov/hg3Ali38/6ZNDGBfdlliZTfiwiwEo8t8/vPAv/4VPivJ52MSyzXXxF5b1BtwVQwhEjt2AA89xPYSwfTvzwwlf9ZUXFRWMk/5k08YGG5oAAYOZLDYUkZTjikEo8uMHs3mkvfey2zBkpJAS5mmJmYE7r8/cP31qZbUXbg2hhDMrl3AL37B4pIOw31QWwvceSfbTU+ZEv+5RZgdlKiCNyNhmEIwusU3vkFl8Pe/0/r3eGgZFBczNviznwWK2AySFjGEp5+mvy+Ur8/R/A8+yIrnVLTeCMYplX/tNeDTTynbxInAccex2tKIGdcoBBEZBuARAAMB+ADcr6p/SK1UBsDf2yef0IW8YwcX+2nTmKaekwN88YtMEZ8/n9lERUXMYhw7NjWdnd2O62MIe/awlXSkwE9REVvffvghi8JSRWsrFdM77zAToqyMF+ycOZwCd+GFjE24YchPGuAahQCgDcA3VXWxiJQAWCQir6jqilQL1pupraVLaNUqbgQLCxkLXLiQNQY338x1o7QUOOOMVEubHjguI9fGELZsiS01LDeXO/NUKoQnn6QyGDmy/aJfWEhl8eSTrMC2VNaYcM0WRVW3qOpi//9rAawEYLkpKaStjVXGa9bQjTxoEH9b5eX8/TU1sQK5ujrVkqYXaeEySgeqq1m1HG7IT04OA9bPPtv1zKhehpsshH2IyEgAkwC8l2JRejUrVrAldsfNl8OAAewpNncucN55yZbOxezZwzTNN95gXq4zhnLqVKCoyP0uo0GDAl0MI1kJzc2pDQwvWRJ9yE9REdNaP/vMgtgx4LorUkSKATwN4BuqWhPi+etFZKGILNxh4/J6lFdfZbwgkvu1spKuWtuA+Vm9mrn2//kPP5SyMgZn//EP4Ic/BDZvdn/aaVkZ3UBbtoQ/pqGBPvtUDPdxqK6mFRCNRA/5yWBcpRBEJAdUBo+p6jOhjlHV+1V1qqpOLbf0lYSzfTvwv/8Bf/oTMHs2YwiRFvu8PLqOGhuTJ6Nr2b6d6ZiFhfSxFRdzwSotpZnV3AzceSe8DVycXGshAAzGFhdTKXQcb1lby2rj665LbYZRcXH0fu0Oqc6EShNc4zISEQHwIICVqnpXquXpbbS1cVbyK69wQ1VYyFT0TZu46T3iCHo+OuLz8RbLRi3jef11as/g4S7BlJcDVVXwrWCehKtjCAMGAN/7HvOJV63iReEohn79mE0wcWJKRcQhh7Ay0umPEormZlZNjh6dXNnSFNcoBADTAVwJYKmILPE/9j1VnZU6kXoPTz7JuQYjRgRcsuPGMY4gwoFfxx3Xea3btYu/y16/AXNSHSujDL0pK4P3vXcAcbmFALDFxK23clfgH+6D8nJWG7qhOdWgQVRKy5aFrjdQTeyQn16AaxSCqs4HYMnCKWD7dloGwcoA4G9s1apALcHq1eyC6tDWRu/BjBnJldeVtLREb+0MAPn58NVsAfq4OIbQkSFD3NuM6rrr6Kb77DMqsOJiKoLqat6mTbN86Dhw+RbFSAZvv01F0HHTV1BABeDEBzZtYrzA+b1VVTG76KCDki+z68jJ4QcYzafd0gJvEXuGR7IQVKmoN2zg8DIjDCUltGKuuYY7l6oq3gYMAG66iX1Tsl2z73U99kkZ2LiRMYNQDBzI+Qhr1rBaee1aHjtqFPCFL9Biz7gi0Pp6YMECzvZtbqapdMwx9EOHe7NZWZwu9NZbkXfTu3fDd/Rk4KOHQsYQVFkkPHMmsHVrwHU/eTJw1lm04owO5Odz1sLxx3PH4vHQUsu4C7PnMYVgIDc3ciZR377MQuzXj5uu0aP5WEb+3lasAP74R5pFffpwod+wgTUFkydzxxlu4PRJJwFvvsm/DTU5aO9eoLAQ3nH7Ax91thBUgUcf5QyZ8nJg2DB+xj4f3eQffMBY7vjxPfC+MwERG8vXTcxlZGDyZKaVR6KhgV2PJ02iYshIZbB+PXDXXTSBRo6k1istBQYP5tZ88WLgb3/rnIbpMGwYcMMNbPi0aVNgNnBzM8/d2Ajccgu8BVQoHWMICxdSGYwaxZd1PmOPh5Za375sI2Ip9UZPYQrBwKGHcgHauzf086p0X5x+eoa7Y59/nm+wpKTzcyJUCosWcXEPx+GHs8XrMccwCLBuHT/Ys85iO+nRo0O2rlDlnPt+/cI3BCwupkdkwYJuvEfDiEAm/7yNGMnNZWv7O+6gJVBZGViUGhqoDCZNokckY6mp4RY9UrtkESqM+fMjO/OHDgWuugq48koGmbOz25lUoVpX1NQwFhpu3LBDaSmnUJ5wQkzvyjDiwhSCAQAYMwb40Y+A557jJtgJZpaUAJdfTmUQrfhs69ZAt4OhQ9NsDkJNDd90tH7dRUWRWzoEIxLyQwvVuqK1lS8dzRWXnU0PlGH0BKYQjH0MHQrceCN7s1VXc/EZNCi6m2jTJuCxxxiPdSamAXRFXX559FotV5CTw+htpKpXgDv+bgYuQ01MKylh/Lq1NbLirasDDjywWy9vGGGxGILRibIyBjaHDYuuDDZsAG6/nemoI0bQ5TFiBP921Srg5z+n5eB6KioYua2tjXxcfT37eHSDUDGEvDxWgm/bFv7vVGkdHHdct17eMMJiCsHoMqrAAw9wZ1tZ2X5j7fHQumhtBR5+OHUyxowIcOaZDAT7fKGPqa5mqk9XhssHEa799amnUjHs3t35b1QZY5gwwdryGD2HKQSjy6xbFygKDUdlJbByJbB5c9LE6jpHHQWccgrNnT17Ar6v1lZW77W0sBAjWnuKKIRrf11eDnz3u7TK1q1j9mp1NV1yVVVMD77hBhtLavQcFkMwukxVFf+N5HIX4W39eqbzuxqPh5lB48YxB3T9ej7m8bBce8YMupa6SagYgsPw4ZxCt3Qps4kaGujJOvpouuIysv7DcA2mEIwuE86zAnBz7fMFdrPharlch8fDGMHhh9NKaGtjxDdcdXIXiDZCMzcXmDKFN8NIJqYQjE5s28bip23bWAw1aRLTUp3dqfPvwIGd/7a6mo0nN22iQsjN5XoarleSaxFhvKAHcP0ITaPXYgrB2EdLC/DII6y78ni4KW5uBh5/nF0X+vdn0HPkSFYtH3ooK2tralgwtXYt8OGHDDIXF/MctbXAzp2cs/Kd77i3i3IycVxGadP+2ug12BbFAECXzt/+BsybRz/28OEMcu7ezYV+40b+W1nJx/70J+APfwAuu4z316/nzPOSEt48HrZZ8Pnofvd62SbIiqqiu4wMI1XErRBEpEjEruRM45NP2O155MiA3//TT3krK6NyaGxkILlvX9YprFrFquZbbmF77OZmFk7t3Uv3uwgwfTotiwEDqDg+/DCFb9IlmMvIcCtRXUYi4gFwKYArABwGoBlAnojsADALwP2quqZHpTR6nLlz6SJy4gNeL/Dxx+27bhYXU3GMHUulMWwYM2FOO4258QcdFOgAUVZGJRCcIllcTAvk8MOT/e56kGiVzSEIl3aaMHkaGylT8BdqGDEQSwxhDoBXAdwGYJkqr2YR6QfgBAC/FpFnVfXRnhPT6GnWrWs/L3n3bsYUgoPB2dks1G1p4VrjLPaLFjFuMGgQb+HIzc2Q6V9VVZyf/O67NIsGDqRWPPzwmKLnkdJOu0xjI/DOO8CsWRx0DTBP9YwzmK7khhnIhuuJRSGcrKqtHR9U1d0AngbwtIhEaXtmuJ2cHPr8HdraOm8uVXkL3vVnZwfa/nu9kdedpqYMmPj1+uuMvOfksCYhO5t+soceAmbPBr79bfrIIpDwGEJtLecKr11L396IEYE5p3/8I3DkkcCXv5zhvcuNRBB1i6KqrSIyTkROEpHi4OdEZIZzTE8JaCSHqVO5fjjk5HSuHWhqoisouPlaWxvXxalT2fUhEvX1ad6HZ/lyLvyDBzNdKieHWrOkhMGXPXuA3/8+8vg59EAM4e9/Z1OpUaPolwMCabP77cfg0MyZiXktI6OJekWKyNcBPAfg/wAsE5Fzgp7+ZU8JZiSXo47iGuJYCf360cXj7P4d13RwPYJTmDZhAtNQW1vDT17bsoUxh4MP7tn30aP8738cqxncuqKhAVi9mqPOFi3iwvvYY/ywwpDQGMKWLZytGW6OgwgV2EsvtTcBDSMEsWxRvgRgiqqeC+B4AD8UkZv8zyU0YiUifxeR7SKyLJHnNaLTvz9w7bXsOVRdzcV+wAAGkVes4ExfgN6RBQs4S37OHGD//fm3I0cCX/sa/3b9eq6TLS3cNK9bx2O+8Y009lrs3s0oe79+gce2bQNee43pVgADKz4f8Je/cLhEGJMpoTEE54uJFDzOy+OX8emn3X89I6OJ5YrMUtU6AFDVdaBSOF1E7kKCFQKAhwDMSPA5jRiZPp3FY8XFjE1+/DEXcJ8vMBfmpZe4Bm3YQItgxQrg7rupACZPBn71K+Dss/k3NTVcP2+4getjFNe6u2loYIDEWXhrahhUzssLWA1ZWQwqFxZSc955Z8jCC8dllJAYQl1d7FrWLAQjCrFcSVtFZKKqLgEAVa0TkTMB/B3AIYkURlXfFJGRiTynER8HH0zX8wEHcDHPyuKa8/rrXAudzeaUKcD48Xxs6VIOf//mN2lVnHsubxlFURFjA06a6Wef8d+OnU+dATqVlYHS7Q55tgl1GfXrx9eMhjP+zjAiEIuFcBWAdiNOVLVNVa8CcGyPSBUBEbleRBaKyMIdO3Yk++UznvXraQHsvz8TVvr1YxJL376sMxg3jkksO3YEOpkOH86/Wb061dL3IH378s3v2kXFsH59IIAbTFtbYDBynz70q3UgoS6jCROY9hUpkF1fz2wAG6RgRCGWLKONqhpy5pWqvpV4kaLKc7+qTlXVqeVpNbQ3PViwoL1nxOvlRjd47cvLo5JwhouJ0EsSYu3LLM4+m66i+vrO+bcAP5CysoBvLD8/5LSbhKadlpVxhkNVVej2s62tHFl3/vlWi2BExWrnjXbs3t2+03Nra+f6AscyaG7m8y0t9Khs3Jh8eZPKgQcy8r5zJ2MKra1chJuaGD0vLGTrbEdRtLSEdNMkPO30oovYMKqqioGexkbKt3Ej285ecgkHKhhGFNI158PoIfr2bR8H9XgCBWmO1aDKTfLixYEuCQBdSnV1oT0pGcPxxzP39gc/oDmVl8c3fNBBLNMOLtLYswe48MJOp0h464rsbOC66yjb66+zsZTHA5x0EhWFtZg1YiRmhSAiAvYz2k9VfyYiwwEMVNX3EyWMiDwOZjENEJGNAH6sqg8m6vxGdA47jMPCHAWQm0sPSG0tN8Cq3HjW1NBb0acPj9u5k1mWt9/OMZDB2ZkZx9ChwI9/DPz0p2xbEWp4zu7d/HAmT+70lBNDkEQm6YlQUY0Zk7hzGr2OeGzW+wAcBeAy//1aAPcmUhhVvUxVB6lqjqoONWWQfEaMoGdk06bAY2PH0mrw+bjO7drFNbGwkOtQSws3ylOmcFN8771pNCGtq4waBXzpS6xF2LQpkOnT1MSAs8/HNrAhehv51AePeCDWeM5wGfG4jI5Q1cki8gEAqGq1iHRv2rjhOkSAr3yFswvWruVOv6KCHRCWLqVCqKxkIzzHddTWxnY5+fncMH/6KbMyMz6pZfp0ll+//jqnCnm9dB+dey5wzDFhJ6551Wutrw1XEo9CaPXPQVAAEJFyABGm6hrpSmkpcOut7MQwezaL0Pr3p0v6jTe48NfUUCEMGsQUVWftE6FL+4MPeoFCAJhi+oUvAFdfTc2YnR215bRPfTYcx3Al8SiEewA8C6BCRH4B4EIAP+gRqYyUk5/PDfD06QH3z6pVTKzp35+b4dzc0O7znBwGl3sFDQ2B6UClpfThR6kc9vrMQjDcSUwKwR9QfhPAIgAngS0rzlXVlT0om+ESnA1vSQmVgxM7CEdzMyuWM5qWFuDZZ4FXXw30ClelUrjgArqMwnxIXvXaPGXDlcSkEFRVReS/qjoFwKoelslwKUOGMJi8dy8zjELh8/F22GFJFS25tLZyqPRHHwXaYDs0NHA4dU0NcOaZIf/cXEaGW4nHbn1XRDL5Z25EQYRp9bt3h+zZBlUm2EyfzsBzxvLuu8CSJUzJClYGAM2n4cOBp59mkVgIzGVkuJV4YggnAPiyiFQBqAfdRqqqh/aIZIYrmTCBxbqPPML7/fqxOO3jj1mLMHgwvSbbtmWoUlBlK9gBA8L7zXJyWBg2bx5w8cWdnvapr73LyOtlgGbOHLaZKCgApk2jmZXRVX6G24hHIZzeY1IYacXxx7Mwd9484PHHuZaVllJZ9O3LWTGzZ9OV/rnPZdic97o6LtpOA7tw9O1Ll1IIhdAu7bS2lmMuV69m/4+iIr7Gww8D//43h0gccEDi30emsm0bL8x58+i+q6hgr6cY5133dmJWCKpa1ZOCGOlFRQWVQEEB0+6DE2v69KGb/YknuME9/vhUSdkDOCXc0bScE2QOwb4YgtdLZfDpp5wwFHzOsjLGIX73O1ZEDx6csLeQsSxcyOFEAC240lIWyjz0EMvvv/1tXrhGWOJpXfGjUI+r6s8SJ47hBnbvZmFZWxtTTEeP7tzYs7kZeOYZBplDZVnm5DDe+tRTjCl0dLWnLcXF1Hj19dzNh2PPnrAN5fbFEFavpq9txIjQCqa0lBbESy8B11yTGPkzlXXrWCJfUcFdikNpacCHedddwM9+1nmGhbGPeFxG9UH/zwdwJgBLO80gamo4Dvh9f3cqZ5NbXg5cfjkwcWLg2BUr2KUhLy/8+QoK2N9o5Urg0EyJNHk8HCD92GNsXxEKr5cm0nHHhXzaB38M4Y03+CFFsjYqKzmv9NJL2y908bJlC5vx7dhBpTZlCjV9pvjzZs/mQh/uM6qspNL46CNg6tSkipZOxOMyujP4voj8DsD/Ei6RkRJqa4Ff/5oL+NCh7dtd19RwTOaNNwaGf+3dG1u/IlUem1EccwwX6Y0baQYFL6ptbWxDfeqp3PmHYJ+FsG1bdL92dnagR0hXFEJTE10m775LZZafzxqKF19kP5Ibb0zz2abge1ywILpbraSEStgUQli60/66EMB+iRLESC0vvMBNZKg1rLSULp8HHuDYzMJCbsZUqUja2vh8UVHoDWfGWegFBcC3vgX84x/s0aFKDer1cgE/7zzgnHPC7r73xRAKC0MO0GmHKgs7uvIh+nzAX/9KGTu6pVTZlO+OO4Af/jC9s5kaGgLfQSTy84Hq6uTIlKbEE0NYCn8fIwBZAMoB3N4TQhnJpbGR/dkGDQp/jOP+WbSIrvGmJqbiA4GZCWVlTIhxzuP18rmMTJIpKQG+/nXu8pcv56LUrx99Y1EW131ZRtOnU8t27BXe0sIPWIT/7r8/tXK8rF7NoRUdA9YA7w8aRDfKvHl0g6UrjuXUcZJTR5qaLKgchXgshOCyyzYA21Q1hunehtvZto27/Gib0MJCxg5qaoAnn6SnwWn7r8rf2zvvcE0cM4Yb0GnTwlc1ZwSVlXEXXHh9/tYVkydTeezZww+pro59kTZsCEwlamlh6qnTOC8eXnstep+RigoGrU87rXPmQLpQUMCYyNKlbLcbjtrasHEdg8RzBXxVVav8t02q2iYiv+kxyYykETwNLRrbtwP/+Q+7Ph95JJVBdTXXq/x83l+yhL/NkSMZjDbas89lVFAA3HwzTTSnMG3DBvrecnL4xey/P3fw993HQHU8VFVFtywKC6nhGxu7/obcwIwZ3JE0NYV+fscOpqJOmJBcudKMeBTCKSEeS2M703CoqKBCaItg7znB4a1buVbl5NCiOPpoxhXa2vi80+V01CjgO9+xWqBQtCtM228/4Ec/4u61qSkQmCktpXk1YQKPWbgQeOWV+F4oO5txhEg4lkg0/7vbGT2aA4u2b6dp2toaCMZXVfGCveWWyGlxRnSXkYh8BcBXAewnIh8FPVUC4O2eEsxIHkVFTJx5803u/IPx+YDNm5k6umNHIFbQ0MA4ZW4uJ6qNHh3obyTCNS1Ua2wjROuK3buplSdPposoK6uz/27QIGYGnXJK7EUdzjzUSPUSu3fzC8yEhXL6dF6Uc+ZwYFFLC+Mzl11Gc7YrcZheRixOyX8BeBHArwDcGvR4rapGSZEw0oWzzgI+/JCZRgMHclH3+RhEXreO/z/8cGDZMq5Hy5dz4zV9Oj0fHk/72F5LS0rfjqvp1Nxu8WIqgKys8KmlBQXUyJs20RcXC0cfDTz/PDV1qAXf52P84rrrMqceYehQ4MorefP50jcukiKiflqquldV16nqZQBqAFQCGAFgvIgc29MCGolHla7qpUvpum5q4kbqttu41lRVUQm8/TafLyjgBmv06MBMhLIyup0XLuxcj1Bba50WItGp/XVTU2wBY48nvjhCRQUnuW3cyIU/+Iuqr+eM1JNPzly/uimDuIkn7fSLAG4CMBTAEgBHAngHwIk9IpnRIyxfzp5pVVWB30tODj0RZ53F0ZmbNjGb6I9/BI49tn2h2tixtBry8pggs2tX5/kI1dUsrDVC02mm8pAhwHvvRf4jVZpeYeY0h+W44/g3Tz/N3uROjnCfPmyHcfzxmWMdGN0mnjy2mwAcBuBdVT1BRMYB+GkihRGRGQD+ANY5PKCqv07k+dON2lpu4pyeQsOHd++3u2AB57qUlbWvU2ppAWbO5GvddBMVQEMDW1Z0bOo5eDDwySdMTCkp4Tm2buU5VRlvGDUKmDSp63JmOp1iCEccwQU7Uh79jh3AwQczU2bXLub3rlvH4ydM4Acezt106KHAIYfwy3GCO8OGpX8g2Ug48SiEJlVtEhGISJ6qrhKRhJUciUgWgHvBbKaNABaIyP9UdUWiXiNdqK9nauf8+XSDinCtGDaM8bGDDor/nLW1HOQ1cGDndSM3l66ipUuBuXPZdaG5ObTyyc5m8sv77zMe2dTEf53EjgMOYDeEYJe1Ki2STz7hMQMHcm3LuArmGOkUQygvp+vmpZf4RXR0ddTV8YM+7zyO7Zw5k48XFfECee89pnN95Sv0/a1Zww96wIDABy1CS8QwIhCPQtgoImUA/gvgFRGpBrA5gbIcDuATVf0MAETkCQDnAOhVCqG+nt0ENmzgbtxxLavSDXzHHaxTCm40FwsLFnCNCLeJFOFCPWsWcOKJTMjw+ULXKOTnMyupupo1B/vvTyVxxBHMkAw+futW4P772T0V4Frn83H9uvxyBqV7m8fCq97OIzQvuYRa//XXuXMvLub9ujp+WN/8Jv19zzxD8y445lBeTgvimmto0pWUBFxDhYX030WY8WwYDvE0tzvP/9+fiMgcAH0AzE6gLEMAbAi6vxHAEQk8f1owaxZ30x0TSUToCs7NZcv3u++Or9fZkiVcJyJRWEhvxM6dXFcGDqRrqE+fzseK8PExY9hRuGP3BYBr1C9/SZdXx1Y6jY1ss9PWlmHzEmLApz7kZnUwj7KzgauuYjDnrbfoDsrNZQXu5MnU5nfdxS+mYwC6sZGZSo2N3DUcckjgw25spGnY2gqcdFIy3p6RxsQchhfyeRH5kaq+AQaWJyZQllDbl079NEXkehFZKCILd+zYkcCXTz1NTew2EClDp6iIx33wQXzndvoKRcNJN3XmJ+/aFXp+ss9HxXX88aGVAUDvhjO0quPmtKCAsYrHHqNV1JuIOFN50CB+8N/6FnslOXm9ixbxQw9Vg7BmDS+KAQPov3OqAwH+7bBhwL/+Re1uGBGIJy/rPgBHAbjMf78W9Pknio0AgsuihiKES0pV71fVqao6tby8PIEvn3q2buVGLpaeQsuWxXfu0aMZR4hEayuVhpPIMmUKvRDbttGFVVvLxXvLFiasHHMMYxqh2LuXHZcjNczLy+NrLlwY33tJdzqlncbC5s2hL4zWVloTxcWBSW4d21Dk5gZiDYYRgXgUwhGqeiOAJgBQ1WoAiQwLLgAwVkRGiUgugEvRy+YtROsy4OAEmeNh2jT+TaS/27IFOOGE9q6oE04AfvMb4MwzAy12DjuMHZO/+MXwRbPbt1POaIksBQWB+EJvoVPaaSzk5YX+8urrO7eeCGUKFhVxOpthRCCeoHKrPxNIAUBEygHEuIRFx98s72sAXgLTTv+uqssTdf50wDF4ojW2bGjgjj8eBg5k/68XXugck1Slv7+0lE0vQ8l13nm8xYozbU2Vm1gRvmZH11E8jfUyhU5pp7EwfjyrjiPh+AVDBX164wdtxE08CuEeAM8CqBCRXwC4EMAPEimMqs4CMCuR50wnSkq4k3/nHfrXQ+G4dY7oQrj94ovpPXjhhYA72rEaRowAvvrV8PGAeCkuZirq0qVUcKo899ixgdYYAOMT48Yl5jXThYgxhHDsvz/9bzt3MlbgUFzMC6KtjbGDsWNDm20NDV3LVzZ6FbE0t/unql4JYACA7wA4CQwAn6uqNlM5wZx9NjOCtm/nzjx4U9fczC4El18eehMYDY8HOP98JpssXsyYRX4+k1ISOV53507gzju5BjU1sahOlevVO+8wNXXCBD5fUBB/Cm2606UYgscDfO1rwK9+xYtg4ECaXNnZ3D0sX06tHmoakdMaw0ZHGlGIxUKYIiIjAFwL4BEAjztPiEg/a3CXWMrLge99j6mlTnsJZwOYm8vMxJNP7t5r9OnD2EBP4POxGrq2lr3V5s1jgLm0lMHwggLg00+5PhUXs6ait3VF7VIMAWBh2Y9+xMK0t98O+OUqK3lxFBV1jh/U1tIfeMMN6T0m00gKsSiEv4D1BvsBWBT0uIDxBJurnGAGDQJ+8hMGW1evpmUwaBB31V2Zs55MPv2USS9OHcXRR9NttGlTwAJRZZbSAw+wq0Jvo0sxBIeKCnYnvfhi5gR7PLw4GhuZw/v+++2P79+f/UimTOm+4EbGE1UhqOo9AO4RkT+r6leSIJMBLp6jR8cfPE41ixa1d2Hn5zMrafx4VjY7xbPhCt56A12KIXSkpKR9pWFODltXXHIJtXJbG4M2Y8ZYzyIjZmKJIYiSsMrAOSaxohnpSG1t6JhmQUF766auLnTBW28gZOuKWGhpYepofT0/zP337+xv69cvcZkBRq8jFpfRHBF5GsBzqrreedBfK3A0gKsBzAHwUI9IaCSdhgYu2Hl59P3HE2wuLw8/1tZBlbGGaK00MpW4XUY+H/Dyy8D//kfXkBM7yM0FTj+dRSKxzFMwjCjEchXNAAPKj4vIKAB7AOSDtQIvA7hbVZf0lIBG8ti4kVMa33svsGiPHQt87nP09ceiGA4/HPjvfyOnve/ZwxjDwIEJFD6NiMtlpAo8/jgwezaDyhUVgedaWtjsbvt2VgnaQBijm8QSQ2gC21bcJyI5YPppo6ru6WHZjCSyYgV7pzkxyqwsrkVbtjCF9PzzgXPOia4UBg9mjcR773VuaAfQTVRdzfWrt9ZJRUw73buXvUJE+GFu3kzrYOTIzrEAp2/5/Pn80DN18pmRNOKyM1W1FcCWHpLFSBG1tcA99zDIG+zGEWGSSp8+3IiOGcPgcDSuuYab18WLGUAuLmZtwpYtPOfVV8d2nkwlZNrpjh3sBvjuuwFN6YzM9HjCB4Y9Hvr1Zs82hWB0G3M8Gnj/fe7cw7lwsrOpKF58MbaFPD8f+L//Y/zzn//kBre5mQW2FRVcu5YvB770pc4T2XoDnSyErVvZJ7yhgW4hZ/FvbaUmLitjrUG4go3+/Tn82obKG93EFIKBt99uPxM5FAMG0K3U0MBdfzScze2mTeyKWloaeE6VKfS//CXwgx+Eb9ORqbSLIahyglBra+cPIieH5lVjI4s5Djss+cIavYoubSdEpJeGAzOThoboSSpOZ+WWltjO2dYGPPggN6/BysA514AB/Pexx7omczrTLsuoqooViMHB4mD69eOXs2lT57bWDk6U3qwDo5t09QrqtQ3oMpFBg6gUItHSwg1rUVFs51y5kvHRSKmlFRX0dGwOmnqxcyf/9uOPo8uUrrSLIXzyCf8NF2EfPZrWA8BofEdU+fjppydeUKPX0VWXUS/ND8lMTjyRAWBn1x6Kbds43THc/IOOVFVFzyJyrI4tW9hx9T//oWfEGQecnc2eS2efHbsiSgfaxRDa2iLv7MvL6UpavZrHtjuRj5OLDjkEmDSp5wQ2eg1dVQh/S6gURko58EDWG3z2Gdeejgt5dTUzHOMZyRtPSummTZyvnJXFaY/O+tjayoD0qlXAd76TOUqhXQyhsjLyZCQRLva1tbytW8cPyJlzevTRwBVXxK6pDSMCXVUIzyRUCpdTW8tMnFde4eJYVMRZwtOn00ee7mRlcXzvH//IjWhBAd9jSwvfe58+XJDjmVg6ahR3+ZHw+WgZzJrFQHXHwHZODmsZqqpYpBtuXGe64VVvIIZw8MH8sBsbw3curKvjxfbd7zI9a+9eBpvHj7c2FUZC6apCmAVgciIFcSsbNwK//S2bsfXvT793czPw3HMcNHPTTZkxd6SkBLj1ViqEN96gi6i4mOvQxInxt6g+4ACuVXv3hm9it20blcyuXZGrlgcPBubMYWFcLBlObqedyyg3l5rur3+leZaX1/7g+nruQr70JWrM6dOTLm/KaGvjxeFMV4o2bNzoNhZDiEB9PfC733EnO2JE4PHCQt5qa4Hf/x64/XZa/umOx0P30YEHdv9cWVnAl7/MecxeL9C3b/v219u28fc9diyVbSRycrg2bNgQev5LutGpdcX06fyQHn2UfrKCAn5Izc3UxDffTEuit9DYCLz2GvDSS4HMgrw8DgI55ZTe2wQrCVgMIQILFgQy+kJRUsId8Jw5wKWXJlOy9GD//YHbbgMeeqh9kFmVSufqq4FXX40tW1Ik9Iz5dENVodD2ze1EgOOO40SzhQsZzBHhhzRxYmerIZNpaOAu7NNPaTY6PtnmZvoN33+fpmy0whmjS3RJIajqfYkWxI289lp0F21FBRXCxRe3X9hU+btevJgu4PJy1hVlgiURD2PG0IJat44FuR4PA8eDB/P5oUOjt8F2Gu0FjxJOV3zKAHLI5nZFRVQMxx2XZKlcxBNPAGvXMggVTF4ed2abNgF//ztwyy0pES/TsUrlCDixu0jk5jL42tIS8LPv2sUxkmvXMnUyJ4eL3tNP0ztw5ZW9a9Mnwt93x984wE3xo4/y8wvnIt6xgx6TcLVb6YSjEDo1t2trY8D4pZcCF86kScy7HTmyd3QCrKkB3norcun64MHARx8xV3nQoOTJ1kswhRCB0lK6MyPFslpbueA7x9TW0m9eU9O526fPxxnDzc0cbmWFpVS4F17IiuVhwzp/1nv38jO++OLUyJdovEq/VzsLoakJuPdeLnQlJTRLVdno7s03gXPPja3VbLqzZg1/JJEmvDmzIFavNoXQA0RdkkSkXwy3su4IISIXichyEfGJyNTunCuRnHACsHt35GO2bweOPTawuL/5Jh8bOLDz79fj4Wbv/fcDBaoGcNppTKXfvj3gWtq8mf8HmG2ZKU3w9lkIwTGEhx9mRd7IkfSLOTuMwYO5W376aba4znRaW2NTeh5P9ClMRpeIxULY7L9F+qayAHTnJ7sMwPkA/tqNcyQcZ9hLdTWzZDrS0MDNyokn8r7XS4s/UpxAhEkkr7/OoKvBz2TGDLrTFi1iADonh26igw/OrGFgXl8HC2HbNuCdd6jxQi2G2dm8oJ57Dpg2LbPnI/ftG714BeAxmRBQciGx/NRWqmrEungR+aA7QqjqSv95unOahFNSwtjV734HrF/PwHB+PjcyzgyTG28MBEid0ZPRAtGlpYHdrxGgpIQFf5mM4zLaF0P44APueCP5D4uLeQFWVQH77ZcEKVPEmDFUCnV14YN3zc0MwPXmgRo9SCwK4agEHZMQROR6ANcDwPAk+BFGjQJ+/nPGul5+mYqgoAA49VQmgwQXVDlTxiKNjwToJrUam95JJ5fR7t2xt52oq4v9hVpaGJN4910W1FRWss3F6NHujUVkZQGXXMKMjJyczpkXbW2sFP385+OvlDRiItYRmiERkWtU9R+Rjgk69lUAoepRv6+qz0X7+yB57gdwPwBMnTo1Bvuy+/TtyznmZ54ZeQZJQQEVyK5doV1MDrt3s2Gb0fvo5DIqLQ10M42EavjWFh1Zv54Vk9XVrKDMyWFe/9y53Fl/5SvubQx1xBHM5PjnP/ljKy2lAqup4Wdw0UXcjRk9Qne9sz8F8I9YDlTVk7v5Wq4gkmUvApxxRmAcZahjnZz73tSBwAjQKe100iTgqacim5UNDVwYQ+XtdmTnTuCOO3jxBZfXl5XxNVas4A78W99ybzzi+OM5DvTdd4Fly6gYnOZh8TTUMuImqkIQkY/CPQWgl5VZRWfyZLqS5s5lVpzTe0eVKZS7d3PAvF3XvYcdO1jxnpMDePp0sBCGDGE18tKloVvNer3Muf/CF2KLrr/2GnfYw4Z1fk6Ej69YwYETiehR0lP07csZDzbnIanEYiFUAjgNQMfpHALg7UQIISLnAfgjgHIAL4jIElU9LRHnTjYeD3+7I0YAzz/PDZsINznDhwPXXgscemiqpTSSwSefMGN01arAjAeU+oC+ADRod/7FLwJ3380/6NeP1oAqL566OublnnBC9BdsbWXZfLQ0t8JCHudmhWCkhFgUwvMAilV1SccnRGRuIoRQ1WcBPJuIc7mBrCz24Tr+eLpzm5v5Gx882L3xPCOxfPgh3fhFRe0zSrc00kKY96YHX5zs99oUF7O/+MKFwIsvMpvI4+Hgm1NPZTvdWC6c+vrIJd8OxcVsAWEYHYglqHxdhOcuT6w4mUV2dmZnCRqhqasD/vxnpsp3jN3mFzCG8NmnWZg/P6htUV4efeRO51OPJ/7dQ04OTdFoaW5tbZ0HXRsGuj5T2TCMMCx4owFNazejaPE8umYWLdrX119BC6FPqQcvvBBmWFpWVtdMyaIiVjtGK6/fswc48sj4z29kPLG0rliciGMMo1ewbBkW/XwWStcvowuntZV9OObNA959F9rKNLPC/Czs3Bl97Y6bz32O2QvheoXX1dGldMQRsZ/TGVSze3fkcZ9G2hNLDOHACJlGAIPLYWZiGUYvYt064K670JJ9IbL6lAD5/hzj3Nx9U4F8K6qAiYDAAxGutQnl0ENZMDNzJmcJ9OkTyGrYvp0xhptvjs1lVFfHHisvvxwYVNO3L62L4cPpEx082JrMZRCxKIRxMRyTAaNLDKObzJwJ5OZiZEUD1q6tRJ/8oEEPIkCfPtCdVQAAX1sWsrJ6YM6LCIu3Ro1imtuGDYEOoVOmUFkE1yeEY88etu3dsoXl+AMGUEEsWMAGX337MtidlcUxdp//fOS21UZaEEtQuSr4voj8DGxmtwTAElVd0zOiGUYasXcv+xINHYqjCzbg5U/HwKeAJzgUIALN4gN792Th/GN7qAODCKcxTZ1KV09LCzOL4gkkP/gg016dcYG1tXR7+XysnaipoUts/HhmRf3858D3vx+6/sFIG+IOKqvqjwDcA6AWwAUi0ivGaRpGRGpquBB7PBhWuhfHDK/Cuj1l8Gn74LAvhz+5/DwPZszoYZlEuLMfPDg+ZbBpEyuEna6NAPNofT4qFo+HnQjXrmWMpLKS7qMHHoitW6nhWmJuXSEivwdws5JtAGb7b4Zh5ObuS/kUEVw14UPkeLyYs24URBQF2W1o83mwOZd5qBddkOXeDs4rVvBfJ9OppobWQp+gUGFWFt/v7t2B2cdVVYyjhGqxsXMn8PbbVDQAMG4cU2x720xZlxNPL6M6AP8TkUtVtV5ETgXwY1W1rjyGUVHB4GptLVBaipwsH66a+BFmjP0U720cgi11xSjIbkNO/ht4HkD5ABdnfDc1tW/EVVPDfzumwooEspmc59avb68QVBmUfvJJ3neUyqefMuZyzjm9YxpcmhCzQlDVH4jI5QDmikgzgHoAt/aYZIaRTogAZ53FUZiOWwVARVE9zjrgYx6zezfe9Qea201Mcxv9+7dPWw3nBlLtHATpmJY6fz6HZnecj9qnD1Osnn6a9ROnnJIY2Y1uEY/L6CQAXwIVwSAA16nq6p4SzMhcdu3ikLAFC+iCHjWKbT5Gj07zOdNHHsl+RC+/zJ5EZWVUFK2tnAuanw/vRRcCs55oP1PZbRx6KGMCzsBwZ1hNcAV0Swt7InWcBhU8IKStDfj3v2k5hWqnkZ0dGBF67LGd5x8YSScel9H3AfxQVeeLyCEAnhSRW1T19R6SrUfw+QLxv5KSNF+A0pC332YCiyozFz0etvCZP5+JMaeeSle0E6scPz7168Tu3ZT7nXfoTRk4kBvagw/uMNtGhOmXBx4IvPAC/ekiPOjkk4GTT4avYRWAoPbXbqS4mNbOU08xy6isjEHppibOZPB6mYJ6+OHt4wwDBjAF1WH1ah7Xv3/418rLY7Ov5cvZKthIKfG4jE4M+v9SETkdwNMApvWEYImmqYlZcy++yBRrgJubM85gbCvVi05vYPly4C9/YfJKsKehqIiu9wceAP71L3ZfcLqDFhRwiNaxx6bGzfzBB+xL1NbG6yUnhwrr7rvZp6pTjZcI0z2nTOEi2dbGBdZ/gXnXLQcAd1sIABVCYyOHhGdlcbzle+8FKp0nTgzUHdTWUmt++9vtd1jODy0aqhzmY6ScLg/IUdUtfjeS66mvB+68k3GsigoWWQK8th96iHM4br459oFUmYoqP6PVq7lpGzyYc0oS8bmo0jNQVtbZ7dzQQAshK4v/HzAg4KVoaqKi8HqBE0/sdNoeZd064I9/5AY3uEldfj5l3LCBz992WwhL01+I1pFOIzSTjSrTSpcv5w+jf38O6emYlurxAJdeChxzDHdSa9bw/Xz2GS+I7GwGkH0+fqnf+hZNpmDimRNrOzJX0K2JaaramChBepJHH+WurmM2XHExf+hr1gBPPAFcc01q5HMDW7ZwJ7x+PdeyrCxubnNzWfh68snd26Fv3cq09VBjsFevpouotJSbyk2bAp6H/HzGI//1L7qUSkq6LkO8vPAC33+4aZNDhvDaWb069tECnUZoJpM9e4C//Y3KQISLelsb8MgjnLlw/vmdh/AMGULF4OD18g2vW0flMnw433yo4T1jxwYykcJNZ/P5eEywq8lIGd0doel6du2iBRCugFKElu/8+cAFF/TOrsA7dgC//CV/tyNGtF/4m5u5Xni96FYhVU1N6I7Ozc1UQs5Cn5PTeZZ8bi5ff8GC5FkJdXVsUhqpG4MIFda8eXEoBKVCSHoMob4e+O1v2c+o45fc1sY2Fw0NwNVXR9b8WVlsWXHQQdFfs6yM/ti33gq9EwDY+G/yZBsh6BJc7sjsPqtXcyMTKXjs1Nis7qU5U//9L93FFRWd14K8PP6W//MfdmcIZutW4NlnmWn54IPAkiXh58Xn5YXOXqyv31fgC4ALfyhPQ2EhE3iSRV1de7nCUVhIhRorKXMZvfEGTa8hQzp/ydnZDB7PnUtTOpFcdhnP/dlngQZ5AC+4tWsZob/66sS+ptFlMt5CaGqK7ThV7lZ7GzU1tKCCuxR0xJm78v77zK5pbaUb7o03uGAWFnKTOW8eXdJf/3rn/mnDhtH6qq9v74Jx+q45+HyhZYmm1BNNfn5ss2acNkGxkhKXkdfL4HBFRfhjPB5+0XPncgZsoigs5DS4OXOA2bMDM2ULC+mLPOGE8D45I+lkvEJwuv9GI0wMMOPZuZP/hnPxOhQWcpOnCjz8MPDmm9z4dVykd+8Gfv1r4Cc/ad+VICuLiSsPPcRYjvN3Tg2X18tNY58+nVPbAW4ukzkCuE8fusC3bImcNVlXR69IrOyzEJLpMqqrYyZQ376Rj+vTp2fMsPx84PTTuZvYsyeQcxwq7mCklIx3GR18MF0QkXb/TU1c8MbF0ug7w+i4Qw+HKhf1TZsYbwmlDAAu5q2twKxZnZ87/nje1q5llqEqN6WDB3Phzclpn9ru0NDANSWZaeoi7BS9d2/4mQW7dlFZHHpo7Od1YghJtRCcLznaF+18yT1FdjbTs8rLTRm4lIxXCPn5wLnnAhs3hv5ht7VxkTv//A5FRr2EQYP4vltaIh/X0EDlOn8+f8uR3DcDBzKOWF/f/vGsLGZyff3rVBzr1zN1c+BAZj6OGdM+JVWVC/K2bcAXv5j8tOBDDgEuvpgybtsW6MrQ1ETZAaYrx5NdmZIYQkkJP+Ta2sjHVVczz9jotbhCTYvIbwGcBaAFwKcArlHVPYk6/2mncXF64QXsG0qiSuvV66UrM9k57m4hP58ppS+8EGh935H6ei7GkybRVRTNZ56dHVjMO7qHPZ5Aq/76en7+RUVUOI8/ztonx2+vynXs299mxXKyEeFEyjFj6IJfsoSPFxYCZ58NHHdcdC9MR1ISQ3DeyP33UzmE8qE6O4Kjj06eXIbrcIVCAPAKgNtUtU1EfgPgNgDfTdTJPR7gwguBo47igraK3QNwxBGsu0n1BEBVWioeT89a7OE44wx2JV6/nu4bx5p3lGZNDfCNb1B55OWFH9cbTDTvg0h7xVJaCnz5y9yRr13Lz6N/f1YDp7IRpghdiePGcc1sbaVy7GqAOyUxBIB9lhYuZOn1kCEBs0aVmnn7duCqqyIHno2MxxUKQVVfDrr7LoALe+J1hgxhFpxbaGxkhs+LLwZSFw89lP18DjooeQthURF34U89xUwhx9Xs8zHl9IYbAgHdqVM5KyVSoLW2NuAqjpe+fePfdSeL3Nz43EOhSEkMAaCWv/FGtpx+7jn6ULdsCaRJXXRRaswww1W4QiF04FoAT6ZaiJ5mzx6201i/ngvniBFcgNes4Sbu9NNZIJospVBczGzDCy7gDt3rpZ9/+PD2MkyezKrh2trQVcOq3Gxed501DgyF4zJKSeuKnBz2IJo9mybZsGH8EouKmEL2wx8C//d/8UXJY8C5rt98k3VoBQXAtGm8lgoLE/pSRjdJmkIQkVcBDAzx1PdV9Tn/Md8H0AbgsQjnuR7A9QAwPFz1o8tRZZuIrVvbt9PweGix9+9Pq2HwYPqpk0lJSeT1oLCQG8277qLfv7w8sPDX1/M9HXUUXXFGZ1LmMgLo+7vzTn6JHYPHw4bxC7znHuD22xPmR62vB+67jy7JggLqnl27WMj4xBPATTcxvddwB0nbw6nqyao6PsTNUQZXAzgTwBWq4fPjVPV+VZ2qqlPL07Tcfd06VkWHKwbLyqJimDmz87wRN3DwwdxMHnAAPQ8bN9LSaW2lG/r661MTC0kHUuYyAhixb2hgVkUonAyAuXMT8nI+H5XBypVMWBg4kBuOfv1oEefksJvGli0JeTkjAbjCZSQiM8Ag8nGq2hDt+HRn0SK6dCO5g4qLA2mZHat+3cDIkQw0795N95dTT2CKIDIp7Xb6+uuRgz8Aqwnnzk2Iv3LNGloGI0eGPlWfPqyZe+mlxBZHG13HFQoBwJ8A5AF4RXjlvKuqN6RWpJ5j797YgpMeDwPPbqZfv9CVxUZoUtrttLY2ejm+U5TS2trtCPobb9BNFEmvVFayZuWSSzK0/XxbG7BiRcA6q6xkAGXYMFfOkXaFQlDVMamWIZn07x+9b5IqA7vx9Mkx3E9KYwhlZdxhRKrAbGlhbnECqjS3bInepig7m66luroMVAibNgF/+ANTCPPz+ZkuW8ag/pQpqam2jILlgaSAww7jYh+pk0BNDdNkhwxJnlxGz5PSGMKJJ9LHF4lt24CTTkrI7rWgIHz3Wweno0bGdQlwmno1NNDnW1lJU3rIEN5ftIjjA10WJDSFkAKGDGE+//r1oZVCSwszMc47z5VWpdENUhpDOPxwppuGUwq1tQwCJSi1bdq02Lpl7LdfBjaWfO21wPi/johQKSxZktye7jFgCiFFXHsti8/WrWPH0ZYW9shx6oU+/3mbOZ6JpDSGUFzMUZcA5x7U1dFUbWjg7qSujs2ZElStPHkyXUYd52g4eL1MSDjjjAzb+LS2UiEEt/vtiAhNqDlzkidXDLgihtAbKSzkb2/FCuDll/n7zM5me/jjjos8qctIX1IaQwAYzPzFL1gi/8ordBEVF7O747RpCc0QKCxkJtodd1DXVFYG+lxVV/N21lnskZVR1NXFFpQvKWEaoYswhZBCsrNZBJbgwlDDxaQ0huBQUsLZBKec0uMvNWYM8NOfMrX0rbcCQ4f224+pppMmZZh1AAQi5dGmK7W1uW5mrykEw0giKW1dkSIGDeLif8kl3Dzn5nIdzDhF4FBcTI23c2fkxlx79rBHjYuwGIJhJJGUu4xSSEEBW53EOsUwbRFhYKS6OnwWUUMDLYkjjkiubFEwhWAYScQVLiOj55k8mS65tWvbp1r5fOz+uGMH8NWvui69ylxGhpFEfOqDQCAZvUU2IAJccQW7Vz7/PLNGPB4qhEMOAc45Bxg9OtVSdsIUgmEkEa/Pa9ZBb8Hj4QS6adOYzdXczOCJi3u9mEIwjCTiU1+vCigboGJI9VjGGLGtimEkEa+ahWC4F7syDSOJ+NTXKzOMjPTAFIJhJBGLIRhuxq5Mw0giFkMw3IwpBMNIIhZDMNyMZRkZaYfXy5nUr7/OGST5+cCRR7LoM9y4YLfg9XkthmC4FlMIRlpRXw/cey+wfDm7aZaUsL3yE08ATz0FfO1rwIQJqZYyPOYyMtyM2a5G2qDKIVOrVnFwe2UllUJpKeeNlJUBv/89uwW4FXMZGW7GrkwjbVi7Fli6NPx88qIiuo9mzky+bLFiaaeGmzGXkZE2zJ/P1smR2gCVlwMffEA3ksv6hgFIYwuhthb46CN28CwsBA4+OPJEMCMtMYVgxExdHUd+er0cFTt4cHLbGG/bxhbKkfB4eKutdadCSLsYgtcLPPssMHs2B7pkZfExgB09r7mGgRwjI3CFQhCR2wGcA8AHYDuAL6jq5tRKZTjU1QFPPw3Mm0c/vnMbPRq47LLkNW0sLuZkwkg4skWbXpgq0qowTRV49FHOBx4+nP37HXw+4MMPgTvvBL773eia2kgL3HJl/lZVD1XViQCeB/CjFMtj+KmrA37zG+CNN4CBA+m/Hz6ct23bOJ535crkyHLkkZwrEom9e4EhQ+g6ciNpFUNYt45D4EeObK8MAJphw4YxsPP226mQzugBXKEQVLUm6G4RAE2VLEZ7Zs4ENm7svEEUAfr354TAe+8FWlp6Xpbx4+mq2rkz9PNeL7BrFwe3u3XcQFrFEObOpanliSBveTkwa1b4yWBGWuGaK1NEfiEiGwBcgQgWgohcLyILRWThjh07kidgL6ShgRvEwYPDH1NSQivio496Xp6cHOAb3+Biv34928sD9Gzs3MkZJJ/7HHDYYT0vS1fx+rzpE0NYvTp6pV9xMQPN0Uw3Iy1ImkIQkVdFZFmI2zkAoKrfV9VhAB4D8LVw51HV+1V1qqpOLXerXyBD2LyZccScnMjH5eUBK1YkR6ahQ4Gf/pQL/549wIYNVA5DhwLf/CYHubvVOgDSzGUkQm0bCSdo4+YP3YiZpAWVVfXkGA/9F4AXAPy4B8UxYsDrje137vEEEk+SQb9+wAUXcAphfT1dWUVFyXv97pBWLqMJE4BXX4384dbU0IQsLEyeXEaP4YorU0TGBt09G8CqVMliBKio4OYvmnu4sZFxx2STnc3U0nRRBkCapZ0eeyxNxHCpXaoM2pxxhlkIGYIrFAKAX/vdRx8BOBXATakWyGDAePJkYOvW8Me0tHBhnjo1eXKlM2mVdjp4MHDhhfTJdYwRtLYyC2nSJHYVNDICV9QhqOoFqZbBCM0FF7CR3K5dzCoKprmZGUhXXmm1SbGSVjEEgMGa0lIWogQnceTk8Llzz+2ckmqkLfZNGhEZNAj43veA++7jhjArizEDJ9h89dXASSelWsr0Ia1iCABdQcceC0ybBnzyCWMGeXnA2LEWN8hATCEYURk2jAVoa9awCK2lhVk9EyfamhAvaRVDCCY7Gxg3LtVSGD2MKQQjJjwe4IADeDO6TlrVIRi9jjSyXQ0j/Um7GILRqzCFYBhJJO1iCEavwq5Mw0gi5jIy3IwpBMNIIuYyMtyMKQTDSCLmMjLcjF2ZhpFE0jbt1OgVmEIwjCSSVq0rjF6HXZmGkUQshmC4GVMIhpFELIZguBm7Mg0jiVgMwXAzphAMI4lYDMFwM3ZlGkYSsRiC4WZEo81MdTEisgNAVdBDAwDsTJE44XCjTIDJFS9ulMuNMgEmV7ykQq4RqtppKH1aK4SOiMhCVXXV7C43ygSYXPHiRrncKBNgcsWLm+Qyl5FhGIYBwBSCYRiG4SfTFML9qRYgBG6UCTC54sWNcrlRJsDkihfXyJVRMQTDMAyj62SahWAYhmF0EVMIhmEYBoAMVQgi8i0RUREZkGpZAEBEbheRj0RkiYi8LCKDUy0TAIjIb0VklV+2Z0WkLNUyAYCIXCQiy0XEJyIpTccTkRkislpEPhGRW1Mpi4OI/F1EtovIslTLEoyIDBOROSKy0v/93eQCmfJF5H0R+dAv009TLVMwIpIlIh+IyPOplgXIQIUgIsMAnAJgfaplCeK3qnqoqk4E8DyAH6VYHodXAIxX1UMBfAzgthTL47AMwPkA3kylECKSBeBeAKcDOAjAZSJyUCpl8vMQgBmpFiIEbQC+qaoHAjgSwI0u+LyaAZyoqhMATAQwQ0SOTK1I7bgJwMpUC+GQcQoBwN0AvgPANdFyVa0JulsEl8imqi+rapv/7rsAhqZSHgdVXamqq1MtB4DDAXyiqp+paguAJwCck2KZoKpvAtidajk6oqpbVHWx//+14EI3JMUyqarW+e/m+G+u+P2JyFAAnwPwQKplccgohSAiZwPYpKofplqWjojIL0RkA4Ar4B4LIZhrAbyYaiFcxhAAG4Lub0SKF7h0QURGApgE4L0Ui+K4ZZYA2A7gFVVNuUx+fg9uXn0plmMf2akWIF5E5FUAA0M89X0A3wNwanIlIpHkUtXnVPX7AL4vIrcB+BqAH7tBLv8x3wfN/ceSIVOscrkACfGYK3aXbkZEigE8DeAbHazjlKCqXgAT/TGyZ0VkvKqmNP4iImcC2K6qi0Tk+FTKEkzaKQRVPTnU4yJyCIBRAD4UEYDuj8Uicriqbk2VXCH4F4AXkCSFEE0uEbkawJkATtIkFqXE8Xmlko0AhgXdHwpgc4pkSQtEJAdUBo+p6jOplicYVd0jInPB+EuqA/LTAZwtImcAyAdQKiKPqurnUylUxriMVHWpqlao6khVHQn+mCcnQxlEQ0TGBt09G8CqVMkSjIjMAPBdAGerakOq5XEhCwCMFZFRIpIL4FIA/0uxTK5FuBN7EMBKVb0r1fIAgIiUO9lzIlIA4GS44Penqrep6lD/WnUpgNdTrQyADFIILufXIrJMRD4CXVopT8fz8ycAJQBe8afE/iXVAgGAiJwnIhsBHAXgBRF5KRVy+APuXwPwEhgg/beqLk+FLMGIyOMA3gFwgIhsFJHrUi2Tn+kArgRwov96WuLfAaeSQQDm+H97C8AYgitSPN2Ita4wDMMwAJiFYBiGYfgxhWAYhmEAMIVgGIZh+DGFYBiGYQAwhWAYhmH4MYVgGIZhADCFYGQIIjJSRBr9PWucxzq1rhaRAn9+fEt326P7z/WGvysqROTr/tbPcbUAEZEyEflqd2SJ4TU6tcwWkVwReVNE0q5jgdEzmEIwMolP/S3Gw7auVtVG/zGJaEFxLYBn/L1yAOCrAM5Q1SviPE+Z/2/jQkisv+GH0KFltr+D62sALon3tY3MxBSCkTb4h6+c4v//z0XkngiHJ6N19RUAnAaBfwGwH4D/icjNIvJ5/2CWJSLy1yAr4r8issg/rOV6/3l+DWC0/9jf+q2d4J38t0TkJ/7/j/RbIfcBWAxgWLjXCiZCy+z/+t+HYZhCMNKKH4MdY68AWyvfHOHYHm1d7e9ttJ+qrgMAVb0BtDpOADAb3HVP91sjXgQW3WtVdQqAqQC+LiL9AdwKv3Wjqt+O4eUPAPCIqk4CUBjhtWJhGYDD4jjeyGDMd2ikDar6pr+B2i0AjndcNSJyO9hULZiebl09AMCeMM+dBGAKgAX+zrsFYC9+gErgPP//hwEYCyDeBoxVqvpuDK8VFVX1+uMpJf6hNkYvxhSCkTb4W5wPArDTWbxEZCBCX8dxt64WkRsBfMl/9wwA5wXfV9Xgv28E2xaHPBWAh1W13UhSf9/7kwEcpaoN/lbMoc7RhvbWe8dj6qO9VpzkAWjqxt8bGYK5jIy0QEQGgQN8zgFQLyKn+Z+aBGBJiD+Ju3W1qt7rd9tMVNXNHe93OLYaQJaIhFrQXwNwoYhU+GXvJyIjAPQBUO1XBuPAucMAUAt2nXXYBqBCRPqLSB44ryIc4V4rJvwuqx2q2hrr3xiZiykEw/WISCGAZ8AB7isB3A7gJ/6nJyKEQkhS6+qXARwd4rVXAPgBgJf9bZdfAS2b2QCy/Y/dDs6xhqruAvCWv0X6b/2L88/A8ZPPI0L//giv1Y4ILbNPADCrK2/eyDys/bWR1ojIg6BbZziA51V1fIx/tw7AVFXd2Y3XngTgFlW9sqvnSDUi8gyA21R1daplMVKPWQhGWqOq16mqD8yu6RNcmBYKpzANQA66OdxcVT8Ah690SvNMB/yutP+aMjAczEIwDMMwAJiFYBiGYfgxhWAYhmEAMIVgGIZh+DGFYBiGYQAwhWAYhmH4MYVgGIZhADCFYBiGYfgxhWAYhmEAAP4f9Oj7C7wVhqoAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x,train_labels,model.layers[0].weights[0],model.layers[0].weights[1])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "dvAiaj_JndyP"
      },
      "source": [
        "## Plotting the training graphs\n",
        "\n",
        "`fit` function returns `history` object as a result, which can be used to observe loss and metrics on each epoch. In the example below, we will re-start the training with small learning rate, and will observe how the loss and accuracy behave.\n",
        "\n",
        "> **Note** that we are using slightly different syntax for defining `Sequential` model. Instead of `add`-ing layers one by one, we can also specify the list of layers right when creating the model in the first place - this is a bit shorter syntax, and you may prefer to use it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "70/70 [==============================] - 1s 5ms/step - loss: 0.6600 - acc: 0.6143 - val_loss: 0.6351 - val_acc: 0.8000\n",
            "Epoch 2/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.6384 - acc: 0.7143 - val_loss: 0.6187 - val_acc: 0.8333\n",
            "Epoch 3/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.6188 - acc: 0.7571 - val_loss: 0.6001 - val_acc: 0.8667\n",
            "Epoch 4/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.6022 - acc: 0.7714 - val_loss: 0.5837 - val_acc: 0.9000\n",
            "Epoch 5/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5860 - acc: 0.8571 - val_loss: 0.5673 - val_acc: 0.9000\n",
            "Epoch 6/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5702 - acc: 0.8571 - val_loss: 0.5597 - val_acc: 0.8667\n",
            "Epoch 7/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5568 - acc: 0.8286 - val_loss: 0.5458 - val_acc: 0.9000\n",
            "Epoch 8/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5430 - acc: 0.8714 - val_loss: 0.5325 - val_acc: 0.9000\n",
            "Epoch 9/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5308 - acc: 0.8714 - val_loss: 0.5234 - val_acc: 0.9000\n",
            "Epoch 10/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.5175 - acc: 0.9143 - val_loss: 0.5170 - val_acc: 0.8667\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(1,input_shape=(2,),activation='sigmoid')])\n",
        "model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.05),loss='binary_crossentropy',metrics=['acc'])\n",
        "hist = model.fit(x=train_x_norm,y=train_labels,validation_data=(test_x_norm,test_labels),epochs=10,batch_size=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x2c41a32fe80>]"
            ]
          },
          "execution_count": 50,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsWklEQVR4nO3dd3xUVfrH8c9DCpDQISAl9AREkBZAmjRBrLhWbLgWEBUVe9t1LbuuXVFwEUFF+SlrQ0CRIiJFQAm9JyEECKEkdALpz++PG9cYAkxgkjuZed6vV14wM/fOPBnIN2fOueccUVWMMcb4r3JuF2CMMaZkWdAbY4yfs6A3xhg/Z0FvjDF+zoLeGGP8XLDbBRSlVq1a2rhxY7fLMMaYMmP58uVpqhpR1GM+GfSNGzcmNjbW7TKMMabMEJFtJ3vMum6MMcbPWdAbY4yfs6A3xhg/Z0FvjDF+zoLeGGP8nAW9Mcb4OQt6Y4zxcxb0xhjjA5ZvO8D787eUyHNb0BtjjMumrEzmxnFL+ey37aRn5nj9+X1yZqwxxgSCvDzljTmbGTNvC12a1GDsLR0JL+/9WLagN8YYFxzLyuGh/65i1vo9DO4UyQuDWhMaXDKdLBb0xhhTylIOHueuibFs2n2Yv1/eiju6N0ZESuz1LOiNMaYUrdx+gKGfLCcjO5cJt3WiT8vaJf6aFvTGGFNKpq7ayWNfraFOlfJ8NrQL0XUql8rretQhJCIDRWSziCSIyJNFPF5dRKaIyBoR+U1EWnt6rjHG+Lu8POXN2Zt5cPIq2jWoxtT7epRayIMHQS8iQcAY4BKgFXCjiLQqdNjTwCpVPR8YAowqxrnGGOO3jmXlcN9nK3jnpwSu69iASXd1oUZ4aKnW4EnXTWcgQVUTAURkMjAI2FDgmFbAvwFUdZOINBaROkBTD841/kwV0tNA89yuBCpWh+DS/QHzWVnpkHnU7Sp8R2g4lK/k9afddeg4Qz+JZX3KYZ659Fzu6tmkRAddT8aToK8P7ChwOxnoUuiY1cDVwCIR6Qw0Ahp4eK7xV/u2wLQHYNsitytxVDoHLn8TWl7mdiXuyc2BJaPh539DTobb1fiOoFC48HHoMRKCQrzylKt3HGToJ7GkZ+YwfkgM/c6t45XnPROeBH1Rv3600O2XgVEisgpYC6wEcjw813kRkWHAMICGDRt6UJbxWbk5sHQMzHsJgspD379BxRru1qR5sHwiTL4JzrsaLnkVKhW5vab/2r0Wpt4Hu1ZDy8uhWV+3K/IdSQth3j9hw7cwaDTUa39WTzd9dQqPfrmaiMrl+eTObrQ8p4p36jxDngR9MhBZ4HYDIKXgAap6GLgdQJzPJVvzv8JOd26B5xgHjAOIiYkp8peBKQN2r4NpIyBlpRMml74OVeq6XZWj41/hl7dh/quQOA8GvgLnXw8ufJQuVTmZsOA1WPSW03113URoNcj/v+/i6HQntLkevn8YPugL3e6H3k9BSMViPY2q8vaP8YyaG09Mo+qMvbUjtSqVL6GiPSeqp85UEQkG4oB+wE5gGXCTqq4vcEw14JiqZonIUKCnqg7x5NyixMTEqG0OXsbkZMKC12HRm06YXPoatLrKN8MkdTNMHQHJv0HUALj8LajawO2qSsaO35zvNW0ztL0RLn4Jwlz+dOXLjh+EOc/CiolQoxlc+Q407uHZqVm5PPrVar5fs4trOjTgpatbUz44qGTrLUBElqtqTJGPnS7o85/gUuBtIAj4UFX/JSLDAVR1rIh0BT4BcnEGWu9U1QMnO/d0r2dBX8aUxTDJy4XfPoC5z4MEQf/noOMdUM5P1vnLPAo//RN+Hev8Erv8bYi6yO2qyo7E+TD9ATiQBDF3wEXPQ4WTd7/sOZzB0E9iWbvzEE8ObMmwC5uW+qDrWQd9abOgLyOy0mHui06YVKkPV7wNUf3drqp4DiTB9Ach8Wdo2A2ufBdqNXe7qrOz5Sfnezq4HToNhYv+AeVL75ptv5GV7owzLX0PKtd1fllGDzjhsLXJh7jrk2Ucychh1OD29G/lzqCrBb3xvi3znBaPP4SJKqz6P5j1tNMF1fsp6DoCgsrYxPHjB2DW32DVJKjZ3Pml1aib21WVfcmxzifW1I1OP/7AlyG8JgAz1u7i4S9WUTO8PONvi+Hcuu4NulrQG+85fhBmPwMrJzl9mING+0+YHNkN3z8Cm76Dum1h0Bg4p43bVXlm43Sn9vQ06P4g9HoCQiq4XZX/yMmChW84XxWqope8yrt72vDmj/F0aFiN92+NIaKyu4OuFvTGOzZ+lx8mqdD9gfwwKd5VCWXChqnw/aNwfD/0eAgufAyC3b9yokhH9sAPjzk1n9MGrhwN9dq5XZX/2rOevG9HUG7XCubkdmRxi6d44oa+VAgpvUHXkzlV0PvJyJMpUUf3whe3wX9vhvAIGDoXLnrOP0MenEsP7/vV+Zi+4DUY29MZcPYlqrDqcxjTGTbPhH7PwtB5FvIlbG/FZlyT/Tz/yrmZ3iHreHbHHVRY86nz7+HDLOjNyanC6sn5YTID+v4dhs0768kkZUJYDfjLf+CWryH7GEwYAD886RvLBhzcDpOugW+HQ0QLGL4Iej7itRmdpmjrdh5i0Jhf2LTnGDE3PkvIfUuQum2dge+JV8D+RLdLPCnrujFFO7gDvhsJCT9CZBenSyAi2u2q3JF5BOa+AL+Ng2oN4YpR7swqzcuDZePhx+ec2xc9B53u8p9LQn3YzHW7eOi/q6keFsIHt8VwXr2qzgN5ebDyE5j9d8jNdmaBX3APlCv9rhzrozeey8uD2AlOmKg6V9N0GmphArBtCUy7H/bFQ7tb4OJ/OpPDSkNavHPlx46lzi+Zy9+G6o1K57UDmKoyZl4Cr8+Oo11kNcYN6UjtykUMch9Oge8ehrgfoH5Hp2FUp3QX6rWgN55Ji3eCbPsSaNrHablamPxZdgbMfwV+GQXhteCyN+DcK0ru9XKzYfE78PMrzpjIwH87k9J8ccaxn8nIzuXJr9fw7aoUBrWrxyvXnH/qQVdVWPc1/PA4ZByGCx+FHg+X2oqpFvTm1HJz8sPkZeeSvIv/De1usjA5lV2rnQXCdq91Bm8veQ0qe3mizK7VTit+95qSew1TpL1HMrj70+Ws3H6Qxy5uwb29m3k+0zV9H8x8AtZ+CbVbOa37Bh1LtmAs6M2p7FrjLEK2a7XTMr30DQsTT53Q2n4Z2g4++1+QBT81hNV0PjW0utI7NZvT2pBymLsmLuPAsWzeuqEtA1uf4aJ8m2fCdw/B0d1wwb3Q5xkIDfNusQVY0JsTZWfAgldh0dv5YfK602o0xZca53R57VgKzfo5S0FUO8OltrcvdVrxbowDGGat381D/11FlQohjL8thtb1q57dE2Yccsa7Yj+E6o2d2cpNLvRGqSewoDd/VjBM2t4EF//L9xch83UFr4gRca6IibnT80Hs/13Z8wFUjXR+WTTvV4IFm4JUlbHzE3l11ibOb1CND27tSO0qXpxZnLTIaQzsT3SWy+7/AlQ4y18ihVjQG0fm0T8uE6waCVe8Bc1tRUOvOrgdpo+ELXMh8gJniYhaUac+J+FH55xDydDlbme+Qglsa+drVJWVOw5yLDPX7VL4ZmUy36zYyRVt6/HatacZdD1TWcecnb2WjIZKdZzlsVtc4rWnt6A3kDA3P0x2QOdhzkzKAAgTV6jC6s9h5lOQfRx6PwHdHjhxQtOx/c5Caqs/h1rRzsf6hhe4U3Mpy8jO5Ymv1zB1VZH7ELni4f7R3N+3eckvL7xzhfOJeu96aH2Ns9tZeK2zfloL+kB2bD/M/puzOmPNKKeFGSBh4ro/rUNzvvPe123rPLb+W5jxqLPiZPeRzno6AbII2d4jGQz7ZDmrdhzkwX5R9Ig6+5A7WzXCQ2kWUYoNn5ysP3Y7K1/ZCfs2157VQL4FfaD6fXGuY/ucTY8vfDxgwsSnbJjmhHp6GnQb4fTTbpzuhP6Vo6Hu+W5XWGrWpxxi6MTYs7+ixV/s3ej03Scvg6iLnc3rz3C3Mwv6QHNktxMsG6fntyTHBFSY+KSCa8UHlYc+T0HX+8vemvdnYdb63YycvIpqYSF8MMQLV7T4i7xcZ9xs7gsQEgYj10BoeLGf5lRBHzj/ywKBKqz6DGY95Vw+2e/ZovuGTemrWB2uGuNccRFeE2o0dbuiUqOqvPfzFl6btZm2kSVwRUtZVy7IWR8neiDsXH5GIX86FvT+4sC2/C3x5kHDrvlb4p3mag9T+iI7uV1BqcrIzuWpb9YyZWUJX9HiD2o0cb5KgAV9WZeXB8s+gB+fdwZyLn29eNdvG1NCUo9kcvensazYfpBH+kczojSuaDFFsqAvy1I358/I/NW5Hv7yt858RqYxXrQh5TBDP4llX3om793cgUvbBPigq8ss6Mui3Ow/Ls0KDYe/vA/n32CLkBmfMGfDHh6cvJLKFYL58u5utGlgg65us6Ava1JWOZMt9qyFVlfBpa9BpdpuV+Xzxi3Ywme/bne7DAA6N6nBw/1bcE5V/xqQVFXeX5DIKzM30aZ+VT4YEkMdG3T1CRb0ZUX2cWcZ4cXvOrPobphUsuug+5H1KYd4+YdNtGlQjcY1S271QE9k5eTx7coUpq1OYVjPptzdqxnh5cv+j2FmTi5Pf7OOr1ckc9n5dXn92rZUDLVBV1/h0f8wERkIjAKCgPGq+nKhx6sCk4CG+c/5uqp+lP9YEnAEyAVyTnadpzmFbYvzdzZKgPa3wABb0dBTeXnKs1PXUz0slE9u70zVMPcvNd2x/xivztrMOz8l8NlvO3hkQDTXx0QSVK5sdr2lHc1k+KfLid12gJEXRfFgvygbdPUxp700Q0SCgDHAJUAr4EYRKbxH1n3ABlVtC/QG3hCRgtuq9FHVdhbyxZRxGL5/BD66BHKz4NZvnclPFvIe+2blTpZvO8ATl7T0iZAHiKwRxrs3tmfKvd1oXDOMp75Zy6WjFvLz5r1ul1Zsm3YfZtDoX1i78xCjb2rPyIuiLeR9kCfX4HUGElQ1UVWzgMlA4YXLFagszr9wJWA/kOPVSgNN/Bx4ryssm+BsWnDvUmjWx+2qypRDx7N5+YeNtG9YjWs7nNm08pLUvmF1vhzelf/c3IGMnFz++tEybp3wKxt3HXa7NI/M3biHa95bTHZuHl/c3ZXLz6/ndknmJDzpuqkP7ChwOxnoUuiY0cA0IAWoDNygqnn5jykwW0QUeF9Vx51dyX7u2H5n1cM1k6FWC7hzNkR2druqMumtOXHsS8/i49s7U85Hu0VEhEva1KXfuXX4dOk23pkbz6XvLOT6jpE8PCDaJwczVZUPFiby7x82cV69Kowf0snvBpb9jSdBX9RPSOEFci4GVgF9gWbAHBFZqKqHge6qmiIitfPv36SqC054EZFhwDCAhg0D8FpwVVg/BWY8BhkHnQXILnwUgsu7XVmZtCHlMJ8sSeLmLg3LxJoqocHluLNHE67t0IB3f4pn4pIkpq1O4e5eTRl2YVPCQn1jwDYrJ49npqzly+XJXNrmHN64rp0NupYBnnTdJAORBW43wGm5F3Q78I06EoCtQEsAVU3J/3MvMAWnK+gEqjpOVWNUNSYiIqJ430VZd3gX/PcW+Op2Z+W6YT9D32cs5M+QqvKPaeuoFhbKowNauF1OsVQNC+Fvl7fix4d70bdlbd7+MZ7er/3MF8t2kJvn7gKE+45mcsv4X/lyeTIP9Iti9I0dLOTLCE+CfhkQJSJN8gdYB+N00xS0HegHICJ1gBZAooiEi0jl/PvDgQHAOm8VX+apwopPYEwXZ5eh/i/AXXPhnDZuV1amTVm5k2VJB3hiYAuqhYWe/gQf1KhmOGNu7sDX93SlfvWKPP71Gi57ZyEL41NdqSduzxGueu8XViUfZNTgdjzcP9pnu8PMiU77eVBVc0RkBDAL5/LKD1V1vYgMz398LPAi8LGIrMXp6nlCVdNEpCkwJX8UPhj4TFVnltD3Urbs3wrTH4CtC6BRd2cRsprN3K6qzDuckc1LMzbRLrIa13WMPP0JPq5joxp8c083ZqzdzcszN3LrhN/oFR3B05eeS4tzKpdKDfM27eX+z1dSMTSI/w67gPYN7aqvssbWoy9tebnw6/vw04sgQdD/eeh4uy1C5iXPT1/Px4uTmHZfD7+bep+Zk8unS5wB26OZOdzQKZKH+kdTu3LJDISqKhMWbeWlGRtpeU4Vxt8WQ71qFUvktczZs/XofcXejc7yBTtjIWqAswjZGe4mY060cddhPlmyjZs6N/S7kAcoHxzEXT2bck2HBrz7UwKfLk1i6qoUhvdqxtCeTb3aX56Vk8ezU9cxedkOBp53Dm/e0NZnBoRN8VmLvjScsD/kK9DmOluEzItUlevfX0LC3qPMe7R3me2bL46ktHRembmJH9btpk6V8jw6oAVXd2hw1jNsD6RnMXzScn7dup8RfZpbf3wZYS16N+1cDlPv/2PH94GvQKUAu6qoFHy7yhmAffnqNgER8gCNa4Xzn1s6sixpP//8fiOPfbWGj35J4pnLzqV78zPbcDth7xHunBjLrkMZvH1DO65qX9/LVRs3WIu+pGQdg59fgiVjoFIduOxNaHmp21X5pSMZ2fR9Yz71qlVkyj3dArL1qapMX7OLV37YxM6Dx+nbsjZPXdKSqDqeD9j+vHkv93+2kvIhQYwb0pEONuhapliLvrQlLXIWIdufCB1ucy6brFjN7ar81ts/xpN2NJMJt8UEZMiDM8P2yrb1GNCqDhMXJzF6XgIDRy1kcKdIRl4UTUTlk8/JUFU+XpzEi99toEX+oGt9G3T1Kxb03pRxCOb8A5Z/BNUbw5Bp0LSX21X5tU27D/Px4iQGd2rI+Q2quV2O6yqEBHF3r2ZcFxPJO3PjmbR0G1NXpXBP72bc2aPJCfu1Zufm8ezU9Xz+23YGtKrDWze084tlk82fWdeNt8TNgukj4ehuZxGyPk+XyG7u5g+qyg3jlhK35wjzHulN9fDA6JsvjsTUo7z8wyZmb9hD3aoVeOziFlzVrj7lygkHj2Vxz6QVLEncx729m/HogBYB+4nIH1jXTUlKT4OZT8LaLyHiXLjhU2hgqzGXhmmrU/ht635e+ksbC/mTaBpRiXFDYliauI+XZmzk4S9W8+EvWxnasylvzYkj5WAGb17flqt9cHVP4z3Woj9TqrDua/jhcWfd+J6POF/BFjil4fcB2LpVKzDl3u5ldtOO0pSXp0xfk8KrMzez8+BxaoaHMm5IRzo2quF2acYLrEXvbYd2wvcPQ9xMqNcBBo2GOue5XVVAGZU/ADt+SIyFvIfKlRMGtavPxeedw/TVKXRrXssGXQOEBX1x5OXBiokw51nIzYYB/4IL7oFytoJfadq8+wgfLU5icKdI2kZWc7ucMqdCSBDXxZT9dYCM5yzoPbVvC0x/EJIWQuOecOU7UKOp21UFHFXl2anrqFwhmMcubul2OcaUCRb0p5OXC0vfg5/+BUEhcMUo59p4W77AFdNWp/Dr1v386y+tqWEDsMZ4xIL+VPZsgKn3QcoKiL4ELn8Tqti+mG45mpnDSzM20qZ+VQZ3CsBdyIw5Qxb0RcnJgoVvOF8VqsA1E5x1aqwV76p35saz53AmY2/paAOwxhSDBX1hybHOUsKpG6HN9TDwZQiv6XZVAS9+zxE+XLSVG2IibeMLY4rJgv53WelOP/zS95zumZu+gOiL3a7K8PsA7HrCywfz+MCytQesMb7Agh4gcb6zrd+BJIi5Ay563umyMT7huzW7WJK4jxevak3NSrZhujHFFdhBf/wgzPm7s0F3jabw1++hcQ+3qzIFHM3M4Z/fb6B1/Src1NkGYI05E4Eb9JtmOLNbj+6Bbg9A76cgNMztqkwh7+YPwP7HBmCNOWOBF/RHU531adZ/A7XPg8GfQf0ObldlipCw9wgTFm3l+pgGtgmGMWchcIJe1Vlh8ocnIPMI9HkGuo+0Rch8lKryj2nrCQsN4omBNgPWmLMRGEF/KBm+ewjiZ0ODTnDlaKht4eHLvl+7i18S9vHCoPNsANaYs+TfQZ+XB8s/hDnPgeY618R3HmaLkPm49Mwc/vndRs6rV4WbuzRyuxxjyrxynhwkIgNFZLOIJIjIk0U8XlVEpovIahFZLyK3e3puidm3BSZeDt8/4vTB37PYVposI979KYHdhzN4YVBrG4A1xgtO26IXkSBgDNAfSAaWicg0Vd1Q4LD7gA2qeoWIRACbReT/gFwPzvWu3BxYMhp+/jcElXe6adrfYssXlBEJe48yfmEi13ZsQMdGNgBrjDd40nXTGUhQ1UQAEZkMDAIKhrUClUVEgErAfiAH6OLBud6ze62zfMGuVdDiMrjsDahSt0ReynifqvLctPVUDA3iyUtsDMUYb/Ek6OsDOwrcTsYJ8IJGA9OAFKAycIOq5omIJ+d6x/EDMOFi51r46z6GVldZK76M+WHdbhYlpPH8ledRywZgjfEaT4K+qLQsvNHsxcAqoC/QDJgjIgs9PNd5EZFhwDCAhg3PYAZkxepwzXhoeAGE2R6YZU16Zg4vfreBc+tW4eYuNgPWGG/yZDA2GSi471gDnJZ7QbcD36gjAdgKtPTwXABUdZyqxqhqTEREhKf1/1nLSy3ky6jR8xLYdSiDFwedR3CQR9cIGGM85MlP1DIgSkSaiEgoMBinm6ag7UA/ABGpA7QAEj081wS4LanOAOw1HRoQ09h+URvjbaftulHVHBEZAcwCgoAPVXW9iAzPf3ws8CLwsYisxemueUJV0wCKOrdkvhVTFv0+AFshxAZgjSkpHk2YUtUZwIxC940t8PcUYICn5xrzu5nrdrMwPo1/XNGKiMo2AGtMSbDOUOOaY1nOAGzLcypz6wU2A9aYkmJBb1wzZl4CKYcyePGq1jYAa0wJsp8u44rE1KOMW5DI1e3r08kGYI0pURb0ptSpKs9N30CF4CCevNQGYI0paRb0ptTNWr+HBXGpPNQ/mtqVK7hdjjF+z4LelKrjWbn/G4Ad0tUGYI0pDf69Hr3xOWPmJbDz4HG+uLurDcAaU0rsJ82Umq1p6YxbkMhf2tencxMbgDWmtFjQm1Khqjw/fT2hweV4ymbAGlOqLOhNqZi9YQ8/b05l5EVR1K5iA7DGlCYLelPijmfl8sL0DbSoU5nbujV2uxxjAo4NxpoS95+fnQHY/w67gBAbgDWm1FnQmxKTl6csS9rP2PmJDGpXjy5Na7pdkjEByYLeeNXeIxksjEtjQXwqi+LT2JeeRbWwEJ6+9Fy3SzMmYFnQm7OSlZNH7Lb9LIhLY0FcKht2HQagZngoPaNq0atFBL2ia1MjPNTlSo0JXBb0ptiS0tJZEJ/KgrhUFm/Zx7GsXILLCR0bVeexi1vQKzqCVnWrUK6cbc5ujC+woDendTQzhyVb9rEgLpX5cals338MgMgaFbm6Q30ujIqga7OaVK4Q4nKlxpiiWNCbE+TlKRt2HWZ+nNNqX7H9ANm5SlhoEF2b1uTOHk24MDqCxjXDELFWuzG+zoLeAJB2NJNF8WnMj0tlYXwqaUezADi3bhXu6NGEXlERdGxcnfLBQS5XaowpLgv6AJWdm8eKbQecVnt8Kut2OoOoNcJD6dG8Fr2iI+gZVctmsRrjByzoA8j2fceYnz+IumTLPo5m5hBUTujQsBqPDojmwugIWteraoOoxvgZC3o/lp6Zw9JEZxB1QXwaW9PSAahfrSJXtqvHhVERdGtekyo2iGqMX7Og91OfLknixe83kpWTR4WQcnRtWpMhXRtxYXQETWuF2yCqMQHEgt7PZOfm8cL0DXy6dBt9WkRwZ4+mxDSuToUQG0Q1JlB5FPQiMhAYBQQB41X15UKPPwbcXOA5zwUiVHW/iCQBR4BcIEdVY7xUuynk0LFs7v1sOb8k7OPuXk15/OKWBFl/uzEB77RBLyJBwBigP5AMLBORaaq64fdjVPU14LX8468AHlLV/QWepo+qpnm1cvMnialHuWtiLDsOHOO1a8/nuphIt0syxvgIT1r0nYEEVU0EEJHJwCBgw0mOvxH43DvlGU/8kpDGPZOWExxUjs+GXkCnxrZNnzHmD54sDl4f2FHgdnL+fScQkTBgIPB1gbsVmC0iy0Vk2MleRESGiUisiMSmpqZ6UJYBmLR0G0M+/I26VSsy9b7uFvLGmBN40qIvqpNXT3LsFcAvhbptuqtqiojUBuaIyCZVXXDCE6qOA8YBxMTEnOz5Tb6c3Dxe/G4DE5dso2/L2owa3M7WmjHGFMmToE8GCnb4NgBSTnLsYAp126hqSv6fe0VkCk5X0AlBbzx36Hg2Iz5bwcL4NIb2bMKTl5xrg67GmJPyJOiXAVEi0gTYiRPmNxU+SESqAr2AWwrcFw6UU9Uj+X8fALzgjcIDVVJaOndMXMaO/cd49Zrzub6TDboaY07ttEGvqjkiMgKYhXN55Yequl5Ehuc/Pjb/0L8As1U1vcDpdYAp+ZNzgoHPVHWmN7+BQLJ4Sxr3TFpBOYFJd3axrfmMMR4RVd/rDo+JidHY2Fi3y/Apn/26nWenrqNJrXAm3NaJhjXD3C7JGONDRGT5yeYp2cxYH5eTm8e/Zmzko1+S6N0igndubG9r0xhjisWC3ocdzshmxGcrWRCXyh3dm/DMZTboaowpPgt6H7VtXzp3TowlKS2df1/dhhs7N3S7JGNMGWVB74OWJu5j+KTlAHx6Zxe6NrNBV2PMmbOg9zGTf9vO375dR6OaYUy4rRONa4W7XZIxpoyzoPcRuXnKSzM2MmHRVi6MjuDdG9tTtaINuhpjzp4FvQ84kpHNA5+vZN7mVP7arTF/u+xcgoM8WYbIGGNOz4LeZdv3HePOictITEvnn1e15pYLGrldkjHGz1jQu+i3rfsZPmk5uXnKp3d0plvzWm6XZIzxQxb0Lvli2Q6e+XYtkdXDmPDXTjSxQVdjTAmxoC9luXnKKzM3MW5BIj2a12LMTR2oGmaDrsaYkmNBX4qOZGQzcvIq5m7ay5Cujfj75a0IsUFXY0wJs6AvJTv2H+OuibEkpB7lhUHnMaRrY7dLMsYECAv6UrAsaT/DP11Odm4eH9/eiZ5REW6XZIwJIBb0Jeyr5ck8/c1a6levyPjbYmgWUcntkowxAcaCvoTk5imvztrE+/MT6dasJu/d3IFqYaFul2WMCUAW9CXgaGYOIyev4seNe7ipS0Oev/I8G3Q1xrjGgt7L0jNzuG7sEjbvPsxzV7Titm6Nyd9K0RhjXGFB72UfL05i467DjB8Sw0Wt6rhdjjHGYP0JXnToeDbvz99Cv5a1LeSNMT7Dgt6LJizayuGMHB7qH+12KcYY8z8W9F5yID2LDxdt5ZLW59C6flW3yzHGmP+xoPeS9xckkp5lrXljjO+xoPeC1COZTFycxJVt6xFdp7Lb5RhjzJ94FPQiMlBENotIgog8WcTjj4nIqvyvdSKSKyI1PDnXH4ydv4XMnFwe7BfldinGGHOC0wa9iAQBY4BLgFbAjSLSquAxqvqaqrZT1XbAU8B8Vd3vybll3e5DGXy6dBvXdGhAU1vewBjjgzxp0XcGElQ1UVWzgMnAoFMcfyPw+RmeW+aMmZdAXp7ygLXmjTE+ypOgrw/sKHA7Of++E4hIGDAQ+PoMzh0mIrEiEpuamupBWe5LPnCMycu2c32nSCJrhLldjjHGFMmToC9q/r6e5NgrgF9UdX9xz1XVcaoao6oxERFlYxnfd+cmICLc37e526UYY8xJeRL0yUBkgdsNgJSTHDuYP7ptintumZKUls5XK5K5qXND6lat6HY5xhhzUp4E/TIgSkSaiEgoTphPK3yQiFQFegFTi3tuWTRqbjwhQcK9fZq5XYoxxpzSaRc1U9UcERkBzAKCgA9Vdb2IDM9/fGz+oX8BZqtq+unO9fY3Udri9xzh21U7GdazKbUrV3C7HGOMOSWPVq9U1RnAjEL3jS10+2PgY0/OLeve/jGesJAg7u5lrXljjO+zmbHFtCHlMN+v3cUdPZpQI9x2jDLG+D4L+mJ668c4KlcI5q4eTd0uxRhjPGJBXwyrdxxkzoY9DOvZlKphIW6XY4wxHrGgL4Y358RRPSyE23s0cbsUY4zxmAW9h2KT9jM/LpW7ezWjUnnbgdEYU3ZY0Hvojdlx1KpUniFdG7ldijHGFIsFvQcWJ6SxJHEf9/ZuRlioteaNMWWLBf1pqCpvzInjnCoVuKlLQ7fLMcaYYrOgP435caks33aAEX2bUyEkyO1yjDGm2CzoT0FVeXNOHA2qV+T6mMjTn2CMMT7Igv4U5mzYw5rkQzzQL4rQYHurjDFlk6XXSeTlOa35JrXCubp9kXulGGNMmWBBfxIz1u1i0+4jjLwoiuAge5uMMWWXJVgRcvOUt3+MJ6p2JS4/v57b5RhjzFmxoC/CtNU7Sdh7lIf6RxNUrqjdEI0xpuywoC8kOzePt3+Mp1XdKgw87xy3yzHGmLNmQV/INyuS2bbvGA/3j6acteaNMX7Agr6AzJxc3pmbQNvIavQ7t7bb5RhjjFdY0BfwxbId7Dx4nEf6RyNirXljjH+woM+XkZ3L6HkJdGpcnZ5RtdwuxxhjvMaCPt+kpdvYcziTRwa0sNa8McavWNADx7JyGDt/C92b1+SCpjXdLscYY7zKFlcHJi7eRtrRLN7v38LtUowxxusCvkV/JCOb9xdsoU+LCDo2qu52OcYY43UeBb2IDBSRzSKSICJPnuSY3iKySkTWi8j8Avcnicja/MdivVW4t3y4KImDx7J52Frzxhg/ddquGxEJAsYA/YFkYJmITFPVDQWOqQa8BwxU1e0iUvgi9D6qmua9sr3j4LEsxi9M5OLz6tCmQVW3yzHGmBLhSYu+M5CgqomqmgVMBgYVOuYm4BtV3Q6gqnu9W2bJ+GBhIkezcniof7TbpRhjTInxJOjrAzsK3E7Ov6+gaKC6iPwsIstFZEiBxxSYnX//sJO9iIgME5FYEYlNTU31tP4ztu9oJh/9ksTl59ej5TlVSvz1jDHGLZ5cdVPUReVaxPN0BPoBFYElIrJUVeOA7qqakt+dM0dENqnqghOeUHUcMA4gJiam8PN73dj5W8jIzmXkRVEl/VLGGOMqT1r0yUDBDVMbAClFHDNTVdPz++IXAG0BVDUl/8+9wBScriBX7T2cwSdLtnFV+/o0i6jkdjnGGFOiPAn6ZUCUiDQRkVBgMDCt0DFTgZ4iEiwiYUAXYKOIhItIZQARCQcGAOu8V/6ZGTMvgdw85cF+1po3xvi/03bdqGqOiIwAZgFBwIequl5Ehuc/PlZVN4rITGANkAeMV9V1ItIUmJK/pEAw8Jmqziypb8YTOw8e5/PfdnBdTAMa1Qx3sxRjjCkVHs2MVdUZwIxC940tdPs14LVC9yWS34XjK0b/lADAiL7WmjfGBIaAmhm7fd8xvozdwY2dI6lfraLb5RhjTKkIqKAfNTeeoHLCfX2au12KMcaUmoAJ+i2pR5myMpkhXRtRu0oFt8sxxphSEzBB//aP8VQICWJ4r2Zul2KMMaUqIIJ+0+7DfLcmhdu7N6ZmpfJul2OMMaUqIIL+rTlxVAoNZmjPpm6XYowxpc7vg37dzkPMWr+HO3s2oVpYqNvlGGNMqfP7oH9zThzVwkK4o0cTt0sxxhhX+HXQL992gJ827WXYhU2pUiHE7XKMMcYVfh30b82Jo1alUP7arbHbpRhjjGv8NuiXJu5jUUIaw3s1IyzU9kA3xgQuvwx6VeXN2XHUqVKeWy5o5HY5xhjjKr8M+kUJafyWtJ8RfZpTISTI7XKMMcZVfhf0qsrrs+OoX60i13eKPP0Jxhjj5/wu6H/atJfVOw7yQL/mlA+21rwxxvhV0OflKW/MjqNRzTCu7tDA7XKMMcYn+FXQz1q/mw27DvNgvyhCgvzqWzPGmDPmN2mYm6e89WMczSLCGdSuvtvlGGOMz/CbC8yPZ+fSPrI6vVtEEFRO3C7HGGN8ht8EfaXywbxy7flul2GMMT7Hb7pujDHGFM2C3hhj/JwFvTHG+DkLemOM8XMeBb2IDBSRzSKSICJPnuSY3iKySkTWi8j84pxrjDGm5Jz2qhsRCQLGAP2BZGCZiExT1Q0FjqkGvAcMVNXtIlLb03ONMcaULE9a9J2BBFVNVNUsYDIwqNAxNwHfqOp2AFXdW4xzjTHGlCBPgr4+sKPA7eT8+wqKBqqLyM8islxEhhTjXABEZJiIxIpIbGpqqmfVG2OMOS1PJkwVNc1Ui3iejkA/oCKwRESWeniuc6fqOGAcgIikisg2D2orSi0g7QzP9Tf2XvyZvR9/Zu/HH/zhvTjpLkueBH0yUHBh9wZAShHHpKlqOpAuIguAth6eewJVjfCgriKJSKyqxpzp+f7E3os/s/fjz+z9+IO/vxeedN0sA6JEpImIhAKDgWmFjpkK9BSRYBEJA7oAGz081xhjTAk6bYteVXNEZAQwCwgCPlTV9SIyPP/xsaq6UURmAmuAPGC8qq4DKOrcEvpejDHGFEFUi+wyL7NEZFh+f3/As/fiz+z9+DN7P/7g7++F3wW9McaYP7MlEIwxxs9Z0BtjjJ/zm6C3NXX+ICKRIjJPRDbmrz30oNs1uU1EgkRkpYh853YtbhORaiLylYhsyv8/0tXtmtwkIg/l/5ysE5HPRaSC2zV5m18EfYE1dS4BWgE3ikgrd6tyVQ7wiKqeC1wA3Bfg7wfAgziX/BoYBcxU1ZY4810C9n0RkfrAA0CMqrbGuTpwsLtVeZ9fBD22ps6fqOouVV2R//cjOD/IAbtjuog0AC4Dxrtdi9tEpApwITABQFWzVPWgq0W5LxioKCLBQBgeTOosa/wl6D1eUyfQiEhjoD3wq8uluOlt4HGcOR6BrimQCnyU35U1XkTC3S7KLaq6E3gd2A7sAg6p6mx3q/I+fwl6j9fUCSQiUgn4GhipqofdrscNInI5sFdVl7tdi48IBjoA/1HV9kA6ELBjWiJSHefTfxOgHhAuIre4W5X3+UvQn9GaOv5MREJwQv7/VPUbt+txUXfgShFJwunS6ysik9wtyVXJQLKq/v4J7yuc4A9UFwFbVTVVVbOBb4BuLtfkdf4S9LamTgEiIjh9sBtV9U2363GTqj6lqg1UtTHO/4ufVNXvWmyeUtXdwA4RaZF/Vz8gkDcC2g5cICJh+T83/fDDwWlPVq/0eSdbj8flstzUHbgVWCsiq/Lve1pVZ7hXkvEh9wP/l98oSgRud7ke16jqryLyFbAC52q1leQvl+5PbAkEY4zxc/7SdWOMMeYkLOiNMcbPWdAbY4yfs6A3xhg/Z0FvjDF+zoLeGGP8nAW9Mcb4uf8HL4zLvVpYuQ8AAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.plot(hist.history['acc'])\n",
        "plt.plot(hist.history['val_acc'])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Multi-Class Classification\n",
        "\n",
        "If you need to solve a problem of multi-class classification, your network would have more that one output - corresponding to the number of classes $C$. Each output will contain the probability of a given class.\n",
        "\n",
        "> Note that you can also use a network with two outputs to perform binary classification in the same manner. That is exactly what we will demonstrate now.\n",
        "\n",
        "When you expect a network to output a set of probabilities $p_1,\\dots, p_C$, we need all of them to add up to 1. To ensure this, we use `softmax` as a final activation function on the last layer. **Softmax** takes a vector input, and makes sure that all components of that vector are transformed into probabilities.\n",
        "\n",
        "Also, since the output of the network is a $C$-dimensional vector, we need labels to have the same form. This can be achieved by using **one-hot encoding**, when the number of a class $i$ is converted to a vector of zeroes, with 1 at the $i$-th position.\n",
        "\n",
        "To compare the probability output of the neural network with expected one-hot-encoded label, we use **cross-entropy loss** function. It takes two probability distributions, and outputs a value of how different they are.\n",
        "\n",
        "So, to summarize what we need to do for multi-class classification with $C$ classes:\n",
        "* The network should have $C$ neurons in the last layer\n",
        "* Last activation function should be **softmax**\n",
        "* Loss should be **cross-entropy loss**\n",
        "* Labels should be converted to **one-hot encoding** (this can be done using `numpy`, or using Keras utils `to_categorical`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "70/70 [==============================] - 1s 6ms/step - loss: 0.6524 - acc: 0.7000 - val_loss: 0.5936 - val_acc: 0.9000\n",
            "Epoch 2/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.5715 - acc: 0.8286 - val_loss: 0.5255 - val_acc: 0.8333\n",
            "Epoch 3/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.4820 - acc: 0.8714 - val_loss: 0.4213 - val_acc: 0.9000\n",
            "Epoch 4/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.4426 - acc: 0.9000 - val_loss: 0.3694 - val_acc: 0.9333\n",
            "Epoch 5/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.3602 - acc: 0.9000 - val_loss: 0.3454 - val_acc: 0.9000\n",
            "Epoch 6/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.3209 - acc: 0.8857 - val_loss: 0.2862 - val_acc: 0.9333\n",
            "Epoch 7/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2905 - acc: 0.9286 - val_loss: 0.2787 - val_acc: 0.9000\n",
            "Epoch 8/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2698 - acc: 0.9000 - val_loss: 0.2381 - val_acc: 0.9333\n",
            "Epoch 9/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2639 - acc: 0.8857 - val_loss: 0.2217 - val_acc: 0.9667\n",
            "Epoch 10/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2592 - acc: 0.9286 - val_loss: 0.2391 - val_acc: 0.9000\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(5,input_shape=(2,),activation='relu'),\n",
        "    keras.layers.Dense(2,activation='softmax')\n",
        "])\n",
        "model.compile(keras.optimizers.Adam(0.01),'categorical_crossentropy',['acc'])\n",
        "\n",
        "# Two ways to convert to one-hot encoding\n",
        "train_labels_onehot = keras.utils.to_categorical(train_labels)\n",
        "test_labels_onehot = np.eye(2)[test_labels]\n",
        "\n",
        "hist = model.fit(x=train_x_norm,y=train_labels_onehot,\n",
        "                 validation_data=[test_x_norm,test_labels_onehot],batch_size=1,epochs=10)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Sparse Categorical Cross-Entropy\n",
        "\n",
        "Often labels in multi-class classification are represented by class numbers. Keras also supports another kind of loss function called **sparse categorical crossentropy**, which expects class number to be integers, and not one-hot vectors. Using this kind of loss function, we can simplify our training code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "70/70 [==============================] - 1s 6ms/step - loss: 0.2353 - acc: 0.9143 - val_loss: 0.2190 - val_acc: 0.9000\n",
            "Epoch 2/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2243 - acc: 0.9286 - val_loss: 0.1886 - val_acc: 0.9333\n",
            "Epoch 3/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2366 - acc: 0.9143 - val_loss: 0.2262 - val_acc: 0.9000\n",
            "Epoch 4/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2259 - acc: 0.9429 - val_loss: 0.2124 - val_acc: 0.9000\n",
            "Epoch 5/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2061 - acc: 0.9429 - val_loss: 0.2691 - val_acc: 0.9000\n",
            "Epoch 6/10\n",
            "70/70 [==============================] - 0s 2ms/step - loss: 0.2200 - acc: 0.9286 - val_loss: 0.2344 - val_acc: 0.9000\n",
            "Epoch 7/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2133 - acc: 0.9286 - val_loss: 0.1973 - val_acc: 0.9000\n",
            "Epoch 8/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2062 - acc: 0.9429 - val_loss: 0.1893 - val_acc: 0.9000\n",
            "Epoch 9/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2060 - acc: 0.9571 - val_loss: 0.2719 - val_acc: 0.9000\n",
            "Epoch 10/10\n",
            "70/70 [==============================] - 0s 3ms/step - loss: 0.2021 - acc: 0.9571 - val_loss: 0.2293 - val_acc: 0.9000\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x2c42267de80>"
            ]
          },
          "execution_count": 55,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.compile(keras.optimizers.Adam(0.01),'sparse_categorical_crossentropy',['acc'])\n",
        "model.fit(x=train_x_norm,y=train_labels,validation_data=[test_x_norm,test_labels],batch_size=1,epochs=10)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Multi-Label Classification\n",
        "\n",
        "Sometime we have cases when our objects can belong to two classes at once. As an example, suppose we want to develop a classifier for cats and dogs on the picture, but we also want to allow cases when both cats and dogs are present.\n",
        "\n",
        "With multi-label classification, instead of one-hot encoded vector, we will have a vector that has 1 in position corresponding to all classes relevant to the input sample. Thus, output of the network should not have normalized probabilities for all classes, but rather for each class individually - which corresponds to using **sigmoid** activation function. Cross-entropy loss can still be used as a loss function.\n",
        "\n",
        "> **Note** that this is very similar to using **different neural networks** to do binary classification for each particular class - only the initial part of the network (up to final classification layer) is shared for all classes."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "BmHNhUU8bqEX"
      },
      "source": [
        "## Summary of Classification Loss Functions\n",
        "\n",
        "We have seen that binary, multi-class and multi-label classification differ by the type of loss function and activation function on the last layer of the network. It may all be a little bit confusing if you are just starting to learn, but here are a few rules to keep in mind:\n",
        "* If the network has one output (**binary classification**), we use **sigmoid** activation function, for **multiclass classification** - **softmax**\n",
        "* If the output class is represented as one-hot-encoding, the loss function will be **cross entropy loss** (categorical cross-entropy), if the output contains class number - **sparse categorical cross-entropy**.  For **binary classification** - use **binary cross-entropy** (same as **log loss**)\n",
        "* **Multi-label classification** is when we can have an object belonging to several classes at the same time. In this case, we need to encode labels using one-hot encoding, and use **sigmoid** as activation function, so that each class probability is between 0 and 1.\n",
        "\n",
        "| Classification | Label Format | Activation Function | Loss |\n",
        "|---------------|-----------------------|-----------------|----------|\n",
        "| Binary      | Probability of 1st class | sigmoid | binary crossentropy |\n",
        "| Binary      | One-hot encoding (2 outputs) | softmax | categorical crossentropy |\n",
        "| Multiclass |  One-hot encoding | softmax | categorical crossentropy |\n",
        "| Multiclass | Class Number | softmax | sparse categorical crossentropy |\n",
        "| Multilabel | One-hot encoding | sigmoid | categorical crossentropy |\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "gZ-kWx84bMDH"
      },
      "source": [
        "**Task**: \n",
        "Use Keras to train a classifier for MNIST handwritten digits:\n",
        "* Notice that Keras contains some standard datasets, including MNIST. To use MNIST from Keras, you only need a couple of lines of code (more information [here](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist))\n",
        "* Try several network configuration, with different number of layers/neurons, activation functions.\n",
        "\n",
        "What is the best accuracy you were able to achieve?"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "yX6hqiafwHl9"
      },
      "source": [
        "## Takeaways\n",
        "\n",
        "* **Keras** is really recommended for beginners, because it allows to construct networks from layers quite easily, and then train it with just a couple of lines of code\n",
        "* If non-standard architecture is needed, you would need to learn a bit deeper into Tensorflow. Or you can ask someone to implement custom logic as a Keras layer, and then use it in Keras models\n",
        "* It is a good idea to look at PyTorch as well and compare approaches. \n",
        "\n",
        "A good sample notebook from the creator of Keras on Keras and Tensorflow 2.0 can be found [here](https://t.co/k694J95PI8)."
      ]
    }
  ],
  "metadata": {
    "celltoolbar": "Slideshow",
    "colab": {
      "collapsed_sections": [],
      "name": "IntroKerasTF.ipynb",
      "provenance": []
    },
    "interpreter": {
      "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
    },
    "kernelspec": {
      "display_name": "Python 3.8.12 64-bit (conda)",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "livereveal": {
      "start_slideshow_at": "selected"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}