{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Perceptron\n",
    "\n",
    "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n",
    "\n",
    "As we have discussed, perceptron allows you to solve **binary classification problem**, i.e. to classify input examples into two classes - we can call them **positive** and **negative**.\n",
    "\n",
    "First, let's import some required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pylab\n",
    "from matplotlib import gridspec\n",
    "from sklearn.datasets import make_classification\n",
    "import numpy as np\n",
    "from ipywidgets import interact, interactive, fixed\n",
    "import ipywidgets as widgets\n",
    "import pickle\n",
    "import os\n",
    "import gzip\n",
    "\n",
    "# pick the seed for reproducability - change it to explore the effects of random variations\n",
    "np.random.seed(1)\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Toy Problem\n",
    "\n",
    "To begin with, let's start with a toy problem, where we have two input features. For example, in medicine we may want to classify tumours into benign and malignant, depending on its size and age.\n",
    "\n",
    "We will generate a random classification dataset using `make_classification` function from SciKit Learn library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features:\n",
      " [[-1.7441838  -1.3952037 ]\n",
      " [ 2.5921783  -0.08124504]\n",
      " [ 0.9218062   0.91789985]\n",
      " [-0.8437018  -0.18738253]]\n",
      "Labels:\n",
      " [-1 -1  1 -1]\n"
     ]
    }
   ],
   "source": [
    "n = 50\n",
    "X, Y = make_classification(n_samples = n, n_features=2,\n",
    "                           n_redundant=0, n_informative=2, flip_y=0)\n",
    "Y = Y*2-1 # convert initial 0/1 values into -1/1\n",
    "X = X.astype(np.float32); Y = Y.astype(np.int32) # features - float, label - int\n",
    "\n",
    "# Split the dataset into training and test\n",
    "train_x, test_x = np.split(X, [ n*8//10])\n",
    "train_labels, test_labels = np.split(Y, [n*8//10])\n",
    "print(\"Features:\\n\",train_x[0:4])\n",
    "print(\"Labels:\\n\",train_labels[0:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also plot the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-4-ade59d2ee664>:11: UserWarning: Matplotlib is currently using module://ipykernel.pylab.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
      "  fig.show()\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 21032 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_dataset(suptitle, features, labels):\n",
    "    # prepare the plot\n",
    "    fig, ax = pylab.subplots(1, 1)\n",
    "    #pylab.subplots_adjust(bottom=0.2, wspace=0.4)\n",
    "    fig.suptitle(suptitle, fontsize = 16)\n",
    "    ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
    "    ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
    "\n",
    "    colors = ['r' if l>0 else 'b' for l in labels]\n",
    "    ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
    "    fig.show()\n",
    "\n",
    "plot_dataset('Training data', train_x, train_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Perceptron\n",
    "\n",
    "Since perceptron is a binary classifier, for each input vector $x$ the output of our perceptron would be either +1 or -1, depending on the class. The output will be computed using the formula\n",
    "\n",
    "$$y(\\mathbf{x}) = f(\\mathbf{w}^{\\mathrm{T}}\\mathbf{x})$$\n",
    "\n",
    "where $\\mathbf{w}$ is a weight vector, $f$ is a step activation function:\n",
    "$$\n",
    "f(x) = \\begin{cases}\n",
    "         +1 & x \\geq 0 \\\\\n",
    "         -1 & x < 0\n",
    "       \\end{cases} \\\\\n",
    "$$\n",
    "\n",
    "However, a generic linear model should also have a bias, i.e. ideally we should compute $y$ as $y=f(\\mathbf{w}^{\\mathrm{T}}\\mathbf{x}+\\mathbf{b})$. To simplify our model, we can get rid of this bias term by adding one more dimension to our input features, which always equals to 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.92180622  0.91789985  1.        ]\n",
      " [-1.06435513  1.49764717  1.        ]\n",
      " [ 0.32839951  2.25677919  1.        ]]\n"
     ]
    }
   ],
   "source": [
    "pos_examples = np.array([ [t[0], t[1], 1] for i,t in enumerate(train_x) \n",
    "                          if train_labels[i]>0])\n",
    "neg_examples = np.array([ [t[0], t[1], 1] for i,t in enumerate(train_x) \n",
    "                          if train_labels[i]<0])\n",
    "print(pos_examples[0:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training Algorithm\n",
    "\n",
    "In order to train the perceptron, we need to find out weights $\\mathbf{w}$ that will minimize the error. The error is defined using **perceptron criteria**:\n",
    "\n",
    "$$E(\\mathbf{w}) = -\\sum_{n \\in \\mathcal{M}}\\mathbf{w}^{\\mathrm{T}}\\mathbf{x}_{n}t_{n}$$\n",
    " \n",
    "  * $t_{n} \\in \\{-1, +1\\}$ for negative and positive training samples, respectively\n",
    "  * $\\mathcal{M}$ - a set of wrongly classified examples\n",
    "  \n",
    "We will use the process of **gradient descent**. Starting with some initial random weights $\\mathbf{w}^{(0)}$, we will adjust weights on each step of the training using the gradient of $E$:\n",
    "\n",
    "$$\\mathbf{w}^{\\tau + 1}=\\mathbf{w}^{\\tau} - \\eta \\nabla E(\\mathbf{w}) = \\mathbf{w}^{\\tau} + \\eta\\sum_{n \\in \\mathcal{M}}\\mathbf{x}_{n} t_{n}$$\n",
    "\n",
    "where $\\eta$ is a **learning rate**, and $\\tau\\in\\mathbb{N}$ - number of iteration.\n",
    "\n",
    "Let's define this algorithm in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def train(positive_examples, negative_examples, num_iterations = 100):\n",
    "    num_dims = positive_examples.shape[1]\n",
    "    \n",
    "    # Initialize weights. \n",
    "    # We initialize with 0 for simplicity, but random initialization is also a good idea\n",
    "    weights = np.zeros((num_dims,1)) \n",
    "    \n",
    "    pos_count = positive_examples.shape[0]\n",
    "    neg_count = negative_examples.shape[0]\n",
    "    \n",
    "    report_frequency = 10\n",
    "    \n",
    "    for i in range(num_iterations):\n",
    "        # Pick one positive and one negative example\n",
    "        pos = random.choice(positive_examples)\n",
    "        neg = random.choice(negative_examples)\n",
    "\n",
    "        z = np.dot(pos, weights)   \n",
    "        if z < 0: # positive example was classified as negative\n",
    "            weights = weights + pos.reshape(weights.shape)\n",
    "\n",
    "        z  = np.dot(neg, weights)\n",
    "        if z >= 0: # negative example was classified as positive\n",
    "            weights = weights - neg.reshape(weights.shape)\n",
    "            \n",
    "        # Periodically, print out the current accuracy on all examples \n",
    "        if i % report_frequency == 0:             \n",
    "            pos_out = np.dot(positive_examples, weights)\n",
    "            neg_out = np.dot(negative_examples, weights)        \n",
    "            pos_correct = (pos_out >= 0).sum() / float(pos_count)\n",
    "            neg_correct = (neg_out < 0).sum() / float(neg_count)\n",
    "            print(\"Iteration={}, pos correct={}, neg correct={}\".format(i,pos_correct,neg_correct))\n",
    "\n",
    "    return weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run the training on our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration=0, pos correct=0.2631578947368421, neg correct=0.6190476190476191\n",
      "Iteration=10, pos correct=0.8947368421052632, neg correct=0.8571428571428571\n",
      "Iteration=20, pos correct=0.8421052631578947, neg correct=1.0\n",
      "Iteration=30, pos correct=0.8947368421052632, neg correct=0.9523809523809523\n",
      "Iteration=40, pos correct=0.8947368421052632, neg correct=0.9523809523809523\n",
      "Iteration=50, pos correct=0.9473684210526315, neg correct=0.9047619047619048\n",
      "Iteration=60, pos correct=0.8947368421052632, neg correct=0.9523809523809523\n",
      "Iteration=70, pos correct=0.8947368421052632, neg correct=0.9047619047619048\n",
      "Iteration=80, pos correct=0.8947368421052632, neg correct=0.6190476190476191\n",
      "Iteration=90, pos correct=0.8421052631578947, neg correct=1.0\n",
      "[[-0.66042328  4.90850882 -1.        ]]\n"
     ]
    }
   ],
   "source": [
    "wts = train(pos_examples,neg_examples)\n",
    "print(wts.transpose())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, initial accuracy is around 50%, but it quickly increases to higher values close to 90%.\n",
    "\n",
    "Let's visualize how classes are separated. Our classification function looks like $\\mathbf{w}^Tx$, and it is greater than 0 for one class, and is below 0 for another. Thus, class separation line is defined by $\\mathbf{w}^Tx = 0$. Since we have only two dimensions $x_0$ and $x_1$, the equation for the line would be $w_0x_0+w_1x_1+w_2 = 0$ (remember that we have explicitly defined an extra dimension $x_2=1$). Let's plot this line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def plot_boundary(positive_examples, negative_examples, weights):\n",
    "    if np.isclose(weights[1], 0):\n",
    "        if np.isclose(weights[0], 0):\n",
    "            x = y = np.array([-6, 6], dtype = 'float32')\n",
    "        else:\n",
    "            y = np.array([-6, 6], dtype='float32')\n",
    "            x = -(weights[1] * y + weights[2])/weights[0]\n",
    "    else:\n",
    "        x = np.array([-6, 6], dtype='float32')\n",
    "        y = -(weights[0] * x + weights[2])/weights[1]\n",
    "\n",
    "    pylab.xlim(-6, 6)\n",
    "    pylab.ylim(-6, 6)                      \n",
    "    pylab.plot(positive_examples[:,0], positive_examples[:,1], 'bo')\n",
    "    pylab.plot(negative_examples[:,0], negative_examples[:,1], 'ro')\n",
    "    pylab.plot(x, y, 'g', linewidth=2.0)\n",
    "    pylab.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 7508 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_boundary(pos_examples,neg_examples,wts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Evaluate on Test Dataset\n",
    "\n",
    "In the beginning, we have put apart some data to the test dataset. Let's see how accurate our classifier is on this test dataset. In order to do this, we also expand the test dataset with an extra dimension, multiply by weights matrix, and make sure that the obtained value is of the same sign as the label (+1 or -1). We then add together all boolean values and divide by the length of test sample, to obtain the accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def accuracy(weights, test_x, test_labels):\n",
    "    res = np.dot(np.c_[test_x,np.ones(len(test_x))],weights)\n",
    "    return (res.reshape(test_labels.shape)*test_labels>=0).sum()/float(len(test_labels))\n",
    "\n",
    "accuracy(wts, test_x, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Observing the training process\n",
    "\n",
    "We have seen before how the accuracy decreases during training. It would be nice to see how the separation line behaves during training. The code below will visualize everything on one graph, and you should be able to move the slider to \"time-travel\" through the training process. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def train_graph(positive_examples, negative_examples, num_iterations = 100):\n",
    "    num_dims = positive_examples.shape[1]\n",
    "    weights = np.zeros((num_dims,1)) # initialize weights\n",
    "    \n",
    "    pos_count = positive_examples.shape[0]\n",
    "    neg_count = negative_examples.shape[0]\n",
    "    \n",
    "    report_frequency = 15;\n",
    "    snapshots = []\n",
    "    \n",
    "    for i in range(num_iterations):\n",
    "        pos = random.choice(positive_examples)\n",
    "        neg = random.choice(negative_examples)\n",
    "\n",
    "        z = np.dot(pos, weights)   \n",
    "        if z < 0:\n",
    "            weights = weights + pos.reshape(weights.shape)\n",
    "\n",
    "        z  = np.dot(neg, weights)\n",
    "        if z >= 0:\n",
    "            weights = weights - neg.reshape(weights.shape)\n",
    "            \n",
    "        if i % report_frequency == 0:             \n",
    "            pos_out = np.dot(positive_examples, weights)\n",
    "            neg_out = np.dot(negative_examples, weights)        \n",
    "            pos_correct = (pos_out >= 0).sum() / float(pos_count)\n",
    "            neg_correct = (neg_out < 0).sum() / float(neg_count)\n",
    "            # make correction a list so it is homogeneous to weights list then numpy array accepts\n",
    "            snapshots.append((np.concatenate(weights),[(pos_correct+neg_correct)/2.0,0,0]))\n",
    "\n",
    "    return np.array(snapshots)\n",
    "\n",
    "snapshots = train_graph(pos_examples,neg_examples)\n",
    "\n",
    "def plotit(pos_examples,neg_examples,snapshots,step):\n",
    "    fig = pylab.figure(figsize=(10,4))\n",
    "    fig.add_subplot(1, 2, 1)\n",
    "    plot_boundary(pos_examples, neg_examples, snapshots[step][0])\n",
    "    fig.add_subplot(1, 2, 2)\n",
    "    pylab.plot(np.arange(len(snapshots[:,1])), snapshots[:,1])\n",
    "    pylab.ylabel('Accuracy')\n",
    "    pylab.xlabel('Iteration')\n",
    "    pylab.plot(step, snapshots[step,1][0], \"bo\")\n",
    "    pylab.show()\n",
    "def pl1(step): plotit(pos_examples,neg_examples,snapshots,step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8561af1ae77c421f9ca068fe0bdac566",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='step', max=6), Output()), _dom_classes=('widget-interact…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.pl1(step)>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interact(pl1, step=widgets.IntSlider(value=0, min=0, max=len(snapshots)-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Limitations of the Perceptron\n",
    "\n",
    "As you have seen above, perceptron is a **linear classifier**. It can distinguish between two classes well if they are **linearly separable**, i.e. can be separated by a straight line. Otherwise, perceptron training process will not converge.\n",
    "\n",
    "A most obvious example of a problem that cannot be solved by a perceptron is so-called **XOR problem**. We want our perceptron to learn the XOR boolean function, which has the following truth table:\n",
    "\n",
    "|   | 0 | 1 |\n",
    "|---|---|---|\n",
    "| 0 | 0 | 1 | \n",
    "| 1 | 1 | 0 |\n",
    "\n",
    "Let's try and do that! We will manually populate all positive and negative training samples, and then call our train function defined above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pos_examples_xor = np.array([[1,0,1],[0,1,1]])\n",
    "neg_examples_xor = np.array([[1,1,1],[0,0,1]])\n",
    "\n",
    "snapshots_xor = train_graph(pos_examples_xor,neg_examples_xor,1000)\n",
    "def pl2(step): plotit(pos_examples_xor,neg_examples_xor,snapshots_xor,step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8f45bf78e4c8471fbc6eea233dde2bf6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='step', max=6), Output()), _dom_classes=('widget-interact…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.pl2(step)>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interact(pl2, step=widgets.IntSlider(value=0, min=0, max=len(snapshots)-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As you can see from the graph above, the accuracy never goes above 75%, because it is impossible to draw a straight line in such a way as to get all possible examples right.\n",
    "\n",
    "The XOR problem is a classical example of perceptron limitations, and it was pointed out by Marvin Minsky and Seymour Papert in 1969 in their book [Perceptrons](https://en.wikipedia.org/wiki/Perceptrons_(book)). This observation limited research in the area of neural networks for almost 10 years, even though - and we will see this in the next section of our course - multi-layered perceptrons are perfectly capable of solving such problems.\n",
    "\n",
    "## Complex Example - MNIST\n",
    "\n",
    "Even though perceptron cannot solve XOR problem, it can solve many more complex problems, such as handwritten character recognition.\n",
    "\n",
    "A dataset that is often used when mastering machine learning is called [MNIST](https://en.wikipedia.org/wiki/MNIST_database). It has been created by Modified National Institute of Standards and Technology, and contains a training set of 60000 handwritten digits, collected from around 250 students and employees of the institute. There is also a test dataset of 10000 digits, collected from different individuals.\n",
    "\n",
    "All digits are represented by grayscale images of size 28x28 pixels.\n",
    "\n",
    "> MNIST Dataset is available as a training competition on [Kaggle](https://www.kaggle.com/c/digit-recognizer), a site that hosts machine learning competitions and contests. Once you learn how to classify MNIST digits, you can submit your solution to Kaggle to see how it is rated among other participants. \n",
    "\n",
    "We start by loading MNIST dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# If you are not running this notebook from a cloned repository, you may need to grab the binary dataset file first\n",
    "# !wget https://github.com/microsoft/AI-For-Beginners/blob/main/data/mnist.pkl.gz?raw=true\n",
    "# In this case correct the link to the dataset below as well.\n",
    "\n",
    "with gzip.open('../../data/mnist.pkl.gz', 'rb') as mnist_pickle:\n",
    "    MNIST = pickle.load(mnist_pickle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now plot the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  0   0 188 255  94   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0 191 250 253  93   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0]\n",
      "1\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 25144 characters]",
      "text/plain": [
       "<Figure size 720x360 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(MNIST['Train']['Features'][0][130:180])\n",
    "print(MNIST['Train']['Labels'][0])\n",
    "features = MNIST['Train']['Features'].astype(np.float32) / 256.0\n",
    "labels = MNIST['Train']['Labels']\n",
    "fig = pylab.figure(figsize=(10,5))\n",
    "for i in range(10):\n",
    "    ax = fig.add_subplot(1,10,i+1)\n",
    "    pylab.imshow(features[i].reshape(28,28))\n",
    "pylab.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because perceptron is a binary classifier, we will limit our problem to recognizing only two digits. The function below will populate positive and negative sample arrays with two given digits (and will also show samples of those digits for clarity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def set_mnist_pos_neg(positive_label, negative_label):\n",
    "    positive_indices = [i for i, j in enumerate(MNIST['Train']['Labels']) \n",
    "                          if j == positive_label]\n",
    "    negative_indices = [i for i, j in enumerate(MNIST['Train']['Labels']) \n",
    "                          if j == negative_label]\n",
    "\n",
    "    positive_images = MNIST['Train']['Features'][positive_indices]\n",
    "    negative_images = MNIST['Train']['Features'][negative_indices]\n",
    "\n",
    "    fig = pylab.figure()\n",
    "    ax = fig.add_subplot(1, 2, 1)\n",
    "    pylab.imshow(positive_images[0].reshape(28,28), cmap='gray', interpolation='nearest')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    ax = fig.add_subplot(1, 2, 2)\n",
    "    pylab.imshow(negative_images[0].reshape(28,28), cmap='gray', interpolation='nearest')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    pylab.show()\n",
    "    \n",
    "    return positive_images, negative_images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by trying to classify between 0 and 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACqCAYAAACTZZUqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAI8UlEQVR4nO3dS6hV1R8H8H3kKuHzopZIiIgEooYRUYN0IGoZCkqiGSIiDq5Y6ERwEBENGog5UtTIBtee+IAok3yEICImSoMiqFEiKb3z+iqse/+TKP6steUc9z33dzz38xl+WXedlZ77ddPae69aX19fAcDAGxK9AIDBSgEDBFHAAEEUMEAQBQwQRAEDBOloZHCtVnPPGk3V19dXG+jP9L2m2cq+166AAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCNLQoZzce06cOJHN582bl2Rr1qzJjt23b1+/romBM3bs2CQbOXJkduwLL7xQ15xPPPFENt+1a1eS9fT0ZMcePXo0yfr6Bt/ZqK6AAYIoYIAgChggiAIGCGITro2cPHkyyZ588sns2N7e3iQbjJsg96JRo0Yl2TPPPJMd+8477yRZR0dzfu0nTpyYZJMmTcqO7e7uTrKtW7dmx3733XeV1tXKXAEDBFHAAEEUMEAQBQwQRAEDBKk1svNdq9Vsk7eAl156KZu//PLLSTZ06NDs2P379yfZunXrsmNv3rzZwOqq6evrqw3Yh/2jVb/XnZ2d2fztt99OskWLFjV5Nc33ww8/ZPMlS5Yk2TfffJMde/Xq1X5dU38p+167AgYIooABgihggCAKGCCITbgWt3Tp0iR7//33s2OHDRuWZF9++WV27Jw5c5Ls2rVrjS2uCWzC/WfhwoXZ/MiRIwO8ktazYcOGbL5nz54BXkl9bMIBtBgFDBBEAQMEUcAAQRQwQBAvZG8RZS+ufuWVV5Isd7dDURTFr7/+mmS5x5OLojXueOA/s2fPTrItW7YErOT/bdq0KckuX76cHbt58+YkKztBuapt27Zl819++SXJDhw40JQ19AdXwABBFDBAEAUMEEQBAwTxKHKAxx9/PMnefPPN7NiZM2fWPe+qVauS7IMPPqh/YS1gsD6KfPDgwSR79tlnK897/vz5JPv888/r/vk33ngjyb766qvs2BEjRiTZ2LFjs2NzG2O534tGHTp0KMmWL19eed6qPIoM0GIUMEAQBQwQRAEDBFHAAEE8itxEq1evzubd3d1JVnY3Su6U1xMnTmTHHj16tIHVEaFWy9/kMWRItWuh3B0wRVEUP/74Y5J99tlnlT6rzI0bN+rKiqIoPv300yR77LHHsmMb+bOZNm1aki1evDg79vDhw3XP2yyugAGCKGCAIAoYIIgCBgjiUeR+MmHChCQ7fvx4dmzu8eKyv4d9+/Yl2dq1axtc3b2j3R9FnjVrVjb/4osvKs07efLkbH7p0qVK8w6kZcuWZfOq7/Mte8y/q6ur0ryN8CgyQItRwABBFDBAEAUMEEQBAwTxKPJd6OzsTLJjx44l2YwZM+qes+yU4o8++qjuOWh9U6ZMqTxHT09Pkt2+fbvyvNHOnDmTzXP/vaNHj272cgaEK2CAIAoYIIgCBgiigAGC2IS7C7nTXxs5vThn0qRJ2bxsc4570++//155jnPnziXZb7/9VnneaFeuXMnmR44cSbKVK1fWPe/TTz+dzUeOHJlk169fr3ve/uAKGCCIAgYIooABgihggCDeB3wH48ePz+a5p94eeeSRuuc9e/Zsks2dOzc79s8//6x73nbQTu8Dzj2t9e2332bHPvDAA5U+qx3eB1xm0aJFSfbxxx9XnnfcuHFJ1qzNTO8DBmgxChggiAIGCKKAAYIoYIAgHkW+g507d2bz3Mm2ubtJyt5vOn/+/CQbbHc7DAYdHemvV9W7HQaj77//PnoJTeMKGCCIAgYIooABgihggCA24f6Re+x46tSpdf987lDErVu3ZsfacBsccu/+fffdd7NjV61a1eTV0IpcAQMEUcAAQRQwQBAFDBBEAQMEGXR3QZQ9Cvree+8l2aOPPpod+8cffyTZ+vXrk+zw4cMNro520tvbm2THjx/Pjq16F8SBAweyee6x94E++bdenZ2d2by7u7vSvHv27Mnm/XFCdVWugAGCKGCAIAoYIIgCBggy6E5F7urqyua7du2qe45Tp04lWdmpxjSmnU5FzhkzZkw2P3nyZJI1ctJ2mfPnzyfZli1b6l5Ds9x///1J9vrrr2fHrl69uu55b926lWTTp0/Pjr148WLd81blVGSAFqOAAYIoYIAgChggiAIGCNLWd0E8//zzSbZ79+7s2FGjRiVZ2anGK1asSLIrV640uDpy2v0uiDKzZ89OsrLv6owZMyp91unTp7P5xo0b6/r5np6ebD5s2LAku++++7Jjc48XP/zww3V9/p0cOnQoyZYvX1553qrcBQHQYhQwQBAFDBBEAQMEaYtNuLLHOy9cuJBkU6ZMqXveZcuWZfMPP/yw7jlozGDdhMvJbfYWRVG89dZbSTZixIhmL+dfP/30UzYfPnx4kg3kuoqiKFauXJlk+/fvH9A15NiEA2gxChggiAIGCKKAAYIoYIAgbXEq8pIlS7J5I3c85IwePbrSz0MVZbv3Dz74YJJt37692cv5V+5l6s109erVJCs7WOGTTz5p9nL6lStggCAKGCCIAgYIooABgrTFJtzt27ezeW9vb5INGZL/N+fvv/9OsoceeqjawqAJ9u7dm2QLFizIjl24cGGzl9Nvbty4kc2fe+65JDt27FizlzMgXAEDBFHAAEEUMEAQBQwQRAEDBGmLF7KX+frrr5OsoyN/48drr72WZLmTW2kuL2S/O2WnD8+fPz/JnnrqqezYF198MclqtfSvo6wzcmN37NiRHfvqq68m2V9//ZUdm3sU+V7jhewALUYBAwRRwABBFDBAkLbehOPeYxOOdmQTDqDFKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCdDQ4/ueiKC42YyFQFMXkoM/1vaaZSr/XDR1LD0D/8b8gAIIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYL8D+KUFeSspSmeAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos1,neg1 = set_mnist_pos_neg(1,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def plotit2(snapshots_mn,step):\n",
    "    fig = pylab.figure(figsize=(10,4))\n",
    "    ax = fig.add_subplot(1, 2, 1)\n",
    "    pylab.imshow(snapshots_mn[step][0].reshape(28, 28), interpolation='nearest')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    pylab.colorbar()\n",
    "    ax = fig.add_subplot(1, 2, 2)\n",
    "    ax.set_ylim([0,1])\n",
    "    pylab.plot(np.arange(len(snapshots_mn[:,1])), snapshots_mn[:,1])\n",
    "    pylab.plot(step, snapshots_mn[step,1], \"bo\")\n",
    "    pylab.show()\n",
    "def pl3(step): plotit2(snapshots_mn,step)\n",
    "def pl4(step): plotit2(snapshots_mn2,step)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afbc754ef0d04c95a1af039574b46a6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='step', max=66), Output()), _dom_classes=('widget-interac…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.pl3(step)>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snapshots_mn = train_graph(pos1,neg1,1000)    \n",
    "interact(pl3, step=widgets.IntSlider(value=0, min=0, max=len(snapshots_mn) - 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note how accuracy goes up to almost 100% very fast.\n",
    "\n",
    "Please, move the slider to some position towards the end of the training, and observe the weight matrix plotted on the left. This matrix will allow you to understand how perceptron actually works. You can see the high weight values in the middle of the field, which correspond to pixels that are typically present for digit 1, and low negative values by the sides, where parts of 0 digit are. So, if the digit presented to the perceptron is in fact 1, middle part of it will be mupliplied by high values, producing positive result. On the contrary, when perceptron observes 0, corresponding pixels will be multiplied by negative numbers.\n",
    "\n",
    "> You may notice that if we give our perceptron a digit 1 slightly shifted horizontally, so that its pixels occupy the place where there are vertical parts of 0, we may receive incorrect result. Since the nature of our MNIST dataset is such that all digits are centered and positioned properly, and perceptron relies on this to distinguish between digits.\n",
    "\n",
    "Now let's try different digits: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACqCAYAAACTZZUqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAI90lEQVR4nO3dT4hVZR8H8HPfphJxLKzQippFZIOGTFEZFNEuxE3YIjAKFxYoKkqMItGqiCjEhSARWRkVgqCbyAKFslqJC/9ubKOgKP6JcuHoQPfdvBAvz+/Uvc2993e98/ksvzxzzsPM8cvB8zznNJrNZgVA7/0newIA05UCBkiigAGSKGCAJAoYIIkCBkgy1M7gRqNhzRpd1Ww2G70+p+uabqu7rt0BAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyQZyp5ANz3yyCNF9sQTT/Ts/CtXrgzz559/vsi2bt0ajj18+HCR7d+/Pxx74cKF1icHLRgdHS2ytWvXhmNvv/32Ips7d244dunSpS3P4dChQ0W2Z8+ecOy+ffuK7OjRoy2fq9fcAQMkUcAASRQwQBIFDJBEAQMkaTSbzdYHNxqtD+6huiet0WqBBQsWdHs6Xffdd9+FeTtPlvtVs9ls9Pqc/Xpdd8vw8HCYv/fee0X22muvFdmsWbNaPlejEf852+mddkxMTBTZ7t27w7ErVqzoyhwidde1O2CAJAoYIIkCBkiigAGSDMRW5E8++STMB+GBW2TRokXZU+AmMTIyUmQ//vhjOPaBBx5o6ZjffvttmE9OThZZrx/CPfbYY0X28ssvh2N///33IhsfHw/H3rhxY2oTq+EOGCCJAgZIooABkihggCQKGCDJQKyCgOkuehl6VVXV119/XWQPPvhgODZambBr164ie/XVV8Of//PPP/9uij0RbZNevnx5OHbZsmVFNnPmzHCsVRAAA0YBAyRRwABJFDBAkoF4H/Drr78e5h999FGPZ9Ib586dC/NWt5L2M+8D/nfqrvXo30bd9uAvv/yyyNavX19kV65caW9yeB8wQL9RwABJFDBAEgUMkEQBAyQZiFUQs2fPDvOjR48WWTsrBU6ePBnmn376aZH99ttvRbZjx46Wz9UOqyA6q1+v63ZcvHgxzO+6664i+/zzz8OxGzZsKLLopeW0zyoIgD6jgAGSKGCAJAoYIMlAvA/4jz/+CPMXX3yxyOq+oPzVV18V2c6dO8Ox0VbMuuNCpy1ZsqTI7rjjjnBs9JA9ethWVVN/4HbnnXcW2dBQXDHRvC5fvjyl89+M3AEDJFHAAEkUMEASBQyQRAEDJBmIrcjtqHtaPNUnwNH24Llz507pmHX2798f5i+88EJXztdLtiL/pe5Lxz/88EORPfXUUy0f95Zbbml57L333ltkq1atCsdGebQVuqqq6vr160X28ccfh2PHx8eLrFtfKe4WW5EB+owCBkiigAGSKGCAJAOxFbkd7TxsGxkZCfO33367yIaHh//1nP7OTz/9VGQrVqzoyrnoL3XXVDsP3L755psiW7lyZTh206ZNRXbPPfe0PK923HbbbUW2Zs2acOylS5eK7J133pnyHPqBO2CAJAoYIIkCBkiigAGSTLudcO04cuRImD/66KMdP9fExESYR+9YnZyc7Pj5+4WdcH+59dZbwzzaCfnss8+2fNxGI/4Vt9oFhw4dCvNjx461PIeXXnqpyOp2qZ4/f77IHn/88XDshQsXWp5DL9kJB9BnFDBAEgUMkEQBAyRRwABJpt1W5DrRE+dou2QnRCsetmzZEo4d5BUP/L26v/1bb71VZAcOHAjHRtf11atXw7HRl8Hff//9Ijtz5kz48+2IVm1EK36qKn4n8UMPPRSO7ddVEHXcAQMkUcAASRQwQBIFDJDEQ7j/id6FOn/+/K6ca86cOUUWfaQQIj///HORLVy4MBwbfYDz2rVr4dhOPFxrVbTtuW4r9OXLl4vs7NmzHZ9TBnfAAEkUMEASBQyQRAEDJFHAAEmm3SqI0dHRMO/Gl4brvsDczkvwoRW//vpr9hRCdf/e7r///paPcfjw4SI7ffr0v55TP3EHDJBEAQMkUcAASRQwQJKBfgg3NjZWZHv27AnHjoyMdPz8b775ZpjfuHGj4+eCfrRz584wnzVrVsvH2Lt3b6em03fcAQMkUcAASRQwQBIFDJBEAQMkGehVEM8880yRdWO1Q1VV1YkTJ4rs4MGDXTlX9KXbqqqqhx9+uCvnO3/+fJFduXKlK+fi5rVhw4Yie/LJJ8Ox0Xb8HTt2hGM/++yzqU2sj7kDBkiigAGSKGCAJAoYIMlAP4TrpeirtFu3bg3HHjlyZErnqtvGuW7duikdt873339fZMuWLQvHTkxMdGUO9I/nnnsuzLds2VJkjUYjHHv16tUie/fdd8Oxk5OTbczu5uIOGCCJAgZIooABkihggCQKGCDJQK+CiLbQRk9fq6qqhoeHO37+pUuXtpVna+d3MzQ00JfOtDRz5swiW7NmTZGNj4+HPx9tL65bwbBx48YiO3PmzD9NceC4AwZIooABkihggCQKGCBJI/qP89rBjUbrg/vUL7/8EuZPP/10j2fSG9GDyKqKt33Wje3lV2mbzWa8d7WLbrbrevHixUV23333hWOjv90bb7wRjl27dm2RLViwoM3Z/b8PPvggzDdv3jyl495s6q5rd8AASRQwQBIFDJBEAQMkUcAASabdftJXXnklzLdt21ZkdS+ernsheq9cv349zA8cOFBkdU+bjx8/3tE50Tvz5s0rsi+++CIce+3atSK7++67w7Gtrog6depUmEdfNf7www9bOuZ05Q4YIIkCBkiigAGSKGCAJNNuK3I76t7bO3v27CJbvXp1kW3fvr3jc6qq+i8P93LLcLfYivzPxsbGiqxui/2MGTOKrO5LxdHXuqNrKnrYVlVVdfbs2TDHVmSAvqOAAZIoYIAkChggiQIGSGIVBH3FKggGkVUQAH1GAQMkUcAASRQwQBIFDJBEAQMkUcAASRQwQBIFDJBEAQMkUcAASRQwQBIFDJBEAQMkUcAASYbaHH+pqqrT3ZgIVFU1knRe1zXdVHtdt/VCdgA6x39BACRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyT5L95QHHzClR5NAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos2,neg2 = set_mnist_pos_neg(2,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bf6f25d14c3b4d548ac026af97f70e2a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='step', max=66), Output()), _dom_classes=('widget-interac…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.pl4(step)>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snapshots_mn2 = train_graph(pos2,neg2,1000)\n",
    "interact(pl4, step=widgets.IntSlider(value=0, min=0, max=len(snapshots_mn2) - 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Discussion\n",
    "\n",
    "For some reason, 2 and 5 are not as easily separable. Even though we get relatively high accuracy (above 85%), we can clearly see how perceptron stops learning at some point.\n",
    "\n",
    "To understand why this happens, we can try to use [Principal Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) (PCA). It is a machine learning technique used to lower the dimensionality of the input dataset, in such a way as to obtain the best separability between classes. \n",
    "\n",
    "In our case, an input image has 784 pixels (input features), and we want to use PCA to reduce the number of parameter to just 2, so that we can plot them on the graph. Those two parameters would be a linear combination of original features, and we can view this procedure as \"rotating\" our original 784-dimensional space and observing it's projection to our 2D-space, until we get the best view that separates the classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "def pca_analysis(positive_label, negative_label):\n",
    "    positive_images, negative_images = set_mnist_pos_neg(positive_label, negative_label)\n",
    "    M = np.append(positive_images, negative_images, 0)\n",
    "\n",
    "    mypca = PCA(n_components=2)\n",
    "    mypca.fit(M)\n",
    "    \n",
    "    pos_points = mypca.transform(positive_images[:200])\n",
    "    neg_points = mypca.transform(negative_images[:200])\n",
    "\n",
    "    pylab.plot(pos_points[:,0], pos_points[:,1], 'bo')\n",
    "    pylab.plot(neg_points[:,0], neg_points[:,1], 'ro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACqCAYAAACTZZUqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAI8UlEQVR4nO3dS6hV1R8H8H3kKuHzopZIiIgEooYRUYN0IGoZCkqiGSIiDq5Y6ERwEBENGog5UtTIBtee+IAok3yEICImSoMiqFEiKb3z+iqse/+TKP6steUc9z33dzz38xl+WXedlZ77ddPae69aX19fAcDAGxK9AIDBSgEDBFHAAEEUMEAQBQwQRAEDBOloZHCtVnPPGk3V19dXG+jP9L2m2cq+166AAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCNLQoZzce06cOJHN582bl2Rr1qzJjt23b1+/romBM3bs2CQbOXJkduwLL7xQ15xPPPFENt+1a1eS9fT0ZMcePXo0yfr6Bt/ZqK6AAYIoYIAgChggiAIGCGITro2cPHkyyZ588sns2N7e3iQbjJsg96JRo0Yl2TPPPJMd+8477yRZR0dzfu0nTpyYZJMmTcqO7e7uTrKtW7dmx3733XeV1tXKXAEDBFHAAEEUMEAQBQwQRAEDBKk1svNdq9Vsk7eAl156KZu//PLLSTZ06NDs2P379yfZunXrsmNv3rzZwOqq6evrqw3Yh/2jVb/XnZ2d2fztt99OskWLFjV5Nc33ww8/ZPMlS5Yk2TfffJMde/Xq1X5dU38p+167AgYIooABgihggCAKGCCITbgWt3Tp0iR7//33s2OHDRuWZF9++WV27Jw5c5Ls2rVrjS2uCWzC/WfhwoXZ/MiRIwO8ktazYcOGbL5nz54BXkl9bMIBtBgFDBBEAQMEUcAAQRQwQBAvZG8RZS+ufuWVV5Isd7dDURTFr7/+mmS5x5OLojXueOA/s2fPTrItW7YErOT/bdq0KckuX76cHbt58+YkKztBuapt27Zl819++SXJDhw40JQ19AdXwABBFDBAEAUMEEQBAwTxKHKAxx9/PMnefPPN7NiZM2fWPe+qVauS7IMPPqh/YS1gsD6KfPDgwSR79tlnK897/vz5JPv888/r/vk33ngjyb766qvs2BEjRiTZ2LFjs2NzG2O534tGHTp0KMmWL19eed6qPIoM0GIUMEAQBQwQRAEDBFHAAEE8itxEq1evzubd3d1JVnY3Su6U1xMnTmTHHj16tIHVEaFWy9/kMWRItWuh3B0wRVEUP/74Y5J99tlnlT6rzI0bN+rKiqIoPv300yR77LHHsmMb+bOZNm1aki1evDg79vDhw3XP2yyugAGCKGCAIAoYIIgCBgjiUeR+MmHChCQ7fvx4dmzu8eKyv4d9+/Yl2dq1axtc3b2j3R9FnjVrVjb/4osvKs07efLkbH7p0qVK8w6kZcuWZfOq7/Mte8y/q6ur0ryN8CgyQItRwABBFDBAEAUMEEQBAwTxKPJd6OzsTLJjx44l2YwZM+qes+yU4o8++qjuOWh9U6ZMqTxHT09Pkt2+fbvyvNHOnDmTzXP/vaNHj272cgaEK2CAIAoYIIgCBgiigAGC2IS7C7nTXxs5vThn0qRJ2bxsc4570++//155jnPnziXZb7/9VnneaFeuXMnmR44cSbKVK1fWPe/TTz+dzUeOHJlk169fr3ve/uAKGCCIAgYIooABgihggCDeB3wH48ePz+a5p94eeeSRuuc9e/Zsks2dOzc79s8//6x73nbQTu8Dzj2t9e2332bHPvDAA5U+qx3eB1xm0aJFSfbxxx9XnnfcuHFJ1qzNTO8DBmgxChggiAIGCKKAAYIoYIAgHkW+g507d2bz3Mm2ubtJyt5vOn/+/CQbbHc7DAYdHemvV9W7HQaj77//PnoJTeMKGCCIAgYIooABgihggCA24f6Re+x46tSpdf987lDErVu3ZsfacBsccu/+fffdd7NjV61a1eTV0IpcAQMEUcAAQRQwQBAFDBBEAQMEGXR3QZQ9Cvree+8l2aOPPpod+8cffyTZ+vXrk+zw4cMNro520tvbm2THjx/Pjq16F8SBAweyee6x94E++bdenZ2d2by7u7vSvHv27Mnm/XFCdVWugAGCKGCAIAoYIIgCBggy6E5F7urqyua7du2qe45Tp04lWdmpxjSmnU5FzhkzZkw2P3nyZJI1ctJ2mfPnzyfZli1b6l5Ds9x///1J9vrrr2fHrl69uu55b926lWTTp0/Pjr148WLd81blVGSAFqOAAYIoYIAgChggiAIGCNLWd0E8//zzSbZ79+7s2FGjRiVZ2anGK1asSLIrV640uDpy2v0uiDKzZ89OsrLv6owZMyp91unTp7P5xo0b6/r5np6ebD5s2LAku++++7Jjc48XP/zww3V9/p0cOnQoyZYvX1553qrcBQHQYhQwQBAFDBBEAQMEaYtNuLLHOy9cuJBkU6ZMqXveZcuWZfMPP/yw7jlozGDdhMvJbfYWRVG89dZbSTZixIhmL+dfP/30UzYfPnx4kg3kuoqiKFauXJlk+/fvH9A15NiEA2gxChggiAIGCKKAAYIoYIAgbXEq8pIlS7J5I3c85IwePbrSz0MVZbv3Dz74YJJt37692cv5V+5l6s109erVJCs7WOGTTz5p9nL6lStggCAKGCCIAgYIooABgrTFJtzt27ezeW9vb5INGZL/N+fvv/9OsoceeqjawqAJ9u7dm2QLFizIjl24cGGzl9Nvbty4kc2fe+65JDt27FizlzMgXAEDBFHAAEEUMEAQBQwQRAEDBGmLF7KX+frrr5OsoyN/48drr72WZLmTW2kuL2S/O2WnD8+fPz/JnnrqqezYF198MclqtfSvo6wzcmN37NiRHfvqq68m2V9//ZUdm3sU+V7jhewALUYBAwRRwABBFDBAkLbehOPeYxOOdmQTDqDFKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCdDQ4/ueiKC42YyFQFMXkoM/1vaaZSr/XDR1LD0D/8b8gAIIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYL8D+KUFeSspSmeAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 14112 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pca_analysis(1,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACqCAYAAACTZZUqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAI90lEQVR4nO3dT4hVZR8H8HPfphJxLKzQippFZIOGTFEZFNEuxE3YIjAKFxYoKkqMItGqiCjEhSARWRkVgqCbyAKFslqJC/9ubKOgKP6JcuHoQPfdvBAvz+/Uvc2993e98/ksvzxzzsPM8cvB8zznNJrNZgVA7/0newIA05UCBkiigAGSKGCAJAoYIIkCBkgy1M7gRqNhzRpd1Ww2G70+p+uabqu7rt0BAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyQZyp5ANz3yyCNF9sQTT/Ts/CtXrgzz559/vsi2bt0ajj18+HCR7d+/Pxx74cKF1icHLRgdHS2ytWvXhmNvv/32Ips7d244dunSpS3P4dChQ0W2Z8+ecOy+ffuK7OjRoy2fq9fcAQMkUcAASRQwQBIFDJBEAQMkaTSbzdYHNxqtD+6huiet0WqBBQsWdHs6Xffdd9+FeTtPlvtVs9ls9Pqc/Xpdd8vw8HCYv/fee0X22muvFdmsWbNaPlejEf852+mddkxMTBTZ7t27w7ErVqzoyhwidde1O2CAJAoYIIkCBkiigAGSDMRW5E8++STMB+GBW2TRokXZU+AmMTIyUmQ//vhjOPaBBx5o6ZjffvttmE9OThZZrx/CPfbYY0X28ssvh2N///33IhsfHw/H3rhxY2oTq+EOGCCJAgZIooABkihggCQKGCDJQKyCgOkuehl6VVXV119/XWQPPvhgODZambBr164ie/XVV8Of//PPP/9uij0RbZNevnx5OHbZsmVFNnPmzHCsVRAAA0YBAyRRwABJFDBAkoF4H/Drr78e5h999FGPZ9Ib586dC/NWt5L2M+8D/nfqrvXo30bd9uAvv/yyyNavX19kV65caW9yeB8wQL9RwABJFDBAEgUMkEQBAyQZiFUQs2fPDvOjR48WWTsrBU6ePBnmn376aZH99ttvRbZjx46Wz9UOqyA6q1+v63ZcvHgxzO+6664i+/zzz8OxGzZsKLLopeW0zyoIgD6jgAGSKGCAJAoYIMlAvA/4jz/+CPMXX3yxyOq+oPzVV18V2c6dO8Ox0VbMuuNCpy1ZsqTI7rjjjnBs9JA9ethWVVN/4HbnnXcW2dBQXDHRvC5fvjyl89+M3AEDJFHAAEkUMEASBQyQRAEDJBmIrcjtqHtaPNUnwNH24Llz507pmHX2798f5i+88EJXztdLtiL/pe5Lxz/88EORPfXUUy0f95Zbbml57L333ltkq1atCsdGebQVuqqq6vr160X28ccfh2PHx8eLrFtfKe4WW5EB+owCBkiigAGSKGCAJAOxFbkd7TxsGxkZCfO33367yIaHh//1nP7OTz/9VGQrVqzoyrnoL3XXVDsP3L755psiW7lyZTh206ZNRXbPPfe0PK923HbbbUW2Zs2acOylS5eK7J133pnyHPqBO2CAJAoYIIkCBkiigAGSTLudcO04cuRImD/66KMdP9fExESYR+9YnZyc7Pj5+4WdcH+59dZbwzzaCfnss8+2fNxGI/4Vt9oFhw4dCvNjx461PIeXXnqpyOp2qZ4/f77IHn/88XDshQsXWp5DL9kJB9BnFDBAEgUMkEQBAyRRwABJpt1W5DrRE+dou2QnRCsetmzZEo4d5BUP/L26v/1bb71VZAcOHAjHRtf11atXw7HRl8Hff//9Ijtz5kz48+2IVm1EK36qKn4n8UMPPRSO7ddVEHXcAQMkUcAASRQwQBIFDJDEQ7j/id6FOn/+/K6ca86cOUUWfaQQIj///HORLVy4MBwbfYDz2rVr4dhOPFxrVbTtuW4r9OXLl4vs7NmzHZ9TBnfAAEkUMEASBQyQRAEDJFHAAEmm3SqI0dHRMO/Gl4brvsDczkvwoRW//vpr9hRCdf/e7r///paPcfjw4SI7ffr0v55TP3EHDJBEAQMkUcAASRQwQJKBfgg3NjZWZHv27AnHjoyMdPz8b775ZpjfuHGj4+eCfrRz584wnzVrVsvH2Lt3b6em03fcAQMkUcAASRQwQBIFDJBEAQMkGehVEM8880yRdWO1Q1VV1YkTJ4rs4MGDXTlX9KXbqqqqhx9+uCvnO3/+fJFduXKlK+fi5rVhw4Yie/LJJ8Ox0Xb8HTt2hGM/++yzqU2sj7kDBkiigAGSKGCAJAoYIMlAP4TrpeirtFu3bg3HHjlyZErnqtvGuW7duikdt873339fZMuWLQvHTkxMdGUO9I/nnnsuzLds2VJkjUYjHHv16tUie/fdd8Oxk5OTbczu5uIOGCCJAgZIooABkihggCQKGCDJQK+CiLbQRk9fq6qqhoeHO37+pUuXtpVna+d3MzQ00JfOtDRz5swiW7NmTZGNj4+HPx9tL65bwbBx48YiO3PmzD9NceC4AwZIooABkihggCQKGCBJI/qP89rBjUbrg/vUL7/8EuZPP/10j2fSG9GDyKqKt33Wje3lV2mbzWa8d7WLbrbrevHixUV23333hWOjv90bb7wRjl27dm2RLViwoM3Z/b8PPvggzDdv3jyl495s6q5rd8AASRQwQBIFDJBEAQMkUcAASabdftJXXnklzLdt21ZkdS+ernsheq9cv349zA8cOFBkdU+bjx8/3tE50Tvz5s0rsi+++CIce+3atSK7++67w7Gtrog6depUmEdfNf7www9bOuZ05Q4YIIkCBkiigAGSKGCAJNNuK3I76t7bO3v27CJbvXp1kW3fvr3jc6qq+i8P93LLcLfYivzPxsbGiqxui/2MGTOKrO5LxdHXuqNrKnrYVlVVdfbs2TDHVmSAvqOAAZIoYIAkChggiQIGSGIVBH3FKggGkVUQAH1GAQMkUcAASRQwQBIFDJBEAQMkUcAASRQwQBIFDJBEAQMkUcAASRQwQBIFDJBEAQMkUcAASYbaHH+pqqrT3ZgIVFU1knRe1zXdVHtdt/VCdgA6x39BACRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyT5L95QHHzClR5NAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 15032 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pca_analysis(2,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As you can see, 0 and 1 can be clearly separated by a straight line. This indicates that in the original 784-dimensional space dots corresponding to digits are also linearly separable. In the case of 2 and 5, we cannot find the good projection that will separate the digits clearly, and thus there are some cases of wrong classification.\n",
    "\n",
    "> Later on this course we will learn how to create non-linear classifiers using Neural Networks, and how to deal with a problem of digits not being properly aligned. Very soon we will reach above 99% accuracy in MNIST digit classification, while classifying them into 10 different classes.\n",
    "\n",
    "## Takeaway\n",
    "\n",
    " * We have leart about the simplest neural network architecture - one-layer perceptron.\n",
    " * We have implemented the perceptron \"by hand\", using simple training procedure based on gradient descent\n",
    " * Despite simplicity, one-layered perceptron can solve rather complex problems of handwritten digit recognition\n",
    " * One-layered perceptron is a liner classifier, and thus it provides the same classification power as logistic regression.\n",
    " * In the sample space, perceptron can separate two classes of input data using hyperplane."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Credits\n",
    "\n",
    "This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners), and has been prepared by [Dmitry Soshnikov](http://soshnikov.com). It is inspired by Neural Network Workshop at Microsoft Research Cambridge. Some code and illustrative materials are taken from presentations by [Katja Hoffmann](https://www.microsoft.com/en-us/research/people/kahofman/), [Matthew Johnson](https://www.microsoft.com/en-us/research/people/matjoh/) and [Ryoto Tomioka](https://www.microsoft.com/en-us/research/people/ryoto/), and from [NeuroWorkshop](http://github.com/shwars/NeuroWorkshop) repository."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "interpreter": {
   "hash": "16aeaa504b544176258e5caf576fc030dfd6fff62d0c15825e7863ff13e121ff"
  },
  "kernelspec": {
   "display_name": "Python 3.8.0 64-bit (conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "livereveal": {
   "start_slideshow_at": "selected"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
