{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Multi-Class Classification with Perceptron\n",
    "\n",
    "Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pickle\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the following perceptron training code from the lecture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def train(positive_examples, negative_examples, num_iterations = 100):\n",
    "    num_dims = positive_examples.shape[1]\n",
    "    weights = np.zeros((num_dims,1)) # initialize weights\n",
    "    \n",
    "    pos_count = positive_examples.shape[0]\n",
    "    neg_count = negative_examples.shape[0]\n",
    "    \n",
    "    report_frequency = 10\n",
    "    \n",
    "    for i in range(num_iterations):\n",
    "        pos = random.choice(positive_examples)\n",
    "        neg = random.choice(negative_examples)\n",
    "\n",
    "        z = np.dot(pos, weights)   \n",
    "        if z < 0:\n",
    "            weights = weights + pos.reshape(weights.shape)\n",
    "\n",
    "        z  = np.dot(neg, weights)\n",
    "        if z >= 0:\n",
    "            weights = weights - neg.reshape(weights.shape)\n",
    "            \n",
    "        if i % report_frequency == 0:             \n",
    "            pos_out = np.dot(positive_examples, weights)\n",
    "            neg_out = np.dot(negative_examples, weights)        \n",
    "            pos_correct = (pos_out >= 0).sum() / float(pos_count)\n",
    "            neg_correct = (neg_out < 0).sum() / float(neg_count)\n",
    "            print(\"Iteration={}, pos correct={}, neg correct={}\".format(i,pos_correct,neg_correct))\n",
    "\n",
    "    return weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def accuracy(weights, test_x, test_labels):\n",
    "    res = np.dot(np.c_[test_x,np.ones(len(test_x))],weights)\n",
    "    return (res.reshape(test_labels.shape)*test_labels>=0).sum()/float(len(test_labels))\n",
    "\n",
    "accuracy(wts, test_x, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading the Dataset\n",
    "\n",
    "This code download the dataset from the repository on the internet. You can also manually copy the dataset from `/data` directory of AI Curriculum repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "!rm *.pkl\n",
    "https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/data/mnist.pkl.gz",
    "!gzip -d mnist.pkl.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('mnist.pkl', 'rb') as mnist_pickle:\n",
    "    MNIST = pickle.load(mnist_pickle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  0   0 188 255  94   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0 191 250 253  93   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0]\n",
      "1\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 25144 characters]",
      "text/plain": [
       "<Figure size 720x360 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(MNIST['Train']['Features'][0][130:180])\n",
    "print(MNIST['Train']['Labels'][0])\n",
    "features = MNIST['Train']['Features'].astype(np.float32) / 256.0\n",
    "labels = MNIST['Train']['Labels']\n",
    "fig = plt.figure(figsize=(10,5))\n",
    "for i in range(10):\n",
    "    ax = fig.add_subplot(1,10,i+1)\n",
    "    plt.imshow(features[i].reshape(28,28))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Code to create *one-vs-other* dataset for two-digit classification. You need to modify this code to create *one-vs-all* dateset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def set_mnist_pos_neg(positive_label, negative_label):\n",
    "    positive_indices = [i for i, j in enumerate(MNIST['Train']['Labels']) \n",
    "                          if j == positive_label]\n",
    "    negative_indices = [i for i, j in enumerate(MNIST['Train']['Labels']) \n",
    "                          if j == negative_label]\n",
    "\n",
    "    positive_images = MNIST['Train']['Features'][positive_indices]\n",
    "    negative_images = MNIST['Train']['Features'][negative_indices]\n",
    "\n",
    "    return positive_images, negative_images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you need to:\n",
    "1. Create 10 *one-vs-all* datasets for all digits\n",
    "1. Train 10 perceptrons\n",
    "1. Define `classify` function to perform digit classification\n",
    "1. Measure the accuracy of classification and print *confusion matrix*\n",
    "1. [Optional] Create improved `classify` function that performs the classification using one matrix multiplication."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "livereveal": {
   "start_slideshow_at": "selected"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
