{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "collapsed": true,
        "id": "4Bta_vJzY-p2"
      },
      "source": [
        "## Object Detection\n",
        "\n",
        "This is a notebooks from [AI for Beginners Curriculum](http://aka.ms/ai-beginners)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "H46ttrmEY-p4"
      },
      "source": [
        "![Image Algorithms](https://cdn-images-1.medium.com/max/840/1*Hz6t-tokG1niaUfmcysusw.jpeg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1RDHP56XY-p6"
      },
      "source": [
        "## Naive Approach to Object Detection\n",
        "\n",
        "* Break image into tiles\n",
        "* Run CNN image classified through each time\n",
        "* Select tiles with activation above the threshold"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "colab_type": "code",
        "id": "jaR_CChGY-qN",
        "outputId": "52b62ed4-6fd9-4e7c-827b-4e6457687e27",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "import cv2\n",
        "from tensorflow import keras\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import os"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's read sample image to play with and pad it to square dimension:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 369
        },
        "colab_type": "code",
        "id": "S0GTVupbY-qa",
        "outputId": "99f192d3-570c-4e5a-d617-6992c5dc9098",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.image.AxesImage at 0x1cb8a16e8e0>"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 77592 characters]",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "img = cv2.imread('images/1200px-Girl_and_cat.jpg')\n",
        "img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)\n",
        "img = np.pad(img,((158,158),(0,0),(0,0)),mode='edge')\n",
        "plt.imshow(img)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will use pre-trained VGG-16 CNN:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "colab_type": "code",
        "id": "0RgUkBkchCxG",
        "outputId": "1d5dc51e-3a19-43e8-8903-b85139f48fe6",
        "trusted": false
      },
      "outputs": [],
      "source": [
        "vgg = keras.applications.vgg16.VGG16(weights='imagenet')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's define a function that will predict the probability of a cat on the image. Since ImageNet contains a number of classes for cats, indexed from 281 to 294, we will just add probabilities for those classes to get overall 'cat' probability:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "colab_type": "code",
        "id": "GzFYd9InhML4",
        "outputId": "329354ce-5759-4236-e16b-d0c716893036",
        "trusted": false
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.61825"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def predict(img):\n",
        "  im = cv2.resize(img,(224,224))\n",
        "  im = keras.applications.vgg16.preprocess_input(im)\n",
        "  pr = vgg.predict(np.expand_dims(im,axis=0))[0]\n",
        "  return np.sum(pr[281:294]) # we know that VGG classes for cats are from 281 to 294\n",
        "\n",
        "predict(img)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Next function will build a heatmap of probabilities, dividing the image into $n\\times n$ squares:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 365
        },
        "colab_type": "code",
        "id": "z5qqyj5uh6GN",
        "outputId": "cf01a729-87e7-4b6e-8a43-89a61247df20",
        "trusted": false
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.image.AxesImage at 0x1cb8983a940>"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 120684 characters]",
            "text/plain": [
              "<Figure size 1080x360 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "def predict_map(img,n):\n",
        "  dx = img.shape[0] // n\n",
        "  res = np.zeros((n,n),dtype=np.float32)\n",
        "  for i in range(n):\n",
        "    for j in range(n):\n",
        "      im = img[dx*i:dx*(i+1),dx*j:dx*(j+1)]\n",
        "      r = predict(im)\n",
        "      res[i,j] = r\n",
        "  return res\n",
        "\n",
        "fig,ax = plt.subplots(1,2,figsize=(15,5))\n",
        "ax[1].imshow(img)\n",
        "ax[0].imshow(predict_map(img,10))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Detecting Simple Objects\n",
        "\n",
        "To give more precise location of a bounding box, we need to run **regression model** to predict bounding box coordinates. Let's start with simple example of having black rectangles in 32x32 images, which we want to detect. The idea and some code are borrowed from [this blog post](https://towardsdatascience.com/object-detection-with-neural-networks-a4e2c46b4491).\n",
        "\n",
        "The following function will generate a bunch of sample images:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "KJSwfU3BGobM",
        "trusted": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Images shape = (100000, 8, 8)\n",
            "BBoxes shape = (100000, 4)\n"
          ]
        }
      ],
      "source": [
        "def generate_images(num_imgs, img_size=8, min_object_size = 1, max_object_size = 4):\n",
        "    bboxes = np.zeros((num_imgs, 4))\n",
        "    imgs = np.zeros((num_imgs, img_size, img_size))  # set background to 0\n",
        "\n",
        "    for i_img in range(num_imgs):\n",
        "        w, h = np.random.randint(min_object_size, max_object_size, size=2)\n",
        "        x = np.random.randint(0, img_size - w)\n",
        "        y = np.random.randint(0, img_size - h)\n",
        "        imgs[i_img, x:x+w, y:y+h] = 1.  # set rectangle to 1\n",
        "        bboxes[i_img] = [x, y, w, h]\n",
        "    return imgs, bboxes\n",
        "\n",
        "imgs, bboxes = generate_images(100000)\n",
        "print(f\"Images shape = {imgs.shape}\")\n",
        "print(f\"BBoxes shape = {bboxes.shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To make outputs of the network in the range [0;1], we will divide `bboxes` by the image size:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0.   , 0.25 , 0.125, 0.25 ])"
            ]
          },
          "execution_count": 114,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "bb = bboxes/8.0\n",
        "bb[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In our simple example, we will use dense neural network. In real life, when objects have more complex shape, it definitely makes sense to use CNNs for the task like this. We will use stochastic gradient descent optimizer and mean squared error (MSE) as the metrics, because our task is **regression**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 144,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_25\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " flatten_24 (Flatten)        (None, 64)                0         \n",
            "                                                                 \n",
            " dense_50 (Dense)            (None, 200)               13000     \n",
            "                                                                 \n",
            " dropout_3 (Dropout)         (None, 200)               0         \n",
            "                                                                 \n",
            " dense_51 (Dense)            (None, 4)                 804       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 13,804\n",
            "Trainable params: 13,804\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    keras.layers.Flatten(input_shape=(8,8)),\n",
        "    keras.layers.Dense(200, activation='relu'),\n",
        "    keras.layers.Dropout(0.2),\n",
        "    keras.layers.Dense(4)\n",
        "])\n",
        "model.compile('sgd','mse')\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's train our network. We will also normalize the input data (by subtracting mean and dividing by standard deviation) for slightly better performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 145,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/30\n",
            "3125/3125 [==============================] - 6s 2ms/step - loss: 0.0562\n",
            "Epoch 2/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0131\n",
            "Epoch 3/30\n",
            "3125/3125 [==============================] - 5s 1ms/step - loss: 0.0076\n",
            "Epoch 4/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0058\n",
            "Epoch 5/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0050\n",
            "Epoch 6/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0044\n",
            "Epoch 7/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0041\n",
            "Epoch 8/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0038\n",
            "Epoch 9/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0036\n",
            "Epoch 10/30\n",
            "3125/3125 [==============================] - 6s 2ms/step - loss: 0.0034\n",
            "Epoch 11/30\n",
            "3125/3125 [==============================] - 8s 3ms/step - loss: 0.0033\n",
            "Epoch 12/30\n",
            "3125/3125 [==============================] - 8s 3ms/step - loss: 0.0031\n",
            "Epoch 13/30\n",
            "3125/3125 [==============================] - 6s 2ms/step - loss: 0.0030\n",
            "Epoch 14/30\n",
            "3125/3125 [==============================] - 6s 2ms/step - loss: 0.0029\n",
            "Epoch 15/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0028\n",
            "Epoch 16/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0028\n",
            "Epoch 17/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0027\n",
            "Epoch 18/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0026\n",
            "Epoch 19/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0026\n",
            "Epoch 20/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0025\n",
            "Epoch 21/30\n",
            "3125/3125 [==============================] - 6s 2ms/step - loss: 0.0024\n",
            "Epoch 22/30\n",
            "3125/3125 [==============================] - 6s 2ms/step - loss: 0.0024\n",
            "Epoch 23/30\n",
            "3125/3125 [==============================] - 6s 2ms/step - loss: 0.0024\n",
            "Epoch 24/30\n",
            "3125/3125 [==============================] - 6s 2ms/step - loss: 0.0023\n",
            "Epoch 25/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0023\n",
            "Epoch 26/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0022\n",
            "Epoch 27/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0022\n",
            "Epoch 28/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0022\n",
            "Epoch 29/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0021\n",
            "Epoch 30/30\n",
            "3125/3125 [==============================] - 5s 2ms/step - loss: 0.0021\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x1cbb7c499a0>"
            ]
          },
          "execution_count": 145,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "imgs_norm = (imgs-np.mean(imgs))/np.std(imgs)\n",
        "model.fit(imgs_norm,bb,epochs=30)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We seem to have relatively good loss, let's see how it translates into more tangible metrics, such as mAP. First, let's define IOU metric between two bounding boxes:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 146,
      "metadata": {},
      "outputs": [],
      "source": [
        "def IOU(bbox1, bbox2):\n",
        "    '''Calculate overlap between two bounding boxes [x, y, w, h] as the area of intersection over the area of unity'''\n",
        "    x1, y1, w1, h1 = bbox1[0], bbox1[1], bbox1[2], bbox1[3]\n",
        "    x2, y2, w2, h2 = bbox2[0], bbox2[1], bbox2[2], bbox2[3]\n",
        "    w_I = min(x1 + w1, x2 + w2) - max(x1, x2)\n",
        "    h_I = min(y1 + h1, y2 + h2) - max(y1, y2)\n",
        "    if w_I <= 0 or h_I <= 0:  # no overlap\n",
        "        return 0.\n",
        "    I = w_I * h_I\n",
        "    U = w1 * h1 + w2 * h2 - I\n",
        "    return I / U"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will now generate 500 test images, and plot first 5 of them to visualize how accurate we are. We will print out IOU metric as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 149,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "pred=[3.7325673 3.6551285 2.0126944 1.030895 ],act=[4. 4. 2. 1.], IOU=0.41607480412565545\n",
            "pred=[2.3762555 3.755858  1.1647941 1.0540264],act=[2. 4. 1. 1.], IOU=0.2932611042051458\n",
            "pred=[-0.04900682 -0.10628867  2.7881489   1.027148  ],act=[0. 0. 3. 1.], IOU=0.7548650953478908\n",
            "pred=[0.96806276 4.267275   1.3179774  1.1021365 ],act=[0. 5. 1. 1.], IOU=0.004833667961256898\n",
            "pred=[2.157959   0.94667876 2.7568097  0.96259   ],act=[2. 1. 3. 1.], IOU=0.7965311752734529\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 5112 characters]",
            "text/plain": [
              "<Figure size 1080x360 with 5 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib\n",
        "\n",
        "test_imgs, test_bboxes = generate_images(500)\n",
        "bb_res = model.predict((test_imgs-np.mean(imgs))/np.std(imgs))*8\n",
        "\n",
        "plt.figure(figsize=(15,5))\n",
        "for i in range(5):\n",
        "    print(f\"pred={bb_res[i]},act={test_bboxes[i]}, IOU={IOU(bb_res[i],test_bboxes[i])}\")\n",
        "    plt.subplot(1,5,i+1)\n",
        "    plt.imshow(test_imgs[i])\n",
        "    plt.gca().add_patch(matplotlib.patches.Rectangle((bb_res[i,1],bb_res[i,0]),bb_res[i,3],bb_res[i,2],ec='r'))\n",
        "    #plt.annotate('IOU: {:.2f}'.format(IOU(bb_res[i],test_bboxes[i])),(bb_res[i,1],bb_res[i,0]+bb_res[i,3]),color='y')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now to calculate mean precision over all cases, we just need to go over all our test samples, compute IoU, and calculate mean:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 151,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.7150587956049862"
            ]
          },
          "execution_count": 151,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "np.array([IOU(a,b) for a,b in zip(test_bboxes,bb_res)]).mean()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Real-Life Object Detection\n",
        "\n",
        "Real object detection algorithms are more complicated. We will recommend you to follow [Keras tutorial on Object Detection with RetinaNet](https://keras.io/examples/vision/retinanet/) if you want to get into the details of RetinaNet implementation, or use [Keras RetinaNet Library](https://github.com/fizyr/keras-retinanet), if you just want to train object detection model."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "ObjectDetection.ipynb",
      "provenance": [],
      "version": "0.3.2"
    },
    "interpreter": {
      "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
    },
    "kernelspec": {
      "display_name": "Python 3.8.12 ('py38')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}
