{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computer Vision and OpenCV\n",
    "\n",
    "This notebook is a part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners).\n",
    "\n",
    "[OpenCV](https://opencv.org/) is considered to be *de facto* standard for image processing. It contains a lot of useful algorithms, implemented in C++. You can call OpenCV from Python as well.\n",
    "\n",
    "In this notebooks, we will give you some examples for using OpenCV. For more details, you can visit [Learn OpenCV](https://learnopencv.com/getting-started-with-opencv/) online course. \n",
    "\n",
    "First, let's `import cv2`, as well as some other useful libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def display_images(l,titles=None,fontsize=12):\n",
    "    n=len(l)\n",
    "    fig,ax = plt.subplots(1,n)\n",
    "    for i,im in enumerate(l):\n",
    "        ax[i].imshow(im)\n",
    "        ax[i].axis('off')\n",
    "        if titles is not None:\n",
    "            ax[i].set_title(titles[i],fontsize=fontsize)\n",
    "    fig.set_size_inches(fig.get_size_inches()*n)\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Images\n",
    "\n",
    "Images in Python can be conveniently represented by NumPy arrays. For example, grayscale image with size of 320x200 pixels would be stored in 200x320 array, and color image of the same dimension would have shape of 200x320x3 (for 3 color channels). \n",
    "\n",
    "Let's start by loading an image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(242, 531, 3)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x19a755cc640>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 79268 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "im = cv2.imread('data/braille.jpeg')\n",
    "print(im.shape)\n",
    "plt.imshow(im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, it is an image of braille text. Since we are not very interested in the actual color, we can convert it to black-and-white:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(242, 531)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1de4dfef6a0>"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 103856 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bw_im = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)\n",
    "print(bw_im.shape)\n",
    "plt.imshow(bw_im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Braille Image Processing\n",
    "\n",
    "If we want to apply image classification to recognize the text, we need to cut out individual symbols to make them similar to MNIST images that we have seen before. This can be done using [object detection](../11-ObjectDetection/README.md) technique which we will discuss later, but also we can try to use pure computer vision for that. A good description of how computer vision can be used for character separation can be found [in this blog post](https://learnopencv.com/image-alignment-feature-based-using-opencv-c-python/) - we will only focus on some computer vision techniques here.\n",
    "\n",
    "First, let's try to enhance the image a little bit. We can use the idea of **thresholding** (well described [in this OpenCV article](https://docs.opencv.org/4.x/d7/d4d/tutorial_py_thresholding.html)): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1de4e26d9a0>"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 27280 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "im = cv2.blur(bw_im,(3,3))\n",
    "im = cv2.adaptiveThreshold(im, 255, cv2.ADAPTIVE_THRESH_MEAN_C,\n",
    "                           cv2.THRESH_BINARY_INV, 5, 4)\n",
    "im = cv2.medianBlur(im, 3)\n",
    "_,im = cv2.threshold(im, 0, 255, cv2.THRESH_OTSU)\n",
    "im = cv2.GaussianBlur(im, (3,3), 0)\n",
    "_,im = cv2.threshold(im, 0, 255, cv2.THRESH_OTSU)\n",
    "plt.imshow(im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To work with images, we need to \"extract\" individual dots, i.e. convert the images to a set of coordinates of individual dots. We can do that using **feature extraction** techniques, such as SIFT, SURF or [ORB](https://docs.opencv.org/4.x/d1/d89/tutorial_py_orb.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 5 points: [(307.20001220703125, 40.80000305175781), (297.6000061035156, 114.00000762939453), (423.6000061035156, 133.20001220703125), (242.40000915527344, 144.0), (103.68000793457031, 57.60000228881836)]\n"
     ]
    }
   ],
   "source": [
    "orb = cv2.ORB_create(5000)\n",
    "f,d = orb.detectAndCompute(im,None)\n",
    "print(f\"First 5 points: { [f[i].pt for i in range(5)]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot all points to make sure we got things right:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29940 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_dots(dots):\n",
    "    img = np.zeros((250,500))\n",
    "    for x in dots:\n",
    "        cv2.circle(img,(int(x[0]),int(x[1])),3,(255,0,0))\n",
    "    plt.imshow(img)\n",
    "\n",
    "pts = [x.pt for x in f]\n",
    "plot_dots(pts)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To separate individual characters, we need to know the bounding box of the whole text. To find it out, we can just compute min and max coordinates: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1de74b0fac0>"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 17968 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "min_x, min_y, max_x, max_y = [int(f([z[i] for z in pts])) for f in (min,max) for i in (0,1)]\n",
    "min_y+=13\n",
    "plt.imshow(im[min_y:max_y,min_x:max_x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, this text can be partially rotated, and to make it perfectly aligned we need to do so-called **perspective transformation**. We will take a rectangle defined by points $(x_{min},y_{min}), (x_{min},y_{max}), (x_{max},y_{min}), (x_{max},y_{max})$ and align it with new image with proportional dimensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1de74cf0f70>"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 18000 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "off = 5\n",
    "src_pts = np.array([(min_x-off,min_y-off),(min_x-off,max_y+off),\n",
    "                    (max_x+off,min_y-off),(max_x+off,max_y+off)])\n",
    "w = int(max_x-min_x+off*2)\n",
    "h = int(max_y-min_y+off*2)\n",
    "dst_pts = np.array([(0,0),(0,h),(w,0),(w,h)])\n",
    "ho,m = cv2.findHomography(src_pts,dst_pts)\n",
    "trim = cv2.warpPerspective(im,ho,(w,h))\n",
    "plt.imshow(trim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we get this well-aligned image, it should be relatively easy to slice it into pieces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 80580 characters]",
      "text/plain": [
       "<Figure size 21600x14400 with 50 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "char_h = 36\n",
    "char_w = 24\n",
    "def slice(img):\n",
    "    dy,dx = img.shape\n",
    "    y = 0\n",
    "    while y+char_h<dy:\n",
    "        x=0\n",
    "        while x+char_w<dx:\n",
    "            # Skip empty lines\n",
    "            if np.max(img[y:y+char_h,x:x+char_w])>0:\n",
    "                yield img[y:y+char_h,x:x+char_w]\n",
    "            x+=char_w\n",
    "        y+=char_h\n",
    "\n",
    "sliced = list(slice(trim))\n",
    "display_images(sliced)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have seen that quite a lot of tasks can be done using pure image processing, without any artificial intelligence. If we can use computer vision techniques to make the work of a neural network simpler - we should definitely do it, because it will allow us to solve problems with smaller number of training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Motion Detection using Frame Difference\n",
    "\n",
    "Detecting motion on video stream is a very frequent task. For example, it allows us to get alerts when something happens on a surveillance camera. If we want to understand what's happening on the camera, we can then use a neural network - but it is much cheaper to use neural network when we know that something is going on.\n",
    "\n",
    "The main idea of motion detection is simple. If the camera is fixed, then frames from the camera should be pretty similar to each other. Since frames are represented as arrays, just by subtracting those arrays for two subsequent frames we will get the pixel difference, which should be low for static frames, and become higher once there is substantial motion in the image.\n",
    "\n",
    "We will start by learning how to open a video and convert it into a sequence of frames:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total frames: 876\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1046632 characters]",
      "text/plain": [
       "<Figure size 2592x1728 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "vid = cv2.VideoCapture('data/motionvideo.mp4')\n",
    "\n",
    "c = 0\n",
    "frames = []\n",
    "while vid.isOpened():\n",
    "    ret, frame = vid.read()\n",
    "    if not ret:\n",
    "        break\n",
    "    frames.append(frame)\n",
    "    c+=1\n",
    "vid.release()\n",
    "print(f\"Total frames: {c}\")\n",
    "display_images(frames[::150])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since color is not so important for motion detection, we will convert all frames to grayscale. Then, we will compute the frame differences, and plot their norms to visually see the amount of activity going on: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24100 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 196956 characters]",
      "text/plain": [
       "<Figure size 2592x1728 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bwframes = [cv2.cvtColor(x,cv2.COLOR_BGR2GRAY) for x in frames]\n",
    "diffs = [(p2-p1) for p1,p2 in zip(bwframes[:-1],bwframes[1:])]\n",
    "diff_amps = np.array([np.linalg.norm(x) for x in diffs])\n",
    "plt.plot(diff_amps)\n",
    "display_images(diffs[::150],titles=diff_amps[::150])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want to create a report that shows what happened in front of the camera by showing the suitable image each time something happens. To do it, we probably want to find out the start and end frame of an \"event\", and display the middle frame. To remove some noise, we will also smooth out the curve above with moving average function: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.lines.Line2D at 0x1de72d816d0>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 26728 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def moving_average(x, w):\n",
    "    return np.convolve(x, np.ones(w), 'valid') / w\n",
    "\n",
    "threshold = 13000\n",
    "\n",
    "plt.plot(moving_average(diff_amps,10))\n",
    "plt.axhline(y=threshold, color='r', linestyle='-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can find out frames that have the amount of changes above the threshold by using `np.where`, and extract a sequence of consecutive frames that is longer than 30 frames: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322]\n"
     ]
    }
   ],
   "source": [
    "active_frames = np.where(diff_amps>threshold)[0]\n",
    "\n",
    "def subsequence(seq,min_length=30):\n",
    "    ss = []\n",
    "    for i,x in enumerate(seq[:-1]):\n",
    "        ss.append(x)\n",
    "        if x+1 != seq[i+1]:\n",
    "            if len(ss)>min_length:\n",
    "                return ss\n",
    "            ss.clear()\n",
    "\n",
    "sub = subsequence(active_frames)\n",
    "print(sub)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally we can display the image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1de74da5b80>"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 140432 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(frames[(sub[0]+sub[-1])//2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may notice that color scheme on this image does not look right! This is because OpenCV for historical reasons loads images in BGR color space, while matplotlib uses more traditional RGB color order. Most of the time, it makes sense to convert images to RGB immediately after loading them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1dec94969d0>"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 140364 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(cv2.cvtColor(frames[(sub[0]+sub[-1])//2],cv2.COLOR_BGR2RGB))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract Motion using Optical Flow\n",
    "\n",
    "While just comparing two consecutive frames allows us to see the amount of changes, it does not give any information on what is actually moving and where. To get that information, there is a technique called **[optical flow](https://docs.opencv.org/3.4/d4/dee/tutorial_optical_flow.html)**:\n",
    "\n",
    "* **Dense Optical Flow** computes the vector field that shows for each pixel where is it moving\n",
    "* **Sparse Optical Flow** is based on taking some distinctive features in the image (eg. edges), and building their trajectory from frame to frame.\n",
    "\n",
    "Read more on optical flow [in this great tutorial](https://learnopencv.com/optical-flow-in-opencv/).\n",
    "\n",
    "Let's compute dense optical flow between our frames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(180, 320, 2)"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flows = [cv2.calcOpticalFlowFarneback(f1, f2, None, 0.5, 3, 15, 3, 5, 1.2, 0) \n",
    "         for f1,f2 in zip(bwframes[:-1],bwframes[1:])]\n",
    "flows[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, for each frame the flow has the dimension of the frame, and 2 channels, corresponding to x and y components of optical flow vector.\n",
    "\n",
    "Displaying optical flow in 2D is a bit challenging, but we can use one clever idea. If we convert optical flow to polar coordinates, then we will get two components for each pixel: *direction* and *intensity*. We can represent intensity by the pixel intensity, and direction by different colors. We will create an image in [HSV (Hue-Saturation-Value) color space](https://en.wikipedia.org/wiki/HSV_color_space), where hue will be defined by direction, value - by intensity, and saturation will be 255. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "195 322\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 207184 characters]",
      "text/plain": [
       "<Figure size 2592x1728 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def flow_to_hsv(flow):\n",
    "    hsvImg = np.zeros((flow.shape[0],flow.shape[1],3),dtype=np.uint8)\n",
    "    mag, ang = cv2.cartToPolar(flow[..., 0], flow[..., 1])\n",
    "    hsvImg[..., 0] = 0.5 * ang * 180 / np.pi\n",
    "    hsvImg[..., 1] = 255\n",
    "    hsvImg[..., 2] = cv2.normalize(mag, None, 0, 255, cv2.NORM_MINMAX)\n",
    "    return cv2.cvtColor(hsvImg, cv2.COLOR_HSV2BGR)\n",
    "\n",
    "start = sub[0]\n",
    "stop = sub[-1]\n",
    "print(start,stop)\n",
    "\n",
    "frms = [flow_to_hsv(x) for x in flows[start:stop]]\n",
    "display_images(frms[::25])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, in those frames greenish color corresponds to moving to the left, while blue - moving to the right.\n",
    "\n",
    "Optical flow can be a great tool to draw conclusions about general direction of movement. For example, if you see that all pixels in a frame are moving in more or less one direction - you can conclude that there is camera movement, and try to compensate for that."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 ('py38')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
