microsoft/AI-For-Beginners
Publicmirrored fromhttps://github.com/microsoft/AI-For-BeginnersAvailable
lessons/4-ComputerVision/06-IntroCV/README.md
109lines · modecode
| 1 | # Introduction to Computer Vision |
| 2 | |
| 3 | [Computer Vision](https://wikipedia.org/wiki/Computer_vision) is a discipline whose aim is to allow computers to gain high-level understanding of digital images. This is quite a broad definition, because *understanding* can mean many different things, including finding an object on a picture (**object detection**), understanding what is happening (**event detection**), describing a picture in text, or reconstructing a scene in 3D. There are also special tasks related to human images: age and emotion estimation, face detection and identification, and 3D pose estimation, to name a few. |
| 4 | |
| 5 | ## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/11) |
| 6 | |
| 7 | One of the simplest tasks of computer vision is **image classification**. |
| 8 | |
| 9 | Computer vision is often considered to be a branch of AI. Nowadays, most of computer vision tasks are solved using neural networks. We will learn more about the special type of neural networks used for computer vision, [convolutional neural networks](../07-ConvNets/README.md), throughout this section. |
| 10 | |
| 11 | However, before you pass the image to a neural network, in many cases it makes sense to use some algorithmic techniques to enhance the image. |
| 12 | |
| 13 | There are several Python libraries available for image processing: |
| 14 | |
| 15 | * **[imageio](https://imageio.readthedocs.io/en/stable/)** can be used for reading/writing different image formats. It also support ffmpeg, a useful tool to convert video frames to images. |
| 16 | * **[Pillow](https://pillow.readthedocs.io/en/stable/index.html)** (also known as PIL) is a bit more powerful, and also supports some image manipulation such as morphing, palette adjustments, and more. |
| 17 | * **[OpenCV](https://opencv.org/)** is a powerful image processing library written in C++, which has become the *de facto* standard for image processing. It has a convenient Python interface. |
| 18 | * **[dlib](http://dlib.net/)** is a C++ library that implements many machine learning algorithms, including some of the Computer Vision algorithms. It also has a Python interface, and can be used for challenging tasks such as face and facial landmark detection. |
| 19 | |
| 20 | ## OpenCV |
| 21 | |
| 22 | [OpenCV](https://opencv.org/) is considered to be the *de facto* standard for image processing. It contains a lot of useful algorithms, implemented in C++. You can call OpenCV from Python as well. |
| 23 | |
| 24 | A good place to learn OpenCV is [this Learn OpenCV course](https://learnopencv.com/getting-started-with-opencv/). In our curriculum, our goal is not to learn OpenCV, but to show you some examples when it can be used, and how. |
| 25 | |
| 26 | ### Loading Images |
| 27 | |
| 28 | Images in Python can be conveniently represented by NumPy arrays. For example, grayscale images with the size of 320x200 pixels would be stored in a 200x320 array, and color images of the same dimension would have shape of 200x320x3 (for 3 color channels). To load an image, you can use the following code: |
| 29 | |
| 30 | ```python |
| 31 | import cv2 |
| 32 | import matplotlib.pyplot as plt |
| 33 | |
| 34 | im = cv2.imread('image.jpeg') |
| 35 | plt.imshow(im) |
| 36 | ``` |
| 37 | |
| 38 | Traditionally, OpenCV uses BGR (Blue-Green-Red) encoding for color images, while the rest of Python tools use the more traditional RGB (Red-Green-Blue). For the image to look right, you need to convert it to the RGB color space, either by swapping dimensions in the NumPy array, or by calling an OpenCV function: |
| 39 | |
| 40 | ```python |
| 41 | im = cv2.cvtColor(im,cv2.COLOR_BGR2RGB) |
| 42 | ``` |
| 43 | |
| 44 | The same `cvtColor` function can be used to perform other color space transformations such as converting an image to grayscale or to the HSV (Hue-Saturation-Value) color space. |
| 45 | |
| 46 | You can also use OpenCV to load video frame-by-frame - an example is given in the exercise [OpenCV Notebook](OpenCV.ipynb). |
| 47 | |
| 48 | ### Image Processing |
| 49 | |
| 50 | Before feeding an image to a neural network, you may want to apply several pre-processing steps. OpenCV can do many things, including: |
| 51 | |
| 52 | * **Resizing** the image using `im = cv2.resize(im, (320,200),interpolation=cv2.INTER_LANCZOS)` |
| 53 | * **Blurring** the image using `im = cv2.medianBlur(im,3)` or `im = cv2.GaussianBlur(im, (3,3), 0)` |
| 54 | * Changing the **brightness and contrast** of the image can be done by NumPy array manipulations, as described [in this Stackoverflow note](https://stackoverflow.com/questions/39308030/how-do-i-increase-the-contrast-of-an-image-in-python-opencv). |
| 55 | * Using [thresholding](https://docs.opencv.org/4.x/d7/d4d/tutorial_py_thresholding.html) by calling `cv2.threshold`/`cv2.adaptiveThreshold` functions, which is often preferable to adjusting brightness or contrast. |
| 56 | * Applying different [transformations](https://docs.opencv.org/4.5.5/da/d6e/tutorial_py_geometric_transformations.html) to the image: |
| 57 | - **[Affine transformations](https://docs.opencv.org/4.5.5/d4/d61/tutorial_warp_affine.html)** can be useful if you need to combine rotation, resizing and skewing to the image and you know the source and destination location of three points in the image. Affine transformations keep parallel lines parallel. |
| 58 | - **[Perspective transformations](https://medium.com/analytics-vidhya/opencv-perspective-transformation-9edffefb2143)** can be useful when you know the source and destination positions of 4 points in the image. For example, if you take a picture of a rectangular document via a smartphone camera from some angle, and you want to make a rectangular image of the document itself. |
| 59 | * Understanding movement inside the image by using **[optical flow](https://docs.opencv.org/4.5.5/d4/dee/tutorial_optical_flow.html)**. |
| 60 | |
| 61 | ## Examples of using Computer Vision |
| 62 | |
| 63 | In our [OpenCV Notebook](OpenCV.ipynb), we give some examples of when computer vision can be used to perform specific tasks: |
| 64 | |
| 65 | * **Pre-processing a photograph of a Braille book**. We focus on how we can use thresholding, feature detection, perspective transformation and NumPy manipulations to separate individual Braille symbols for further classification by a neural network. |
| 66 | |
| 67 |  |  |  |
| 68 | ----|-----|----- |
| 69 | |
| 70 | > Image from [OpenCV.ipynb](OpenCV.ipynb) |
| 71 | |
| 72 | * **Detecting motion in video using frame difference**. If the camera is fixed, then frames from the camera feed should be pretty similar to each other. Since frames are represented as arrays, just by subtracting those arrays for two subsequent frames we will get the pixel difference, which should be low for static frames, and become higher once there is substantial motion in the image. |
| 73 | |
| 74 |  |
| 75 | |
| 76 | > Image from [OpenCV.ipynb](OpenCV.ipynb) |
| 77 | |
| 78 | * **Detecting motion using Optical Flow**. [Optical flow](https://docs.opencv.org/3.4/d4/dee/tutorial_optical_flow.html) allows us to understand how individual pixels on video frames move. There are two types of optical flow: |
| 79 | |
| 80 | - **Dense Optical Flow** computes the vector field that shows for each pixel where is it moving |
| 81 | - **Sparse Optical Flow** is based on taking some distinctive features in the image (eg. edges), and building their trajectory from frame to frame. |
| 82 | |
| 83 |  |
| 84 | |
| 85 | > Image from [OpenCV.ipynb](OpenCV.ipynb) |
| 86 | |
| 87 | ## ✍️ Example Notebooks: OpenCV [try OpenCV in Action](OpenCV.ipynb) |
| 88 | |
| 89 | Let's do some experiments with OpenCV by exploring [OpenCV Notebook](OpenCV.ipynb) |
| 90 | |
| 91 | ## Conclusion |
| 92 | |
| 93 | Sometimes, relatively complex tasks such as movement detection or fingertip detection can be solved purely by computer vision. Thus, it is very helpful to know the basic techniques of computer vision, and what libraries like OpenCV can do. |
| 94 | |
| 95 | ## 🚀 Challenge |
| 96 | |
| 97 | Watch [this video](https://docs.microsoft.com/shows/ai-show/ai-show--2021-opencv-ai-competition--grand-prize-winners--cortic-tigers--episode-32?WT.mc_id=academic-77998-cacaste) from the AI show to learn about the Cortic Tigers project and how they built a block-based solution to democratize computer vision tasks via a robot. Do some research on other projects like this that help onboard new learners into the field. |
| 98 | |
| 99 | ## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/12) |
| 100 | |
| 101 | ## Review & Self Study |
| 102 | |
| 103 | Read more on optical flow [in this great tutorial](https://learnopencv.com/optical-flow-in-opencv/). |
| 104 | |
| 105 | ## [Assignment](lab/README.md) |
| 106 | |
| 107 | In this lab, you will take a video with simple gestures, and your goal is to extract up/down/left/right movements using optical flow. |
| 108 | |
| 109 | <img src="images/palm-movement.png" width="30%" alt="Palm Movement Frame"/> |