microsoft/AI-For-Beginners
Publicmirrored fromhttps://github.com/microsoft/AI-For-BeginnersAvailable
lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb
1737lines · modecode
| 1 | { |
| 2 | "cells": [ |
| 3 | { |
| 4 | "cell_type": "markdown", |
| 5 | "metadata": { |
| 6 | "id": "En2vX4FuwHlu" |
| 7 | }, |
| 8 | "source": [ |
| 9 | "## Introduction to PyTorch\n", |
| 10 | "\n", |
| 11 | "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n", |
| 12 | "\n", |
| 13 | "### Neural Frameworks\n", |
| 14 | "\n", |
| 15 | "We have learnt that to train neural networks you need:\n", |
| 16 | "* Quickly multiply matrices (tensors)\n", |
| 17 | "* Compute gradients to perform gradient descent optimization\n", |
| 18 | "\n", |
| 19 | "What neural network frameworks allow you to do:\n", |
| 20 | "* Operate with tensors on whatever compute is available, CPU or GPU, or even TPU\n", |
| 21 | "* Automatically compute gradients (they are explicitly programmed for all built-in tensor functions)\n", |
| 22 | "\n", |
| 23 | "Optionally:\n", |
| 24 | "* Neural Network constructor / higher level API (describe network as a sequence of layers)\n", |
| 25 | "* Simple training functions (`fit`, as in Scikit Learn)\n", |
| 26 | "* A number of optimization algorithms in addition to gradient descent\n", |
| 27 | "* Data handling abstractions (that will ideally work on GPU, too)" |
| 28 | ] |
| 29 | }, |
| 30 | { |
| 31 | "cell_type": "markdown", |
| 32 | "metadata": { |
| 33 | "id": "8cACQoFMwHl3" |
| 34 | }, |
| 35 | "source": [ |
| 36 | "### Most Popular Frameworks\n", |
| 37 | "\n", |
| 38 | "* Tensorflow 1.x - first widely available framework (Google). Allowed to define static computation graph, push it to GPU, and explicitly evaluate it\n", |
| 39 | "* PyTorch - a framework from Facebook that is growing in popularity\n", |
| 40 | "* Keras - higher level API on top of Tensorflow/PyTorch to unify and simplify using neural networks (Francois Chollet)\n", |
| 41 | "* Tensorflow 2.x + Keras - new version of Tensorflow with integrated Keras functionality, which supports **dynamic computation graph**, allowing to perform tensor operations very similar to numpy (and PyTorch)\n", |
| 42 | "\n", |
| 43 | "In this Notebook, we will learn to use PyTorch. You need to make sure that you have recent version of PyTorch installed - to do it, follow the [instructions on their site](https://pytorch.org/get-started/locally/). It is normally as simple as doing\n", |
| 44 | "```\n", |
| 45 | "pip install torch torchvision\n", |
| 46 | "```\n", |
| 47 | "or\n", |
| 48 | "```\n", |
| 49 | "conda install pytorch -c pytorch\n", |
| 50 | "```" |
| 51 | ] |
| 52 | }, |
| 53 | { |
| 54 | "cell_type": "code", |
| 55 | "execution_count": 1, |
| 56 | "metadata": { |
| 57 | "colab": { |
| 58 | "base_uri": "https://localhost:8080/", |
| 59 | "height": 35 |
| 60 | }, |
| 61 | "id": "xwqVx9-bwHl3", |
| 62 | "outputId": "38564a63-0567-4406-ee1a-1d3618f27351", |
| 63 | "tags": [] |
| 64 | }, |
| 65 | "outputs": [ |
| 66 | { |
| 67 | "data": { |
| 68 | "text/plain": [ |
| 69 | "'1.8.2'" |
| 70 | ] |
| 71 | }, |
| 72 | "execution_count": 1, |
| 73 | "metadata": {}, |
| 74 | "output_type": "execute_result" |
| 75 | } |
| 76 | ], |
| 77 | "source": [ |
| 78 | "import torch\n", |
| 79 | "torch.__version__" |
| 80 | ] |
| 81 | }, |
| 82 | { |
| 83 | "cell_type": "markdown", |
| 84 | "metadata": { |
| 85 | "id": "6tp2xGV7wHl4" |
| 86 | }, |
| 87 | "source": [ |
| 88 | "## Basic Concepts: Tensor\n", |
| 89 | "\n", |
| 90 | "**Tensor** is a multi-dimensional array. It is very convenient to use tensors to represent different types of data:\n", |
| 91 | "* 400x400 - black-and-white picture\n", |
| 92 | "* 400x400x3 - color picture \n", |
| 93 | "* 16x400x400x3 - minibatch of 16 color pictures\n", |
| 94 | "* 25x400x400x3 - one second of 25-fps video\n", |
| 95 | "* 8x25x400x400x3 - minibatch of 8 1-second videos" |
| 96 | ] |
| 97 | }, |
| 98 | { |
| 99 | "cell_type": "markdown", |
| 100 | "metadata": { |
| 101 | "id": "qG2bsaR7wHl4" |
| 102 | }, |
| 103 | "source": [ |
| 104 | "### Simple Tensors\n", |
| 105 | "\n", |
| 106 | "You can easily create simple tensors from lists of np-arrays, or generate random ones:" |
| 107 | ] |
| 108 | }, |
| 109 | { |
| 110 | "cell_type": "code", |
| 111 | "execution_count": 2, |
| 112 | "metadata": { |
| 113 | "colab": { |
| 114 | "base_uri": "https://localhost:8080/" |
| 115 | }, |
| 116 | "id": "ybpnk08HwHl4", |
| 117 | "outputId": "54e2c89b-b373-4389-b285-49b0510be931", |
| 118 | "trusted": true |
| 119 | }, |
| 120 | "outputs": [ |
| 121 | { |
| 122 | "name": "stdout", |
| 123 | "output_type": "stream", |
| 124 | "text": [ |
| 125 | "tensor([[1, 2],\n", |
| 126 | " [3, 4]])\n", |
| 127 | "tensor([[ 1.3577, 0.7550, -1.7503],\n", |
| 128 | " [-0.7006, 0.1918, -0.2571],\n", |
| 129 | " [ 0.2964, -0.3188, 0.4575],\n", |
| 130 | " [-0.1524, 1.3446, 0.7218],\n", |
| 131 | " [-1.4642, -1.3296, -0.5098],\n", |
| 132 | " [ 2.2282, 0.5065, 0.6176],\n", |
| 133 | " [-0.3013, 0.9485, 0.1195],\n", |
| 134 | " [ 1.0261, 1.5614, -0.1013],\n", |
| 135 | " [-0.2211, -0.4294, -2.2319],\n", |
| 136 | " [ 1.4257, -0.6976, 0.0656]])\n" |
| 137 | ] |
| 138 | } |
| 139 | ], |
| 140 | "source": [ |
| 141 | "a = torch.tensor([[1,2],[3,4]])\n", |
| 142 | "print(a)\n", |
| 143 | "a = torch.randn(size=(10,3))\n", |
| 144 | "print(a)" |
| 145 | ] |
| 146 | }, |
| 147 | { |
| 148 | "cell_type": "markdown", |
| 149 | "metadata": { |
| 150 | "id": "AXFMsV3r09Ux" |
| 151 | }, |
| 152 | "source": [ |
| 153 | "You can use arithmetic operations on tensors, which are performed element-wise, as in numpy. Tensors are automatically expanded to required dimension, if needed. To extract numpy-array from tensor, use `.numpy()`:" |
| 154 | ] |
| 155 | }, |
| 156 | { |
| 157 | "cell_type": "code", |
| 158 | "execution_count": 3, |
| 159 | "metadata": { |
| 160 | "colab": { |
| 161 | "base_uri": "https://localhost:8080/" |
| 162 | }, |
| 163 | "id": "e5Nu5Xgj1DnQ", |
| 164 | "outputId": "c1fbcd86-dde6-40b6-8edf-7a37f9d60901" |
| 165 | }, |
| 166 | "outputs": [ |
| 167 | { |
| 168 | "name": "stdout", |
| 169 | "output_type": "stream", |
| 170 | "text": [ |
| 171 | "tensor([[ 0.0000, 0.0000, 0.0000],\n", |
| 172 | " [-2.0583, -0.5631, 1.4932],\n", |
| 173 | " [-1.0613, -1.0738, 2.2078],\n", |
| 174 | " [-1.5101, 0.5896, 2.4722],\n", |
| 175 | " [-2.8219, -2.0846, 1.2405],\n", |
| 176 | " [ 0.8706, -0.2485, 2.3679],\n", |
| 177 | " [-1.6590, 0.1935, 1.8698],\n", |
| 178 | " [-0.3316, 0.8065, 1.6490],\n", |
| 179 | " [-1.5788, -1.1844, -0.4816],\n", |
| 180 | " [ 0.0680, -1.4526, 1.8159]])\n", |
| 181 | "[3.887189 2.1276016 0.17371987]\n" |
| 182 | ] |
| 183 | } |
| 184 | ], |
| 185 | "source": [ |
| 186 | "print(a-a[0])\n", |
| 187 | "print(torch.exp(a)[0].numpy())" |
| 188 | ] |
| 189 | }, |
| 190 | { |
| 191 | "cell_type": "markdown", |
| 192 | "metadata": { |
| 193 | "id": "uQ5zN6cVyrG7" |
| 194 | }, |
| 195 | "source": [ |
| 196 | "## In-place and out-of-place Operations\n", |
| 197 | "\n", |
| 198 | "Tensor operations such as `+`/`add` return new tensors. However, sometimes you need to modify the existing tensor in-place. Most of the operations have their in-place counterparts, which end with `_`:" |
| 199 | ] |
| 200 | }, |
| 201 | { |
| 202 | "cell_type": "code", |
| 203 | "execution_count": 4, |
| 204 | "metadata": { |
| 205 | "colab": { |
| 206 | "base_uri": "https://localhost:8080/" |
| 207 | }, |
| 208 | "id": "Mjkbcw3-ACKS", |
| 209 | "outputId": "ca021008-9ab6-4b09-c5a5-bbe854cd1493" |
| 210 | }, |
| 211 | "outputs": [ |
| 212 | { |
| 213 | "name": "stdout", |
| 214 | "output_type": "stream", |
| 215 | "text": [ |
| 216 | "Result when adding out-of-place: tensor(8)\n", |
| 217 | "Result after adding in-place: tensor(8)\n" |
| 218 | ] |
| 219 | } |
| 220 | ], |
| 221 | "source": [ |
| 222 | "u = torch.tensor(5)\n", |
| 223 | "print(\"Result when adding out-of-place:\",u.add(torch.tensor(3)))\n", |
| 224 | "u.add_(torch.tensor(3))\n", |
| 225 | "print(\"Result after adding in-place:\", u)" |
| 226 | ] |
| 227 | }, |
| 228 | { |
| 229 | "cell_type": "markdown", |
| 230 | "metadata": { |
| 231 | "id": "DLPUcVsXACKT" |
| 232 | }, |
| 233 | "source": [ |
| 234 | "This is how we can compute the sum or all rows in a matrix in a naive way:" |
| 235 | ] |
| 236 | }, |
| 237 | { |
| 238 | "cell_type": "code", |
| 239 | "execution_count": 5, |
| 240 | "metadata": { |
| 241 | "colab": { |
| 242 | "base_uri": "https://localhost:8080/" |
| 243 | }, |
| 244 | "id": "7pu0UZ-_yqfB", |
| 245 | "outputId": "bd2e8c6a-39e1-4f29-990b-9591e866936c" |
| 246 | }, |
| 247 | "outputs": [ |
| 248 | { |
| 249 | "name": "stdout", |
| 250 | "output_type": "stream", |
| 251 | "text": [ |
| 252 | "tensor([ 3.4945, 2.5325, -2.8684])\n" |
| 253 | ] |
| 254 | } |
| 255 | ], |
| 256 | "source": [ |
| 257 | "s = torch.zeros_like(a[0])\n", |
| 258 | "for i in a:\n", |
| 259 | " s.add_(i)\n", |
| 260 | "\n", |
| 261 | "print(s)" |
| 262 | ] |
| 263 | }, |
| 264 | { |
| 265 | "cell_type": "markdown", |
| 266 | "metadata": { |
| 267 | "id": "rIh1EHcezlNo" |
| 268 | }, |
| 269 | "source": [ |
| 270 | "But it is much better to use" |
| 271 | ] |
| 272 | }, |
| 273 | { |
| 274 | "cell_type": "code", |
| 275 | "execution_count": 6, |
| 276 | "metadata": { |
| 277 | "colab": { |
| 278 | "base_uri": "https://localhost:8080/" |
| 279 | }, |
| 280 | "id": "aQIdWZ1kzn6P", |
| 281 | "outputId": "89000bb4-f45e-493b-a7b0-39fa4e7d92c1" |
| 282 | }, |
| 283 | "outputs": [ |
| 284 | { |
| 285 | "data": { |
| 286 | "text/plain": [ |
| 287 | "tensor([ 3.4945, 2.5325, -2.8684])" |
| 288 | ] |
| 289 | }, |
| 290 | "execution_count": 6, |
| 291 | "metadata": {}, |
| 292 | "output_type": "execute_result" |
| 293 | } |
| 294 | ], |
| 295 | "source": [ |
| 296 | "torch.sum(a,axis=0)" |
| 297 | ] |
| 298 | }, |
| 299 | { |
| 300 | "cell_type": "markdown", |
| 301 | "metadata": { |
| 302 | "id": "5UzUmEZhACKT" |
| 303 | }, |
| 304 | "source": [ |
| 305 | "You can read more on PyTorch tensors in the [official documentation](https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html)" |
| 306 | ] |
| 307 | }, |
| 308 | { |
| 309 | "cell_type": "markdown", |
| 310 | "metadata": { |
| 311 | "id": "U-auwezDwHl6" |
| 312 | }, |
| 313 | "source": [ |
| 314 | "## Computing Gradients\n", |
| 315 | "\n", |
| 316 | "For back propagation, you need to compute gradients. We can set any PyTorch Tensor's attribute `requires_grad` to `True`, which will result in all operations with this tensor being tracked for gradient calculations. To compute the gradients, you need to call `backward()` method, after which the gradient will become available using `grad` attribute:\n" |
| 317 | ] |
| 318 | }, |
| 319 | { |
| 320 | "cell_type": "code", |
| 321 | "execution_count": 7, |
| 322 | "metadata": { |
| 323 | "colab": { |
| 324 | "base_uri": "https://localhost:8080/" |
| 325 | }, |
| 326 | "id": "m8vFOXr7wHl6", |
| 327 | "outputId": "7054c2b1-0b61-4938-937d-813f75f0b195", |
| 328 | "trusted": true |
| 329 | }, |
| 330 | "outputs": [ |
| 331 | { |
| 332 | "name": "stdout", |
| 333 | "output_type": "stream", |
| 334 | "text": [ |
| 335 | "tensor([[-0.1728, 0.0913],\n", |
| 336 | " [-0.1666, -0.1942]])\n" |
| 337 | ] |
| 338 | } |
| 339 | ], |
| 340 | "source": [ |
| 341 | "a = torch.randn(size=(2, 2), requires_grad=True)\n", |
| 342 | "b = torch.randn(size=(2, 2))\n", |
| 343 | "\n", |
| 344 | "c = torch.mean(torch.sqrt(torch.square(a) + torch.square(b))) # Do some math using `a`\n", |
| 345 | "c.backward() # call backward() to compute all gradients\n", |
| 346 | "# What's the gradient of `c` with respect to `a`?\n", |
| 347 | "print(a.grad)" |
| 348 | ] |
| 349 | }, |
| 350 | { |
| 351 | "cell_type": "markdown", |
| 352 | "metadata": { |
| 353 | "id": "nPj3rtrtACKU" |
| 354 | }, |
| 355 | "source": [ |
| 356 | "To be more precise, PyTorch automatically **accumulates** gradients. If you specify `retain_graph=True` when calling `backward`, computational graph will be preserved, and new gradient is added to the `grad` field. In order to restart computing gradients from scratch, we need to reset `grad` field to 0 explicitly by calling `zero_()`: " |
| 357 | ] |
| 358 | }, |
| 359 | { |
| 360 | "cell_type": "code", |
| 361 | "execution_count": 8, |
| 362 | "metadata": { |
| 363 | "colab": { |
| 364 | "base_uri": "https://localhost:8080/" |
| 365 | }, |
| 366 | "id": "z_VIw8MoACKU", |
| 367 | "outputId": "36a28b11-6919-47ab-c3f9-c7f1d8500423" |
| 368 | }, |
| 369 | "outputs": [ |
| 370 | { |
| 371 | "name": "stdout", |
| 372 | "output_type": "stream", |
| 373 | "text": [ |
| 374 | "tensor([[-0.5185, 0.2739],\n", |
| 375 | " [-0.4998, -0.5826]])\n", |
| 376 | "tensor([[-0.1728, 0.0913],\n", |
| 377 | " [-0.1666, -0.1942]])\n" |
| 378 | ] |
| 379 | } |
| 380 | ], |
| 381 | "source": [ |
| 382 | "c = torch.mean(torch.sqrt(torch.square(a) + torch.square(b)))\n", |
| 383 | "c.backward(retain_graph=True)\n", |
| 384 | "c.backward(retain_graph=True)\n", |
| 385 | "print(a.grad)\n", |
| 386 | "a.grad.zero_()\n", |
| 387 | "c.backward()\n", |
| 388 | "print(a.grad)" |
| 389 | ] |
| 390 | }, |
| 391 | { |
| 392 | "cell_type": "markdown", |
| 393 | "metadata": { |
| 394 | "id": "HM9sUkVgCiG9" |
| 395 | }, |
| 396 | "source": [ |
| 397 | "To compute gradients, PyTorch creates and maintains **compute graph**. For each tensor that has the `requires_grad` flag set to `True`, PyTorch maintains a special function called `grad_fn`, which computes the derivative of the expression according to chain differentiation rule:" |
| 398 | ] |
| 399 | }, |
| 400 | { |
| 401 | "cell_type": "code", |
| 402 | "execution_count": 9, |
| 403 | "metadata": { |
| 404 | "colab": { |
| 405 | "base_uri": "https://localhost:8080/" |
| 406 | }, |
| 407 | "id": "PcxHb-7jC7Vv", |
| 408 | "outputId": "3b3fa138-6d09-4636-8a71-f4a4051c7827" |
| 409 | }, |
| 410 | "outputs": [ |
| 411 | { |
| 412 | "name": "stdout", |
| 413 | "output_type": "stream", |
| 414 | "text": [ |
| 415 | "tensor(0.9143, grad_fn=<MeanBackward0>)\n" |
| 416 | ] |
| 417 | } |
| 418 | ], |
| 419 | "source": [ |
| 420 | "print(c)" |
| 421 | ] |
| 422 | }, |
| 423 | { |
| 424 | "cell_type": "markdown", |
| 425 | "metadata": { |
| 426 | "id": "rvLfNiblACKV" |
| 427 | }, |
| 428 | "source": [ |
| 429 | "Here `c` is computed using `mean` function, thus `grad_fn` point to a function called `MeanBackward`.\n", |
| 430 | "\n", |
| 431 | "In most of the cases, we want PyTorch to compute gradient of a scalar function (such as loss function). However, if we want to compute the gradient of a tensor with respect to another tensor, PyTorch allows us to compute the product of a Jacobian matrix and a given vector.\n", |
| 432 | "\n", |
| 433 | "Suppose we have a vector function $\\vec{y}=f(\\vec{x})$, where\n", |
| 434 | "$\\vec{x}=\\langle x_1,\\dots,x_n\\rangle$ and\n", |
| 435 | "$\\vec{y}=\\langle y_1,\\dots,y_m\\rangle$, then a gradient of $\\vec{y}$ with respect to $\\vec{x}$ is defined by a **Jacobian**:\n", |
| 436 | "\n", |
| 437 | "$$\n", |
| 438 | "\\begin{align}J=\\left(\\begin{array}{ccc}\n", |
| 439 | " \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{1}}{\\partial x_{n}}\\\\\n", |
| 440 | " \\vdots & \\ddots & \\vdots\\\\\n", |
| 441 | " \\frac{\\partial y_{m}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n", |
| 442 | "\\end{array}\\right)\\end{align}\n", |
| 443 | "$$\n", |
| 444 | "\n", |
| 445 | "Instead of giving us access to the whole Jacobian, PyTorch computes the product $v^T\\cdot J$ of Jacobian with some vector\n", |
| 446 | "$v=(v_1 \\dots v_m)$. In order to do that, we need to call ``backward`` and pass `v` as an argument. The size of `v` should be the same as the size of the original tensor, with respect to which we compute the gradient.\n" |
| 447 | ] |
| 448 | }, |
| 449 | { |
| 450 | "cell_type": "code", |
| 451 | "execution_count": 10, |
| 452 | "metadata": { |
| 453 | "colab": { |
| 454 | "base_uri": "https://localhost:8080/" |
| 455 | }, |
| 456 | "id": "VUNYiQCOACKV", |
| 457 | "outputId": "e3127c21-fce6-420d-f347-ec40cc827e7e" |
| 458 | }, |
| 459 | "outputs": [ |
| 460 | { |
| 461 | "name": "stdout", |
| 462 | "output_type": "stream", |
| 463 | "text": [ |
| 464 | "tensor([[-0.8642, 0.0913],\n", |
| 465 | " [-0.1666, -0.9710]])\n" |
| 466 | ] |
| 467 | } |
| 468 | ], |
| 469 | "source": [ |
| 470 | "c = torch.sqrt(torch.square(a) + torch.square(b))\n", |
| 471 | "c.backward(torch.eye(2)) # eye(2) means 2x2 identity matrix\n", |
| 472 | "print(a.grad)" |
| 473 | ] |
| 474 | }, |
| 475 | { |
| 476 | "cell_type": "markdown", |
| 477 | "metadata": { |
| 478 | "id": "dGHlkVlvACKV" |
| 479 | }, |
| 480 | "source": [ |
| 481 | "More on computing Jacobians in PyTorch can be found in [official documentation](https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html)" |
| 482 | ] |
| 483 | }, |
| 484 | { |
| 485 | "cell_type": "markdown", |
| 486 | "metadata": { |
| 487 | "id": "FnVvj4LkD15r" |
| 488 | }, |
| 489 | "source": [ |
| 490 | "# Example 0: Optimization Using Gradient Descent\n", |
| 491 | "\n", |
| 492 | "Let's try to use automatic differentiation to find a minimum of a simple two-variable function $f(x_1,x_2)=(x_1-3)^2+(x_2+2)^2$. Let tensor `x` hold the current coordinates of a point. We start with some starting point $x^{(0)}=(0,0)$, and compute the next point in a sequence using gradient descent formula:\n", |
| 493 | "$$\n", |
| 494 | "x^{(n+1)} = x^{(n)} - \\eta\\nabla f\n", |
| 495 | "$$\n", |
| 496 | "Here $\\eta$ is so-called **learning rage** (we will denote it by `lr` in the code), and $\\nabla f = (\\frac{\\partial f}{\\partial x_1},\\frac{\\partial f}{\\partial x_2})$ - gradient of $f$.\n", |
| 497 | "\n", |
| 498 | "To begin, let's define starting value of `x` and the function `f`:" |
| 499 | ] |
| 500 | }, |
| 501 | { |
| 502 | "cell_type": "code", |
| 503 | "execution_count": 11, |
| 504 | "metadata": { |
| 505 | "id": "nDw5mV9KEeOa" |
| 506 | }, |
| 507 | "outputs": [], |
| 508 | "source": [ |
| 509 | "x = torch.zeros(2,requires_grad=True)\n", |
| 510 | "f = lambda x : (x-torch.tensor([3,-2])).pow(2).sum()\n", |
| 511 | "lr = 0.1" |
| 512 | ] |
| 513 | }, |
| 514 | { |
| 515 | "cell_type": "markdown", |
| 516 | "metadata": { |
| 517 | "id": "Wt815LWdEj77" |
| 518 | }, |
| 519 | "source": [ |
| 520 | "Now let's do 15 iterations of gradient descent. In each iteration, we will update `x` coordinates and print them, to make sure that we are approaching the minimum point at (3,-2):" |
| 521 | ] |
| 522 | }, |
| 523 | { |
| 524 | "cell_type": "code", |
| 525 | "execution_count": 12, |
| 526 | "metadata": { |
| 527 | "colab": { |
| 528 | "base_uri": "https://localhost:8080/" |
| 529 | }, |
| 530 | "id": "KfwMf555EyWJ", |
| 531 | "outputId": "67e2199c-61ff-4ad1-9c48-b4a646bf8bbd" |
| 532 | }, |
| 533 | "outputs": [ |
| 534 | { |
| 535 | "name": "stdout", |
| 536 | "output_type": "stream", |
| 537 | "text": [ |
| 538 | "Step 0: x[0]=0.6000000238418579, x[1]=-0.4000000059604645\n", |
| 539 | "Step 1: x[0]=1.0800000429153442, x[1]=-0.7200000286102295\n", |
| 540 | "Step 2: x[0]=1.4639999866485596, x[1]=-0.9760000705718994\n", |
| 541 | "Step 3: x[0]=1.7711999416351318, x[1]=-1.1808000802993774\n", |
| 542 | "Step 4: x[0]=2.0169599056243896, x[1]=-1.3446400165557861\n", |
| 543 | "Step 5: x[0]=2.2135679721832275, x[1]=-1.4757120609283447\n", |
| 544 | "Step 6: x[0]=2.370854377746582, x[1]=-1.5805696249008179\n", |
| 545 | "Step 7: x[0]=2.4966835975646973, x[1]=-1.6644556522369385\n", |
| 546 | "Step 8: x[0]=2.597346782684326, x[1]=-1.7315645217895508\n", |
| 547 | "Step 9: x[0]=2.677877426147461, x[1]=-1.7852516174316406\n", |
| 548 | "Step 10: x[0]=2.7423019409179688, x[1]=-1.8282012939453125\n", |
| 549 | "Step 11: x[0]=2.793841600418091, x[1]=-1.8625609874725342\n", |
| 550 | "Step 12: x[0]=2.835073232650757, x[1]=-1.8900487422943115\n", |
| 551 | "Step 13: x[0]=2.868058681488037, x[1]=-1.912039041519165\n", |
| 552 | "Step 14: x[0]=2.894446849822998, x[1]=-1.929631233215332\n" |
| 553 | ] |
| 554 | } |
| 555 | ], |
| 556 | "source": [ |
| 557 | "for i in range(15):\n", |
| 558 | " y = f(x)\n", |
| 559 | " y.backward()\n", |
| 560 | " gr = x.grad\n", |
| 561 | " x.data.add_(-lr*gr)\n", |
| 562 | " x.grad.zero_()\n", |
| 563 | " print(\"Step {}: x[0]={}, x[1]={}\".format(i,x[0],x[1]))" |
| 564 | ] |
| 565 | }, |
| 566 | { |
| 567 | "cell_type": "markdown", |
| 568 | "metadata": { |
| 569 | "id": "8sfjBMBu59B5" |
| 570 | }, |
| 571 | "source": [ |
| 572 | "## Example 1: Linear Regression\n", |
| 573 | "\n", |
| 574 | "Now we know enough to solve the classical problem of **Linear regression**. Let's generate small synthetic dataset:" |
| 575 | ] |
| 576 | }, |
| 577 | { |
| 578 | "cell_type": "code", |
| 579 | "execution_count": 13, |
| 580 | "metadata": { |
| 581 | "id": "j723455WwHl7", |
| 582 | "trusted": true |
| 583 | }, |
| 584 | "outputs": [], |
| 585 | "source": [ |
| 586 | "import numpy as np\n", |
| 587 | "import matplotlib.pyplot as plt\n", |
| 588 | "from sklearn.datasets import make_classification, make_regression\n", |
| 589 | "from sklearn.model_selection import train_test_split\n", |
| 590 | "import random" |
| 591 | ] |
| 592 | }, |
| 593 | { |
| 594 | "cell_type": "code", |
| 595 | "execution_count": 14, |
| 596 | "metadata": { |
| 597 | "colab": { |
| 598 | "base_uri": "https://localhost:8080/", |
| 599 | "height": 282 |
| 600 | }, |
| 601 | "id": "WJNK_J6v6I-Z", |
| 602 | "outputId": "09e6386e-a6d4-4b81-c8d2-153f0acf9696" |
| 603 | }, |
| 604 | "outputs": [ |
| 605 | { |
| 606 | "data": { |
| 607 | "text/plain": [ |
| 608 | "<matplotlib.collections.PathCollection at 0x20b8e1f1ca0>" |
| 609 | ] |
| 610 | }, |
| 611 | "execution_count": 14, |
| 612 | "metadata": {}, |
| 613 | "output_type": "execute_result" |
| 614 | }, |
| 615 | { |
| 616 | "data": { |
| 617 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbVklEQVR4nO3df4ylVXkH8O+zw0XuAjJaphVGx8XULKkQdtmJoZnGyGrFij82YNTGtKltsmmTNkIo7egfRf9iEmPVxqbNxNJqSnVpFzYGVDRZDHWj2Bl2EZClUXErs7aMZQfEmbqzw9M/7n2XO++85z3nve857z3n3u8n2bC795075903PPfMc57zHFFVEBFRvLYNegBERFSOgZqIKHIM1EREkWOgJiKKHAM1EVHkzgnxphdffLHu2LEjxFsTEQ2lxcXFn6nqRNFrQQL1jh07sLCwEOKtiYiGkoicML3G1AcRUeQYqImIIsdATUQUOQZqIqLIMVATEUUuSNUHEdEoOXR0CZ+4/0mcXFnDpeNt3HrdTuzbPent/RmoiYhqOHR0CR+5+1GsrW8AAJZW1vCRux8FAG/BmqkPIqIaPnH/k2eDdGZtfQOfuP9Jb9+DgZqIqIaTK2uV/r4fDNRERDVcOt6u9Pf9YKAmIkIn1zwzdxiXzd6HmbnDOHR0yenrbr1uJ9qtsU1/126N4dbrdnobGxcTiWjk1VkQzF5n1QcRUUBlC4IuAXff7kmvgTmPqQ8iGnlNLAjWYQ3UIrJTRI71/HpeRG5qYGxERI1oYkGwDmugVtUnVXWXqu4CsAfAKoB7Qg+MiKgpTSwI1lE1R/0WAD9UVWODayKi1FRZEOzdLn5RuwURYGV1PcgiYkZU1f1ikTsAPKyqny14bT+A/QAwNTW158QJxnIiGi756pC8dmsMt99wZV/BWkQWVXW66DXnxUQRORfAuwH8a9HrqjqvqtOqOj0xUXjsFxFR9MrqqYuqQ3r53jqeqZL6+B10ZtP/430UREQRsNVTu1SBhKgUqVKe97sAvuh9BEREkbA1WHKpAglRKeIUqEVkO4DfBnC39xEQEUXCVk9dVB3SK1SliFPqQ1VXAfyK9+9ORBSRS8fbWCoI1tksOV8d0lTVB7eQExF13Xrdzi1VHflZcujt4kUYqImIupposNQPBmoioh6DmDHbsCkTEVHkGKiJiCLHQE1EFDnmqIloJPQ2U4plkdAVAzURDYWyQNzvUVvZey6trGFMBBuqmBxAkGegJqJKYpyZlgViALjlrkewkesUajtqK/+e2ddXOU/Rl0ptTl1NT0/rwsKC9/closEqavNZp7WnLzNzhwt3FI63W/jlmRdLO95NjrcLP3RM79n7dUdm99YffJeXNqdERLamRYNi6tGxsrZeGqQFnRmy4qWZctbW1NYFr8nzFBmoichZrIfA9tOxTgDk8wlVOuU1eZ4iAzUROTMFp4vaLWOz/SaYzjx8xfZW4fVjIluCdMalU17T5ykyUBORs6Lg1dom+MXpM8YUQgj5U1gA4PYbrsTkeBuCTv749huuxG3vekNhAP/k+67CpOXk8X27J8++J9AJ7uh5b1Z9EFGUipoWrZ4+g1Or65uus1VU1GGq8Lj9hiuNi3umKpUYO+UVYdUHEdVy2ex9hWkEAfDU3PXev5+pGqOfKoyYSg3Lqj44oyaiWmzN9n3zuaAZy4zZhjlqIqrFtJAXarHN9AHQZBVG0xioiaiW3kW33oW8UDPVpj8YYsDUBxHV1mQKIdZTWEJyCtQiMg7gcwCuQKdG/A9V9dsBx0VEZJRKbtkX1xn1ZwB8TVXfKyLnAtgecExERIViqtJokjVQi8jLAbwJwB8AgKqeBnA67LCIiDbrt1XpMHBZTHwdgGUA/ygiR0XkcyJyfv4iEdkvIgsisrC8vOx9oEQ0WPndgE1vE4+1IVQTXAL1OQCuBvB3qrobwC8AzOYvUtV5VZ1W1emJiQnPwySiQcpms01uE+/93mUtRwfdEKoJLjnqpwE8raoPdf/8bygI1EQ0vEyz2Y99+fHSU1Xq5pOL+l/nDXP9dMYaqFX1v0XkJyKyU1WfBPAWAN8PPzQiikVZv+eVtU6fj/ypKj7yyUUfEL2GvX4641r18WcA7uxWfPwIwIfCDYmIYmPaJp7XmzM25ZOrBOqytMYgzi4cFKdArarHABQ2CyGi4XfrdTutKYhMWXCtmk82fUD4PgYrdtxCTkRWRdvETU35Lx1vO/XjcKkiGcXt4kW4hZyInOR3A5oOus2CaNlrLjXR2WLk2voGxkSwoTpS6Y5eDNREI6xOZYZLzw3Ta2U10ft2T24J5BuqZwP9qAVpgAcHEI0s04y4iWOmTIcNAJ20imnhsig3PSzbyssODmCOmmhEDXKnnymHLUBpdUl+MXKQG3GaxEBNNKJ8npRSVdEioQDGWXYmH+BHZVs5AzXRkDNVV5hmtQoE7+WRryIZb7esQbqo2mOQHzZN4mIi0RArq64oq4323ZnOlEfuXTgsY6r2aPq8xkFhoCYaQllgLApiWWogW5SzXVc3UNtK8cq2idsWN4s+bNqtMVx7+QRm5g4nv8CYYeqDyINBtwDNjyVbYDPJUgP7dk/iyOxeiOW6Omx55LLvYatAKdqIc+OeSRxcXBqqBUbOqIlqiq2hva2REbA1NeAzhZBPc9jak5ZtE3f598tvxJmZO+ylz0hMOKMmqmkQlQdlM3jbLLhoUc7XVu2icjnTbD37EPC9TdxlgTGmn4BccEZNVFPTlQe2GXzZLHZMZNOHSDbD9HWyd9GHlqK49G719BkcOrrk/VRx208Hsf0E5IKBmqimpisPbNuvixbYWtsEEGB9oxMui4KTj5O9TR9Oik4JXta7GgBOra5vGoOvIGlaYMxm6LZ/vxgx9UFUU9Md3mwz+KIFtgvOO+dskM6Y0jN10gKmD6fJ8TbOf9nWeWGIFFHR/fcuSqZYe80ZNVFNvn90t3GZwednqJfN3lf4XqYt2f2mBcpmszcfOOY0Bh/KZugp1l5zRk3kQVbm9tTc9Tgyuzfoj9D9zOBd+kMD9RdGy2azrmMILcUe15xREyWmnxm8LW+b8ZEWMM1mXccQWtM/AfnAQE2UoKqLb67BKWRaIKYA6XPxsgnsR00UqUH0WR5kj+pRV9aPmjNqoggNqtY3plkvvcQpUIvIjwH8HMAGgDOmqE9Efgyy1je1tMAoqDKjvlZVfxZsJER0Voq1vhQOUx9EEcjno8e3t3BqdX3LdTHX+lI4rnXUCuDrIrIoIvuLLhCR/SKyICILy8vL/kZINOSKGhm98H9n0Brb3M4o9lpfCsd1Rj2jqidF5FcBfENEjqvqg70XqOo8gHmgU/XheZxEQyM/e149fWZLPnr9RcV4u4XzX3YOF/XILVCr6snuf58RkXsAvBHAg+VfRUR5RdUcJs+trePYbW9ramgUMWvqQ0TOF5ELs98DeBuAx0IPjGgYuTT1zzAfTRmXGfWvAbhHRLLr/0VVvxZ0VERDyrVqg/lo6mUN1Kr6IwBXNTAWoqFn2qLNfDSVYXkeJWkQ26t9MDUm+ti732Acf++9XtRuQQRYWV1P6r6pHgZqSk6T26ttHwhVPzCqbtHO32vvCSlF953qBxiVY1MmSs7M3GHjqdVHZvd6+z62BkVNNDAy3Wuv7L7ZUCltZU2ZeHAAJaep7dW2JvpNnD7uck/ZNYM4DZ2awUBNyWnqpBDbB0ITHxgu95Rdw/4gw4uBmpLT1FFKtg+EJj4wiu61V+99x3LUFfnHQE3JsZ0y7YvtA6GJD4z8vY63W3jF9lbhfYcYT50TyckfLiYSlfBd9THo8VZ9Ly5ONqdsMZGBmihxoT4smqquoQ4exUWUiKpBN2RNORcn48EcNVHOoPKyRX2pP3L3o6XfP2RJHhcn48FATUkJHUT7CZau72sbdz9BN+Sst6nqGrJjoKZkhAqivULMUF3H3U/QDTnrbaq6huyYo6ZkNHEyd4gZquu4TZ31yoKuqcmTr1kvTySPAwM1JaOpnYBVg6WNaXxLK2uYmTt8duHw2ssncHBxqVLQrdrkidLEQE3JCBFE80LMUE3jFrx0FNfSyhoOLi7hxj2TeOD4cqWgy1nv8GOgpmSE/jEfKJ+h9luvXDRuAZDfwbC2voEHji83UqMc20YdKscNL5SUQQWYol16APCK7S3c9q7ipv9lDf9NrUsFwFNz14e4hU3j4o7D+HDDCw0N24/5oQK56VDaU6vrhRtMihr+t1tj+NT7d2Hf7knjrr8mapSbWJQlv1ieR17E0LwnZPle2YJlUfmercxvkDXK3HGYHudALSJjInJURO4NOSBKTxP1zS5MwfGWux6pPRbbTDcf5GzBcJA1ytxxmJ4qqY8PA3gCwMsDjYUSFcuP0qbguKFau/9F0YJgr94gd+joEraJYKNg/af3ukFVazSxKEt+Oc2oReTVAK4H8Lmww6EUxfKjdNmMsO7uwmwGPN5ubXmtN8hlP10UBelYgiF3HKbHdUb9aQB/AeBC0wUish/AfgCYmpqqPTBKRxP1zS5ss966HxzZDLhswdK06DgmElUwZO11WqyBWkTeCeAZVV0UkTebrlPVeQDzQKc8z9cAKX6x/CidBZ5b7nrEmnao+31MQc70YfCiKgMj9c0l9TED4N0i8mMAXwKwV0T+OeioKCkx/Si9b/ckPvm+qwZWUcGFOgqh0oaX7oz6z1X1nWXXccMLDVpMG2O4mYRccMMLjZxB5WDZJIlC4BZyGphB9ptgrwuKDWfUFB2fZ/35Omdw4cSzlTvXETWBM2oaCFOvC6CzGOkaJKvkhLOAXtYQqff/BuaWqUllM2r2+qCBKKtprrIF3eXorENHl7Dr41/HTQeOGYM0UNx21MchsUR1MVDTQNjK1VyDpG1XZDbjXllbrz7IkvcnahIDNTnz2SGvqHtcnkuQtNUtm3YK5knF9ydqEgM1OfHdIa93k4yJS5C0tQt1CfaT42188JqpgW2SIbJh1Qc5CdEhr7d3Rr9b0G11y2WnqeQXC6df+0qW7FGUGKjJScgOeXU3iZRtbjE1aio6QouNiihWDNRDoInNG6E75IUKktwpSMOAgTpxPjeOlImlQ14/OFOm1HExMXEudcQ+xNQhj2jUcEaduCZPV2liZtqbxrmo3YJI56Tvse7RVlV2LRINCwbqxMVyuoqNSx49n8bp3aSSHQQQKrVDFDOmPhJnqyOOgWsNtuvmFG7tplHDQJ24FHLHpjz6TQeObdrhWCVdw63dNEqY+hgCsVc1uDRgAso3p+TFltohComBmrzL56PHt7dwatXcFClLZdhOEc/EltohCo2BmrwqqutubRO0xgTrG+be5ydX1rZsTmHVB1EHAzV5VZSPXn9RMd5u4fyXnWNMbWSpjKbTODySi1LAxUTyypSPfm5tHUdm9+LT798VTZWK746ARKFYA7WInCci3xWRR0TkcRH5eBMDo5f47AMdmq0/dExVKk3t6iSqyyX18UsAe1X1BRFpAfiWiHxVVb8TeGyE5np5+OLSEySWKpUmd3US1WEN1No5/faF7h9b3V/+T8SlQj77QDeRj02pW10quzqJnE4hF5ExAIsAfh3A36rqXxZcsx/AfgCYmprac+LECc9DHU2Xzd5X+KkoAJ6au975fYqa82enbhdVUozCIluVE8yJQis7hdyp6kNVNwDsEpFxAPeIyBWq+ljumnkA8wAwPT3NGbcnvmZ9RTPz7CHl0ym2dEtR46SV1fXkAnpKs38abZXK81R1RUS+CeDtAB6zXD60mpxt+uoDbcu79qZTbItspsZJsefPi8SSLycq41L1MdGdSUNE2gDeCuB44HFFq+mSLl9VEi4z8CyYly2y2RonsWqCyD+XGfUlAD7fzVNvA3CXqt4bdljxCnHIq42PWZ/L9uwsmJvSLdtEnHpxsGqCyC+Xqo/vAdjdwFiSkGJJV5aqWVvfOLsVO1tIzPSmU0xBvejrimQBfxQWJImawJ2JFdk2dMSmN1UDdIJtuzWGD14zZUynZOmWMZEt76foVIuYZAGfu/6I/GGvj4piOuTVZcZqStU8cHwZR2b3Gt973+5J3HzgWOFrWUlfWdXHzNzhxlNERMOKgbqiWEq6XHcs1knVmHLVk+Pt0iBf9/sS0WYM1H2IoaTLdVGzTh12nZ8euOuPyB/mqAML1VDJdcZa50zFOqWBKZzlSJQKzqgDCtlQyXXGWjdV0+9PD7GkiIiGgVOvj6qmp6d1YWHB+/umZmbucGEwHRPBJ993Va2gxT4VRMOldq8P6o8pPbGhWntmzRkr0ehgoA6o7FRtH6Vqvhc1uUGFKE5cTAyoaEGtV0ylatygQhQvBuqAynb4AXGVqvFYKqJ4MfURWJY6aHo3Y9U0BjeoEMWLgboBLgt/PvPD/ZQFcoMKUbwYqBtStvDnu966n1asMfUwIaLNmKOOgM/88KGjS8ZKk7I0hq8DCojIP86oI+ArP5zNzE1saYwYepgQ0VacUUfAV4/rsmOymMYgShdn1DX5WAQ05YevvXwCM3OHa1duAGAagyhhDNQ1+FoELKoKufbyCRxcXPJSuTE53maQJkrY0AbqujPdOqenuG4NL/se/ZyQwsoNouFkDdQi8hoAXwDwKgAvAphX1c+EHlgddWe6TZyeYvse/bx3P42a2N+DKH4uM+ozAG5R1YdF5EIAiyLyDVX9fuCx9a3uTLeJ01Ns36Pf965SuRGyXzYR+WOt+lDVn6rqw93f/xzAEwCi/r+4brlbE6en2L5HEyeksL8HURoqleeJyA4AuwE8VPDafhFZEJGF5eVlT8PrT91yN9evr7NJxPY9mtiAwv4eRGlwXkwUkQsAHARwk6o+n39dVecBzAOdE168jdBBPs+ar5gAqs1GqyzK9btJxOV7hN6Awv4eRGlwmlGLSAudIH2nqt4ddkjVFPVRPri4hBv3TPY9G+1nNlv1ENsYtmzzAFqiNFjPTBQRAfB5AM+q6k0ub9rkmYmmcwknx9s4Mru39vu7VEWkfH4hqz6I4lB2ZqJLoP4tAP8O4FF0yvMA4KOq+hXT14QM1PnAYmpAJACemru+8vv1BqqiACwAFJ0Pguza0B8WRDT8ah1uq6rfQic+DVxROVkWOPNc8qy28rSiqojse/Vey0U5IgopqaZMpsCZ/xRxzbPaytNsgTa71ldTJSKiIkkFalPgzFIRAmC83cJ5rW24+cAx66KebSbsEmhPrqxxUY6IgkoqUJsCZ5YL/tT7d+GXZ17EqdV1p5O0bTNh2yni2bUxVHAQ0fBKqimTrfa46tZx2/v19s4oyofnr2VgJqIQkgrUtqZDVRf1XJoY9QbgGErZYhgDETXLWp7XjybrqHsNe5lcyvXaRFSurDwvqRy1TSqLelV3MWbYRIloNCWV+rDppx9z0+q0FmW9NtFoSjpQm/K1/QZmX/nfsvep0yubTZSIRlOygdp303tf71f35JayIM+jtohGU7I5at/5Wl/vZ3ufstrtok6AvXXgrNcmGk3Rzahd0w++87W+3s/l5BbTrNglLcJ6baLRE1WgNqUNFk48iweOL28K3r7ztb7ez/Y+ZQueNx84VvieXCwkGm1RBWrTjPLO7/zXlq51N+6ZrHWKS56v/G+dk1u4WEhERaLKUZc1Xeq1tr6BB44ve83X+sr/1nmfVOrAiahZUe1MNO0sLOJ6MEBquEWcaDTVOjigSUVpgzoHA6SIi4VElBdV6qMobfDBa6aYDiCikRbVjBoonlFOv/aVTAcQ0ciKLlAXYTqAiEaZNVCLyB0A3gngGVW9IvyQwuFCHRGlyGVG/U8APgvgC2GHspXPwOq7NwgRUVOsi4mq+iCAZxsYyya2vhdVsZczEaUqqqqPXr4DK3s5E1GqvAVqEdkvIgsisrC8vFz7/XwHVtuJ40REsfIWqFV1XlWnVXV6YmKi9vv5Dqzcnk1EqYo29eEzsGaLkmvrGxgTAQCMt1s4r7UNNx84VuncQiKiplkDtYh8EcC3AewUkadF5I/CD8tfk6TeRUkA2FBFa5vgF6fP4NTqupeFSiKikKJqyhRClUZPk+NtHJndG3hERERblTVlijb14UuVxUdWgBBRjIY+UFdZfGQFCBHFaOgDddGiZGuboDUmm/6OFSBEFKskmjLVYTqjsOjvuJWciGI09IuJREQpSOKEF3a2IyIqFkWgZmc7IiKzKBYT2dmOiMgsikDNznZERGZRBGp2tiMiMosiULOzHRGRWRSLiaZaZy4kEhFFEqgBnjRORGQSReqDiIjMGKiJiCLHQE1EFDkGaiKiyDFQExFFLkj3PBFZBnCizy+/GMDPPA5nkIblXoblPgDeS4yG5T6AevfyWlWdKHohSKCuQ0QWTK3+UjMs9zIs9wHwXmI0LPcBhLsXpj6IiCLHQE1EFLkYA/X8oAfg0bDcy7DcB8B7idGw3AcQ6F6iy1ETEdFmMc6oiYioBwM1EVHkBhKoReTtIvKkiPxARGYLXhcR+Zvu698TkasHMU4XDvfyZhF5TkSOdX/91SDGaSMid4jIMyLymOH1lJ6J7V5SeSavEZEHROQJEXlcRD5ccE0Sz8XxXlJ5LueJyHdF5JHuvXy84Bq/z0VVG/0FYAzADwG8DsC5AB4B8Bu5a94B4KsABMA1AB5qepwe7+XNAO4d9Fgd7uVNAK4G8Jjh9SSeieO9pPJMLgFwdff3FwL4z4T/X3G5l1SeiwC4oPv7FoCHAFwT8rkMYkb9RgA/UNUfqeppAF8C8J7cNe8B8AXt+A6AcRG5pOmBOnC5lySo6oMAni25JJVn4nIvSVDVn6rqw93f/xzAEwDyTduTeC6O95KE7r/1C90/trq/8lUZXp/LIAL1JICf9Pz5aWx9YC7XxMB1nL/Z/THpqyLyhmaG5l0qz8RVUs9ERHYA2I3O7K1Xcs+l5F6ARJ6LiIyJyDEAzwD4hqoGfS6DOOFFCv4u/2nkck0MXMb5MDp7+F8QkXcAOATg9aEHFkAqz8RFUs9ERC4AcBDATar6fP7lgi+J9rlY7iWZ56KqGwB2icg4gHtE5ApV7V0T8fpcBjGjfhrAa3r+/GoAJ/u4JgbWcarq89mPSar6FQAtEbm4uSF6k8ozsUrpmYhIC53Adqeq3l1wSTLPxXYvKT2XjKquAPgmgLfnXvL6XAYRqP8DwOtF5DIRORfABwB8OXfNlwH8fnfl9BoAz6nqT5seqAPrvYjIq0REur9/Izr/5v/b+EjrS+WZWKXyTLpj/AcAT6jqXxsuS+K5uNxLQs9lojuThoi0AbwVwPHcZV6fS+OpD1U9IyJ/CuB+dKom7lDVx0Xkj7uv/z2Ar6CzavoDAKsAPtT0OF043st7AfyJiJwBsAbgA9pdFo6JiHwRnVX3i0XkaQC3obNIktQzAZzuJYlnAmAGwO8BeLSbDwWAjwKYApJ7Li73kspzuQTA50VkDJ0Pk7tU9d6QMYxbyImIIsediUREkWOgJiKKHAM1EVHkGKiJiCLHQE1EFDkGaiKiyDFQExFF7v8B4SC4LI9GLoEAAAAASUVORK5CYII=", |
| 618 | "text/plain": [ |
| 619 | "<Figure size 432x288 with 1 Axes>" |
| 620 | ] |
| 621 | }, |
| 622 | "metadata": { |
| 623 | "needs_background": "light" |
| 624 | }, |
| 625 | "output_type": "display_data" |
| 626 | } |
| 627 | ], |
| 628 | "source": [ |
| 629 | "np.random.seed(13) # pick the seed for reproducibility - change it to explore the effects of random variations\n", |
| 630 | "\n", |
| 631 | "train_x = np.linspace(0, 3, 120)\n", |
| 632 | "train_labels = 2 * train_x + 0.9 + np.random.randn(*train_x.shape) * 0.5\n", |
| 633 | "\n", |
| 634 | "plt.scatter(train_x,train_labels)" |
| 635 | ] |
| 636 | }, |
| 637 | { |
| 638 | "cell_type": "markdown", |
| 639 | "metadata": { |
| 640 | "id": "Ng4rZmGc6oxk" |
| 641 | }, |
| 642 | "source": [ |
| 643 | "Linear regression is defined by a straight line $f_{W,b}(x) = Wx+b$, where $W, b$ are model parameters that we need to find. An error on our dataset $\\{x_i,y_u\\}_{i=1}^N$ (also called **loss function**) can be defined as mean square error:\n", |
| 644 | "$$\n", |
| 645 | "\\mathcal{L}(W,b) = {1\\over N}\\sum_{i=1}^N (f_{W,b}(x_i)-y_i)^2\n", |
| 646 | "$$\n", |
| 647 | "\n", |
| 648 | "Let's define our model and loss function:" |
| 649 | ] |
| 650 | }, |
| 651 | { |
| 652 | "cell_type": "code", |
| 653 | "execution_count": 15, |
| 654 | "metadata": { |
| 655 | "id": "QxhI4GlB6aiH" |
| 656 | }, |
| 657 | "outputs": [], |
| 658 | "source": [ |
| 659 | "input_dim = 1\n", |
| 660 | "output_dim = 1\n", |
| 661 | "learning_rate = 0.1\n", |
| 662 | "\n", |
| 663 | "# This is our weight matrix\n", |
| 664 | "w = torch.tensor([100.0],requires_grad=True,dtype=torch.float32)\n", |
| 665 | "# This is our bias vector\n", |
| 666 | "b = torch.zeros(size=(output_dim,),requires_grad=True)\n", |
| 667 | "\n", |
| 668 | "def f(x):\n", |
| 669 | " return torch.matmul(x,w) + b\n", |
| 670 | "\n", |
| 671 | "def compute_loss(labels, predictions):\n", |
| 672 | " return torch.mean(torch.square(labels - predictions))" |
| 673 | ] |
| 674 | }, |
| 675 | { |
| 676 | "cell_type": "markdown", |
| 677 | "metadata": { |
| 678 | "id": "JUxwj3367gD2" |
| 679 | }, |
| 680 | "source": [ |
| 681 | "We will train the model on a series of minibatches. We will use gradient descent, adjusting model parameters using the following formulae:\n", |
| 682 | "$$\n", |
| 683 | "\\begin{array}{l}\n", |
| 684 | "W^{(n+1)}=W^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial W} \\\\\n", |
| 685 | "b^{(n+1)}=b^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial b} \\\\\n", |
| 686 | "\\end{array}\n", |
| 687 | "$$" |
| 688 | ] |
| 689 | }, |
| 690 | { |
| 691 | "cell_type": "code", |
| 692 | "execution_count": 16, |
| 693 | "metadata": { |
| 694 | "id": "-991PErM7fJU" |
| 695 | }, |
| 696 | "outputs": [], |
| 697 | "source": [ |
| 698 | "def train_on_batch(x, y):\n", |
| 699 | " predictions = f(x)\n", |
| 700 | " loss = compute_loss(y, predictions)\n", |
| 701 | " loss.backward()\n", |
| 702 | " w.data.sub_(learning_rate * w.grad)\n", |
| 703 | " b.data.sub_(learning_rate * b.grad)\n", |
| 704 | " w.grad.zero_()\n", |
| 705 | " b.grad.zero_()\n", |
| 706 | " return loss" |
| 707 | ] |
| 708 | }, |
| 709 | { |
| 710 | "cell_type": "markdown", |
| 711 | "metadata": { |
| 712 | "id": "idr2VEWb9rr0" |
| 713 | }, |
| 714 | "source": [ |
| 715 | "Let's do the training. We will do several passes through the dataset (so-called **epochs**), divide it into minibatches and call the function defined above:" |
| 716 | ] |
| 717 | }, |
| 718 | { |
| 719 | "cell_type": "code", |
| 720 | "execution_count": 17, |
| 721 | "metadata": { |
| 722 | "id": "nOuu0qpx-wAp" |
| 723 | }, |
| 724 | "outputs": [], |
| 725 | "source": [ |
| 726 | "# Shuffle the data.\n", |
| 727 | "indices = np.random.permutation(len(train_x))\n", |
| 728 | "features = torch.tensor(train_x[indices],dtype=torch.float32)\n", |
| 729 | "labels = torch.tensor(train_labels[indices],dtype=torch.float32)" |
| 730 | ] |
| 731 | }, |
| 732 | { |
| 733 | "cell_type": "code", |
| 734 | "execution_count": 18, |
| 735 | "metadata": { |
| 736 | "colab": { |
| 737 | "base_uri": "https://localhost:8080/" |
| 738 | }, |
| 739 | "id": "3zdIf6c_85Ht", |
| 740 | "outputId": "6520288c-da59-4a9f-c37e-cd99779c3073" |
| 741 | }, |
| 742 | "outputs": [ |
| 743 | { |
| 744 | "name": "stdout", |
| 745 | "output_type": "stream", |
| 746 | "text": [ |
| 747 | "Epoch 0: last batch loss = 94.5247\n", |
| 748 | "Epoch 1: last batch loss = 9.3428\n", |
| 749 | "Epoch 2: last batch loss = 1.4166\n", |
| 750 | "Epoch 3: last batch loss = 0.5224\n", |
| 751 | "Epoch 4: last batch loss = 0.3807\n", |
| 752 | "Epoch 5: last batch loss = 0.3495\n", |
| 753 | "Epoch 6: last batch loss = 0.3413\n", |
| 754 | "Epoch 7: last batch loss = 0.3390\n", |
| 755 | "Epoch 8: last batch loss = 0.3384\n", |
| 756 | "Epoch 9: last batch loss = 0.3382\n" |
| 757 | ] |
| 758 | } |
| 759 | ], |
| 760 | "source": [ |
| 761 | "batch_size = 4\n", |
| 762 | "for epoch in range(10):\n", |
| 763 | " for i in range(0,len(features),batch_size):\n", |
| 764 | " loss = train_on_batch(features[i:i+batch_size].view(-1,1),labels[i:i+batch_size])\n", |
| 765 | " print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))" |
| 766 | ] |
| 767 | }, |
| 768 | { |
| 769 | "cell_type": "markdown", |
| 770 | "metadata": {}, |
| 771 | "source": [ |
| 772 | "We now have obtained optimized parameters $W$ and $b$. Note that their values are similar to the original values used when generating the dataset ($W=2, b=1$)" |
| 773 | ] |
| 774 | }, |
| 775 | { |
| 776 | "cell_type": "code", |
| 777 | "execution_count": 19, |
| 778 | "metadata": { |
| 779 | "colab": { |
| 780 | "base_uri": "https://localhost:8080/" |
| 781 | }, |
| 782 | "id": "US6q0nCBD-LL", |
| 783 | "outputId": "c804b779-3231-4f6f-c854-032d211b2853" |
| 784 | }, |
| 785 | "outputs": [ |
| 786 | { |
| 787 | "data": { |
| 788 | "text/plain": [ |
| 789 | "(tensor([1.8617], requires_grad=True), tensor([1.0711], requires_grad=True))" |
| 790 | ] |
| 791 | }, |
| 792 | "execution_count": 19, |
| 793 | "metadata": {}, |
| 794 | "output_type": "execute_result" |
| 795 | } |
| 796 | ], |
| 797 | "source": [ |
| 798 | "w,b" |
| 799 | ] |
| 800 | }, |
| 801 | { |
| 802 | "cell_type": "code", |
| 803 | "execution_count": 20, |
| 804 | "metadata": { |
| 805 | "colab": { |
| 806 | "base_uri": "https://localhost:8080/", |
| 807 | "height": 282 |
| 808 | }, |
| 809 | "id": "_e6xRMZFDnyI", |
| 810 | "outputId": "79e6c360-265a-401d-ce39-8f211917a13d" |
| 811 | }, |
| 812 | "outputs": [ |
| 813 | { |
| 814 | "data": { |
| 815 | "text/plain": [ |
| 816 | "[<matplotlib.lines.Line2D at 0x20b8e30a850>]" |
| 817 | ] |
| 818 | }, |
| 819 | "execution_count": 20, |
| 820 | "metadata": {}, |
| 821 | "output_type": "execute_result" |
| 822 | }, |
| 823 | { |
| 824 | "data": { |
| 825 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAArG0lEQVR4nO3deXhU1f3H8fdJiBBBCSKLBBGUxQUEBK02rmhdQePSuvxcaq1YxSouSLQu4BpFpXWpihbBrW5IQEGpNoqIoobFFVFBUYMKFMMaIAnn98fNxGEyy53Mncm94fN6njwNmTt3zvU+/d4z53zP9xhrLSIi4l9Zjd0AERGJT4FaRMTnFKhFRHxOgVpExOcUqEVEfK5ZOk668847265du6bj1CIiTdLcuXNXWmvbRXstLYG6a9eulJWVpePUIiJNkjFmaazXNPQhIuJzCtQiIj6nQC0i4nMK1CIiPqdALSLic2nJ+hAR2ZaUzC9nzIxFLKuopFNeLiOO6UVh/3zPzq9ALSKSgpL55Vz70idUVtUAUF5RybUvfQLgWbDW0IeISArGzFhUF6RDKqtqGDNjkWefoUAtIpKCZRWVSf29IRSoRURS0CkvN6m/N4QCtYgIzlhzQXEp3YqmUVBcSsn8clfvG3FML3Jzsrf6W25ONiOO6eVZ2zSZKCLbvFQmBEOvK+tDRCSN4k0Iugm4hf3zPQ3MkTT0ISLbvExMCKYiYaA2xvQyxiwI+1ljjBmegbaJiGREJiYEU5EwUFtrF1lr+1lr+wEDgA3A5HQ3TEQkUzIxIZiKZMeojwQWW2tjFrgWEQmaZCYEw5eLt87NwRio2FCVlknEEGOtdX+wMeOBedbaB6K8NhQYCtClS5cBS5cqlotI0xKZHRIpNyebO07p06BgbYyZa60dGO0115OJxpjtgBOBF6K9bq0dZ60daK0d2K5d1G2/RER8L14+dbTskHBeLx0PSWbo4zic3vTPnrdCRMQHEuVTu8kCSUemSDLpeWcC//a8BSIiPpGowJKbLJB0ZIq4CtTGmO2B3wEved4CERGfSJRPHS07JFy6MkVcDX1YazcAbT3/dBERH+mUl0t5lGAd6iVHZodkKutDS8hFRGqNOKZXvayOyF5yupeLR6NALSJSKxMFlhpCgVpEJExj9JgTUVEmEREvzJsHjzySllMrUIuIpOLrr+GMM2DAABg9GjZs8PwjFKhFRBrip59g2DDYay94+WW4/npYuBC2397zj9IYtYhsE8KLKaU0SbhmDYwZA2PHwqZNcOGFcOON0LGj942upUAtIk1CvEDc0K22Qucsr6gkt6aaM+dN46/vP0+b9avh9NPh1luhe/e0X5sCtYgkxbOeqcdtihWIAa56/iNqIiqFJtpqK3TOTZs2c/LnM7lq1lN0XrOcWbv14+9n/Ilzhp1CYffMXHdSZU7dGjhwoC0rK/P8vCLSuKKV+UyltKdXCopLo64ozMvNYVP1lrgV7/LzcqM+dAru+C89583impkT2WvFt3zcsTt3HvZHZnftV/e+2UWDPLuGeGVO1aMWEddS3QQ2XWLV6KiorIr7PgN1AX6r4ZBN33PvQ8P5zfef8k2bXbj0xGuYtufBWPNr/kUm91NUoBYR1/y6CWysGh3xGCByPKHTj9/S5uxb4fPZdG/VhuuPvoRn9z2a6uz6oTKT+ykqUIuIa7ECYuvcHAqKSxtt3DpWjY4WOVn8sqF+rzrbmK3GrDuuWcnw2c/w+0/eYENOc7jlFt494g9Mem0J1VGGTTK9n6LyqEXEtWhlPnOyDOs3V1NeUYnl1yGE8J1RvBa5CwvAHaf0IT8vF4MzfnzHKX24acg+UTetvecPfcnPy6V15VqK3hzPW48O5eTPSnl8wBBOv+ZpuP56hhT0rDsnOMGdsHNn8kGkHrWIuBataNGGzdX1eq3pHLeOleFxxyl9Yk7u1ctS6dWGHhPeoPO4+9lh43om9z6Cew8+m1U778Idp/TZ6nobO6MFlPUhIinqVjSt3lgvOGPA3xSf4PnnxcrwcJWFUV0Njz8Oo0bBsmX8dMhRjNjvdN5psUujpxoq60NE0iZRsX2vNWhC01qYPBmuuw4WLYKDDoJnn6XjIYfwZFpa6S2NUYtISqKNW6dzsi3WAyDmg+Gtt5zAfOqpkJ0NJSUwezYcckha2pcOCtQikpLC/vlRJ/LSNYTg+sHw0Udw3HFwxBFQXg7jx8PHH8NJJ0HtxGBQaOhDRFKWyUm3hLuwfPMN3HADPPMM5OU5BZSGDYPczOU9e81VoDbG5AGPAb1xcsT/ZK19L43tEhGJKeqDYflyp0jSww9Ds2YwcqTzk5fXKG30ktse9T+A16y1pxljtgO8L7gqIpJA1IJQ3XeEe++Fu++Gykq44AK46Sbo1Kmxm+uZhIHaGLMjcCjwRwBr7WZgc3qbJSKytcj86eX/W8On197KcR+8QPNf/udMFt52G/TK3IrBTHHTo94dWAE8bozpC8wFLrfWrg8/yBgzFBgK0KVLF6/bKSKNrLHLm4YKQhm7hSEL3+aqWU+xW8VPzN29HwNemw4HHJCxtmSam6yPZsB+wEPW2v7AeqAo8iBr7Thr7UBr7cB27dp53EwRaUyh3mwml4mHf3ZBcSnlv2zgsCVzeWXCcO57+W7Wb5fLub8fzWmn3dKkgzS461H/APxgrX2/9t8vEiVQi0jTFau86aipn8XdVSXVHnjoAdFz6eeMmTmR3373Md+17sBlQ67m5b0OxZqsulocTVnCQG2t/ckY870xppe1dhFwJPB5+psmIn4Rr95zqOZz5K4qDdn6KtKzT73B3dMe5YRFs1m5fWtuOuoinul3LFXZOUDmq9g1FrdZH38Fnq7N+FgCnJ++JomI37it9xwqxhT6PdprrgL1smUwejRPPfoYG3OaM7bgLB7bv5D1zX9NOMv3yTZgmeAqUFtrFwBRi4WISNMXrd5zLPFqbiTcYKCiAu66C/7+d6iuZvKBJ1E84FT+1zJvq8O83gbL77QyUUQSclveFH6tuZGoUFP4GHbXltncv2o2vSc+6ATrs86Cm28mZ3VzNrz0CURsCLAtDHeEU6AWEVciVwPG2ug2FETjvRa+w/dpn5ZyxTtP02ntSn7+7eF0eHAs9OtXF8grq2rqdmTZloY7wilQi2zDUsnMSFhzI85rY177goM/n82ImU/Q83/fsWCXnlw5+Eq+3/c3zK4N0uGBvsbaukC/rQVp0MYBItusWD3itG8zNWsWZWdexMDyhSzeqTN3HXouM3oeVFfRLj/OxGW0senGXojjFW0cICL1xMqNTtcWWnzyiVO4/5VX2G3HthQdcykv7Ps7arJ+LVlqiD62HRI5GRlrWy5ILg3Q71SPWmQb1aCdUhpi6VI47zzo2xdmzYLiYua8Oocp+59QL0gn+n4fuTlAvIdNU6JALdLERe7YHVr2HWtHFAtbHddgK1fClVdCz57w3HNw9dWwZAmMHMmQ33bfarOBvNychEE6WrZHxh42jUyBWqQJi1ejI9pOKSEp1fJYv96pC73HHvCPf8DZZzNj8iwKdjqWbne9V/cQKOyfz+yiQYw9vR+bqrfEPWWsXWOS3pYroDRGLdIEhSbYoo33hoYGQpNyiY5zPdZbVQWPPQY33ww//QSFhXDbbZRsah13HDna8EVIosnNaAtxcnOyOWLPdhQUlwZ+gjFEPWoRD8QaXmistoR60bGEhgZCvdpYOwi6GkLYssUZ2th7b7jkEujRA95919n1e++9E44jx/uMRBko0fZrPHVAPpPmljdKpb90UY9aJEV+yzyI10MNiRwaiFXLI+EQwhtvQFERzJ0LffrAK69QsktfxvznS5ZNmRa3RkgoQMc6Jj8v19V/v8iFOAXFpZnNZskA9ahFUtQYmQfxevCJesHRJuVc7+wdMncu/O53zs/KlfDEEzB/PiWd+nHt5E+36s3G6q2HHgJJf3YCbiYY/fQNyA31qEVSlOnMg0Q9+Hi92GxjtnqIhHqYblYZAvD113D99c5QR9u2MHYsXHwxNG9e9/7Ih5Yleurdhs3VdZOKrj7bpUTfDvz2DcgNBWqRFDV42KCBEi1UiTbBlpNlwEBVjRMuowWnqDt7h/z0kzNJ+OijsN12TrC++mpo3Xqrw2I9nCxOCl6odjXALxuqtmqDV0Ey1gRjqIee8YU+HtDQh0iKvP7qnkiiHny0CbZWLZrVBemQWMMz4cMCvxv1MosuHO6k2j36KAwdCosXwy231AvSEPvhlJ+XS8vm9fuF6Rgiinb94ZOSQcy9Vo9aJEVef3VPxE0PPrKH2q1oWtRzxVqSXVO5kfPnT+fS955jp8o1/HD0iXR+8B7o3j1u2+L1Zq94boGrNnghXg8909+AvKBALeIBL7+6J5Loq300boPTPa9+znHz/8OVs56m85rlvN21P3cddh6/7NmH2QmCNMR/aMXK1850gGzIf7/GpkAtEjAN6cEnDE7WwvTpPDp2GHuuXMrHHbsz8rjLmN21HwAmiV5vrIeWXwJkpr8BeUGBWiSAku3Bxw1O770HI0fCrFm0bJvPsBNHMn3PAqz5dQrLi16vnwJkJr8BeUGBWsSnvK6zXC84ff45FA6DKVOgY0d46CHm7XcspS9/gU1TrzdoAdIvFKhFfCitub7ffw+jRsGECdCqlVNAafhwaNmSkwCbk+OLXq/8ylWgNsZ8C6wFaoDqWLsQiIg30pLru2oV3HEH3H+/MyY9fDhcey3svPNWh6nX6z/J9KiPsNauTFtLRKSOp7m+GzbAffdBcTGsWQPnngujR8Nuu6XYSskUDX2I+EDkeHTe9jn8sqGq3nFJTepVV8P48U5QXrYMBg+G2293iidJoLhdmWiB/xhj5hpjhkY7wBgz1BhTZowpW7FihXctFGniohX3X7exmpzsrcsZuZ7UsxYmTYLeveGii6BrV2cLrJdfVpAOKLc96gJr7TJjTHvgdWPMF9bat8MPsNaOA8aBswu5x+0UaTIie88bNlfXG4+u2mLJy82hZfNmyU3qvfmmU3b0gw+c+tBTpsCQIXU7fEswuQrU1tpltf+73BgzGTgAeDv+u0QkUrRsjlhWV1ax4Kaj3Z14wQJnYvC112DXXZ0hj3PPhezoW21JsCQc+jDGtDTG7BD6HTga+DTdDRNpitwU9Q9xNR69ZAn83/9B//7w/vtw993w5Zdw/vkK0k2Imx51B2Cycb46NQOesda+ltZWiTRRbrM2Eo5HL1/u5D8//DA0a+b0pq+5BvLyvGmo+ErCQG2tXQL0zUBbRJq8WMWRXI9Hr10L99zj/FRWwgUXwE03QadOGWi9NBal50kgeb28OlNiFSYadeI+MdtfMr+csdM+5YiZk7ns3WfZacNqpvcq4MkTLuT0s4+iUEG6yVOglsDJ5FZKiR4IyT4wki1MVDL3e9659QGefHMiXVb/zLtd9uXOw87jo07OsMiCiOsO6gNM4jPWep9JN3DgQFtWVub5eUXA2WU61q7Vs4sGefY5kQ8EcHq/od1CEr2eEmthxgy+/NNf6fnj13zWfnfuPOw83u62X71Uu9B1p7U9knbGmLmxynNoKy4JnExtpZRod/G07T7+wQcwaBAcdxzNK9dx2ZARDP7j33l79wFR86FD190Yu6FLZmjoQwInU1spJXogeP7AWLQI/vY3Z1Vh+/Zw//2cW9GDpeuq474tdN1B3AtQ3FGPWgInU5vJxgr8ob8net218nJn09h99oEZM5zaHF9/DZdeyhUn9K53reHCr9uz9ojvKFBL4CTaZdoriR4IKT8wKiqc/OcePZza0MOGOTt833gj7LADUP9a83JzaLN9TtTrTscDLHxH8oLiUkrmlzf4XNJwmkwUicPrrA/AyX9+4AGnNnRFhbOy8OaboVu3tLc32XNpcjJz4k0mKlCLZEp1NTzxhLNA5Ycf4LjjnGDdN7X1ZOlKyctUdo044gVqTSaKpJu1ThW7666DhQvhgAPgySfh8MPrHZps0E1nTrkmJ/1DY9QiETwdl501CwoK4OSTYcsWJ6NjzpyYQTqyLvW1L30S9/PTmZKnyUn/UKCWQEn35FZDgmVUn3zi7Khy6KGwdCnzb7iLQ8+5n24fNKfgzjejnq8hQTedvd5MZddIYgrUEhieBdE4Uu6hLl0K553njDvPng3Fxbz8wkzOsn34bu3muO1uSNBNZ683U9k1kpjGqCUw0rIzd4QG91BXroTbboN//hOysmDECGenlTZtKC4uddXuhizkiVXkyater3Yk9wcFagmMTExuJR0s16+HsWPhrruc388/H0aNgs6dE7avvKKSguLSuonDI/Zsx6S55UkF3WSLPEkwKVBLYGRi6bjrHmpVFTz6qJP//PPPUFjo7PC9116u2234dSuu8opKJs0t59QB+bz5xYqkgq56vU2fArUERrq/5kP8HmrJ/HLufnUh/d5/naJ3nqLzqmXOZOHkyXDQQUm12wCRKxgqq2p484sVGclRVjnUYFGglsDI1Nf8aD3UkvnlvHz3RB7673j6/LyYhe268sfTbuKj3gdxU4suFEY5T3gwbJ2bQ4ucLCo2VMXsYUNmcpQzWc9bvKFALYGS6Gt+WnqKZWXkn3ER/1o8jx92bM8VJ1zJlL0PY0tWNlRWRw1ykcGworKK3Jxsxp7ej8L++TFX/WUiRzkTk7LiLaXniSf8ULzH8/S9r76C00+H/fdn92VfM/rICxl04SNM7j3ICdK1oqXvJUrza8wcZa04DB7XPWpjTDZQBpRbawenr0kSNH75Kh0rOF71/EfJteWnn5xJwkcfhebN4YYbOCNrf77aGLtfExnkEgXDxszWyFQ9b/FOMkMflwMLgR3T1BYJKL98lY4VHGusdffgWL0axoxx0u02b4aLLoIbboAOHRgWpZJcuPAgVzK/nCxjqIlS8Cz8uMbK1sjEpKx4y9XQhzGmM3AC8Fh6myNB5Jev0vF6hHFXF27c6ATnPfZwFq2ceKJTPOmBB6BDB+DXVXp5uTn13h4e5ELfLqIFab8EQ604DB63Peq/A9cAO8Q6wBgzFBgK0KVLl5QbJsHhl6/S0XqK4eo9OGpq4KmnnEL9330HRx/tlB3db7+o7w/1gONNWEb7dgGQbYyvgqFyr4MlYaA2xgwGlltr5xpjDo91nLV2HDAOnHrUXjVQ/M8vX6VDgeeq5z+KP+xgLUyb5uyu8umnMHAgjB8PRx7p+nNiBblY3yK2WKvAKA3mZuijADjRGPMt8CwwyBjzVFpbJYHip6/Shf3zuecPfWNnVLz7rrNIZcgQZ8jj+eedXb9dBulEVBpU0iGpHV5qe9RXJ8r60A4v0tgihydu7g5HPvkPmDoVOnZ0dlm54ALIqT/mnOrnavsqaQjt8CLbnLrhie+/d4Ly3yZCq1bOZOHll0PLlmn7XFCRJPGW9kyURpPWehOrVjkTg/ff74xJX3qpsxVW27bp/2yRBlCPWnzHy0Uy4UG32/aG+1fMYp8nH4I1a5wi/qNHQ1gmUqzPLlu6KunKdSKZoB61NIpYtS7AmYx0GyRDQXfzps384ePXGT77GTqsW8WPh/6OXR68F3r33urYMTMWxfzcyIp2GluWTFKPWnwn3mKYZHrXY177gsM/mcnVs55kj1XllOXvxbCTRvJj74HMrg3SJfPLGTX1Myoqq+KeK1rZURUqEj9QoJZGEa/UJ7gMkm++yT/vv5i+P37Fl2278OdTbuCN7geAMZjac0fLwkiGChWJHyhQi2teTsAlWkUIcYLk/PnOYpUZM+jQuj1XHz+cl/Y5YquKdqG85VgrBSNFK+Qffh6RxqQyp+KK1yVEwxfJxFIvSC5ZAmed5Szx/vBDuPtuPnjtXabtd8xWQTp8VaSbHnF+Xi7/d2CXRis7KpKIetTiSjoq5IXXzoi7BP3nn+HWW+GRR6BZMyfNbsQIyMvjRGBL8xYxe/rxhlgiJwsH7raTUvbElxSoxZV0VsiLuUik+47OYpV77nGWe//5z04BpU6d6r0/VkCNNcTSZvscbhqyz1bvU6Ei8SsF6iYgE4s30l0hb6sguWmT03s+5lZYsQJ+/3unR92zZ4POC1opKMGmQB1wmdpdJSMV8rZsgX//G66/Hr79FgYNguJi2H//lE6rnrIEnSYTAy7R3nxeSWuFPGvh1VedScKzz4a8PJgxA954I+UgLdIUqEcdcJncXSUtPdP334eRI2HmTNh9dz68/QGusL0oL91E6/dexxj4ZUMV2bVbWyWzalGkqVCgDji/7K6SSOQ4+uie2Rz19H3w0kvQvj088ABT9j+eopcXUVm1CWCrlYShjQAaa+NckcakoY+AG3FML9/n/4bnYLdfu5JL/30nh582iKrXZjgFkxYvhmHDuKv0G1eLU9IxtCPiZ+pRB1wQshrGzFhEztrVXDbnRc6fO5WsLVt4Yr/BPPDb08ndriMjvlpNYf9WSQ3XaGm3bEsUqJsAX2c1VFYyZMaT/GXOi+y4cT0l+xzOvQf/Hz/kdXReDxvKSFT/I5zfhnZE0kmBWjxXMr+ce6d/zkHvvMJV7z5D0ZqVlO4+kLsOO48v2nerd3xoKMNN/Q/w39COSLopUIunSub9wBvF4/hX6QR6/O975u/SiyuGXM0Hu/WhqiZ27fNlFZX1hnFa5+Yo60MEBWrx0ttvs8c5F1P43ed8vVNnLjr5Omb0OAiMIW+7ZrRs3izm0EZoKCPTwzjakkuCQIFaUvfxx07Z0enT2blVW0Ye+1de7HMUNWEV7VZXVrHgpqMTF2DKoEyt6hRJVcJAbYxpAbwNNK89/kVr7U3pbpj8yre9vm+/dYokPfUUtG4Nd97JWZv68M36LfUODe8xgz+yVNJREVAkHdz0qDcBg6y164wxOcA7xphXrbVz0tw2wae9vhUr4Lbb4KGHICvLKTlaVARt2nC5ix6zX7JUMrmqUyQVCQO1dXa/XVf7z5zaH+93xJWovOz1pdwzX7cOxo6FMWNg/Xr405+cMqSdO9cd4qcecyJBWdUp4moXcmNMNjAX6A48aK0dGeWYocBQgC5dugxYunSpx03dNnUrmhb1qWiAb4pPcH2eaGPDoe2nomVShAf1Lq2acd+6MvpOuN8p4n/yyU6Peq+9GnxdfhBrvFw7j0tjSHkXcmttDdDPGJMHTDbG9LbWfhpxzDhgHMDAgQPV4/aIV72+aD3z0E2KHE4JBbCNm6sYvHAWV816iq4VP7JywIHsXFJCSfNdGTNlEcsmLqlLoavYUOXr3nM0Qer9y7YtqawPa22FMeYt4Fjg0wSHN1mZnNzzqg50onHX8OGUMTMWMeDLMopmTqD3z4tZ2K4rfzxtFF/tdzAjmu+6VXvCCyf5Yvw8SX4ZLxeJx03WRzugqjZI5wJHAXemvWU+lenJPa96fW6WZy+rqISyMu58+EoOXvoRP+zYnuGDr2LK3odhTRZm9caEu3ora0LEe2561LsAE2vHqbOA5621r6S3Wf7VGCldXvT6Ei3P7rqqnBvmPAN3zmSflq0ZfeSFPN3veDY3y6k7JssYV7U4lDUh4i03WR8fA/0z0JZACGJKV2ioprKqpm4pdmgisd26VVw++9+c8dEMaNECbryR2UedybP/+ZbNEUE9/H3xhMbPfZv/LRIwWpmYpKCldEUO1dRYS25ONmftuSNdHnuQ38+exHY11Xz3+3PY/b47oUMHBgPVrXbgquc/qivYH2IhbrAOjZ/7Mv9bJKC0cUCS/FSov2R+OQXFpXQrmkZBcSkl88vrHRM5VNO8ejNnzX6Ryy48lvPeeobtTzuFZl8uYvfnJkCHDnXHFfbPZ0uM1M1QSp8B8nJzaLN9Tr19FDO1l6PItkA96iT5JaXLbY81NCSTtaWGUz57k+HvPE3nNSt4u2t/Dp30mLOhbAyxvj3k5+Uyu2hQ3PYFcYhIxK8UqBvADyldbic1O7VuwZ5zZ3LNzIn0WvkdH3XswYjjh/Nd3wOZHSdIQ2qpgUEbIhLxMwXqNEvXhJqrHuvs2Ux9/lraLviQJW06cclJRUzvVUDuds24w0WwTeXbg1f53yKiQJ1W6ZxQi9tj/ewzuO46mDqVtrvswoLrirls+/58v7Yq6cL7Df324JchIpGmwFWtj2QNHDjQlpWVeX7eoCkoLo0aTLON4Z4/9E0paEWrU7H7hv/x+LfT2G3ai9CqFYwcCZdfDi1bNvhzRCQzUq71IQ0Ta3iixtqUe9bhPdYNP/7MiPmTOf39qWQb4IornEL+bds26Nwi4i8K1GkUb9m2F6sZC3vmUfjqHJhwp1OC9NxzYfRo6NKlQefTAhURf1IedRpFy7kO1+BUtaoqeOQR6NED/vY3OPxwZzusxx9PKUhf+9InlFdUYvl1PD1abraIZJYCdRoV9s/njlP6kG1M1NeTTlWzFl54AfbZB/7yF9h9d3jnHZgyxflbCrRARcS/NPSRZqGhg5RT1UpLne2uPvzQCcpTp8LgwRDjIZDsMIYWqIj4lwJ1BrhJVYsZWOfPdwL0f/4Du+4KEybA2WdDduwhlYakBWqBioh/KT3PB6Kl2vVc+zP/WjKVXV+bAjvt5IxFX3KJU+EugVhpgfGWfmtbKpHGpfQ8nwsfH955/S/89d1nOWvBa9RkN3MWrlxzDbRu7epcJfPLY2aaxBvG0AIVEf9SoPaBZRWVtNq0gQs/mMyfP5xM8+rNPNv3GO4rOJMPbjvH9XlCveJYEg1j+KGGiYjUp0Dd2DZt4vLPpnPOf5+ibeUaXul1MPcceg7f7JRPvgcb2IaozoZIcClQp6jBi0S2bIFnnoEbbmD4t9/yXte+3H7oH/lklx6AE1iP2LMdBcWlKWduABprFgkw5VGnoEGLRKyF6dOhf3845xxo0wZmzODnSa+waq996wrwnzogn0lzy5M6d6yhjfy8XAVpkQBrsj3qVJdDu3l/0hvdzpnjpNrNnAl77MGHdzzIFVt6Ul66iU55X271GQXFpUlvoqvSoiJNU8JAbYzZFXgC6AhsAcZZa/+R7oalItXyosnunhKp3t+/+MLJ3pg8Gdq3hwceYMr+x1P08iIqqzZF/YyGLEBpSOaG6nuI+J+bHnU1cJW1dp4xZgdgrjHmdWvt52luW4Ml3dNt4PsTLhIpL4dRo2D8eKfU6M03O5XtWrXirgQ95oYuQEkmc0Mb0IoEQ8Ixamvtj9baebW/rwUWAr7+f3Gqy6Hdvj/WRrfXHdTBqQXdvTtMnAh//SssXgw33ODUiXbxGZnYRFf1PUSCIakxamNMV6A/8H6U14YCQwG6NLCCm1dSXQ7t9v2RQw1dW2Zx/8pZ9C78J6xe7Sz1vvlm6No16c/IxAIU1fcQCQbXgdoY0wqYBAy31q6JfN1aOw4YB84Scs9a6ELkOOsRe7Zj0tzyBk+qJTMpV9g/n8I+HZwaHKNGOcMdxx8Pd9wB++6b0mekewGK6nuIBIOr9DxjTA5OkH7aWvtSepuUnGgpcpPmlnPqAGfBSCjdLZk84lB50oTvt9aZIOzTBy68kE+zduT0s4opOOQqSmri767i+jPSKBPDKyKSuoRFmYwxBpgIrLLWDndz0kwWZWpIAaJkxMyKmDnTSbWbM4e1Xbtz7f5n8kq3A+rKjgaloJGyPkT8IV5RJjeB+mBgFvAJTnoewHXW2umx3pPOQB0ZWGIVIDLAN8UnJH2+8EAVraLcXsu/YcTMiQxaUkZl+47k3n4rh/7Uhe/Wbq53bq8eFiLS9KVUPc9a+w5O3Gt00dLJDBDtUeNmnDVRelp4VkTn1T9z5aynKPzsLdY2357bDz+f539zEqP2G8j3zy2Ien5NyomIFwK1MjFaOpmFesHa7ThronzpZRWV7LRhNZe++xxnz5/OlqwsHvnNqTx04GmsadGq7hyalBORdApUrY9YPVQLdZNyebk5tMjJ4ornFlBQXBq3Nkbc9LR16/jb3BeY+cifOW/eK0zqPYjDLxzHnYf/sS5Ih47VpJyIpFOgetSxeq6hseBkV9pFO19OTRUXLyqFPf7En5cv5z97FnDnwWezuO2uMdukovsikk6BCtSJco+TXToefj5jtzBk4SyunvUkXSp+gsMOgylT2NB8VzbOWARRxsPDP1tF90UkXQIVqBP1XJNdaVfYPx+sZeYDz3DB9HH0/nkxq3vsBc+Mh2OPBWMoDPtcP6Sy+aENIpJZgQrUEL/nmvSk3ocfUlhURGFpqbPM+8knaX3WWZAVfei+sXvNKqIksm0K1GRiIq4n9b78Ev7wBzjgAPj4Y/jHP5xSpGefHTNIe6lkfjkFxaV0K5qWcMIznIooiWybAtejjifhpN6PP8Lo0fDYY9CiBdx4I1x1Fey4Y8bamEqvWEWURLZNgQ7UscZr6wW81avhrrtg7FioroaLL4brr4cOHVydz6t2QWq1spWvLbJtCmygdtUz3bgRHnwQbr8dVq2CM8+EW26BPfZo2Pk8aFeiXnG8IK+ttkS2TYEdo447XltT45Qd7dkTrr4a9t8f5s1zdv2OEqQTns+rdhG799spLzfhZrl+qLgnIpnnux612+GHqD1Ta9n7w7eg72Xw2WdOgJ4wAQYlLozk1fivm51bYvWK3QyLNHbmiYhknq8Cdaxhg7Klq3jzixVbBe/I8dqBP3zGyLcmsn/5505P+oUX4NRT68qOJuLV+G8qO7dcoeJOIhKFrwJ1rB7l03O+q1sRGArepw7IZ9LccnZdtpgRbz/B777+gOWtdmL+34rpf9OVkJOT1Gd7Nf6bys4tmiwUkWh8FajjFV0KV1lVw+dzPuW1z0vY9ZUXWLfd9jx8zAXk31jEkN92b9Bne1WvI5XzaLJQRKJJuHFAQzR044BYu7WEy6tcw7D3nufcedNo3izL2eG7qAjaxt/6Kii0RFxk25TSxgGZFK1HGSqElLt5I38qm8JF70+iZdVGXt3vaAa/9Ag08o7nXtNkoYhE8lWgjjZscGT3PLLGP84lbz9N+/W/8Hr333DfkedzwUWDoYsCmog0fb4K1BDWo7TWydy47i/w1Vd81LU3lxRey4+9B2g4QES2Kb4L1AD897/OuHNZGeyzD0ydSt/Bg3nRZaqdiEhTkjBQG2PGA4OB5dba3mltzbx5ToB+/XVn7HnCBKeiXXZ2wre6oYk6EQkiN0vIJwDHprkdUFEBhxwCc+fCvffCokWU7HsUBWNmJl0ONJpEy7NFRPwqYY/aWvu2MaZr2luSlweTJ8NvfgOtW3teJD+VqnUiIo3JX0WZjj4aWrcGvC+Sr1rOIhJUngVqY8xQY0yZMaZsxYoVKZ/P68Aar2qdiIifeRaorbXjrLUDrbUD27Vrl/L5vA6srrfpEhHxGX8NfYTxMrCGsj0qq2rIrk3xy8vNoUVOFlc8tyDliUoRkXRKGKiNMf8G3gN6GWN+MMZckP5meVckPzzbA6DGWnKyDOs3V/PLhiplgIiI7/mqKFM6uCn0FJKfl8vsosSbDIiIeC1eUSbfDn14JZnJR2WAiIgfNflAnczkozJARMSPmnygjjYpmZNlyMneum6IMkBExK/8WZTJQ7F2XIn2N61QFBE/avKTiSIiQRCIHV5U2U5EJDpfBGqvCzCJiDQlvphM9LoAk4hIU+KLQK3KdiIisfkiUKuynYhIbL4I1KpsJyISmy8mE2PlOmsiUUTEJ4EanGCtwCwiUp8vhj5ERCQ2BWoREZ9ToBYR8TkFahERn1OgFhHxubRUzzPGrACWNvDtOwMrPWxOY2oq19JUrgN0LX7UVK4DUruW3ay17aK9kJZAnQpjTFmsUn9B01SupalcB+ha/KipXAek71o09CEi4nMK1CIiPufHQD2usRvgoaZyLU3lOkDX4kdN5TogTdfiuzFqERHZmh971CIiEkaBWkTE5xolUBtjjjXGLDLGfG2MKYryujHG3Ff7+sfGmP0ao51uuLiWw40xq40xC2p/bmyMdiZijBlvjFlujPk0xutBuieJriUo92RXY8ybxpiFxpjPjDGXRzkmEPfF5bUE5b60MMZ8YIz5qPZaRkc5xtv7Yq3N6A+QDSwGdge2Az4C9o445njgVcAABwLvZ7qdHl7L4cArjd1WF9dyKLAf8GmM1wNxT1xeS1DuyS7AfrW/7wB8GeD/r7i5lqDcFwO0qv09B3gfODCd96UxetQHAF9ba5dYazcDzwInRRxzEvCEdcwB8owxu2S6oS64uZZAsNa+DayKc0hQ7ombawkEa+2P1tp5tb+vBRYCkUXbA3FfXF5LINT+t15X+8+c2p/IrAxP70tjBOp84Puwf/9A/Rvm5hg/cNvOg2q/Jr1qjNknM03zXFDuiVuBuifGmK5Af5zeW7jA3Zc41wIBuS/GmGxjzAJgOfC6tTat96UxdngxUf4W+TRyc4wfuGnnPJw1/OuMMccDJUCPdDcsDYJyT9wI1D0xxrQCJgHDrbVrIl+O8hbf3pcE1xKY+2KtrQH6GWPygMnGmN7W2vA5EU/vS2P0qH8Adg37d2dgWQOO8YOE7bTWrgl9TbLWTgdyjDE7Z66JngnKPUkoSPfEGJODE9ietta+FOWQwNyXRNcSpPsSYq2tAN4Cjo14ydP70hiB+kOghzGmmzFmO+AMYGrEMVOBc2tnTg8EVltrf8x0Q11IeC3GmI7GGFP7+wE4/83/l/GWpi4o9yShoNyT2jb+C1horb03xmGBuC9uriVA96VdbU8aY0wucBTwRcRhnt6XjA99WGurjTGXAjNwsibGW2s/M8b8pfb1h4HpOLOmXwMbgPMz3U43XF7LacDFxphqoBI4w9ZOC/uJMebfOLPuOxtjfgBuwpkkCdQ9AVfXEoh7AhQA5wCf1I6HAlwHdIHA3Rc31xKU+7ILMNEYk43zMHneWvtKOmOYlpCLiPicViaKiPicArWIiM8pUIuI+JwCtYiIzylQi4j4nAK1iIjPKVCLiPjc/wPCCvjKMZGi9wAAAABJRU5ErkJggg==", |
| 826 | "text/plain": [ |
| 827 | "<Figure size 432x288 with 1 Axes>" |
| 828 | ] |
| 829 | }, |
| 830 | "metadata": { |
| 831 | "needs_background": "light" |
| 832 | }, |
| 833 | "output_type": "display_data" |
| 834 | } |
| 835 | ], |
| 836 | "source": [ |
| 837 | "plt.scatter(train_x,train_labels)\n", |
| 838 | "x = np.array([min(train_x),max(train_x)])\n", |
| 839 | "with torch.no_grad():\n", |
| 840 | " y = w.numpy()*x+b.numpy()\n", |
| 841 | "plt.plot(x,y,color='red')" |
| 842 | ] |
| 843 | }, |
| 844 | { |
| 845 | "cell_type": "markdown", |
| 846 | "metadata": { |
| 847 | "id": "0giuwC9GHzi8" |
| 848 | }, |
| 849 | "source": [ |
| 850 | "## Computations on GPU\n", |
| 851 | "\n", |
| 852 | "To use GPU for computations, PyTorch supports moving tensors to GPU and building computational graph for GPU. Traditionally, in the beginning of our code we define available computation device `device` (which is either `cpu` or `cuda`), and then move all tensors to this device using a call `.to(device)`. We can also create tensors on the specified device upfront, by passing the parameter `device=...` to tensor creation code. Such code works without changes both on CPU and GPU: " |
| 853 | ] |
| 854 | }, |
| 855 | { |
| 856 | "cell_type": "code", |
| 857 | "execution_count": 21, |
| 858 | "metadata": { |
| 859 | "colab": { |
| 860 | "base_uri": "https://localhost:8080/" |
| 861 | }, |
| 862 | "id": "HK7HPLz3Hyrl", |
| 863 | "outputId": "7e14cccb-d376-4e59-be66-4ab3f5c3f6f4" |
| 864 | }, |
| 865 | "outputs": [ |
| 866 | { |
| 867 | "name": "stdout", |
| 868 | "output_type": "stream", |
| 869 | "text": [ |
| 870 | "Doing computations on cpu\n", |
| 871 | "Epoch 0: last batch loss = 94.5247\n", |
| 872 | "Epoch 1: last batch loss = 9.3428\n", |
| 873 | "Epoch 2: last batch loss = 1.4166\n", |
| 874 | "Epoch 3: last batch loss = 0.5224\n", |
| 875 | "Epoch 4: last batch loss = 0.3807\n", |
| 876 | "Epoch 5: last batch loss = 0.3495\n", |
| 877 | "Epoch 6: last batch loss = 0.3413\n", |
| 878 | "Epoch 7: last batch loss = 0.3390\n", |
| 879 | "Epoch 8: last batch loss = 0.3384\n", |
| 880 | "Epoch 9: last batch loss = 0.3382\n" |
| 881 | ] |
| 882 | } |
| 883 | ], |
| 884 | "source": [ |
| 885 | "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n", |
| 886 | "\n", |
| 887 | "print('Doing computations on '+device)\n", |
| 888 | "\n", |
| 889 | "### Changes here: indicate device\n", |
| 890 | "w = torch.tensor([100.0],requires_grad=True,dtype=torch.float32,device=device)\n", |
| 891 | "b = torch.zeros(size=(output_dim,),requires_grad=True,device=device)\n", |
| 892 | "\n", |
| 893 | "def f(x):\n", |
| 894 | " return torch.matmul(x,w) + b\n", |
| 895 | "\n", |
| 896 | "def compute_loss(labels, predictions):\n", |
| 897 | " return torch.mean(torch.square(labels - predictions))\n", |
| 898 | "\n", |
| 899 | "def train_on_batch(x, y):\n", |
| 900 | " predictions = f(x)\n", |
| 901 | " loss = compute_loss(y, predictions)\n", |
| 902 | " loss.backward()\n", |
| 903 | " w.data.sub_(learning_rate * w.grad)\n", |
| 904 | " b.data.sub_(learning_rate * b.grad)\n", |
| 905 | " w.grad.zero_()\n", |
| 906 | " b.grad.zero_()\n", |
| 907 | " return loss\n", |
| 908 | "\n", |
| 909 | "batch_size = 4\n", |
| 910 | "for epoch in range(10):\n", |
| 911 | " for i in range(0,len(features),batch_size):\n", |
| 912 | " ### Changes here: move data to required device\n", |
| 913 | " loss = train_on_batch(features[i:i+batch_size].view(-1,1).to(device),labels[i:i+batch_size].to(device))\n", |
| 914 | " print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))" |
| 915 | ] |
| 916 | }, |
| 917 | { |
| 918 | "cell_type": "markdown", |
| 919 | "metadata": { |
| 920 | "id": "A10prCPowHl7" |
| 921 | }, |
| 922 | "source": [ |
| 923 | "## Example 2: Classification\n", |
| 924 | "\n", |
| 925 | "Now we will consider binary classification problem. A good example of such a problem would be a tumour classification between malignant and benign based on it's size and age.\n", |
| 926 | "\n", |
| 927 | "The core model is similar to regression, but we need to use different loss function. Let's start by generating sample data:\n" |
| 928 | ] |
| 929 | }, |
| 930 | { |
| 931 | "cell_type": "code", |
| 932 | "execution_count": 23, |
| 933 | "metadata": { |
| 934 | "id": "j0OTPkGpwHl7", |
| 935 | "scrolled": false, |
| 936 | "trusted": true |
| 937 | }, |
| 938 | "outputs": [], |
| 939 | "source": [ |
| 940 | "np.random.seed(0) # pick the seed for reproducibility - change it to explore the effects of random variations\n", |
| 941 | "\n", |
| 942 | "n = 100\n", |
| 943 | "X, Y = make_classification(n_samples = n, n_features=2,\n", |
| 944 | " n_redundant=0, n_informative=2, flip_y=0.1,class_sep=1.5)\n", |
| 945 | "X = X.astype(np.float32)\n", |
| 946 | "Y = Y.astype(np.int32)\n", |
| 947 | "\n", |
| 948 | "split = [ 70*n//100, (15+70)*n//100 ]\n", |
| 949 | "train_x, valid_x, test_x = np.split(X, split)\n", |
| 950 | "train_labels, valid_labels, test_labels = np.split(Y, split)" |
| 951 | ] |
| 952 | }, |
| 953 | { |
| 954 | "cell_type": "code", |
| 955 | "execution_count": 24, |
| 956 | "metadata": { |
| 957 | "id": "c-_BjSHPwHl8", |
| 958 | "scrolled": false, |
| 959 | "trusted": true |
| 960 | }, |
| 961 | "outputs": [], |
| 962 | "source": [ |
| 963 | "def plot_dataset(features, labels, W=None, b=None):\n", |
| 964 | " # prepare the plot\n", |
| 965 | " fig, ax = plt.subplots(1, 1)\n", |
| 966 | " ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n", |
| 967 | " ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n", |
| 968 | " colors = ['r' if l else 'b' for l in labels]\n", |
| 969 | " ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n", |
| 970 | " if W is not None:\n", |
| 971 | " min_x = min(features[:,0])\n", |
| 972 | " max_x = max(features[:,1])\n", |
| 973 | " min_y = min(features[:,1])*(1-.1)\n", |
| 974 | " max_y = max(features[:,1])*(1+.1)\n", |
| 975 | " cx = np.array([min_x,max_x],dtype=np.float32)\n", |
| 976 | " cy = (0.5-W[0]*cx-b)/W[1]\n", |
| 977 | " ax.plot(cx,cy,'g')\n", |
| 978 | " ax.set_ylim(min_y,max_y)\n", |
| 979 | " fig.show()" |
| 980 | ] |
| 981 | }, |
| 982 | { |
| 983 | "cell_type": "code", |
| 984 | "execution_count": 25, |
| 985 | "metadata": { |
| 986 | "colab": { |
| 987 | "base_uri": "https://localhost:8080/", |
| 988 | "height": 283 |
| 989 | }, |
| 990 | "id": "tq0vFchQwHl8", |
| 991 | "outputId": "919f1922-f789-4779-cbdc-4f9e742c358b", |
| 992 | "scrolled": false, |
| 993 | "trusted": true |
| 994 | }, |
| 995 | "outputs": [ |
| 996 | { |
| 997 | "name": "stderr", |
| 998 | "output_type": "stream", |
| 999 | "text": [ |
| 1000 | "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_89704/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n", |
| 1001 | " fig.show()\n" |
| 1002 | ] |
| 1003 | }, |
| 1004 | { |
| 1005 | "data": { |
| 1006 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABPHElEQVR4nO2dd3zcdf3HX+9cdtI0bVZX0rSllFE6IGUIlD1lCKKADAeKA1DABeJGFMWJCtIfAqKIioJI2UihjIIU6KSFriTdTbN3crn374/XfXuX5HKjuZm8n4/HPZL73vfu3rc+7897i6rCMAzDMNISLYBhGIaRHJhCMAzDMACYQjAMwzC8mEIwDMMwAJhCMAzDMLyYQjAMwzAAAOmJFmA4FBcXa2VlZaLFMAzDSCnefvvtvapaMvB4SiuEyspKLF++PNFiGIZhpBQiUhPouLmMDMMwDACmEAzDMFKLvj6guxuIQZeJlHYZGYZhjBo2bwZeeAF4803A4wHy8oDTTweOPx4YNy4qT2EKwTAMI9lZsgT405+AzExg0iTA5QK6uoDHHgOefx74xjeA8vJhP425jAzDMJKZDz4A7r+fisBRBgCQnQ1Mncr/f/UrKohhYgrBMAwjmXnqKSA/n9ZBIIqKgIYGYMWKYT+VKQTDMIxkpb0dWLUKKC4Ofl5BAfDKK8N+OlMIhmEYyYrjBkoLsVRnZgItLcN+uqRRCCKSLSL/E5GVIrJWRH6QaJkMwzASSl4eIMJU02B0ddF1NEySRiEA6AZwsqrOBTAPwJkicnRiRTIMw0gg2dnA0UcDe/YEP6+tDTjxxGE/XdIoBCVt3qsZ3ovN9zQMY3Rz5pmA2814QiB27QKmTAEOPXTYT5U0CgEARMQlIisA7AHwvKq+GeCcq0VkuYgsr6uri7uMhmEYcaW8HPjKV4DmZqC2loqhpwdobAS2bAHGjwduuAHIyBj2U4nGoPx5uIhIIYDHAFynqmuGOq+qqkqtuZ1hGKOC+npg2TJmE3V2AqWlwGmnAfPmAVlZET2UiLytqlUDjydlpbKqNonISwDOBDCkQjAMwxg1FBUB55zDS4xIGpeRiJR4LQOISA6AUwGsT6hQhmEYo4hkshAmAviTiLhARfUPVV2cYJkMwzBGDUmjEFR1FYD5iZbDMAxjtJI0LiPDMAwjsSSNhWAYRoLo62NHzZ072UmzogKorGSFbLLR1wesWwds2MCZAFOnAocdFnGWjREYUwjGkHg8rIgXYcFkMq4PxjBZuxa47z52y/RPQa+oAD73uaj02I8aGzYAd9/N/Pu0NF56e4HcXOBTnwKOPDLREqY8SVmHEC5WhxAbOjuB119n192GBh6bOhU4+2zgiCN87diNFGfNGuDnP2dhU0GB77gqc97dbuDb32YVbKLZvBm47TbKOXZs/9s6Olite+21phTCZKg6BIshGP1oaQF+8hPgwQdpEUydys1iYyPw298CixZxnTBSHLcbuPde5rb7KwOAH3xxMf8+9FBi5PNHFfjLX4CcnMHKAKCFMGECJ4r19MRfvhGEKQSjH/feC+zYAUybxpkcANeFceOA6dOBN96g5WCkOOvWAU1NwJgxQ59TWsrzdu6M/vN3dACrVwPvvMPdv8cz9LnbtvGcYN08c3PZ0mHt2ujLOoqwGIKxj+3bOYvDmco3EBFO8HvmGeCMMyyOl9LU1obusS/Cc3buBCZOjM7zdnUB//438OKLtFJEqAzKyoCLLwbmB8g837OHcoQKYrlc/BIHegwjLMxCMPaxejV/c8F+d1lZ/E1v3hw/uYwEEs0YY08P8JvfcEdRUkJfZHk5dyA9PZwLHGjqV1paeHJ4PBbgGiZmIRj7aGsD0sP4RohEZZ63kUimTQvupgF4uyoweXJ0nvOll+jSmTZt8K6joIBTvx54gGboli0MXBUU+DKdPJ7gVo0q/ZrGfmMKwdjH+PHM4guFxxPc9WykALNm0Sff3Bw4UAvQVXPYYXTnDJe+PuDpp/lYQ5mg6enA+vXAF75AJZSR4ftCtrfTzVVZGfi+DQ10a82cOXxZRzHmMjL2MX8+N2DBpvW1tVFx2EYsxXG5gKuvZlrZwBoEVWD3bi7Ql10WnedraOBz5eUFvt3jAZYv53l9fVz4J0/m36lTaT1s2ADU1PT/gjqydncDn/986LiIERSzEIx9jBsHnHwy8Nxz/B0O/G319vK398Uv2u9uRDBrFnDTTUzXrKnx7dxVgYMPBq68kumc0SCUe6qujsHrgoLBFoQIcMABVARFRQwcO8c9HuCQQ4BLLkmuIroUxRSC0Y+LL6Z1/vrrrE4uLOT60NDApJBLLwWOOSbRUhpRY+ZM4NZb6bPfvZuLbHl59OIGDuPG0QXU3R04PW3zZloBPT00QQciQpdQTg7wi18AW7fyi1lWFj2lZZhCMPqTkUFPwsknMzNw40Z6F049FVi4kPE+Y4QhQh9gLP2AmZmc7rV4ceC85oYGntPbO3Tec0EBLZmCAmDOnNjJOooxhWAMQoQbR4vPGVAFqquBTZtoIk6YwGHu+zO/99RTWdm4cycfx981pMpCuUMO8VVEBpIlVF60MSxMIRiGEZgdO4B77uGuHGDgyOPhgn355cDRR0f2eGPHMmZxzz3sripC89Ptpito/HgqhKGor6cysgBWzDCFYBjGYHbvBn78Y+7Kp07tvyvv6ADuuovK4UMfiuxxi4qAm29mDGDtWnZSLClhsOoXv2DgOFAxjMcDtLayRN6IGaYQjLBpbuZvMivL1/vMGKE88ggDvIGCRrm5PP6nPzFXOScnsscWYZVyRYXvmCpw3nnAY4/RneSfntrZSWvllFNoIRgxI2kUgoiUA3gQwAQAHgCLVPU3iZXKAJgA8p//sM+R4zUoLwfOPZftsE0xjDAaGth0Lljb6+xsZgy9+27kVkIgRIALLqAF8dhj/dNg8/JYD3H66eYuijFJoxAAuAF8VVXfEZExAN4WkedV9b1ECzaaeftt4He/4yZwyhRfW5nmZralueACXkwpjCB27fI1tgtGVhbTVaOhEAA+5wknAMcdx11IWxu/eDNm7F8Q24iYpFEIqroTwE7v/60isg7AZACmEBJEQwPwhz/QxZub6zsuQpdvfj4bVx54IDB7dqKkTEJUuaAtX04fW1ERB7dEO7c/VoiE10xONTY7dpfLUtwSRNIoBH9EpBLAfABvBrjtagBXA0CFvw/SiDqvv84Yn78y8Cc9nT2NnnnGFMI+GhuB3/+eBRzp6cyt7+oCHn8cqKoCrroqcp97vJk8mUrB7Q7e7bCnh9XOxogh6RSCiOQD+BeA61W1ZeDtqroIwCKAIzTjLN6Ip74e+N//OJPkiSeYKRisyWRREScxDlWAOqpobwd+9jO+iQMzc1Tpf+vqAm64IbnbNBcUAMceC7z22tDtINraaCLaTmBEkVQKQUQyQGXwkKo+mmh5RhN9fcCjj7IhJcCYodNHbONGejzGjRt8P/9Z56NeISxb5hs3NxAns2b1ak4hS/aF9MILKef27WwZ4ewInAKylhbgxhtpARkjhqQJ2YuIAPgjgHWq+stEyzPa+Oc/mUk0eTLXrdJSZv/l5NBCeO01usMH0tXFc5LdCxJzVKlNS0uHPkeEu+rnn4+fXPtLYSHwrW8B8+axZqC2lpeaGvoJb7qJrbGNEUUyWQjHArgCwGoRWeE99i1VtQm+MaaujnGAysr+nozp05lwUlhID8H779MN7s+ePUwfT2YPSFzo7mYUfqg+PA5OP55UYNw44Jpr6AKrrqa5WFLCL4qllY1IkkYhqOqrAOxblgDeeMPXRcCfkhIWoDU0cGO7fTs9HdnZvH3vXm4WTzgh/jLHHLeb7p/eXi6MgTpw+uO8eU6/naEYqhLXD4+HbroVK1gUXFYGLFjAzyIhFBUFH3BvjBiSRiEYiaOmJvDckrQ04KijgLfeohXR2cmOBjk5XKhKS4Hrrw8cW0hZ3G7ghRfo/mlt9U0MmjOHBRdDTezKyGAVbXV18JW7vp4FVkOwZw+TlGpqqDcyMuiW+/vf2YH20kstJd+IHaYQDGRkDD0lLTOTdUeNjSxKnTiRMYZjjmEfshG1OLndwN13M81q4kSfpvN42Izt1luBr32Nw2MCccYZwB138H6BfGg9PXyshQsD3r2pCbj9diregXrH46Ge6ulh5qp5bIxYkDRBZSNxzJ/PHf9QiNAqmDcP+MEPOPJ27twRpgwAYOlSKoPp0/sXX6Sl0W9TWAj89rdcsQMxezaVQnU1gy4OTmbO1q3sEjpxYsC7//e/PC3QCOO0NCqJV19NnRCEkXqYQjAwdy5jBC2Dqj6IKlvYn3nmCFQCDn19HN4SbAj8mDHUnO++G/h2Efp0PvtZXq+p8WXojBkDfPWrbNAWgJ4eJh8Fm2eflsb3f+nSCF6XYUSAuYwMZGUB113HmiqnG7GTdt7VRWVw2GEceDUUqgw679zJdXHy5CE3wslJXR2356Gq3/Pz2ZJiqP49aWm+fjzbtvENzM9nd9Agfp7WViqFULUcBQVsH2QYscAUggGAHQi+8x0Wp61e7Wtil5MDfPSjtA6GqkGqrQX+/Ge62dPS+s8+v+KKFFEMbnd4fXlcLq7c4ZwXKgXVD+f9DpWk5PGETFIyjP3GvlrGPiorWXy6dy+DyOnp3OkHK0atqeEclfT0/unpTn+3H/0IuOWWFJjFXFhIoUP172lvj2ihj+TpJ0ygpVBQMPR5TU02I8aIHRZDMAZRXMxmk9OmBVcGqsC99/KckpL+O1sRLnAeD/Dgg7GXedjk59MNtHv30Od4PFQYxx4b9acXAT78YXquhmo02tlJwyPSyZWGES6mEIz9ZvNmxkyD1WyVlrLCefv2+Mm135xzDqO2jY2Db/N4mD100kkxM3eOOYaL/ZYt/ROZVFkcuGsX49WFhTF5esMwl5Gx/9TW8m8wn7dz29atKTAOoKwM+OY3gTvv5OKfnU33UUcHFcKppwKf+ETMnt7lAj7/ebrennmG1oITj6ms5G02QdKIJaYQjEF4PNyltrTQHTRjhq9dhT+BXBseD6ttt29n14cxY3yLWkowdSqrw957z9euesoUlmwHywmNEunpdB2dfjoVbk8PW5BPnGjFaEbsMYVg7EOVdVmPPMIOC84C5PFwTczJ4bEZM4Djjx+cPdTSwr5IHR1c2FwuuuRbW1lle/jhgRVL0pGRweKMuXMTKsKMGQl7emOUYgrB2McLLzB9tKTEl0hTU8M6rFdeoVKYNw/YtImtfk4/nT3Pmpu5gL36Ku8z0Mc9aRLv83//B1x7re10DSNZsaCyAYBunocf5qI/ZgyP7dwJvPMOr0+ZwlhrayuHaJWXUylMncpUyDVr6CLy7/jQ1cVj8+bRB/7221QMhmEkJ6YQDADc3Yv40kxVucjn5tL9I8L/N2zgbU7d1apVbLa2bRszMpuafBeXiwW748fz/llZwMsvJ/BFRhNVFmzs3MnaBMMYAZjLyADA3vv+rp6mJsYCxo71HcvKYpzAabGQnu5TAkccwRiD0w+psJDuJH/3UH4+s42GhSozgJYsAdav57HZs9kuoqIi9v4oj4eBksWLqQyc0uwPfQg466wUKcs2jMCYQjAAcJ3z79zQ1TX4HP8qZIf0dObIA4w9BJsg2dc3zBG8fX30az3/PDXSuHEUZulStgo991z22YiVUvB4gD/9CXjxRb5YRwG53Zyn/OabwDe+YdFgI2Uxl5EBgGtYc7PveqB2/r29XND9F3WPh5XNFRX97x+I5uZhVtkuXgw8+yyfbNIkmiS5uQxwlJcDjz8OPPfcMJ4gBMuWURlMm8b+Ev5pWCI0lb773f6trw0jhUgqhSAi94nIHhFZk2hZRhsnnsixwE69gDMbpqmJBVL19bxMm+ZzndfXc9M+ezZz5+vrh643aG+nIlmwYD8F7OgAnnySyiCQtnIaLz3+eHjN5yJFFXjiif6tYN1uBlGeeYb5uu+/TyvhU5+ixTJUDwrDSFIidhmJSB6ALlUdYsbWsHgAwO8ApEL3mxFFZSWVwpIlXHO3baMycFJK3W5eGhtpEWRmAh0ditJxbmxf2YwjF2bjlFPy8d//0m3kZCp5PFQgnZ3ADTf4jkfM2rVc6IMNZMjOZrrU+vUceRlN6ur42OXlvN7XRyWwezcDLY6SyMzkG/fAAwyoXHBBdOUw9o/6elZLqjLOE8y3OYoJqRBEJA3AJQAuA7AAQDeALBGpA/AUgEWquiEawqjqUhGpjMZjGZEhwlbVubnA/fczmyg/n1aD01cnK4vrXn29oji/E8XdezEHH+CuG3LROuN1XHluLqafeyEWL5+A2lo+pipnKZx/PgeR7Teh/FEOqsyNjTa9vXxBjpto2zYqg8LC/jELJ8g8dSqtlQUL6NIyEkNdHfC3vzF/2j8IdthhwCWXpEAb3vgSjoWwBMALAG4GsEZVPQAgIuMBnATgdhF5TFX/EjsxjXiQng6cfTY9M0VFtAhE6B5au9a7CVZFVkcjKnrex9zpbcjKS8dYjxsP7zoZ8956GAtd38JxX74eu0rn7Gu74LifhoV/gUMoYlEO7aRb9fXxjdiwgTINDGD39NAMSk/n5ZVXOEXNiD979gC33UZ3Y3m5z4pzZmT/6EfAt75lCtuPcGIIp6rqraq6ylEGAKCqDar6L1X9KIC/x07E/ojI1SKyXESW19XVxetpRw3Ll9NFP3MmZ8nPmsXfzwEHMPB8yIS9OD17KcZmdMKVnQGIINPlQVoa8GpXFVBUhLS7f49JuU2orIySMgCAgw7i4tsXxFPpdlP4WbOi9KR+5Oezn9Hu3bQW2toGp0yp8s1ypq6NG8dpQ0ZiePBBpstNntw/hS4tjb3ZRdi/3WI9+wipEFS1V0QOEpFTRCTf/zYROdM5J1YCBpBnkapWqWpVSUlJvJ521LB5M5N3HHp6uPZlZdF9P7F1A5CeAaSloauLm6+ODiA/vQsrdpUBeXm805tvRlew8eM5h2DbtsA/YFXedtJJXLxjwYc/zOdpbu7vPnKev6mJi49/8YYtNolh1y5WVk6YMPQ5xcWsaamujpdUSU9IhSAiXwbwOIDrAKwRkfP9bv5xrAQzEkNaWv9MIWc9EwHS+nqR174H3em56OxkN07nUlOjqN0qdPWPG8cUzWhz2WXAgQf6WrE6AjY389icOcBFF0X/eR0mTwa+/nX+39vL5+3qoizNzbx9/nyfomhqomVjxJ+amsFKeyDObTU18ZEpBQjHZfQ5AEeo6kcAnAjgOyLyFe9tUa0AEpGHASwDMEtEtonIVdF8fCM0hx7avyjNqTvo7QVEPVAVNDcLurtpNWRn89LrykKhpwFLlwKNren9J7xEi5wczvj87GcpVE0NL3l5HBZw3XWhp9QPlwMPBH7+c+Dqq+meystjAPnEExlAdsZv9vXRUjrxxNjKYwQmkn7rwdyQo4xwgsouVW0DAFWtFpETAfxTRKYiygpBVS36lmDmzuUa19ZGz0taGuMHa9cCmWMz0NHjgrvbjbz89H3lAB4VuDUdR0+oQQaANW+240OfmokA1QLDJysLWLiQ/bc7O7nLy86ObwvVnBwqhJYW+timTOnvo+7tpdl05pm+eIIRX0pLaT2qhv5uBHMrjTLCsRB2icg854pXOZwDoBjAYTGSy0gQWVnAF7/ItO2GBv6epk5lYW5DUxre752OAlf7vqSfPhXs6hqLeYVbUJbVhOwshXS0Y2PFKbEV1Om25wxpiDfZ2SysOPJINmiqrqa1Ul3NVMePfYzZRdbrOzFMn86U0qamoc9pbaV709x6+wjHQrgSgNv/gKq6AVwpIvfERCojocyeDdx8M/DXv3J9S0vjRnfVKmBn9nSUajPSOz3oceVCRPGhog9wfMl6CBRjWrahdtxM1HYcjBjk+iQXjqvqxBN9eblTptDvFkmarBF9RFgx/pOf0LVXUND/9vZ2VkzeeGPgyvdRSkiFoKrbgtz2WnTFMZKFAw8Evvc9Ju7s3cvfTE8P8POfZ6OgewYq1z6JUs9OTC9oQE5mHzLaOuDq68be4oPwUsU1mO8Z4X0TnfFyTzzBCliXi37ruXNZxDGsKjwjKhx4IJMAFi2i9eYs/G43a0Wuvz6hU/GSkRH+qzWGg4hvGA5AxZCTA0w4sBR9sy+DZ9dKdG19HZ7edrTnlaK24ng0FM1ES23avolrIxJV4O9/ZwWf09nPGRy9fj17iX/lK5wMZCSWQw5hEsB773E6kyobcs2eHbwNyijFFIIRNsXFrPjfsAEoK8vEjskLsGNy/251vb30nFRVJUjIeLBqFfDUU2wA5e9uSEsDyspYmPH73wN33DF4nmgonM6pFnuIHunpTEmOdn+rEUjYCkFEBOxnNF1VfygiFQAmqOr/YiadkXRcdBEr/puauNZ5PIyhbtzIv21tTAKqrqYrPS2ctIVU46mnWHw2lO85N5dvxptvAmecEfrxWls5dOfZZ2mGZWVx4M7JJ/vMM8OIA5H8XO8CcAwAJzW0FcDvoy6RkdRUVADf/CY3sJs3czzAiy/6JqEtWMAEnDvuAP7v/+iuHVF0dLDN9fjxwc8Ltzhv1y4Ga/76V19TvOJizjT97nc5/Mcw4kQkLqOjVPVwEXkXAFS1UUSGM//KSFFmzOCC/4tfsED4oINoLUya5KsLKy4GXnuNfz/60YSKG116e2kZhHLppKezVWwwenqAX/6SlYD+QZeMDFY9d3cDf/wj2zXPnDl82Q0jBJEohF4RcQFQABCREgARlAMaqUBtLfDSS0yg6e3lunTGGezI4N/LrauLsYQTTvAV5/qTlkZvxzPPcNTwiMnCzMvjG+GUag9Fayt9ZsFYtYodOSsrA9+elcU37qmnGKQeLnV1lCsri0omWfx5TU1sqQtQrkjjLkbUiEQh3AngMQClInIbgIsAfDsmUhkJ4bnn6LlIT2ehJ2cfAHfdxSzKG27wpXO/9x4r/gMpAwdnsM66dcARR8TnNcSc9HTg1FN909sCoUrX0sknB3+sV14J3YivpARYudJXOr4/rF8PPPooNbjTrKq0lDOojz02cQHs+nrgn/9krMV/VsGCBQxWWfPKuBOWQvAGlJcCeBvAKWDLio+o6roYymbEkRUrgD//mbt6f0tg3Dhetm4F7r6bM+RFuN6p+iapOe3/A9HREZeXED9OOolm1N699In5o0oza/bs0G24m5tD915yBu50du6fQli2DPjDH6jJnfRYgG037rmHsiaiorqujrMK2ttphjoB+r4+DrNZv56zCsrK4ivXKCcshaCqKiL/VtUjAKyPsUxGnFEFHnuMcdKBLf4dJk+mVbBlC62F5mbOGFm50ndOeTnjC/7dn4ER5C5yGD8euOkm4Ne/ZjpVbi7NoY4O+tkOO4z9P0JVwI4bx4Vx4FxRt5suKREqDNX9exP37mW//4kTBw8NKiig++uZZ6i84p2S+cADVHKTJ/c/7nLx2K5dHN3nZDAYcSESl9EbIrJAVd+KmTRGQti9mxvFYH3YRKgs/vc/rjOPPMJeR8XFtAw8Hhbsbt3K9j4TJ3JtzMjgoJ0Rx+TJwI9/zJYVb7xBl05ZGV0w06aFt4idcALw7rs+10hHB4unqqv7d+s88cT9mwL32mtUJkPd12np8PTT8VUIzgi+YNWLZWW0ErZvt4lmcSQShXASgM+LSA2AdtBtpKpq1R4pTnu7zzMRjKwsppo+/zzXQyewXFjI+48ZQyXw1ltcw+rrgfPOG4EWgkNGBquR97ci+dBDmZq1axd366++ysyj/Hwu1r29DLhu2cL2C07L7XB5883Q6bFFRVx4u7piM3o0EFu28G84swqqq00hxJFIFMJZMZPCSCi5udyQBusU3NkJ7NjBpBiXi2vHwQfz+LZttB5ycnhbZyfw9tvA5ZcD558f+PEMUKHccAPws5/RdePs2D0e+uQA4JhjqH2XLaOvLpxCN4eentAKxPnA4zkTIJJZBZGcawybsBWCqtpYoRHKhAlcc1paBvv/u7o4iXDbNnpFHHf2rl10lR9xBF1NmzbRlQTQXTRxInDVVcmT2Zi0lJQAl1xCF0pbG8219HQOoZg61RdInjCB6aennBI8tcuf8nIW0fnPRB2IE6wOdk60CTd7SGRw0N6IKZG0rvhuoOOq+sPoiWMkAhHu5O+8k54LZ73p6mJmZGenr7NzUxM3se3tvO3YY+nu9U8G8XhoTZgyCJP33uObO2nS0Ofk5lLjbts2dN3CQE49lRk7wUy/PXtYORjPD2vmTC70LS2D21I7OLMKQmVqGVElkm9Bu9+lD3QhVcZAJiOG9PQAy5ezTfyXv8zuwI89xs3oxz/OoPDOnfQgrF1LBdDXx03k0UdzXXK7+Tc7m4810Kpvb7eNXUR0dITXeTMtjXGFcDnoIJpxtbW+4dj+7N7NGMPCheE/ZjRISwOuvJJBpvb2wbd3dFD5XXmlzSqIM5G4jH7hf11Efg7gP1GXyNiHMz/e7eZGaqiU0HBpaQF+9SsGhseO5SLvdrOl/+LFwOc+B3znO8B//8sElfXrWb904IHcvGZk0I29Zg1lycqiwtizp/8Uwvp6/paNMJk4kelbwVClZi4s5P+bN/sCOpWV/KAG4nIBX/oSm0q9+y5Nv+xsKpXublolX/7y0Lv0WDJnDquv//hHpt469Rjd3XRfXXedtQ9PAMNpf50LIKpTQETkTAC/AeACcK+q3h7Nx08VPB5m6ixezKw7gAvwySfTCxAqcWSox/ztb2kBTJvmO56ZyRhAZycLz77zHQ4BO+44tpEfmBlYXs7Oph0dtBKcamZHIdTVUb4jj+x/v+ZmJoz09dF6KC+39PJ9HHUUzTSPZ2jXTX09d/x1dex/tHNn/zdw7lzgiiu44G/ZQk0/fjzf6C9/mR/8smXceefn8wM68MDE7sAPP5yZVitX+mYVHHAAX0uogj0jJkQSQ1gNbx8jcMEuAXBrtATx9kn6PYDTAGwD8JaI/EdV34vWc6QCHg/rcV56iRmBzsLZ08NElFdfZU1UMHdzIDZs4GWo1O+cHMYPnniCiS+qgdcmpzPzsmW0DtxuxhoaGuj2LSoCvvpVXyy0tRX4xz9ocfi/xqlTgU98wlzEABiAWbgQWLKEu/2Bb3x7OwPOhx7KjKTCQr6BjkLweNgX6bLLGLDNyOBtfX0875JLmBIWrNAkUWRlUTkN3EEYCSESC+Ecv//dAHZ7ZytHiyMBbFTVzQAgIn8DcD6AUaUQXn6ZymDatP7rgrOT37sX+M1vWBMVyebu1Vf52wu2Ky8uBlav5m6+rIzrTKBNa0EBk1127qQnwul0etJJ3Nw56extbcDtt/M8/+4EqkBjI2+78Ua6uUc9l1/OBfzVV+naKSjg9ZYWfvjXXgv86U/8kAa2sOjro1WwfTs/LGdxVaXW/ulPOS7SXDBGCCIJKn9JVWu8l+2q6haRn0ZRlskAtvpd3+Y91g8RuVpElovI8rq6uig+feLp6+MOfcKEoT0HxcV0Ha+LsIuUM/4yGGlpvLS1caN52GF8rkCkp/OcBQvoor7pJno+/Gub/vMfZhtVVPRXXiL0ZhQVsc1OqC7Ro4KMDObp/vCHrGAuKqIWveIK+u7S0mhuBepntHEjNeyECdS+XV08LsJMnZISvtGdnfF9TUbKEYlCOC3AsWgWqwXauw5KjVDVRapapapVJSOsG+KOHdzQ5eUFPy8zk9k9kVBQQLdTMJy4pbOoX3wx/9bXDz63s5Mb0ssvD1zg2tFBSyeYays/n94Q/35IoxoRas8rrgBuuYWdBE8+mSXgGzYEziro62OAOT/ft4tobe1/Tl4elcTbb8f+NRgpTUiFICJf9MYPZonIKr/LFgCroyjLNgD+8wKnANgRxcdPenp6wksHz8gInK0XjA99KHTX0cZGNqdzgtaTJwM338y1prqa2YvbtvH/1lbgC19g8DkQO3ZwrQqVTZmdzTR8IwRDVey2tfnazToESjHNy6M/0DCCEE4M4a8AngbwEwA3+R1vVdWGKMryFoCZIjINwHYAlwD4RBQfP+kZO5aLaLA6IoC784kTI3vsQw+lR2HPnsAZik7bnKuu6v/cU6eyS7ETlO7r465/zpzgiSBOa+zaWrrBXS56LoqK+j++iHUnCIvp0zlzeSD+i7/z/8DuqYC90UZYhFQIqtoMoBnApSIyDsBMANkAICJQ1agMffXGJK4F8CyYxXSfqq6NxmOnCsXFTAapqRm6ul+Vv+ujj47ssdPTGVf82c98j+/0MNqzh0rm4osDN71MS2M2ULgZQarcjL7xBi0Apxvq++/T2liwwNcio6urfxqsMQROtN7J93Vw/vd4aDZOnBg4WNTRQaViGEEIO4YgIp8Fh+Q8C+AH3r/fj6YwqvqUqh6oqjNU9bZoPnaqcOGF/F0Hcu+o0l1zzDGRp50CtBC+/31WJHd3UzFs28bg8S23AB/+cHRqA559lmn106bRMhgzhgqgsJBusVdeocvJ6b22YMHwn3PEk50NfOYzbCLl7y/MzGRucl0d38xDDhl8X6e6OdJdhDHqEA3kbwx0IuMICwC8oarzROQgAD9Q1YtjKWAwqqqqdHmk0dUU4J13mBTS28uF1OXyLaBHHkm3znDrdlT5eOnp0a1NamlhHUNZGR9/6VJfQzxH2Tj1ChMmAJ/8JFNYjTBZvpyj7VpafC6iri6mnR58cP/6BIAfwtat3AWce25iZDaSDhF5W1WrBh6PpA6hS1W7RAQikqWq60XEyopiwOGHM9PwzTd9w+5nz2Y2YrizV0LhDOOKNk5vo8xMXo47jsecbs5paYxD1NQwtT7U2GFjAFVVrCdYt469iFwufikyM7mLqK3lh+vMU8jIYAXgmWcmWnIjBYhEIWwTkUIA/wbwvIg0YpRlAcWTggLgtNN4SSW2bOnvwi4oYMFaYyNrITwexhF6e7mhtfYVEdDayo6Dzc18E6uq6Idz+MEP2AJi3Tr6BCdMAObPD53HbBheImlud4H33++LyBIAYwE8ExOpjJTF5RqczOIUovn3YKqttfbYYdPbCzz6KEfVud18Q53eIgsXApde6itDP+AAXgxjP4ikl5EAuAzAdFX9oYhUAJgHIESbRmM0MXs2C9KC0dNDT8b+BMZHHR4PcN99bAZVUdG/3qCvj71OGhrYwC7cwTmGMQSR7NHuAnAMgEu911vBZnSGsY85c+jNGFgsC3BTq8qitZNPju+QrpRl/Xoqg8rKwQu+y0UlsWIFm0oZxjCJZEtxlKoeLiLvAoCqNorIMDv0GyONzEzgmmuAO+6gG7uwkJmSGzdyI9vZybWtsjJ4t2fDy3PP9W9LMRCnX9HTT1v+rjFsIvk59npbVCsAiEgJACt9NAZx8MGsayguZsvul17yzXI59FAmxfz+92zzHc/Z7inJhg1c8INRWMhofrwqkVWp5Tdv5t8wU9eN5CcSC+FOAI8BKBWR2wBcBODbMZHKSHlmzGCMYMYMrmcuF2sqnP5sHg/d3xMmsCDOGAIngJwMqNI19fjjvqwAj4duq498hOmwljaW0oTT3O7P3n+LAXwD7Gm0E8BHVPWRGMpmpDB793KIzqxZXPRLSvo360xLY/O8J5/s3/66tZXV2Fu3RjY+eMRyyCH0tQWjoYFvdKz9b08/Dfz610x7rahghXRFBa//6lc0B42UJhwL4QgRmQrgMwAeBPCwc4OIjI9ygztjhLB69dBT1xyyslhku3Ej3Uv/+Q/7Hzmb4pwc4IwzgNNPH8UTFU89lRWKfX2BS8qdwdtXXRVbObZs4ei78vL+LWxF6LLKywP+/nf6CysrYyuLETPCUQh/AOsNpgPwb6guYDxh1HTM6upiUDQrq39/MWMwTofTcNiyBbjrLqajTprku19XF/DPf7IW6/rrA89dGPHMnEml8NxzwJQp/TVjTw+HUhxzTOzHzi1ZQhNvqH7mGRm8/cUX2XPJSEnC6XZ6J4A7ReRuVf1iHGRKOrZtY8O2ZcvoMlVlAehZZ/H3moq43Vxo33iDvdLKyjgzobIyOm5gp5V3KFSBf/+b1sDkAfPxsrMpz7p1wOLFwEUXDV+ulEOEs5KLijhOz5mGBnARPu884Pzzo9uQKhD/+9/QLXgdSkp4nimElCWkQhARUTKkMnDOia5oycHq1XSbulz0hTutnNet4wCqT38aOPHEREsZGdu38zXV1XEhzsjg63nuOcYFP//54VtAc+f65rwPtVZ1dvo6nk6ZEvgcESqK558HzjlnlFoJaWnA2WezC+D773MoTm4u4wbxKubo7Q2tdJz+SVHA4wE++IDxJI+HnqqDD7bau1gTztu7RET+BeBxVa11DnprEI4D8EkASwA8EBMJE0hDA/Db37Llgn87mLQ07qi7uzn3fOrU1Onp39DA4faqg129qsCqVXTf3Hjj8GKU48ZRUb74YmCro6+PBWqVlUPPbXbIzOQ6U1sLHHjg/suU8mRlBR5YEQ8mTWLE3xlkEYjW1sFm3n6wcSOwaBG/FyK+2T6FhTQ+EvUWjAbC+cmfCaAPwMMiskNE3hORzQA2gFXLv1LVB2IoY8J4/XW6VobqDZaVxcXqhRcC3+7x0N20aVPoRS9eLFlCF1Fx8eDbnJG+a9ZwIzpcLr2U/de2bKE14nbTIti5k4v72WfT5RaOt0OE9x+x7NlDv9iiRcBf/kJ/XjIVaZx5Zuhsp8bGYXdV3bKFG5bubm4Wpk7ld7Kykt+TX/7SJoHGknBiCF1g24q7RCQDTD/tVNWmGMuWcF55ha7bYJSW0g//mc/4FjaPh90GnniC6ZdOy+eZMzkA56CDYi97INxuKq+ysqHPEaEXYskSmujDITMT+NKXODP5uee480tLY/zllFPYg23JEs5MCIYzJc6/Od6IobcXePhhYMkStLuz8HbrTNQ2u+DyLMWhM57HId//ONIrkqDp0xFHcHXevp3Wgr/J5/Qjqahg7/b9RBV48EF+/wLV4jmTQe+7j+3hYx02GY1E5JFT1V6wBmFU0NUVunOw093T7eb/qsBDD3EBLC3lbwTg8Z07gZ/8BPjiFxMzvKqtjTv0UCmc+fm0bKKBy8UEmKGSYI44ghvigXPi/amvp6towoToyJQ0eFdAfellvKgn4W/vzUGvJw1Zrj4oBM8u60Xhee/jmkU5mHlkiGrlWJOdDXz1qywx37CBH1ZWFrfybjd3O9dcM6wgz9attBCmTh36nDFjOEtj/XpWvRvRJSlCNCLyMXAc58EAjlTVpBiDVlJCV0dmkI5NXV2M7znnrFzJAOi0af198CK0NnJzgXvv5e8nlPURbTIyaKmoBs8k6uujUogHY8cyUeZf/+JCMFAptLXRxfXxj8dHnriydSvwyitYgpNw/6rDUV7Qgqx0PzfRGKC5rhc//WY9vn3/uMSn948dC9x8M32gb7wBNDVxK3/00ZzXPMz0tJ07fTGDUGzfbgohFiSFQgCwBsCFAO5JtCD+nHoqh1AFayWzZw+z/pxiqief5O9mqIBsTg4timXLmDUTT3JzudPeuTO4MmpsZEFYvDjvPL4nTzzB9zEvzzczPjeXG9MZM+InT9x45RV0SB7+9t6cwcrAy9jxLvTs3oq/PzQV37xliBqAeBLDmQvh6pNQBY/G/rNfCkFEJqjqrmgJoarrvI8brYeMCvPn0126a1dgd0VDAxeshQt5vbOTfnLHTTQU48ax+DTeCkGEgdxf/IIZG4F8sB0d3KUfdVT85EpLY2zlhBOoKKurac3MmcPPYMSmmtbU4J32WejtSwuoDAAALheKM5uxbpUbu3ZljDy3mR8VFb4W6UMtBU5yezC3krH/7K+F8BSA/Y8epQhZWdyd/vKXXKTy87nD7+lht4DCQuBrX/NZEM4wq1B6zeXiYySCuXNZUPf008w0KijwpfXt3Uuldt11/SczxouiovgryYSSkYFtLQXIcAXPJhIBXOn8fMJWCDt3ss3sq6/yQy0uZg+Qo4+Onz8wQiZM4ICljRuHfp2Njdyk2VC42LC/CiHirbyIvAAg0Md8i6o+HsHjXA3gagCoCLUVjwJFRcD3v89UzCVLaBUUFQEXX8zdq39dUF4eLYauruC72tbW2HcaGAoRpoNOm0YXTW2tLzB+2GF034xI90wyUlWF9CfXwqNBfk7d3UBuLjQzK3w3yZtvAvfcQ9OrpIRf2I4ORu+ffBL4xjeAiROj8hKizRVXAD/6EfXZhAm+zZUqFWJvL2tkksyZMGLYX4Xwf5HeQVVP3c/nGvg4iwAsAoCqqqq4VEdnZHDxnz8/+HkuFzdhjz02tEmryt/mySdHX85wEWH7m6OPZgyku5uWQiKsglHNggU4uORlLN47xBwDVaC9He65RyDNlRbSFQmAaTp/+ANT3Px3K/n5vNTV0Wd4221J2TGwrAz49repu9as4TEnPnfAAVQYcdgHjlr2VyE8GlUpRhAnnEBLfffuwfn+qkyZmzcvOSpuRYLXJBjDw+1mDcYLLzAOlZ1NRXzMMV7lm5+Pg276CIqv3InGhmy6Hp2tb28vU6ymTMEOVzlOXBimp+fZZ5nyNlRLi5IS+j9XrgSOPDIqrzPalJXRVbt7N9OfVWnQDCx/MKKP7E8LIhF5R1WjFkMQkQsA/BZACYAmACtUNWSeS1VVlS5fnhQZqv3Yswe4805+mbOyaGF0dXGBqKpip+KRPE+4r48/3NGcCdLaCvzmN0zZdzbnbjddjhkZwLXX+lowbHlxC26/uQlpTY0oyWlFmgDIzETvtAOxI7sSk6e4cNNNYSiEri5WAk6eHLxqq6GBzaO+8Y1ovVwjxRCRt1W1auDxuMUQgqGqj4HT2EYEpaXAD3/I9g//+x83emVldNFMnjwydzk9PWz299RTTK8XYaXzmWcyUDialIPHwx5YW7YM7uM0ZgzTaX/9a+B73/P2wTp5Gr79T+CR+1qxeqUHaRlpQG4e0tLTcPJC4IILwrQOOju9EegQJbxZWawhMIwBxC2GMNpIS+OCONz2D6lARwd3w+vWsb2EEz+prWWLgRNOAD71qdHTamDDmm588FozpvZtgdS6GaCpqNiX0pWXRwviySe5oQfYzfPG741BXR3d/GlpPBaqUr4fOTn0rwRrMQswaFRaOqzXaIxM9kshqOpd0RbESF0efJCukWnT+u+Gi4upIEbV7OQNG7D0+reQvX0iZGwnV/b6elb3TpnCzASXCyUlwPLltB79d/8lJaHHDgxJdjYLSN55J3gWUUsLtXS4tLVRS4nwgxyxhSFGslQqGynKnj3MciwvD+wK85+dfNppwduApDzbtwM/+xnquj+CnHHZQLb3DcnO5s5961ZeP+IIuFwCkcEKYdiccQbbSnR2Bg5U7dlDjTN3bujH2ruXc01ff91XEeZyse/K5Ml8XdOm0Sc41CQ1I6UwhWAMi1Wr+Dfc2cmHHBIfuRKCt/dG3hjBjoYBLhtn9vC2bcDMmdCCsfB4YpD5WVlJP9Qf/sDrpaVcrDs6aKmMG8dE/lBPvHs38OMfM+DhTIbas4dBsRdfpBZzpiDl5wOf/SzT54yUJpyJaeE0HfaMhnbYxmBaW8OPDXR0xFaWhNLaysVy8mR8SLfh3V0TUTxw6pyTerV1KxonjcWMGTGq/ViwgO6pl1/2VSoXFQGXX06XUiiTRJWFbT09vlF2e/fSUsjJ4bHmZpYNH344lcavfsVcUZtek9KEYyHs8F6C5ca4AFi5yChk3LjwZycnaceE6NDcvC/DZ+6E3Rif04n6jhwU5Xb2Py8jA+6mVjTlcoZGzDLOJk4ELrmEl0iprgY2b/ZlB6gCK1bQReRYFmPG0AV2yCGMfBcXAw88ANxxx9A7hMZGzk1wCgvi3e7XCEk4CmGdqgat0RWRd6Mkj5FiOBvCULOTCwpGeP+Z9PR9fvZMVx9uOPoN/PS1Y1HbXICyvHZkpffBo8De9ly06ThceG14bvyE8MEH/Otoq8ZGBjv8x2empfH1NjZycXcGFbz//mC/4N69wCOP0IJy7qfKAPvFF1t1ZBIRTnb4MVE6xxiBjB/P2cm1tb64oz/O7OTzzhvhA9JLS7lLbm0FAJSPbcEPT3oJ5xy4AU1dWahtHoutLWNxQM4O3HRjz76W6UlJT09/7d7ezr8DBXZ6SjiosiTbn7o6Nid65x26msrLmYJbXs7eFLfeysZFRlIQ7gjNgIjIp1X1/mDnGCOfSy9lJuNbb9EtNG4ci7Pq65nyfu65/Xs3dXezc8KKFezQUFHBdg6B5jynDGlpzKu99166UNLSMD6nExcevA7nz1qPjt4MZLQ1Ilu6gQsujHJpZ5SZMKG/HzCQ5nJ2+QMzmQaaiQ8+SBNx8uT+x9PS2Itizx6+Z9/+dhJryNHDcPdsPwBwfzQEiRduN61cJ+ljRO9a44QzO3ndOk6Lc2YnL1hAReA/TGv9euB3v6MHIjeX5739NiemnXUW77N5MxVFWRmnYiW6B1tLC70dy5ZxbZs4ka/roIMGrH/HHceCjJdfpsXgHQLsUjfG1G/nOd/8ZuJfUCgOO8w3HjMry+cq8h9U0N3N1+dExR1LwX+s265dwOrVwYcXlJSwRqO21oYcJAHhZBmtGuomACnj/OvoYLbcs8/6sl3y89la4cQTR3ZvoXjgcjEdffbsoc+prmbMcexYDBoH2dLCedOFhVwXnJbcubmMix5/fGI2kGvXsg1Fd7dvA7FuHYvKZs0Cvvxlv2C5ywV8+tPsXPjkk/SpO4O2jz6aFsSkSfF/EZGSnU3f/v3307UzZgwX7oYG/t/bS804f77vQ6mro+b3b0VaXR16QIhzW3W1KYQkIJz9cRmAMwA0DjguAF6PukQxoLUV+NnPuAmZMMGX3NDRATz8MF0dX/1qhG0CRiBtbVzoli/3uXKOP37oorNI+de/uOEsKOh/vKODGY0uF623o47yKeiuLnoU3O74twzfupXDkcaN6z+wJS+Pa/ymTZw5//Wv+9VhuFwcoXfccVwk3W5qwGRJsXL8da+/7qsxOP54Rvz9i0lOOolfgn/8g5q5vJw7/h07qBSOOoomnMdDt4/LxVoE/y+KZ4i23oEIJ1XNiDnhKITFAPJVdcXAG0TkpWgLFAsefJBxq2nT+h/PzeWmprqaiuGzn02IeEnBqlXAXXf5PAEuF103zz/Pte2TnxxeMerevYwhlpcPvu399xnHLChg9ub27b6MpOxsup//+le6k7xemLjw5JO0CAKt5SKMka5bRxfZoHbmaWnJlz1TU8N6gaYmvqiMDO6SXnmFTbeuucb3YkVY9Xz00fSXbdpE/922bfww3W7e1+NhutQllwxul1FaGjjTYCDWhz1pCCeofFWQ2z4RXXGiT10dd7yBFiKHyZO5YbrootE5JGbTJq4TxcX9d8Jjx/L3vnSpzxsykPZ2unvS03n/oSyJhgaukQMrmru7uRN31qH0dD6eP1lZ3EC+9Vb8rIT2dj5fMA+PCGVbujQ55lsEpa6OZrLL1d9fN3YsF+3332dw5+tf7x8YGTuWPUdOO813rL6eWhugEhiq+dL06fxCNTf3T1n1x0lnPeigYb08IzqM+JDqBx/w+x6stYLj5v3gg6SdGRJTHnuMLppALrO0NK4fS5cy6OsojB072Op62TJeV+Vt55zDTeXA99svTb8fTkajc77HEzjQn5vLnXi8FII3ezRkFXZeHrs8JD3//S/9b4F2RiI8vm5d4DqCgRQVhVdUlpYGXHmlr1htoKnV0UFFdf31o6cVbpITsg5BRN6JxjmJors7vPNUEzf4PpHs3cvAabAOm87O/s03eX3TJuAHP6AnYdIkriXl5Vxv7r6bLrqB7uMpU+j+6RqQoDwwlb2vL7D3IJRSjzZZWXwNoTwePT0pEHvq7eVA8GBuGRFq3RdfjO5zH3IIeyd1d9M3u307dxPV1bQOrrsu9GxaI26EYyEcHCTTCGBweQh7MPGMGxfeeU4a6mijqYkLbaigcXY2f8ddXRzukpvb3wsgwhhAfj43ozNnAsce67s9M5Pzph99tH+b7Px8Pn9fHxfXnJzAyqmzM76zJQoL6fHYuzf4d6itDfjQh+Im1v7R3k6lEKrVbH5+bIrEDjuM0fk1a5iWC/BLMGdO8qfgjjLCUQjhOPeSNkXgkEO4yDgp1YHo7ORvYdas+MqWDGRmhhf3c7upBFaupDtlYNqoQ1oaF/TFi7lQ+iuas87ierB6NV3POTmMa1ZU0EopKGCB2kBLoKODn93hURvaGhoRZon++tcMZAdyYzU2UnEkbQsKh4wMn7kTTPO73YNTwKIpw/z5Zg0kOSGNcFWt8b8AuArA1QCOBJDpPb4t1oLuL1lZwEc/yuQIt3vw7b293Pl+7GOjs6X7pElcA0J1Iu3p4YL8xhuhMyjHjGGGYl1d/+NZWcBXvgJ8/ON8vtpaX0B5+nQGZgdmETU300f/2c/Gv1Zk/nzgIx+hnHv3+hRnTw/ldruBG25IgU1uXh6Dtg0Nwc9rakoBc8eIJREHlVX1uyJSBmA+gI+KyAxV/dxwhBCROwCcC6AHwCYAn45mO+1TTuEC9Nhj3H06m6DmZv697DKmYieK9nbuNl0uZurFM76Wns6d8IMP9q8o9qe+nhlEBx/MQHKo6m6ny3Nv7+DbMjMZeD7jDHon+vp88cmHH2acwt9iKSsDvvY1eh3ijQjnGc+cyde9bh1fV0YG3V+nnJJC7TbOPpvzTAsLA3/B2tqo2RYsiLtoRvIgGo6/AICI/BrADRruHSIRQuR0AC+qqltEfgoAqvrNUPerqqrS5cuXh/08e/awPfz69bx+6KH0cyfqR71nD3PdX3uN1z0eKquzz2ZdULwslr4+YNEipt76dVxAby93+jk5wE03MTD8l7+wM8PA1jQDH2/HDs5ZjrQeq7GR9Q+Oopg2Lb7B5GB0dPiCyClnTaoygPP44wyKFBZS4/X18YvodjP4e+ihsZOht5dvYmamtQZIMCLytqpWDTweiYXQBuA/InKJqrZ7F/Hvqeqxoe4YClV9zu/qGwAuGu5jBqK0FLjwwlg8cuRs28ZWDd3dvoFUAK2Fv/yF8bdrr43PyEmXC7j6asZbnnqKLhJnl3/SSdzNO4He444DXnghuDt69+7w5rAEYtw44Igj9v+1xJLcXF5SEhF++adNY4Bnyxafpl2wgLsQ/7YT0WTvXn5pliyhUlCl4jnrLH7prKld0hC2hQAAIvIJADcA6AbQDuBHqvpKVAUSeQLA31X1L0PcfjUYw0BFRcURNTU10Xz6uNDXB9x8s2+Q1UBUmZV30UXsFBpPPB66mp2OCwM3cqps1/DWWwwsD/wtt7TQ+/C97wW3IowE4swx6Omh1o5lW43qahbEdXfT/5eZ6WuF29rq+5KbUogrw7YQROQUAJ8DFcFEAFep6vsR3P8FABMC3HSLqj7uPecWAG4ADw31OKq6CMAigC6jcJ8/mVi/ngHXoXp5iTAL55lnuDuP52D6tLTgLjQRX4uP5cvpOsnLo5JrbeXa8vWvmzJIakQ4yCLWdHWxBD4jg+a5g5OKNm4c8M9/0mpJRJDIGEQkLqNbAHxHVV8VkcMA/F1EblTVsCpZVPXUYLeLyCcBnAPglFjEKZKJlStD+6Czs+na3boVmDEjPnKFS3Y2297U1LANzvbtPHbUUZyzbu5hAwAHXjQ3D52jnJ5OM3Tx4pGtEFpa2AbBadg1a1bSBqHCVgiqerLf/6tF5CwA/wIw7Dw1ETkTwDcBnKCqI3kUOwBunMLJJBIJnCqbDIjwdz7Ub90w8Oqroesaxo9ncUqwfkepSlcXu8W+/HL/0v38fObCn3BC0rnK9ruXkaru9LqRosHvAGQBeF74Br2hql+I0mMnHZMmhW6pocrv0GisnjZGCO3toXfCzryEkdY3pqeHVY3r1zM9zz9Xu7MTuO8++ljjHSQMwbAS+lS1MxpCqOoBqlquqvO8lxGrDABfA71gLeDr61lLZF2BjZSltDR0xWNfHxVC0jeEipDXXmP5/dSpgwt3cnLY/OvRRwfPoE4wSZLhPboYP56FTTU1gZVCWxs3VxfFJPnWMOLECSdQIQQLCTo5yimbzxsAj4cFRmVlQ7uEMjIYXH8lqkmaw8YUQoL42Mc4vnPrVl4aG5muXVNDZXDjjckXTDaMiJg1i/1Itm8PrBTa2rh4nnVW/GWLJa2tNPFDpfMWFrLgKIkY8fMQkhWXC/jEJ9jf//XXma6dns5+QYcfPrI2TMYoxeVie+s772TgOD+frqHeXvZNyspiM6gpUxItaXQJ1UTQYWDv9yTAFEKCmTAheaqnDSPqFBSw78m6deyLvmsXd8Yf/jCDabHqrppIxoxhxlR7e/DYSFNT0pXlm0IwDCO2pKezzmAk1xr443LRDfbQQ4MHuTv09TGnfOHC+MoWAoshGIZhRJvjj6cyqK0dPD6wp4c+4rPPDj60OwGYhWAYhhFtcnLYt/3BB9n4y+PxxRWysoBLLqEVMVIK0wzDMIwg5OcDX/oS0wfXr2flcmEhO70maX8XUwiGYRixpLiYfeNTAIshGIZhGABMIRiGYRheTCEYhmEYACyGYKQoHg+waRNre9LT2eZjJNY4GUY8MYVgpBzvvAP87W+cOgf4Oigfdxxw8cXW9sMw9hdTCEZK8dprwD33cAKj/whStxtYupSNAr/+9aTN6jOMpMZiCEbK0NIC3H8/izvHjOl/W3o6FcSmTWyZYxhG5JiFYKQMb73FFjDZ2UOfM3Ei8MwzwBlnJO3Y2tSkpQVYtQpoaKBPbvZsdmY0RhSmEIyw2b4d2LKFi3JxMdvdDxwGFUtWrRpsGQwkO5uxhbq6pGsTk5r09QH/+hfw7LP83+XyTXWaPx/4zGdCfyhGypAUCkFEbgVwPgAPgD0APqWqOxIrleFQV8cRsOvW8boqhz0VFABXXAFUVcVHjhRuM5+aqAJ//jPw4otARUV/7a9KDf3zn7O9tQVtRgTJEkO4Q1XnqOo8AIsBfDfB8hhe6uuB224DNm+mj76ykk0cp06lS+bOO4Fly+Ijy8yZHLIVjJ4ebmKLiuIj04hmyxbgpZf4oQ80BUU4F7i6mhOejBFBUigEVW3xu5oHwPZ3ScK//81FeOLEwbvz/Hwef+ABoLMz9rIccww3pr29Q5+zcyen0AWLMxhh8tJLQGYmzcGhKC3l/OCBLZ6NlCQpFAIAiMhtIrIVwGUIYiGIyNUislxEltc5iehGTGhp4eYvWOwwJwfo7mZtQKwpLgYuuIAt5ru7+9+mSmVQVMRZ1UYU+OADducMRl4eqwM7OuIhkRFj4qYQROQFEVkT4HI+AKjqLapaDuAhANcO9TiqukhVq1S1qqSkJF7ij0p27eLfUIHjnByOzI0H55wDXH45k12qq6kcqquBmhq6sm6+mdMLjSjgcoUOxqiGH9wxkp64BZVV9dQwT/0rgCcBfC+G4hhRJJ4BXBGmlC5cCKxezYB3VhZw0EHA5Mm2LkWVOXOA558PPhe4pYVvvJWHjwiSJctopqo6e8zzAKxPpDwGmTiRf93u4FZCVxdw4IHxkckhJ4cz2o0YsnAh8PTTDNoEKupQ5fCXj33MNPEIIVliCLd73UerAJwO4CuJFshgevmxx9I3PxQdHQzgHn54/OQy4sTEiVzsa2uB9vb+tzlzgauqTDOPIJLCQlDVjyZaBiMwF1wArF0L7NjB4LJ/wklrK102111nWT0jlrPPZsHJo48yUOMUeWRmAueeC5x/fnyrE42YIprCFTxVVVW6fPnyRIsx4qmv56zwlSt9x0SA8eNZmDZvXsJEM+JFXx+wcSN3AVlZwAEHWDFaCiMib6vqoJJSU+1GSIqKgBtuAHbvZoGa07pi5kwmohijAJeLvUqMEY0pBCNsysp4MQxjZJIsQWXDMAwjwZhCMAzDMACYQjAMwzC8mEIwDMMwAJhCMAzDMLyYQjAMwzAAmEIwDMMwvJhCMAzDMACYQjAMwzC8mEIwDMMwAJhCMAzDMLyYQjAMwzAAmEIwDMMwvJhCMAzDMACYQjAMwzC8JJVCEJGviYiKSHGiZUl2VDnWcsUKYNUqoKkp0RIZhpHqJM2AHBEpB3AagNpEy5LsVFcDf/0rsGEDZxw7U1CPOgq4+GKgsDCR0hmGkaokjUIA8CsA3wDweKIFSWY2bgRuv51jbSsqONsY4FjLt94CNm0CvvUtUwqGYUROUriMROQ8ANtVdWXIk0cxfX3A3XcD+flASYlPGQAceTtlClBfDzzySOJkNAwjdYmbhSAiLwCYEOCmWwB8C8DpYT7O1QCuBoCKioqoyZcKrFvHBb+ycuhzJk0Cli0DPv5xYOzYuIlmGMYIIG4KQVVPDXRcRA4DMA3ASuGWdwqAd0TkSFXdFeBxFgFYBABVVVUaO4mTjw8+ANJDfGIuFy2HrVtNIRiGERkJjyGo6moApc51EakGUKWqexMmVJLS19ffTTQUqoDHE3t5DMMYWSRFDMEIj4oKoLc3+DmOMigri49MhmGMHBJuIQxEVSsTLUOyMncukJ0NdHYCOTmBz6mrAw491BSCYRiRYxZCCpGdDVxxBQvSOjsH397YCLjdrEUwDMOIlKSzEIzgHHss//75z8Du3Qwyq1IRlJUB119P15JhGEakmEJIQY49FqiqYsuKrVupFGbOBGbNYuWyYRjG/mAKIUXJygIWLODFMAwjGth+0jAMwwBgCsEwDMPwYgrBMAzDAACIaup2fxCROgA1foeKASRbhXMyygSYXJGSjHIlo0yAyRUpiZBrqqqWDDyY0gphICKyXFWrEi2HP8koE2ByRUoyypWMMgEmV6Qkk1zmMjIMwzAAmEIwDMMwvIw0hbAo0QIEIBllAkyuSElGuZJRJsDkipSkkWtExRAMwzCM/WekWQiGYRjGfmIKwTAMwwAwQhWCiHxNRFREihMtCwCIyK0iskpEVojIcyIyKdEyAYCI3CEi672yPSYihYmWCQBE5GMislZEPCKS0HQ8ETlTRN4XkY0iclMiZXEQkftEZI+IrEm0LP6ISLmILBGRdd7P7ytJIFO2iPxPRFZ6ZfpBomXyR0RcIvKuiCxOtCzACFQIIlIO4DQAtYmWxY87VHWOqs4DsBjAdxMsj8PzAGar6hwAHwC4OcHyOKwBcCGApYkUQkRcAH4P4CwAhwC4VEQOSaRMXh4AcGaihQiAG8BXVfVgAEcDuCYJ3q9uACer6lwA8wCcKSJHJ1akfnwFwLpEC+Ew4hQCgF8B+AaApImWq2qL39U8JIlsqvqcqrq9V98AMCWR8jio6jpVfT/RcgA4EsBGVd2sqj0A/gbg/ATLBFVdCqAh0XIMRFV3quo73v9bwYVucoJlUlVt817N8F6S4vcnIlMAfBjAvYmWxWFEKQQROQ/AdlVdmWhZBiIit4nIVgCXIXksBH8+A+DpRAuRZEwGsNXv+jYkeIFLFUSkEsB8AG8mWBTHLbMCwB4Az6tqwmXy8mtw8+pJsBz7SLl5CCLyAoAJAW66BcC3AJweX4lIMLlU9XFVvQXALSJyM4BrAXwvGeTynnMLaO4/FA+ZwpUrCZAAx5Jid5nMiEg+gH8BuH6AdZwQVLUPwDxvjOwxEZmtqgmNv4jIOQD2qOrbInJiImXxJ+UUgqqeGui4iBwGYBqAlSIC0P3xjogcqaq7EiVXAP4K4EnESSGEkktEPgngHACnaByLUiJ4vxLJNgDlftenANiRIFlSAhHJAJXBQ6r6aKLl8UdVm0TkJTD+kuiA/LEAzhORswFkAygQkb+o6uWJFGrEuIxUdbWqlqpqpapWgj/mw+OhDEIhIjP9rp4HYH2iZPFHRM4E8E0A56lqR6LlSULeAjBTRKaJSCaASwD8J8EyJS3CndgfAaxT1V8mWh4AEJESJ3tORHIAnIok+P2p6s2qOsW7Vl0C4MVEKwNgBCmEJOd2EVkjIqtAl1bC0/G8/A7AGADPe1Ni/5BogQBARC4QkW0AjgHwpIg8mwg5vAH3awE8CwZI/6GqaxMhiz8i8jCAZQBmicg2Ebkq0TJ5ORbAFQBO9n6fVnh3wIlkIoAl3t/eW2AMISlSPJMRa11hGIZhADALwTAMw/BiCsEwDMMAYArBMAzD8GIKwTAMwwBgCsEwDMPwYgrBMAzDAGAKwRghiEiliHR6e9Y4xwa1rhaRHG9+fM9w26N7H+tlb1dUiMiXva2fI2oBIiKFIvKl4cgSxnMMapktIpkislREUq5jgREbTCEYI4lN3hbjQ7auVtVO7znRaEHxGQCPenvlAMCXAJytqpdF+DiF3vtGhJBwf8MPYEDLbG8H1/8CuDjS5zZGJqYQjJTBO3zlNO//PxKRO4OcHo/W1ZcBcBoE/gHAdAD/EZEbRORy72CWFSJyj58V8W8Reds7rOVq7+PcDmCG99w7vNaO/07+ayLyfe//lV4r5C4A7wAoH+q5/AnSMvvf3tdhGKYQjJTie2DH2MvA1so3BDk3pq2rvb2NpqtqNQCo6hdAq+MkAM+Au+5jvdZIH3yL7mdU9QgAVQC+LCJFAG6C17pR1a+H8fSzADyoqvMB5AZ5rnBYA2BBBOcbIxjzHRopg6ou9TZQuxHAiY6rRkRuBZuq+RPr1tXFAJqGuO0UAEcAeMvbeTcH7MUPUAlc4P2/HMBMAJE2YKxR1TfCeK6QqGqfN54yxjvUxhjFmEIwUgZvi/OJAPY6i5eITEDg73HEratF5BoAn/NePRvABf7XVdX//p1g2+KADwXgT6rabySpt+/9qQCOUdUObyvmQI/hRn/rfeA57aGeK0KyAHQN4/7GCMFcRkZKICITwQE+5wNoF5EzvDfNB7AiwF0ibl2tqr/3um3mqeqOgdcHnNsIwCUigRb0/wK4SERKvbKPF5GpAMYCaPQqg4PAucMA0Ap2nXXYDaBURIpEJAucVzEUQz1XWHhdVnWq2hvufYyRiykEI+kRkVwAj4ID3NcBuBXA9703z0MAhRCn1tXPATguwHO/B+DbAJ7ztl1+HrRsngGQ7j12KzjHGqpaD+A1b4v0O7yL8w/B8ZOLEaR/f5Dn6keQltknAXhqf168MfKw9tdGSiMifwTdOhUAFqvq7DDvVw2gSlX3DuO55wO4UVWv2N/HSDQi8iiAm1X1/UTLYiQesxCMlEZVr1JVD5hdM9a/MC0QTmEagAwMc7i5qr4LDl8ZlOaZCnhdaf82ZWA4mIVgGIZhADALwTAMw/BiCsEwDMMAYArBMAzD8GIKwTAMwwBgCsEwDMPwYgrBMAzDAGAKwTAMw/BiCsEwDMMAAPw/yjotrlRUg/EAAAAASUVORK5CYII=", |
| 1007 | "text/plain": [ |
| 1008 | "<Figure size 432x288 with 1 Axes>" |
| 1009 | ] |
| 1010 | }, |
| 1011 | "metadata": { |
| 1012 | "needs_background": "light" |
| 1013 | }, |
| 1014 | "output_type": "display_data" |
| 1015 | } |
| 1016 | ], |
| 1017 | "source": [ |
| 1018 | "plot_dataset(train_x, train_labels)" |
| 1019 | ] |
| 1020 | }, |
| 1021 | { |
| 1022 | "cell_type": "markdown", |
| 1023 | "metadata": { |
| 1024 | "id": "SjPlpf2-wHl8" |
| 1025 | }, |
| 1026 | "source": [ |
| 1027 | "## Training One-Layer Perceptron\n", |
| 1028 | "\n", |
| 1029 | "Let's use PyTorch gradient computing machinery to train one-layer perceptron.\n", |
| 1030 | "\n", |
| 1031 | "Our neural network will have 2 inputs and 1 output. The weight matrix $W$ will have size $2\\times1$, and bias vector $b$ -- $1$.\n", |
| 1032 | "\n", |
| 1033 | "To make our code more structured, let's group all parameters into a single class:" |
| 1034 | ] |
| 1035 | }, |
| 1036 | { |
| 1037 | "cell_type": "code", |
| 1038 | "execution_count": 26, |
| 1039 | "metadata": { |
| 1040 | "id": "J1KaixW-cMWJ" |
| 1041 | }, |
| 1042 | "outputs": [], |
| 1043 | "source": [ |
| 1044 | "class Network():\n", |
| 1045 | " def __init__(self):\n", |
| 1046 | " self.W = torch.randn(size=(2,1),requires_grad=True)\n", |
| 1047 | " self.b = torch.zeros(size=(1,),requires_grad=True)\n", |
| 1048 | "\n", |
| 1049 | " def forward(self,x):\n", |
| 1050 | " return torch.matmul(x,self.W)+self.b\n", |
| 1051 | "\n", |
| 1052 | " def zero_grad(self):\n", |
| 1053 | " self.W.data.zero_()\n", |
| 1054 | " self.b.data.zero_()\n", |
| 1055 | "\n", |
| 1056 | " def update(self,lr=0.1):\n", |
| 1057 | " self.W.data.sub_(lr*self.W.grad)\n", |
| 1058 | " self.b.data.sub_(lr*self.b)\n", |
| 1059 | "\n", |
| 1060 | "net = Network()" |
| 1061 | ] |
| 1062 | }, |
| 1063 | { |
| 1064 | "cell_type": "markdown", |
| 1065 | "metadata": { |
| 1066 | "id": "rQ7W6TOacIAI" |
| 1067 | }, |
| 1068 | "source": [ |
| 1069 | "> Note that we use `W.data.zero_()` instead of `W.zero_()`. We need to do this, because we cannot directly modify a tensor that is being tracked using *Autograd* mechanism.\n", |
| 1070 | "\n", |
| 1071 | "Core model will be the same as in previous example, but loss function will be a logistic loss. To apply logistic loss, we need to get the value of **probability** as the output of our network, i.e. we need to bring the output $z$ to the range [0,1] using `sigmoid` activation function: $p=\\sigma(z)$.\n", |
| 1072 | "\n", |
| 1073 | "If we get the probability $p_i$ for the i-th input value corresponding to the actual class $y_i\\in\\{0,1\\}$, we compute the loss as $\\mathcal{L_i}=-(y_i\\log p_i + (1-y_i)log(1-p_i))$. \n", |
| 1074 | "\n", |
| 1075 | "In PyTorch, both those steps (applying sigmoid and then logistic loss) can be done using one call to `binary_cross_entropy_with_logits` function. Since we are training our network in minibatches, we need to average out the loss across all elements of a minibatch - and that is also done automatically by `binary_cross_entropy_with_logits` function: \n", |
| 1076 | "\n", |
| 1077 | "> The call to `binary_crossentropy_with_logits` is equivalent to a call to `sigmoid`, followed by a call to `binary_crossentropy`" |
| 1078 | ] |
| 1079 | }, |
| 1080 | { |
| 1081 | "cell_type": "code", |
| 1082 | "execution_count": 27, |
| 1083 | "metadata": { |
| 1084 | "id": "kdDxWeCqwHl8", |
| 1085 | "trusted": true |
| 1086 | }, |
| 1087 | "outputs": [], |
| 1088 | "source": [ |
| 1089 | "def train_on_batch(net, x, y):\n", |
| 1090 | " z = net.forward(x).flatten()\n", |
| 1091 | " loss = torch.nn.functional.binary_cross_entropy_with_logits(input=z,target=y)\n", |
| 1092 | " net.zero_grad()\n", |
| 1093 | " loss.backward()\n", |
| 1094 | " net.update()\n", |
| 1095 | " return loss" |
| 1096 | ] |
| 1097 | }, |
| 1098 | { |
| 1099 | "cell_type": "markdown", |
| 1100 | "metadata": { |
| 1101 | "id": "zAAgw0h6KzUd" |
| 1102 | }, |
| 1103 | "source": [ |
| 1104 | "To loop through our data, we will use built-in PyTorch mechanism for managing datasets. It is based on two concepts:\n", |
| 1105 | "* **Dataset** is the main source of data, it can be either **Iterable** or **Map-style**\n", |
| 1106 | "* **Dataloader** is responsible for loading the data from a dataset and splitting it into minibatches.\n", |
| 1107 | "\n", |
| 1108 | "In our case, we will define a dataset based on a tensor, and split it into minibatches of 16 elements. Each minibatch contains two tensors, input data (size=16x2) and labels (a vector of length 16 of integer type - class number)." |
| 1109 | ] |
| 1110 | }, |
| 1111 | { |
| 1112 | "cell_type": "code", |
| 1113 | "execution_count": 28, |
| 1114 | "metadata": { |
| 1115 | "colab": { |
| 1116 | "base_uri": "https://localhost:8080/" |
| 1117 | }, |
| 1118 | "id": "PfyqjVb2wHl8", |
| 1119 | "outputId": "b3a685a9-304c-4e7e-adf9-2858cc47c3a5", |
| 1120 | "trusted": true |
| 1121 | }, |
| 1122 | "outputs": [ |
| 1123 | { |
| 1124 | "data": { |
| 1125 | "text/plain": [ |
| 1126 | "[tensor([[ 1.5442, 2.5290],\n", |
| 1127 | " [-1.6284, 0.0772],\n", |
| 1128 | " [-1.7141, 2.4770],\n", |
| 1129 | " [-1.4951, 0.7320],\n", |
| 1130 | " [-1.6899, 0.9243],\n", |
| 1131 | " [-0.9474, -0.7681],\n", |
| 1132 | " [ 3.8597, -2.2951],\n", |
| 1133 | " [-1.3944, 1.4300],\n", |
| 1134 | " [ 4.3627, 3.1333],\n", |
| 1135 | " [-1.0973, -1.7011],\n", |
| 1136 | " [-2.5532, -0.0777],\n", |
| 1137 | " [-1.2661, -0.3167],\n", |
| 1138 | " [ 0.3921, 1.8406],\n", |
| 1139 | " [ 2.2091, -1.6045],\n", |
| 1140 | " [ 1.8383, -1.4861],\n", |
| 1141 | " [ 0.7173, -0.9718]]),\n", |
| 1142 | " tensor([1., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 0., 1., 1., 1., 1.])]" |
| 1143 | ] |
| 1144 | }, |
| 1145 | "execution_count": 28, |
| 1146 | "metadata": {}, |
| 1147 | "output_type": "execute_result" |
| 1148 | } |
| 1149 | ], |
| 1150 | "source": [ |
| 1151 | "# Create a tf.data.Dataset object for easy batched iteration\n", |
| 1152 | "dataset = torch.utils.data.TensorDataset(torch.tensor(train_x),torch.tensor(train_labels,dtype=torch.float32))\n", |
| 1153 | "dataloader = torch.utils.data.DataLoader(dataset,batch_size=16)\n", |
| 1154 | "\n", |
| 1155 | "list(dataloader)[0]" |
| 1156 | ] |
| 1157 | }, |
| 1158 | { |
| 1159 | "cell_type": "markdown", |
| 1160 | "metadata": { |
| 1161 | "id": "xrwgkbQjhkEp" |
| 1162 | }, |
| 1163 | "source": [ |
| 1164 | "Now we can loop through the whole dataset to train our network for 15 epochs:" |
| 1165 | ] |
| 1166 | }, |
| 1167 | { |
| 1168 | "cell_type": "code", |
| 1169 | "execution_count": 78, |
| 1170 | "metadata": { |
| 1171 | "colab": { |
| 1172 | "base_uri": "https://localhost:8080/" |
| 1173 | }, |
| 1174 | "id": "QGchp9D6gVJa", |
| 1175 | "outputId": "b4c4751d-cb56-4104-d5b5-f1ae9d3d858d" |
| 1176 | }, |
| 1177 | "outputs": [ |
| 1178 | { |
| 1179 | "name": "stdout", |
| 1180 | "output_type": "stream", |
| 1181 | "text": [ |
| 1182 | "Epoch 0: last batch loss = 0.6491\n", |
| 1183 | "Epoch 1: last batch loss = 0.6064\n", |
| 1184 | "Epoch 2: last batch loss = 0.5822\n", |
| 1185 | "Epoch 3: last batch loss = 0.5679\n", |
| 1186 | "Epoch 4: last batch loss = 0.5592\n", |
| 1187 | "Epoch 5: last batch loss = 0.5537\n", |
| 1188 | "Epoch 6: last batch loss = 0.5501\n", |
| 1189 | "Epoch 7: last batch loss = 0.5478\n", |
| 1190 | "Epoch 8: last batch loss = 0.5463\n", |
| 1191 | "Epoch 9: last batch loss = 0.5454\n", |
| 1192 | "Epoch 10: last batch loss = 0.5447\n", |
| 1193 | "Epoch 11: last batch loss = 0.5443\n", |
| 1194 | "Epoch 12: last batch loss = 0.5441\n", |
| 1195 | "Epoch 13: last batch loss = 0.5439\n", |
| 1196 | "Epoch 14: last batch loss = 0.5438\n" |
| 1197 | ] |
| 1198 | } |
| 1199 | ], |
| 1200 | "source": [ |
| 1201 | "for epoch in range(15):\n", |
| 1202 | " for (x, y) in dataloader:\n", |
| 1203 | " loss = train_on_batch(net,x,y)\n", |
| 1204 | " print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))" |
| 1205 | ] |
| 1206 | }, |
| 1207 | { |
| 1208 | "cell_type": "markdown", |
| 1209 | "metadata": {}, |
| 1210 | "source": [ |
| 1211 | "Obtained parameters:" |
| 1212 | ] |
| 1213 | }, |
| 1214 | { |
| 1215 | "cell_type": "code", |
| 1216 | "execution_count": 29, |
| 1217 | "metadata": { |
| 1218 | "colab": { |
| 1219 | "base_uri": "https://localhost:8080/" |
| 1220 | }, |
| 1221 | "id": "5QaDiCQUkFOT", |
| 1222 | "outputId": "45b4a66b-1222-40f4-c758-d58f1c7daf8c" |
| 1223 | }, |
| 1224 | "outputs": [ |
| 1225 | { |
| 1226 | "name": "stdout", |
| 1227 | "output_type": "stream", |
| 1228 | "text": [ |
| 1229 | "tensor([[ 0.1330],\n", |
| 1230 | " [-0.2810]], requires_grad=True) tensor([0.], requires_grad=True)\n" |
| 1231 | ] |
| 1232 | } |
| 1233 | ], |
| 1234 | "source": [ |
| 1235 | "print(net.W,net.b)" |
| 1236 | ] |
| 1237 | }, |
| 1238 | { |
| 1239 | "cell_type": "markdown", |
| 1240 | "metadata": { |
| 1241 | "id": "s4_Atvn5K4K9" |
| 1242 | }, |
| 1243 | "source": [ |
| 1244 | "To make sure our training worked, let's plot the line that separates two classes. Separation line is defined by the equation $W\\times x + b = 0.5$" |
| 1245 | ] |
| 1246 | }, |
| 1247 | { |
| 1248 | "cell_type": "code", |
| 1249 | "execution_count": 30, |
| 1250 | "metadata": { |
| 1251 | "colab": { |
| 1252 | "base_uri": "https://localhost:8080/", |
| 1253 | "height": 283 |
| 1254 | }, |
| 1255 | "id": "PgRTHttLwHl9", |
| 1256 | "outputId": "d9abf92f-cb70-4c56-ccd0-5e027239da58", |
| 1257 | "trusted": true |
| 1258 | }, |
| 1259 | "outputs": [ |
| 1260 | { |
| 1261 | "name": "stderr", |
| 1262 | "output_type": "stream", |
| 1263 | "text": [ |
| 1264 | "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_89704/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n", |
| 1265 | " fig.show()\n" |
| 1266 | ] |
| 1267 | }, |
| 1268 | { |
| 1269 | "data": { |
| 1270 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABbH0lEQVR4nO2dd3jUVfaH3zvpPSEJJbTQERAIBFGwIYJYsCL2QnbXLe6u61Zdt7tVV1fX1d+uqwMiit21gIgFRFGUktBBioReQkivM3N/f5wZ0yaTSTItk/s+Tx7I5JuZkyn33HvK5yitNQaDwWAwWIJtgMFgMBhCA+MQDAaDwQAYh2AwGAwGJ8YhGAwGgwEwDsFgMBgMToxDMBgMBgMAkcE2oDNkZGTo7OzsYJthMBgMXYr169cXaa0zm9/epR1CdnY269atC7YZBoPB0KVQShW6u92EjAwGg8EAdPETgsFgMHQrampg716orYWUFMjOBovv9vXGIRgMBkOoU18Pb70F774r/wfQGnr2hGuvhdxcnzyMCRkZDAZDKGOzweOPwxtvQHo6DBggXwMHinN49FFYscInD2UcgsFgMIQyn34KGzZIeCg6uunPkpOhXz9YtAiKijr9UMYhGAwGQ6iiNSxdChkZoJT7a2Ji5LpPP+30wxmHYDAYDKFKaSkcOyYnAU+kpkJBQacfzjgEg8FgCFW0bv1k0BiLBez2Tj9cyDgEpVSsUuoLpdRGpdRWpdTvg22TwWAwBJWkJEhIgOpqz9eVlMCQIZ1+uJBxCEAtcIHWehwwHpillDozuCYZDAZDEImMhIsukrBRazgcUol03nmdfriQcQhaqHB+G+X8MvM9DQZD9+b886F3bzh8WEJIjbHZYN8+mDZNSlE7Scg4BAClVIRSqgA4Dryntf7czTV3KKXWKaXWnThxIuA2GgwGQ0BJTIRf/AIGD4bCQti/Hw4dEkdw+DBceincdJN3uYY2ULq5xwkBlFKpwOvAD7TWW1q7Ljc3VxtxO4PB0C3QGg4ehE2boKpKSlEnTBAJi3ailFqvtW7R3hyS0hVa6xKl1EpgFtCqQzAYDIZug1LQv798+YmQCRkppTKdJwOUUnHAhcCOoBplMBgM3YhQOiH0AZ5RSkUgjuolrfXbQbbJYDAYug0h4xC01puAnGDbYTAYDN2VkAkZGQwGgyG4GIdgMBgMBsA4BIPB0BVxOHyi3WNoSsjkEAwGQxBwOGD7dnjvPdi9W0TSTj8dLrhAGqF80OzkMxwO2LIFli0Tm7WWITEXXyz1+M1nBRjajXEIhlY5fly+LBaZwdGWAq+hi1FXB//9L3z+uXTDpqbKIrtuHXzyiXTAXnttaDgFux3mz4dVq+SN6JJpKC2FJ56A0aPhhz+EuLjg2tnFMQ7B0IKDB+Gll2Dz5qbzu6dOhWuu6VBjpCEUeeEF+OILGDSo6aLfp48swG+9BT16wIUXBs9GF0uXijNoPlQ+NVXekDt2wMKF8O1vB8vCsMDkEAxN2LcP7r8fdu1qaIrs31/WiNWr4S9/kU2ZoYtTUgIrV8pO290JICICsrJkjq9rqHuwqK0Vh9C3b1Nn4MLVwbtmDRh9s05hHILhaxwO+L//k4l8vXo1/exFRspnrqhITg+GLs7GjfKCR0S0fk1cHFRUSG7B19TWwpdfwtatciT1pKm2a5dcHxPT+jUWi9zH5s2+t7UbYUJGhq/58kuRXc/Obv2aPn3gs89g7lwTOurSlJZ6dgYulBKn4Cvq6mS3v3w51NTI/dvt8qa79lrJBTSnpsa7+46IgPJy39naDTEnBMPXfPll22uE6+eFhf63x+BHkpK8L9v0VaK2vh4efxxef112EwMGyLFz4EA4dQoeeADWrm35ewkJ3t2/3Q5pab6xtZtiTgiGr7HZvCsoUUqiDYYuzJgxDS+ku7g8SJgmOhqGDfPNY378MWzY0LKcVSlZyGNj4cknYcQIeTNWVoozGjKkYYxka87Jbm8omTV0GOMQDF/Tr598Dj2htXz2MjMDY5PBT2RmwuTJUnLqLrHscMgQljlzPMfuvcXhkFBRr16t7zri4iRJ9eMfy5ssIkL+HTxYQkmffCIVUc2PsVrL0Jjp080JoZMYh2D4mrFjZZNWUyP/uuPkSdkwZmUF1jaDH7jtNqk22rZNyktdSaGTJ6GsDM45R3oRfMHJk1Bc7HnM4/79Uj6algYzZojj0FqcxJdfyo5l/355c6any89PnZIcxxlnwPXX+8bWboxxCIaviY2VSXxPPikVfs2dQmmpOIsbbwyNXiVDJ4mLk934hg2wZIkkhpSCkSNh1iwJv7QWTmovDofnN015OeTnS2goLq7hWlc4KTERDhwQJ7Znj1QTORyyO5kxA047zbskucEjxiEYmnD22bIpe+45KQiJipLvbTbpAfr5z+XUbggToqPhzDPly26XBdhXTqAxqalSu1xX515iwlWl4HC4D/tERYmj2L3bNJ/5EeMQDE1QCs49FyZNkg1bYaFsvEaOhFGj5DNtCFP8ucOOiYFp06Tc1F3Y6OBBWfCrqloPK/XsKTmPO+4wR1Q/YT7eBrfExcGUKfJlMPiEiy6SBf3YMVncGy/q9fVyehg2rHXRLIulQeXU7Ez8gnlWDQaDZw4fFk0TV3nZsGEdO02kpcE994gYnevo6QojaS2JqzFjWv/9qipJJhtn4DfMM2swGNxz4gQsWCDyEo1JT5fk7tix7b/PXr3gd7+TXMDmzbLI9+4t1QzPPus5f3HiBNxyS/sf0+A1xiEYDIaWnDwJf/qTNIMNHNg0vFNeDg89JHLTEye2/76VklNG44a3ujppXDt4UMpLm3P0aEPvhMFvGOkKg8HQkldflU7hPn1aJnCTkiQH8PTT0s3sC6KjpQR20CAJTx05Ij0GR4/K9xkZUuKWmOibxzO4JWROCEqp/sBCoDfgAJ7UWj8aXKsMIHm83btlA3fihHwmp0yRMvWoqGBbZ/A5JSUiJd23b+vXJCRIw9jGjdIU5gtSUuDee+XNtnq1nFJSUuTNNnKk6TMIACHjEAAb8BOt9QalVBKwXin1ntZ6W7AN686Ul4se2Y4dUjkYHy/9QevWyabt7rs9rxuGLsiRI3IqaGsBjo4WaWpfOQSQHMLw4fJlCDghEzLSWh/RWm9w/r8c2A6YpSaI2Gzwz3/KZ37gQIkepKRIKDc7W7qW//pXOdkbDIauTyidEL5GKZUN5ACfB9mUbs22bSIhk53tvg8oI0OkZVauhKuuCrR1IUxJCXz6KXz0kejsuMZQ5uZ6L+UcTPr0aVAx9HRKqK31nRKqISQImROCC6VUIvAq8COtdZmbn9+hlFqnlFp3wozL8yvvvy/5Ak9Nob16wXvveS+tH/bs3Cm19i+/LE9KaqokZ+fPh1//Wmr6Q53UVAkDHTnS+jVVVdK9OG5cwMwy+J+QcghKqSjEGTyntX7N3TVa6ye11rla69xMo8Hsc44fhzffhH/9C959Vza4nhb7mBgJHVVXB87GkOX4cSnHjI+XGFtiomTdk5PlmFVbKz+vqgq2pW0zZ47Yf+RIy/GW5eXSbfyNb/hGGtsQMoRMyEgppYCnge1a64eDbU93w2aTWcnvvScngvh4KSI5eFASypMnS+SjOQ6HfJlqI+DDD8V7tia9kJkpHbobNoiKYCiTkQG//CVYrfIGcElRg7wR7r4bxo8PqokG3xMyDgGYCtwCbFZKFThv+6XWemnwTOo+vPiinAgGDmwIG48cKU2qSkkV4HnntVzrTp6U8tNuv1F0OGDFComheSI1VWJxoe4QQHoN7rlHBuU0lq4YPtyUgIYpIeMQtNafAEbCMAgcPy4ng8bOAKRhdMeOBjWBnTtFBdWFzSbRg1mzAmtvSFJX17q0c2NiY7teWVbfvqa2uJsQUjkEQ3D49FNxBM03fXFxUhjjyg8cOiT5Aq1lTSsslOqiUaMCb3PIERUlT2BbM0jr6qTTtw20Fkd94IAMLzMYAkHInBAMwePgQckZuKNPH5mPsGuXNJDu3SuVk4MGwe23Sxg57KTpKyth7Vr44gtJBPfrJ+Mkhwxp/Y+NiJAw0OrVnnfTxcVw882t/lhraRJ+6y1RbXCF7idMgNmz5RRnMPgL4xAMREd7riRKS5MqxPR00TMbMkRuCztHANJ88dhjcixKSZGF/sAB6SmYMEGGs7Q2cHr6dFi1Sn43Lq7lz0tLxfM2jrs1QmtYtEhmyGRmQv/+8hw7HLBliwwsuvtuzwrRBkNnMCEjAxMmtF0JWVUlxSU5OfJvWDqD/fvh4Ydl0c7OFq+XnAxZWbI137AB/vvflmWYLvr3h+98RwSfDh2SoS8gp4z9+8VR/PjHrVYhrVsnzmDQILnE9RxbLKIQnZYmMiIVFb7/0w0GMCcEAyJrn5wsG9iUlJY/11rCFzfeGOazSd5+W/5AdzF+pcQprF8vi3trsZszzhAH8uGHogZYXy8OZvZsib2lp7v9Na1lzn2PHq2PBEhMlKqutWtlGqXB4GvC+eNt8JLoaAkFPfCAnAR69WpYlKqqxBnk5EhEJGwpK5MtujstfhdKicP45BPPwfx+/eDWW2WYi80mv9PGkaqsTJL0rY0TdpGcLFMojUMw+APjEAwADB0Kv/kNvPGGbIJdycykJDkZTJ/edvPZ0aMNagf9+kkcvMtQViZ/tKeJXSAZdU+SDo1RyuuOvfp6eei2QnGRkb4bQWAwNMc4BMPX9OsHd94p2mynTsni06dP22GiQ4fgueckH2uxNITYx44VZ9JWr1ZIEBUl2VutPa/KNpv7hHEnSUqS/HV9vWcfUlEBp53m84c3GACTVDa4ITVVEpv9+7ftDA4cgPvvh6++kijKgAHyb//+0tT2xz/KySHk6dlTMrfl5Z6vq6z0yxjHmBjpBD92rPVrtJbTwXnn+fzhDQbAOARDJ9AannpKdra9ejXdWFsscrqor4dnngmejV6jFFx2mXSDORzurzl1Skp9OjJc3gtmzhTHUFzc8mdaS45h3Dgp+zUY/IFxCIYOs2+fLFIZGa1f06sXbN/eNVSfOessmDFDjjslJQ2xr/p66d6rq4O77mpbnqKDZGbCL34hp7J9+6R69dQpCckVFkp58He+03aaw2DoKCaHYOgwhYXyr6eQu1LytX+/VGOGNBaLVAaNHCk1oPv3y20Wi5SMzpoloSU/MmCATKHbvFmqiaqqJJJ19tkSigvL/g9DyGAcgqHDtBZZAdlcOxwNu9nWerlCDotFcgRnnCGnBJtNMr6tdSf7gehomDhRvgyGQGIcgqEFx45J89OxY9IMlZMjZamu3anr3969W/7uqVOid3TokDiE6GhZT1vTSgpZlJJ8gcHQjTAOwfA1dXWwcKH0XVkssimurYXFi0V1ISNDFvjsbLj4Ysmt9ughJfzJyRJ637hRksyJiXIf5eUyaMdqhZ//3KgoGwyhjElPGQAJ6fz3v6K2MGCAfGVmSsXLV19JTnXPHkkSFxfLiM1HH4UbbpDv9++HggI5DSQliTOoqZFTwrnninjeww+bpiqDIZRpt0NQSiUopcy4pDBj925Re87Oboj779kjX6mp4hxqaiSRnJYmfQo7dkhX849/LPLYtbXSOFVaKuF3pWDqVJHvycgQx7FxYxD/SIPB4JE2Q0ZKKQtwPXATMAmoBWKUUieApcCTWutdfrXS4HdWrpQQkSs/YLfDl182Vd1MTBTHMWyYOI3+/aUS5qKLpDZ+1KgGBYjUVHECjUskExPlBHLGGYH+6/xIW53NBkMXwpscwgrgfeBeYIvW2gGglOoBTAP+qpR6XWu9yH9mGvzNvn1NVZmLixuEOl1ERkqjbl2dOA/XYr9hg+QN+vSRr9aIjvbR9K+aGhkOsHu3LMbDh0vHVqAGOxcWyvzkNWvkWNS7t3jFM87ogtlzg6EBbxzChVrr+uY3aq2LgVeBV5VS3il4GUKWqChZZ124mwSptXw13vU3Fluz2z3PXq+p8cHErzVrYMECubPYWDHogw9EX+iOO2SEmz/58EPJvEdFSU9CZKTEyRYsgGXL4Gc/a1Xi2mAIddrMIWit65VSI5VS05VSiY1/ppSa5brGXwYaAkNubtPZ71FRLXsHamokFNRYfM1mk0Rzbq6oPniisrKTOjzr1sETT8jQhuxs2Zn36SNeJiEBHnlERov5i61bZeHPypJyqagoOaEkJYk9JSVig6fxcwZDCNOmQ1BK/RB4A/gBsEUpdUWjH//ZX4YZAstZZ8na5jol9OghIR7X0C+tpfR02LCGkLmrMW3cOClDra9vffLakSOScxg9uoMG2mzw7LPifdypjSYkiNHPPuu5Y64zvPmmOKPG0hVVVbBzp4w6W79ehiE/95w8WQZDF8ObKqNvARO11lcC5wO/Vkrd5fyZT7NpSimrUuq4UsqP2zyDO9LTIS9PNIdOnZI1NSNDqoy2bWvYeFdUSNPa6tUSRh8+XH43Oxu+/3353f37ZZ2sq5NN8759cs2PftSJiWs7dkgCIiGh9WuSk+WYsmdPBx/EA8XFkmXv0aPhtmPHJFy1Y4d8Hxsrp4N//1uGS7R1ZDIYQgxvPp4RWusKAK31PqXU+cArSqmB+NghAAuAfwELfXy/Bi+YOlU2wIsXw9Klsv5GRMjmPDFRdvn798s1cXGQmmxj2ydl/OPucr5zWw0Txg7hL3+J5JNPpLmtrEzC7NdfL6H9Tqk/HD/uvf7FiRNylPElVVXyZLiOR2Vlks+Ii2t6YkhIkMRyRQU89BD84Q+BS3Yb3FNXJzuaXbvkPTR4cGCLELoQ3jiEo0qp8VrrAgCtdYVS6jLACpzuS2O01quUUtm+vE9D+xg9WvoMRoyQzXBEhMT+339f1sLYWKirdTCl11cMs21HHbSx+ct0Hv/kMD+Z8n9kzLmaK684hyuv9PFeoT1HC0+Z7Y6SkCC7f1eZ6d698m9z5VObTRxCr14NrdthVWfbxdi0STouy8sbXqtly+Q1mjcPJk0Krn0hhjcho1uBJiNOtNY2rfWtwLl+scoDSqk7lFLrlFLrTpw4EeiHD3v275fN1PDh0ozmkqbo0UP6DE4b4eDcmC9I3rcJlRCHSktlQJadLbYR7KzsJx++JUt8b9igQfKvp1OCK3fgutaXpKWJCurJk+IY9u+XY1NzbDZJloAcpVas8L0tBu/Ytg3+8Q85CWRnSzFAVpb8PzERHntM8j6Gr/Gmyuig1trtzCut9Wrfm9SmPU9qrXO11rmZXWpob9dg7dqmkRG7XTa6rrUvqewgaTWHKVWp1NlkJ64UxEfWs+LoSNG8eOUV3w9A6NdPwkCe4vJHj4rAkr8kqi+/XLxjZWXL+luQXWhqakPZaWys+2k3Bv/jcEiBQUqKe8cdHy/vk4UL3ddYd1OMlpGhCcXFTWP99fWN+gu0Jv3kLuqj4kEp7Ha+/kqIquNgWbKUYkZEwKpVvjVMKfjmN+X+Dx1qWtpps8ksz6QkuPVW3z5uY047TTLvRUWSU6ivl4Wnpkay5/HxIp3tchR1dWKTIfDs3SsbhNTU1q9JTBSdFVdRgMGonRqakpbWVIDOYmloSItw1BNdV05NTAr11ZJkdpWl1mBhQM866uogukcPmfBy/fW+Na5XL/j1r+G110Qzw2UcwJQpcPXV/pesPv980QL/9a9F/CkmRhaWUaOkJ6Jxk0ZJCcyZ4197DO5pTzj52DEYM8Z/tnQhvHYISimF6BkN1lr/QSk1AOittf7CV8YopRYjpa0ZSqmDwG+11k/76v4NbTNpkqQAXLnT6GiJgJSXQ1IMgKKsTH0tXxETI9cVV8WSVrmLjz6Csyco4iL91AuQmQnf/jZcd53sAEHiwo11N/xNv35SVvr730tznLvyqeJiCVdMmBA4uwwNeDtnVGv/FCF0UdoTMnoCOAu4wfl9OfC4L43RWt+gte6jtY7SWvczziDwDBwokZFDhxpuGzZMTg31RFFhi8VWVUdSUkOjbo09iugIO5P6HKKmBrZ/VoL2ddlnc1JTJck7cmRgnYGLQYPgW9+S3eWhQw1x6JoaSTg7HCIDa7SNgkN2dsPYvtZwnS79UYTQRWmPQ5istb4TqAHQWp8C/DNt3BA0lILvflc2wV99JSHWnj2ldPvwEUV+7UjSoiuJiQGHVhTXJVBaH8+VWWtJjKwlKcFBVUkt+wdPC/af4n+mTpVTwtSpclopLJSE85VXwv33+0C4ydBhevWSXoNjx1q/5vhxkekdMCBwdoU47ckh1DvnIGgApVQm4Ke4gCGYJCfDPfdIRd6yZZKvTU+HCy6A1SuzOFF2kuiKCuqiEhmaeJQp6V/SJ64Ei8NGSul+1mVdwNET2XSL5XDAALj9drjtNjklREYaOexQ4ZZb4E9/khNc794NoSGHQxx4dLSc8szr9TXtcQj/BF4Heiql/gTMAX7lF6sMQSc2Vja+U6c2nKx37ACbLYreSSMYuelFsovWkRRRha6LwFJrx2GJZMfwy1nf4yrOqOwmH7KqqobpQMnJknDusD6HwadkZMCvfiVl0J9/3lSEa+JEuPZaOUkYvsard64zobwKWA9MRyQrrtRab/ejbYYQwfU5SkoS5xCVmsDe8/I4VnEpmSe2ElVXRU1cGsd6jaUuJoma/fJZDGvq6uD116WF22aTJ0lrcQrXXAPnnGN2nqFAenpDEYIrMdanT1NNKsPXeOUQtNZaKfU/rfVEwBTtdlP69pXcQmmp5HQrE3tRmdh0h+VwyFdYKwLU18tQ6U2bGmSwXVRVSbd2WRlcdlnwbDQ0JTXVc0+CAWhfUnmNUiqcP+aGNlBKyuqLi5v2KrjQWgpspk4N85P4mjVQUCBJ48bOAKSqaMAAePVVadQwGLoQ7Ql2TgO+rZQqBCqRsJHWWo/1i2WGkGTcOGnWXejUo+3RQ6T/v/xSGnhdLQHHjoWpU9BapGAzMloPCUVFSR38xx/D3Llt36fdLgmaFSsk2RkXJ412kya5l10wGPyE0l5KCjvlrlugtS70qUXtIDc3V69bty5YD9+tOX5c1rvFi2UtS04WQby0NImWOBwSSr/00jALpZeXww9/KKcAT39YRYVk5v/4x7bv77HHZMhOQoJ81ddLl3NcnAyRGDHCl3+BwYBSar3WOrf57V6fEIK58BtCj549xQnExUnZfePCmpQUWdNeeEE2uOefHywr/YCrhbstL+dKMnvCbhdnsGePNFI1vs/UVPGsf/+79DpkZXXWcoOhTdojXfEbd7drrf/gO3MMoUBxsWiD2WxSpDFkSEslgNpakRTq1899lWVUlORbX3lFcgrNQ+1dlsRE8XiVlZ6nt5WUwNlne76vnTsl1jZwoHsHk5wsJ4h33xXtfoPBz7Qnh1DZ6P+xwGWAKTsNI8rKZBzwF051KtcmNzMTbrxRpp652LZNVBo8DZ2Ki5PQ0vbtokodFlgsMkD6uedalzyw2+WIdN55nu/ro4/kSfJ02ujVS+aVXn+9+1nS3nLkiGibnzghTm3iRPH0wY7nlZZKj8D27fJmGzkSzjzTVAQFifaEjB5q/L1S6u/Amz63yBAUysvhr3+VBbxfv6Z6X2VlMmfkzjsbhn+Vlno30VJruTasOOccWaQPHpRjUONF1WYTCYuZM9uWrjh2rG2to8hIeRIrKzvmEGpqYMECqYyyWJwj7+rgnXdEj+TOOxvmNwQSrWHlSli0SBJOruT5xo3w8stwww0wfXrwHVY3ozMtlfHAYF8ZYgguS5bIJtLdGpbsHHPw1FOiEhwfL13/Wosjsdnk5wkJ7j+/zadMdnni4uCnP4X58yE/v0Ex026XBfyqq+CKK9pezOLj2x6g4xJo68iT6HDAf/4jNjYPS2ktjVoPPCBS3oGuZvr0U3n++vZtesxMTxeH9cwzcvs55wTWrm5Oe3IIm3HqGAERQCZwvz+MMgSW6mr48ENp4GwNV/hn/XoJjdfUSCk+NIwlcAmQ9u4tt9vt8rOwLJJJSpJqo2PHYOtWaUjr0UNiY94urlOnipdt3jVbVydPsFLy7/DhHVN03bkTNmxombAG+b5PH9i3T8rFLr64/fffUWw2eOkleaO4izlGR0sS/aWXJHwUNgmo0Kc9J4TGbZc24JjW2syeCwOOHZPPaFub0Ph4yR2UlcGLL8pm7tQpybFqLWvXp5/Kmjh0qGxAp0wJ83Bwr14db7iYMEGcR0mJPEkVFaKLdOBAw1SiujopPXUJ57WHDz6QF83TSaVnT0laX3SR9zMEOsuOHfIm8hRSi4uTN2ZYJaBCn/a8A76ntS50fh3SWtuUUn/zm2WGgOGqpPSG48clxNu/v2zeUlLEKdhsEp5OSZGTw+bNsjG98UZ/Wt7FiYuDu++WI5qrMe3AAYm9RUXJCzN8uOzgn3iiYTydtxQWtn2yiI+Xxbm6uuN/R3s5dcq7BJTrWkPAaI9DmOHmtgCeMw3+omdPcQieZo27ksNHj8paFRUlJ4qpUyWvYLPJzysq5PpBg+DnPzfzYdpk8GCZvlZeLkcsV2ImOVmOV+PGyTXr1sF777XvviMjPQ+IgUbzUQM4Naw9ISCjHBtQ2ny2lVLfBb4HDFZKbWr0oyTgU38ZZggcCQmSu1u1Snb+jXE44PBhObmfONGQK6iqkhN/dLRMVBsypEHfSClZ09xNljS4obhYvPKECRIiiohoGb/r00cqg2bM8H5Bdc1D9dQvUVwsL6Cn+mFfM3SovEkcjtbDVA6HXOPvyXuGJnhzQngemI2UmM5u9DVRa32TH20zBJDZsyXcc+RIw2ne4ZAk8mefybpxxhkN0YytW8WBuCINFotEQOLi5Od1dcH7W7ocGzaIA4iIkCfQXTInLk5KTxvPNm2Ls8+WF9OdEiHIC1xSApdcEtjyzowMyM2VnUZrHD4MOTniKA0Bo02HoLUu1Vrv01rfAJQBvYCBwBil1Ln+NtDge7SWUPXmzRK6rqmRQpd775W4f2GhFJ98+qn8PC5O8gVDhjTMREhNFWewbl3LcHB5uVFaaBc1Nd6FRiyW9uURevaUSW4HD8rC3/iFqqyUGakXXihhqUBzyy1i3759TR1Wba28ATMy4NZbA29XN6c9ZaffBO4C+gEFwJnAZ8AFfrHM4Be2bpVqvsLChtN6VJREImbPltGZhw5JNdFjj8G55zZtVBs2TE4NMTFSIHPyZMN8BBenTkljrcFL+vaVbl1PaC11vGlp7bvv886T33n1VdEmd9UIp6SIHMb55wen+Ss5GX75S6lweu+9BkcXFSUnllmzZPdhCCjtydjcBUwC1mitpymlRgK/96UxSqlZwKNIn8NTWuu/+vL+uxrl5bKJc2kKtSWw2RZr18pcl9TUpn1KdXXw1lvyWHfdJQ6gqkokK5rPH8/Kgt27pTAlKUnu4+hRuU+t5aQ/aJCc9g1eMnmyLNh2e+vJ3RMnYPRo2TmfPClxvH375Ppx4+QJb62TeexYOP10eXFcyZ3+/QObSHZHYqJI4l52mWing/x9gcxnGJrQHodQo7WuUUqhlIrRWu9QSvms5UgpFQE8jlQzHQTWKqXe1Fpv89VjdBUqK6W085NPGnJrdrt8hm+4AUaNav99lpfLIK/evVuuG9HREiravFnUBGbOlJO7O+cTGSnFL198IXmFmhr599Ah2eSNGCFqCI0/01rLiWT3brmmd29Z28Kug7mjZGZK6Obdd+WFaJ5oraiQJ/qqq2Rs51tvye0JCfIG+fxzKef67ncl9rdrlzzRGRkNT7RSchIJRWJiQte2bkZ7HMJBpVQq8D/gPaXUKcBDVqjdnAHs1lrvBVBKvQBcAXQrh1BZKWoCBw7IbtwVWtZawsAPPCB9So2F5rxh7VpZI1rbRColC/XSpXDBBXKidzjc9yjExkpV0qlT0nMwfLg4icmTpUKy8fVHj8KTT4p6Ksha53DI+nXjjVK2auRqkJm/dru0jEdEyO7ZbhdnEB8PP/mJxPtee02Od41zDpmZcoKYN0+OdElJDaGh+HiJ35kZzwYvaI+43VXO//5OKbUCSAGW+dCWvsCBRt8fBCb78P67BEuXym46O7vp7UpJKDg6Gv79bxGba4/WWUFB2yHZ+HiJRhQVybrSu7eEhlJSWl6rlNw+dCj84Q/uZ5afOAF//rOEvJpL6VRXi8yOzRZm8xI6SmSkJFFnzBDhvH375MWeOFHKUevr4eGH5YVpnoCurha9oupq8dKnn97wZFdXy9Gwvl7E4gwGD3jdmKaEm5VSv9Faf4Qklsf70BZ325cW7YxKqTuUUuuUUutOnDjhw4cPPjU1ojbgqUInIUGuy89v3327dIXawlUe7pqffPKk+6pFh0Mc1/nnu3cGINGNqqqGxrfGxMVJruK55+RUZHDSp4888T/9qWglTZ0qT9b69fKku+tB2LVLFv6MDHEIru5AkN/t3x+ef168u8HggfZ0Kj8BnAXc4Py+HIn5+4qDQOO2qH64CUlprZ/UWudqrXMzMzN9+PDB5+hR2ch5oym0ZUv77nvIEMkjeKK+XpyGq5Bl4kSJQhw7JiGs8nJZvI8ckYKVc86RnIY7SktFcdmTYF5MjDymmYLqBYcPu39j1NfLaSIxsWGSW3MZiujohlyDweCB9jiEyVrrO4EaAK31KcCXacG1wDCl1CClVDRwPd1s3kJbKgMuXEnm9jBlivyOp987cgSmTWsaipo2Df72NykEcTWlTZokisnf/GbrTbPHj4udbRWyxMU15BcMHoiJcf/iVVa2lJ5wdxRMSJDpbAaDB9qTVK53VgJpAKVUJuDlEtY2TrG87wPvImWnVq31Vl/df1fAdeBpS9iyqkp2/O2hd28p7V6ypGVOUmuJ9ycni+ilO7uuukq+vMU1bU1r2cQqJY/ZPHTUHmG9bs2YMfD2256vccUF3SV9zBNt8IL2OIR/Aq8DPZVSfwLmAL/ypTFa66XAUl/eZ1ciKUl28p99JvF1d7jCOpM7kG6fO1eiB0uWNISjXaeGgQPhe99rPR/QXhITpRR182ZxcFrLfQ8bJs7JtTbV1soMBUMbDB8u8beiIskVuEhMlDeEzSa5g6FD3R/bqqo6Vq9s6FZ4I273rNb6FiAD+DkwHUkAX6m1NjOVfczll0tF0PHjsjNvvKmrrRUVghtvdL8JbAuLBa6+WopNNmyQnEVsrBSl+HK8blERPPSQrEE1NdJUp7WsV599JqWp48bJz+Pi2l9C2y2xWOD734e//EXeBL17y5ErMlJ2D1u3ild3511d0hi5uYG329ClULoNXXKl1DZE5vpN4HyaVQNprduYAeg/cnNz9bowzEgeOSKlpS55CdcGMDoarr1WephC9fTvcEgZ6tGjkpz++GMJcycnN4SRioulgS0xsWM9Fd2a48elMe3TTxue0IgIqS5KTJSKosY5hPJyiQd+5zty/DQYAKXUeq11ix2CNw7hh8B3kfnJjaUWFaC11kGbqxyuDgHkc753r0xBrK2VaMG4cR2bsx5Idu2CP/2poY+ipkbCRocONTix+nqplHrqKTMMq8OUl0tNsMUib47qaqnh/eKLptelp0sp2MSJwbEzhLE77JTXlZMamxpsUwJOhx1Cozv4P631d31uWScIZ4fQVXnhBemlaK5E4OqZcjXPlpXB737neYqioQMUF8OePXKk7NFDcgrB1iwKMfae2suCggUsKFjABYMuYMGVC4JtUsBpzSF4k0NQWmjVGbiu6ayRhq5Pebn7nKZrVoKLiorWZfoNrVBXJ6WjlZXyZA4f3nIKUY8evqsMCCOq6qt4bftrWPOtrNi3AoVi5pCZXDnyymCbFlJ4U2W0Qin1KvCG1nq/60Znr8DZwG3ACmCBXyw0dCkyMyVM5AmtJddg1I29xOGA5cvhzTflqOXKHURHw8UXS5OIGTXZAq01aw+vxZpvZfGWxZTVljE4bTD3T7uf28bdRv+U/m3fSTfDm3fRLCAPWKyUGgSUALFIr8By4B9a6wJ/GWjoWpxxBvzvf57L3ktKJMfQu3cADeuqaA2LF8OyZRKHazxBrK5OxO6OH5cuQW+0SboBxyuPs2jTIqz5Vrae2EpcZBzXjr6WvPF5nDPwHCzKPE+t0aZD0FrXILIVTyilopDy02qtdYmfbTN0QbKypEfi889bCtqBhIlOnZL1K1QrpUKK3bvldJCd3TIX4NIt/+QTedKDMfksRLA5bCzbvQxrvpW3vnwLm8PGmf3O5MnLnmTu6LmkxHagTrsb0q5zpta6HjjiJ1sMYcK8ebJ53bBBEsiJidKbcOSIOIHbbpPGW4MXfPCB5AlaSwxbLFLTu2xZt3QIO4t2Mr9gPs9sfIajFUfpmdCTH03+EfNy5jEq0zTitRcTeDT4nNhY+MEPJP/57LOywa2tlQbbnj1l7dq6Fb71rZYT2QzN2LJFSkc9kZ4uw68djm4RNiqvLeflbS9jzbey+sBqIlQElw6/lLzxeVwy7BKiIloR2DK0iXEIBr/gmgd/6JCooiYnN/xMaymh//Of4Ve/al2mw0BoahCdPAmbNkntcHKyNJO05bQ6idaa1QdWY8238tLWl6isr2RE+ggeuPABbhl3C70TTULKF3TIISilemutj/raGEP4YLPB00/LOtG8mkgpOS0cOya9VL/4RXBs7BIMHSrdfo2Tyc1xZen9fTqorYVFi2SAj8MhlU02mzzu1Klw880+n4d8uPwwCzcuxJpvZVfxLhKjE7lhzA3k5eRxZr8zUaHmLLs4HT0hLAUm+NIQQ3ixfbvMRPDUeNazp0Q6Dh9uGApUVCRKCxERcnKIjw+MvSHLzJkyDam5sJULrSVLf9NN/rXDZoPHH5eTwYABTZ2PwyEaJaWlMtSnkyWwdfY63v7ybaz5Vt7Z/Q4O7eDcgedy3zn3MWfUHBKiEzr5xxhao6OvnHHLBo8UFrYd6XDNczlyRBRXX35ZZC5c44AjI2Uew+WXi5x/t+S00+Css0QVcODApsllh0MmF51+OuTk+NeOzZtFdXHQoJYvrMUitm3cKNd10JbNxzYzv2A+z256lqKqIrKSsrhn6j3cPv52hqUP6/zfYGiTjjqE//rUCkPY0Z6T/KFDMl85IqKpNlt9vSSkd+yAn/+8mzoFi0VqdJOT4cMPG2YeuOacnn22nA5am1TkK5YtE4nd1l5YpRqqndrhEEpqSli8eTHWAivrDq8jyhLFFSOvIG98HjOHzCTCYmQ3AklHHcJrPrUixCkvF82w996T03lCgswSnjrV77k0v1JUJIttdTWkpkopqK/E8wYNkl2+JxwOWd+WLpXQUGpq059HRcnGs7BQmnRbG9cZ9kRFyaJ/2WVSdVRaKrW8Y8YETqZizx7Pw75B5G337GnzrhzawYqvVmAtsPLa9teosdVwes/TeeSiR7hp7E1kxGe0eR8G/2ByCG1w8CA8+KAUVKSnS9y7thbeeEMGzdx1V9ebO1JZKeWga9Y0vT06WsIzl1zS+fzkiBGyVpWWtj674dgxCY2fPOm5azkrC1asgCuu6OY5hZQU2YUEA5dchidcp5ZWKCwpZEHBAuYXzKewtJDU2FTyxueRl5PHhD4TWiSIbTZ5b7iGK7U1a9zQeUwOwQOVlfD3v8v7vHFyND5evsrL4ZFH4P77oVevoJnZLmpqZHhNYWHL3GBdHbz4ogjPXXdd56odIyLg29+Wecx2u2weXfentTiD6GiZoFZW5vm+oqJkcThwQByNIQicfrpUCnh6o588Kdc1osZWw+vbX8daYOWDvR+g0Vw4+EL+Mv0vXDnySuKiWh5Jq6ulH+/dd2WIEkjx0oUXwowZRgPLn5gcggfWrm2o6HNHUpLsgFesgOuvD6RlHWf1alFDGOxmioVLCeGdd2Qj2r+T2l/Dh8O998KCBU2TzFpLrvS22+D99707jSjlfsZ8WGGzyWoYFdVSxTTYzJgB69a13vzmcMgOasYMtNZsOLIBa76V57c8T0lNCQNTBvLb837LbeNvIzs1u9WHqaqSTdiePXJqdIVka2slbPjFF3DPPS3Diwbf0CGHoLV+wteGhCIffNB2iLZnT3EIc+c2/Zy4Btxs2CA77sxMmDQpuCcJh0PCXJ5siIgQx/DRR1JW3lmGDpUT1L59MkXNYhFH4wpH9+vXtgy2Sx01I1xDyydPwsqV4h1ra+UPHj1alExHjQqNxrSRI2WL/t57IrLX2GHV1MChQxRdcCbPlb6L9T/Xs+nYJmIiYrhm1DXkjc9j2qBpXonKvfACfPWV5KAaExMjm5VDh8BqhR//2Ld/nkEwncoecOXuPBEdLaGWurqGz8jJk/Cvf8kbOzJSNny1tfDqq7LzvuUWn/fveEVFhZx42pKLSE2VZLOvUEo+4M0/5CBjfhctkuevtRjxiROyPnrqzeqyHDgAf/sbtspatqoxvHvgNL46lUrkukpyXvuUabcdIvtbM1CWIDsFpWSHkJkJb78tCquAHQfLU4qwTirljYMvUr+/ntysXJ645AmuH3M9aXFpXj9EWZmcYD11rmdlSSvEkSMyKM7gW4xD8EByspzgPSWz6utlwXddU14ucfOyspZqn67+ndpa+O53Q1d2JpCjjhITYc4c6Vju37/lc11aKs/x3LmBsylg1NbCP/5BjSOaxw9dxqZjvUmKqaVHXA06LpI1FeNY9UAtV1bt44ofDQr+QcFikVPL9OnsLviA+V++zDOHl3Go+hjpNencOelO5uXMY2yvjs1F3bVLPiOeBry5cts7dxqH4A+8mZjmTV2bozNy2Eqpa4HfAacBZ2itQ2Iu5rRpsnv1lMQ6fhzOO69hcV+1Sm5zl3ewWOT2L76QkOzw4f6wunUSEyXsUl7u+W8qKZHoQKC46CL595VXJIweGysLQ12dJKN/8YswFcHbtAmKi3mm6Go2H+9FdmpJk0U/K6UKW2Qtrz5TTnqu5pxzgusRKusqeWXbK1gLrKwqXIVFWZg1dBaPjs9j9ojZREd0rgyovt676JjF0vYQJkPH8OaEcNj55emligA685HdAlwN/KcT9+FzXMNeTp2Shak5VVWyW7ngAvnebpfKCE8xeqWk1v/DDwPvECwWKSl9+mlxDu4+fDab/B3nnhs4u5SCWbMknLZ+vSSgo6IkTDR6dBgPA1u9mmOWPnx2sD8DUkrdvh6R8dH0KjrEGy8OZ8qU2ICPR9Zas+bgGqz5Vl7Y+gIVdRUM6zGMv0z/C7eMvYW+yX3bvhMvSUvz7nSqdRjnk4KMNx+17Vprj62HSqn8zhihtd7uvJ/O3I3PSUqS5NXf/w7790v4NDZWdjLHjslCduedDQnSqiqJ07eViE5OliRrMJgyRYbX7NghucHGi211tegKXXtt2z1I/iApSRr+ug2VleSfysaioNUUgVIkRtex/6SDwkL31WH+4GjFUZ7d+CzWAis7inaQEJXA3NFzycvJY2r/qX75rA4dKk6hoqL13F1treTfzDwN/+CNQzjLR9f4BKXUHcAdAAMCEEcYNAj++EdJdi1fLo4gLk40x847r2lDVUSE7F7aUix2OILXZBMdLc10L70klUQOh9yutXwI582TUJkhAPTsSXGZIsrioZ7W9QJFRVFR0Y77rquTkNSaNVIO2quXyFwMGdLqm7PeXs/SXUuxFlhZ8uUS7NrO1P5Tefryp7l21LUkxfi3ASAiQvpf/vUvOSE2L7yw2aRR9OabQ68qN1zwdoSmW5RS87TW8z1d0+ja9wF3/aj3aa3faOv3G9nzJPAkQG5ubkDSn2lpohpw2WWeZ5DExYkDOXnSfYjJRXGxdAQHi9hYuPVWuOoqGWJTWyunlhEj/C+JY2jEOeeQvHgJ9Q4P1QWVlZCVhY6I8l5WZP9+6Zg8dUo6KKOipLB/5UrZWn/3u02Eobad2Mb8/Pks3LSQ45XH6Z3Ym59O+Snzxs9jREZgOwEnT5aT6rPPymctOVn8V1mZbFquvVY2Ywb/0Nno7O+B+d5cqLUOYJrSf3iqDFJKYvT//KeoDLi71lVzHywFgsYkJcHEicG2ohszfDg5k5bzymvV6CTdsrS0rg4cDqr6DSc51n3ZbguKiuCBBxoUSF2kpsqKum0b/OtflP3gDl7cLgniNQfXEGmJZPbw2eTl5DFr6CwiLcFL3Jx/vkwDXbNGpJscjgbtsMzMoJnVLfCmymhTaz8CuohgQ+CYMEFCSStXSlmcS3tHaymhLC4W8Urzxu4+nDghlVtRUVJj/3XexmKh76/mMX7zZ2z+MoZ+qRWomGhZAaurISIC+6QzOVKVwu1zvUyuf/CB/K6bNnOtYFV/O9YTT/Lyw7+j2l7DqMxRPDTzIW4eezM9E0Kn0SMtTSpcL7442JZ0L7x5i/UCLgJONbtdAZ/6wgil1FXAY0AmsEQpVaC1vsgX9x1oLBa4/XbZnL39tmzYlJLP+IABkJcnEwcN4c/u3dKMuGNHw4yHpCRZ5C680LnAJyXxzecv4B+/KmL358X0qC0iOd6OHjSEorj+VNTEcNFFXuZ16uulbb5ZmdtByniGAuZTwB7LKZITork16gzy5j3EpKxJIVfMYQge3jiEt4FErXVB8x8opVb6wgit9evA6764r1AgIkI+8OefL+FcV4w+Kys0VAj8SVWV5FCUks7i7qpQuXGjhPETEmQj4Hrdq6rg+eclpP+d78h7JTE1kp8/1Jt163rzzjtQeFAcyOljJF7utXpFZeXXLd+12HiTnVgp4F12oxVM09n8Tp/P1VUDiLclQN8z/PkUGLog3iSVv+HhZzf61pzwIjIycGWCwaa4WHSSVq1qKIyJiZGms5kzfTdnoStQUQH/939SK998qE98vOQC1qyR/O5558ntMTESI586tWEGTrs3D1FRFEQWYWUrz7GZYlVNf53MrziX2/V4BuOsdLCVQrIp0zG0JFxbfgwB5Phx+MtfJEfSp09DpVJNDbz2mkxe/NnPus8sg7UfVVHzVQm9bXukVjI5WVrUe/QApb4+PS1ZAuec07L4oL3NZ8XVxTy/+Xms+Vby++YTrS1cxWnk6RymM4gImj1ASYlUPxgMzfAmqbxBa+1xGI431xjCE61l/GV1dUt5iVhnZUxhocxLvu224NgYULZsYf0fd5Bckgwpzhrlw4dFxK5XL1Hzi4oiKUnCicXFHeu6tTvsfPDVB1jzrby+43Xq7HXk9M7hsbH3cOMbe+nRf7h7z1JRIXG8yZO9fzCbTby9UlKtFKoiXIZO480J4TQPlUYgyeVWZmIZwp3CQomHN65wbE5Wloj6XXNN2+qxXZp9++Dhh6mLnENEShLEOmuMo6MbpgJt2CCaKM6Tgs3WvofYe2ovCwoWsKBgAQfKDpAWm8a3J36beePnkdMnRx6n/mV46y0ZJuCag+xwyFGurg7uvltOLW1RUSEaK8uXN0yqSUuDM88U7x8ZKS+uUZkLG7xxCCO9uCbcR5cYWmHHDllvPMW7IyNlPdqzR+rLw5a33oLoaLJ7VvHVV71IcTkEkCcoJUV0m0tLqU9IJSLCu0EvVfVVvLb9Naz5VlbsW4FCMXPITP4+8+9cPuJyYiMb5QOUku6tQYOkzO3AgQaJ0IkTpbvSk/d2UVIisr1Hjkg7fkaGOIi1a0XgKy1Nst0REdLRePPNnnWrDV0Cb5LKhY2/V0r9ARGzKwAKtNa7/GOaoStQW+t9zLu9u+EuRWkp5OdDv36cHXeA5XuG4tDNNIqUknDL/v0cTUtl2rTWJRi01qw9vBZrvpXFWxZTVlvG4LTB3D/tfm4bdxv9UzyMs1NKpjHl5krJV12dHM28ORW4ePppqZl2yfaWl8sxz+EQEayyMqlqGjNGjol//CPcd1/nx+wZgkq7k8pa698opXoBOcA1SqkhWutv+d40Q1egZ8+2R1u69J3aEv3r0pSVfb3g908u5ZwBhXxUOJDs1FIsqpHCSlQURcdsxPcVhdfmHK88zqJNi7DmW9l6YitxkXHMGTWHvJw8zh14rldTx75GqY4lKA4dkhbhxkmhjRvFGbhifklJMgFqxAjJjRQVwVNPwe9+F/611WGM1w5BKfUIcLcWjgHLnF+Gbsy4cVJV5GniWVmZhJlbm00dFkQ7O4y1RinFreM2EmWxs2LfIJTSxEXasDks1FbG0GsA/PCehrXa5rCxbPcyrPlW3vryLWwOG5P7TuY/l/2H60ZfR0psgFN027bJv66FvaxMFvyURnZERMjfW1zcMPy4sFDyKO40NoqK4NNPxdGAjOScOjW4M2UNLWjPCaECeFMpdb3WulIpNRP4rdY6BFR5DMEiPh6uvlomng0Y0FIcr7pa1oyf/jTMN449e4rXKy+H5GSiIhzcOn4Ts4bt4fODfTlSkUhcpI0JjnWM/NUcIvrCzqKdzC+YzzMbn+FoxVEy4zO5a/JdzBs/j9E9Rwfvb6mpaVpJVFYm/zZ/AZVqOB66frZ/f1OHoLUkpV98Ub53OZU9eyTncsUV8hXWb46ug9cOQWv9K6XUjcBKpVQtUAnc4zfLDF2Giy4S1YTXXpPPdUKCrAOVleIg7rwTTj892Fb6GaVg9mx4/HEJqzgX1J4Jlcwe8aVcU1xMeZTmGdZhtf6Q1QdWE6EiuHT4peSNz+OSYZcQFRECcrPp6U3jgK1NrdG6ZRLE1ZXo4pNPZOxg8/moKSmSVHr1VXnDzJjhG9sNnaI9IaPpwLcQR9AH+IbWeqe/DDN0HVxr4ZQp8NlnsHevrIdjxkhes3mp6cmTct3ateJIBg0SmY8hQ7p4ifuZZ4qA0fLlkjBJTQWl0PV1rD61EWvyXl5K2k/l0ipGpI/ggQsf4JZxt9A70Z0qfBAZO1ZKw1wDw10vYONBH3V1cjxsnhhqPCDEZpPBG336uI8nRkZKZdKrr8qIvuYDEAwBpz0ho/uAX2utP1FKnQ68qJT6sdb6Qz/Z5hccjob8X1JSF1+AQoz0dKlq9MSnn0oBi9ZSuWixwLp1spHMzZXTRmGhrEW9eolTCfY6UVwsdn/2mURTeveWDe3o0c1CZEpJ+eVpp8GSJRw+sI2FiXuwJu1mV69SEqMSuGHMjeTl5HFmvzNDV1QuMVE8/CuvSOInNVUqlGpqRIPEbpcSVGc/BSAfqowMSTK72LlTrktPb/2xYmKkVG3rVpEKNgSV9oSMLmj0/81KqYuBV4Ep/jDM19TUSNXcO+9IiTXI5uaSSyS3FexFpzuwdSv8+9/Sy9Q40pCQIKH3p5+GxYtl1rRLHTQuTqZonXtucMLM+fmiS2SzyfslKkoc1j/+ITpVzXu86hz1vJ2wH+uwtbyj3sGhHZzb/2zum/BN5oyaQ0J0QusPFkrMni0JoHfflQTy0KEye9XV6Tx+fEPfQXm5eM2f/azpDsv1QWsLrWWYjyHodFjLSGt9xBlGCnkqK+GhhySP1bNnQzVdRQUsWCBCY3ff3b0E2NyhtTxHO3fKpi0rS6qIfPG8aC2RgdTUlmHnqioZURoRIa9VRkZDlKKmRqoZ7Xa44IIWd+tX9u2Dxx6TDW5jkbrYWLHxwAH5+b33wraiLVjzrTy76VmKqorISsrinqn3cPv42xmWPiywhntCaykr3bpVnuz0dMjJadmjYLHA9deL2NLHH8OuXRL337tX3hCRkZJAdjjkRf3pT+XI1Jj2SN2aHVlI0ClxO611ta8M8SeLFsmurnk1XGKifNB37YIXXpB5wt2VI0dkJ7x/v+zEIyJkVxwdLY2vF17YuR360aNStu5uDPbOnRKSTk6W/q5DhxoiD7Gxko98/nnptUry71jfJixZIn9/c8VSF+l9S3j76Au8+LiVzcVribJEccXIK8gbn8eMITOCOnXMLSUl8N//ijNQShZ1mw0WLpRY3dVXt5zC07evOAYXdru8YPv2iXMZMEBCZO6m9wwb1lCJ1Fr3osMh1zQONRmCRoi9Y33PyZNyAmitgVIpOfl+8olo7bSnmTNcOHEC/vxn+dwOHNh04a+tlfXCbnffSOUtZWXuJZ1ra2Wn7ToRREbSYph8dLQ8/tq1gTslVFTA+vUt1Rg0DvaxknysbFevYsuooW/Z6Txy0SPcNPYmMuI70AgWCCor4cEHRc+o+Ytss4nMRVWVKBB68vwRESJZMWpU24+Zmirx2NWr3e8EQIT/JkwwIwRDhLB3CDt3ykbGU/LY1WOzc6fsQrsb//ufhIvdSdHExMhn+eWX4ayzmvYmHT0qidbDh2UnP3Gim0Rro/txV71YWSn/ul4fu9199CA+Xgp4AukQXEoTACUUstE5d6xE7SNGpzCePIaX53Fa6gTuOjNEE8QuPvpIjl7uugMjI+X2lSul3MuXHYQ33CCPu3u3ZONdGujV1fIG6t+/m8jgdg3C3iHU1Hh3ndayW+1ulJXJCSorq/VroqLEYX7xhVTX1NdLGO6jj2TBjI+XTebHH0tI+oc/bKmf1r+/nL4qK5uGYFy6ay4cDvfimW05dV8TGwt1uobN+nUK1Hz28j4ozWB9IRfoPzOSK4kijqKawIaxOoTdLsnhnh5mJlss8kKvXCkzYH1FfDz8/Ocy2nPZsoaZsvHxEoucNq31mJwh4IS9Q3Cp/7aFS4yyu1FUJP+2JVAXHy/5RK3hmWdkMlp2dstFurgY/vpXkbRprEoQESGFKwsWSC7H9XuuHi67XTaNKSnuNY+qqiRU7W+01mw4soGn8628MvB5aighRQ/kPH7LeH0bqWQ3ub6iQqIiIU1FhVQCpaV5vi4lRXbyviY2VgZJz5gheQxXzbG7vIMhqIT9KzJ6tMSga2tbL2SoqZEFb6Q3Qt9hRvMdemtoLYv6oUOSb3HnDEAW84MHYenSlkn688+XXOTKlQ19W1FRcjrZvl3CyI1L211UVcma4s8y9aKqIp7b9BzWAiubjm0iJiKG6f2uoe6LPM7InEZUZMs/9uRJORGNHes/u3yC60Vu3FjmDteL7C8iIzsmtmcIGGHvEGJj4corpUpl4MCWmxKbTRa52293H/sOd1wjLz2J04EsyqNHizOIjPQcvundW/KIc+c2jQZERIiTGDtWHMa+fXI/vXuLs46Pb1qSqrWEtIqL4Qc/8H1ZsN1hZ/me5VgLrLyx4w3qHfXkZuXyxCVPcP2Y60mNTWNJL8mfxMeLw7JYZANx/Lj8bXff3b7qyqCQlCRPslNnqVVOneoCxx2DPwkJh6CUehCYDdQBe4B5WusSX93/RRdJ7HrJEr4eSqK1nF7tdgllBrrGPVSIjZWS0iVLWs8lVlbKYpyTI6GitqaeRUbK81ta2jI8bLE0SPVXVsrzn5AgDmfxYul9cm1ktZZ17Gc/k45lX7G7eDfz80VU7lD5IdLj0rlz0p3My5nH2F5Nt/uXXio9We++K7OhQZzD5ZfDeee1HYUJCZSSP+TJJ8U5uDsl1NXJv2efHVjbDCFFSDgE4D3gXq21TSn1N+Be4Be+unOLBebMkSqZVatkyhfIWNlzzgn+BECt5aRisfj3xN4al1wiqsT790v4xnWKcjnNsjL40Y/EecTEtD3/wPW7nv4WpZo6luRk+Pa35VTx1VfyfKSnSzewLzqUK+sqeWXbK1gLrKwqXIVFWZg1dBaPznqU2SNmEx3hfpuvlIQSR46UNbO+Xpxjl5M8OfNM0QjJz5fegsbHmooKOfLceqvnxLMh7FHamwByAFFKXQXM0Vrf1Na1ubm5et26dQGwyj9UV0uFzzvvSC8ASDhl5kwp8w6kVENFhUjXfPxxQ07B4ZCS0xtuaEjofvKJSEx4msJYXi7O4K9/De7CqbVmzcE1WPOtvLj1RcrryhnaYyh54/O4ddyt9E3uGzzjgkF9vUhOv/uu/B/kxc7IkB2TuwSOISxRSq3XWue2uD0EHcJbwIta60VtXduVHUJJichp7N8vsenERFmAi4pkcb74YmkQDfTns7xcduh2uyR+BwxoakNVlagUxMW5L7fUWqqRvvENSSIHg6MVR3l247NYC6zsKNpBfFQ8c0fPJW98HmcPODt0ReUCRU2NaJTU1srRbPBg/3puh0PkAFatkqaVuDiRxp0woaEvwRBQgu4QlFLvA+50fu/TWr/hvOY+IBe4WrdimFLqDuAOgAEDBkwsLCx0d1lIo7XsnvfuldN7c+x2kdr4xjckTh1qbN0KDz8sUQdXohUkJ3D0qITmvv3twIa/6u31LN21FGuBlSVfLsGu7UzpP4W88XnMHT2XpJhQbxYIUyor4YknJCYZFycJo/p62XkkJMBdd4nEhSGgBN0htIVS6jbgO8B0rXWVN7/TVU8IX30Fv/99SwWBxrjkGx54IDTj1fv2SYfzpk1in8Mhm81LL5Veo0CVmG87sY35+fNZuGkhxyuP0zuxN7eOvZV5OfMYmdEN64hDCYdDjsHbt0tnYvM3e2mpOIzf/z74ibxuRmsOISSSykqpWUgS+TxvnUFXZv16WTA9RS4SEyWcdOCA53h9sMjOlkRzcbGEv1z9BIE4FZTVlvHilhexFlhZc3ANkZZIZg+fTV5OHrOGzgo9Ubnuyq5dcjLIznb/Zk9JkZ3Pu+/6tjva0GFC5ZPzLyAGeM8Z312jtf5OcE3yH6Wl3tWuWyySeA5levRw31nsa7TWrCpchbXAystbX6baVs2ozFE8NPMhbh57Mz0TTHVMyPHRRxIm8rTz6dVLmlauuy489edtNti2Teqpq6rk750yxf2JKQQICYegtR4abBsCSXp627pJWksuoa2a/3DnYNlBnil4hvkF89lzag/JMcncOu5W8nLymJQ1ySSIQ5kjR9rWKYqMlNBSRUX4OYRDh+DRR6WEMDZWjtFbtoim08SJ8M1vhtzfHBIOobsxaZLE3z0pCZSVScLZXdI53Km11fLmzjexFlhZvmc5Du1gWvY0fnveb7lm1DXER5nKlC5BXJzoe3jCJakRbjIBLlEvrVvGfLWWuLHNJkn1EEoSGocQBPr2lU7d9etblnWCNECdPCnjebvTBnjj0Y1Y860s2ryI4upi+iX3475z7uP28bczOG1wsM0ztJcpU6RpxVNM8dQpKXsNN2XJDz6QEJG7QSxKiZMoKBAxweHDA25eaxiHECTy8iQ/sGWL1PMnJzf0Idjt4gy6w8zx4upiFm9ejLXAyoYjG4iOiOaqkVeRl5PH9EHTibAEoXXb4BsmTJBRhKWl7hd8u10qEm6/Pbx2PvX14hAay/02Ryk5Qa1YYRyCQfpx7r5b8k3Ll0vfQWSklGyed577YTXhgt1h58OvPsRaYOX17a9Ta68lp3cOj138GDeefiM94gKQpTb4n/h4KUV74AHJEfTq1SB0deqUfM2eLSJZ4URFhTiFtipHkpKkjDCEMA4hiERGilRFyMsn+4ivTn3FgoIFLNi4gP2l+0mLTeOOiXcwb/w8cvqE2aJgEIYOlT6Dd9+VaiKHQxzC4MFyMsjJCa/TATQkytuSG7fZQm5mr3EIBr9SVV/Fa9tfw5pvZcW+FSgUM4fM5MEZD3L5iMuJjYxt+04MXZs+fWTxv+462T1HR8tCGG6OwEVioji8oiLPcrglJaJRE0IYh2DwOVpr1h5eizXfyuItiymrLWNw2mDun3Y/t427jf4pbhJthvAnLi7kyiz9glIiIfzoo5I7cVdFVFUlJ4nJkwNvnweMQzD4jOOVx1m0aRHWfCtbT2wlLjKOOaPmkJeTx7kDz8WiQqe8zmDwKxMmyMjQ5ctFUtylBOmqHKmuluHjIVZdZRyCoVPYHDaW7V6GNd/KW1++hc1hY3Lfyfznsv9w3ejrSIkNrTe8wRAQlIKbbpIB4m+/LVUjLtGv00+HK66AIUOCbWULjEMwdIidRTuZXzCfhRsXcqTiCJnxmdw1+S7mjZ/H6J6jg22ewRB8LBaZQDdlChw71iA3Hgitlw5iHILBa8pry3l528tY862sPrCaCBXBJcMuIS8nj0uHXUpURJh1mxoMvsBi6TJqrsYhGDyitWb1gdVY8628tPUlKusrGZE+gr9d+DduGXsLfZK6xhvdYDC0jXEIBrccLj/Mwo0LseZb2VW8i8ToRK4fcz15OXmc1e8sIypnMIQhxiEYvqbOXsfbX76NNd/KO7vfwaEdnDPgHH55zi+ZM2oOidHdXHrVYAhzjEMwsOX4Fqz5Vp7d9CxFVUVkJWVxz9R7uH387QxLN+MNDYbugnEI3ZSSmhJe2PIC1nwraw+vJcoSxRUjryBvfB4zhswwU8cMhm6I+dR3Ixzawcp9K7HmW3l1+6vU2Go4vefpPHLRI9w09iYy4jOCbaJX2O2wcyd8+KHMIImNhTPPlKbP1NRgW2cwdF2MQ+gGFJYU8sxGmTq2r2QfKTEp5I3PIy8njwl9JnSpBHFlJTz+OGzdKmKaSUmirvzCC/DKK/D978O4ccG20mDomhiHEKbU2Gr4347/Yc238v7e99FoLhx8IX++4M9cOfJK4qK6nqaM1vDvf8OOHS3nticni7N45BH4zW+kQdRgMLQP4xDCCK01+UfzseZbeW7zc5TUlDAwZSC/Pe+33Db+NrJTs4NtYqf46ivYvFmGTbk71CQkiJjmW2+JTIzBYGgfxiGEASerTvLc5uew5lvZeGwjMRExXDPqGvLG5zFt0LSwEZX75BNRTvYU4crMhPz81od0GTpIeTls2iRDbeLjYfRozxPBDF0S4xC6KHaHnff2voc138obO9+gzl5HblYuT1zyBNePuZ60OA867B2kogL27ZOkbkYGZGUFVtL+2LG21ZMtFvkqLzcOwSfY7fD667BsmQx0iYiQ20AUPefNa1DyNHR5QsIhKKXuB64AHMBx4Hat9eHgWhWa7C7ezfz8+Tyz8RkOlR8iPS6d7+V+j3k58xjbyz+j1yoq4NVX4eOPJY7v+hoyBG64IXCijYmJMpnQEy7b2ppeaPACrWHRIpkPPGCA6Pe7cDhg40Z46CH4xS+6x5yDbkBIOATgQa31rwGUUj8EfgN8J7gmhQ6VdZW8su0VrAVWVhWuwqIszBo6i0dnPcrsEbOJjvDf6ldRAX/7m5R3ZmU1rAlay479T3+Cn/0MTjvNbyZ8zZlnwuefe76mtBT69pXQkaGT7NsnQ+Czs1sOebFYoH9/Sex8+ilMnx4MCw0+JiQcgta6rNG3CYAOli2hgtaaNQfXYM238uLWFymvK2doj6H8+YI/c+u4W+mb3Dcgdrz1Fhw8KIncxigF6emyE3/8cXj4Yf/vyseMkVBVUZH82xy7HU6ehBtvDN/pjAFl5Up5Ud1N/HKRmQlLl8K0aZ6vM3QJQsIhACil/gTcCpQC0zxcdwdwB8CAAQMCY1wAOVpxlGc3Pou1wMqOoh3ER8Uzd/Rc8sbncfaAswPaM1BVJRvErKzWr0lKkkV40ybIzfWvPVFR8KMfyYll/37JacbEyGnl5EkoK4NLL4VJk/xrR7dh5862O/0SE+XFqKqS/xu6NAFzCEqp94Hebn50n9b6Da31fcB9Sql7ge8Dv3V3P1rrJ4EnAXJzc8PiJFFvr2fprqVYC6ws+XIJdm1nSv8pPDX7KeaOnktSTHCSdocPSx4xqo0xBzExsG2b/x0CQL9+8Pvfi6N6/32oq5Nw9ogRMq987FhzOvAZSom39YQraWOe9LAgYA5Ba32hl5c+DyyhFYcQTmw/sR1rvpWFmxZyvPI4vRJ68ZOzfsK8nHmMzBgZbPOw2737nFssDYUngaBHD7jmGplCWFkpeY2EhMA9frdh3Djxup6e3LIyOULGxwfOLoPfCImQkVJqmNZ6l/Pby4EdwbTHn5TVlvHilhexFlhZc3ANkZZILht+GXnj85g1dFZITR3r2VM2fw6H5/BwdbXkHQNNZKQpLfUr554r5ab19e6Pia5Y3Zw55oQQJoSEQwD+qpQagZSdFhJmFUZaa1YVrsJaYOXlrS9TbatmVOYo/j7j79w89mZ6JYZmg09ampSab9rUeh6hrk4W5kCEiwwBJitLFvsXX5QRkI1PAfX1Um2QkyOqgoawICQcgtb6mmDb4A8Olh3kmQIRldtzag9J0UncMvYW8nLyOKPvGV1CVO6aa0RI7uRJqSpqTG2trAm33GJ6k8KWSy8VoahXX4UTJxpuj4qSn115ZdP+BEOXRum2kkYhTG5url63bl2wzWhCra2WN3e+ibXAyvI9y3FoB9OypzFv/DyuGXUN8VFdL9Z64AA88QQcOSKNqhZLQ7J57lwpQe8Cvs3QGWw22L1bcgYxMTBsmMkbdGGUUuu11i3O9cYh+IiNRzdizbeyaPMiiquL6Zfcj3nj53H7+NsZnDY42OZ1GocDdu2C7dslTNSvH4wfb9YEg6Er0ppDMGe9TlBcXczizYuxFljZcGQD0RHRXDXyKvJy8pg+aDoRlohgm+gzLBYp7RwxItiWGAwGf2EcQjtxaAcf7P0Aa4GV17e/Tq29lvG9x/PYxY9x4+k30iOuR7BNNBgMhg5hHIKXfHXqKxYULGDBxgXsL91PWmwad0y8g3nj55HTJyfY5hkMBkOnMQ7BA9X11by2/TWsBVY+/OpDFIqZQ2by4IwHuXzE5cRGxgbbRIPBYPAZxiE0Q2vNusPrsOZbWbxlMaW1pQxKHcT90+7n1nG3MiAl/PSTDAaDAYxD+JoTlSdYtGkR1gIrW45vIS4yjjmj5pCXk8e5A88Nm6ljBoPB0Brd2iHYHDaW7V7G/IL5vLnzTWwOG5P7TuY/l/2H60ZfR0qs0UUwGAzdh27pEHYW7WR+wXwWblzIkYojZMZnctfku5g3fh6je44OtnkGg8EQFLqlQ7julevYcnwLlwy7hLycPC4ddmlIicoZDAZDMOiWDuGpy5+ib1Jf+iT1CbYpBoPBEDJ0S4eQm2WkOQ0Gg6E5pnTGYDAYDIBxCAaDwWBw0qXVTpVSJ5CBOi4ygKIgmdMaoWgTGLvaSyjaFYo2gbGrvQTDroFa68zmN3Zph9AcpdQ6d5KuwSQUbQJjV3sJRbtC0SYwdrWXULLLhIwMBoPBABiHYDAYDAYn4eYQngy2AW4IRZvA2NVeQtGuULQJjF3tJWTsCqscgsFgMBg6TridEAwGg8HQQYxDMBgMBgMQpg5BKfVTpZRWSmUE2xYApdT9SqlNSqkCpdRypVRWsG0CUEo9qJTa4bTtdaVUarBtAlBKXauU2qqUciilglqOp5SapZTaqZTarZS6J5i2uFBKWZVSx5VSW4JtS2OUUv2VUiuUUtudr99dIWBTrFLqC6XURqdNvw+2TY1RSkUopfKVUm8H2xYIQ4eglOoPzAD2B9uWRjyotR6rtR4PvA38Jsj2uHgPGKO1Hgt8CdwbZHtcbAGuBlYF0wilVATwOHAxMAq4QSk1Kpg2OVkAzAq2EW6wAT/RWp8GnAncGQLPVy1wgdZ6HDAemKWUOjO4JjXhLmB7sI1wEXYOAfgH8HMgZLLlWuuyRt8mECK2aa2Xa61tzm/XAP2CaY8LrfV2rfXOYNsBnAHs1lrv1VrXAS8AVwTZJrTWq4DiYNvRHK31Ea31Buf/y5GFrm+QbdJa6wrnt1HOr5D4/Cml+gGXAk8F2xYXYeUQlFKXA4e01huDbUtzlFJ/UkodAG4idE4IjckD3gm2ESFGX+BAo+8PEuQFrquglMoGcoDPg2yKKyxTABwH3tNaB90mJ48gm1dHkO34mi4nf62Ueh/o7eZH9wG/BGYG1iLBk11a6ze01vcB9yml7gW+D/w2FOxyXnMfctx/LhA2eWtXCKDc3BYSu8tQRimVCLwK/KjZ6TgoaK3twHhnjux1pdQYrXVQ8y9KqcuA41rr9Uqp84NpS2O6nEPQWl/o7nal1OnAIGCjUgok/LFBKXWG1vposOxyw/PAEgLkENqySyl1G3AZMF0HsCmlHc9XMDkI9G/0fT/gcJBs6RIopaIQZ/Cc1vq1YNvTGK11iVJqJZJ/CXZCfipwuVLqEiAWSFZKLdJa3xxMo8ImZKS13qy17qm1ztZaZyMf5gmBcAZtoZQa1ujby4EdwbKlMUqpWcAvgMu11lXBticEWQsMU0oNUkpFA9cDbwbZppBFyU7saWC71vrhYNsDoJTKdFXPKaXigAsJgc+f1vperXU/51p1PfBhsJ0BhJFDCHH+qpTaopTahIS0gl6O5+RfQBLwnrMk9t/BNghAKXWVUuogcBawRCn1bjDscCbcvw+8iyRIX9Jabw2GLY1RSi0GPgNGKKUOKqW+EWybnEwFbgEucL6fCpw74GDSB1jh/OytRXIIIVHiGYoY6QqDwWAwAOaEYDAYDAYnxiEYDAaDATAOwWAwGAxOjEMwGAwGA2AcgsFgMBicGIdgMBgMBsA4BEOYoJTKVkpVOzVrXLe1kK5WSsU56+PrOiuP7ryvj5yqqCilfuiUfm6XBIhSKlUp9b3O2OLFY7SQzFZKRSulVimlupxigcE/GIdgCCf2OCXGW5Wu1lpXO6/xhQRFHvCaUysH4HvAJVrrm9p5P6nO320XSvD2M7yAZpLZTgXXD4Dr2vvYhvDEOARDl8E5fGWG8/9/VEr908PlgZCuvglwCQT+GxgMvKmUulspdbNzMEuBUuo/jU4R/1NKrXcOa7nDeT9/BYY4r33QedppvJP/qVLqd87/ZztPIU8AG4D+rT1WYzxIZv/P+XcYDMYhGLoUv0UUY29CpJXv9nCtX6WrndpGg7XW+wC01t9BTh3TgGXIrnuq8zRip2HRzdNaTwRygR8qpdKBe3CebrTWP/Pi4UcAC7XWOUC8h8fyhi3ApHZcbwhjTOzQ0GXQWq9yCqj9GDjfFapRSt2PiKo1xt/S1RlASSs/mw5MBNY6lXfjEC1+ECdwlfP//YFhQHsFGAu11mu8eKw20VrbnfmUJOdQG0M3xjgEQ5fBKXHeByhyLV5Kqd64fx+3W7paKXUn8C3nt5cAVzX+Xmvd+PerEdlit3cFPKO1bjKS1Kl7fyFwlta6yinF7O4+bDQ9vTe/prKtx2onMUBNJ37fECaYkJGhS6CU6oMM8LkCqFRKXeT8UQ5Q4OZX2i1drbV+3Bm2Ga+1Ptz8+2bXngIilFLuFvQPgDlKqZ5O23sopQYCKcAppzMYicwdBihHVGddHAN6KqXSlVIxyLyK1mjtsbzCGbI6obWu9/Z3DOGLcQiGkEcpFQ+8hgxw3w7cD/zO+ePxuHEIAZKuXg6c7eaxtwG/ApY7ZZffQ042y4BI5233I3Os0VqfBFY7JdIfdC7Of0DGT76NB/1+D4/VBA+S2dOApR354w3hh5G/NnRplFJPI2GdAcDbWusxXv7ePiBXa13UicfOAX6stb6lo/cRbJRSrwH3aq13BtsWQ/AxJwRDl0Zr/Q2ttQOprklp3JjmDldjGhBFJ4eba63zkeErLco8uwLOUNr/jDMwuDAnBIPBYDAA5oRgMBgMBifGIRgMBoMBMA7BYDAYDE6MQzAYDAYDYByCwWAwGJwYh2AwGAwGwDgEg8FgMDgxDsFgMBgMAPw/B/f7IPKRLS0AAAAASUVORK5CYII=", |
| 1271 | "text/plain": [ |
| 1272 | "<Figure size 432x288 with 1 Axes>" |
| 1273 | ] |
| 1274 | }, |
| 1275 | "metadata": { |
| 1276 | "needs_background": "light" |
| 1277 | }, |
| 1278 | "output_type": "display_data" |
| 1279 | } |
| 1280 | ], |
| 1281 | "source": [ |
| 1282 | "plot_dataset(train_x,train_labels,net.W.detach().numpy(),net.b.detach().numpy())" |
| 1283 | ] |
| 1284 | }, |
| 1285 | { |
| 1286 | "cell_type": "markdown", |
| 1287 | "metadata": { |
| 1288 | "id": "1W4TZfXOmIlS" |
| 1289 | }, |
| 1290 | "source": [ |
| 1291 | "Not let's compute the accuracy on the validation dataset:" |
| 1292 | ] |
| 1293 | }, |
| 1294 | { |
| 1295 | "cell_type": "code", |
| 1296 | "execution_count": 31, |
| 1297 | "metadata": { |
| 1298 | "colab": { |
| 1299 | "base_uri": "https://localhost:8080/" |
| 1300 | }, |
| 1301 | "id": "HUjdeIefsIsg", |
| 1302 | "outputId": "a1a363d4-a307-4769-9ccf-fe8a857b62af" |
| 1303 | }, |
| 1304 | "outputs": [ |
| 1305 | { |
| 1306 | "data": { |
| 1307 | "text/plain": [ |
| 1308 | "tensor(0.7333)" |
| 1309 | ] |
| 1310 | }, |
| 1311 | "execution_count": 31, |
| 1312 | "metadata": {}, |
| 1313 | "output_type": "execute_result" |
| 1314 | } |
| 1315 | ], |
| 1316 | "source": [ |
| 1317 | "pred = torch.sigmoid(net.forward(torch.tensor(valid_x)))\n", |
| 1318 | "torch.mean(((pred.view(-1)>0.5)==(torch.tensor(valid_labels)>0.5)).type(torch.float32))" |
| 1319 | ] |
| 1320 | }, |
| 1321 | { |
| 1322 | "cell_type": "markdown", |
| 1323 | "metadata": {}, |
| 1324 | "source": [ |
| 1325 | "Let's explain what is going on here:\n", |
| 1326 | "* `pred` is the vector of predicted probabilities for the whole validation dataset. We compute it by running original validation data `valid_x` through our network, and applying `sigmoid` to get probabilities.\n", |
| 1327 | "* `pred.view(-1)` creates a flattened view of the original tensor. `view` is similar to `reshape` function in numpy.\n", |
| 1328 | "* `pred.view(-1)>0.5` returns a boolean tensor or truth value showing the predicted class (False = class 0, True = class 1)\n", |
| 1329 | "* Similarly, `torch.tensor(valid_labels)>0.5)` creates the boolean tensor of truth values for validation labels\n", |
| 1330 | "* We compare those two tensors element-wise, and get another boolean tensor, where `True` corresponds to correct prediction, and `False` - to incorrect.\n", |
| 1331 | "* We convert that tensor to floating point, and take it's mean value using `torch.mean` - that is the desired accuracy " |
| 1332 | ] |
| 1333 | }, |
| 1334 | { |
| 1335 | "cell_type": "markdown", |
| 1336 | "metadata": { |
| 1337 | "id": "_95qF9lY2kHp" |
| 1338 | }, |
| 1339 | "source": [ |
| 1340 | "## Neural Networks and Optimizers\n", |
| 1341 | "\n", |
| 1342 | "In PyTorch, a special module `torch.nn.Module` is defined to represent a neural network. There are two methods to define your own neural network:\n", |
| 1343 | "* **Sequential**, where you just specify a list of layers that comprise your network\n", |
| 1344 | "* As a **class** inherited from `torch.nn.Module`\n", |
| 1345 | "\n", |
| 1346 | "First method allows you to specify standard networks with sequential composition of layers, while the second one is more flexible, and gives an opportunity to express networks of arbitrary complex architectures. \n", |
| 1347 | "\n", |
| 1348 | "Inside modules, you can use standard **layers**, such as:\n", |
| 1349 | "* `Linear` - dense linear layer, equivalent to one-layered perceptron. It has the same architecture as we have defined above for our network\n", |
| 1350 | "* `Softmax`, `Sigmoid`, `ReLU` - layers that correspond to activation functions \n", |
| 1351 | "* There are also other layers for special network types - convolution, recurrent, etc. We will revisit many of them later in the course.\n", |
| 1352 | "\n", |
| 1353 | "> Most of the activation function and loss functions in PyTorch are available in two form: as a **function** (inside `torch.nn.functional` namespace) and **as a layer** (inside `torch.nn` namespace). For activation functions, it is often easier to use functional elements from `torch.nn.functional`, without creating separate layer object.\n", |
| 1354 | "\n", |
| 1355 | "If we want to train one-layer perceptron, we can just use one built-in `Linear` layer:" |
| 1356 | ] |
| 1357 | }, |
| 1358 | { |
| 1359 | "cell_type": "code", |
| 1360 | "execution_count": 32, |
| 1361 | "metadata": { |
| 1362 | "colab": { |
| 1363 | "base_uri": "https://localhost:8080/" |
| 1364 | }, |
| 1365 | "id": "D77pXPR6oFRs", |
| 1366 | "outputId": "efa49e5c-72d4-4781-89d4-4ab6597d2b0e" |
| 1367 | }, |
| 1368 | "outputs": [ |
| 1369 | { |
| 1370 | "name": "stdout", |
| 1371 | "output_type": "stream", |
| 1372 | "text": [ |
| 1373 | "[Parameter containing:\n", |
| 1374 | "tensor([[-0.0422, 0.1821]], requires_grad=True), Parameter containing:\n", |
| 1375 | "tensor([0.6582], requires_grad=True)]\n" |
| 1376 | ] |
| 1377 | } |
| 1378 | ], |
| 1379 | "source": [ |
| 1380 | "net = torch.nn.Linear(2,1) # 2 inputs, 1 output\n", |
| 1381 | "\n", |
| 1382 | "print(list(net.parameters()))" |
| 1383 | ] |
| 1384 | }, |
| 1385 | { |
| 1386 | "cell_type": "markdown", |
| 1387 | "metadata": { |
| 1388 | "id": "0tbe0Et_oiNo" |
| 1389 | }, |
| 1390 | "source": [ |
| 1391 | "As you can see, `parameters()` method returns all the parameters that need to be adjusted during training. They correspond to weight matrix $W$ and bias $b$. You may note that they have `requires_grad` set to `True`, because we need to compute gradients with respect to parameters.\n", |
| 1392 | "\n", |
| 1393 | "PyTorch also contains built-in **optimizers**, which implement optimization methods such as **gradient descent**. Here is how we can define a **stochastic gradient descent optimizer**:" |
| 1394 | ] |
| 1395 | }, |
| 1396 | { |
| 1397 | "cell_type": "code", |
| 1398 | "execution_count": 33, |
| 1399 | "metadata": { |
| 1400 | "id": "B4AxyrFMozh0" |
| 1401 | }, |
| 1402 | "outputs": [], |
| 1403 | "source": [ |
| 1404 | "optim = torch.optim.SGD(net.parameters(),lr=0.05)" |
| 1405 | ] |
| 1406 | }, |
| 1407 | { |
| 1408 | "cell_type": "markdown", |
| 1409 | "metadata": { |
| 1410 | "id": "6eB8v58eo9pp" |
| 1411 | }, |
| 1412 | "source": [ |
| 1413 | "Using the optimizer, our training loop will look like this:" |
| 1414 | ] |
| 1415 | }, |
| 1416 | { |
| 1417 | "cell_type": "code", |
| 1418 | "execution_count": 34, |
| 1419 | "metadata": { |
| 1420 | "colab": { |
| 1421 | "base_uri": "https://localhost:8080/" |
| 1422 | }, |
| 1423 | "id": "ups7nlV22ofp", |
| 1424 | "outputId": "503d8ae9-35f3-4ecb-e2ff-4da2ec2914eb" |
| 1425 | }, |
| 1426 | "outputs": [ |
| 1427 | { |
| 1428 | "name": "stdout", |
| 1429 | "output_type": "stream", |
| 1430 | "text": [ |
| 1431 | "Epoch 0: last batch loss = 0.7596041560173035, val acc = 0.5333333611488342\n", |
| 1432 | "Epoch 1: last batch loss = 0.6602361798286438, val acc = 0.6000000238418579\n", |
| 1433 | "Epoch 2: last batch loss = 0.5847358107566833, val acc = 0.6666666865348816\n", |
| 1434 | "Epoch 3: last batch loss = 0.5263020992279053, val acc = 0.7333333492279053\n", |
| 1435 | "Epoch 4: last batch loss = 0.48015740513801575, val acc = 0.800000011920929\n", |
| 1436 | "Epoch 5: last batch loss = 0.4430023431777954, val acc = 0.8666666746139526\n", |
| 1437 | "Epoch 6: last batch loss = 0.41254672408103943, val acc = 0.8666666746139526\n", |
| 1438 | "Epoch 7: last batch loss = 0.3871781527996063, val acc = 0.800000011920929\n", |
| 1439 | "Epoch 8: last batch loss = 0.3657420873641968, val acc = 0.800000011920929\n", |
| 1440 | "Epoch 9: last batch loss = 0.34739670157432556, val acc = 0.800000011920929\n" |
| 1441 | ] |
| 1442 | } |
| 1443 | ], |
| 1444 | "source": [ |
| 1445 | "val_x = torch.tensor(valid_x)\n", |
| 1446 | "val_lab = torch.tensor(valid_labels)\n", |
| 1447 | "\n", |
| 1448 | "for ep in range(10):\n", |
| 1449 | " for (x,y) in dataloader:\n", |
| 1450 | " z = net(x).flatten()\n", |
| 1451 | " loss = torch.nn.functional.binary_cross_entropy_with_logits(z,y)\n", |
| 1452 | " optim.zero_grad()\n", |
| 1453 | " loss.backward()\n", |
| 1454 | " optim.step()\n", |
| 1455 | " acc = ((torch.sigmoid(net(val_x).flatten())>0.5).float()==val_lab).float().mean()\n", |
| 1456 | " print(f\"Epoch {ep}: last batch loss = {loss}, val acc = {acc}\")" |
| 1457 | ] |
| 1458 | }, |
| 1459 | { |
| 1460 | "cell_type": "markdown", |
| 1461 | "metadata": { |
| 1462 | "id": "vRLXEQ4Qrcvx" |
| 1463 | }, |
| 1464 | "source": [ |
| 1465 | "> You may notice that to apply our network to input data we can use `net(x)` instead of `net.forward(x)`, because `nn.Module` implements Python `__call__()` function\n", |
| 1466 | "\n", |
| 1467 | "Taking this into account, we can define generic `train` function:" |
| 1468 | ] |
| 1469 | }, |
| 1470 | { |
| 1471 | "cell_type": "code", |
| 1472 | "execution_count": 35, |
| 1473 | "metadata": { |
| 1474 | "colab": { |
| 1475 | "base_uri": "https://localhost:8080/" |
| 1476 | }, |
| 1477 | "id": "5c6WsBhlrlIs", |
| 1478 | "outputId": "54de8404-4170-4a15-abba-039d06d5e946" |
| 1479 | }, |
| 1480 | "outputs": [ |
| 1481 | { |
| 1482 | "name": "stdout", |
| 1483 | "output_type": "stream", |
| 1484 | "text": [ |
| 1485 | "Epoch 0: last batch loss = 0.48486900329589844, val acc = 0.7333333492279053\n", |
| 1486 | "Epoch 1: last batch loss = 0.41338109970092773, val acc = 0.800000011920929\n", |
| 1487 | "Epoch 2: last batch loss = 0.35756850242614746, val acc = 0.800000011920929\n", |
| 1488 | "Epoch 3: last batch loss = 0.31495171785354614, val acc = 0.800000011920929\n", |
| 1489 | "Epoch 4: last batch loss = 0.2824164032936096, val acc = 0.800000011920929\n", |
| 1490 | "Epoch 5: last batch loss = 0.2572754919528961, val acc = 0.800000011920929\n", |
| 1491 | "Epoch 6: last batch loss = 0.23751722276210785, val acc = 0.800000011920929\n", |
| 1492 | "Epoch 7: last batch loss = 0.2217157930135727, val acc = 0.800000011920929\n", |
| 1493 | "Epoch 8: last batch loss = 0.2088666558265686, val acc = 0.800000011920929\n", |
| 1494 | "Epoch 9: last batch loss = 0.19824868440628052, val acc = 0.800000011920929\n" |
| 1495 | ] |
| 1496 | } |
| 1497 | ], |
| 1498 | "source": [ |
| 1499 | "def train(net, dataloader, val_x, val_lab, epochs=10, lr=0.05):\n", |
| 1500 | " optim = torch.optim.Adam(net.parameters(),lr=lr)\n", |
| 1501 | " for ep in range(epochs):\n", |
| 1502 | " for (x,y) in dataloader:\n", |
| 1503 | " z = net(x).flatten()\n", |
| 1504 | " loss = torch.nn.functional.binary_cross_entropy_with_logits(z,y)\n", |
| 1505 | " optim.zero_grad()\n", |
| 1506 | " loss.backward()\n", |
| 1507 | " optim.step()\n", |
| 1508 | " acc = ((torch.sigmoid(net(val_x).flatten())>0.5).float()==val_lab).float().mean()\n", |
| 1509 | " print(f\"Epoch {ep}: last batch loss = {loss}, val acc = {acc}\")\n", |
| 1510 | "\n", |
| 1511 | "net = torch.nn.Linear(2,1)\n", |
| 1512 | "\n", |
| 1513 | "train(net,dataloader,val_x,val_lab,lr=0.03)" |
| 1514 | ] |
| 1515 | }, |
| 1516 | { |
| 1517 | "cell_type": "markdown", |
| 1518 | "metadata": { |
| 1519 | "id": "KzuIDqJ8sFYm" |
| 1520 | }, |
| 1521 | "source": [ |
| 1522 | "## Defining Network as a Sequence of Layers\n", |
| 1523 | "\n", |
| 1524 | "Now let's train multi-layered perceptron. It can be defined just by specifying a sequence of layers. The resulting object will automatically inherit from `Module`, e.g. it will also have `parameters` method that will return all parameters of the whole network." |
| 1525 | ] |
| 1526 | }, |
| 1527 | { |
| 1528 | "cell_type": "code", |
| 1529 | "execution_count": 36, |
| 1530 | "metadata": { |
| 1531 | "colab": { |
| 1532 | "base_uri": "https://localhost:8080/" |
| 1533 | }, |
| 1534 | "id": "tBtytmEAsq-O", |
| 1535 | "outputId": "06ad840b-c2b7-409e-e01e-a9170548151d" |
| 1536 | }, |
| 1537 | "outputs": [ |
| 1538 | { |
| 1539 | "name": "stdout", |
| 1540 | "output_type": "stream", |
| 1541 | "text": [ |
| 1542 | "Sequential(\n", |
| 1543 | " (0): Linear(in_features=2, out_features=5, bias=True)\n", |
| 1544 | " (1): Sigmoid()\n", |
| 1545 | " (2): Linear(in_features=5, out_features=1, bias=True)\n", |
| 1546 | ")\n" |
| 1547 | ] |
| 1548 | } |
| 1549 | ], |
| 1550 | "source": [ |
| 1551 | "net = torch.nn.Sequential(torch.nn.Linear(2,5),torch.nn.Sigmoid(),torch.nn.Linear(5,1))\n", |
| 1552 | "print(net)" |
| 1553 | ] |
| 1554 | }, |
| 1555 | { |
| 1556 | "cell_type": "markdown", |
| 1557 | "metadata": { |
| 1558 | "id": "5r5RbLB1s6YB" |
| 1559 | }, |
| 1560 | "source": [ |
| 1561 | "We can train this multi-layered network using the function `train` that we have defined above:" |
| 1562 | ] |
| 1563 | }, |
| 1564 | { |
| 1565 | "cell_type": "code", |
| 1566 | "execution_count": 37, |
| 1567 | "metadata": { |
| 1568 | "colab": { |
| 1569 | "base_uri": "https://localhost:8080/" |
| 1570 | }, |
| 1571 | "id": "ogXKdcfIs_ND", |
| 1572 | "outputId": "957ccd8d-0076-4e9b-89f1-edc1de75f18e" |
| 1573 | }, |
| 1574 | "outputs": [ |
| 1575 | { |
| 1576 | "name": "stdout", |
| 1577 | "output_type": "stream", |
| 1578 | "text": [ |
| 1579 | "Epoch 0: last batch loss = 0.5835739970207214, val acc = 0.800000011920929\n", |
| 1580 | "Epoch 1: last batch loss = 0.4642275869846344, val acc = 0.800000011920929\n", |
| 1581 | "Epoch 2: last batch loss = 0.35158076882362366, val acc = 0.800000011920929\n", |
| 1582 | "Epoch 3: last batch loss = 0.26132312417030334, val acc = 0.800000011920929\n", |
| 1583 | "Epoch 4: last batch loss = 0.19465585052967072, val acc = 0.800000011920929\n", |
| 1584 | "Epoch 5: last batch loss = 0.14735405147075653, val acc = 0.800000011920929\n", |
| 1585 | "Epoch 6: last batch loss = 0.11454981565475464, val acc = 0.800000011920929\n", |
| 1586 | "Epoch 7: last batch loss = 0.09244414418935776, val acc = 0.800000011920929\n", |
| 1587 | "Epoch 8: last batch loss = 0.07805468142032623, val acc = 0.800000011920929\n", |
| 1588 | "Epoch 9: last batch loss = 0.06894762068986893, val acc = 0.800000011920929\n" |
| 1589 | ] |
| 1590 | } |
| 1591 | ], |
| 1592 | "source": [ |
| 1593 | "train(net,dataloader,val_x,val_lab)" |
| 1594 | ] |
| 1595 | }, |
| 1596 | { |
| 1597 | "cell_type": "markdown", |
| 1598 | "metadata": { |
| 1599 | "id": "jY4R1XEGtEzJ" |
| 1600 | }, |
| 1601 | "source": [ |
| 1602 | "## Defining a Network as a Class\n", |
| 1603 | "\n", |
| 1604 | "Using a class inherited from `torch.nn.Module` is a more flexible method, because we can define any computations inside it. `Module` automates a lot of things, eg. it automatically understands all internal variables that are PyTorch layers, and gathers their parameters for optimization. You just need to define all layers of the network as members of the class:" |
| 1605 | ] |
| 1606 | }, |
| 1607 | { |
| 1608 | "cell_type": "code", |
| 1609 | "execution_count": 38, |
| 1610 | "metadata": { |
| 1611 | "colab": { |
| 1612 | "base_uri": "https://localhost:8080/" |
| 1613 | }, |
| 1614 | "id": "SlsJmGu0tMsZ", |
| 1615 | "outputId": "240d5c89-096c-4392-99cd-1ade5ff3e3e1" |
| 1616 | }, |
| 1617 | "outputs": [ |
| 1618 | { |
| 1619 | "name": "stdout", |
| 1620 | "output_type": "stream", |
| 1621 | "text": [ |
| 1622 | "MyNet(\n", |
| 1623 | " (fc1): Linear(in_features=2, out_features=10, bias=True)\n", |
| 1624 | " (func): ReLU()\n", |
| 1625 | " (fc2): Linear(in_features=10, out_features=1, bias=True)\n", |
| 1626 | ")\n" |
| 1627 | ] |
| 1628 | } |
| 1629 | ], |
| 1630 | "source": [ |
| 1631 | "class MyNet(torch.nn.Module):\n", |
| 1632 | " def __init__(self,hidden_size=10,func=torch.nn.Sigmoid()):\n", |
| 1633 | " super().__init__()\n", |
| 1634 | " self.fc1 = torch.nn.Linear(2,hidden_size)\n", |
| 1635 | " self.func = func\n", |
| 1636 | " self.fc2 = torch.nn.Linear(hidden_size,1)\n", |
| 1637 | "\n", |
| 1638 | " def forward(self,x):\n", |
| 1639 | " x = self.fc1(x)\n", |
| 1640 | " x = self.func(x)\n", |
| 1641 | " x = self.fc2(x)\n", |
| 1642 | " return x\n", |
| 1643 | " \n", |
| 1644 | "net = MyNet(func=torch.nn.ReLU())\n", |
| 1645 | "print(net)" |
| 1646 | ] |
| 1647 | }, |
| 1648 | { |
| 1649 | "cell_type": "code", |
| 1650 | "execution_count": 39, |
| 1651 | "metadata": { |
| 1652 | "colab": { |
| 1653 | "base_uri": "https://localhost:8080/" |
| 1654 | }, |
| 1655 | "id": "HwdapRxft-7M", |
| 1656 | "outputId": "6eb900cf-4902-4a04-c62b-497b68455406" |
| 1657 | }, |
| 1658 | "outputs": [ |
| 1659 | { |
| 1660 | "name": "stdout", |
| 1661 | "output_type": "stream", |
| 1662 | "text": [ |
| 1663 | "Epoch 0: last batch loss = 0.7821246981620789, val acc = 0.46666666865348816\n", |
| 1664 | "Epoch 1: last batch loss = 0.7457502484321594, val acc = 0.5333333611488342\n", |
| 1665 | "Epoch 2: last batch loss = 0.7120334506034851, val acc = 0.5333333611488342\n", |
| 1666 | "Epoch 3: last batch loss = 0.6811249256134033, val acc = 0.6666666865348816\n", |
| 1667 | "Epoch 4: last batch loss = 0.6533011794090271, val acc = 0.7333333492279053\n", |
| 1668 | "Epoch 5: last batch loss = 0.627849280834198, val acc = 0.7333333492279053\n", |
| 1669 | "Epoch 6: last batch loss = 0.6030643582344055, val acc = 0.800000011920929\n", |
| 1670 | "Epoch 7: last batch loss = 0.5775002837181091, val acc = 0.800000011920929\n", |
| 1671 | "Epoch 8: last batch loss = 0.5522137880325317, val acc = 0.8666666746139526\n", |
| 1672 | "Epoch 9: last batch loss = 0.5250465869903564, val acc = 0.8666666746139526\n" |
| 1673 | ] |
| 1674 | } |
| 1675 | ], |
| 1676 | "source": [ |
| 1677 | "train(net,dataloader,val_x,val_lab,lr=0.005)" |
| 1678 | ] |
| 1679 | }, |
| 1680 | { |
| 1681 | "cell_type": "markdown", |
| 1682 | "metadata": { |
| 1683 | "id": "dvAiaj_JndyP" |
| 1684 | }, |
| 1685 | "source": [ |
| 1686 | "**Task 1**: Plot the graphs of loss function and accuracy on training and validation data during training\n", |
| 1687 | "\n", |
| 1688 | "**Task 2**: Try to solve MNIST classificiation problem using this code. Hint: use `crossentropy_with_logits` as a loss function." |
| 1689 | ] |
| 1690 | }, |
| 1691 | { |
| 1692 | "cell_type": "markdown", |
| 1693 | "metadata": {}, |
| 1694 | "source": [ |
| 1695 | "## Takeaways\n", |
| 1696 | "\n", |
| 1697 | "* PyTorch allows you to operate on tensors at low level, you have most flexibility.\n", |
| 1698 | "* There are convenient tools to work with data, such as Datasets and Dataloaders.\n", |
| 1699 | "* You can define neural network architectures using `Sequential` syntax, or inheriting a class from `torch.nn.Module`\n", |
| 1700 | "* For even simpler approach to defining and training a network - look into PyTorch Lightning" |
| 1701 | ] |
| 1702 | } |
| 1703 | ], |
| 1704 | "metadata": { |
| 1705 | "accelerator": "GPU", |
| 1706 | "celltoolbar": "Slideshow", |
| 1707 | "colab": { |
| 1708 | "collapsed_sections": [], |
| 1709 | "name": "IntroPyTorch.ipynb", |
| 1710 | "provenance": [] |
| 1711 | }, |
| 1712 | "interpreter": { |
| 1713 | "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5" |
| 1714 | }, |
| 1715 | "kernelspec": { |
| 1716 | "display_name": "Python 3.8.12 64-bit (conda)", |
| 1717 | "name": "python3" |
| 1718 | }, |
| 1719 | "language_info": { |
| 1720 | "codemirror_mode": { |
| 1721 | "name": "ipython", |
| 1722 | "version": 3 |
| 1723 | }, |
| 1724 | "file_extension": ".py", |
| 1725 | "mimetype": "text/x-python", |
| 1726 | "name": "python", |
| 1727 | "nbconvert_exporter": "python", |
| 1728 | "pygments_lexer": "ipython3", |
| 1729 | "version": "3.8.12" |
| 1730 | }, |
| 1731 | "livereveal": { |
| 1732 | "start_slideshow_at": "selected" |
| 1733 | } |
| 1734 | }, |
| 1735 | "nbformat": 4, |
| 1736 | "nbformat_minor": 0 |
| 1737 | } |
| 1738 | |