{"cells":[{"cell_type":"markdown","metadata":{"id":"INuiHpSiH58F"},"source":["#PyTorch Tutorial\n","This colab notebook is based on the official [PyTorch tutorial ](https://docs.pytorch.org/tutorials/beginner/basics/intro.html). We may reference a condensed version of the final training loop, which is viewable [here](https://colab.research.google.com/drive/1gkMO8fOiZ4rZ8zP2mny344Xkf3YAm7Mf)."]},{"cell_type":"markdown","metadata":{"id":"SCXBFC4hNqzm"},"source":["##1) Tensors"]},{"cell_type":"markdown","metadata":{"id":"id6qrtKFNlu2"},"source":["\n","\n","Tensors are a specialized data structure that are very similar to arrays\n","and matrices. In PyTorch, we use tensors to encode the inputs and\n","outputs of a model, as well as the model's parameters.\n","\n","Tensors are similar to [NumPy's](https://numpy.org/) ndarrays (see Python Tutorial), except\n","that tensors can run on GPUs or other hardware accelerators. In fact,\n","tensors and NumPy arrays can often share the same underlying memory,\n","eliminating the need to copy data. Tensors are also\n","optimized for automatic differentiation (we\\'ll see more about that\n","later in the [Autograd](autogradqs_tutorial.html) section). If you're\n","familiar with ndarrays, you'll be right at home with the Tensor API. If\n","not, follow along!\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"643jWuWjNlu3"},"outputs":[],"source":["import torch\n","import numpy as np"]},{"cell_type":"markdown","metadata":{"id":"WT98ByXzNlu4"},"source":["###Initializing a Tensor\n","\n","Tensors can be initialized in various ways. Take a look at the following\n","examples:\n","\n","**Directly from data**\n","\n","Tensors can be created directly from data. The data type is\n","automatically inferred.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1JXQOAG4Nlu4"},"outputs":[],"source":["data = [[1, 2],[3, 4]]\n","x_data = torch.tensor(data)"]},{"cell_type":"markdown","metadata":{"id":"Im2XbI9TNlu5"},"source":["**With random or constant values:**\n","\n","`shape` is a tuple of tensor dimensions. In the functions below, it\n","determines the dimensionality of the output tensor.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"r_h4H3RmNlu5","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772054155954,"user_tz":300,"elapsed":76,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"d798341c-351d-4705-e98d-2ddca56b5938"},"outputs":[{"output_type":"stream","name":"stdout","text":["Random Tensor: \n"," tensor([[0.5923, 0.5808, 0.7881],\n","        [0.1156, 0.5566, 0.9350]]) \n","\n","Ones Tensor: \n"," tensor([[1., 1., 1.],\n","        [1., 1., 1.]]) \n","\n","Zeros Tensor: \n"," tensor([[0., 0., 0.],\n","        [0., 0., 0.]])\n"]}],"source":["shape = (2,3,)\n","rand_tensor = torch.rand(shape)\n","ones_tensor = torch.ones(shape)\n","zeros_tensor = torch.zeros(shape)\n","\n","print(f\"Random Tensor: \\n {rand_tensor} \\n\")\n","print(f\"Ones Tensor: \\n {ones_tensor} \\n\")\n","print(f\"Zeros Tensor: \\n {zeros_tensor}\")"]},{"cell_type":"markdown","metadata":{"id":"S8BWELaDNlu6"},"source":["###Attributes of a Tensor\n","\n","Tensor attributes describe their shape, datatype, and the device on\n","which they are stored.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jzDEX6fENlu6","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772054207203,"user_tz":300,"elapsed":36,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"cc3ab9ac-69be-4d73-b6f3-3474d2d8313f"},"outputs":[{"output_type":"stream","name":"stdout","text":["Shape of tensor: torch.Size([3, 4])\n","Datatype of tensor: torch.float32\n","Device tensor is stored on: cpu\n"]}],"source":["tensor = torch.rand(3,4)\n","\n","print(f\"Shape of tensor: {tensor.shape}\")\n","print(f\"Datatype of tensor: {tensor.dtype}\")\n","print(f\"Device tensor is stored on: {tensor.device}\")"]},{"cell_type":"markdown","metadata":{"id":"U4DIsvetNlu6"},"source":["------------------------------------------------------------------------\n"]},{"cell_type":"markdown","metadata":{"id":"TPr1eaa0Nlu6"},"source":["###Operations on Tensors\n","\n","Over 1200 tensor operations, including arithmetic, linear algebra,\n","matrix manipulation (transposing, indexing, slicing), sampling and more\n","are comprehensively described\n","[here](https://pytorch.org/docs/stable/torch.html).\n","\n","Each of these operations can be run on the CPU and\n","[Accelerator](https://pytorch.org/docs/stable/torch.html#accelerators)\n","such as CUDA, MPS, MTIA, or XPU. If you're using Colab, allocate an\n","accelerator by going to Runtime \\> Change runtime type \\> GPU.\n","\n","By default, tensors are created on the CPU. We need to explicitly move\n","tensors to the accelerator using `.to` method (after checking for\n","accelerator availability). Keep in mind that copying large tensors\n","across devices can be expensive in terms of time and memory!\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VY-QB02-Nlu6","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772054315323,"user_tz":300,"elapsed":155,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"2aec7871-1e3a-4ffe-f79c-c8903e587584"},"outputs":[{"output_type":"stream","name":"stdout","text":["Tensor now lives on device: cuda:0\n"]}],"source":["# We move our tensor to the current accelerator if available\n","if torch.accelerator.is_available():\n","    tensor = tensor.to(torch.accelerator.current_accelerator())\n","\n","print(f\"Tensor now lives on device: {tensor.device}\")"]},{"cell_type":"markdown","metadata":{"id":"-3XwLGDQNlu6"},"source":["Try out some of the operations from the list. If you\\'re familiar with\n","the NumPy API, you\\'ll find the Tensor API a breeze to use. They are nearly identical, with a small number of notable differences (such as dimensions being referred to with ``dim`` rather than ``axis``.)\n"]},{"cell_type":"markdown","metadata":{"id":"NKhpd-dUNlu6"},"source":["**Standard numpy-like indexing and slicing:**\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lZA-WfEANlu7","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772054429371,"user_tz":300,"elapsed":12,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"567c1cb2-3165-4bd2-f9bb-fb9c2805b3fb"},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[0.1969, 0.1583, 0.6543, 0.1055],\n","        [0.8903, 0.3037, 0.9863, 0.0601],\n","        [0.2858, 0.0979, 0.8000, 0.7475],\n","        [0.5590, 0.2498, 0.8525, 0.0892]])\n","First row: tensor([0.1969, 0.1583, 0.6543, 0.1055])\n","First column: tensor([0.1969, 0.8903, 0.2858, 0.5590])\n","Last column: tensor([0.1055, 0.0601, 0.7475, 0.0892])\n","tensor([[0.1969, 0.0000, 0.6543, 0.1055],\n","        [0.8903, 0.0000, 0.9863, 0.0601],\n","        [0.2858, 0.0000, 0.8000, 0.7475],\n","        [0.5590, 0.0000, 0.8525, 0.0892]])\n"]}],"source":["tensor = torch.rand(4, 4)\n","print(tensor)\n","print(f\"First row: {tensor[0]}\")\n","print(f\"First column: {tensor[:, 0]}\")\n","print(f\"Last column: {tensor[..., -1]}\")\n","tensor[:,1] = 0\n","print(tensor)"]},{"cell_type":"markdown","metadata":{"id":"t32EZb2mNlu7"},"source":["**Arithmetic operations**\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"boMeMUphNlu7","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772054571792,"user_tz":300,"elapsed":19,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"5e50db9e-7cff-4e71-b765-123af3517d29"},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[0.4780, 0.8270, 0.6586, 0.6773],\n","        [0.8270, 1.7690, 1.0884, 1.3439],\n","        [0.6586, 1.0884, 1.2804, 0.9085],\n","        [0.6773, 1.3439, 0.9085, 1.0473]])\n","tensor([[0.0388, 0.0000, 0.4281, 0.0111],\n","        [0.7926, 0.0000, 0.9728, 0.0036],\n","        [0.0817, 0.0000, 0.6400, 0.5588],\n","        [0.3125, 0.0000, 0.7268, 0.0080]])\n"]}],"source":["# Torch provides similar arithmetic operations on tensors as numpy but with automatic differentiation built in!\n","y1 = tensor @ tensor.T\n","z1 = tensor * tensor\n","print(y1)\n","print(z1)"]},{"cell_type":"markdown","metadata":{"id":"JKGXEfA4Nlu7"},"source":["**Single-element tensors** If you have a one-element tensor, for example\n","by aggregating all values of a tensor into one value, you can convert it\n","to a Python numerical value using `item()`:\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"949nXxw_Nlu7","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772054693153,"user_tz":300,"elapsed":14,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"3ecec7ee-acd5-4325-a266-2a7518b6ca6f"},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor(6.2275) <class 'torch.Tensor'>\n","6.227480411529541 <class 'float'>\n"]}],"source":["agg = tensor.sum()\n","agg_item = agg.item()\n","print(agg, type(agg))\n","print(agg_item, type(agg_item))"]},{"cell_type":"markdown","metadata":{"id":"dJKpvpimNlu8"},"source":["------------------------------------------------------------------------\n"]},{"cell_type":"markdown","metadata":{"id":"t58oqjijNlu8"},"source":["###Bridge with NumPy\n","\n","Tensors on the CPU and NumPy arrays can share their underlying memory\n","locations, and changing one will change the other.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Vf1HAvaaNlu8","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772054801674,"user_tz":300,"elapsed":6,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"f56061cf-3b95-477a-e361-48f8e6bff141"},"outputs":[{"output_type":"stream","name":"stdout","text":["t: tensor([1., 1., 1., 1., 1.])\n","n: [1. 1. 1. 1. 1.]\n","t: tensor([0., 1., 1., 1., 1.])\n","n: [0. 1. 1. 1. 1.]\n","t2: tensor([0., 1., 1., 1., 1.])\n"]}],"source":["t = torch.ones(5)\n","print(f\"t: {t}\")\n","\n","# Tensor to NumPy array\n","n = t.numpy()\n","print(f\"n: {n}\")\n","\n","t[0] = 0\n","print(f\"t: {t}\")\n","print(f\"n: {n}\")\n","\n","# NumPy array to Tensor\n","t2 = torch.from_numpy(n)\n","print(f\"t2: {t2}\")"]},{"cell_type":"markdown","metadata":{"id":"OITGcG2INIxo"},"source":["##2) Datasets & DataLoaders\n"]},{"cell_type":"markdown","metadata":{"id":"osDmRDXPNIxq"},"source":["Code for processing data samples can get messy and hard to maintain; we\n","ideally want our dataset code to be decoupled from our model training\n","code for better readability and modularity. PyTorch provides two data\n","primitives: `torch.utils.data.Dataset` and `torch.utils.data.DataLoader`\n","that allow you to use pre-loaded datasets as well as your own data.\n","`Dataset` stores the samples and their corresponding labels, and\n","`DataLoader` wraps an iterable around the `Dataset` to enable easy\n","access to the samples.\n","\n","PyTorch domain libraries provide a number of pre-loaded datasets (such\n","as FashionMNIST) that subclass `torch.utils.data.Dataset` and implement\n","functions specific to the particular data. They can be used to prototype\n","and benchmark your model. You can find them here: [Image\n","Datasets](https://pytorch.org/vision/stable/datasets.html), [Text\n","Datasets](https://pytorch.org/text/stable/datasets.html), and [Audio\n","Datasets](https://pytorch.org/audio/stable/datasets.html)\n"]},{"cell_type":"markdown","metadata":{"id":"qo_2tVm_NIxq"},"source":["###Loading a Dataset\n","\n","Here is an example of how to load the\n","[Fashion-MNIST](https://research.zalando.com/project/fashion_mnist/fashion_mnist/)\n","dataset from TorchVision. Fashion-MNIST is a dataset of Zalando's\n","article images consisting of 60,000 training examples and 10,000 test\n","examples. Each example comprises a 28×28 grayscale image and an\n","associated label from one of 10 classes.\n","\n","We load the [FashionMNIST Dataset](https://pytorch.org/vision/stable/datasets.html#fashion-mnist) with the following parameters:\n","\n","*   `root` is the path where the train/test data is stored,\n","*   `train` specifies training or test dataset,\n","*   `download=True` downloads the data from the internet if it\\'s\n","    not available at `root`.\n","*   `transform` and `target_transform` specify the feature and label\n","    transformations; `torchvision.transforms` implements a reasonably comprehensive library of image transformations\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"A8TNtWcUNIxq","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772055122541,"user_tz":300,"elapsed":12914,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"afc40872-a3dc-493f-81a0-5df022a81148"},"outputs":[{"output_type":"stream","name":"stderr","text":["100%|██████████| 26.4M/26.4M [00:02<00:00, 11.9MB/s]\n","100%|██████████| 29.5k/29.5k [00:00<00:00, 188kB/s]\n","100%|██████████| 4.42M/4.42M [00:01<00:00, 3.50MB/s]\n","100%|██████████| 5.15k/5.15k [00:00<00:00, 23.9MB/s]\n"]}],"source":["import torch\n","from torch.utils.data import Dataset\n","from torchvision import datasets\n","from torchvision.transforms import ToTensor\n","import matplotlib.pyplot as plt\n","\n","\n","training_data = datasets.FashionMNIST(\n","    root=\"data\",\n","    train=True,\n","    download=True,\n","    transform=ToTensor()\n",")\n","\n","test_data = datasets.FashionMNIST(\n","    root=\"data\",\n","    train=False,\n","    download=True,\n","    transform=ToTensor()\n",")"]},{"cell_type":"markdown","metadata":{"id":"yQQ_t4KONIxr"},"source":["###Iterating and Visualizing the Dataset\n","\n","We can index `Datasets` manually like a list: `training_data[index]`. We\n","use `matplotlib` to visualize some samples in our training data.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5Liz_ULsNIxr","colab":{"base_uri":"https://localhost:8080/","height":675},"executionInfo":{"status":"ok","timestamp":1772055305840,"user_tz":300,"elapsed":454,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"55e66505-9a0c-4ba4-dd62-458ac2c11643"},"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 800x800 with 9 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAn4AAAKSCAYAAABMVtaZAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAZcJJREFUeJzt3Xl4VeXV//9PSMg8MQQCARIGQQapAyoWkUGEKopSQLEqqIi2zlP7qE+1Wlstap1QHPpVVFqrRaFFARULah1wFkXUIgQEhTAGEgIJJOf3hz/zGHOvG84xkIT7/bquXldZ+6xz7nOy99nLnay14yKRSEQAAADY7zWp7wUAAABg36DwAwAACASFHwAAQCAo/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgaDw2wfi4uJ04403Vv/7scceU1xcnFasWFFvawIAYG/77nz33nvv7faxAwcO1MCBA/f+ogJH4efw3Y763f+Sk5PVtWtXXXzxxSoqKqrv5QFBcR2Pbdu21bBhw3TvvfeqpKSkvpcINDrfP6Z8/3vllVec+VVVVXriiSd05JFHqnnz5srIyFDXrl01btw4LVy4cK+vf8mSJbrxxhu5gBKDhPpeQEP2+9//Xh07dtSOHTv0+uuv64EHHtCcOXO0ePFipaam1vfygKB8dzzu3LlTa9eu1SuvvKLLL79cd955p2bNmqXevXvX9xKBRmPatGk1/v3EE09o3rx5teLdu3d35l966aW6//77dfLJJ+uMM85QQkKCvvjiC82dO1edOnVS3759o17TSy+9tMePXbJkiW666SYNHDhQBQUFUb9WyCj8PI4//nj16dNHknTeeeepRYsWuvPOO/Wvf/1Lp59+ej2vbu/Ztm2b0tLS6nsZQA3fPx4l6dprr9X8+fN14oknasSIEfrss8+UkpLizGWfBmo688wza/x74cKFmjdvXq24S1FRkaZMmaKJEyfq4YcfrrHt7rvv1vr162NaU2Ji4m4fs2PHjj16HGz8qjcKgwcPliQVFhaaf4tw9tlnx/xfH1OmTFHPnj2VlJSktm3b6qKLLlJxcXH19osvvljp6ekqKyurlXv66acrNzdXlZWV1bG5c+eqf//+SktLU0ZGhoYPH65PP/201nrT09O1bNkynXDCCcrIyNAZZ5wR0/qBfW3w4MG6/vrrtXLlSv31r3+V5N+nq6qqdPfdd6tnz55KTk5W69atdcEFF2jz5s01nve9997TsGHD1LJlS6WkpKhjx44699xzazzmqaee0mGHHaaMjAxlZmbqoIMO0j333LNv3jhQjwoLCxWJRNSvX79a2+Li4tSqVata8fLycl155ZXKyclRWlqaRo4cWatA/OF59ZVXXlFcXJyeeuop/fa3v1VeXp5SU1N17733asyYMZKkQYMG7fbX0qiJwi8Ky5YtkyS1aNGizp/7xhtv1EUXXaS2bdvqz3/+s0aNGqWHHnpIQ4cO1c6dOyVJp512mrZt26bZs2fXyC0rK9Nzzz2n0aNHKz4+XtK3l/GHDx+u9PR0TZo0Sddff72WLFmio48+utbfROzatUvDhg1Tq1atdMcdd2jUqFF1/v6AveWss86SVPPXRNY+fcEFF+jXv/61+vXrp3vuuUfnnHOO/va3v2nYsGHVx9m6des0dOhQrVixQtdcc40mT56sM844o8bfLc2bN0+nn366mjVrpkmTJulPf/qTBg4cqDfeeGMfvnOgfuTn50uSpk+f7rwQ4XLJJZdo0aJF+t3vfqdf/epXeu6553TxxRfvUe7NN9+s2bNn6+qrr9Ytt9yioUOH6tJLL5UkXXfddZo2bZqmTZtm/loaPxBBLVOnTo1Iirz88suR9evXR1atWhV56qmnIi1atIikpKREVq9eHRkwYEBkwIABtXLHjx8fyc/PrxGTFPnd735X6/kLCwsjkUgksm7dukhiYmJk6NChkcrKyurH3XfffRFJkUcffTQSiUQiVVVVkby8vMioUaNqPP8//vGPiKTIa6+9FolEIpGSkpJIdnZ2ZOLEiTUet3bt2khWVlaN+Pjx4yOSItdcc020HxOwT3x3vLz77rvmY7KysiKHHHJIJBKx9+n//Oc/EUmRv/3tbzXiL7zwQo34zJkzd/t6l112WSQzMzOya9euWN8W0KBcdNFFkWhKgnHjxkUkRZo1axYZOXJk5I477oh89tlntR733fE7ZMiQSFVVVXX8iiuuiMTHx0eKi4urYz88ry5YsCAiKdKpU6dIWVlZjeedPn16RFJkwYIFe/4mEYlEIhGu+HkMGTJEOTk5at++vcaOHav09HTNnDlTeXl5dfo6L7/8sioqKnT55ZerSZP/+5FMnDhRmZmZ1Vf44uLiNGbMGM2ZM0elpaXVj3v66aeVl5eno48+WtK3VyOKi4t1+umna8OGDdX/i4+P15FHHqkFCxbUWsOvfvWrOn1PwL6Unp5eq7v3h/v09OnTlZWVpeOOO67GcXHYYYcpPT29+rjIzs6WJD3//PPVVwF/KDs7W9u2bdO8efPq/s0AjcDUqVN13333qWPHjpo5c6auvvpqde/eXccee6y+/vrrWo8///zzFRcXV/3v/v37q7KyUitXrtzta40fP978+11Ej8LP4/7779e8efO0YMECLVmyRMuXL9ewYcPq/HW+2/G7detWI56YmKhOnTrVODBOO+00bd++XbNmzZIklZaWas6cORozZkz1QbV06VJJ3/79U05OTo3/vfTSS1q3bl2N10lISFC7du3q/H0B+0ppaakyMjKq/+3ap5cuXaotW7aoVatWtY6L0tLS6uNiwIABGjVqlG666Sa1bNlSJ598sqZOnary8vLq57rwwgvVtWtXHX/88WrXrp3OPfdcvfDCC/vmzQL7SGlpqdauXVv9v+//TV6TJk100UUX6f3339eGDRv0r3/9S8cff7zmz5+vsWPH1nquDh061Ph3s2bNJKnW39e6dOzY8Ue+E3wfXb0eRxxxRI0uwu+Li4tTJBKpFf9+c8Xe0LdvXxUUFOgf//iHfvGLX+i5557T9u3bddppp1U/pqqqStK3f+eXm5tb6zkSEmr+2JOSkmpcaQQak9WrV2vLli3q0qVLdcy1T1dVValVq1b629/+5nyenJwcSd8e288884wWLlyo5557Ti+++KLOPfdc/fnPf9bChQuVnp6uVq1a6aOPPtKLL76ouXPnau7cuZo6darGjRunxx9/fO+9WWAfuuOOO3TTTTdV/zs/P985N69FixYaMWKERowYoYEDB+rVV1/VypUrq/8WUFL135//kOs8+kNc7atbFH4xatasmZYvX14rvieXrX/ou4Pjiy++UKdOnarjFRUVKiws1JAhQ2o8/tRTT9U999yjrVu36umnn1ZBQUGNmUmdO3eWJLVq1apWLrC/+W7u2O6uxnfu3Fkvv/yy+vXrt0cnkr59+6pv37764x//qCeffFJnnHGGnnrqKZ133nmSvr0if9JJJ+mkk05SVVWVLrzwQj300EO6/vrraxShQGM1bty46j8hkvasAOvTp49effVVrVmzpkbhV9e+/2tjRIfLPDHq3LmzPv/88xqXvhctWhRTV9+QIUOUmJioe++9t8Z//TzyyCPasmWLhg8fXuPxp512msrLy/X444/rhRde0Kmnnlpj+7Bhw5SZmalbbrnF+TdKsc5YAhqa+fPn6+abb1bHjh13O4bo1FNPVWVlpW6++eZa23bt2lU9Omnz5s21rkIcfPDBklT9696NGzfW2N6kSZPqAdLf/5Uw0Jh16tRJQ4YMqf7fd+Nb1q5dqyVLltR6fEVFhf7973+rSZMme/0/fr6by/n9kWfYM1zxi9G5556rO++8U8OGDdOECRO0bt06Pfjgg+rZs6e2bt0a1XPl5OTo2muv1U033aSf/exnGjFihL744gtNmTJFhx9+eK2Bmoceeqi6dOmi//3f/1V5eXmNX/NKUmZmph544AGdddZZOvTQQzV27Fjl5OToq6++0uzZs9WvXz/dd999P/ozAPaluXPn6vPPP9euXbtUVFSk+fPna968ecrPz9esWbOUnJzszR8wYIAuuOAC3Xrrrfroo480dOhQNW3aVEuXLtX06dN1zz33aPTo0Xr88cc1ZcoUjRw5Up07d1ZJSYn+8pe/KDMzUyeccIKkbwe6b9q0SYMHD1a7du20cuVKTZ48WQcffDAjJbDfW716tY444ggNHjxYxx57rHJzc7Vu3Tr9/e9/16JFi3T55ZerZcuWe3UNBx98sOLj4zVp0iRt2bJFSUlJGjx4sHOGIGqi8ItR9+7d9cQTT+iGG27QlVdeqR49emjatGl68sknYxoieeONNyonJ0f33XefrrjiCjVv3lznn3++brnlFjVt2rTW40877TT98Y9/VJcuXXTooYfW2v6LX/xCbdu21Z/+9CfdfvvtKi8vV15envr3769zzjknlrcM1KsbbrhB0re/Ym3evLkOOugg3X333TrnnHNqNHb4PPjggzrssMP00EMP6brrrlNCQoIKCgp05plnVl/NGDBggN555x099dRTKioqUlZWlo444gj97W9/q/4j8zPPPFMPP/ywpkyZouLiYuXm5uq0007TjTfeyN/LYr/XrVs33X333ZozZ46mTJmioqIiJScnq1evXvrLX/6iCRMm7PU15Obm6sEHH9Stt96qCRMmqLKyUgsWLKDw2wNxkT35y0oAAAA0evynKQAAQCAo/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgdjjAc51eV8833NZYwVjyalr1r1Av38vwx9KSkpyxisrK82chAT3j+W74bEu1mfwj3/8w8yZPn26uS0UDXGMJfegjM2kSZOc8f/5n/+J+rl8Q5itfca3L1k/04a4/+0tDfG9hnKs+d7nKaec4oz/8pe/NHPS09Odcd9t2saMGeOMjxgxwsw5+eSTo36dWbNmOeOPPPJI1DmN1e6ONa74AQAABILCDwAAIBAUfgAAAIGg8AMAAAhEXGQP/+K2Mf4R7JAhQ8xthx9+uDPevn17M8f6g++NGzeaOTk5Oc54u3btzJyysjJn/KuvvjJz1q9f74y3bt3azLEaT7755hszZ8GCBc7466+/buY0ZPzBef2Jj483t1nNT1lZWWaOtQ8edNBB0S0sRg2hAa0ha4ifQUM+1goKCpzx008/3cwZNGiQM+5rDCwtLXXGfecB67ziO9Z+/vOfO+OXX365mdOnTx9nvLCw0Mz5+uuvnfHevXubOdb3zSeffGLm3HLLLc7422+/bebsKzR3AAAAQBKFHwAAQDAo/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEol7GucQy9iAlJcXMsdrBW7ZsaeZYLew7d+40c7Zt2+aMJyYmmjnWaJYdO3aYOVVVVc54ZmammVOX9wROTU01c9LS0pxx3+cWy71S99W9TRkxUX9iGefSqVMnM8caveAb41BcXOyMV1RUmDklJSXmNtg41mrz3af24osvdsZ93+nWfrt9+/boFib73CXZP8vmzZubOYcccogzvmnTJjNnxYoVzviGDRvMHN/IJ4u1H/jO7c2aNXPG//3vf5s5v//976NbWIwY5wIAAABJFH4AAADBoPADAAAIBIUfAABAICj8AAAAAuFu69zLYunumjBhgrnN6t71df5YHa2+jiCrs7i8vDzqHF83mdXxm56ebuZYnV6+bq6MjAxn3NfNZXU2WzftlqSJEyc643/5y1/MnIbYAYj61717d3Pbrl27nHFf56TVWezr6rVuXu/r5tu8ebMzHsuEA+w/xowZY24rKipyxq39XIptuoOlSRP7upB13KxZs8bMsd6PNcVCkpKTk51x35QP6/PxvY7Fmv4hSevWrXPG+/XrZ+Ycf/zxzvjcuXOjW9iPxBU/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAg6mWcSyyskS2SPX6kadOmZo7Vjn7nnXdGnfPb3/7WzHnqqaeccd+N3i+77DJn/LbbbjNzBg8e7Iy3a9fOzPnggw+c8TZt2pg51k24fW3vvucDojFw4EBz25YtW5zxt956y8zJz893xn2jLKwxSH379jVz9vW4BjQsZ555pjOelpZm5lj7s2/MjzW2xTcCxhpz4jt/Ws/nGzlmjS7yjZqx3qvvM7BGMflGJ1lriGWk0sqVK81tP/vZz5xxxrkAAABgr6DwAwAACASFHwAAQCAo/AAAAAJB4QcAABCIBtfVa3XO+jqMLL4Oo/fee88Z79mzp5lz6KGHOuM7d+40c95++21nPDEx0cyx3uvGjRvNnNzcXGc8JyfHzHnnnXec8QEDBpg5bdu2dca3b99u5lg3Dk9IsHc/Xxca9g+xdMx169bN3GZ1SDZr1szMsW727tv/duzY4Yz7OoERtn79+jnjsXzP+bpTY3k+63xjdfv61PX7sfg6ga3ns2oLyf4MysvLo1vYbl6nRYsWzrjvM4jle3J3+KYCAAAIBIUfAABAICj8AAAAAkHhBwAAEAgKPwAAgEBQ+AEAAASiwY1zadeunTPua5G2xjhYYxd8Oc8++6yZU1JS4oz36tXLzLHGn5SWlpo5K1ascMZ9LexPPPGEMz5hwgQzx2phT05ONnOsFn/f6Byrxb9NmzZmzqpVq8xt2D/EMqbAl2ONSMrLyzNzrBu6+0azWMeh7/hE2KwxYWvXrjVzUlNTnXHrPCTZ+7Pv/GmN2/KNc6nLESN1/T0QyzgXaySbb1SbbyRbtDk/+clPzJyPPvoo6tfZHa74AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgGlxXb0FBgTPu6zDKzs52xouLi82c3NxcZ3zixIlmTocOHZxx342pL7zwQmd8w4YNZo7VseTr0F2yZIkzvnXrVjPnmGOOccYTEuzdwupS9n3W1ufj67akq3f/F0s3X6tWrcxtVpe6bz9buXKlMx5LB+D27dvNHMveuAE76seBBx4YdU4s3am+KRLr1693xpctW2bmWJ2mvnOuJZacumYdu76u+02bNjnjrVu3NnOsqSG+esBaW7du3cwcunoBAAAQMwo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAhEgxvnYo1r8LW9W2MccnJyzJxPP/3UGe/atauZ8+GHHzrjvvZta1vHjh3NnE8++cQZt26mLdlt59ZIAEnq1KmTM7569Wozp2XLls74xo0bzRxLs2bNos5B2NLS0sxt1s3r161bZ+aUl5c7476xFBUVFc54VlaWmYP9n2+cy9q1a51x37nDGrPi+063xm2tWLHCzKnLkUJNmtjXkqxjypdjvVdrpJJkj22xPk9JKi0tdcZ79+5t5ljfA2vWrDFzrBEw1qi4vYUrfgAAAIGg8AMAAAgEhR8AAEAgKPwAAAACQeEHAAAQiAbX1Zudne2M+26abnXXWB1OklRcXOyMWze5lqR27do547F0Mvk6qfLz853xWDqZEhLsH7F1Y2qrS1qyu3qtLmnf89EFiWgddNBB5ratW7c641bHnmR3+lnfKZJ9TFsTCSQpIyPDGbc6kdH4+CY1WN271ve2ZHeAZmZmmjnJycnOeIsWLcwc61zk62zfvn17VM8l+buRo+Xr0LXOedbnKdk/H2tahmR/31gd3JL9uVnxvYUrfgAAAIGg8AMAAAgEhR8AAEAgKPwAAAACQeEHAAAQCAo/AACAQDS4cS7p6enOuNWmLtkjEXyt2FZrua8d3Rop42t7t0a9+Nr4rdfxtbBba/CNZrGez/cZpKamRv061nu1RvcAlqlTp5rbDjzwQGfcNyohLS3NGfd931ijoLZt22bmtG/f3hlfsmSJmYPGJS8vz9zWrFkzZzyWESNWXLLPN7792fq+952jysvLo3p9yV63b8yLb4ybxTfCzJKUlOSM+85r1vHuy7HeT5s2bTyrq3tc8QMAAAgEhR8AAEAgKPwAAAACQeEHAAAQCAo/AACAQDS4rl6ru8Z3Y+pNmzY541lZWWaO9Xy+Dt1YOoGt5/N1MlndVL61WWuIpaPRx1qDr5PK+plaN64HLIsWLTK3zZkzxxmPpTPQ10Fv8XVBduvWzRmnq3f/cdNNN5nbRo0a5YwfccQRZo7VCT579mwz55hjjnHGfftmRUWFM56SkmLmWHznwlhyrGPXd16z+M5RLVq0cMbfffddM+eAAw5wxsvKyswca9tHH31k5uwNXPEDAAAIBIUfAABAICj8AAAAAkHhBwAAEAgKPwAAgEBQ+AEAAASiwY1zscacpKammjlbt251xjds2GDm+G6kHC3faJa6fD7fDbCtHN9NwK0bh1s3oZfscS7p6elmjiWWcQEIgzWKqXPnzmaO9T2wc+dOM8c6bmIZ5+Ibt8Toov1fSUmJue2xxx6LKh6rDz/80Bn37c/W8eE7biy7du0yt1mjWXw51rp942ksvuPTqi+KiorMnLPOOivqNTQUXPEDAAAIBIUfAABAICj8AAAAAkHhBwAAEAgKPwAAgEA0uK5eq8PHd6N16wbLn376qZmzefNmZ9zqdPWtLRa+Dt26fD5fZ5bVMeXr6l24cKEz3qlTJzOnsLDQGU9KSjJzEDbrBvW+LvVYuhCt7sDk5GQzJ5Yb0fu+v7D/s76ffZ2msVi7dq0znpBgn+qtfXP79u1mjvV+fPt5LMdnLJ+PtYby8nIzx/pe8X0GjRlX/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgWhw41ystnPfGIe8vDxn/IknnjBzrNEsKSkpZo51E3jrRu/7kjViIpaxMdu2bTO3LVu2zBkfMGCAmfPRRx8544xzgSU3N9cZb9q0qZljjYvw7WfWcWPdtN3HN+YlPT3dGfetzTd+Ao2LNZbEd+6IZWzQpk2bnHHfmLKKigpn3HfuKCkpccat/VyyP4NYRrb4RqvFMnbNGt/05ZdfRv1cdf0z3Ru44gcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgaiXrl7fDdCtbjqro1aSDjvsMGfc6nCS9r+bplvdQr4uyLKyMmfc6vKS7G4uXweY9XxpaWlmjtXtSKdjGKyuXt/N5q39zHcMWHyvEwtrf05MTDRz2Ncbl1imO9R1l6d1nszOzq7TNVjHRyzHZ2VlZZ29vhTb+7E6izds2BD1czWUzl0frvgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAo/AAAAAJRL+NcMjIyzG3W6IXS0lIzxxol4hvZYrWD+1rLY2nX31esFnLfmq2b2vs+t+3btzvj27ZtM3OsVnnf61hjfRhxEYa8vDxn3DcqIZaxEFaObzyRleM71qyb18dyg3o0TA1hjIf1ne5j7be7du0yc6zztO87vS5HqMVyXvONTrJ+dtb4Mh/f2hrCPiJxxQ8AACAYFH4AAACBoPADAAAIBIUfAABAICj8AAAAAlEvXb3WDcsluyPG1/1mdQv5OnKsm8D7OgOtNfg6AH3b6pL1ufluUG+9V1/ON99844z7OsCsz8D3M7W6ejdv3mzmYP/Rpk0bZ9zX1R1Lh2wsN5uPpTsxLS3NGY+lExmwxNKhG8tECGu/9XWtWseN71iz+HKsCSDWOUWy309ZWVl0CxNdvQAAAGhAKPwAAAACQeEHAAAQCAo/AACAQFD4AQAABILCDwAAIBD1Ms4llvZt31gUq0V6+fLlZk7Hjh2jXkNDZn0GsYy4sG4oL0kvvPBC1M9n3Rx7x44dZo5vpAz2f61bt3bGfce0xTcyxdrPYvmO8u2zycnJznhd3rgesPjOA77xI9HyPZe1r/tyrHWnpKSYOZs2bXLGKyoqzBzr2LW+h3waysgWH674AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAg6qWr1+pwk+yOGF/H3LZt25xxXwfgSSed5IzvbzdN93UYWZ3SVheuJK1fv94Z93WNWR2Svhtg+/YR7P+sznLffmYdu75jOiMjwxn3HQNWF6L1XJKUlpbmjMfSdQ/sC7FM0ojl/Ok7BmKZVpGUlOSM79y508yxjvdmzZqZORa6egEAANBgUPgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCDqZZyLb1SC1XJtjUOQpIMPPrjO1uC7YXQsN7PeVzfAtrb5bkxttcS3aNEiuoVJOuCAA8xt7733njNeUlJi5lgjYBAG6ybs1pgXScrKynLGfSMZcnJynPHMzMyoX8eKS/aICca5oC4VFxc74+3btzdzrLEtvvONNV7NNwIm2tf3vY5vbIx1rPnOhampqVE9V2PHFT8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACES9tE7GcvNnq7tHktasWRP1Gqxuvi1btpg51tp83U91mRPLzZ99OWVlZc54LF29RUVF5rbk5GRnfNeuXWYOXb1hs7rsfBMBrM5/374Uy03grS7h0tJSM2fTpk3OuO97rby83NwGRCOW6RKxnG98+7P1fe+rB6x1+9ZmfUfs2LHDzLGmiVjfQ40dV/wAAAACQeEHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIFocONcrDEKvpslL168OOo1xHLTdGssRCyt8rHczLqu2+uttndfjmXVqlXmNmvMhtVCL0nx8fFRrwH7j4yMDGd8w4YNZo51E3bfCJjNmzc7477RD9Z3xLZt28wca93WsSH5x8MALm3btnXGfecOa3/2jduytu2rMWW+84O1Bt8513o/1iiyxo4rfgAAAIGg8AMAAAgEhR8AAEAgKPwAAAACQeEHAAAQiEbT1evrzHvvvfeiXkPHjh2d8bKyMjPHWpuvkymW7l1LLN3Dvi5lq3u3oKAg6tcpLCw0t+Xk5Djjvm6uWDqL0bj49mer697X6VpcXOyM+260bnXqN2vWzMzJzMyM6vUl+3ugd+/eZs68efPMbdg/xNIF62PlWB3vkr1v+iZpxDJ1wfpOt45BX47v9a2OfN/rWOdJa7pAY8cVPwAAgEBQ+AEAAASCwg8AACAQFH4AAACBoPADAAAIBIUfAABAIOplnIuvRTqWtuoVK1ZEvYZp06Y5475xEbG0iccyzsVqya/rcS5Wu34s43E++ugjc9txxx0X9fNlZ2dHnYPGxTdmZefOnc74hg0bzJwOHTo44y1btox6DdYIIknq1q2bM56WlmbmWKNefGOqsP+r63Eu1v7key5rDb7zmvV8vjErsZzXrPOXb9SM9Tq+Y62ysjKq52rsuOIHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGol67eWPg65latWhX1802fPv3HLAc/sHbtWnOb1TFl3YBbotsxBM2bNze3vf766854u3btzJx+/fo5488//7yZU1ZW5oz79uf169c741u2bDFzSkpKnHHr2EAYYpnU4MuxumorKiqifh1fh661hl27dpk5Voesb22xdBzH8pla73X79u1RP1djwBU/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAg6mWcS25urrnNumm6b/THsmXLol6D9Xy+8QqN8YbNsbS2N2li//eA1a6/Zs0aM2fnzp3OuG9kS2ZmprkN+wdrLIokLVmyxBlv27atmWMd082aNTNzysvLzW0Wa9/0jaXIyMhwxt99992oXx9h841BSk9Pd8Z9566qqipn3HeejuU73VpbcXGxmWPxvY51LvLlWOe8rKys6Ba2G9b5eF/XFlzxAwAACASFHwAAQCAo/AAAAAJB4QcAABAICj8AAIBA1EtX72effWZu27FjhzPuu1ny0qVLo16D1Z3aGDt3fWJ5P7Hk/Pe//zW3ffTRR874pk2bzJzly5dHvQY0LtaxLkn33HOPM+7rUm/Tpo0z7usE7tixozP+5ptvmjlW93BJSYmZs2jRImfc6o5EGKyOWp+NGzea2+68805n3JqWIUnJyclRr8E6BnzHp9X1Hssx4JvyYb0f38SOpKQkZ7ywsDC6hck/FSOWn/fewBU/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAg4iL72/wSAAAAOHHFDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgKPwAAAACQeEHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgKPwAAAACQeHXwDz22GOKi4vTe++9t9vHDhw4UAMHDtz7iwIAYC9ZsWKF4uLidMcdd9T3UoJA4beH4uLi9uh/r7zyijO/qqpKTzzxhI488kg1b95cGRkZ6tq1q8aNG6eFCxfu9fUvWbJEN954o1asWLHXXwuIxo89tgDs3ieffKLRo0crPz9fycnJysvL03HHHafJkyfX99KwjyXU9wIai2nTptX49xNPPKF58+bVinfv3t2Zf+mll+r+++/XySefrDPOOEMJCQn64osvNHfuXHXq1El9+/aNek0vvfTSHj92yZIluummmzRw4EAVFBRE/VrA3vJjjy0Afm+++aYGDRqkDh06aOLEicrNzdWqVau0cOFC3XPPPbrkkkvqe4nYhyj89tCZZ55Z498LFy7UvHnzasVdioqKNGXKFE2cOFEPP/xwjW1333231q9fH9OaEhMTd/uYHTt27NHjgPoS67FVVlam1NTUvbm0vWLbtm1KS0ur72UgIH/84x+VlZWld999V9nZ2TW2rVu3rn4WtY811u+LvYFf9e4DhYWFikQi6tevX61tcXFxatWqVa14eXm5rrzySuXk5CgtLU0jR46sVSD+8G/8XnnlFcXFxempp57Sb3/7W+Xl5Sk1NVX33nuvxowZI0kaNGgQvzpDozNw4ED16tVL77//vo455hilpqbquuuuk/TtiWvChAlq3bq1kpOT9ZOf/ESPP/54jfzvjo0f7vPf/W3RY489Vh1bu3atzjnnHLVr105JSUlq06aNTj755Fp/JjF37lz1799faWlpysjI0PDhw/Xpp5/WeMzZZ5+t9PR0LVu2TCeccIIyMjJ0xhln1NnnAuyJZcuWqWfPnrWKPkk1zj9xcXG6+OKL9c9//lO9evVSUlKSevbsqRdeeKFW3tdff61zzz1XrVu3rn7co48+WuMxFRUVuuGGG3TYYYcpKytLaWlp6t+/vxYsWLDbNUciEZ1//vlKTEzUjBkzquN//etfddhhhyklJUXNmzfX2LFjtWrVqhq5vu8LcMVvn8jPz5ckTZ8+XWPGjNmj/+q45JJL1KxZM/3ud7/TihUrdPfdd+viiy/W008/vdvcm2++WYmJibr66qtVXl6uoUOH6tJLL9W9996r6667rvpXZvzqDI3Jxo0bdfzxx2vs2LE688wz1bp1a23fvl0DBw7Ul19+qYsvvlgdO3bU9OnTdfbZZ6u4uFiXXXZZ1K8zatQoffrpp7rkkktUUFCgdevWad68efrqq6+q/0xi2rRpGj9+vIYNG6ZJkyaprKxMDzzwgI4++mh9+OGHNf6cYteuXRo2bJiOPvpo3XHHHVx1wD6Xn5+vt956S4sXL1avXr28j3399dc1Y8YMXXjhhcrIyNC9996rUaNG6auvvlKLFi0kfftbrL59+1YXijk5OZo7d64mTJigrVu36vLLL5ckbd26Vf/v//0/nX766Zo4caJKSkr0yCOPaNiwYXrnnXd08MEHO9dQWVmpc889V08//bRmzpyp4cOHS/r2yuX111+vU089Veedd57Wr1+vyZMn65hjjtGHH35Yo7B1fV/g/xdBTC666KJINB/fuHHjIpIizZo1i4wcOTJyxx13RD777LNaj5s6dWpEUmTIkCGRqqqq6vgVV1wRiY+PjxQXF1fHBgwYEBkwYED1vxcsWBCRFOnUqVOkrKysxvNOnz49IimyYMGCPX+TQD1wHVsDBgyISIo8+OCDNeJ33313RFLkr3/9a3WsoqIictRRR0XS09MjW7dujUQi/3ds/HD/LywsjEiKTJ06NRKJRCKbN2+OSIrcfvvt5vpKSkoi2dnZkYkTJ9aIr127NpKVlVUjPn78+IikyDXXXLPH7x+oay+99FIkPj4+Eh8fHznqqKMiv/nNbyIvvvhipKKiosbjJEUSExMjX375ZXVs0aJFEUmRyZMnV8cmTJgQadOmTWTDhg018seOHRvJysqqPv/s2rUrUl5eXuMxmzdvjrRu3Tpy7rnnVse+Ow5vv/32yM6dOyOnnXZaJCUlJfLiiy9WP2bFihWR+Pj4yB//+Mcaz/fJJ59EEhISasSt7wt8i1/17iNTp07Vfffdp44dO2rmzJm6+uqr1b17dx177LH6+uuvaz3+/PPPV1xcXPW/+/fvr8rKSq1cuXK3rzV+/HilpKTU6fqB+paUlKRzzjmnRmzOnDnKzc3V6aefXh1r2rSpLr30UpWWlurVV1+N6jVSUlKUmJioV155RZs3b3Y+Zt68eSouLtbpp5+uDRs2VP8vPj5eRx55pPPXWL/61a+iWgdQl4477ji99dZbGjFihBYtWqTbbrtNw4YNU15enmbNmlXjsUOGDFHnzp2r/927d29lZmZq+fLlkr79Feyzzz6rk046SZFIpMYxMGzYMG3ZskUffPCBJCk+Pr76b8yrqqq0adMm7dq1S3369Kl+zPdVVFRozJgxev755zVnzhwNHTq0etuMGTNUVVWlU089tcZr5ubm6oADDqh13Lm+L/AtftVbh0pLS1VaWlr97/j4eOXk5EiSmjRpoosuukgXXXSRNm7cqDfeeEMPPvig5s6dq7Fjx+o///lPjefq0KFDjX83a9ZMksyT0fd17Njxx74VoMHJy8ur1ai0cuVKHXDAAWrSpOZ/w373Zwx78h9K35eUlKRJkybpqquuUuvWrdW3b1+deOKJGjdunHJzcyVJS5culSQNHjzY+RyZmZk1/p2QkKB27dpFtQ6grh1++OGaMWOGKioqtGjRIs2cOVN33XWXRo8erY8++kg9evSQVPvcI317/vnu3LN+/XoVFxfr4YcfrtWs+J3vN4w8/vjj+vOf/6zPP/9cO3furI67zlO33nqrSktLNXfu3FozapcuXapIJKIDDjjA+ZpNmzat8W/X9wW+ReFXh+644w7ddNNN1f/Oz893zs1r0aKFRowYoREjRmjgwIF69dVXtXLlyuq/BZS+LRpdIpHIbtfB1T7sj37Mfv39q+ffV1lZWSt2+eWX66STTtI///lPvfjii7r++ut16623av78+TrkkENUVVUl6du/8/uuGPy+hISaX6tJSUm1ClOgviQmJurwww/X4Ycfrq5du+qcc87R9OnT9bvf/U7S7s893+3/Z555psaPH+98bO/evSV924hx9tln65RTTtGvf/1rtWrVSvHx8br11lu1bNmyWnnDhg3TCy+8oNtuu00DBw5UcnJy9baqqirFxcVp7ty5zjWmp6fX+DfnQRuFXx0aN26cjj766Op/78mO16dPH7366qtas2ZNjcKvrlknPqAxy8/P18cff6yqqqoaxdXnn39evV36vyvmxcXFNfKtK4KdO3fWVVddpauuukpLly7VwQcfrD//+c/661//Wv1rsFatWmnIkCF1/ZaAfaZPnz6SpDVr1uxxTk5OjjIyMlRZWbnb/f+ZZ55Rp06dNGPGjBrnoO+KzB/q27evfvnLX+rEE0/UmDFjNHPmzOr/kOrcubMikYg6duyorl277vF6URv/GVqHOnXqpCFDhlT/77vxLWvXrtWSJUtqPb6iokL//ve/1aRJE3Xp0mWvru27uWE/PPEBjdkJJ5ygtWvX1uh237VrlyZPnqz09HQNGDBA0rcFYHx8vF577bUa+VOmTKnx77KyMu3YsaNGrHPnzsrIyFB5ebmkb69KZGZm6pZbbqnxq6vvxDqXE9hbFixY4Pxt0Zw5cyRJ3bp12+Pnio+P16hRo/Tss89q8eLFtbZ/f///7src91/77bff1ltvvWU+/5AhQ/TUU0/phRde0FlnnVV9hfHnP/+54uPjddNNN9V6L5FIRBs3btzj9xA6rvjtA6tXr9YRRxyhwYMH69hjj1Vubq7WrVunv//971q0aJEuv/xytWzZcq+u4eCDD1Z8fLwmTZqkLVu2KCkpSYMHD3bOEAQai/PPP18PPfSQzj77bL3//vsqKCjQM888ozfeeEN33323MjIyJElZWVkaM2aMJk+erLi4OHXu3FnPP/98reG1//3vf3Xsscfq1FNPVY8ePZSQkKCZM2eqqKhIY8eOlfTt3/A98MADOuuss3TooYdq7NixysnJ0VdffaXZs2erX79+uu+++/b5ZwFYLrnkEpWVlWnkyJE68MADVVFRoTfffFNPP/20CgoKom6C+NOf/qQFCxboyCOP1MSJE9WjRw9t2rRJH3zwgV5++WVt2rRJknTiiSdqxowZGjlypIYPH67CwkI9+OCD6tGjR42/h/+hU045RVOnTtW4ceOUmZmphx56SJ07d9Yf/vAHXXvttVqxYoVOOeUUZWRkqLCwUDNnztT555+vq6+++kd9TqGg8NsHunXrprvvvltz5szRlClTVFRUpOTkZPXq1Ut/+ctfNGHChL2+htzcXD344IO69dZbNWHCBFVWVmrBggUUfmjUUlJS9Morr+iaa67R448/rq1bt6pbt26aOnWqzj777BqPnTx5snbu3KkHH3xQSUlJOvXUU3X77bfXmGvWvn17nX766fr3v/+tadOmKSEhQQceeKD+8Y9/aNSoUdWP+8UvfqG2bdvqT3/6k26//XaVl5crLy9P/fv3p5MQDc4dd9yh6dOna86cOXr44YdVUVGhDh066MILL9Rvf/tb52Bnn9atW+udd97R73//e82YMUNTpkxRixYt1LNnT02aNKn6cWeffbbWrl2rhx56SC+++KJ69Oihv/71r5o+ffpubyBw5plnqqSkRBdeeKEyMzN1++2365prrlHXrl111113Vf89ffv27TV06FCNGDEi2o8lWHGRPekWAAAAQKPH3/gBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAo/AAAAAJB4QcAABCIPR7gXN/3erVuHC39302jf6iuRxQWFBQ441dccYWZs337dmf8gw8+MHO2bNnijPvu5XvxxRc74//617/MnOuvv97cVpd+eNP671RWVpo5+2q8ZEMcY1nfx9q+8vHHH5vbrH3D9/PatWtX1DmLFi1yxn3HmvU61veQ9O3AW5e+ffuaOb7na4w41oB9Y3fHGlf8AAAAAkHhBwAAEAgKPwAAgEBQ+AEAAASCwg8AACAQcZE9bLXaV91P1uvE0hHmW3P//v2d8ZEjR5o5Vlev73Xat2/vjHfo0MHMSUxMdMYzMzPNnNWrVzvj8+fPN3OaNHHX/b6c559/3hlfv369mRMLa2113elIp+He16VLF2f8008/NXO+/PJLZzw1NdXMsfaNNm3amDkrVqwwt1msn09ZWZmZYx27V111lZkza9as6BbWwHGsAfsGXb0AAACQROEHAAAQDAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIFIqO8F/JDVhpybm2vmnH/++c547969zZyKioroFiZ7ZIpvxIg1sqJly5ZmjjX6YdOmTWbO2rVrnfH09HQzp1WrVs74KaecYuaMHTvWGV+5cqWZ8+GHHzrjDz30kJljfaa+8QsNcVwEpI4dOzrjpaWlZs6GDRuc8aSkpKhfv7Kysk5fx9oHd+3aZeZs377dGW/btq2ZAwB7A1f8AAAAAkHhBwAAEAgKPwAAgEBQ+AEAAASCwg8AACAQDa6rNyMjwxm/6667zJzk5GRnvKSkxMyxOv3i4+PNHKvbtkkTu35u3ry5M75582Yz5+uvv3bGfZ2GVuekr9O1vLzcGbe6lyUpMTHRGU9JSTFzjj/+eGe8R48eZs4ll1zijNO52/h0797dGd+xY4eZYx2f1j7ry/Ed02VlZVGvzfoe2LZtm5mTmprqjPft29fMefDBB81tABArrvgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAo/AAAAALR4Ma5XH/99c54y5YtzZy1a9c64wkJ9tvzjXiwVFVVRZ1jycnJiTrHdxP4iooKZ9w3/sT6DKyRLb7n862tqKjIGT/ggAPMHGsEzNy5c80cNEz9+/d3xn37THp6ujPu2zetESy+Y90as7J9+3Yzp7i42BnPzs42c6yxMdbrA8DewhU/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAhEg+vqtToA161bZ+YkJyc7476O1p07d0a3sDrm62iMpXM2Li4u6jUkJSU5402a2P89YN2IvrKy0syxuip9N7U/+uijnXG6ehsfa3/asmWLmWN1zqakpJg51vdA06ZNzZzc3Fxn3Nc9bK1t48aNZk7z5s2dcV/3MADsDVzxAwAACASFHwAAQCAo/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEol7GuXTr1s3clpWV5YyXlJSYOQkJ7rfhGxdhjTKxbqa+L1njT3wjW6zxNBkZGWbOzJkznfGCggIzp3v37s54UVGRmdO+fXtzm6VHjx5R56Bh2rx5szPuG83SqlUrZ7xZs2Z1sqbvWKNefN831jHlG7dkjTtavXq1Z3UAUPe44gcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgaiXrt7evXub22LpAOzQoYMz/sUXX5g5Vgee1VFb13wdur5tFmvdVjehJPXv398Z993U3uqUtm52L9k3qF+zZo2Zk5+f74y3bNnSzNmwYYO5DfXH+pn59nOrq3bjxo1mzkEHHeSM+7pt161b54zn5eWZOd98840zXlFRYeZYncDWdAGgvjVpYl8Xqqqq2ocrqc039eHnP/+5M/6HP/zBzInlnBuJRKLOycnJccZ9565YXmd3uOIHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAhEvYxz6dixo7nNGr2wY8cOM8calWCNBJGkr776yhn3jXOx2tt97dZW27tvzEos7dtWO7qv7T4rKyvqHGuci++ztn52vp/pqlWrnPHDDz/czJk7d665DfXHGsGSlpZm5lgjgDZt2mTmWPvmm2++aeZYI1i6d+9u5hQXFzvjCQn212l6eroz7htpBNSVWMaVxOLkk082t7Vr184Z951z33//fWf8jDPOMHPatGljbrNY53bfedqSnZ1tbjvmmGOccd/oucWLFzvjvnE7u8MVPwAAgEBQ+AEAAASCwg8AACAQFH4AAACBoPADAAAIRL109fo6QFNSUqKKS3YH6Pz5882ck046yRm3bsAuSU2bNnXGfZ0/Voesr3PW6myOpYvH12lYWlrqjPu6rKyb199www1mzrXXXuuMWzesluz3evTRR5s5dPU2TNYx5ds3y8vLnXFfJ3jr1q2d8Q8++MDMOfTQQ53xxMREM8eyc+dOc5v1fNZ3FxCLuuze9Z2jrO/hK664wsyxunp95/bXXnvNGbc6+CX7u8Mnlu7dZ5991hnv0qWLmTN79mxn3NfVuzdwxQ8AACAQFH4AAACBoPADAAAIBIUfAABAICj8AAAAAkHhBwAAEIh6Gefym9/8xtx27LHHOuMXXXSRmfPWW28543PmzDFzLrzwQme8sLDQzPGNn7BEIpGo4pI9ysTXXm+tzdfen5aW5oxv27bNzCkoKHDGfTebLykpccZ94y9mzZrljD/99NNmDhoma3/yHU++fd2Snp7ujPvG/BQXFzvj48aNM3OscUu+kRDW8W4dG0AsfOeVunT88cc74xUVFWZOWVmZM/7cc8+ZOQcddJAz3qFDBzPnkEMOccZ9o2buuusuZ3zQoEFmTu/evZ1x33fXkUce6Yxfd911Zo4llu/I73DFDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgKPwAAAACUS9dvb6uUaub04r7jBw50txmdZT6bs5udcj6OqmsHKtz17c2q5tQsjt8mjZtauZYfN2WVteW72bzo0ePjnoN2H9s37496hzruPHtz9Yx8N///tfMiWVtsXTqW+v++uuvo359hME3kaEuWfvtiSeeaOacf/75zvjSpUvNnJUrVzrj7dq1M3Oys7Od8Q0bNpg5r776qjP+hz/8wcy5+OKLnfEWLVqYOdYEkKSkJDMnJSXF3LYvccUPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAo/AAAAAJB4QcAABCIehnnEh8fb26zRjLEcvPp5ORkc9vatWudcd/arJuw+3KslnxrZIvkH6discbD+NZmjWbxvb71OqmpqWaOdXNuH2sN1s9A2nc3KEd01q1bF3WO9bP0jUooKSmJ+nVWr14ddY4llmPt448/rrPXR/TqckSXLyeW72frXOj7DozF5MmTnfGhQ4eaOZ9//rkzvnHjRjPHOnYPOeQQM8cat+TL+eCDD5zxRx991Mw57LDDnHHfCDVrm/Vzk6T8/HxnPCcnx8xZv369M+4bCbc7XPEDAAAIBIUfAABAICj8AAAAAkHhBwAAEAgKPwAAgEDUS1dvLF1JvhtWx9LNGUuO1YHley7rvcbSkRNL97Dvdawbx/u6klatWhX12iy+HF83FRoXq4M+MTHRzLGOG2uflaTy8vLoFqbYvgesY8r3HbVt27aoXwd7Xyzfm7Gcv6zvVN93rSUtLc3cZnW7XnfddWaO9V4fe+wxM2f06NHOeEFBgZljdehaHe+SfR549dVXzZzFixc7476JAB9++KEz3rp1azOnd+/ezvjmzZvNnB07djjj/fr1M3P++c9/OuM/ZooFV/wAAAACQeEHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGol3Eu+0osN9r2tfFb23bu3Bn1GhIS7I/eep1YbgLuez/W8/lGDFgjWGIZT1PXIxPQMC1YsMAZLy0tNXOs48N3rMUyUqgu+V5/69at+3Al2FOxjFOJRbt27ZzxLl26mDldu3Z1xtu0aWPmdO/e3RlfsWKFmVNYWOiMH3PMMWaOde7473//a+YUFxdH9fqStHHjRmfc93Pr1KmTM56ammrmLFq0yBn/+OOPzZzly5c740cddZSZY9mwYUPUOYxzAQAAwG5R+AEAAASCwg8AACAQFH4AAACBoPADAAAIxH7d1RsL343WY+mcjYXVseRbm9VRGEvXmq9bKJYuZcDl7bffNrdZ3Y7l5eVmjnVD97pmdZz7unp9XZVoeIYPH25us7ptO3ToYOZkZWU545s3bzZzVq5c6Yz79qWKigpnfPTo0WZOdna2M+471qzu+vfff9/Msbr409PTzZw+ffo440OHDjVzkpOTnXHfOWrs2LHO+KuvvmrmfPbZZ8647+djfdZXXnmlmfP666+b22LFFT8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCCYwREFa5xK06ZNzRxrnIpvZEosOdbafONcrDE01rgKH99nALhYNzmX7JEZvpEM27Zt+9Fr2hOxjG9as2bNXlgJfqxrrrnGGbf2P0lav369M+4bf/Luu+8649b4Fckef+Lb/7Zv3+6Mv/baa2ZOfn6+M37AAQeYOdYajj32WDPHOj59Y5hat27tjL/yyitmzv333++Mv/HGG2bOKaec4ozfcsstZs5//vMfZ3zOnDlmzssvv+yMX3LJJWaONVpo9uzZZs7ucMUPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAo/AAAAAJBV28diKXb1teZZXUuWjfG9r1OLN22vpvNW2vzdVtarDUjDCUlJeY2ax+0bsAu2Z2Tdc06dn3HdFFR0d5aDn6Eb775xhnPy8szczp06OCM+743mzVr5oz7OmeXLl3qjC9evNjM6d27tzPepUsXM8fqLPZ1tM6aNcsZb9++vZlz4IEHOuNZWVlmzquvvuqMr1ixwsyx+L47pk2bFlU81texOpgff/xxM6dly5ZRr2F3uOIHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAhEsONcfCNYohXLWBJf639lZaUzXtdjY6qqqqJ6LskeDxPLjesRtszMTHObta/79k3fzd6jFcv3g2900pdffvljloO95IknnogqLkk9evRwxlu3bm3mFBQUOONvvPGGmbN161Zn3DfK5JlnnnHG09LSzJxNmzZFFZekxMREZ9y3n3/xxRfOuO9Ys17HN+Jk27Ztzrjv+LS2+UaoWWNbrPOqJGVkZDjjvs+grKzMGU9NTTVzdoezNQAAQCAo/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEItiu3li6YKN9Lh9f54+1Bl8ncCzvx8rxdRhZHcfp6elmjqUuO6vR+GzZssXcFsu+2a5dux+9pt29vmQfAzt27DBz1qxZ86PXhH3H9725ZMmSqOJ1zXcesLp3N27caOZYnaYHHnigmWN10Ps6660uWJ/S0tKon8tat/Vcvm2+yQNW17PvOyohwV1y+c6fbdu2dcZ/zHcKV/wAAAACQeEHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIEIdpyL1RLva5W3WrF9o1l82yzWKAHfiIlYRqNYzxfLKIvs7OyoXx9h++STT8xtZ5xxhjPu2zdjGSlk8Y2lsI7P8vJyM2fDhg0/ek3Yd2L53vbtm02bNnXGExMTzRzruzaW84DvdbZt2+aMW+NKfHxri+UztfjO09ZnYP0MJGn79u3OeG5urplTVFTkjPu+h/Lz853xrVu3mjmLFi1yxr/55hszZ3e44gcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgdivu3p9na6+7iOL1Unke6667ND1vR9rm+9m41aXlW9t1mdg3egbsKxbty7qHF9n4L7q6rWUlZWZ277++usfsxw0Ar7v54qKiqjijVUs0yViYXU8S9KyZcvq7HXq+rj94osv6vT5YsUVPwAAgEBQ+AEAAASCwg8AACAQFH4AAACBoPADAAAIBIUfAABAIBrNOJdYxp/EclPohAT7I7HW4BuZYq1tX41z8d3M2lq3LycpKckZb9GihZljiWWkDvYfy5cvN7dZYy58x1pycvKPXtOevI51rJWUlJg5xcXFP3ZJAFAnuOIHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIFoNF29sfB1pzZt2tQZ93WaWtti6bb1dQ9b3ci+HOu9+m5mbXUuWp+NJJWXlzvj27ZtM3MAl9WrV5vbdu3aFfXzxZJj2b59u7nNOtY2bdpUZ68PAHsLV/wAAAACQeEHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIFoNONcfKNMrBu6W2NRfHwjYKzxJ7617dy5M+o1JCYmRp0Ty3NZIytiGTUTyygN3xgc7P98P39rbFBqaqqZ4xvFFK3PPvvM3Na6dWtnvKSkpM5eHwD2Fq74AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgGk1Xbyxi6Rr1dadaXYO+7mHr+Xzdw9br+NZWWVnpjPu6IK01WM8lSSkpKc641ekIxGLNmjXOeK9evcwca9+MhdXB77N06dKoc3ydyHS9A9gbuOIHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAjEfj3OJTk52dxmjX5o2bKlmWONXkhKSjJzrBu3+8Y4WGNWfONcduzY4YxnZGSYOWVlZVG9vu91vv76azPH4hsbg7Bt3brVGfftz9nZ2XX2+llZWea2iooKZ7yoqKjOXh8A9hau+AEAAASCwg8AACAQFH4AAACBoPADAAAIBIUfAABAIBpNV6+vo9Uybdo0c9uSJUuc8e7du5s5BQUFznjHjh3NHKvTMCHB/uirqqqccd9nYHUPr1y50sz55ptvnPHCwkIz56WXXnLGly9fbuZYrPeJ/YvVwR6JRMycFStWOOO+7vG67Kq1utclqWnTps74li1bon4d32cAAHsDV/wAAAACQeEHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGIizBPAAAAIAhc8QMAAAgEhR8AAEAgKPwAAAACQeEHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgKPwAAAACQeEHAAAQCAo/AACAQFD4AYDhscceU1xcnFasWBF17tlnn62CgoI6XxMA/BgUfnvBsmXLdMEFF6hTp05KTk5WZmam+vXrp3vuuUfbt2/fK6/55JNP6u67794rzw3sS5988olGjx6t/Px8JScnKy8vT8cdd5wmT55c30sDgsV5bf+RUN8L2N/Mnj1bY8aMUVJSksaNG6devXqpoqJCr7/+un7961/r008/1cMPP1znr/vkk09q8eLFuvzyy+v8uYF95c0339SgQYPUoUMHTZw4Ubm5uVq1apUWLlyoe+65R5dcckl9LxEIDue1/QuFXx0qLCzU2LFjlZ+fr/nz56tNmzbV2y666CJ9+eWXmj17dj2uEGjY/vjHPyorK0vvvvuusrOza2xbt25d/SwKCBjntf0Pv+qtQ7fddptKS0v1yCOP1Dg4vtOlSxdddtllkqRdu3bp5ptvVufOnZWUlKSCggJdd911Ki8vr5Hzr3/9S8OHD1fbtm2VlJSkzp076+abb1ZlZWX1YwYOHKjZs2dr5cqViouLU1xcHH9bhEZp2bJl6tmzZ62iT5JatWpV/f+nTp2qwYMHq1WrVkpKSlKPHj30wAMP1MopKCjQiSeeqNdff11HHHGEkpOT1alTJz3xxBO1Hvvpp59q8ODBSklJUbt27fSHP/xBVVVVtR63J8cksL/gvLb/4YpfHXruuefUqVMn/fSnP93tY8877zw9/vjjGj16tK666iq9/fbbuvXWW/XZZ59p5syZ1Y977LHHlJ6eriuvvFLp6emaP3++brjhBm3dulW33367JOl///d/tWXLFq1evVp33XWXJCk9PX3vvElgL8rPz9dbb72lxYsXq1evXubjHnjgAfXs2VMjRoxQQkKCnnvuOV144YWqqqrSRRddVOOxX375pUaPHq0JEyZo/PjxevTRR3X22WfrsMMOU8+ePSVJa9eu1aBBg7Rr1y5dc801SktL08MPP6yUlJRar70nxySwv+C8th+KoE5s2bIlIily8skn7/axH330UURS5LzzzqsRv/rqqyOSIvPnz6+OlZWV1cq/4IILIqmpqZEdO3ZUx4YPHx7Jz8+Pef1AQ/DSSy9F4uPjI/Hx8ZGjjjoq8pvf/Cby4osvRioqKmo8znVcDBs2LNKpU6casfz8/IikyGuvvVYdW7duXSQpKSly1VVXVccuv/zyiKTI22+/XeNxWVlZEUmRwsJC72u7jsnx48dzTKJR47y2f+JXvXVk69atkqSMjIzdPnbOnDmSpCuvvLJG/KqrrpKkGn8v8f0rDiUlJdqwYYP69++vsrIyff755z963UBDctxxx+mtt97SiBEjtGjRIt12220aNmyY8vLyNGvWrOrHff+42LJlizZs2KABAwZo+fLl2rJlS43n7NGjh/r371/975ycHHXr1k3Lly+vjs2ZM0d9+/bVEUccUeNxZ5xxRq01ckwiFJzX9k8UfnUkMzNT0rc78e6sXLlSTZo0UZcuXWrEc3NzlZ2drZUrV1bHPv30U40cOVJZWVnKzMxUTk6OzjzzTEmqdYID9geHH364ZsyYoc2bN+udd97Rtddeq5KSEo0ePVpLliyRJL3xxhsaMmSI0tLSlJ2drZycHF133XWSah8XHTp0qPUazZo10+bNm6v/vXLlSh1wwAG1HtetW7daMY5JhILz2v6Jv/GrI5mZmWrbtq0WL168xzlxcXHe7cXFxRowYIAyMzP1+9//Xp07d1ZycrI++OAD/c///I/zD8+B/UViYqIOP/xwHX744eratavOOeccTZ8+XWeeeaaOPfZYHXjggbrzzjvVvn17JSYmas6cObrrrrtqHRfx8fHO549EIlGviWMSIeG8tn+i8KtDJ554oh5++GG99dZbOuqoo8zH5efnq6qqSkuXLlX37t2r40VFRSouLlZ+fr4k6ZVXXtHGjRs1Y8YMHXPMMdWPKywsrPWcuzvYgMasT58+kqQ1a9boueeeU3l5uWbNmlXjat6CBQtifv78/HwtXbq0VvyLL76o8e9ojklgf8B5bf/Dr3rr0G9+8xulpaXpvPPOU1FRUa3ty5Yt0z333KMTTjhBkmpNJL/zzjslScOHD5f0f1cqvn9loqKiQlOmTKn13GlpaVwiR6O3YMEC55W47/5+qFu3bs7jYsuWLZo6dWrMr3vCCSdo4cKFeuedd6pj69ev19/+9rcaj4vmmAT2B5zX9j9c8atDnTt31pNPPqnTTjtN3bt3rzHh/M0339T06dN19tln67LLLtP48eP18MMPV1/2fuedd/T444/rlFNO0aBBgyRJP/3pT9WsWTONHz9el156qeLi4jRt2jTnifGwww7T008/rSuvvFKHH3640tPTddJJJ+3rjwD4US655BKVlZVp5MiROvDAA6uPnaeffloFBQU655xzVFRUpMTERJ100km64IILVFpaqr/85S9q1aqV1qxZE9Pr/uY3v9G0adP0s5/9TJdddln1OJf8/Hx9/PHH1Y+L5pgE9gec1/ZD9dZPvB/773//G5k4cWKkoKAgkpiYGMnIyIj069cvMnny5OpW9Z07d0ZuuummSMeOHSNNmzaNtG/fPnLttdfWaGWPRCKRN954I9K3b99ISkpKpG3bttXjLSRFFixYUP240tLSyC9+8YtIdnZ2RBIt8GiU5s6dGzn33HMjBx54YCQ9PT2SmJgY6dKlS+SSSy6JFBUVVT9u1qxZkd69e0eSk5MjBQUFkUmTJkUeffTRWqNX8vPzI8OHD6/1OgMGDIgMGDCgRuzjjz+ODBgwIJKcnBzJy8uL3HzzzZFHHnmk1nPu6THJOBfsTziv7T/iIhH+UxUAACAE/I0fAABAICj8AAAAAkHhBwAAEAgKPwAAgEBQ+AEAAASCwg8AACAQFH4AAACB2OM7d4Ryz7wmTexaOCMjwxn3jUIsKSlxxlNSUsyczMxMZ3zHjh1mjrXNl7OvWPtOQxgh2RDW8EOhHGsIC8da3fCtuS4/4wEDBpjbjj/+eGe8efPmZk5CgrvcWL16tZnz3a0af2jhwoVmDna/H3DFDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAg4iJ7+NegDfmPYK2GDN+aKysrnfGWLVuaOZs2bXLG27VrZ+Z06NDBGf/888/NnNzcXGd87dq1Zs7WrVud8cTERDOntLTUGY+PjzdzrM96586dZk5Dxh+cA/sGx1r9ufXWW81tvXv3dsa3bNli5mzbts0Z9507KioqnPG0tDQzJy8vzxlv2rSpmfPoo48641OnTjVzGnIDYixo7gAAAIAkCj8AAIBgUPgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACMR+Mc4lFtZYkqysLDPHuu9uVVWVmfPzn//cGb/gggvMnEmTJjnjL7/8splj3SPRaqGX7HEuIWmI7fr727EGSBxr0bJGlvhGZw0dOtQZt85DkvT6668749bIFklq0aKFM26NL5OkwsJCZ7y4uNjM6dWrlzPeqVMnM6dnz57O+BFHHGHmWKw6QfKf9+sb41wAAAAgicIPAAAgGBR+AAAAgaDwAwAACASFHwAAQCAS6nsBe9Mtt9xibuvTp48zbnVFSXaXla+755lnnnHGf/KTn5g5mzZtMrdZNm/e7Iz7unt+9rOfOeNXX321mWN1Iy9btsyzOgBANHwdpZYePXo44zt27DBzrGkVvg5da/pFfHy8mZOfn++M+7qUrW27du0yc7Kzs53xww47zMx5//33nfGEBLtE8k3MaOi44gcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACMR+Mc7lgQcecMZPPvlkM8d3Y2iL1Vo+aNAgM8cajWLd5FqSfvnLXzrj69evN3P+8Y9/OOM33nijmdOtWzdnvKCgwMx58sknnfF+/fqZOVbrve8G6Q3xhu4A0JB16tTJGT/++OPNnFWrVjnjixcvNnOs0WYffPCBmdO2bVtnPDk52cz55JNPnPErr7zSzLFGyowdO9bMsca5+EbNNGZc8QMAAAgEhR8AAEAgKPwAAAACQeEHAAAQCAo/AACAQMRF9rB90teBWd/Wrl3rjC9dutTM8d182WJ1C7Vp08bMKS0tdcYfeeQRM2fMmDHOuO+m3VY3V3l5uZmzceNGZ3z16tVRv46ve/jvf/+7M+57P1VVVea2utQQu4cb8rEGxIpjbe+74YYbnPFzzjnHzNmyZYszvm3bNjPHOuf6pmWce+65zvjzzz9v5qSkpDjjOTk5Zk5iYqIz/uqrr5o51iSNxmp3xxpX/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgYh+pkk96dq1q7nNGpniGwlSUlLijGdnZ5s5lZWVzvh7771n5jRv3twZ//zzz80ca2xMYWGhmbNy5Upn3GqHl6Rdu3Y549aaJXv8wZFHHmnmWONc9tXIFqCh2lcjjYYPH25uO+6445xx3/iLmTNn/ug1ITZ9+vQxtx1zzDHO+NatW80c63zjO+cmJyc749Z5VZI++ugjZ9x3zrXWZo1skexj6qCDDjJzevXq5YwvXrzYzGnMuOIHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIFoNF29xx57rLktNTXVGfd1MqWlpTnjmzZtMnOsTiKrw0myO/OsTjpJKisrc8YzMjLMHKuTydcZaHX8JiUlmTnl5eXO+KGHHmrmACHwdRpWVFQ447F07v7qV78yt/32t791xt966y0z54033nDGhw4dauYsXLjQGV+zZo2Zg+g0a9bMGX/kkUfMnB07djjj1uQLyf6+/+qrr8wc6/xp7eeSPUXCei5fjq972DpHJSTY5c6zzz7rjF966aVmzosvvmhua+i44gcAABAICj8AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACESjGecyYMAAc1tlZaUznp6ebuZYrd2RSMTMiWVkitVG/5Of/MTMsezcudPc1rRpU2fc115vtb372usthxxySNQ5QENlHeuSfbz7jjXLxIkTzW0jRoxwxq0xTJJ0yy23OON9+/Y1c1q2bOmM33///WYOY1v2vvPPP98Z9+2b1ggz38gxa3yYb/yJtQZfjjVqxrc/b9u2zRnPzs42c6zn8x2f1trGjRtn5jDOBQAAAA0ehR8AAEAgKPwAAAACQeEHAAAQCAo/AACAQDSart7OnTub26xO3KysLDMnlpujx8Lqctq+fbuZY93s3deZFRcX54z7OoGt50tNTTVzrE5gX8fUcccd54zPmzfPzMH+L5bO2X31OrG8/qBBg8xtV155pTPu+x5Yv369M251+0pSUlKSM/7888+bOZMmTTK3WWKZcIDodOjQwRm3zg++bSUlJWaONcXB16FbXFzsjH/44YdmjnWO8k2R6NatmzPum77Rtm1bZ9zqXpbs9+P7rBszrvgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAo/AAAAALRaMa5dOrUydy2ZcsWZ9x3I2fr5s+7du0yc6x29FatWkWds3nzZjPHGqfiW1ssLfnW55OXl2fmvPDCC8647/2MGTPGGWecS9hiGf2xr0aztG/f3tz2pz/9yRlv06aNmbN06VJn/PPPPzdzrO+Vzz77zMyZMWOGM37XXXeZORbrZveSPdYJdSczM9MZr6ysNHOsEWbr1q0zczIyMpxx38/Y2tdLS0vNHGsNBx54oJljjQn76quvzBzr/OkbbbZhwwZnPD093cxpzLjiBwAAEAgKPwAAgEBQ+AEAAASCwg8AACAQFH4AAACBaHBdvfHx8c64r0O3sLDQGT/ooIOizrFuci75u6ks1vM1b97czLE6dH0dU1bHktW9LNldW+3atTNzrJtZ+15n1KhRzvj5559v5qD++DrBfZ3l0dpXHbpW16Jkd+gef/zxZs78+fOdcV+n4U9/+lNnvG/fvmbOzJkznfH+/fubOXVp+/bt++R14NaiRQtn3Ndta52jmjZtauZYx5TVHSvZkzSsNUv294pvbdb50/cdtWLFCmc8MTHRzLH2dd93h7W2xtDxzhU/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgGtw4F+vm6JFIxMyx2tF9beLWyJS4uDgzZ8eOHVGvzRpP43sda8yFddNuSVq7dq0z7mtHt7b5RmZYo17mzp1r5lg3DsfeF8tolroc2eITy2gW3w3dr7jiCmf85JNPNnM+/PBDZ/zPf/6zmWOt2zfK4ssvv3TGP/30UzPHGufiY31/+o5Ba7SUb/xFSUmJM/722297Vocfatu2rbnN+vx9x43vnGexzjfp6elmTllZmTPuG3lmjT/xjSmz1uDbN62RY77zpzWCJSUlxcyxjvdvvvnGzGkouOIHAAAQCAo/AACAQFD4AQAABILCDwAAIBAUfgAAAIFocF29HTt2jDonlk4mqws2Ly/PzKmoqIj6dXzduxare9jXodmsWTNnPDk52cyxPjdf11ibNm2ccd/P7Z133jG3Ye/ydeha+9PAgQPNnBNPPNEZHzdunJlj7WeLFy82c/r27euM+26A/uabbzrjTz75pJmzadMmZ7x169ZmjmXz5s3mNqsL8rzzzjNz/vCHPzjjvu+71atXR52zdetWZ3zNmjVmjtXt6PuZorZWrVqZ26yJEL4pEtbP2dedanW7btu2zcyxOnG3b99u5ljfRb7zjbVvWuc7yZ5WYXWvS3aXuu/8bXVk09ULAACABoPCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgKPwAAAAC0eDGubRr184Z941Ssdq3v/76azPn8ccfd8Z//etfmzlW27l1k2vJbsn3sVrIfTfAtkZzWKNhJHttvhEg1hp8rf85OTnmNuxdN954o7lt1KhRzrhvf/7ggw+c8ddee83MsX7+vjEO1vH51VdfmTnWfpudnW3mWN8dvrVZN4737efW2BhfjvWZWiMuJPuY9uVY79U3Oscae/Xxxx+bOajNGo8l2eNCioqKzBxrnIvv/GkdN8XFxWZOYmKiM+4731mv4xs1ZJ1vfCNtdu7c6Yz7jgFr3I3vdTp37uyMv/fee2ZOQ8EVPwAAgEBQ+AEAAASCwg8AACAQFH4AAACBoPADAAAIRIPr6rVuGO3rMLNy3n77bTOnQ4cOzrjVsSdJ69atc8aTk5PNHKtDN5Ybbfs6pqzuXV+3bVJSUlSvL0ktW7Z0xlNTU82cWG54j7px4oknmttWrFjhjH/xxRdRv86yZcvMbdax69vPrH3Td3z6OnGjfZ0WLVqYOVbXs+8G9dZx8+abb5o51vFudRP6+L4/Lb79wHq+DRs2RP06IfN1nFvnjlgmRVjd65JUWloa9etY57yysjIzxzpurPcp2dMqfOdPi9W563s+q0NYkpo3bx71GhoKrvgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAo/AAAAALR4Ma5WG3nvrZqq0177dq1Zk737t2jW1iMrBETvtET1o2pfTlW671vZIY1GuP22283c371q18546effrqZU1JSYm5DbRMnTnTGL730UjPHGi0wY8YMM2f9+vXOeFpampljjT8pKCiIOsc3/sS6QbzvGLBGP/hGwFg2btxobrOONd9oDmuci/U+Jfum9tb7lOzP1Pc9YD2fNSZLst9PUVGRmYPafD9LaxSXbyyJdexax7pkj1nxrc3aN33Hp7UP+kYNWedCa82S/X3jy7E+08TExKhfpzHgih8AAEAgKPwAAAACQeEHAAAQCAo/AACAQFD4AQAABKLBdfXm5uY6476OHOsGy/n5+WaO1WG0Y8cOM8e6MbWvk8m6AbXv/VjbfDkVFRVR51hdSX369DFzLr74Ymf83nvvNXN8NwhHbUceeaQz7ut+++yzz5zx9u3bmznWz8Xq2JPsLjvf2qzuRF9Xr8W3P1t83YmpqanOuK+j1foMfF1+Vvfu1q1bzZxmzZo549axLtk/O9/Px9rm+6ytnG+++cbMQW1Wd7Rk/yx9P/927do549Y+K9nHh++48T2fxToXWudVyf7usM75kv25lZaWmjmtW7d2xmM51hoDrvgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAo/AAAAALR4Ma5WCMRdu7caeZYN0333ZzdatP2tXxbreW+0Q9WS7xvBIzVqu5rYbc+A1/O8uXLnfGBAweaOR06dHDGV69ebeb06tXL3Ibann32WWf8qKOOMnOs8URt2rQxc6wRIz7W+BHfaBbruPGNP7GOG2v8imTfUD2WG8f7xlVY3x3Wjd4l+71ax61kj/rwjWaxPjffaJZNmzY5477vNWsNvvEXqM13fFrjT2IZt+T7ufiOKYt1XrHWLNlr872+dXz4vm+sHN/3jbU/W8eG5P/ZNXRc8QMAAAgEhR8AAEAgKPwAAAACQeEHAAAQCAo/AACAQDS4rl5fB160fJ15Vsear/MnLS3NGffdZNrqGvR1P1ndgb6bZludk74ca21FRUVmTteuXZ3xwsJCMwfRWbx4sTO+fv16M2fDhg3OuG9/bt68uTNu7ReS1KJFC3ObxeqU93XBWnzdfNZ3h+87xTo+rONJsr8HfFMENm7c6IxbPwPJfq++CQdWx6fvs7Y6Gn3dw75uR+y5rKwsc5v18/ftz9Y237nQ6ob35Vh8UySsc57vmM7IyHDGfV3KVlev73Oz1mB9NpLUrl07c1tDxxU/AACAQFD4AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgGtw4F6uF3HeTcWucinVDeUnasWOHM96+fXvP6tysUQ2S3Q7ua0e3xl/4buhuvY4vp6yszBn3tfFbIx5iGcPjG2njGwuwv1u1apUzftZZZ5k5hxxyiDNeUFBg5ljjYbp3727mWGOQfMeaNfbAl2Px7WfWGCLreJKk5cuXO+PZ2dlRrUvyvx/rc/ONQercubMz7vsMrPEwvrEU1veXb5zLunXrzG3Yc75xPtZoHh/ru9t3HohlbIv1fL7vdGubL8dam2+EmvW5+c4pqampzvjatWvNHF9N0tA13pUDAAAgKhR+AAAAgaDwAwAACASFHwAAQCAo/AAAAALRaLp6fV1JVjed1ekoSbm5uc64r8OoZcuWzniHDh3MHOv5fJ15vjXUZY7Ft7aSkhJnPJYOJ19OLJ1m+zur23d326Ll+7mkpKQ449u2bTNz+vXr54y/8cYbZk6bNm2ccV/nrLUGX0err7u+vi1cuLDOnsv3GcTSPdqYOxobEl93qtWFmpSUFNPzRSuWqQsJCXZJsXPnTmfc+k6R7HORb/+zagVfV2/Tpk2dcWv6hyRlZGSY2xo6jl4AAIBAUPgBAAAEgsIPAAAgEBR+AAAAgaDwAwAACASFHwAAQCAa3DgXa7SAb5yL1d6+YsUKM+cXv/iFM56WlmbmWO3tsYw2sMaiSPbN2X0jTr755htn3HcT8KVLlzrjgwYNMnNuu+02Z3zTpk1mTiw350b98Y3z8Y1tsfjGtljWrFkTdY6lIY9s2Vfq+jPw7SPYc77P0fq+951vYhkxEst5zTpPW2NRpNhGdFnPV15ebuZYtYLvs7ZyfONpGvPIMa74AQAABILCDwAAIBAUfgAAAIGg8AMAAAgEhR8AAEAgGlxXr9Up4+vIsTqMtm/fbua88sorUa1rX1qwYEG9vv4nn3xibrO6emMRy83hAWB/4euctTpNExMTzZwNGzY4477uVGsNsUxd8E3fsNbgex0rx9dRa51XkpOTzZxYJk/4JoA0dFzxAwAACASFHwAAQCAo/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEosGNcykoKHDGy8rKzByrvT0lJSXq1/e118fSWm49XyQSiW5hu8mx2s59bfw7duxwxvPy8syc4uJiZ9z3uVlry8rKMnO2bNlibgOA/YFvpJX1neobbWblpKammjnW8/m+02NhnYt8I2CszyeW8TRNmzY1c6xzq+91KioqzG0NHVf8AAAAAkHhBwAAEAgKPwAAgEBQ+AEAAASCwg8AACAQDa6r97bbbnPG//Wvf5k5SUlJzvijjz4a9ev7Omdj6eLxdfzuC74OMMvy5cvNbaWlpc647wbYhYWFzjiduwBC5vsObNmypTNuTWOQpOzsbGe8devWZk55ebkz7jt3WOc13/nOmu4QyyQN67kkKT093Rn3dTZb5zUf3/M1dFzxAwAACASFHwAAQCAo/AAAAAJB4QcAABAICj8AAIBAUPgBAAAEosGNc/nggw+c8bS0NDMnKyvLGe/WrVvUrx/LDaMbMt94mlhYn0Hbtm3NnLq+2TcA7A/ef/99c1v//v2dcd956Oqrr3bG27RpY+acc845znhxcbGZk5OT44z7xrnEci6yRsoccMABZs7999/vjL/zzjtmzh133OGMW6PiJGn27NnmtoaOMzIAAEAgKPwAAAACQeEHAAAQCAo/AACAQFD4AQAABKLBdfVarE4dyb6Z9axZs6J+nbrugt3fPPPMM854cnKymbNz5869tRwAaLQmT55sbktNTY0qLkkPPPBA1Gu44YYbnPHExEQzp1WrVlG/TnZ2tjO+detWM6eoqMgZLy8vj/r1fY466ihn3DdN5H/+53/qdA37Elf8AAAAAkHhBwAAEAgKPwAAgEBQ+AEAAASCwg8AACAQFH4AAACBiIswvwQAACAIXPEDAAAIBIUfAABAICj8AAAAAkHhBwAAEAgKPwAAgEBQ+AEAAASCwg8AACAQFH4AAACBoPADAAAIxP8H0G4ie11ckQIAAAAASUVORK5CYII=\n"},"metadata":{}}],"source":["labels_map = {\n","    0: \"T-Shirt\",\n","    1: \"Trouser\",\n","    2: \"Pullover\",\n","    3: \"Dress\",\n","    4: \"Coat\",\n","    5: \"Sandal\",\n","    6: \"Shirt\",\n","    7: \"Sneaker\",\n","    8: \"Bag\",\n","    9: \"Ankle Boot\",\n","}\n","figure = plt.figure(figsize=(8, 8))\n","cols, rows = 3, 3\n","for i in range(1, cols * rows + 1):\n","    sample_idx = torch.randint(len(training_data), size=(1,)).item()\n","    img, label = training_data[sample_idx]\n","    figure.add_subplot(rows, cols, i)\n","    plt.title(labels_map[label])\n","    plt.axis(\"off\")\n","    plt.imshow(img.squeeze(), cmap=\"gray\")\n","plt.show()"]},{"cell_type":"markdown","metadata":{"id":"XCooMfqYNIxr"},"source":["###Creating a Custom Dataset for your files\n","\n","A custom Dataset class must implement three functions:\n","`__init__`, `__len__`, and\n","`__getitem__`. Take a look at this implementation; the\n","FashionMNIST images are stored in a directory `img_dir`, and their\n","labels are stored separately in a CSV file `annotations_file`. To create your own custom datasets, you must implement these three methods (`__init__`, `__len__`, and `__getitem__`).\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"QJ-7W5_WNIxs"},"outputs":[],"source":["import os\n","import pandas as pd\n","from torchvision.io import decode_image\n","\n","class CustomImageDataset(Dataset):\n","    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):\n","        self.img_labels = pd.read_csv(annotations_file)\n","        self.img_dir = img_dir\n","        self.transform = transform\n","        self.target_transform = target_transform\n","\n","    def __len__(self):\n","        \"\"\"\n","        Returns the number of samples in our dataset.\n","        \"\"\"\n","        return len(self.img_labels)\n","\n","    def __getitem__(self, idx):\n","        \"\"\"\n","        Loads and returns a sample from the dataset at the given index `idx`.\n","        \"\"\"\n","        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])\n","        image = decode_image(img_path)\n","        label = self.img_labels.iloc[idx, 1]\n","        if self.transform:\n","            image = self.transform(image)\n","        if self.target_transform:\n","            label = self.target_transform(label)\n","        return image, label"]},{"cell_type":"markdown","metadata":{"id":"KGNl4ZoUNIxt"},"source":["###Preparing your data for training with DataLoaders\n","\n","The `Dataset` retrieves our dataset\\'s features and labels one sample at\n","a time. While training a model, we typically want to pass samples in\n","\\\"minibatches\\\", reshuffle the data at every epoch to reduce model\n","overfitting, and use Python\\'s `multiprocessing` to speed up data\n","retrieval.\n","\n","`DataLoader` is an iterable that abstracts this complexity for us in an\n","easy API.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"RDoBwDBGNIxt"},"outputs":[],"source":["from torch.utils.data import DataLoader\n","\n","train_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)\n","test_dataloader = DataLoader(test_data, batch_size=64, shuffle=True)"]},{"cell_type":"markdown","metadata":{"id":"ddqkoXHmNIxt"},"source":["###Iterate through the DataLoader\n","\n","We have loaded that dataset into the `DataLoader` and can iterate\n","through the dataset as needed. Each iteration below returns a batch of\n","`train_features` and `train_labels` (containing `batch_size=64` features\n","and labels respectively). Because we specified `shuffle=True`, after we\n","iterate over all batches the data is shuffled (for finer-grained control\n","over the data loading order, take a look at\n","[Samplers](https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler)).\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"a5K1iyTzNIxt","colab":{"base_uri":"https://localhost:8080/","height":482},"executionInfo":{"status":"ok","timestamp":1772055863058,"user_tz":300,"elapsed":161,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"26788258-08c0-4f43-87e5-529e19ac7c09"},"outputs":[{"output_type":"stream","name":"stdout","text":["Feature batch shape: torch.Size([64, 1, 28, 28])\n","Labels batch shape: torch.Size([64])\n"]},{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAHRNJREFUeJzt3X1slfX5x/HPobQH0PbUUvpwBEoBhUUelqF0jcowNEC3GBFM1PkHGoKBFaPiw8YyReeSbmxxxoXh/lhkZqLOZED0D4xWW7KtYEAZYc6GsmrLSovgek4pttT2+/uDn2c78vi9OadXW96v5JvQc+6r99W7d8+Hc87dqyHnnBMAAANshHUDAIDLEwEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEyOtG/i6/v5+tba2Kjs7W6FQyLodAIAn55w6OzsVjUY1YsS5n+cMugBqbW3VhAkTrNsAAFyilpYWjR8//pz3D7qX4LKzs61bAACkwIUez9MWQBs3btSkSZM0atQolZWV6f3337+oOl52A4Dh4UKP52kJoNdee01r167V+vXr9cEHH2j27NlatGiRjh49mo7dAQCGIpcGc+fOdVVVVYmP+/r6XDQaddXV1ResjcViThKLxWKxhviKxWLnfbxP+TOgU6dOae/evaqoqEjcNmLECFVUVKi+vv6M7Xt6ehSPx5MWAGD4S3kAHTt2TH19fSosLEy6vbCwUG1tbWdsX11drUgkklhcAQcAlwfzq+DWrVunWCyWWC0tLdYtAQAGQMp/Dyg/P18ZGRlqb29Pur29vV1FRUVnbB8OhxUOh1PdBgBgkEv5M6CsrCzNmTNHNTU1idv6+/tVU1Oj8vLyVO8OADBEpWUSwtq1a7V8+XJdf/31mjt3rp577jl1dXXpvvvuS8fuAABDUFoC6M4779Rnn32mJ598Um1tbfrmN7+pHTt2nHFhAgDg8hVyzjnrJv5XPB5XJBKxbgMAcIlisZhycnLOeb/5VXAAgMsTAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATKQ8gJ566imFQqGkNX369FTvBgAwxI1Mxye97rrr9M477/x3JyPTshsAwBCWlmQYOXKkioqK0vGpAQDDRFreAzp48KCi0agmT56se+65R83NzefctqenR/F4PGkBAIa/lAdQWVmZNm/erB07dmjTpk1qamrSzTffrM7OzrNuX11drUgkklgTJkxIdUsAgEEo5Jxz6dxBR0eHSkpK9Oyzz2rFihVn3N/T06Oenp7Ex/F4nBACgGEgFospJyfnnPen/eqA3NxcXXvttWpsbDzr/eFwWOFwON1tAAAGmbT/HtCJEyd06NAhFRcXp3tXAIAhJOUB9Oijj6qurk6ffPKJ/va3v+n2229XRkaG7r777lTvCgAwhKX8JbjDhw/r7rvv1vHjxzVu3DjddNNN2rVrl8aNG5fqXQEAhrC0X4TgKx6PKxKJWLcBALhEF7oIgVlwAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwMRI6wYApE9lZWWgur///e/eNa2trd41GRkZ3jX9/f3eNc4575rhKsgx7+vrS0MnPAMCABghgAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABggmGkwBAxZswY75qFCxcG2tenn37qXRNkGOlgHywaCoW8a4L0N5ADQtM1WDQIngEBAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwwTBSwMCvfvUr75oggzv/85//eNdI0kcffRSoztdADfsMaqD2FWRAaJBjJ0lTpkzxrmlsbAy0rwvhGRAAwAQBBAAw4R1AO3fu1K233qpoNKpQKKRt27Yl3e+c05NPPqni4mKNHj1aFRUVOnjwYKr6BQAME94B1NXVpdmzZ2vjxo1nvX/Dhg16/vnn9cILL2j37t264oortGjRInV3d19yswCA4cP7IoTKykpVVlae9T7nnJ577jn95Cc/0W233SZJeumll1RYWKht27bprrvuurRuAQDDRkrfA2pqalJbW5sqKioSt0UiEZWVlam+vv6sNT09PYrH40kLADD8pTSA2traJEmFhYVJtxcWFibu+7rq6mpFIpHEmjBhQipbAgAMUuZXwa1bt06xWCyxWlparFsCAAyAlAZQUVGRJKm9vT3p9vb29sR9XxcOh5WTk5O0AADDX0oDqLS0VEVFRaqpqUncFo/HtXv3bpWXl6dyVwCAIc77KrgTJ04kjWVoamrSvn37lJeXp4kTJ+qhhx7Sz372M11zzTUqLS3VE088oWg0qiVLlqSybwDAEOcdQHv27NEtt9yS+Hjt2rWSpOXLl2vz5s16/PHH1dXVpfvvv18dHR266aabtGPHDo0aNSp1XQMAhryQG8jJfhchHo8rEolYtzEojBjh/wppkIGVA2mwD58M4kc/+pF3zf/+J+5iTZw40bumo6PDu0YKNox0xYoVgfY1EDIyMgLVBRkSOlAmTZoUqO7TTz/1rhk7dqzX9v39/fr8888Vi8XO+76++VVwAIDLEwEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADAhPefY0CwKdVBpkAP1H4GcuLvQE22zsvLC1T3+9//3rumoKDAu+bo0aPeNd3d3d41QY/33LlzvWv+8Y9/eNdcd9113jVBDOap1pK0cuVK75r58+cH2ldbW5t3zXvvvee1fW9vr956660LbsczIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACZCbqCmQ16keDyuSCRi3cZ5DcchoQNl3Lhx3jVPPPGEd80dd9zhXSNJBw4c8K4pLS31rgkyuDPIj2p+fr53jSSFw2HvmpycHO+a7Oxs75pHHnnEu+bVV1/1rpGkSZMmedc888wz3jVXX321d82oUaO8ayRpypQp3jVz5szx2r6/v1+tra2KxWLnPS94BgQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMDEsBlGGmTY5yD70s3cd999A1ZXUlLiXdPb2+tdU19f710jSZ999pl3TWFhoXfN+PHjvWuCHIcgvUnBBu4GGZ47evRo75qioiLvms8//9y7RpJOnTrlXROPx71rgjwWjRw50rtGkq666irvmqVLl3pt39fXp3379jGMFAAwOBFAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADARbJrdIDTYB4sGGe64evVq75oHH3zQu+bYsWPeNZLU1tbmXbN3717vmiBDOIMMtJWkmTNnetcE+d4GqQkyWLS/v9+7RpK+/PJL75ogX1NnZ6d3TSwW867JzMz0rpGCPa4EOQ6jRo3yrgkyKDWogoICr+0v9meWZ0AAABMEEADAhHcA7dy5U7feequi0ahCoZC2bduWdP+9996rUCiUtBYvXpyqfgEAw4R3AHV1dWn27NnauHHjObdZvHixjhw5klivvPLKJTUJABh+vC9CqKysVGVl5Xm3CYfDgf5qIQDg8pGW94Bqa2tVUFCgadOmafXq1Tp+/Pg5t+3p6VE8Hk9aAIDhL+UBtHjxYr300kuqqanRL37xC9XV1amysvKcfy++urpakUgksSZMmJDqlgAAg1DKfw/orrvuSvx75syZmjVrlqZMmaLa2lotWLDgjO3XrVuntWvXJj6Ox+OEEABcBtJ+GfbkyZOVn5+vxsbGs94fDoeVk5OTtAAAw1/aA+jw4cM6fvy4iouL070rAMAQ4v0S3IkTJ5KezTQ1NWnfvn3Ky8tTXl6enn76aS1btkxFRUU6dOiQHn/8cU2dOlWLFi1KaeMAgKHNO4D27NmjW265JfHxV+/fLF++XJs2bdL+/fv1hz/8QR0dHYpGo1q4cKGeeeYZhcPh1HUNABjyvANo/vz55x3Q99Zbb11SQ0H9byherOuvvz7QvqLR6IDUjB071rvmo48+8q45evSod42kc17ZeD5XXnmld01+fr53zRVXXOFdIwXrL8hQyKysLO+a7u5u75qRI4NdZxRkiGmQwZ1BaoJ8TUEGhErBjkOQwadB+gs6YDXI8fN9LLrYnwlmwQEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATKT8T3KnysqVK70mBq9atcp7H4cPH/aukaSenh7vmiATk48fP+5dE2S6cNC/Qhtkgm+QSbxBaoIcB0k6efKkd01ubq53zahRo7xrRo8e7V0TdGJykEnnJSUl3jVBfpaCTAUPsh9J+vLLL71rMjIyAu3L1yeffBKobtq0ad41vo9Fvb29F7Udz4AAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYCLmgUxvTJB6PKxKJKC8vz2vY5dSpU733NWvWLO8aSbr22mu9a8aPH+9dM1ADK4MOT8zOzvauCfI1BRFkiKQUbGhle3v7gNQ0NDR41/z73//2rpGk5uZm75qOjg7vmlgsNiD76e/v966RpFAoFKjOV5CBu0EGHEvBhg8fO3Ys0L5isdh598czIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACb8J+ANkMzMTK9hpP/617+897F7927vmqB8vpavBBkSGmSoYVZWlneNFGxQY5ChkEHm5QYdRnry5MlAdcNNkPM1MzPTuybIuZebm+tdE2RIrxTsHB+o8zXoOR5kiHBnZ6fX9s65ixqWyjMgAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgbtMNL29nav7ceOHeu9j/Hjx3vXSMEGFAYZ1NjX1+ddE2RAYZDhiVKwgZUDJcj3SAo2zDVIzUAJ+j0KOujSV5CBu0G/t4NZkHMoyOODFOyY+55HF/uYMngfQQAAwxoBBAAw4RVA1dXVuuGGG5Sdna2CggItWbJEDQ0NSdt0d3erqqpKY8eO1ZVXXqlly5Z5v5wGABj+vAKorq5OVVVV2rVrl95++2319vZq4cKF6urqSmzz8MMP64033tDrr7+uuro6tba2aunSpSlvHAAwtIVc0HegJX322WcqKChQXV2d5s2bp1gspnHjxmnLli264447JEkff/yxvvGNb6i+vl7f/va3L/g54/G4IpGIdy9BLkIYM2aMd43ERQhf4SKE4DUDhYsQhoYgfyk46EUI4XDYu6apqclre+ecuru7FYvFlJOTc87tLukRJBaLSZLy8vIkSXv37lVvb68qKioS20yfPl0TJ05UfX39WT9HT0+P4vF40gIADH+BA6i/v18PPfSQbrzxRs2YMUOS1NbWpqysrDP+ZnthYaHa2trO+nmqq6sViUQSa8KECUFbAgAMIYEDqKqqSgcOHNCrr756SQ2sW7dOsVgssVpaWi7p8wEAhoZAL16vWbNGb775pnbu3Jn0y5xFRUU6deqUOjo6kp4Ftbe3q6io6KyfKxwOB3pNEgAwtHk9A3LOac2aNdq6daveffddlZaWJt0/Z84cZWZmqqamJnFbQ0ODmpubVV5enpqOAQDDgtczoKqqKm3ZskXbt29XdnZ24n2dSCSi0aNHKxKJaMWKFVq7dq3y8vKUk5OjBx54QOXl5Rd1BRwA4PLhFUCbNm2SJM2fPz/p9hdffFH33nuvJOnXv/61RowYoWXLlqmnp0eLFi3Sb3/725Q0CwAYPi7p94DSIejvAQEABpe0/h4QAABBEUAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwIRXAFVXV+uGG25Qdna2CgoKtGTJEjU0NCRtM3/+fIVCoaS1atWqlDYNABj6vAKorq5OVVVV2rVrl95++2319vZq4cKF6urqStpu5cqVOnLkSGJt2LAhpU0DAIa+kT4b79ixI+njzZs3q6CgQHv37tW8efMSt48ZM0ZFRUWp6RAAMCxd0ntAsVhMkpSXl5d0+8svv6z8/HzNmDFD69at08mTJ8/5OXp6ehSPx5MWAOAy4ALq6+tz3/ve99yNN96YdPvvfvc7t2PHDrd//373xz/+0V199dXu9ttvP+fnWb9+vZPEYrFYrGG2YrHYeXMkcACtWrXKlZSUuJaWlvNuV1NT4yS5xsbGs97f3d3tYrFYYrW0tJgfNBaLxWJd+rpQAHm9B/SVNWvW6M0339TOnTs1fvz4825bVlYmSWpsbNSUKVPOuD8cDiscDgdpAwAwhHkFkHNODzzwgLZu3ara2lqVlpZesGbfvn2SpOLi4kANAgCGJ68Aqqqq0pYtW7R9+3ZlZ2erra1NkhSJRDR69GgdOnRIW7Zs0Xe/+12NHTtW+/fv18MPP6x58+Zp1qxZafkCAABDlM/7PjrH63wvvviic8655uZmN2/ePJeXl+fC4bCbOnWqe+yxxy74OuD/isVi5q9bslgsFuvS14Ue+0P/HyyDRjweVyQSsW4DAHCJYrGYcnJyznk/s+AAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYGXQA556xbAACkwIUezwddAHV2dlq3AABIgQs9nofcIHvK0d/fr9bWVmVnZysUCiXdF4/HNWHCBLW0tCgnJ8eoQ3sch9M4DqdxHE7jOJw2GI6Dc06dnZ2KRqMaMeLcz3NGDmBPF2XEiBEaP378ebfJycm5rE+wr3AcTuM4nMZxOI3jcJr1cYhEIhfcZtC9BAcAuDwQQAAAE0MqgMLhsNavX69wOGzdiimOw2kch9M4DqdxHE4bSsdh0F2EAAC4PAypZ0AAgOGDAAIAmCCAAAAmCCAAgIkhE0AbN27UpEmTNGrUKJWVlen999+3bmnAPfXUUwqFQklr+vTp1m2l3c6dO3XrrbcqGo0qFApp27ZtSfc75/Tkk0+quLhYo0ePVkVFhQ4ePGjTbBpd6Djce++9Z5wfixcvtmk2Taqrq3XDDTcoOztbBQUFWrJkiRoaGpK26e7uVlVVlcaOHasrr7xSy5YtU3t7u1HH6XExx2H+/PlnnA+rVq0y6vjshkQAvfbaa1q7dq3Wr1+vDz74QLNnz9aiRYt09OhR69YG3HXXXacjR44k1l/+8hfrltKuq6tLs2fP1saNG896/4YNG/T888/rhRde0O7du3XFFVdo0aJF6u7uHuBO0+tCx0GSFi9enHR+vPLKKwPYYfrV1dWpqqpKu3bt0ttvv63e3l4tXLhQXV1diW0efvhhvfHGG3r99ddVV1en1tZWLV261LDr1LuY4yBJK1euTDofNmzYYNTxObghYO7cua6qqirxcV9fn4tGo666utqwq4G3fv16N3v2bOs2TElyW7duTXzc39/vioqK3C9/+cvEbR0dHS4cDrtXXnnFoMOB8fXj4Jxzy5cvd7fddptJP1aOHj3qJLm6ujrn3OnvfWZmpnv99dcT2/zzn/90klx9fb1Vm2n39ePgnHPf+c533IMPPmjX1EUY9M+ATp06pb1796qioiJx24gRI1RRUaH6+nrDzmwcPHhQ0WhUkydP1j333KPm5mbrlkw1NTWpra0t6fyIRCIqKyu7LM+P2tpaFRQUaNq0aVq9erWOHz9u3VJaxWIxSVJeXp4kae/evert7U06H6ZPn66JEycO6/Ph68fhKy+//LLy8/M1Y8YMrVu3TidPnrRo75wG3TDSrzt27Jj6+vpUWFiYdHthYaE+/vhjo65slJWVafPmzZo2bZqOHDmip59+WjfffLMOHDig7Oxs6/ZMtLW1SdJZz4+v7rtcLF68WEuXLlVpaakOHTqkH//4x6qsrFR9fb0yMjKs20u5/v5+PfTQQ7rxxhs1Y8YMSafPh6ysLOXm5iZtO5zPh7MdB0n6/ve/r5KSEkWjUe3fv18//OEP1dDQoD//+c+G3SYb9AGE/6qsrEz8e9asWSorK1NJSYn+9Kc/acWKFYadYTC46667Ev+eOXOmZs2apSlTpqi2tlYLFiww7Cw9qqqqdODAgcvifdDzOddxuP/++xP/njlzpoqLi7VgwQIdOnRIU6ZMGeg2z2rQvwSXn5+vjIyMM65iaW9vV1FRkVFXg0Nubq6uvfZaNTY2Wrdi5qtzgPPjTJMnT1Z+fv6wPD/WrFmjN998U++9917Sn28pKirSqVOn1NHRkbT9cD0fznUczqasrEySBtX5MOgDKCsrS3PmzFFNTU3itv7+ftXU1Ki8vNywM3snTpzQoUOHVFxcbN2KmdLSUhUVFSWdH/F4XLt3777sz4/Dhw/r+PHjw+r8cM5pzZo12rp1q959912VlpYm3T9nzhxlZmYmnQ8NDQ1qbm4eVufDhY7D2ezbt0+SBtf5YH0VxMV49dVXXTgcdps3b3YfffSRu//++11ubq5ra2uzbm1APfLII662ttY1NTW5v/71r66iosLl5+e7o0ePWreWVp2dne7DDz90H374oZPknn32Wffhhx+6Tz/91Dnn3M9//nOXm5vrtm/f7vbv3+9uu+02V1pa6r744gvjzlPrfMehs7PTPfroo66+vt41NTW5d955x33rW99y11xzjevu7rZuPWVWr17tIpGIq62tdUeOHEmskydPJrZZtWqVmzhxonv33Xfdnj17XHl5uSsvLzfsOvUudBwaGxvdT3/6U7dnzx7X1NTktm/f7iZPnuzmzZtn3HmyIRFAzjn3m9/8xk2cONFlZWW5uXPnul27dlm3NODuvPNOV1xc7LKystzVV1/t7rzzTtfY2GjdVtq99957TtIZa/ny5c6505diP/HEE66wsNCFw2G3YMEC19DQYNt0GpzvOJw8edItXLjQjRs3zmVmZrqSkhK3cuXKYfeftLN9/ZLciy++mNjmiy++cD/4wQ/cVVdd5caMGeNuv/12d+TIEbum0+BCx6G5udnNmzfP5eXluXA47KZOneoee+wxF4vFbBv/Gv4cAwDAxKB/DwgAMDwRQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAw8X+1N48hxpb7cwAAAABJRU5ErkJggg==\n"},"metadata":{}},{"output_type":"stream","name":"stdout","text":["Label: 7\n"]}],"source":["# Display image and label.\n","train_features, train_labels = next(iter(train_dataloader))\n","print(f\"Feature batch shape: {train_features.size()}\")\n","print(f\"Labels batch shape: {train_labels.size()}\")\n","img = train_features[0].squeeze()\n","label = train_labels[0]\n","plt.imshow(img, cmap=\"gray\")\n","plt.show()\n","print(f\"Label: {label}\")"]},{"cell_type":"markdown","metadata":{"id":"QSituGbeM6Ms"},"source":["##3) Transforms"]},{"cell_type":"markdown","metadata":{"id":"mH-Ny97jM2t7"},"source":["\n","\n","Data does not always come in its final processed form that is required\n","for training machine learning algorithms. We use **transforms** to\n","perform some manipulation of the data and make it suitable for training, including data augmentation.\n","\n","All TorchVision datasets have two parameters:\n","- `transform` to modify the\n","features\n","- `target_transform` to modify the labels that accept\n","callables containing the transformation logic. The\n","[torchvision.transforms](https://pytorch.org/vision/stable/transforms.html)\n","module offers several commonly-used transforms out of the box.\n","\n","The FashionMNIST features are in PIL Image format, and the labels are\n","integers. For training, we need the features as normalized tensors, and\n","the labels as one-hot encoded tensors. To make these transformations, we\n","use `ToTensor` and `Lambda`.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"8hhw7gXjM2t8"},"outputs":[],"source":["import torch\n","from torchvision import datasets\n","from torchvision.transforms import ToTensor, Lambda, RandomCrop\n","\n","ds = datasets.FashionMNIST(\n","    root=\"data\",\n","    train=True,\n","    download=True,\n","    transform=ToTensor(),\n","    target_transform=Lambda(lambda y: torch.zeros(10, dtype=torch.float).scatter_(0, torch.tensor(y), value=1))\n",")"]},{"cell_type":"markdown","metadata":{"id":"zBWxI2ltM2t8"},"source":["**ToTensor()**\n","\n","[ToTensor](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ToTensor)\n","converts a PIL image or NumPy `ndarray` into a `FloatTensor`. and scales\n","the image\\'s pixel intensity values in the range \\[0., 1.\\]\n"]},{"cell_type":"markdown","metadata":{"id":"LAR6X-c9M2t9"},"source":["**Lambda Transforms**\n","\n","Lambda transforms apply any user-defined lambda function. Here, we\n","define a function to turn the integer into a one-hot encoded tensor. It\n","first creates a zero tensor of size 10 (the number of labels in our\n","dataset) and calls\n","[scatter\\_](https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_.html)\n","which assigns a `value=1` on the index as given by the label `y`.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"09RiGJ7jM2t9"},"outputs":[],"source":["target_transform = Lambda(lambda y: torch.zeros(\n","    10, dtype=torch.float).scatter_(dim=0, index=torch.tensor(y), value=1))"]},{"cell_type":"markdown","source":["### Data augmentation with transforms\n","\n","`torchvision.transforms` also includes a relatively comprehensive library of image transformations; you will rarely need to write your own image transformations. These can be used to resize images to the expected input shape for models like vision transformers that need fixed input sizes, but can also be used to **augment** data during training. Here, we transform an image with `RandomCrop`."],"metadata":{"id":"_hZUpluATmPq"}},{"cell_type":"code","source":["aug = RandomCrop((14, 14))\n","img = ds[0][0][0] # Get first image, with shape (28, 28)\n","\n","fig, axs = plt.subplots(1, 4)\n","axs[0].imshow(img, cmap=\"gray\")\n","axs[0].set_title('Original Image')\n","\n","for i in range(1, 4):\n","  axs[i].set_title('Random Crop')\n","  axs[i].imshow(aug(img), cmap=\"gray\")"],"metadata":{"id":"1Fj_IsiSUCvO","colab":{"base_uri":"https://localhost:8080/","height":190},"executionInfo":{"status":"ok","timestamp":1772056271304,"user_tz":300,"elapsed":334,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"5d928469-a3c3-484a-f62e-4b11b0641057"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 4 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAiEAAACtCAYAAABm+q8NAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAMVdJREFUeJzt3XtYVVX+P/A3IByQy0EFuagoKt5NCy+haF5IHidNSrOwDCu1DJsxzb45pZb6iIOTOSpemmnUnmpsdLx0USczsxwvU2YpWuQFDUUQUa4KKqzfH/7OmbPXOWdvDufgPuD79Tw8j3uvc/ZZe/M5m+Ve67OWhxBCgIiIiOgO89S7AkRERHR3YiOEiIiIdMFGCBEREemCjRAiIiLSBRshREREpAs2QoiIiEgXbIQQERGRLtgIISIiIl2wEUJERES6aFCNkDfffBMeHh61eu+6devg4eGBs2fPurZSFs6ePQsPDw+sW7euzj6D6s6ECRPQpk0bvatBOmMcEGPAddyiEXL8+HE89dRTaNGiBQwGAyIjI/Hkk0/i+PHjeldNF19//TU8PDywadMmvauiC1OD0PTTqFEjtGjRAhMmTMCFCxf0rp7bOX36NJ5//nm0bdsWvr6+CAoKQv/+/fGXv/wF169f17t6tcY4cExDjAPGgGPqYww00rsCmzdvRnJyMpo2bYrnnnsO0dHROHv2LN577z1s2rQJGzZswCOPPFKjY73xxht47bXXalWP8ePH44knnoDBYKjV+8n15s2bh+joaFRUVODgwYNYt24d9u3bh8zMTPj6+updPbfw+eef47HHHoPBYMDTTz+Nbt264caNG9i3bx9mzpyJ48eP491339W7mk5hHGhr6HHAGNBWb2NA6OjUqVOicePGolOnTuLSpUuKsoKCAtGpUyfh7+8vTp8+rXqcsrKyuqymy2RnZwsAYu3ataqv27NnjwAgNm7ceGcq5mbWrl0rAIjvvvtOsf///u//BADx8ccf61KvlJQU0bp1a10+25YzZ86IgIAA0alTJ5Gbm2tVfvLkSbF06VK776+qqhLXr1+vyyo6hXFQMw05DhgDNVOfY0DX7pjFixfj2rVrePfddxEaGqooCwkJwZo1a1BeXo709HTzftO4jxMnTmDcuHFo0qQJ4uPjFWWWrl+/jt///vcICQlBYGAgHn74YVy4cAEeHh548803za+zNSakTZs2GDFiBPbt24c+ffrA19cXbdu2xfvvv6/4jCtXruCVV15B9+7dERAQgKCgIAwfPhw//fSTi67U/87t119/xVNPPQWj0YjQ0FDMnj0bQgjk5ORg1KhRCAoKQnh4ON5++23F+2/cuIE5c+YgNjYWRqMR/v7+GDBgAPbs2WP1WYWFhRg/fjyCgoIQHByMlJQU/PTTTzbHs/zyyy8YM2YMmjZtCl9fX/Tq1QuffPKJy87b0oABAwDcfuTo6HmZxuP8+c9/xrvvvot27drBYDCgd+/e+O6776w+a+vWrejWrRt8fX3RrVs3bNmyxWadysvLMWPGDLRq1QoGgwEdO3bEn//8ZwhpcWoPDw9MnToVGzduRJcuXeDn54e4uDgcO3YMALBmzRq0b98evr6+GDRoUI3GJqWnp6OsrAzvvfceIiIirMrbt2+PP/zhD1Z1+PDDD9G1a1cYDAbs3LkTAHDkyBEMHz4cQUFBCAgIwNChQ3Hw4EHF8UzfkW+++QbPP/88mjVrhqCgIDz99NO4evWqZn1dhXGgdDfGAWNAqV7HgC5Nn/8vMjJStGnTRvU1bdq0ES1btjRvz507VwAQXbp0EaNGjRIrV64UGRkZijJLY8eOFQDE+PHjRUZGhhg7dqzo0aOHACDmzp1rfp2pxZ2dnW3e17p1a9GxY0cRFhYm/vjHP4oVK1aI++67T3h4eIjMzEzz67777jvRrl078dprr4k1a9aIefPmiRYtWgij0SguXLhgfp0zT0JM59azZ0+RnJwsVq5cKR566CEBQCxZskR07NhRTJkyRaxcuVL0799fABB79+41v7+goEBERESI6dOni1WrVon09HTRsWNH4e3tLY4cOWJ+XVVVlYiLixNeXl5i6tSpYsWKFeLBBx80XzPLumdmZgqj0Si6dOki/vSnP4kVK1aIgQMHCg8PD7F582bVc1Rj738/K1asEADEqlWrHD4v07W/9957Rfv27cWf/vQnkZ6eLkJCQkTLli3FjRs3zK/997//LTw9PUW3bt3EkiVLxOuvvy6MRqPo2rWr4n8/1dXVYsiQIcLDw0NMnDhRrFixQowcOVIAENOmTVPUHYC45557RKtWrcSiRYvEokWLhNFoFFFRUWLFihWiS5cu4u233xZvvPGG8PHxEYMHD9a8Ti1atBBt27at8XUFIDp37ixCQ0PFW2+9JTIyMsSRI0dEZmam8Pf3FxEREWL+/Pli0aJFIjo6WhgMBnHw4EHz+02/l+7du4sBAwaIZcuWidTUVOHp6SkGDhwoqqura1yXmmAcMA4YAw0/BnRrhBQVFQkAYtSoUaqve/jhhwUAUVJSIoT43x/j5ORkq9fKjZDDhw/bDIIJEybUuBECQHzzzTfmfZcuXRIGg0HMmDHDvK+iokJUVVUpPiM7O1sYDAYxb948xT5nGyGTJ08277t165Zo2bKl8PDwEIsWLTLvv3r1qvDz8xMpKSmK11ZWVio+5+rVqyIsLEw8++yz5n3/+te/BADFo7uqqioxZMgQq7oPHTpUdO/eXVRUVJj3VVdXi379+omYmBjVc1Rj+l18+eWXoqCgQOTk5IhNmzaJ0NBQYTAYRE5OjsPnZbr2zZo1E1euXDHv37ZtmwAgPv30U/O+nj17ioiICFFUVGTe98UXXwgAihvP1q1bBQCxYMECxeePGTNGeHh4iFOnTpn3ARAGg0ERX2vWrBEARHh4uDm+hRBi1qxZVrEoKy4urtH3xxIA4enpKY4fP67Yn5SUJHx8fBTdnrm5uSIwMFAMHDjQvM/0e4mNjVXcqNPT0wUAsW3bthrXpSYYB4wDxkDDjwHdumNKS0sBAIGBgaqvM5WXlJQo9r/wwguan2F6vPTiiy8q9r/00ks1rmeXLl3Mj/4AIDQ0FB07dsSZM2fM+wwGAzw9b1/KqqoqFBYWIiAgAB07dsQPP/xQ48+qiYkTJ5r/7eXlhV69ekEIgeeee868Pzg42KqOXl5e8PHxAQBUV1fjypUruHXrFnr16qWo486dO+Ht7Y1JkyaZ93l6eiI1NVVRjytXruCrr77C2LFjUVpaisuXL+Py5csoLCxEYmIiTp486fTo9YSEBISGhqJVq1YYM2YM/P398cknn6Bly5YOn5fJ448/jiZNmpi3Tb9b07W6ePEifvzxR6SkpMBoNJpf9+CDD6JLly6KY23fvh1eXl74/e9/r9g/Y8YMCCGwY8cOxf6hQ4cq0vr69u0LABg9erTie2Dab/n7k5m+D1rfH9kDDzygOI+qqip88cUXSEpKQtu2bc37IyIiMG7cOOzbt8/quzd58mR4e3ubt6dMmYJGjRph+/btDtWlphgHjAPGQMONAd0aIaYLZmqM2GOvsRIdHa35GefOnYOnp6fVa9u3b1/jekZFRVnta9KkiaLfq7q6Gu+88w5iYmJgMBgQEhKC0NBQHD16FMXFxTX+rNrUx2g0wtfXFyEhIVb75b659evX45577oGvry+aNWuG0NBQfP7554o6njt3DhEREWjcuLHivfI1O3XqFIQQmD17NkJDQxU/c+fOBQBcunTJqXPNyMjArl27sGnTJvzud7/D5cuXbWYv1eS8TOTrZ7oJma7VuXPnAAAxMTFW7+3YsaNi+9y5c4iMjLSKzc6dOyuOZe+zTTe2Vq1a2dyv1rcaFBQEQPv7I5O/CwUFBbh27ZrVuQG3z6O6uho5OTmK/fK1CQgIQERERJ3NscM4YBwwBhpuDOiWoms0GhEREYGjR4+qvu7o0aNo0aKF+UKb+Pn51WX1zLy8vGzuFxaDjRYuXIjZs2fj2Wefxfz589G0aVN4enpi2rRpqK6urvP61KSOH3zwASZMmICkpCTMnDkTzZs3h5eXF9LS0hSDu2rKdF6vvPIKEhMTbb7GkcaeLX369EGvXr0AAElJSYiPj8e4ceOQlZWFgIAAAI6fV02uVV2x99m1qVNQUBAiIyORmZnpUB3u1PfGlRgHjAPGQMONAV2zY0aMGIHs7Gzs27fPZvm3336Ls2fPYsSIEbU6fuvWrVFdXY3s7GzF/lOnTtXqePZs2rQJgwcPxnvvvYcnnngCw4YNQ0JCAoqKilz6Oc7YtGkT2rZti82bN2P8+PFITExEQkICKioqFK9r3bo1Ll68iGvXrin2y9fM9LjO29sbCQkJNn8cfTyoxnQzyc3NxYoVKxw+r5pq3bo1AODkyZNWZVlZWVavzc3NtfofyC+//KI4Vl0ZMWIETp8+jQMHDtT6GKGhoWjcuLHVuQG3z8PT09Pqf2fytSkrK8PFixfvyAySjANrd1scMAas1ecY0LURMnPmTPj5+eH5559HYWGhouzKlSt44YUX0LhxY8ycObNWxzf9D33lypWK/cuXL69dhe3w8vKyaqlu3LjRrWb0M7WwLet56NAhq6BNTEzEzZs38de//tW8r7q6GhkZGYrXNW/eHIMGDcKaNWtw8eJFq88rKChwZfUBAIMGDUKfPn2wdOlS842lpudVUxEREejZsyfWr1+veIS7a9cunDhxQvHa3/3ud6iqqlLcCAHgnXfegYeHB4YPH16rOtTUq6++Cn9/f0ycOBH5+flW5adPn8Zf/vIX1WN4eXlh2LBh2LZtm+IRan5+Pj766CPEx8dbPYV89913cfPmTfP2qlWrcOvWrTo/XxPGgdLdGAeMAaX6HAO6zpgaExOD9evX48knn0T37t2tZky9fPky/vGPf6Bdu3a1On5sbCxGjx6NpUuXorCwEPfffz/27t2LX3/9FQBqvc6MbMSIEZg3bx6eeeYZ9OvXD8eOHcOHH36oGNyjtxEjRmDz5s145JFH8NBDDyE7OxurV69Gly5dUFZWZn5dUlIS+vTpgxkzZuDUqVPo1KkTPvnkE1y5cgWA8pplZGQgPj4e3bt3x6RJk9C2bVvk5+fjwIEDOH/+vEvnSTGZOXMmHnvsMaxbtw4vvPBCjc/LEWlpaXjooYcQHx+PZ599FleuXMHy5cvRtWtXxTFHjhyJwYMH4/XXX8fZs2fRo0cPfPHFF9i2bRumTZtW67itqXbt2uGjjz7C448/js6dOytmSdy/fz82btyICRMmaB5nwYIF2LVrF+Lj4/Hiiy+iUaNGWLNmDSorKxVz9JjcuHEDQ4cOxdixY5GVlYWVK1ciPj4eDz/8cB2cpW2Mg/+5W+OAMfA/9ToGapxHU4eOHj0qkpOTRUREhPD29hbh4eEiOTlZHDt2zOq1plTVgoICu2WWysvLRWpqqmjatKkICAgQSUlJIisrSwBQpLXaS9F96KGHrD7ngQceEA888IB5u6KiQsyYMUNEREQIPz8/0b9/f3HgwAGr17kiRVc+75SUFOHv72+zjl27djVvV1dXi4ULF4rWrVsLg8Eg7r33XvHZZ5/ZnPmvoKBAjBs3TgQGBgqj0SgmTJgg/vOf/wgAYsOGDYrXnj59Wjz99NMiPDxceHt7ixYtWogRI0aITZs2qZ6jGntzAwhxO124Xbt2ol27duLWrVs1Pi/TtV+8eLHVMSGlawtxO1W5c+fOwmAwiC5duojNmzfbvFalpaXi5ZdfFpGRkcLb21vExMSIxYsXW+XJAxCpqamKffbq5OiMub/++quYNGmSaNOmjfDx8RGBgYGif//+Yvny5Yr0aVt1MPnhhx9EYmKiCAgIEI0bNxaDBw8W+/fvV7zG9HvZu3evmDx5smjSpIkICAgQTz75pCgsLKxRXR3BOGAcMAYafgy4RSPkTjty5IgAID744AO9q1JvbNmyRQAQ+/bt07sqpBO1Pwh092AckCtjwC1W0a1LtlYOXLp0KTw9PTFw4EAdauT+5GtWVVWF5cuXIygoCPfdd59OtSIiooZG91V061p6ejoOHz6MwYMHo1GjRtixYwd27NiByZMnW430pdteeuklXL9+HXFxcaisrMTmzZuxf/9+LFy40G3SuoiIqP5r8I2Qfv36YdeuXZg/fz7KysoQFRWFN998E6+//rreVXNbQ4YMwdtvv43PPvsMFRUVaN++PZYvX46pU6fqXTUiImpAPIS4AzOzEBEREUka/JgQIiIick911h2TkZGBxYsXIy8vDz169MDy5cvRp08fzfdVV1cjNzcXgYGBLpvHg+qOEAKlpaWIjIw0L+LnCoyD+qUu4oAxUL/wXkC1igGn82ts2LBhg/Dx8RF///vfxfHjx8WkSZNEcHCwyM/P13xvTk6OAMCfevZjuaS2KzAO6uePK+OAMVA/f3gv4I8jMVAnY0L69u2L3r17m6ewra6uRqtWrfDSSy/htddeU7y2srISlZWV5u3i4mKbK9eSeysqKlIsdw3U/mkYcDsOgoOD66Cmt1kuP22L5VTEtowfP95u2aJFi1Tfe/nyZdVytUUPtb6uWqs237p1S7U8NDTUbpnaNSkrK8PgwYNdGgd1HQPkuG3bttktu3btGpKTk+vdvUBP9hasM6mqqqr1sWNjY1XLN23apFpua3oLE3v1LisrQ2xsrM0YsMfl3TE3btzA4cOHMWvWLPM+T09PJCQk2JzDPy0tDW+99Zarq0F3mPyY9OOPP8b06dOxevVq9O3bF0uXLkViYiKysrLQvHlzh4/nas4e38fHx26ZvL6CzLLRbYtaI0RrVWatm5ZW40pt0cEbN26ovhdwbRzw0bv78ff313xNfbsX6Kkuz61RI/U/71oLjKq9X6vx5Mh5uXxg6uXLl1FVVYWwsDDF/rCwMOTl5Vm9ftasWSguLjb/5OTkuLpKpIMlS5Zg0qRJeOaZZ9ClSxesXr0ajRs3xt///nebr6+srERJSYnih+o/R+KAMdAw8V5AanTPjjEYDAgKClL8UP1mehqWkJBg3qf2NAy4/UTMaDSafziRXP3naBwwBhoe3gtIi8sbISEhIfDy8rJaTjg/Px/h4eGu/jhyQ44+DQP4RKwh4lNR4r2AtLh8TIiPjw9iY2Oxe/duJCUlAbjdj717927OuEl2GQwGGAwGvatBOmIMEMA4uNvUyTwh06dPR0pKCnr16oU+ffpg6dKlKC8vxzPPPFMXH0duhk/DCGAcEGOAtNVJI+Txxx9HQUEB5syZg7y8PPTs2RM7d+60eiRHDZM7PA3TGhlek0wPNZmZmXbLtLJftNKD1Uaea2W/aGUbVFRUqJYXFhbaLVOrt606u0MckGs1btzYbpmt9HFXx4C9rAu174zWpFlamRxaGWVaGWtqtFLmnbF9+3bV8vLyctVytRRde9mBWvc+W+psxtSpU6fyRnMX49MwAhgHxBggdQ1+FV3SB5+GEcA4IMYAqWMjhOoMn4YRwDggxgDZx0aIC8l9i1pTbMsz1sXHxyu2d+zYUePPkvtEnelr1OojrYOZ/omI6C6k+2RlREREdHdiI4SIiIh0we4YF5JTweR0yvbt2yu2J06cqNiWU6IsU6jk1Mr//ve/im2t7he5i0Wuq2W51rEsu36EEE6lqNWVukx9A9TT35xJfQPUF8fT6gorKChQLddaeMrX19dumVqqojvGQEOl1V2qlZKqleYdEBBgt6xFixZ2y0pLS1WP6wr24r+uv+91ZeDAgarlo0ePtlvWv39/1fdeu3ZNtVwtHR9Qvw/ZmwKhNvcBPgkhIiIiXbARQkRERLpgI4SIiIh0wTEhLiT3t8t9r0OGDFFsWy5vDQDnz59XbFsu4iRPl/zggw8qtv/2t78ptuW1GuS+VLV+YblPWO7n0+prJCIiqgk+CSEiIiJdsBFCREREumAjhIiIiHTBMSEupLU8fO/evRXbbdq0UWzLY0os8/3//e9/K8ruvfdexXZ6erpi+/vvv1dsHzt2TLH9888/K7b79Oljt5779+9XbB84cMD8byEESkpK4G6cyb8HnMvBdyb/HrCfgw9oz/GgNV5Ha54Qy3FIstrOIUKupTVXjNbvWCuGkpOT7ZZFRETYLfP391c9rl6aNm2qWh4ZGalaHhMTU+v3P/roo6rv7dChg2p5ZWWl3TKt+WC07gXNmjVTLc/NzbVbJs9bZVJWVqZ6TFv4JISIiIh0wUYIERER6YKNECIiItIFx4Q4QV7DQe6rlefy6NWrl2JbXmtB7lO17C+U+w6/++47xfapU6cU2/JcH3FxcYptua/Ssk9fPra8xo1lP+WtW7fw7bffgoiIyFF8EkJERES6YCOEiIiIdMFGCBEREemCY0I0yOM+HDF//nzFtlqOPWC9PsytW7fM/5bnIImPj1dsy+NN5PVefvjhB8W2PIbE8rNSU1MVZW3btlVsjxkzxqrudWXSpEl259VQy8F3Jv8ecC4H35n8e8B+Dj6gPcdI8+bNVcu15rKRY9CSPF+MJa4ndOeozSMDaP+OtchzCllSi021MlfYunWr3blIQkND7b4vODhY9bha86ZozbtSVFRkt8zyvmqLPC5Qpva71PrbdP36ddVyte8zAIwdO9ZumTwPlUl5ebnqMW3hkxAiIiLSBRshREREpAs2QoiIiEgXHBOiQWudBjVXr15VbMtjQuQ+O3ndDsu+X3neD7n/1c/PT7EtjwkZMGCAYrtfv36KbcsxEPK4gp07d4KIiMjV+CSEiIiIdOFwI+Sbb77ByJEjERkZCQ8PD2zdulVRLoTAnDlzEBERAT8/PyQkJODkyZOuqi8RERE1EA43QsrLy9GjRw9kZGTYLE9PT8eyZcuwevVqHDp0CP7+/khMTKzz9C0iIiKqXxweEzJ8+HAMHz7cZpkQAkuXLsUbb7yBUaNGAQDef/99hIWFYevWrXjiiSecq209I8+5IM89IW/Lcy0UFxeb/11YWKgoa9OmjWJbHrsi55DLnyXXzTJXXh5P0qpVKzjizTffxFtvvaXY17FjR/zyyy8OHQcAUlJSrMbDmKjl4DuTfw84l4PvTP49YD8HHwACAwNV36s1/4kcN47o3r273TJb19uVcVAbWr9DrfkftOaKUTu+5VpMtsjfMUdozT3hrF27dtktU5sHwtZ3wpUx0KFDB7vxrzbXh9Y8IM7OcaM2d4/WZ2vN5aHGaDSqlrdu3Vq1fNGiRarlanWbMmWKzf0lJSWqx7TFpQNTs7OzkZeXh4SEBPM+o9GIvn374sCBAzYbIZWVlYobZ21OgtxP165d8eWXX5q3tSZYooaJcUCMAVLj0mjIy8sDAISFhSn2h4WFmctkaWlpVi1lqv8aNWqE8PBwvatBOmMcEGOA1OjeJJ01axamT59u3i4pKXH48X9dUuvWkB+1yd0GkZGRim35Ubm8LafoWnYZyI8M5amI5e4aubtFfmQoPz63fLR39OhRRZl8XpZTxFdVVeHIkSOQnTx5EpGRkfD19UVcXBzS0tIQFRVl9ToTPhFrmByJA8ZAw8R7AalxaYquqbWbn5+v2J+fn2+3JWwwGBAUFKT4ofqtb9++WLduHXbu3IlVq1YhOzsbAwYMUB2nkZaWBqPRaP5xp4Yo1Y6jccAYaHh4LyAtLm2EREdHIzw8HLt37zbvKykpwaFDhxAXF+fKjyI3Nnz4cDz22GO45557kJiYiO3bt6OoqAj//Oc/7b5n1qxZKC4uNv/k5OTcwRpTXXA0DhgDDQ/vBaTF4e6YsrIyxQqs2dnZ+PHHH9G0aVNERUVh2rRpWLBgAWJiYhAdHY3Zs2cjMjISSUlJrqw31SPBwcHo0KGD1cq9lgwGg1V3FDUsWnHAGGj4eC8gmcONkO+//x6DBw82b5vGc6SkpGDdunV49dVXUV5ejsmTJ6OoqAjx8fHYuXMnfH19XVfrO0hOfbVM65PHhDz++OOKbbkLqqCgQLGtNdW65bLV8iNJOcVU/tLKKYLyiHT5sy2Xn5fngOnZs6fqsbSUlZXh9OnTGD9+vEPvA26PZbGXAqeW/uZM6hvgXPqbM6lvgP30NwDIzc1Vfa/WfDyWTyltOXPmjN2ymJgYu2U1WcLbmTiwRy3NVis9sq5TXevKAw88oFqulQIeHx+vWq4Wn/I9zFJZWZnqcU2vqW0MXLhwQXFPtGQvjV+rDACaNm3qcF0seXt72y3Tuo9oPeVR+77L4/5k8rAI2fr161XL1R4cfPrppzb3a6Wm2+JwI2TQoEGq66l4eHhg3rx5mDdvnsOVoYbhlVdewciRI9G6dWvk5uZi7ty58PLyQnJyst5VozuIcUCMAdKie3YMNTznz59HcnIyCgsLERoaivj4eBw8eBChoaF6V43uIMYBMQZICxsh5HIbNmzQuwrkBhgHxBggLWyEaJDHP6hN952ZmanYlucBkfsO5f5suQ+7efPm5n/Lff3yvCDyseUxOHJf6tWrVxXb58+fN/973LhxirLFixcrtg8ePAgiIiJnuTRFl4iIiKim2AghIiIiXbARQkRERLqol2NC5PVcLMdWyMtvy6+V85i1ltR2ZC6B7du3K7bluRPk/Ht5/gs59dkyJ18ePyKP+dDKz9Y6b8vj33PPPYqy4uJi1WPXpWbNmtldvlstB9+Z/HvAuRx8Z/LvAfs5+ADQpk0b1fdqzYkQGxurWm45B5BMbWl7tWm465LWXCDO0Jo/Ql4bylKnTp1q/V4AGDNmTK2PrTVXjHxPlKnN+aKW1VLXc0EFBATYjW95/J0lrXuB1nfG3jxFJmpLjVy5ckX1vefOnVMtV6ub1nxDWnGg9bdty5YtdsuOHTtmc395eblixeSa4JMQIiIi0gUbIURERKQLNkKIiIhIF/ViTIjWfBp1uQbEwIEDFdujR482/7t///6KsmvXrim25bk85L5FeQ4S+bwsjydfA3mtGLk/Vh5fItdNZlk3eQ2IRx99VLGtNmaBiIiopvgkhIiIiHTBRggRERHpgo0QIiIi0kW9GBPiyFwAcm6/nI8fExOjWi6Pf+jQoYNi2zIfXZ43QR530axZM8W2PDeFnMctjxmxXDtGXrNGnsdi//79im05v1we2yLPE2I5F4g8p8j9998PvVy/ft1q7IyJWg6+M/n3ps9Vo5aD70z+PWA/Bx/QnidEa24LtbWPAKCoqMhumdpcNFrz7dSVfv362S1buHCh6nu1VnJt0qSJarnafUkewyWT126SqcVQSUmJ6nvV5swAtOcJUYv9EydO1Op9rnDhwgWrNbBM7O0HgJCQENXjqsU8AFy+fFm13HI+J5m9e5eJPLZPJq8JZklrXhZ78yuZqM37A6ifd+fOnW3ur818QXwSQkRERLpgI4SIiIh0wUYIERER6aJejAmRxyTMnz9fsW3ZtxscHKwok/tt5b5auT9Q7ouV+7gs+9TlvlW5T1QepzF27FjF9vfff6/YlvvwLPt2tcYCdO/eXfVY8voJ8vgVPz8/87/l8RKtW7dW/WwiIqLa4JMQIiIi0gUbIURERKQLt+2O8fT0NHd3LFu2TFEWERGh2LbsclGb+twWOS1Wfr9a2pm8lLzcbbFo0SLVY02ZMkWxrZbCu3v3bkXZmTNnFNty6rGcHiynZsqpX5bpWnIqploKWl3Lzs62m36nlv7mTOob4Fz6mzOpb4D99DdAfZl1QHvZcq3UULXrplZvrXo5w/JeIMvIyLD7vhYtWqgeVyuVWmtqAGfOWSs+1T5b656mRe6ylql1/S5YsMBumVoKtyu0bdvW7vdOvndaku+VMq0l77XS+dXuJZbd3LbIf39kaqneWqnYWvErL+0hU4uzixcv2txfm+8En4QQERGRLtgIISIiIl2wEUJERES6cNsxIcnJyeb+MnmsxenTpxXbln12cv+d1jTWcn+ePM5D7mO37HuUp07Pz89XbK9fv16xnZSUpNj+9NNPFdtyX6zlucTGxirKBg8erNiWxyHIY0DkPmi1vki5L1G+Rq1atTL/u7q6GhcuXLB7LCIiInv4JISIiIh0wUYIERER6cKhRkhaWhp69+6NwMBANG/eHElJScjKylK8pqKiAqmpqWjWrBkCAgIwevRoq24KIiIiIofGhOzduxepqano3bs3bt26hT/+8Y8YNmwYTpw4YZ7L4eWXX8bnn3+OjRs3wmg0YurUqXj00Ufxn//8x6GKFRQUmMciyOMy1KY3l18rjxGRx0IEBQUptuXl4eUl4S2PJ8/7Ieeby/MQyEu4y0u2y2NCLMezyGM85Onm5Rx9+bPlpdblcR6W5fKcDPI169Chg+Jz6nJMiLe3t908fLUcfGfy7wHncvCdyb8H7Ofg1+TYWnMDaC0trjZngtp8CVrLwzvjqaeesvv7VJvT4tSpU6rH1Zr/QatcnovHEVrz1Mhj0yxpzQWj9X1UW/YesB7bZumjjz6yW1ZSUoKwsDDVYzvj1VdftXvdXnnlFbvv01ryQmveHvleK1ObG0PrPqN1n1L7vmodW+s7qXUvUYtRe2VadbLFoSchO3fuxIQJE9C1a1f06NED69atw2+//YbDhw8DAIqLi/Hee+9hyZIlGDJkCGJjY7F27Vrs378fBw8etHnMyspKlJSUKH7IvV29ehU//fQTvv32W+zdu9eqXAiBOXPmICIiAn5+fkhISMDJkyd1qCnVlSNHjmDmzJl4+OGH0a9fP6v/ZDAG7g779u3D6NGjER0dbbMBwjggLU6NCSkuLgbwv/+xHz58GDdv3kRCQoL5NZ06dUJUVBQOHDhg8xhpaWkwGo3mH8vMC3JPVVVVCAgIQMeOHW2Wp6enY9myZVi9ejUOHToEf39/JCYmas5MSPVHRUUF2rdvjxkzZtgsZwzcHcrLy9G9e3csXbrUZjnjgLTUOkW3uroa06ZNQ//+/dGtWzcAQF5eHnx8fKymBQ4LC0NeXp7N48yaNQvTp083b5eUlLAh4uZCQkIQEhJis0wIgaVLl+KNN97AqFGjAADvv/8+wsLCsHXrVjzxxBM231dZWanoAuETMfcWFxeHuLg4m2WMgbtHYmIiEhMTbZYxDqgmat0ISU1NRWZmJvbt2+dUBQwGg811FC5evGjuX5L7rs6fP6/YtuzjlP84yv15cv+fvC6K3Acn182yL0xeX0QeqyLP3SF/trxGiNy3aNn3K6/5IddLPrbWGBG53HJ8RXh4uKLM9MTLpGfPnuZ/y2vaZGdnIy8vT/E0zGg0om/fvjhw4IDdG09aWhreeustm2VUvzAGCGAcUM3Uqjtm6tSp+Oyzz7Bnzx60bNnSvD88PBw3btyw+sOfn59v9YeNGibTEy+5f1jtaRhw+4lYcXGx+Udr8B25L8YAAYwDqhmHnoQIIfDSSy9hy5Yt+PrrrxEdHa0oj42Nhbe3N3bv3o3Ro0cDALKysvDbb7/ZfXRLBNh/IkZ3D8YAAYyDu41DT0JSU1PxwQcf4KOPPkJgYCDy8vKQl5dnTlU1Go147rnnMH36dOzZsweHDx/GM888g7i4ONx///11cgLkXkxPvOQ0Pz4Nu3swBghgHFDNOPQkZNWqVQCAQYMGKfavXbsWEyZMAAC888478PT0xOjRo1FZWYnExESsXLnS4YpZzqGxefNmRdmzzz6r2LZcz+XMmTOKMnkUtpz7L+c7y3NPyHnclnnQ8lwS8hwN8lgWeX4IeT4I+fWWx5PHqmidl9a8ImrzjMjjR+QnXmrzCERHRyM8PBy7d+82jx0pKSnBoUOHMGXKFLvvs+fSpUtWa/SYqOWkO5N/r3VsQD0H35n8e61yrfPSOrYz83movddyfFRdxIC98/rtt9/svk+eA0imlaGh1Q2gNo+I1u9JbR4QwHq+Iktnz56tdb0A7Xlq1K6LPJZMjavj4Msvv7RbtmPHDrtl8jpbsrS0NNVyee0ymdrvUh4XKNO6z6jdp7TmBNJy6dIl1XK1+5i9uWgciQ8Th7tjtPj6+iIjIwMZGRkOV4bqh5s3b6K0tFSx7+jRo4iKikJUVBSmTZuGBQsWICYmBtHR0Zg9ezYiIyOtFvCj+uvatWuKBoDpPwI5OTno2rUrY+AuUV5ebjWhI+8F5Ai3XUWX3FdhYaHV/0oGDBiAlJQUrFu3Dq+++irKy8sxefJkFBUVIT4+Hjt37rTKJqL66/jx44onksuWLQMALFy4EB9++CFj4C6RmZmJ8ePHK/bxXkCOYCOEHBYeHo6nnnoKwO1un3/+858oLi42P/728PDAvHnzMG/ePD2rSXWod+/eii7TsrIyxMXFmbtsGQN3h759++LXX38FcDsG7rvvPt4LyCH1ohEi99n9+OOPim3LdQPkdQLk+TPksRDy3BxyH53ct2vZRye/Vu4zl7uv5H5teVv+LMtyrb58uVwetyH3E1uuSwMo146RB40dPXpUsf3BBx+o1oWIiKgmnJq2nYiIiKi22AghIiIiXbARQkRERLpw2zEhHh4e5nEOluMVAOuccMttOSdcHk8i53zLOd5yXrc87sNyTIhWnrachy2PEZFzreV5R8rKyuzWQyYfW87XlucGkM9z165d5n///PPPirL9+/erfnZdGjBggN35HtRy8J3Jvwecy8F3Jv8esJ+DD1jHiMwyZmzRui5q1OqtNfeEMy5cuGC33mp10prnw3LNKVvsLdJoIo8vsySPRZPJa1bJ1OJTazZRrbli5LmQZGrzq6h95+T5hdzFnj17VMudnUizU6dOdsuciSEAimVRZFrzxWjN2XH69GnV8juFT0KIiIhIF2yEEBERkS7ctjtGCFGjGVpl8qM3rUdt8qM0+fGZ/LjM8vGY/DhMfvzlLo+7iIiI3BGfhBAREZEu2AghIiIiXbARQkRERLpw2zEhd8ovv/zi0OszMzPrqCZky7Bhw+ymZ6qlvzmT+gY4l/7GsUCuZblGjexf//qX3bKJEyeqHlctFRoAzpw5o1p+/fp1u2WBgYGq75WXaJCpLfCm9V6tNGy1egPqqbZq4/TkJTDuFo7+DXHE3fD3hk9CiIiISBdshBAREZEu2AghIiIiXbARQkRERLpgI4SIiIh0wUYIERER6cLtUnRrM1U76c/VvzfT8dRWs1VLJdRaBVdrxU+t98srO9NtroyDmhxLbVXh0tJS1fdqpZRqlVdUVNgtU1ttFoB5hXB71OLT2RRdrZWY1VbhVfudmFZwrqt7AdUfjvzOPISb/YbPnz+PVq1a6V0NclBOTo7m3BuOYBzUT66MA8ZA/cR7ATkSA27XCKmurkZubi6EEIiKikJOTg6CgoL0rla9UFJSglatWt3RayaEQGlpKSIjIzX/9+cIUxwEBgbCw8NDl3Or7+7kNauLOGAMOK++xwCgjIPS0lLGgIPcPQbcrjvG09MTLVu2RElJCQAgKCiIweagO33NjEajy49pigMZ48Fxd+qauToOGAOuU19jAFDGgakbizHgOHeNAQ5MJSIiIl2wEUJERES6cNtGiMFgwNy5c2EwGPSuSr3RkK9ZQz63utLQrllDO587oaFds4Z2PneCu18ztxuYSkRERHcHt30SQkRERA0bGyFERESkCzZCiIiISBdshBAREZEu2AghIiIiXbhtIyQjIwNt2rSBr68v+vbti//+9796V8ltpKWloXfv3ggMDETz5s2RlJSErKwsxWsqKiqQmpqKZs2aISAgAKNHj0Z+fr5ONXYOY8G+b775BiNHjkRkZCQ8PDywdetWRbkQAnPmzEFERAT8/PyQkJCAkydP6lNZJzAG1DEOqL7GgFs2Qj7++GNMnz4dc+fOxQ8//IAePXogMTERly5d0rtqbmHv3r1ITU3FwYMHsWvXLty8eRPDhg1TrPr58ssv49NPP8XGjRuxd+9e5Obm4tFHH9Wx1rXDWFBXXl6OHj16ICMjw2Z5eno6li1bhtWrV+PQoUPw9/dHYmKi6gqw7oYxoI1xQPU2BoQb6tOnj0hNTTVvV1VVicjISJGWlqZjrdzXpUuXBACxd+9eIYQQRUVFwtvbW2zcuNH8mp9//lkAEAcOHNCrmrXCWKg5AGLLli3m7erqahEeHi4WL15s3ldUVCQMBoP4xz/+oUMNa4cx4BjGAdWnGHC7JyE3btzA4cOHkZCQYN7n6emJhIQEHDhwQMeaua/i4mIAQNOmTQEAhw8fxs2bNxXXsFOnToiKiqpX15Cx4Jzs7Gzk5eUprp/RaETfvn3rzfVjDDiPcUDuHANu1wi5fPkyqqqqEBYWptgfFhaGvLw8nWrlvqqrqzFt2jT0798f3bp1AwDk5eXBx8cHwcHBitfWt2vIWHCO6RrV5+vHGHAe44DcOQYa6frp5LTU1FRkZmZi3759eleFiIjIIW73JCQkJAReXl5WmRz5+fkIDw/XqVbuaerUqfjss8+wZ88etGzZ0rw/PDwcN27cQFFRkeL19e0aMhacY7pG9fn6MQacxzggd44Bt2uE+Pj4IDY2Frt37zbvq66uxu7duxEXF6djzdyHEAJTp07Fli1b8NVXXyE6OlpRHhsbC29vb8U1zMrKwm+//VavriFjwTnR0dEIDw9XXL+SkhIcOnSo3lw/xoDzGAfk1jGg67BYOzZs2CAMBoNYt26dOHHihJg8ebIIDg4WeXl5elfNLUyZMkUYjUbx9ddfi4sXL5p/rl27Zn7NCy+8IKKiosRXX30lvv/+exEXFyfi4uJ0rHXtMBbUlZaWiiNHjogjR44IAGLJkiXiyJEj4ty5c0IIIRYtWiSCg4PFtm3bxNGjR8WoUaNEdHS0uH79us41rznGgDbGAdXXGHDLRogQQixfvlxERUUJHx8f0adPH3Hw4EG9q+Q2ANj8Wbt2rfk1169fFy+++KJo0qSJaNy4sXjkkUfExYsX9au0ExgL9u3Zs8dmLKSkpAghbqfmzZ49W4SFhQmDwSCGDh0qsrKy9K10LTAG1DEOqL7GgIcQQty55y5EREREt7ndmBAiIiK6O7ARQkRERLpgI4SIiIh0wUYIERER6YKNECIiItIFGyFERESkCzZCiIiISBdshBAREZEu2AghIiIiXbARQkRERLpgI4SIiIh08f8ASob/tEk9YdIAAAAASUVORK5CYII=\n"},"metadata":{}}]},{"cell_type":"markdown","metadata":{"id":"11eBoRSTMR1B"},"source":["##4) Build the Neural Network"]},{"cell_type":"markdown","metadata":{"id":"7OHAeb1cJ-4e"},"source":["Neural networks comprise of layers/modules that perform operations on\n","data. The [torch.nn](https://pytorch.org/docs/stable/nn.html) namespace\n","provides all the building blocks you need to build your own neural\n","networks. Every module in PyTorch subclasses the\n","[nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html).\n","A neural network is a `Module` itself that consists of other Modules\n","(layers). This nested structure allows for building and managing complex\n","architectures easily.\n","\n","Minimally, a `Module` should implement a `forward` method that implements its forward-pass behavior (with the backward pass being computed by autodiff, as we'll see later.)\n","\n","In the following sections, we\\'ll build a neural network to classify\n","images in the FashionMNIST dataset.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"t0hJdB_SJ-4f"},"outputs":[],"source":["import os\n","import torch\n","from torch import nn\n","from torch.utils.data import DataLoader\n","from torchvision import datasets, transforms"]},{"cell_type":"markdown","metadata":{"id":"MzzZRuT5J-4g"},"source":["###Get Device for Training\n","\n","We want to be able to train our model on an\n","[accelerator](https://pytorch.org/docs/stable/torch.html#accelerators)\n","such as CUDA, MPS, MTIA, or XPU. If the current accelerator is\n","available, we will use it. Otherwise, we use the CPU.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"8vZVg6LqJ-4g","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772057127318,"user_tz":300,"elapsed":7,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"ccdbd0d5-f036-4e1a-aaa3-ab453e9e8aea"},"outputs":[{"output_type":"stream","name":"stdout","text":["Using cuda device\n"]}],"source":["device = torch.accelerator.current_accelerator().type if torch.accelerator.is_available() else \"cpu\"\n","print(f\"Using {device} device\")"]},{"cell_type":"markdown","metadata":{"id":"zn-Dx7LdJ-4h"},"source":["###Define the Class\n","\n","We define our neural network by subclassing `nn.Module`, and initialize\n","the neural network layers in `__init__`. Every `nn.Module` subclass\n","implements the operations on input data in the `forward` method.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"eiZilc-QJ-4h"},"outputs":[],"source":["class NeuralNetwork(nn.Module):\n","    def __init__(self):\n","        super().__init__()\n","        self.flatten = nn.Flatten()\n","        self.linear_relu_stack = nn.Sequential(\n","            nn.Linear(28*28, 512),\n","            nn.ReLU(),\n","            nn.Linear(512, 512),\n","            nn.ReLU(),\n","            nn.Linear(512, 10),\n","        )\n","\n","    def forward(self, x):\n","        x = self.flatten(x)\n","        logits = self.linear_relu_stack(x)\n","        return logits"]},{"cell_type":"markdown","metadata":{"id":"-YFZHRy_J-4h"},"source":["We create an instance of `NeuralNetwork`, and move it to the `device`,\n","and print its structure.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"nu08e5i1J-4h","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772057127333,"user_tz":300,"elapsed":5,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"6accfd7d-51df-44f6-df28-b9a013c57802"},"outputs":[{"output_type":"stream","name":"stdout","text":["NeuralNetwork(\n","  (flatten): Flatten(start_dim=1, end_dim=-1)\n","  (linear_relu_stack): Sequential(\n","    (0): Linear(in_features=784, out_features=512, bias=True)\n","    (1): ReLU()\n","    (2): Linear(in_features=512, out_features=512, bias=True)\n","    (3): ReLU()\n","    (4): Linear(in_features=512, out_features=10, bias=True)\n","  )\n",")\n"]}],"source":["model = NeuralNetwork().to(device)\n","print(model)"]},{"cell_type":"markdown","metadata":{"id":"W53O-21DJ-4h"},"source":["To use the model, we pass it the input data. This executes the model\\'s\n","`forward`, along with some [background\n","operations](https://github.com/pytorch/pytorch/blob/270111b7b611d174967ed204776985cefca9c144/torch/nn/modules/module.py#L866).\n","Do not call `model.forward()` directly!\n","\n","Calling the model on the input returns a 2-dimensional tensor with dim=0\n","corresponding to each output of 10 raw predicted values for each class,\n","and dim=1 corresponding to the individual values of each output. We get\n","the prediction probabilities by passing it through an instance of the\n","`nn.Softmax` module.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iqwMcFIOJ-4i","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772057127307,"user_tz":300,"elapsed":20,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"b78ae34f-a85e-4fdc-9ed3-4dc2d5701e39"},"outputs":[{"output_type":"stream","name":"stdout","text":["Logits shape: torch.Size([1, 10])\n","Predicted class: tensor([6], device='cuda:0')\n"]}],"source":["X = torch.rand(1, 28, 28, device=device)\n","logits = model(X)\n","print(f\"Logits shape: {logits.shape}\")\n","pred_probab = nn.Softmax(dim=1)(logits)\n","y_pred = pred_probab.argmax(1)\n","print(f\"Predicted class: {y_pred}\")"]},{"cell_type":"markdown","metadata":{"id":"faLPM-LdJ-4i"},"source":["------------------------------------------------------------------------\n"]},{"cell_type":"markdown","metadata":{"id":"s9yS6FQSJ-4i"},"source":["###Model Layers\n","\n","Let\\'s break down the layers in the FashionMNIST model. To illustrate\n","it, we will take a sample minibatch of 3 images of size 28x28 and see\n","what happens to it as we pass it through the network.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"PhFekq1zJ-4i","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772057127287,"user_tz":300,"elapsed":8,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"375d3993-c38c-42ed-f102-4f41b6bee32c"},"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([3, 28, 28])\n"]}],"source":["input_image = torch.rand(3,28,28)\n","print(input_image.size())"]},{"cell_type":"markdown","metadata":{"id":"wk8EfAd-J-4i"},"source":["####nn.Flatten\n","\n","We initialize the\n","[nn.Flatten](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html)\n","layer to convert each 2D 28x28 image into a contiguous array of 784\n","pixel values ( the minibatch dimension (at dim=0) is maintained).\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"v3ffYfsIJ-4i","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772057127308,"user_tz":300,"elapsed":4,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"f928442d-6a81-4e79-cacb-27a999da3ed8"},"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([3, 784])\n"]}],"source":["flatten = nn.Flatten()\n","flat_image = flatten(input_image)\n","print(flat_image.size())"]},{"cell_type":"markdown","metadata":{"id":"TV28IJVIJ-4i"},"source":["####nn.Linear\n","\n","The [linear\n","layer](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)\n","is a module that applies a linear transformation on the input using its\n","stored weights and biases.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MgNb7W9IJ-4j","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772057127310,"user_tz":300,"elapsed":3,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"8cde9af9-c239-46a1-e0f8-b17b3417fe65"},"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([3, 20])\n"]}],"source":["layer1 = nn.Linear(in_features=28*28, out_features=20)\n","hidden1 = layer1(flat_image)\n","print(hidden1.size())"]},{"cell_type":"markdown","metadata":{"id":"SiNV2CCQJ-4j"},"source":["####nn.ReLU\n","\n","Non-linear activations are what create the complex mappings between the\n","model\\'s inputs and outputs. They are applied after linear\n","transformations to introduce *nonlinearity*, helping neural networks\n","learn a wide variety of phenomena.\n","\n","In this model, we use\n","[nn.ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html)\n","between our linear layers, but there\\'s other activations to introduce\n","non-linearity in your model.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"6fEqHx22J-4j","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772057208218,"user_tz":300,"elapsed":9,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"92d12346-86f8-4977-8abd-e9972c36201d"},"outputs":[{"output_type":"stream","name":"stdout","text":["Before ReLU: tensor([[-0.7859,  0.2442, -0.2362, -0.0423, -0.1368,  0.0877,  0.4403,  0.2558,\n","         -0.0643, -0.2168,  0.2097,  0.5608,  0.1453,  0.1093,  0.5298,  0.2701,\n","         -0.4077, -0.0118, -0.0943, -0.1792],\n","        [-0.6290,  0.2783, -0.4732,  0.3477, -0.2831,  0.1792,  0.2705, -0.0025,\n","         -0.0753,  0.1558,  0.0826,  0.3272,  0.0158,  0.1041,  0.3259,  0.1702,\n","         -0.5146, -0.1213,  0.0657,  0.1780],\n","        [-0.4968,  0.0218, -0.2450,  0.0556, -0.2566,  0.1782,  0.5621,  0.0166,\n","         -0.1295,  0.1388, -0.2622,  0.1579, -0.2071,  0.0069,  0.4944,  0.0158,\n","         -0.6976, -0.0158,  0.3636, -0.1702]], grad_fn=<AddmmBackward0>)\n","\n","\n","After ReLU: tensor([[0.0000, 0.2442, 0.0000, 0.0000, 0.0000, 0.0877, 0.4403, 0.2558, 0.0000,\n","         0.0000, 0.2097, 0.5608, 0.1453, 0.1093, 0.5298, 0.2701, 0.0000, 0.0000,\n","         0.0000, 0.0000],\n","        [0.0000, 0.2783, 0.0000, 0.3477, 0.0000, 0.1792, 0.2705, 0.0000, 0.0000,\n","         0.1558, 0.0826, 0.3272, 0.0158, 0.1041, 0.3259, 0.1702, 0.0000, 0.0000,\n","         0.0657, 0.1780],\n","        [0.0000, 0.0218, 0.0000, 0.0556, 0.0000, 0.1782, 0.5621, 0.0166, 0.0000,\n","         0.1388, 0.0000, 0.1579, 0.0000, 0.0069, 0.4944, 0.0158, 0.0000, 0.0000,\n","         0.3636, 0.0000]], grad_fn=<ReluBackward0>)\n"]}],"source":["print(f\"Before ReLU: {hidden1}\\n\\n\")\n","hidden1 = nn.ReLU()(hidden1)\n","print(f\"After ReLU: {hidden1}\")"]},{"cell_type":"markdown","metadata":{"id":"piYPTgHqJ-4j"},"source":["####nn.Sequential\n","\n","[nn.Sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html)\n","is an ordered container of modules. The data is passed through all the\n","modules in the same order as defined. You can use sequential containers\n","to put together a quick network like `seq_modules`.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"61PiZbKCJ-4j"},"outputs":[],"source":["seq_modules = nn.Sequential(\n","    flatten,\n","    layer1,\n","    nn.ReLU(),\n","    nn.Linear(20, 10)\n",")\n","input_image = torch.rand(3,28,28)\n","logits = seq_modules(input_image)"]},{"cell_type":"markdown","metadata":{"id":"LNhl0ZVpJ-4j"},"source":["####nn.Softmax\n","\n","The last linear layer of the neural network returns logits—raw values in \\[-infty, infty\\] somewhat like log-probabilities—which are passed to the\n","[nn.Softmax](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html)\n","module. The logits are scaled to values \\[0, 1\\] representing the\n","model\\'s predicted probabilities for each class. `dim` parameter\n","indicates the dimension along which the values must sum to 1.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"V8fnWOPFJ-4j"},"outputs":[],"source":["softmax = nn.Softmax(dim=1)\n","pred_probab = softmax(logits)"]},{"cell_type":"markdown","metadata":{"id":"p5MdShUxJ-4j"},"source":["###Model Parameters\n","\n","To train a neural network, PyTorch needs to know which tensors are trainable parameters (such as weights and biases) and should be optimized. This matters because many tensors which appear during the forward pass, such as activations or intermediate results, that should **not** be treated as learnable parameters.\n","\n","In PyTorch, trainable parameters are represented by `torch.nn.Parameter`, which is a subclass of `torch.Tensor`. When a `Parameter` is assigned as an attribute of an `nn.Module`, PyTorch automatically **registers** it as part of the model, so it will appear in `model.parameters()` and can be updated by an optimizer.\n","\n","In practice, you usually do not need to create `Parameter`s manually: built-in layers (e.g., `nn.Linear`, `nn.Conv2d`) create and register their own weights and biases, and a submodule's parameters are included automatically in the primary module upon creation.\n","\n","In this example, we iterate over each parameter in our current model, and print its size and\n","a preview of its values."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MBMlzG-_J-4j","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772057305352,"user_tz":300,"elapsed":320,"user":{"displayName":"Kathy Choi","userId":"06877605307764605557"}},"outputId":"4b48334d-3803-4c3b-856d-5a26536db028"},"outputs":[{"output_type":"stream","name":"stdout","text":["Model structure: NeuralNetwork(\n","  (flatten): Flatten(start_dim=1, end_dim=-1)\n","  (linear_relu_stack): Sequential(\n","    (0): Linear(in_features=784, out_features=512, bias=True)\n","    (1): ReLU()\n","    (2): Linear(in_features=512, out_features=512, bias=True)\n","    (3): ReLU()\n","    (4): Linear(in_features=512, out_features=10, bias=True)\n","  )\n",")\n","\n","\n","Layer: linear_relu_stack.0.weight | Size: torch.Size([512, 784]) | Values : tensor([[-0.0061,  0.0052, -0.0012,  ...,  0.0330,  0.0027,  0.0042],\n","        [-0.0280, -0.0311, -0.0343,  ..., -0.0199,  0.0174,  0.0297]],\n","       device='cuda:0', grad_fn=<SliceBackward0>) \n","\n","Layer: linear_relu_stack.0.bias | Size: torch.Size([512]) | Values : tensor([0.0103, 0.0344], device='cuda:0', grad_fn=<SliceBackward0>) \n","\n","Layer: linear_relu_stack.2.weight | Size: torch.Size([512, 512]) | Values : tensor([[ 0.0119,  0.0436, -0.0337,  ...,  0.0261,  0.0420, -0.0336],\n","        [ 0.0100,  0.0237, -0.0323,  ..., -0.0109,  0.0355, -0.0379]],\n","       device='cuda:0', grad_fn=<SliceBackward0>) \n","\n","Layer: linear_relu_stack.2.bias | Size: torch.Size([512]) | Values : tensor([-0.0021,  0.0031], device='cuda:0', grad_fn=<SliceBackward0>) \n","\n","Layer: linear_relu_stack.4.weight | Size: torch.Size([10, 512]) | Values : tensor([[ 0.0236,  0.0377,  0.0080,  ..., -0.0088, -0.0365, -0.0262],\n","        [-0.0208,  0.0102, -0.0124,  ..., -0.0007,  0.0436,  0.0180]],\n","       device='cuda:0', grad_fn=<SliceBackward0>) \n","\n","Layer: linear_relu_stack.4.bias | Size: torch.Size([10]) | Values : tensor([-0.0404,  0.0059], device='cuda:0', grad_fn=<SliceBackward0>) \n","\n"]}],"source":["print(f\"Model structure: {model}\\n\\n\")\n","\n","for name, param in model.named_parameters():\n","    print(f\"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \\n\")"]},{"cell_type":"markdown","metadata":{"id":"TlCkBVwkJ-4k"},"source":["------------------------------------------------------------------------\n"]},{"cell_type":"markdown","metadata":{"id":"nNYCdIcTJnPW"},"source":["##5) Automatic Differentiation with `torch.autograd`\n"]},{"cell_type":"markdown","metadata":{"id":"iFWD8J7TJ7Od"},"source":["When training neural networks, the most frequently used algorithm is\n","**back propagation**. In this algorithm, parameters (model weights) are\n","adjusted according to the **gradient** of the loss function with respect\n","to the given parameter.\n","\n","To compute those gradients, PyTorch has a built-in differentiation\n","engine called `torch.autograd`. It supports automatic computation of\n","gradient for any computational graph. Gradient tracking is enabled for any `Tensor` which has `requires_grad=True`.\n","\n","Consider the simplest one-layer neural network, with input `x`,\n","parameters `w` and `b`, and some loss function. It can be defined in\n","PyTorch in the following manner:"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"Siewhg_xJnPX","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772143370193,"user_tz":300,"elapsed":3391,"user":{"displayName":"Ali Cy","userId":"13619003554061403378"}},"outputId":"3d5be652-e606-434a-c835-3f2fcfa2b308"},"outputs":[{"output_type":"stream","name":"stdout","text":["Observe that z has a grad_fn stored for the backward pass:\n","tensor([-0.9373,  0.7791,  2.2106], grad_fn=<AddBackward0>)\n"]}],"source":["import torch\n","\n","x = torch.ones(5)  # input tensor\n","y = torch.zeros(3)  # expected output\n","w = torch.randn(5, 3, requires_grad=True)\n","b = torch.randn(3, requires_grad=True)\n","z = torch.matmul(x, w)+b\n","print('Observe that z has a grad_fn stored for the backward pass:')\n","print(z)\n","loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)"]},{"cell_type":"markdown","metadata":{"id":"OuXQs6t3JnPY"},"source":["This code defines the following **computational graph**:\n","\n","![](https://pytorch.org/tutorials/_static/img/basics/comp-graph.png)\n","\n","The graph is built forward during the forward pass, and then traced backwards during the backward pass (using hte `grad_fn` seen above) to calculate gradients with the chain rule. In this network, `w` and `b` are **parameters**, which we need to\n","optimize. Thus, we need to be able to compute the gradients of loss\n","function with respect to those variables. In order to do that, we set\n","the `requires_grad` property of those tensors.\n"]},{"cell_type":"markdown","metadata":{"id":"W2-UDBq-JnPZ"},"source":["###Computing Gradients\n","\n","To optimize weights of parameters in the neural network, we need to\n","compute the derivatives of our loss function with respect to parameters,\n","namely, we need $\\frac{\\partial loss}{\\partial w}$ and\n","$\\frac{\\partial loss}{\\partial b}$ under some fixed values of `x` and\n","`y`. To compute those derivatives, we call `loss.backward()`, and then\n","retrieve the values from `w.grad` and `b.grad`:\n"]},{"cell_type":"code","execution_count":2,"metadata":{"id":"MmHujVT4JnPZ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772143440416,"user_tz":300,"elapsed":38,"user":{"displayName":"Ali Cy","userId":"13619003554061403378"}},"outputId":"d88bcc6d-a72a-4811-c238-a0d4b78c4dae"},"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[0.0938, 0.2285, 0.3004],\n","        [0.0938, 0.2285, 0.3004],\n","        [0.0938, 0.2285, 0.3004],\n","        [0.0938, 0.2285, 0.3004],\n","        [0.0938, 0.2285, 0.3004]])\n","tensor([0.0938, 0.2285, 0.3004])\n"]}],"source":["loss.backward()\n","print(w.grad)\n","print(b.grad)"]},{"cell_type":"markdown","metadata":{"id":"CnsmGVBcJnPZ"},"source":["###Disabling Gradient Tracking\n","\n","By default, all tensors with `requires_grad=True` are tracking their\n","computational history and support gradient computation. However, there\n","are some cases when we do not need to do that, for example, when we have\n","trained the model and just want to perform inference on some data for evaluation, i.e. we\n","only want to do *forward* computations through the network. We can stop\n","tracking computations by surrounding our computation code with\n","`torch.no_grad()` block:\n"]},{"cell_type":"code","execution_count":3,"metadata":{"id":"0wRcGjfdJnPZ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772143476740,"user_tz":300,"elapsed":5,"user":{"displayName":"Ali Cy","userId":"13619003554061403378"}},"outputId":"6cdb0528-fcc2-476c-9b10-7315626c41fc"},"outputs":[{"output_type":"stream","name":"stdout","text":["True\n","False\n"]}],"source":["z = torch.matmul(x, w)+b\n","print(z.requires_grad)\n","\n","with torch.no_grad():\n","    z = torch.matmul(x, w)+b\n","print(z.requires_grad)"]},{"cell_type":"markdown","metadata":{"id":"ulfdqu_cJnPZ"},"source":["Another way to achieve the same result is to use the `detach()` method\n","on the tensor:\n"]},{"cell_type":"code","execution_count":4,"metadata":{"id":"q-d2FqDCJnPa","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772143508869,"user_tz":300,"elapsed":16,"user":{"displayName":"Ali Cy","userId":"13619003554061403378"}},"outputId":"b378f2c7-e94a-4245-da5f-ff36ed8302bd"},"outputs":[{"output_type":"stream","name":"stdout","text":["False\n"]}],"source":["z = torch.matmul(x, w)+b\n","z_det = z.detach()\n","print(z_det.requires_grad)"]},{"cell_type":"markdown","metadata":{"id":"_V576NujJnPa"},"source":["There are reasons you might want to disable gradient tracking:\n","\n","  * To convert tensors into NumPy arrays.\n","   *   To mark some parameters in your neural network as **frozen\n","        parameters**.\n","  *   To **reduce memory usage** and **speed up computations** when you are only doing forward\n","        pass, because autograd uses a significant amount of memory, and computations on tensors that do not track\n","        gradients would be more efficient.\n","  * `detach()` is used for some special training techniques to encourage better loss behavior or to disable unimportant & poorly behaved gradients; e.g. [SimSiam](https://arxiv.org/abs/2011.10566).\n"]},{"cell_type":"markdown","metadata":{"id":"llc_21hOJCD8"},"source":["##6) Optimizing Model Parameters"]},{"cell_type":"markdown","metadata":{"id":"47m7o7pgI_dk"},"source":["Now that we have a model and data it\\'s time to train, validate and test\n","our model by optimizing its parameters on our data. Training a model is\n","an iterative process; in each iteration the model makes a guess about\n","the output, calculates the error in its guess (*loss*), collects the\n","derivatives of the error with respect to its parameters (as we saw in\n","the [previous section](autogradqs_tutorial.html)), and **optimizes**\n","these parameters using gradient descent. For a more detailed walkthrough\n","of this process, check out this video on [backpropagation from\n","3Blue1Brown](https://www.youtube.com/watch?v=tIeHLnjs5U8).\n","\n","### Prerequisite Code\n","\n","We load the code from the previous sections on [Datasets &\n","DataLoaders](data_tutorial.html) and [Build\n","Model](buildmodel_tutorial.html).\n"]},{"cell_type":"code","execution_count":5,"metadata":{"id":"z1NkVgqfI_dl","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772143926897,"user_tz":300,"elapsed":16301,"user":{"displayName":"Ali Cy","userId":"13619003554061403378"}},"outputId":"1c496322-a1a7-46c8-f4df-29d184cc274e"},"outputs":[{"output_type":"stream","name":"stderr","text":["100%|██████████| 26.4M/26.4M [00:02<00:00, 9.41MB/s]\n","100%|██████████| 29.5k/29.5k [00:00<00:00, 149kB/s]\n","100%|██████████| 4.42M/4.42M [00:01<00:00, 2.70MB/s]\n","100%|██████████| 5.15k/5.15k [00:00<00:00, 12.8MB/s]\n"]}],"source":["import torch\n","from torch import nn\n","from torch.utils.data import DataLoader\n","from torchvision import datasets\n","from torchvision.transforms import ToTensor\n","\n","training_data = datasets.FashionMNIST(\n","    root=\"data\",\n","    train=True,\n","    download=True,\n","    transform=ToTensor()\n",")\n","\n","test_data = datasets.FashionMNIST(\n","    root=\"data\",\n","    train=False,\n","    download=True,\n","    transform=ToTensor()\n",")\n","\n","train_dataloader = DataLoader(training_data, batch_size=64)\n","test_dataloader = DataLoader(test_data, batch_size=64)\n","\n","class NeuralNetwork(nn.Module):\n","    def __init__(self):\n","        super().__init__()\n","        self.flatten = nn.Flatten()\n","        self.linear_relu_stack = nn.Sequential(\n","            nn.Linear(28*28, 512),\n","            nn.ReLU(),\n","            nn.Linear(512, 512),\n","            nn.ReLU(),\n","            nn.Linear(512, 10),\n","        )\n","\n","    def forward(self, x):\n","        x = self.flatten(x)\n","        logits = self.linear_relu_stack(x)\n","        return logits\n","\n","model = NeuralNetwork()"]},{"cell_type":"markdown","metadata":{"id":"-CpXaqpOI_dl"},"source":["###Hyperparameters\n","\n","Hyperparameters are settings of the model optimization process which are determined **before** any optimization begins, and are themselves **not optimizable through gradient descent**. Different hyperparameter values can significantly impact model\n","training and convergence rates ([read\n","more](https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html)\n","about automatic hyperparameter tuning in torch).\n","\n","We define the following hyperparameters for training:\n","\n","-   **Number of Epochs** - the number of times to iterate over the\n","        dataset\n","-   **Batch Size** - the number of data samples propagated through\n","        the network before the parameters are updated\n","-   **Learning Rate** - how much to update models parameters at each\n","        batch/epoch. Smaller values yield slow learning speed, while\n","        large values may result in unpredictable behavior during\n","        training.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"C1_FvqKlI_dm"},"outputs":[],"source":["learning_rate = 1e-3\n","batch_size = 64\n","epochs = 5"]},{"cell_type":"markdown","source":["\n","Selecting high-performing hyperparameters is **extremely important** for any machine learning pipeline. When starting out, always try to reuse hyperparameters from relevant baseline methods whenever possible, and keep an eye out for poor training behavior such as loss spikes or slow convergence. At a more advanced level, this [repo by Google](https://github.com/google-research/tuning_playbook) has a lot of good advice on when and how to tune hyperparameters."],"metadata":{"id":"AzzhvkZWvWGz"}},{"cell_type":"markdown","metadata":{"id":"8O3jrmx7I_dm"},"source":["###Optimization Loop\n","\n","Once we set our hyperparameters, we can then train and optimize our\n","model with an optimization loop. Each iteration of the optimization loop\n","is called an **epoch**.\n","\n","Each epoch consists of two main parts:\n","\n","-   **The Train Loop** - iterate over the training dataset and try to converge to optimal parameters.\n","-   **The Validation/Test Loop** - iterate over the test dataset to check if model performance is improving.\n","\n","Let\\'s briefly familiarize ourselves with some of the concepts used in\n","the training loop.\n"]},{"cell_type":"markdown","source":["###Loss Function\n","\n","When presented with some training data, our untrained network is likely\n","not to give the correct answer. The **loss function** measures the degree of\n","dissimilarity of obtained result to the target value, and it is the loss\n","function that we want to minimize during training, with zero loss typically meaning a perfect answer. To calculate the loss\n","we make a prediction using the inputs of our given data sample and\n","compare it against the true data label value.\n","\n","Common loss functions include\n","[nn.MSELoss](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss)\n","(Mean Square Error) for regression tasks, and\n","[nn.NLLLoss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss)\n","(Negative Log Likelihood) for classification.\n","[nn.CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss)\n","combines `nn.LogSoftmax` and `nn.NLLLoss`.\n","\n","We pass our model\\'s output logits to `nn.CrossEntropyLoss`, which will\n","normalize the logits and compute the prediction error."],"metadata":{"id":"eu9xPy-pWdh7"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"5HDKgKQ2I_dm"},"outputs":[],"source":["# Initialize the loss function\n","loss_fn = nn.CrossEntropyLoss()"]},{"cell_type":"markdown","metadata":{"id":"nK55fyThI_dn"},"source":["###Optimizer\n","\n","Optimization is the process of adjusting model parameters to reduce\n","the loss in each training step. **Optimization algorithms** define\n","how this process is performed. In torch, optimization is handled by the `torch.optim.Optimizer` object, which stores the actual optimization logic and exactly which parameters are to be optimized, as well as keeping track of any relevant historical state (e.g. [momentum](https://distill.pub/2017/momentum/)).\n","\n","Here, we use the Adam optimizer. For most ML applications, Adam will provide reasonable performance and is much less sensitive to changes in the learning rate than vanilla stochastic gradient descent (SGD).  There\n","are many [different\n","optimizers](https://pytorch.org/docs/stable/optim.html) available in\n","PyTorch such as SGD, RMSProp and the recent Muon, that each have their advantages and disadvantages.\n","\n","We initialize the optimizer by registering the model\\'s parameters that\n","need to be trained and then passing in relevant hyperparameters. At minimum, this includes the learning rate, but other common hyperparameters we can set include weight decay and $\\beta_2$ for Adam."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"aJJTXzx-I_dn"},"outputs":[],"source":["optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)"]},{"cell_type":"markdown","metadata":{"id":"CnPDl6bsI_dn"},"source":["Inside the training loop, optimization happens in three steps:\n","\n","-   Call `optimizer.zero_grad()` to reset the gradients of model\n","        parameters. By default, gradients will add up, which enables us to use multiple loss functions, or to perform gradient accumulation. Normally, however, to prevent\n","        double-counting, we must explicitly zero them at each iteration.\n","-   Backpropagate the prediction loss with a call to\n","        `loss.backward()`. At this point, PyTorch deposits the gradients of the loss\n","        w.r.t. each parameter in the `grad` attribute, as seen earlier.\n","-   Once we have our gradients, we call `optimizer.step()` to adjust\n","        the parameters by the gradients collected in the backward pass. This is the point at which the actual parameter values are updated. (Note that `loss.backward()` must happen first, so that the gradients are actually computed.)"]},{"cell_type":"markdown","metadata":{"id":"IOOwTWaTI_dn"},"source":["###Full Implementation\n","\n","We define `train_loop` that loops over our optimization code, and\n","`test_loop` that evaluates the model\\'s performance against our test\n","data.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Pfpc6svyI_dn"},"outputs":[],"source":["def train_loop(dataloader, model, loss_fn, optimizer):\n","    size = len(dataloader.dataset)\n","    # Set the model to training mode - important for batch normalization and dropout layers\n","    # Unnecessary in this situation but added for best practices\n","    model.train()\n","\n","    for batch, (X, y) in enumerate(dataloader):\n","        # Compute prediction and loss\n","        pred = model(X)\n","        loss = loss_fn(pred, y)\n","\n","        # Backpropagation\n","        loss.backward()\n","        optimizer.step()\n","        optimizer.zero_grad()\n","\n","        if batch % 100 == 0:\n","            loss, current = loss.item(), batch * batch_size + len(X)\n","            print(f\"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]\")\n","    return loss # NZ: recording loss\n","\n","\n","def test_loop(dataloader, model, loss_fn):\n","    # Set the model to evaluation mode - important for batch normalization and dropout layers\n","    # Unnecessary in this situation but added for best practices\n","    model.eval()\n","    size = len(dataloader.dataset)\n","    num_batches = len(dataloader)\n","    test_loss, correct = 0, 0\n","\n","    # Evaluating the model with torch.no_grad() ensures that no gradients are computed during test mode\n","    # also serves to reduce unnecessary gradient computations and memory usage for tensors with requires_grad=True\n","    with torch.no_grad():\n","        for X, y in dataloader:\n","            pred = model(X)\n","            test_loss += loss_fn(pred, y).item()\n","            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n","\n","    test_loss /= num_batches\n","    correct /= size\n","    print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")"]},{"cell_type":"markdown","metadata":{"id":"Y_wyMv70I_dn"},"source":["We initialize the loss function and optimizer, and pass it to\n","`train_loop` and `test_loop`. Feel free to increase the number of epochs\n","to track the model\\'s improving performance.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NAdhw0LxI_do"},"outputs":[],"source":["loss_fn = nn.CrossEntropyLoss()\n","optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n","\n","epochs = 10\n","for t in range(epochs):\n","    print(f\"Epoch {t+1}\\n-------------------------------\")\n","    train_loop(train_dataloader, model, loss_fn, optimizer)\n","    test_loop(test_dataloader, model, loss_fn)\n","print(\"Done!\")"]},{"cell_type":"markdown","metadata":{"id":"0f1gD9FFeJel"},"source":["### (Optional) Hand-tuning the parameters\n","\n","To get a feel for the impact of specific design choices on the performance of the network, we can experiment with the following hyperparameters / techniques:\n","\n","-   Data augmentation\n","-   Weight initialization\n","-   Number of layers, or number of layer features\n","-   Type of optimizer\n","-   Learning rate and/or schedule\n","-   Regularization"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"background_save":true},"id":"1Im86rmaeI3X"},"outputs":[],"source":["import matplotlib.pyplot as plt\n","\n","learning_rates = [1e-3, 1e-4]\n","\n","optimizers = [torch.optim.SGD, torch.optim.Adam]\n","\n","num_epochs = 10\n","\n","for lr in learning_rates:\n","  for opt in optimizers:\n","    print(f\"Learning rate: {lr}  Optimizer: {opt}\")\n","    losses = []\n","    for t in range(num_epochs):\n","      print(f\"Epoch {t+1}\\n-------------------------------\")\n","      epoch_loss = train_loop(train_dataloader, model, loss_fn, optimizer=opt(model.parameters(), lr=lr))\n","      losses.append(epoch_loss.item())\n","      print(losses)\n","    plt.plot(range(0, num_epochs), losses, label = f\"lr: {lr}, optimizer: {opt}\")\n","\n","plt.xlabel('Epoch')\n","plt.ylabel('Training Loss')\n","plt.legend()\n","figsizem = (10, 5)\n","plt.show()\n"]},{"cell_type":"markdown","metadata":{"id":"a2R5EvniIpzT"},"source":["##7) Save and Load the Model"]},{"cell_type":"markdown","metadata":{"id":"4d-pfynyIVcw"},"source":["In this section we will look at how to persist model state with saving,\n","loading and running model predictions.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NQAfYmu0IVcx"},"outputs":[],"source":["import torch\n","import torchvision.models as models"]},{"cell_type":"markdown","metadata":{"id":"yowyErQwIVcx"},"source":["###Saving and Loading Model Weights\n","\n","PyTorch models store the learned parameters in an internal state\n","dictionary, called `state_dict`. These can be persisted via the\n","`torch.save` method:\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"T_nTk1mqIVcx"},"outputs":[],"source":["model = models.vgg16(weights='IMAGENET1K_V1')\n","torch.save(model.state_dict(), 'model_weights.pth')"]},{"cell_type":"markdown","metadata":{"id":"2mfsEJs6IVcy"},"source":["To load model weights, **you need to create an instance of the same model\n","first**, and then load the parameters using `load_state_dict()` method.\n","\n","In the code below, we set `weights_only=True` to limit the functions\n","executed during unpickling to only those necessary for loading weights.\n","Using `weights_only=True` is considered a best practice when loading\n","weights.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"GwxRQmTTIVcy"},"outputs":[],"source":["model = models.vgg16() # we do not specify ``weights``, i.e. create untrained model\n","model.load_state_dict(torch.load('model_weights.pth', weights_only=True))\n","model.eval()"]},{"cell_type":"markdown","metadata":{"id":"XkGck9tLIVcy"},"source":["**Note:** be sure to call <code>model.eval()</code> method before inferencing to set the dropout and batch normalization layers to evaluation mode. Failing to do this will yield inconsistent inference results.\n","\n"]},{"cell_type":"markdown","metadata":{"id":"NKd7oZP9h14B"},"source":["##8) Experiment Tracking"]},{"cell_type":"markdown","metadata":{"id":"8p94aMZNiV73"},"source":["This part is adapted from the official Weights & Biases Tutorial\n","https://colab.research.google.com/github/wandb/examples/blob/master/colabs/intro/Intro_to_Weights_&_Biases.ipynb#scrollTo=O6sNs0CEi-TU"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"uPAMl2anh4UJ"},"outputs":[],"source":["# Log in to your W&B account\n","import wandb\n","import random\n","import math"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"KEExgZXui1gT"},"outputs":[],"source":["run = wandb.init()"]},{"cell_type":"markdown","metadata":{"id":"3SbVl_jIi-TV"},"source":["### Simulate and track a machine learning experiment with W&B\n","\n","Create, track, and visualize a machine learning experiment. To do this:\n","\n","1. Initialize a [W&B run](https://docs.wandb.ai/guides/runs) and pass in the hyperparameters you want to track.\n","2. Within your training loop, log metrics such as the accuracy and loss."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"PbC8n9yei-TV"},"outputs":[],"source":["import random\n","import math\n","\n","# Launch 5 simulated experiments\n","total_runs = 5\n","for run in range(total_runs):\n","  # 1️. Start a new run to track this script\n","  wandb.init(\n","      # Set the project where this run will be logged\n","      project=\"basic-intro\",\n","      # We pass a run name (otherwise it’ll be randomly assigned, like sunshine-lollypop-10)\n","      name=f\"experiment_{run}\",\n","      # Track hyperparameters and run metadata\n","      config={\n","        \"learning_rate\": 0.02,\n","        \"architecture\": \"CNN\",\n","        \"dataset\": \"CIFAR-100\",\n","        \"epochs\": 10,\n","      })\n","\n","  # This simple block simulates a training loop logging metrics\n","  epochs = 10\n","  offset = random.random() / 5\n","  for epoch in range(2, epochs):\n","      acc = 1 - 2 ** -epoch - random.random() / epoch - offset\n","      loss = 2 ** -epoch + random.random() / epoch + offset\n","\n","      # 2️. Log metrics from your script to W&B\n","      wandb.log({\"acc\": acc, \"loss\": loss})\n","\n","  # Mark the run as finished\n","  wandb.finish()"]},{"cell_type":"markdown","metadata":{"id":"9fiSvNO0wI1-"},"source":["### Tracking on Fashion MNIST"]},{"cell_type":"markdown","source":["Let's add tracking to the Fashion MNIST Training:"],"metadata":{"id":"aeyQQQ2DYquM"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"-pISq5UsqSqc"},"outputs":[],"source":["def train_loop_with_tracking(dataloader, model, loss_fn, optimizer, epoch):\n","    size = len(dataloader.dataset)\n","    # Set the model to training mode - important for batch normalization and dropout layers\n","    # Unnecessary in this situation but added for best practices\n","    model.train()\n","    for batch, (X, y) in enumerate(dataloader):\n","        # Compute prediction and loss\n","        pred = model(X)\n","        loss = loss_fn(pred, y)\n","\n","        # Backpropagation\n","        loss.backward()\n","        optimizer.step()\n","        optimizer.zero_grad()\n","\n","        if batch % 100 == 0:\n","            loss, current = loss.item(), batch * batch_size + len(X)\n","            ############tracking################\n","            # Log train metrics to wandb\n","            wandb.log({\n","                \"train/loss\": loss,\n","                \"train/epoch\": epoch\n","            })\n","            ############tracking################\n","            print(f\"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]\")\n","\n","\n","def test_loop_with_tracking(dataloader, model, loss_fn, epoch):\n","    # Set the model to evaluation mode - important for batch normalization and dropout layers\n","    # Unnecessary in this situation but added for best practices\n","    model.eval()\n","    size = len(dataloader.dataset)\n","    num_batches = len(dataloader)\n","    test_loss, correct = 0, 0\n","\n","    # Evaluating the model with torch.no_grad() ensures that no gradients are computed during test mode\n","    # also serves to reduce unnecessary gradient computations and memory usage for tensors with requires_grad=True\n","    with torch.no_grad():\n","        for X, y in dataloader:\n","            pred = model(X)\n","            test_loss += loss_fn(pred, y).item()\n","            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n","\n","    test_loss /= num_batches\n","    correct /= size\n","    print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")\n","    ############tracking################\n","    wandb.log({\n","        \"eval/loss\": test_loss,\n","        \"eval/epoch\": epoch,\n","        \"eval/accuracy\": 100*correct\n","    })\n","    ############tracking################"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"qh0Cw5InqSqd"},"outputs":[],"source":["model = NeuralNetwork()\n","\n","for learning_rate_loop in [1e-3, 1e-1, 1e-5]:\n","  wandb.init(\n","      project=\"pytorch-intro\",\n","      config={\n","          \"epochs\": 10,\n","          \"batch_size\": batch_size,\n","          \"lr\": learning_rate_loop,\n","    })\n","  wandb.define_metric(\"eval/loss\", summary=\"min\")\n","  wandb.define_metric(\"eval/accuracy\", summary=\"max\")\n","\n","  loss_fn = nn.CrossEntropyLoss()\n","  optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate_loop)\n","\n","  epochs = 3\n","  for t in range(epochs):\n","      print(f\"Epoch {t+1}\\n-------------------------------\")\n","      train_loop_with_tracking(train_dataloader, model, loss_fn, optimizer, t)\n","      test_loop_with_tracking(test_dataloader, model, loss_fn, t)\n","  print(\"Done!\")\n","  # Close your wandb run\n","  wandb.finish()"]},{"cell_type":"markdown","metadata":{"id":"Td0mQW91wx2L"},"source":["### Optional: Hyperparameter Optimization\n","This part is adapted from the official Weights & Biases Tutorial: https://docs.wandb.ai/models/sweeps/walkthrough"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"DXKXxRNdw2aD"},"outputs":[],"source":["# Import the W&B Python Library and log into W&B\n","import wandb\n","\n","# 1: Define objective/training function\n","def objective(config):\n","    score = config.x**3 + config.y\n","    return score\n","\n","def main():\n","    with wandb.init(project=\"my-first-sweep\") as run:\n","        score = objective(run.config)\n","        run.log({\"score\": score})\n","\n","# 2: Define the search space\n","sweep_configuration = {\n","    \"method\": \"random\",\n","    \"metric\": {\"goal\": \"minimize\", \"name\": \"score\"},\n","    \"parameters\": {\n","        \"x\": {\"max\": 0.1, \"min\": 0.01},\n","        \"y\": {\"values\": [1, 3, 7]},\n","    },\n","}\n","\n","# 3: Start the sweep\n","sweep_id = wandb.sweep(sweep=sweep_configuration, project=\"my-first-sweep\")\n","\n","wandb.agent(sweep_id, function=main, count=10)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"zbDo7ndIyyvt"},"outputs":[],"source":[]}],"metadata":{"colab":{"collapsed_sections":["SCXBFC4hNqzm","MzzZRuT5J-4g","zn-Dx7LdJ-4h","s9yS6FQSJ-4i","llc_21hOJCD8","a2R5EvniIpzT","yowyErQwIVcx","abznZAVuIVcy","NKd7oZP9h14B","3SbVl_jIi-TV","9fiSvNO0wI1-","Td0mQW91wx2L"],"provenance":[{"file_id":"1yrpzA6V0ALlV-kmriVwJxuLMJO5MYzjZ","timestamp":1772049201265}],"gpuType":"T4"},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}