{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[{"file_id":"1OHWLzpLiRvRft8qW8roWF9a2cYKQeGjw","timestamp":1705611473328}],"collapsed_sections":["01M2gpL0MDdx"]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"cells":[{"cell_type":"markdown","source":["# Recitation 0: CONTENTS\n","1. Pytorch basics\n","2. Model training\n","3. LSTM Basics\n","4. Wandb logging\n","5. einops library"],"metadata":{"id":"18iFNel8sVzV"}},{"cell_type":"markdown","metadata":{"id":"F61A7Pmh6cl2"},"source":["**Note: please make a COPY of the notebook if you'll be making any changes or running any cells**\n","\n","We'll be using pytorch for several coding assignments in this class. This tutorial covers:\n","* Setting up a training and evaluation loop\n","* Defining a data iterator, nn.Module, loss and evaluation functions, etc.\n","* Implementing a forward and a backward pass in custom torch autograd functions\n","\n","We'll be using sentiment classification of IMDB reviews as our running example."]},{"cell_type":"markdown","source":["# Pytorch basics"],"metadata":{"id":"TtpfXF5tsqml"}},{"cell_type":"markdown","source":["## Tensors\n","\n","PyTorch defines and uses Tensors for all of its operations, which are very similar to numpy ndarrays. Like ndarrays, PyTorch tensors are also multidimensional arrays of data with a specified datatype. But PyTorch tensors and operations defined in PyTorch over them let you easily transition to using GPUs or TPUs to speed up computation. And most importantly, PyTorch has autograd, which automatically constructs a computational graph making it easy to compute gradients for each tensor since most supported tensor operations come with gradient functions defined."],"metadata":{"id":"fvIZwmnq6hVw"}},{"cell_type":"markdown","source":["#### Creating Tensors"],"metadata":{"id":"7Agp8mhm9oIC"}},{"cell_type":"code","source":["import torch\n","import numpy as np\n","\n","# You can make a tensor directly from data\n","data = [[1., 2.], [3., 4.]]  # list\n","x_data = torch.tensor(data)\n","print(x_data)\n","print(x_data.dtype)\n","print()\n","\n","# Call tensor.numpy() to get a numpy array.\n","# (Note that this only works on detached cpu tensors, should be clear after autograd is explained)\n","print(x_data.numpy())"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"GmXnL0zP5csi","outputId":"78ca9ef7-e6e0-450b-803a-1461eb6ee044","executionInfo":{"status":"ok","timestamp":1756487218407,"user_tz":240,"elapsed":5331,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[1., 2.],\n","        [3., 4.]])\n","torch.float32\n","\n","[[1. 2.]\n"," [3. 4.]]\n"]}]},{"cell_type":"code","source":["# You can also make a tensor from a numpy array\n","data = [[1., 2.], [3., 4.]]\n","np_data = np.asarray(data, dtype=np.float32)\n","x_data = torch.from_numpy(np_data)\n","\n","print(x_data)\n","print(x_data.dtype)\n","\n","# Note:\n","# While this method to construct arrays is very common,\n","# the np array and tensor share the same underlying memory.\n","# So, changing the ndarray would change the tensor and vice-versa.\n","np_data[0][0] = -1\n","print(x_data)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"K56lhW905cqW","outputId":"0e7d17dd-cf04-4a51-d2d0-56709be21453","executionInfo":{"status":"ok","timestamp":1756474786968,"user_tz":240,"elapsed":8,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[1., 2.],\n","        [3., 4.]])\n","torch.float32\n","tensor([[-1.,  2.],\n","        [ 3.,  4.]])\n"]}]},{"cell_type":"code","source":["# A few other useful ways to initialize tensors with constant or random values\n","\n","# Create an array of all ones:\n","print(\"torch.ones((2, 2, 2)) creates a (2, 2, 2) tensor of ones:\\n\", torch.ones((2, 2, 2)))\n","print()\n","\n","# Create an array of all zeros:\n","print(\"torch.zeros((2, 2, 2)) creates a (2, 2, 2) tensor of zeros:\\n\", torch.zeros((2, 2, 2)))\n","print()\n","\n","# Create an array of random floats ~ Unif[0, 1):\n","print(\"torch.rand((2, 2, 2)) creates a (2, 2, 2) tensor from the uniform distribution :\\n\", torch.rand((2, 2, 2)))\n","print()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"jzu58Ys56em7","outputId":"98cf3e6b-82b5-4ee6-e738-30c28d1de706","executionInfo":{"status":"ok","timestamp":1756474786971,"user_tz":240,"elapsed":6,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["torch.ones((2, 2, 2)) creates a (2, 2, 2) tensor of ones:\n"," tensor([[[1., 1.],\n","         [1., 1.]],\n","\n","        [[1., 1.],\n","         [1., 1.]]])\n","\n","torch.zeros((2, 2, 2)) creates a (2, 2, 2) tensor of zeros:\n"," tensor([[[0., 0.],\n","         [0., 0.]],\n","\n","        [[0., 0.],\n","         [0., 0.]]])\n","\n","torch.rand((2, 2, 2)) creates a (2, 2, 2) tensor from the uniform distribution :\n"," tensor([[[0.9982, 0.6047],\n","         [0.3191, 0.4977]],\n","\n","        [[0.1317, 0.9866],\n","         [0.7986, 0.5984]]])\n","\n"]}]},{"cell_type":"markdown","source":["### Tensor operations"],"metadata":{"id":"tblCvmoS9sdW"}},{"cell_type":"code","source":["# clone() creates a copy of a tensor\n","x = torch.rand((2,))\n","x_clone = x.clone()\n","\n","print(x, x_clone)\n","x_clone[0] = 0.\n","\n","print(x, x_clone)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"HO8kSYo16ekx","outputId":"da73b024-b0d6-4d99-ea82-a63258b3e68e","executionInfo":{"status":"ok","timestamp":1756474786976,"user_tz":240,"elapsed":5,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([0.6560, 0.0473]) tensor([0.6560, 0.0473])\n","tensor([0.6560, 0.0473]) tensor([0.0000, 0.0473])\n"]}]},{"cell_type":"code","source":["# Standard numpy style slicing and indexing works, as well as standard arithmetic operations\n","x = torch.rand((4, 4))\n","print(\"x:\", x)\n","\n","print(\"Slice:\\n\", x[:, 1: 3])\n","\n","print(\"\\nElement-wise arithmetic ops:\")\n","\n","x = torch.ones((2, 2))\n","y = torch.ones((2, 2))\n","print(\"x:\", x)\n","print(\"y:\", y)\n","print(\"x + y:\\n\", x + y)\n","print(\"x - y:\\n\", x - y)\n","print(\"x * y:\\n\", x * y) # element-wise multiplication, equivalent to torch.mul(x, y)\n","print(\"x / y:\\n\", x / y)\n","\n","print(\"Matrix multiplication:\")\n","print(\"x @ y:\\n\", x @ y) # equivalent to torch.matmul(x, y)\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mNRMVMqc5YpB","outputId":"848845ae-8902-471b-e297-200d4cbabee8","executionInfo":{"status":"ok","timestamp":1756474786993,"user_tz":240,"elapsed":16,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["x: tensor([[0.8086, 0.1223, 0.0185, 0.1304],\n","        [0.4712, 0.3871, 0.5572, 0.5455],\n","        [0.4246, 0.2166, 0.6691, 0.6405],\n","        [0.5200, 0.9163, 0.0260, 0.5459]])\n","Slice:\n"," tensor([[0.1223, 0.0185],\n","        [0.3871, 0.5572],\n","        [0.2166, 0.6691],\n","        [0.9163, 0.0260]])\n","\n","Element-wise arithmetic ops:\n","x: tensor([[1., 1.],\n","        [1., 1.]])\n","y: tensor([[1., 1.],\n","        [1., 1.]])\n","x + y:\n"," tensor([[2., 2.],\n","        [2., 2.]])\n","x - y:\n"," tensor([[0., 0.],\n","        [0., 0.]])\n","x * y:\n"," tensor([[1., 1.],\n","        [1., 1.]])\n","x / y:\n"," tensor([[1., 1.],\n","        [1., 1.]])\n","Matrix multiplication:\n","x @ y:\n"," tensor([[2., 2.],\n","        [2., 2.]])\n"]}]},{"cell_type":"code","source":["# Use torch.cat to concatenate two tensors along an existing dimension\n","\n","x = torch.zeros((2, 2))\n","y = torch.ones((2, 2))\n","\n","print(\"torch.cat((x, y), dim=0):\\n\", torch.cat((x, y), dim=0))\n","print(\"torch.cat((x, y), dim=1):\\n\", torch.cat((x, y), dim=1))\n","# print(\"torch.cat((x, y), dim=1):\\n\", torch.cat((x, y), dim=2))\n","\n","\n","print()\n","\n","# Use torch.stack to stack tensors of equal dimension along a new axis\n","\n","a = torch.zeros((2, 2))\n","b = torch.ones((2, 2))\n","\n","stacked = torch.stack((a, b), dim=0)\n","print(\"stacked tensor:\\n\", stacked)\n","print(\"stacked tensor shape:\\n\", stacked.shape)\n","new = torch.stack((stacked, stacked), dim=0)\n","print(new.shape)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Upw27Rhb8Y0w","outputId":"ea6ddd83-4d18-43e5-81b9-18154d1f2ff1","executionInfo":{"status":"ok","timestamp":1756491067523,"user_tz":240,"elapsed":13,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["torch.cat((x, y), dim=0):\n"," tensor([[0., 0.],\n","        [0., 0.],\n","        [1., 1.],\n","        [1., 1.]])\n","torch.cat((x, y), dim=1):\n"," tensor([[0., 0., 1., 1.],\n","        [0., 0., 1., 1.]])\n","\n","stacked tensor:\n"," tensor([[[0., 0.],\n","         [0., 0.]],\n","\n","        [[1., 1.],\n","         [1., 1.]]])\n","stacked tensor shape:\n"," torch.Size([2, 2, 2])\n","torch.Size([2, 2, 2, 2])\n"]}]},{"cell_type":"code","source":["# Use tensor.reshape to reshape a tensor to a required dimension\n","\n","# Create tensor\n","tensor = torch.tensor([1, 2, 3, 4, 5, 6])\n","\n","# Reshape to 2x3\n","reshaped_tensor = tensor.reshape(2, 3)\n","print(reshaped_tensor)"],"metadata":{"id":"3S_o8Rkntp-s","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1756474787003,"user_tz":240,"elapsed":3,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}},"outputId":"55f769c4-42a1-4f73-e899-5ae7543e2095"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[1, 2, 3],\n","        [4, 5, 6]])\n"]}]},{"cell_type":"code","source":["# Use tensor.view to view a tensor in a particular dimension\n","\n","# Create tensor\n","tensor = torch.tensor([1, 2, 3, 4, 5, 6])\n","\n","# View as 2x3\n","viewed_tensor = tensor.view(2, 3)\n","print(viewed_tensor)"],"metadata":{"id":"l_311sh2uDuj","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1756474787026,"user_tz":240,"elapsed":22,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}},"outputId":"41d41ca1-d301-42a0-cacc-105fcf0d2f60"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[1, 2, 3],\n","        [4, 5, 6]])\n"]}]},{"cell_type":"code","source":["# Use tensor.permute to reorder dimensions without changing the data\n","\n","# Create tensor\n","tensor = torch.tensor([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])\n","print(f\"{tensor}\\n\")\n","\n","# Permute dimensions (0, 2, 1 means we swap dimensions 1 and 2)\n","permuted_tensor = tensor.permute(0, 2, 1)\n","print(permuted_tensor) # This creates a non-contiguous tensor"],"metadata":{"id":"WBGGAb-Ewibf","executionInfo":{"status":"ok","timestamp":1756474787032,"user_tz":240,"elapsed":6,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}},"outputId":"4da8c931-5a1c-40f3-ce99-325813b0c997","colab":{"base_uri":"https://localhost:8080/"}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[[1, 2],\n","         [3, 4]],\n","\n","        [[5, 6],\n","         [7, 8]]])\n","\n","tensor([[[1, 3],\n","         [2, 4]],\n","\n","        [[5, 7],\n","         [6, 8]]])\n"]}]},{"cell_type":"markdown","source":["Some other useful operations to read into from the pytorch documentation\n","- view (reshapes array, but only works on contiguous arrays: https://discuss.pytorch.org/t/difference-between-view-reshape-and-permute/54157)\n","- reshape\n","- transpose\n","- squeeze\n","- unsqueeze\n","- sum\n","- mean\n"],"metadata":{"id":"7U1LbZRQ_Y5e"}},{"cell_type":"markdown","source":["##Building Computation Graphs with Tensors"],"metadata":{"id":"DlV13l-Z7LVq"}},{"cell_type":"code","source":["# Automatic differentiation is enabled on most pytorch operations, which get tracked through the computation graph\n","x = torch.rand((2, 2), requires_grad=True)\n","x"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9Rg3AJ8E_Yr7","outputId":"dff5a350-fc94-4b82-eb0a-39db75db4b03","executionInfo":{"status":"ok","timestamp":1756474787049,"user_tz":240,"elapsed":16,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[0.0533, 0.6825],\n","        [0.0477, 0.6766]], requires_grad=True)"]},"metadata":{},"execution_count":10}]},{"cell_type":"code","source":["# For example, the slice \"operation\" has its own backward gradient function defined\n","x[1:2]"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"C63Cc441A95E","outputId":"6e93d275-81c3-4b12-8936-201b106abe10","executionInfo":{"status":"ok","timestamp":1756474787064,"user_tz":240,"elapsed":15,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[0.0477, 0.6766]], grad_fn=<SliceBackward0>)"]},"metadata":{},"execution_count":11}]},{"cell_type":"code","source":["# Or the sum operation\n","x.sum()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9eu08aeyA92d","outputId":"f788c3ce-4c4e-401b-eac1-4564c681c902","executionInfo":{"status":"ok","timestamp":1756474787083,"user_tz":240,"elapsed":18,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor(1.4601, grad_fn=<SumBackward0>)"]},"metadata":{},"execution_count":12}]},{"cell_type":"code","source":["# We can call a backward() pass on any pytorch scalar attached to a computation graph\n","x = torch.rand((2, 2), requires_grad=True)\n","print(\"x: \", x)\n","\n","y = x * 2\n","print(\"y: \", y)\n","\n","z = y[0]\n","print(\"z: \", z)\n","\n","loss = z.sum()\n","print(loss)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"RMYIy_fBB_Zs","outputId":"d2f1d3c8-e799-4f27-ce68-46b2ad0b5e83","executionInfo":{"status":"ok","timestamp":1756474787106,"user_tz":240,"elapsed":24,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["x:  tensor([[0.2740, 0.1585],\n","        [0.9953, 0.0444]], requires_grad=True)\n","y:  tensor([[0.5480, 0.3170],\n","        [1.9906, 0.0888]], grad_fn=<MulBackward0>)\n","z:  tensor([0.5480, 0.3170], grad_fn=<SelectBackward0>)\n","tensor(0.8650, grad_fn=<SumBackward0>)\n"]}]},{"cell_type":"code","source":["# Leaf variables in the computation graph (like x here) have a gradient attached\n","#  which is by default None\n","print(x.grad)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"aMwDzRhw8YwI","outputId":"e73ae514-f39a-4c55-9675-ea61cfc7a855","executionInfo":{"status":"ok","timestamp":1756474787149,"user_tz":240,"elapsed":45,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["None\n"]}]},{"cell_type":"code","source":["# A .backward() call runs a backward pass through the computation graph and populates the .grad for leaf variables\n","loss.backward()\n","\n","print(x.grad)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"SFuRJMdeDTjq","outputId":"151e3ba5-6d75-496a-af37-f930fcb92c6f","executionInfo":{"status":"ok","timestamp":1756474787149,"user_tz":240,"elapsed":8,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[2., 2.],\n","        [0., 0.]])\n"]}]},{"cell_type":"code","source":["# Non leaf variables do not have their .grad populated after a backward pass\n","#  If interested, look into \"retain_graph\" here: https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html\n","print(y.grad)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"0HNiPCqzDAX0","outputId":"199abc9e-40ca-4619-a499-53f214f4e18a","executionInfo":{"status":"ok","timestamp":1756474787187,"user_tz":240,"elapsed":41,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["None\n"]},{"output_type":"stream","name":"stderr","text":["/tmp/ipython-input-1776939999.py:3: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /pytorch/build/aten/src/ATen/core/TensorBody.h:489.)\n","  print(y.grad)\n"]}]},{"cell_type":"markdown","source":["# Model training"],"metadata":{"id":"PtmpU8a-syUk"}},{"cell_type":"markdown","source":["## Datasets & DataLoaders\n","\n","Code for processing data samples can get messy and hard to maintain; we ideally want our dataset code to be decoupled from our model training code for better readability and modularity. PyTorch provides two data primitives: torch.utils.data.DataLoader and torch.utils.data.Dataset that allow you to use pre-loaded datasets as well as your own data. Dataset stores the samples and their corresponding labels, and DataLoader wraps an iterable around the Dataset to enable easy access to the samples.\n","\n","PyTorch domain libraries provide a number of pre-loaded datasets (such as FashionMNIST) that subclass torch.utils.data.Dataset and implement functions specific to the particular data. They can be used to prototype and benchmark your model. You can find them here: [Image Datasets](https://pytorch.org/vision/stable/datasets.html),\n","[Text Datasets](https://pytorch.org/text/stable/datasets.html), and\n","[Audio Datasets](https://pytorch.org/audio/stable/datasets.html)"],"metadata":{"id":"3tqeTXf8L1JO"}},{"cell_type":"markdown","source":["#### Loading a Dataset\n","\n","Here is an example of how to load the [Fashion-MNIST](https://research.zalando.com/project/fashion_mnist/fashion_mnist/) dataset from TorchVision.\n","Fashion-MNIST is a dataset of Zalando’s article images consisting of 60,000 training examples and 10,000 test examples.\n","Each example comprises a 28×28 grayscale image and an associated label from one of 10 classes.\n","\n","We load the [FashionMNIST Dataset](https://pytorch.org/vision/stable/datasets.html#fashion-mnist) with the following parameters:\n"," - ``root`` is the path where the train/test data is stored,\n"," - ``train`` specifies training or test dataset,\n"," - ``download=True`` downloads the data from the internet if it's not available at ``root``.\n"," - ``transform`` and ``target_transform`` specify the feature and label transformations"],"metadata":{"id":"5rleJcblMDPU"}},{"cell_type":"code","source":["import torch\n","from torch.utils.data import Dataset\n","from torchvision import datasets\n","from torchvision.transforms import ToTensor\n","import matplotlib.pyplot as plt\n","\n","\n","training_data = datasets.FashionMNIST(\n","    root=\"data\",\n","    train=True,\n","    download=True,\n","    transform=ToTensor()\n",")\n","\n","test_data = datasets.FashionMNIST(\n","    root=\"data\",\n","    train=False,\n","    download=True,\n","    transform=ToTensor()\n",")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"AhhYuRs0MPAT","executionInfo":{"status":"ok","timestamp":1756487241191,"user_tz":240,"elapsed":11856,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"f517fc98-fdde-46b0-bab5-2ac742d9ffdb"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["100%|██████████| 26.4M/26.4M [00:01<00:00, 21.2MB/s]\n","100%|██████████| 29.5k/29.5k [00:00<00:00, 350kB/s]\n","100%|██████████| 4.42M/4.42M [00:00<00:00, 6.27MB/s]\n","100%|██████████| 5.15k/5.15k [00:00<00:00, 6.08MB/s]\n"]}]},{"cell_type":"markdown","source":["#### Iterating and Visualizing the Dataset\n","\n","We can index ``Datasets`` manually like a list: ``training_data[index]``.\n","We use ``matplotlib`` to visualize some samples in our training data."],"metadata":{"id":"D87wZsrWMXYr"}},{"cell_type":"code","source":["labels_map = {\n","    0: \"T-Shirt\",\n","    1: \"Trouser\",\n","    2: \"Pullover\",\n","    3: \"Dress\",\n","    4: \"Coat\",\n","    5: \"Sandal\",\n","    6: \"Shirt\",\n","    7: \"Sneaker\",\n","    8: \"Bag\",\n","    9: \"Ankle Boot\",\n","}\n","figure = plt.figure(figsize=(8, 8))\n","cols, rows = 3, 3\n","for i in range(1, cols * rows + 1):\n","    sample_idx = torch.randint(len(training_data), size=(1,)).item()\n","    img, label = training_data[sample_idx]\n","    figure.add_subplot(rows, cols, i)\n","    plt.title(labels_map[label])\n","    plt.axis(\"off\")\n","    plt.imshow(img.squeeze(), cmap=\"gray\")\n","plt.show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":675},"id":"O14Te5l0MTBt","executionInfo":{"status":"ok","timestamp":1756474857143,"user_tz":240,"elapsed":455,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}},"outputId":"3839fdfe-41da-4e5c-b126-3db0d1d8d86c"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 800x800 with 9 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAn4AAAKSCAYAAABMVtaZAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbDtJREFUeJzt3Xl4VeW1+PEVIDmZB0KYIYEAMjlcEEFQAUW5AuIEglYFq2Cd0Ivz0Cra1utUtVpEWqsUQRAFcWAoCmgFW8CBKyBTDCBTQoBA5kCyf3/wcH7GvOuFszlhyPv9PA9/sPZZ57zn5Oy9F5ustSM8z/MEAAAAtV6dE70AAAAAHB8UfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR1D4AQAAOILCDwAAwBEUfgAAQEaOHCnx8fFHfFyfPn2kT58+Nb8g1AgKvxqQlZUlt956q7Ru3Vqio6MlMTFRevXqJS+//LKUlJTUyGtOnTpVXnrppRp5bqAmRUREHNWfxYsXn+ilAied8ePHS0REhHTv3v1EL8W3kSNHVtnX69WrJy1atJDhw4fLmjVravS1i4uL5YknnnDq+FLvRC+gtvnkk09k6NChEggE5MYbb5TOnTtLeXm5fPnll3L//ffL6tWrZeLEiWF/3alTp8qqVavknnvuCftzAzVp8uTJVf7+j3/8QxYsWFAt3qFDh+O5LOCUMGXKFMnIyJBly5bJxo0bpU2bNid6Sb4EAgH529/+JiIiBw8elKysLJkwYYLMmzdP1qxZI02bNq2R1y0uLpZx48aJiDhzFZPCL4yys7Nl+PDhkp6eLgsXLpQmTZoEt91xxx2yceNG+eSTT07gCoGTz/XXX1/l7//+979lwYIF1eK/VFxcLLGxsTW5tBpRVFQkcXFxJ3oZqAWys7Nl6dKlMnPmTLn11ltlypQp8vjjj5/oZflSr169avt8jx49ZNCgQfLJJ5/IqFGjTtDKah/+qzeMnn32WSksLJQ33nijStF3WJs2beTuu+8WkUP/onnqqackMzNTAoGAZGRkyCOPPCJlZWVVcmbPni0DBw6Upk2bSiAQkMzMTHnqqaekoqIi+Jg+ffrIJ598Ips3bw5eKs/IyKjR9wocT3369JHOnTvL119/LRdccIHExsbKI488IiIiubm5cvPNN0ujRo0kOjpazjzzTJk0aVKV/MWLFxv/u3jTpk0SEREhb731VjC2c+dOuemmm6R58+YSCASkSZMmcvnll8umTZuq5M6dO1fOP/98iYuLk4SEBBk4cKCsXr26ymMO/85UVlaWDBgwQBISEuRXv/pV2D4XuG3KlCmSkpIiAwcOlCFDhsiUKVOqPebwd/z555+XiRMnBs853bp1k+XLlx/xNb777jtJS0uTPn36SGFhofq4srIyefzxx6VNmzYSCASkRYsW8sADD1Q7p4WicePGInKoKPy5H3/8UYYOHSr169eX2NhY6dGjh/GiypGODZs2bZK0tDQRERk3blzw/PnEE0/4XvOpgCt+YfTRRx9J69atpWfPnkd87C233CKTJk2SIUOGyL333iv/+c9/5Omnn5YffvhBZs2aFXzcW2+9JfHx8TJ27FiJj4+XhQsXyu9+9zvZv3+/PPfccyIi8uijj8q+fftk69at8uKLL4qIHNUv6AKnkt27d8ull14qw4cPl+uvv14aNWokJSUl0qdPH9m4caPceeed0qpVK5kxY4aMHDlS8vPzg//QCsXVV18tq1evlrvuuksyMjIkNzdXFixYIFu2bAn+g2ry5MkyYsQI6d+/vzzzzDNSXFwsr732mpx33nny7bffVvmH18GDB6V///5y3nnnyfPPP39KXqXEyWnKlCly1VVXSVRUlFx77bXy2muvyfLly6Vbt27VHjt16lQpKCiQW2+9VSIiIuTZZ5+Vq666Sn788UeJjIw0Pv/y5culf//+cvbZZ8vs2bMlJibG+LjKykoZPHiwfPnllzJ69Gjp0KGDfP/99/Liiy/K+vXr5YMPPjiq95OXlyciIhUVFfLjjz/Kgw8+KKmpqTJo0KDgY3JycqRnz55SXFwsY8aMkdTUVJk0aZIMHjxY3nvvPbnyyitFRI7q2JCWliavvfaa3HbbbXLllVfKVVddJSIiZ5xxxlGt95TlISz27dvniYh3+eWXH/Gx3333nSci3i233FIlft9993ki4i1cuDAYKy4urpZ/6623erGxsV5paWkwNnDgQC89Pd33+oGTxR133OH98tDUu3dvT0S8CRMmVIm/9NJLnoh4b7/9djBWXl7unXvuuV58fLy3f/9+z/M8b9GiRZ6IeIsWLaqSn52d7YmI9+abb3qe53l79+71RMR77rnn1PUVFBR4ycnJ3qhRo6rEd+7c6SUlJVWJjxgxwhMR76GHHjrq9w8cjRUrVngi4i1YsMDzPM+rrKz0mjdv7t19991VHnf4O56amurt2bMnGJ89e7YnIt5HH30UjI0YMcKLi4vzPM/zvvzySy8xMdEbOHBglXON5x3aH3v37h38++TJk706dep4//rXv6o8bsKECZ6IeEuWLLG+l8P7yS//NGvWzPv666+rPPaee+7xRKTKaxUUFHitWrXyMjIyvIqKCs/zjv7YsGvXLk9EvMcff9y6xtqE/+oNk/3794uISEJCwhEfO2fOHBERGTt2bJX4vffeKyJS5ZL1z/+FVVBQIHl5eXL++edLcXGxrF279pjXDZwqAoGA3HTTTVVic+bMkcaNG8u1114bjEVGRsqYMWOksLBQPv/885BeIyYmRqKiomTx4sWyd+9e42MWLFgg+fn5cu2110peXl7wT926daV79+6yaNGiajm33XZbSOsAjmTKlCnSqFEj6du3r4gc6o4fNmyYTJs2rcqvAh02bNgwSUlJCf79/PPPF5FD/236S4sWLZL+/fvLRRddJDNnzpRAIGBdy4wZM6RDhw7Svn37KvvEhRdeGHy+I4mOjpYFCxbIggULZP78+fL6669LfHy8DBgwQNavXx983Jw5c+Scc86R8847LxiLj4+X0aNHy6ZNm4JdwOE+NtQm/FdvmCQmJorIoeLsSDZv3ix16tSp1n3VuHFjSU5Ols2bNwdjq1evlscee0wWLlwYLC4P27dvXxhWDpwamjVrJlFRUVVimzdvlrZt20qdOlX/DXu4A/jn+9LRCAQC8swzz8i9994rjRo1Cv5y+Y033hj8faMNGzaIiARPar90+FhwWL169aR58+YhrQOwqaiokGnTpknfvn0lOzs7GO/evbu88MIL8tlnn8kll1xSJadly5ZV/n64CPzlP3BKS0tl4MCB0rVrV3n33Xer/X6dyYYNG+SHH34I/r7cL+Xm5h7xOerWrSv9+vWrEhswYIC0bdtWHn74YXn//fdF5NA+bRpd8/N9vnPnzmE/NtQmFH5hkpiYKE2bNpVVq1YddU5ERIR1e35+vvTu3VsSExPlySeflMzMTImOjpZvvvlGHnzwQamsrDzWZQOnDO33i46Gtq+Zrozcc889ctlll8kHH3wg8+fPl9/+9rfy9NNPy8KFC+W//uu/gvvd5MmTg8Xgz/3yRBkIBKqdfIBjsXDhQtmxY4dMmzZNpk2bVm37lClTqhV+devWNT6X53lV/h4IBGTAgAEye/ZsmTdvXpXfr9NUVlbK6aefLn/605+M21u0aHHE5zBp3ry5nHbaafLFF1/4yocZhV8YDRo0SCZOnChfffWVnHvuuerj0tPTpbKyUjZs2FBlNllOTo7k5+dLenq6iBzqRNy9e7fMnDlTLrjgguDjfv4vvMOOVEQCtVF6err83//9n1RWVlYprg7/GsThfenw1Y38/Pwq+dq/+jMzM+Xee++Ve++9VzZs2CBnnXWWvPDCC/L2229LZmamiIg0bNiw2hUK4HiYMmWKNGzYUP7yl79U2zZz5kyZNWuWTJgwwdc/liIiImTKlCly+eWXy9ChQ2Xu3LlHnG+XmZkpK1eulIsuuijs56KDBw9W6SZOT0+XdevWVXvcL/f5oz02uHju5J+hYfTAAw9IXFyc3HLLLZKTk1Nte1ZWlrz88ssyYMAAEZFqd9o4/K+lgQMHisj//xfaz/9FVl5eLuPHj6/23HFxcfzXL5wzYMAA2blzp0yfPj0YO3jwoLzyyisSHx8vvXv3FpFDB/m6detWu3Lwy32puLhYSktLq8QyMzMlISEhOJaif//+kpiYKH/84x/lwIED1da0a9eusLw3wKSkpERmzpwpgwYNkiFDhlT7c+edd0pBQYF8+OGHvl8jKipKZs6cKd26dZPLLrtMli1bZn38NddcI9u2bZO//vWvxvUWFRX5Wsf69etl3bp1cuaZZwZjAwYMkGXLlslXX30VjBUVFcnEiRMlIyNDOnbsGHzc0RwbDnfZ//IfhbUZV/zCKDMzU6ZOnSrDhg2TDh06VLlzx9KlS4Ot5HfffbeMGDFCJk6cGPzv3GXLlsmkSZPkiiuuCP6ybs+ePSUlJUVGjBghY8aMkYiICJk8eXK1S/MiIl27dpXp06fL2LFjpVu3bhIfHy+XXXbZ8f4IgONq9OjR8vrrr8vIkSPl66+/loyMDHnvvfdkyZIl8tJLLwWbrZKSkmTo0KHyyiuvSEREhGRmZsrHH39c7XeP1q9fLxdddJFcc8010rFjR6lXr57MmjVLcnJyZPjw4SJy6Nc6XnvtNbnhhhukS5cuMnz4cElLS5MtW7bIJ598Ir169ZJXX331uH8WcMOHH34oBQUFMnjwYOP2Hj16SFpamkyZMkWGDRvm+3ViYmLk448/lgsvvFAuvfRS+fzzz6Vz587Gx95www3y7rvvym9+8xtZtGiR9OrVSyoqKmTt2rXy7rvvyvz58+Xss8+2vt7Bgwfl7bffFpFD/3W8adMmmTBhglRWVlYZSv3QQw/JO++8I5deeqmMGTNG6tevL5MmTZLs7Gx5//33g1f3jvbYEBMTIx07dpTp06dLu3btpH79+tK5c2f1vdYKJ7iruFZav369N2rUKC8jI8OLioryEhISvF69enmvvPJKsC3+wIED3rhx47xWrVp5kZGRXosWLbyHH364Wtv8kiVLvB49engxMTFe06ZNvQceeMCbP39+tdEUhYWF3nXXXeclJyd7IsJoF5yytHEunTp1Mj4+JyfHu+mmm7wGDRp4UVFR3umnnx4cz/Jzu3bt8q6++movNjbWS0lJ8W699VZv1apVVca55OXleXfccYfXvn17Ly4uzktKSvK6d+/uvfvuu9Web9GiRV7//v29pKQkLzo62svMzPRGjhzprVixIviYn4/HAMLhsssu86Kjo72ioiL1MSNHjvQiIyO9vLy84DgX04gi+cUYE9P3NS8vz+vYsaPXuHFjb8OGDZ7nVR/n4nmHRqU888wzXqdOnbxAIOClpKR4Xbt29caNG+ft27fP+p5M41wSExO9iy66yPv000+rPT4rK8sbMmSIl5yc7EVHR3vnnHOO9/HHH1d73NEeG5YuXep17drVi4qKcmK0S4TnGS4fAQAAoNbhd/wAAAAcQeEHAADgCAo/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHDEUd+541S8n51tzeEcX9ilSxd129ChQ43xxYsXqznardfatWun5mhT0ceMGaPm+OHne3Ayj4o8Gdd2Ku5r4Xbvvfca49rdCkSkyv04f8623xQXFxvjpnuBHnb41m2/tH79ejXn/vvvV7e5gn0NOD6OtK9xxQ8AAMARFH4AAACOoPADAABwBIUfAACAIyK8o/yNW34JViQ1NdUYnzVrlpoTCASM8eTkZDWnvLzcGM/IyFBz9uzZY4yPGjVKzdEaTLTX90v7pfvKysqwvo4f/ML5yWnq1KnGeEJCgpqzevVqY/yss85Sc1q3bm2M//TTT2pObGysMd6sWTM1Z8CAAcb4qlWr1Jzahn0NOD5o7gAAAICIUPgBAAA4g8IPAADAERR+AAAAjqDwAwAAcASFHwAAgCOO+l69tU2bNm2M8dGjR6s5gwYNMsa3bNmi5tStW9cY10bDiOgjYLR7hIqIfPjhh8a4dq9gEZGxY8ca48uWLQv5dVasWKHmnAxjW3DysY0nqlfPfGjav3+/mhMZGWmM//jjj2qOn3Ee2rijbdu2qTn169cP+XUAoCZwxQ8AAMARFH4AAACOoPADAABwBIUfAACAIyj8AAAAHFErunq7dOlijN93331qjtY1ePDgQTXniy++MMa1Lj8RvUNX60AUETnzzDON8Tlz5qg52k3lbTe137lzpzGenJys5tx+++3GeGFhoZoza9YsY3zRokVqTp065n+T0CFce5x++unqtiZNmhjj3377rZrTvHlzY9y2r2n7bkFBgZpTUVFhjPfu3VvNufjii41x7ZgCADWFK34AAACOoPADAABwBIUfAACAIyj8AAAAHEHhBwAA4AgKPwAAAEecMuNcunfvrm575JFHjPE9e/aoOXv37g15DX5GiWhjVnJzc9WcNWvWGOMrV65Uc/zcBN7zPGM8Pz9fzSktLTXGtbE1IiK/+c1vjHHbKAttZEZERISao70fnJxiYmLUbTt27DDG8/Ly1JxmzZoZ40VFRWpOWlqaMa6NExLR94F169apOS1btlS3AcDxxBU/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHDEKdPV+9hjj6nbtC7UwsJCNUfbZuvmO3jwoDFu6/aNj483xuPi4tScAwcOGONaB6KN9lwieodsQkKCmqN129o6qLXPbeDAgWrOhx9+aIzTuVt72PaBffv2GePa/iQiEh0dbYzb9oF69cyHQNvaiouLjfHy8vKQc4DaxHb+1M43tvOnn+P9oEGDjPG5c+eqOdoabOdc7Vxo+wySkpKM8Y0bN6o5V1xxhTG+detWNedIuOIHAADgCAo/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHBEjY5z0dq3bS3abdq0McZtYxx27txpjNvGkmijF7Q123K0Fm1bjnajdxF9xIQWFxEpKyszxiMjI9UcbfyFNkpDRP/Z2T43rb29R48eao42zgW1h21kirZ/pKamqjna2JaOHTuqOdo+sGrVqpDX1rBhQzVnzZo16jagtrCNZtHYzh2a2NhYddv9999vjNtGwmnnqMzMTDVHO95ERUWpOSkpKcb4V199peZoNcS4cePUnCPhih8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR1D4AQAAOKJGu3r9OO+884xxWwegxnaD5V27dhnjdevWDfl1bLQbOWtxEZHc3FxjPBAIhPw6thvUa53ANkVFRcZ4enq6mqN1V9u6oVH72b5/2n5o2z8zMjKM8XfeeUfN6d69uzHeqlUrNWfz5s3GuLZviIjk5+er24Dawtahq02EsE350Dz44IPqtpKSEmNc644V0bttN23apOZok0Zsr6MdO2zHteHDhxvj2jSTo8EVPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOoPADAABwBIUfAACAI2p0nIufNu0zzzzTGNdatEX0US8tW7ZUc/bs2WOMb926Vc2JjIw0xm3t29rIClv7tp+bP2vPZxsbo72OLUf7mSYnJ6s5Bw8eNMbr1dO/ftHR0cZ4aWmpmoNTi+2G7tpYCFuOduwYMmSImqPdnP30009Xc7Q12I5RfkYnAacaP+Nc/GjdurW6rXHjxsa4bf/U1q2d80X0c5HtM9DYRs8dy9gWDVf8AAAAHEHhBwAA4AgKPwAAAEdQ+AEAADiCwg8AAMARNdrV60f79u2N8cLCQjWnuLg45Nfp1auXMb548WI1JycnJ+TX1zpnbZ0/Woeu9ly259M6am2vY/use/ToYYzbOqi//vprY7xp06ZqTnp6ujG+bt06NQe1h9YBaOuy07ptN27cqOYsXbrUGL/99tvVHD/Hm71794acA5xqbF33x4u2fwYCATVHm8xhO+dqz2ebiqFN5tCmjNQUrvgBAAA4gsIPAADAERR+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxx0o1z0dqnbS3SWvv2d999p+ZcfvnlxnirVq3UnOjoaGPcdvPn/fv3G+O2MSvaNq3lXEQf52K7MXZsbKwxHhMTo+akpqYa47t37w55bVpru4h+o23GudQetu9zvXrmQ5Pt++xnzMratWuNcdu4pfr16xvj2ppF7O8VgNlzzz1njGtjxUT040BZWZmak5aWZozb6g5tm3bOF9HPrfn5+WpOTeCKHwAAgCMo/AAAABxB4QcAAOAICj8AAABHUPgBAAA44qTr6k1ISDDGbTcxbtiwoTFeUFCg5miduLYOXa2rt27dumqOnw5dbd22G2Br3Y627kRtm63jeMOGDcZ4ixYt1BztZta2z03rskLtUVpaqm7T9jVbV6+tG12j7Wu2Y0dkZKQxbttvbO8VqC3q1NGvJdnOX5qOHTsa4wsWLFBztIkQzZo1U3N27dpljCclJak5Wq2gHR9sbNNEagJX/AAAABxB4QcAAOAICj8AAABHUPgBAAA4gsIPAADAERR+AAAAjjgh41xSUlLUbdoYh71796o52hiH5ORkNSc2NtYY37Jli5qjjWSwjUzRxrbYxlJoYyFsN4H3k3PgwAFj3PZZn3HGGcZ406ZN1ZyffvrJGLfdmJpxLrWf9v0T0Uc/2G6abtsPNYWFhca4bdyStgbbvna8b8ION2n7gO1844c2isu2f2quuOKKkHP69u2rbtPOX7ZRbdrYluLiYjVHG1NWVlam5uzevdsYb9++vZqjbVu7dq2acyRc8QMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR5yQrl7txss2tk4ZbZvtxsdaF2ynTp3UHK3jd9++fWqOxk93ou3mz9pnYOt01HJsN9rWOjHr16+v5vjp9GrYsGHIOTi12LpgtW46rRtfxN/3zI+oqChj3NYBqHXz4cTy0wkOf/vaNddcY4w/8sgjas7q1auN8dzcXDUnPj7eGNe6cEX0c54tR/sMbBMptM5iW9f14MGDjXG6egEAAHBEFH4AAACOoPADAABwBIUfAACAIyj8AAAAHEHhBwAA4IgTMs6lTZs26raioiJjvEWLFmrO9u3bQ17D119/bYzv2bNHzdFau6Ojo9Uc7WbvtpEpWpu4dmNs2zZbm7ifm0xrn/Vll12m5mgjE2zvxza2A7WDbb/Rxi0lJCSoOYWFhce8psNsN3TXxrnY9rXS0tJjXhPCz/Yz0/gZAePndfy8vvY6fnJstOPz1KlT1RzteP+vf/1LzTnnnHOM8Z9++knN0d5PamqqmhMTE2OM2443Gtt4t61btxrjWt0jInLuueeGvIYj4YofAACAIyj8AAAAHEHhBwAA4AgKPwAAAEdQ+AEAADjihHT1duzYUd2Wn59vjNtultysWTNjfP78+WrO2LFjjfEFCxaoOVqnn9blJ6J37x44cEDN0d6rrTNLu+G9rZtQW1tiYqKao3UfFRQUqDlah6bt/WhdVqg9bJ3txcXFxnhcXJyak5eXd8xrOkz7ztrYuvm07n6cerTvrZ9JDbaOWm2bny5cPzl33323uu2ss84yxm3ngZYtWxrjPXv2VHO0/eaCCy5Qc7RzoW2fjoyMNMZt+7S2TftZ27ZpxzsRkfbt26vb/OKKHwAAgCMo/AAAABxB4QcAAOAICj8AAABHUPgBAAA4gsIPAADAESdknEurVq3UbdrIFNvoD+1Gytu3b1dzkpKSjPEGDRqoOdookz179qg5ftr4tXZ022egjYfRnktEHz9g+ww+//xzY/zPf/6zmnP++ecb49u2bVNztHEutjEv2ncHJ6f9+/er2xo3bmyM237+2igoP7Zs2aJu0242b+NnPAxOTtox3TbG40QbOHCguu2mm24yxm3nG20fOOOMM9QcbfyJbUSTNlps7969ao42GsV2zq2srAz5dbTjV3x8fMivYzt2JScnhxQ/GlzxAwAAcASFHwAAgCMo/AAAABxB4QcAAOAICj8AAABHnJCu3iZNmqjbVq9ebYzbOli0DryOHTuqObm5uca4rZNJ22a7Obft+TRa97DWvSyid++WlpaqOSkpKcZ4o0aN1JyoqChjPDo6Ws3RupzKysrUHO35tDWL0NV7qrF16GodgLZO4J07dx7zmg7Tbg4von+fCwsL1Rw/ncA4OWnnooYNG6o52nHT1gGqnSczMjLUnNNPP90Ytx3Tc3JyjHFbR2taWpoxrp27RPTO1a1bt6o5Wje8bVqF9vOxvR+t29bWqa39TAOBgJqj1QO7d+9Wc7Tjijb54GhwxQ8AAMARFH4AAACOoPADAABwBIUfAACAIyj8AAAAHEHhBwAA4IgTMs7FNsbhwIEDxnhsbKya89133xnjtlZ5ra3adiNnLcd2k+mCggJj3HbTdq1N3DYaJjIy0hi3jZHQPmsb7TM9++yz1ZwGDRoY47ZxLlqOdtNuEZHt27er23Dyse032vfZdhxYtWrVMa/pMG1chYg++sE2AkZ7Pzg5XX755eq2/v37G+O28412TLeNNtOO97t27VJztDFlX3zxhZrTrFkzYzw1NVXN0cbDZGVlqTk7duwwxouLi9Wc1q1bG+MtWrRQc7T903bs0M4rtlFtfkabaSNtbMebtm3bGuO2WuVIuOIHAADgCAo/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6o0a5e7UbKtm4+rYvG1s357bffGuN9+/ZVc7QOH1sXrNaJa+v80T4DWyeT1vVs68LVXsfWyaS9n9LSUjVH65j66quv1BytK8n2OtrPwfZ+cGqxdQ1qN7y3HQfy8vKOeU1H81za2rROdBG9AxAnp8GDB6vbmjdvbozburo1+/btU7dpXZv79+8POUfrjhURCQQCxvhPP/2k5mjnAVtnc5s2bYzxpk2bqjnaPmWbiqGdO2znT+1cpJ1Xbc+nde6KiJSUlBjjtgkX2s/U1gl8JFzxAwAAcASFHwAAgCMo/AAAABxB4QcAAOAICj8AAABHUPgBAAA4okbHuWjt27YWaa0lPjk5Wc1ZsmSJMT569Gg1p6Kiwhi33fhYG9tiay3XRrMUFRWpOdrnY1ubtgZbjtaSbxtpo93Qe+HChWrO448/bozb2t61tdlGZuDUoo2REBFJSEgwxm3fzd27dx/zmg6z7Td+Rg1p32ecWO3atTPGbaNZVq5caYzXr19fzdHGEGmjgUT00Si2Y6A2FsT2/dO+t40aNVJztFEittFm2j5dUFCg5nz//ffGuO38qX0GthFq2jbb62g1hG1MmTa+Z+fOnWqOdm63jfU5Eq74AQAAOILCDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjarSrV+s+2rRpU8jP5acrrlWrVuo2rTMvMjJSzYmIiAh5Ddq6bR2AWrej7TPQOoxsnY7a+7HdzPr00083xv/+97+rOdnZ2ca47SbT2hpSUlLUHJxabPua1h2ofc9F7DeVD5Xtpunaum3TCqKioo55TQi/DRs2GONPPfWUmtO0aVNjPDU1Vc3RjvfR0dGW1ZklJSWp27Q12HK0841tbevWrTPGta5VEb1Dds+ePWqO9nw7duxQc/Ly8kJeW2FhoTFu6+7XzlG286d2DvdTW9jWdiRc8QMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR1D4AQAAOKJGx7nExMQY4+3bt1dztJsy226wrLG1o2sjZWw3PtZarm03ptZytBsvi+jt9bY2cY3tJuBaC/v27dvVnDZt2hjjthtTa2382k3IRfT3qt3oG6ce2/gTjfadFRHJyso6luVUYRsNox2/bCMZ/LxX1DxtJIbtGGjbptFGAP3qV79Sc7SRY7Z9ICcnxxgvKSlRc7RttpFG2vnLdl7TPus1a9aoOdoYJNt5QNvXbCPUtDFhtvFR2jmvdevWao42bqe8vFzN0Y4rtpE2R8IVPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOoPADAABwRI22mmmdcbYOI61bZ9euXSG/vq37SeuY0jqRRfRuIVvnj9atY+t+KigoMMZtnUxah5Gt41i7cbf22YjoXc87d+5Ucxo1amSM2z5r7fOxdWah9tC6IG2ds7abvYdq9+7d6jZtWoBtbX6mEqDmaT8zrQPVL21KwVtvvaXmaGtr1aqVmtO4cWNjXNufRPT3qnW6iujnDm0ihYi+f9q68bXOYtv0De392H6m2mddWVkZ8uvYzu3aOdzW1aud223n3CPhih8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR1D4AQAAOILCDwAAwBE1Os7lxhtvNMaXL1+u5mjjR/Ly8kJ+/b59+6rb/vWvfxnjtnZ0W2u3RmtHt41+0D4Dra1bRL+Zta2N37YGjXaDettN6LV1a2MxRPTxPS1btrSsDqcS276mfTdtY1FsY6JCZRuDpO1Tfm5QjxPrZP65aGv78ccf1RzbttpEG49zMti2bduJXsIRccUPAADAERR+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxRo129ycnJxnhsbGzIObYbOWuefPLJkHNsnaYNGjQwxm03zda6EG2v06JFi5Cey/Z8tg4jrWusadOmak5OTo4xbutonDJlijHeq1cvNUfr6vzggw/UHJxaKioq1G1a157te+anS12jfc9F9LXZOg137959zGsCgHDgih8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR1D4AQAAOILCDwAAwBE1Os5l6tSpxvif/vQnNaewsNAYf/XVV0N+/ccffzzknOMlJiZG3aaNpbDdUDwqKsoY37dvX2gLqwH//Oc/jfEuXbqoOQUFBcb4ggULwrImnHi277M26kUb9yQiEggEjPHy8vKQ1iUiUlJSom7T1mYbJ2MbQwMAxxNX/AAAABxB4QcAAOAICj8AAABHUPgBAAA4gsIPAADAETXa1bt27VpjfO7cuWpOUlJS2F6/Xj397VVWVhrjtk5D27ZQ2boGT4bnC6dFixaFFIcbli1bpm5r2LChMW7r6i0rKzvWJQXt2LFD3ZaVlWWM+zneAMDxxhU/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjIrxwzigBAADASYsrfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/ACe1kSNHSnx8/BEf16dPH+nTp0/NLwhwXEREhDzxxBPBv7/11lsSEREhmzZtOmFrwtGj8KsBWVlZcuutt0rr1q0lOjpaEhMTpVevXvLyyy9LSUlJjbzm1KlT5aWXXqqR5wZCNX78eImIiJDu3buf6KX4NnLkSImIiAj+qVevnrRo0UKGDx8ua9asqdHXLi4ulieeeEIWL15co68DNxwuzA7/iY6Olnbt2smdd94pOTk5J3p5OM7qnegF1DaffPKJDB06VAKBgNx4443SuXNnKS8vly+//FLuv/9+Wb16tUycODHsrzt16lRZtWqV3HPPPWF/biBUU6ZMkYyMDFm2bJls3LhR2rRpc6KX5EsgEJC//e1vIiJy8OBBycrKkgkTJsi8efNkzZo10rRp0xp53eLiYhk3bpyICFcxETZPPvmktGrVSkpLS+XLL7+U1157TebMmSOrVq2S2NjYE708HCcUfmGUnZ0tw4cPl/T0dFm4cKE0adIkuO2OO+6QjRs3yieffHICVwjUvOzsbFm6dKnMnDlTbr31VpkyZYo8/vjjJ3pZvtSrV0+uv/76KrEePXrIoEGD5JNPPpFRo0adoJUBobv00kvl7LPPFhGRW265RVJTU+VPf/qTzJ49W6699toTvLqaU1RUJHFxcSd6GScN/qs3jJ599lkpLCyUN954o0rRd1ibNm3k7rvvFpFDVw+eeuopyczMlEAgIBkZGfLII49IWVlZlZzZs2fLwIEDpWnTphIIBCQzM1OeeuopqaioCD6mT58+8sknn8jmzZuDl/IzMjJq9L0CmilTpkhKSooMHDhQhgwZIlOmTKn2mE2bNklERIQ8//zzMnHixOB+0K1bN1m+fPkRX+O7776TtLQ06dOnjxQWFqqPKysrk8cff1zatGkjgUBAWrRoIQ888EC1/SwUjRs3FpFDReHP/fjjjzJ06FCpX7++xMbGSo8ePYz/0MvNzZWbb75ZGjVqJNHR0XLmmWfKpEmTgts3bdokaWlpIiIybty44D7989+pAsLhwgsvFJFD/1jTfkd25MiRvs8n48ePl06dOkkgEJCmTZvKHXfcIfn5+cHtd955p8THx0txcXG13GuvvVYaN25c5Vw3d+5cOf/88yUuLk4SEhJk4MCBsnr16mrrjY+Pl6ysLBkwYIAkJCTIr371K1/rr6244hdGH330kbRu3Vp69ux5xMfecsstMmnSJBkyZIjce++98p///Eeefvpp+eGHH2TWrFnBx7311lsSHx8vY8eOlfj4eFm4cKH87ne/k/3798tzzz0nIiKPPvqo7Nu3T7Zu3SovvviiiMhR/TI8UBOmTJkiV111lURFRcm1114rr732mixfvly6detW7bFTp06VgoICufXWWyUiIkKeffZZueqqq+THH3+UyMhI4/MvX75c+vfvL2effbbMnj1bYmJijI+rrKyUwYMHy5dffimjR4+WDh06yPfffy8vvviirF+/Xj744IOjej95eXkiIlJRUSE//vijPPjgg5KamiqDBg0KPiYnJ0d69uwpxcXFMmbMGElNTZVJkybJ4MGD5b333pMrr7xSRERKSkqkT58+snHjRrnzzjulVatWMmPGDBk5cqTk5+fL3XffLWlpafLaa6/JbbfdJldeeaVcddVVIiJyxhlnHNV6gaOVlZUlIiKpqalhf+4nnnhCxo0bJ/369ZPbbrtN1q1bFzwWLFmyRCIjI2XYsGHyl7/8JfgrUocVFxfLRx99JCNHjpS6deuKiMjkyZNlxIgR0r9/f3nmmWekuLhYXnvtNTnvvPPk22+/rVKcHjx4UPr37y/nnXeePP/88/w39i95CIt9+/Z5IuJdfvnlR3zsd99954mId8stt1SJ33fffZ6IeAsXLgzGiouLq+XfeuutXmxsrFdaWhqMDRw40EtPT/e9fiAcVqxY4YmIt2DBAs/zPK+ystJr3ry5d/fdd1d5XHZ2ticiXmpqqrdnz55gfPbs2Z6IeB999FEwNmLECC8uLs7zPM/78ssvvcTERG/gwIFVvv+e53m9e/f2evfuHfz75MmTvTp16nj/+te/qjxuwoQJnoh4S5Yssb6XESNGeCJS7U+zZs28r7/+uspj77nnHk9EqrxWQUGB16pVKy8jI8OrqKjwPM/zXnrpJU9EvLfffjv4uPLycu/cc8/14uPjvf3793ue53m7du3yRMR7/PHHrWsEjsabb77piYj36aefert27fJ++uknb9q0aV5qaqoXExPjbd26tdr+c9iIESOqnVt++d08/PzZ2dme53lebm6uFxUV5V1yySXB777ned6rr77qiYj397//3fO8Q8eHZs2aeVdffXWV53/33Xc9EfG++OILz/MO7UvJycneqFGjqjxu586dXlJSUpX44f32oYceCvVjcgb/1Rsm+/fvFxGRhISEIz52zpw5IiIyduzYKvF7771XRKTKfw/9/GpGQUGB5OXlyfnnny/FxcWydu3aY143EE5TpkyRRo0aSd++fUXk0NiHYcOGybRp06r8l81hw4YNk5SUlODfzz//fBE59N+mv7Ro0SLp37+/XHTRRTJz5kwJBALWtcyYMUM6dOgg7du3l7y8vOCfw/+9tWjRoiO+n+joaFmwYIEsWLBA5s+fL6+//rrEx8fLgAEDZP369cHHzZkzR8455xw577zzgrH4+HgZPXq0bNq0KdgFPGfOHGncuHGV36eKjIyUMWPGSGFhoXz++edHXBPgV79+/SQtLS3YnR4fHy+zZs2SZs2ahfV1Pv30UykvL5d77rlH6tT5/2XGqFGjJDExMXiOi4iIkKFDh8qcOXOq/MrG9OnTpVmzZsH9acGCBZKfny/XXnttlX25bt260r17d+O+fNttt4X1PdUm/FdvmCQmJorIoeLsSDZv3ix16tSp1unYuHFjSU5Ols2bNwdjq1evlscee0wWLlwYLC4P27dvXxhWDoRHRUWFTJs2Tfr27SvZ2dnBePfu3eWFF16Qzz77TC655JIqOS1btqzy98NF4N69e6vES0tLZeDAgdK1a1d59913q/1+ncmGDRvkhx9+CP6+3C/l5uYe8Tnq1q0r/fr1qxIbMGCAtG3bVh5++GF5//33ReTQPm0aXdOhQ4fg9s6dO8vmzZulbdu2VU6Gv3wcUFP+8pe/SLt27aRevXrSqFEjOe2006p9F8Ph8Pf4tNNOqxKPioqS1q1bV/meDxs2TF566SX58MMP5brrrpPCwkKZM2dO8Nc/RA7tyyL//3cSf+nw+fewevXqSfPmzcP2fmobCr8wSUxMlKZNm8qqVauOOufwl1qTn58vvXv3lsTERHnyySclMzNToqOj5ZtvvpEHH3xQKisrj3XZQNgsXLhQduzYIdOmTZNp06ZV2z5lypRqhd/h39/5Jc/zqvw9EAjIgAEDZPbs2TJv3rwqv1+nqayslNNPP13+9Kc/Gbe3aNHiiM9h0rx5cznttNPkiy++8JUPnCjnnHNOsKv3lyIiIqrtdyJivFIfTj169JCMjAx599135brrrpOPPvpISkpKZNiwYcHHHD7XTZ48Odhc9XO//IdgIBCokYK2tqDwC6NBgwbJxIkT5auvvpJzzz1XfVx6erpUVlbKhg0bgv/SFzn0C+L5+fmSnp4uIiKLFy+W3bt3y8yZM+WCCy4IPu7nV1MOO1IRCdS0KVOmSMOGDeUvf/lLtW0zZ86UWbNmyYQJE9RmDJuIiAiZMmWKXH755TJ06FCZO3fuEefbZWZmysqVK+Wiiy4K+/5x8ODBKv81lZ6eLuvWrav2uMO/jnF4n05PT5f/+7//k8rKyionpl8+jv0Zx1tKSorxVyz8XIU+/D1et26dtG7dOhgvLy+X7OzsalfRr7nmGnn55Zdl//79Mn36dMnIyJAePXoEt2dmZoqISMOGDavlInSUxGH0wAMPSFxcnNxyyy3GaehZWVny8ssvy4ABA0REqt1p4/CViYEDB4rI/78a8vN/hZWXl8v48eOrPXdcXBz/9YsTpqSkRGbOnCmDBg2SIUOGVPtz5513SkFBgXz44Ye+XyMqKkpmzpwp3bp1k8suu0yWLVtmffw111wj27Ztk7/+9a/G9RYVFflax/r162XdunVy5plnBmMDBgyQZcuWyVdffRWMFRUVycSJEyUjI0M6duwYfNzOnTtl+vTpwccdPHhQXnnlFYmPj5fevXuLiAS7EH8++gKoSZmZmbJ27VrZtWtXMLZy5UpZsmRJyM/Vr18/iYqKkj//+c9Vzl9vvPGG7Nu3L3iOO2zYsGFSVlYmkyZNknnz5sk111xTZXv//v0lMTFR/vjHP8qBAweqvd7P14wj44pfGGVmZsrUqVNl2LBh0qFDhyp37li6dGlwbMPdd98tI0aMkIkTJwb/O3fZsmUyadIkueKKK4K/GN+zZ09JSUmRESNGyJgxYyQiIkImT55svBzftWtXmT59uowdO1a6desm8fHxctlllx3vjwCO+vDDD6WgoEAGDx5s3N6jRw9JS0uTKVOmVPkvnFDFxMTIxx9/LBdeeKFceuml8vnnn0vnzp2Nj73hhhvk3Xffld/85jeyaNEi6dWrl1RUVMjatWvl3Xfflfnz56v/7XXYwYMH5e233xaRQ//dtGnTJpkwYYJUVlZWGUr90EMPyTvvvCOXXnqpjBkzRurXry+TJk2S7Oxsef/994NX90aPHi2vv/66jBw5Ur7++mvJyMiQ9957T5YsWSIvvfRSsDksJiZGOnbsKNOnT5d27dpJ/fr1pXPnzup7BY7Vr3/9a/nTn/4k/fv3l5tvvllyc3NlwoQJ0qlTp2q/X34kaWlp8vDDD8u4cePkv//7v2Xw4MGybt06GT9+vHTr1q3aUPQuXbpImzZt5NFHH5WysrJqx4jExER57bXX5IYbbpAuXbrI8OHDJS0tTbZs2SKffPKJ9OrVS1599dVj/gyccUJ7imup9evXe6NGjfIyMjK8qKgoLyEhwevVq5f3yiuvBEdQHDhwwBs3bpzXqlUrLzIy0mvRooX38MMPVxtRsWTJEq9Hjx5eTEyM17RpU++BBx7w5s+f74mIt2jRouDjCgsLveuuu85LTk72RITRLjiuLrvsMi86OtorKipSHzNy5EgvMjLSy8vLC45zee6556o9Tn4xKuLn41wOy8vL8zp27Og1btzY27Bhg+d51ce5eN6hUSnPPPOM16lTJy8QCHgpKSle165dvXHjxnn79u2zvifTOJfExETvoosu8j799NNqj8/KyvKGDBniJScne9HR0d4555zjffzxx9Uel5OT4910001egwYNvKioKO/000/33nzzzWqPW7p0qde1a1cvKiqK0S44JofHrSxfvtz6uLfffttr3bq1FxUV5Z111lne/PnzfY1zOezVV1/12rdv70VGRnqNGjXybrvtNm/v3r3G13700Uc9EfHatGmjrm/RokVe//79vaSkJC86OtrLzMz0Ro4c6a1YsSL4GNPxAlVFeJ7h8hEAAABqHX7HDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/AAAARxz1nTu4dyRqo5NxjCX7Gmoj9rWT0+Fbg/5S9+7d1ZwhQ4YY499//72aU1JSYozb7t09dOhQY/yNN95Qc95//311myuOtK9xxQ8AAMARFH4AAACOoPADAABwBIUfAACAIyj8AAAAHHHUXb2uiIyMVLcdOHDAGO/cubOa88477xjjqampao7WZVVZWanmREVFGeObN29Wc7p06aJu02hrq6ioCPm5gBOtXj3zIfDgwYNqTrNmzYzx66+/Xs1ZtWqVMb527Vo1Z9euXcb4/v371Rw/2KdPLfXr11e3tWzZ0hhPSkpScwKBgDGel5en5pSVlRnjY8eOVXO0/SYlJUXNWbBggTG+fPlyNad9+/bGuPY9FxHJzc01xvPz89UcrR44FXDFDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgiAjvKO+cfSrezNrPaBY/bO3o2jiV3bt3qznaGAXbjyo+Pj7knKZNm6rbwqlOHfO/L2zjaY4Xbhx/4tjep5+fi5/PTRvxYBvn0qZNG2N83Lhxas7KlSuN8R07dqg5ffr0McYnTZqk5nzxxRfqtnDSPmvbz419rbpGjRqp27RxW9oIIhH9mFpaWqrm+DkXat/n4uJiNadJkybGuG080b59+4zxBg0aqDna+Vg7D4no508bbdzSli1b1JzjtQ8c6XW44gcAAOAICj8AAABHUPgBAAA4gsIPAADAERR+AAAAjtDbg04yfjoA/XQrJSYmqttuvvlmY/yWW25Rc7RuKluHUXl5ubpNo3UhRkdHqznaja7vvfdeNcdP16DWaRburk6cWsL9M9aez/Y9s3XvagoKCoxxW0fj2rVrjfGGDRuqOVFRUcb4FVdcoeZ89913xnj9+vXVnEAgYIyvW7dOzWH/DI32s7z44ovVHK2j1TYRQjvWxsTEqDna/mHrBD7zzDNDXpv2fI0bN1ZztG1+zu22nLy8PGNc6/oX0T9TW8ex1gl8vHHFDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgiAjvKPvyT/TNrP3o27evuu23v/2tMX7aaaepOdpoFNtIiJKSEmPcNs6loqIipLiI3lpuW5s2xsF2E3CtHf3TTz9Vc+666y51m0b7fLRxBX6djGMpTsV97WQW7rFBL7zwgjFuG+eyePFiY3zQoEEhv742TkZEJDMz0xjXxlWI6PvaH/7wBzVn586dIT2XiP34daIcr33tjDPOMMY7duyo5mjHWtsxMDIy0hjfv3+/ZXWh084rtlFkfn7+Wo5tNEs4f6a2kTYa7bwqIrJlyxZj3M94GpsjHde44gcAAOAICj8AAABHUPgBAAA4gsIPAADAERR+AAAAjtDbN08hWvfurFmz1BytA6+oqEjNKSwsNMZtHTRal5vt5s/a89m6orS12WgdS7b3o3WN3XLLLWpOUlKSMX7jjTeqOeHu3kXtp3Xz2fY1rTuxefPmak7Dhg2N8X/+858h5+Tm5qo5WsenrXv4p59+MsZtXYOJiYnGuK07UUNHupn2GduO6doxMDY2Vs3ROr5t3dbaNtt3RttvbMdt7b3a9s9Qn8vGtjbt/fg5F9rWdrLsH1zxAwAAcASFHwAAgCMo/AAAABxB4QcAAOAICj8AAABHUPgBAAA4olaMc3n00UeNcdvYA+2m1VFRUWpOvXrmj8vWvq21xPtpe7e15GvrtrWPx8XFGeO2G21r27SbT4uI9OnTxxi3fda2NQAm2ugFPyMUWrZsqW7T9mnb62ijUbSRLbbXiY6OVnO00Rh+RkykpaWpOZs3bw75dVyWkJBgjNtGptSvX98Yt53XtNFZe/fuVXO0Ndh+ltr4Ey1u42ekjW3MSkxMjDFu+9z8HDu0tWljXmzbjvf5jit+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxB4QcAAOCIU6ar19Zd07p1a2Pc1i2kdZTaum1tnUSacN6U2bY2rWOptLRUzdm2bVvIr5OSkmKMax3CtpwOHTqoOStXrlS3AaGwdU5q9u3bp25r0aKFMW672XxycrIxbltbWVmZMW47rmndgXv27FFzmjZtaoxrnagIXXx8vDG+bt06NWf48OHG+LfffqvmaMdurdtXRCQnJ0fdptHOhbZzh20qhcb2fKHm+NnXbPu01qlvo+UUFRWF/FzHgit+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxB4QcAAOAICj8AAABHnDLjXJo0aaJu00Yl2FqktbZq242ctXZ025gXP+NctNexPZfWqr5jxw41Z9OmTca47Qb12o3DtRtji+gt8V26dFFzGOdS+9m+z35GJ4WTNt5BRF+37Xhz2mmnGeO2cRHa82n7oIh+HDh48KCaox1vtDEvCF10dLQxbhu3VVJSYoxr47FERHbu3GmMa+NkRPTRRba1+Tmv+RnNoh0H6tXTSxftHG57fW2b9nMT0WsI29gY2/5+PHHFDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcccp09Xbr1k3dpnXX2LqStG4drZPKL637yda1qHX+2DoNtedr1aqVmnPWWWcZ47YOQD83vNekp6eH7blw6jlenbt+uod79uyp5uzdu9cYb926tZpz/vnnG+OFhYUhr832frTuRFsnsNbVaTt2oLrIyEh1m3ZMteVs2bLFGLd19WrnjvLycjXHD62r1nbu0LbZjgN+zjcVFRXGuK2rV8tp0KCBmqM9n62G0Droj7eTYxUAAACocRR+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxB4QcAAOCIU2acy7nnnqtu09qqbTdy1kYiaG3dIvrYGFsLu5+bWWvt7drr29jWtmfPnpCfLykpKeQcjW0sAXAinXHGGeq2li1bGuO2UQ3amI0VK1aoOdrIKdtYCm2ElW1siHaMSktLU3NQnfbzEtHHqcTExKg52dnZxniTJk3UnOTkZGN89+7dao7t+6TRvuva91xEPxfZzrmacI4Vs2natKm6bf/+/cZ4Tk6OmuPnHF4TuOIHAADgCAo/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI44Zbp6W7RooW7TuoJsN3+Oj483xvPy8tQcrZPJz03gbR2AWpeV7XX8rE3LsXVDa51rts9a+/nYOqZQe2jfQdt3M5ydhn6e68ILL1S35ebmGuPffPONmvP9998b4wkJCWqOn3WXlZUZ41FRUWrOrl27jPHevXurOVqX8PHqtjwZ2Tqnta7euLg4NUfrxLUdn7WpCzt37lRzbOeiUHNs+7Rt3Rrt++TnvGabcKGdoxo1aqTmaPuarUvZz2ddE06OVQAAAKDGUfgBAAA4gsIPAADAERR+AAAAjqDwAwAAcASFHwAAgCNOmXEu2o3RRfRRIrb2ba213NZeX1xcbIzbbkztZzSLtjbbqARtbbbX0dbtZ222FnZt3bYRPag9tP3TNgLIDz/jT5555hljvKioSM2ZN2+eMd64cWM1RxvnUVpaqub4eT+JiYnGuG3USEpKijGujW4SEenUqZMx/t133+mLq+W0EWEi+jHQNt5DO3+1a9dOzcnOzg7p9UX074btu6mx7dN+vs/aucjPODTb2rTPunXr1mrOli1b1G0abZ+y1RC2c6tfXPEDAABwBIUfAACAIyj8AAAAHEHhBwAA4AgKPwAAAEecMl29aWlp6jat68XWKbN3715jPCcnR81p0qSJMa511Ir4u5m11n1kez/azd5t3VyBQCDktWndT7Yc7eeTnp6u5qD287MP+HHbbbep23r06GGMr1ixIuScrKwsNUfbb2yds9r+7ud4Y/s8tefbsGGDmjN69Ghj/Pbbb1dzaruoqCh1W0lJiTFu69jUJigkJSWpOdr3ydaha+v4DlW4O1C1762tq7esrMwYt/18tNfJzc1Vc7TPzbY2bQ1+Op6PBVf8AAAAHEHhBwAA4AgKPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOOGXGucTFxanbtFZobVyJiMiXX35pjO/cuVPNufHGG43x/Px8Nce2Bo12Q3fbOBft8/EzFkMbPSCij6WIiYlRc7QW/9TU1NAWhlolnCNbRET++Mc/GuPdu3dXc1avXm2MJyYmqjmrVq0yxm1jNrTn279/v5qj7Te2sRTaiAnbuIjY2FhjvGnTpmoO+2512udoYxuzoh3T33nnHTWnT58+Ib+Odo6yfWf8fM+OF22cim3MinZeW7dunZoTHx9vjNtqFe11jjeu+AEAADiCwg8AAMARFH4AAACOoPADAABwBIUfAACAI06Zrt6GDRuq27QbKWtdNyIimzZtMsZtHXNaV62tk8lPl5OfG61rncAREREhv45tzVo3X1FRkZqjrXv79u1qDk4ttu+Zxk9X7+jRo9Vtbdq0McbXrl2r5mjdttrxQcTfez1w4IAxbuvU1zrl69XTD9vacSAnJ0fN0TpBbe9TO7aeddZZak5tZ+vq1X6WeXl5ak5KSoox/umnn6o5w4cPD3ltGlsXrNZxbjt3aN/b49UJ7Gdf27p1q5qTkZFhjCckJKg5Wlev1iUtou/Tx4IrfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR5x041y0FnJtZIuI3lpua5HevXu3MZ6ZmanmaG3ntjZxP6MftLZ32/gL7XOzvb72udnk5+cb47bPWltby5YtQ359hMbPd9PPjcT9jGaxjYv4/e9/b4x36NBBzcnKygp5DWVlZcZ4YWGhmlO/fn1j3LYPlJaWGuO2MRvayBTbOBdtm21Mlbbu4uJiNUd7Ptv3rbaLi4tTt2n7lPb9E9HPN9q5S0Rk48aNxnhaWpqas3nzZmPc9p3Rvs+276bGlqONQbIdO7RttvOd9r3duXOnmtO8eXNjPDk5Wc3ZsWOHMW57PzWBK34AAACOoPADAABwBIUfAACAIyj8AAAAHEHhBwAA4IiTrqu3c+fOxrit80fr1rHlJCUlhbYw0Tuwwt2Ro3UY2W5m7afjWOsaXL9+vZqzZs0aY/yqq65Sc7ROJhut49RP96jL/HRu+9GuXTt1269//Wtj3NYFWVBQYIz/8MMPao7WnWr7zuzdu9cYb9y4sZoTHR1tjNuOA1qOTVFRUcjPpd3Q3dY9qq07NTVVzWnYsKExbvv51Ha2rm7t+BwIBNQc7Wdp+z7n5OQY4+E+R2nPZzvn+pkW4GcqhvZZ255LO0/u27dPzdF+3lonsoj+GfiZCHAsuOIHAADgCAo/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHDESTfOJTEx0Rj309Zts2fPHmP8rLPOUnNs41RCZWvJ11q+bZ+B1l5vex1tXIA2qkFE5Ntvv1W3afy08Tdo0MAY37VrV8jP5TLbeIXLLrvMGF+2bJmas23btpBfZ/v27cb45ZdfruasXbvWGN+6dauao31vtdcXESkuLjbGbTda13L8jIgqKSlRt2lr0PZbEX0sREpKiprTunVrY1z7WYuI5ObmGuM33XSTmlPb2UZyaGNBYmJi1Jy8vLyQ16CN7bGN89GOz7Z9WjsXRkVFhfw6tpFjGluO7ecQao7t/KmNQ7ONc9G2+fkMjgVX/AAAABxB4QcAAOAICj8AAABHUPgBAAA4gsIPAADAESddV6/WKWPrMPJj/fr1xritK62wsNAYt90AW+vEtXUL+aE9n61bSOvMsnU0ajeO98PWtZaenm6M09VrNmbMGGP8uuuuU3O0G8TbOme1HFvntrZN6+AXEenfv78x/tFHH6k5WrdrRkZGyDnR0dFqjra/++kEth07tM+toqJCzdGOA02aNFFztGOh7dgRFxdnjNu6Oms723dm9+7dxrjt89JybGxdwhrbdzCcz6V9n2xdsNp53/Y6Wo7tGOWnE1hbg+25/OTUBK74AQAAOILCDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAccdKNc0lISDDGtbEoR9qm0W6OXr9+fTVHu9G27fW1kSm2UQlay7f2XCL2VnWNtgbbSABt/ICflnyb1NTUkHNqu9/97nfqtmuvvdYY18aviOijP7p27RrawizPJaLvH9o+KKLvAxdccIGa8+GHHxrjtu+ftjbbvqbd8H7Tpk1qjjbOpUGDBiHnJCUlqTna6BrbcU07Dtg+A23slp/jUG1h+7z80H7+Nn7GgvgZweNnBIx2XtH2JxF/5w7t+6yNbvL7OuvWrTPG27dvr+Zon1u4x7sdCVf8AAAAHEHhBwAA4AgKPwAAAEdQ+AEAADiCwg8AAMARJ11Xr+3G7Ro/HUbajc5jY2PVnF27dhnjfjp0bV082jY/N6a2vU5paakxbusMS0tLM8ZzcnJCXputG1p7HZf17NlT3abdIH7z5s1qzsyZM43xZs2aqTlad6itO1Xr+LZ1gGrd/Y0bN1ZztH3X1h3ppztV2w9txy7tdWzHmy5duoSco/3sioqK1Bzt52Pr1Nc6Qb/++ms1p7azdfVqx0Bbp6l2jvLDT9eo7XyjnfNsa9Zy/HTU+uGn49lmy5Ytxvg555yj5viZQFITuOIHAADgCAo/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHDESTfORRtVYGtH127+nJ2dreZoYy5sreXaiAfbTa61kQi296O1vdvGxvi52byfFn8tZ/Xq1WrOf/3Xfxnjttb2uLi40BZWi7z88svGeEFBgZqzdOlSY9z2nbnggguMcdtoFm3/tP0stXEq2n4rIpKUlGSMt2vXTs3Jz883xn/44Qc1R3s/tnEu2hgk283mGzVqZIz369dPzZk0aZIx/j//8z9qzpNPPmmM9+3bV82ZMWOGMa6N+xER2bp1q7oNR8/2PfNDG1liGwGkjW2xnTs0thxtRJNt1JD2+Wjjq2xs+6ftOKnZs2dPyDnaWCfbyKmawBU/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHDESdfV27BhQ2Pc1i2kdfjs2LFDzUlOTjbGbZ2uWoeRn24hWxek9l5ta9Nujm3rUta27dq1S83RusNsPx/tM7DlhPuG2qeSFStWGONax6aI3mlq2we0Ltjt27erOdp30NZxrH2fAoFAyK/TuHFjNUe7abptbVq3rdZVLCKSmppqjGdkZKg52r7WsmVLNaewsFDdpvnd734Xcg7Cw3as1ZSUlIR1DdoUCVsXrHbu8NPpqj2XiP08GU7alA3b2vz87DS2n6l2XrOtrSZwxQ8AAMARFH4AAACOoPADAABwBIUfAACAIyj8AAAAHEHhBwAA4IiTbpyLn3Eh2mgUbcSFiEiLFi1Cfh1tjIOf0SO20SzaTbPDTfvctHZ4EZEePXoY47axIX5u9q2NJXDB5MmTjfHy8nI1Z8CAAcZ4/fr11RxtlIh2I3ER/YbqtpEpl1xyiTG+f/9+NUcbAWMbu6C9H9v3OTEx0RjfuXOnmqM932effabmPPHEE+o2jfZe/Yx1stGON+EeOeUybTTKvn37wvo62sgxP+cU23lN+znbjvXa+dh2vAn1uWy02iLcbMdCbayOn9E5x4IrfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgiJOuqzecNzG2dfO98847xvgNN9yg5mjdOg0aNFBz4uLijHFb95vW0WjrsvND68Cy3aB+3rx5xnj//v3VHD8/O1s3qqumT5/ua1uokpOT1W3FxcXGuK0z7+KLLzbGbTeO17odL7roIjVn+/btxvi3336r5mj74ZIlS9ScwsJCdVs4aR2aNtoxwna88dN1j9BoXZtlZWVhfZ0VK1YY4+eee66ak5uba4z7mVZh67bVvme2jlZtkkEgEFBztIkQ2kQCEZGSkhJ1W6hsndq2qQTHE1f8AAAAHEHhBwAA4AgKPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOODl6i39Ga9NOTU1Vc/y0nWu0m93Dzva5NWrUyBi3tdfbxg+4yjbOxzauI1T5+fkh5+zZs0fdFs5RMx988EHYnssv7edg+/looyzC/TMN5/fgeH3fagvbWC9tbEs4z10iIp9++qkx/tlnn6k5MTExIb9OOMcG2cZ9aTm2EUQn+rtZp45+PS0tLc0Yz8rKqqnlGHHFDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAccdJ19T799NPGuHZzeBGR9PR0Y/z3v/99yK9PJ5s/1113nbrtvvvuM8aXL1+u5oSzE7S24Pt3ctB+Die6CzfcTua1nYw2bNigbistLTXG9+3bV1PLqcL2s7SdWxG6zZs3q9uaNm1qjBcWFtbUcoy44gcAAOAICj8AAABHUPgBAAA4gsIPAADAERR+AAAAjqDwAwAAcESER88+AACAE7jiBwAA4AgKPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOoPADAABwBIUfAACAIyj8AAAAHEHhBwAA4AgKPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOoPA7ybz11lsSEREhK1asOOJj+/TpI3369Kn5RQG1EPsacHLYtGmTREREyPPPP3+il+IECr+jFBERcVR/Fi9ebMyvrKyUf/zjH9K9e3epX7++JCQkSLt27eTGG2+Uf//73zW+/jVr1sgTTzwhmzZtqvHXAo4F+xoQft9//70MGTJE0tPTJTo6Wpo1ayYXX3yxvPLKKyd6aTjO6p3oBZwqJk+eXOXv//jHP2TBggXV4h06dDDmjxkzRv7yl7/I5ZdfLr/61a+kXr16sm7dOpk7d660bt1aevToEfKa/vnPfx71Y9esWSPjxo2TPn36SEZGRsivBRwv7GtAeC1dulT69u0rLVu2lFGjRknjxo3lp59+kn//+9/y8ssvy1133XWil4jjiMLvKF1//fVV/v7vf/9bFixYUC1ukpOTI+PHj5dRo0bJxIkTq2x76aWXZNeuXb7WFBUVdcTHlJaWHtXjgJMF+xoQXn/4wx8kKSlJli9fLsnJyVW25ebmnphFHWfFxcUSGxt7opdxUuC/eo+D7Oxs8TxPevXqVW1bRESENGzYsFq8rKxMxo4dK2lpaRIXFydXXnlltZPWL3/vaPHixRIRESHTpk2Txx57TJo1ayaxsbHy5z//WYYOHSoiIn379j3if5UBpyr2NaC6rKws6dSpU7WiT0Sq7BMRERFy5513ygcffCCdO3eWQCAgnTp1knnz5lXL27Ztm/z617+WRo0aBR/397//vcpjysvL5Xe/+5107dpVkpKSJC4uTs4//3xZtGjREdfseZ6MHj1aoqKiZObMmcH422+/LV27dpWYmBipX7++DB8+XH766acquX369JHOnTvL119/LRdccIHExsbKI488csTXdAVX/I6D9PR0ERGZMWOGDB069Kj+1XHXXXdJSkqKPP7447Jp0yZ56aWX5M4775Tp06cfMfepp56SqKgoue+++6SsrEwuueQSGTNmjPz5z3+WRx55JPhfZNp/lQGnKvY1oLr09HT56quvZNWqVdK5c2frY7/88kuZOXOm3H777ZKQkCB//vOf5eqrr5YtW7ZIamqqiBy6st6jR49goZiWliZz586Vm2++Wfbv3y/33HOPiIjs379f/va3v8m1114ro0aNkoKCAnnjjTekf//+smzZMjnrrLOMa6ioqJBf//rXMn36dJk1a5YMHDhQRA5dufztb38r11xzjdxyyy2ya9cueeWVV+SCCy6Qb7/9tkphu3v3brn00ktl+PDhcv3110ujRo2O+XOsNTz4cscdd3ihfHw33nijJyJeSkqKd+WVV3rPP/+898MPP1R73JtvvumJiNevXz+vsrIyGP+f//kfr27dul5+fn4w1rt3b693797Bvy9atMgTEa9169ZecXFxleedMWOGJyLeokWLjv5NAicB9jXg2Pzzn//06tat69WtW9c799xzvQceeMCbP3++V15eXuVxIuJFRUV5GzduDMZWrlzpiYj3yiuvBGM333yz16RJEy8vL69K/vDhw72kpKTgPnHw4EGvrKysymP27t3rNWrUyPv1r38djGVnZ3si4j333HPegQMHvGHDhnkxMTHe/Pnzg4/ZtGmTV7duXe8Pf/hDlef7/vvvvXr16lWJ9+7d2xMRb8KECaF+VE7gv3qPkzfffFNeffVVadWqlcyaNUvuu+8+6dChg1x00UWybdu2ao8fPXq0REREBP9+/vnnS0VFhWzevPmIrzVixAiJiYkJ6/qBUwX7GlDVxRdfLF999ZUMHjxYVq5cKc8++6z0799fmjVrJh9++GGVx/br108yMzODfz/jjDMkMTFRfvzxRxE59F+w77//vlx22WXieZ7k5eUF//Tv31/27dsn33zzjYiI1K1bN/h7r5WVlbJnzx45ePCgnH322cHH/Fx5ebkMHTpUPv74Y5kzZ45ccsklwW0zZ86UyspKueaaa6q8ZuPGjaVt27bV/vs4EAjITTfdFJ4PsJbhv3rDqLCwUAoLC4N/r1u3rqSlpYmISJ06deSOO+6QO+64Q3bv3i1LliyRCRMmyNy5c2X48OHyr3/9q8pztWzZssrfU1JSRERk7969R1xHq1atjvWtACc19jUgNN26dZOZM2dKeXm5rFy5UmbNmiUvvviiDBkyRL777jvp2LGjiFTfH0QO7ROH94ddu3ZJfn6+TJw4sVoD1WE/bxiZNGmSvPDCC7J27Vo5cOBAMG7ad55++mkpLCyUuXPnVpubuWHDBvE8T9q2bWt8zcjIyCp/b9asGc1WCgq/MHr++edl3Lhxwb+np6cbZ3mlpqbK4MGDZfDgwdKnTx/5/PPPZfPmzcHfTxI5dCIz8TzviOvgCgRqO/Y1wJ+oqCjp1q2bdOvWTdq1ayc33XSTzJgxQx5//HEROfL+UFlZKSKHuu9HjBhhfOwZZ5whIocaMUaOHClXXHGF3H///dKwYUOpW7euPP3005KVlVUtr3///jJv3jx59tlnpU+fPhIdHR3cVllZKRERETJ37lzjGuPj46v8nX1TR+EXRjfeeKOcd955wb8fzRfv7LPPls8//1x27NhR5WQUbj//ryzgVMe+Bhy7s88+W0REduzYcdQ5aWlpkpCQIBUVFdKvXz/rY9977z1p3bq1zJw5s8p+cbjI/KUePXrIb37zGxk0aJAMHTpUZs2aJfXqHSpTMjMzxfM8adWqlbRr1+6o14vq+B2/MGrdurX069cv+OfwSImdO3fKmjVrqj2+vLxcPvvsM6lTp460adOmRtcWFxcnIiL5+fk1+jrA8cC+Bhy9RYsWGa9gz5kzR0RETjvttKN+rrp168rVV18t77//vqxatara9p+PQjp8Ze7nr/2f//xHvvrqK/X5+/XrJ9OmTZN58+bJDTfcELzCeNVVV0ndunVl3Lhx1d6L53mye/fuo34PruOK33GwdetWOeecc+TCCy+Uiy66SBo3biy5ubnyzjvvyMqVK+Wee+6RBg0a1OgazjrrLKlbt64888wzsm/fPgkEAnLhhRca55oBpyr2NaC6u+66S4qLi+XKK6+U9u3bS3l5uSxdulSmT58uGRkZITdB/O///q8sWrRIunfvLqNGjZKOHTvKnj175JtvvpFPP/1U9uzZIyIigwYNkpkzZ8qVV14pAwcOlOzsbJkwYYJ07Nixyu/o/tIVV1whb775ptx4442SmJgor7/+umRmZsrvf/97efjhh2XTpk1yxRVXSEJCgmRnZ8usWbNk9OjRct999x3T5+QKCr/j4LTTTpOXXnpJ5syZI+PHj5ecnByJjo6Wzp07y1//+le5+eaba3wNjRs3lgkTJsjTTz8tN998s1RUVMiiRYs4GaFWYV8Dqnv++edlxowZMmfOHJk4caKUl5dLy5Yt5fbbb5fHHnvMONjZplGjRrJs2TJ58sknZebMmTJ+/HhJTU2VTp06yTPPPBN83MiRI2Xnzp3y+uuvy/z586Vjx47y9ttvy4wZM4441Pz666+XgoICuf322yUxMVGee+45eeihh6Rdu3by4osvBn/Ht0WLFnLJJZfI4MGDQ/1YnBXhHc1vMAMAAOCUx+/4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgiKMe4OzK/Sc///xzdduyZcuM8fvvv7+mllOFdvNsEZGKigpjfODAgWrOqFGjjPErrrgipHWdyk7GMZau7GvhdtZZZxnj2n1BRUTq1DH/27eoqEjN+e6774zxZ599Vs3R+NmnT1Xsa8DxcaR9jSt+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxx1M0drkhMTFS3nX766cdxJdVVVlaGnNO2bVt128UXX2yMN2zYUM3Jzc0NeQ1AuPTp00fdtmjRopCfb/jw4cZ4q1at1JxnnnnGGI+Li1NztAaT2tbAAeDkxxU/AAAAR1D4AQAAOILCDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjIryjvIGiK/c0nDt3rrqtQ4cOxnibNm3UnIMHDxrj9erpk3S0sS1+xrnMnj1b3da5c2djPDMzM+TXOVVx/9AT55VXXlG39ejRwxhv166dmrN//35jPD8/P+ScAwcOqDnauKMmTZqoOXv27DHG//rXv6o5//u//6tuOxWxrwHHB/fqBQAAgIhQ+AEAADiDwg8AAMARFH4AAACOoPADAABwBF29v7BkyRJ1W7NmzYzx1atXqzkDBw485jUdjTPOOMMY//DDD9WcoqIiY7xTp05hWdOpgE7DmnfBBRcY459//rmas2vXLmO8pKREzdG63m0duikpKca41o1vW0OdOvq/o2NiYkJ6fRGRFStWGOM9e/ZUc05m7GvA8UFXLwAAAESEwg8AAMAZFH4AAACOoPADAABwBIUfAACAIyj8AAAAHFHvRC/gROnYsaMx3rJlSzVn9+7dxnj79u3VnOXLlxvjb7zxhppTXFxsjHfp0kXNufjii43xgoICNScpKckYb9OmjZqzceNGdRtgkpuba4zv2bNHzdH2gcjISDXHzzgXbZ+uW7eumhMIBELO0UbA2MbG2LYBgF9c8QMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/AAAARzjb1fv73/8+5Byto9DWOZucnGyMP/roo2pO8+bNjfGtW7eqOfv37zfGbTdr1t7P3Xffrebcdddd6jbApE4d878vbV2wERERxrjWuWt7vqioKDVH67bV1ux3bdp+mJqaquZ888036jYA8IsrfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAcQeEHAADgCAo/AAAARzg7ziU9Pd0Yt90YXRvxYLtxfGlpqTGujV8REdmxY4cxXq+e/uOKjo42xm2jLLS1XXXVVWoO41wQqs2bNxvjSUlJao42IsnPOBfb2Bhtn7bta35o41xs+2dOTk5Y1wAAIlzxAwAAcAaFHwAAgCMo/AAAABxB4QcAAOAICj8AAABHONvVq3Xt2W7OrnUH2nK0br5AIKDmaDeB1+IiIhUVFca4rTtRez8bNmxQc4BQHThwIGzPpe1Pttex7Z8xMTHGuG1f09jWpnUj214nKysr5DUAJ5JtX7N15IdT3759jfHzzjtPzXnqqadCfh3tvYb7fWqvYzveHPE5fWcCAADglELhBwAA4AgKPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOcHacS5MmTYxxW4u0NnrB1r6tbfPTim1rldfW5meURf369UNbGGCh7QM7duwI+bls+422zbYPhJPtOOBnf9+4ceOxLAc47mz7gHaO8rNvXHLJJeq2fv36GeMtWrRQcx544AFj/Nlnn1VzjmWcSihqYgwOV/wAAAAcQeEHAADgCAo/AAAAR1D4AQAAOILCDwAAwBHOdvVu2bLFGM/MzFRzDh48aIzXq6d/jFqOn+4n2w3dtc5FPznbtm1Tc4BQJScnG+O272ZFRYUxfuDAATVH29fi4+PVHNsaNNraSkpK1JxAIBDy6zRo0CDkHOBE8nMutJk4caIxrk2kEBHZv3+/MZ6SkqLm5OTkhLYw8delrJ1zbfVA165djfGvv/7asjo7rvgBAAA4gsIPAADAERR+AAAAjqDwAwAAcASFHwAAgCMo/AAAABzh7DiXTz/91Bg/7bTT1Bw/N0vWcsI9zqVu3brGuDZ6QkRvLf/ss8/UHCBUeXl5xrg2dkFEJDY21hi3jYvwM5pF2wf8jJ4I99q++OKLkHOAcNHOKSL6ecXPfjN48GB1W1xcnDG+Z88eNSc9Pd0Yt50LJ02apG7ThLMesLn22muN8ZUrV4b8XIdxxQ8AAMARFH4AAACOoPADAABwBIUfAACAIyj8AAAAHOFsV+/nn39ujN9+++1qjtYVZOt+0rp4bDdy9sNP93BZWZkxvmDBgrCsCbDJz89XtzVp0sQYt3XzRUZGHuuSgvx03dtePxAIGOO2zubVq1er2+AuW4e41qWuxUVEDhw4YIzbumD9uOmmm4zxJ554Qs2ZP3++Md6+fXs1R+uuz87OVnNatGhhjOfk5Kg52jncduxo0KCBMZ6UlKTmNG/e3BjXjilHgyt+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxB4QcAAOAICj8AAABHODvOJTc31xjXWttF/LXK27aFk9ZCbmv9Ly0tNcbXrl0bljUBNjNmzFC3PfXUU8a4nzFItvEK2v4Z7hFN0dHRxvhXX32l5uDU4uc8YPvO+BkXoo1g8TOaJSMjQ9122mmnGeMjR45Uc7744gtj/MEHH1Rzhg8fbowXFBSoOdpYpdTUVDVn/PjxxrifY0d5ebmao63btra9e/ca43FxcWrOkXDFDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAc4WxXb1lZmTFu6+I5Fdm6E7XuI+2zAcJp6dKl6raDBw8a49oN2EX8ddD76YbX9iltzbacefPmWVaHE8X2XdK+M7Zzx4k+r4wYMULd1rx5c2O8cePGak5eXp4x/tBDD6k5Xbp0McavuuoqNadJkybGuO0clZSUZIwXFxerOdqEC9s+7adTOiUlxRjPz89Xc2JiYozx+Pj4kF//MK74AQAAOILCDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjKPwAAAAc4ew4F+0Gx9oNnkXs7eAnmjZ+wtZy7mf8BRAutu/miR5/YaONlLGNftCsW7fuWJeDGhDu7582xqNFixZqTseOHY3xTp06qTkJCQnGeHJyspqzYMECY3z27NlqTs+ePY3xF154Qc3p0KGDMW4bObZjxw5j3Hbs0EajeJ6n5kRHRxvj2sgz2/NpzyWivx/b2hITE41x7WdwNDjzAwAAOILCDwAAwBEUfgAAAI6g8AMAAHAEhR8AAIAjnO3qPf30041xW4eRrfNGo3XO2rrG/LyO1tVre53Y2FhjPCMjQ83ZtGlTKMsCVJ07d1a3ad31frotbd3rfp7PT47Whagdh0RE5s6dG/LrIDwuueQSdZvWVasdT0X8TZHQOnRtXaPaecDWca59B6+55pqQ12Z7nbVr1xrjtv3TTwd9IBAwxsvKykLOsXVDa93DWVlZak5UVJQxnpSUpOYUFRUZ47aO8CPhih8AAIAjKPwAAAAcQeEHAADgCAo/AAAAR1D4AQAAOILCDwAAwBHOjnMZPHiwMe7nRut+RrP4Gdnih+11tBb27t27qzmMc0G42G4yrn1vbaMfNNqIC7/8jHM5cOCAMX711VerOc8++2zIr4PQtGzZ0hi/+OKL1RxtbEu4R4FpOeXl5WqO9t3UjvUiIm3btjXGS0tL1ZySkhJj3DaeRhtDo404EdE/A9v70T4D2xgc7We3fft2NUfb1qBBAzVHe6+2z02TmJgYcs5hXPEDAABwBIUfAACAIyj8AAAAHEHhBwAA4AgKPwAAAEdEeEfZbhTuzrgTTbvBcnFxsZoTzo5fW2egn89ay7G9jnaj7eXLl6s5AwYMCG1hJ7nj1V0ditq2r2lsHXNa95uto1H73Gw/44qKCnVbOGmvk56erubUtu/Bybiv1atnHmzRokULNee//uu/jPHWrVurOVqnZ3x8vJqjbbN1p2rnKFuOn45SrQvW1nWvrUH7GYjo78f289H2G9v5W+tSzs3NVXO0rlrbZ1BYWGiM245D2hreeustNefTTz9Vt4lwxQ8AAMAZFH4AAACOoPADAABwBIUfAACAIyj8AAAAHEHhBwAA4Ai9j7oW0G6mLSKSlJRkjO/fv1/N8TMuws9N5bXnC/d4B210zaWXXhrW14Hbzj77bGO8SZMmas7WrVuNcdvoCW3/sI1K0PZP2xgkjS3HzyiT/v37G+Pz588P+blgFhMTE3LOvHnzjHFtJIhf2viTlJQUNScQCIQUt72On/PNgQMH1G3aflhaWqrmaOco2z6trcH2OtqYKNtnoNUXRUVFao7GT53g5xgVfD3fmQAAADilUPgBAAA4gsIPAADAERR+AAAAjqDwAwAAcESt7urVuuJsbJ0y2o2p/XQ/2bp4wnkzc9vr+OkK0joxd+zYEfJzwQ3Dhw83xvfs2aPmaF17frrfbLTXKSsrU3O0Ndj2W+0G8bb95rzzzjPG6eoNn8LCQmM8Pj5ezWnXrp0xbusQ1jpKbZ2m2tps300tx0bbB/xMq7CtLSoqyhi3depr27TnEtF/DomJiWqONkVg7Nixas7zzz9vjF9++eVqzj//+U9jvHnz5mqO9l20fXeOhCt+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxB4QcAAOAICj8AAABH1OpxLtrN4U9VtrExfm5Q70ePHj2M8VmzZoX1dVB7XHDBBca4bRyBNi7iWG5MHsrz2W4272esk58b1Pfr188Y/+1vf6vmIDx27twZ8jbteyEikpSUZIzHxcWpObGxsSE9l4g+/sQ2msXPODLte1uvnl5SaPt0eXm5mqONQbK9H+0zCAQCak5JSYkxvnnzZjVn/PjxxviPP/6o5iQnJxvj0dHRas7evXuN8e3bt6s5R8IVPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOoPADAABwRK3u6m3btq26TesWOl78dOja2LqcNH46JFu2bBlyDty2ePFiY/w3v/mNmpOXl2eM2zrzNLbvudaJqd3oXUTv0LV1AmtsXZ1r1qwJ+flw4tgmKOzZsyekOE4O7733nq9tobJ1kdcErvgBAAA4gsIPAADAERR+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxRq8e5pKamqtu0m0zbbrSt8TNKJdy0NWg3xhbxN9KmYcOGIefAbQ888IAx3qdPHzWnQ4cOxnhubq6ao41gse0D2qgXPzebt41zSUtLM8YTExPVnJtvvlndBgB+ccUPAADAERR+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxRq7t6bTdn19g6dG2dfhrbjbv9rEFj61zUREREGONlZWVqTrNmzUJ+HcDknHPOUbetXbvWGM/MzFRztBud2/YNrYvftt9q3buNGjVSc7Zt22aMsz8BON644gcAAOAICj8AAABHUPgBAAA4gsIPAADAERR+AAAAjqDwAwAAcEStHudiU15eboxr4x1stLEofvkZG6OxjaWIjIw0xvPz89Uc7WbzgEbbP2xji9q3b2+MP/bYY2rOqFGjjPFAIKDmlJSUhLy21NRUYzw7O1vN6dKli7oNAI4nrvgBAAA4gsIPAADAERR+AAAAjqDwAwAAcASFHwAAgCMiPFv72s8fGObO1eNhyZIl6rbExERjPC4uTs3RumC1uIi/jsbKysqQc7TuXdvPzU/ODz/8YIz/93//t5pzMjvKr/9xdSrua37YOuht3eiamJgYY3zZsmVqToMGDYxx29qeeuopY/yVV16xrA7sa8DxcaR9jSt+AAAAjqDwAwAAcASFHwAAgCMo/AAAABxB4QcAAOAICj8AAABH1DvRC6hJc+fOVbelp6cb48nJyWpOeXm5MZ6QkKDmaKNZDh48GHJOYWGhmqONlLGNxWjatKkxXlRUpOZkZWWp24BQ2L6bdeqE/m/SkpISY3zYsGFqzurVq43xqVOnqjl+xrb4GesEADWBK34AAACOoPADAABwBIUfAACAIyj8AAAAHEHhBwAA4IgI7yjbyly5mXViYqK6rW3btsZ4hw4d1Jw1a9YY4w0bNlRziouLjXFbJ7D289m3b5+as2fPHmN8+/btak5tczJ2Vbqyr50M3nvvPWP8jTfeUHO0aQG2n5v2PbN1L2vd/acq9jXg+DjSvsYVPwAAAEdQ+AEAADiCwg8AAMARFH4AAACOoPADAABwBIUfAACAI456nAsAAABObVzxAwAAcASFHwAAgCMo/AAAABxB4QcAAOAICj8AAABHUPgBAAA4gsIPAADAERR+AAAAjqDwAwAAcMT/A6Z2iwRMsNQsAAAAAElFTkSuQmCC\n"},"metadata":{}}]},{"cell_type":"markdown","source":["#### Creating a Custom Dataset for your files\n","\n","A custom Dataset class must implement three functions: `__init__`, `__len__`, and `__getitem__`.\n","Take a look at this implementation; the FashionMNIST images are stored\n","in a directory ``img_dir``, and their labels are stored separately in a CSV file ``annotations_file``.\n","\n","In the next sections, we'll break down what's happening in each of these functions.\n","\n","##### ``__init__``\n","\n","The __init__ function is run once when instantiating the Dataset object. We initialize\n","the directory containing the images, the annotations file, and both transforms (covered\n","in more detail in the next section).\n","\n","##### ``__len__``\n","\n","The __len__ function returns the number of samples in our dataset.\n","\n","##### ``__getitem__``\n","\n","The __getitem__ function loads and returns a sample from the dataset at the given index ``idx``.\n","Based on the index, it identifies the image's location on disk, converts that to a tensor using ``read_image``, retrieves the\n","corresponding label from the csv data in ``self.img_labels``, calls the transform functions on them (if applicable), and returns the\n","tensor image and corresponding label in a tuple."],"metadata":{"id":"dH3bsTn4MspB"}},{"cell_type":"code","source":["import os\n","import pandas as pd\n","from torchvision.io import read_image\n","\n","class CustomImageDataset(Dataset):\n","    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):\n","        self.img_labels = pd.read_csv(annotations_file)\n","        self.img_dir = img_dir\n","        self.transform = transform\n","        self.target_transform = target_transform\n","\n","    def __len__(self):\n","        return len(self.img_labels)\n","\n","    def __getitem__(self, idx):\n","        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])\n","        image = read_image(img_path)\n","        label = self.img_labels.iloc[idx, 1]\n","        if self.transform:\n","            image = self.transform(image)\n","        if self.target_transform:\n","            label = self.target_transform(label)\n","        return image, label"],"metadata":{"id":"fcuxZDgMMqyI"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["#### Preparing your data for training with DataLoaders\n","The ``Dataset`` retrieves our dataset's features and labels one sample at a time. While training a model, we typically want to\n","pass samples in \"minibatches\", reshuffle the data at every epoch to reduce model overfitting, and use Python's ``multiprocessing`` to\n","speed up data retrieval.\n","\n","``DataLoader`` is an iterable that abstracts this complexity for us in an easy API."],"metadata":{"id":"fLGnm_BxM7U_"}},{"cell_type":"code","source":["from torch.utils.data import DataLoader\n","\n","train_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)\n","test_dataloader = DataLoader(test_data, batch_size=64, shuffle=True)"],"metadata":{"id":"Kr00KinsMxJk"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["#### Iterate through the DataLoader\n","\n","We have loaded that dataset into the ``DataLoader`` and can iterate through the dataset as needed.\n","Each iteration below returns a batch of ``train_features`` and ``train_labels`` (containing ``batch_size=64`` features and labels respectively).\n","Because we specified ``shuffle=True``, after we iterate over all batches the data is shuffled (for finer-grained control over\n","the data loading order, take a look at [Samplers](https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler))."],"metadata":{"id":"LqMCsSptNDPu"}},{"cell_type":"code","source":["# Display image and label.\n","train_features, train_labels = next(iter(train_dataloader))\n","print(f\"Feature batch shape: {train_features.size()}\")\n","print(f\"Labels batch shape: {train_labels.size()}\")\n","img = train_features[0].squeeze()\n","label = train_labels[0]\n","plt.imshow(img, cmap=\"gray\")\n","plt.show()\n","print(f\"Label: {label}\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":482},"id":"zrabfya1NAws","executionInfo":{"status":"ok","timestamp":1756474864033,"user_tz":240,"elapsed":215,"user":{"displayName":"Natalie Sarabosing","userId":"15403124364672772589"}},"outputId":"9d259df8-bc51-4ef5-9d86-8b4a2edf4007"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Feature batch shape: torch.Size([64, 1, 28, 28])\n","Labels batch shape: torch.Size([64])\n"]},{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAHvBJREFUeJzt3X1slfX9//HXaWkPN7YHC/SO2xYVjNw4UDqC4g2VGxcHwh/e/QGbweCKUZm6sE3RbUkVE77GheGSbTAzQacZENlCAlVKNgFDhTDnhrTrBAMtSuw5tKU3tJ/fH/w825Hbz8U5fbfl+Ug+CT3nevd69+rleXn1XH035JxzAgCgi6VZNwAAuDIRQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADDRx7qBb+rs7NTRo0eVlZWlUChk3Q4AwJNzTidPnlRhYaHS0s5/ndPtAujo0aMaPny4dRsAgMt05MgRDRs27LzPd7sfwWVlZVm3AABIgou9nqcsgFavXq1Ro0apb9++Kikp0YcffnhJdfzYDQB6h4u9nqckgN566y0tW7ZMK1as0EcffaSJEydq1qxZOn78eCp2BwDoiVwKTJkyxZWVlcU/7ujocIWFha68vPyitdFo1ElisVgsVg9f0Wj0gq/3Sb8CamtrU1VVlUpLS+OPpaWlqbS0VLt27Tpr+9bWVsVisYQFAOj9kh5AX375pTo6OpSXl5fweF5enurq6s7avry8XJFIJL64Aw4Argzmd8EtX75c0Wg0vo4cOWLdEgCgCyT994AGDx6s9PR01dfXJzxeX1+v/Pz8s7YPh8MKh8PJbgMA0M0l/QooMzNTkydPVkVFRfyxzs5OVVRUaOrUqcneHQCgh0rJJIRly5Zp4cKFuummmzRlyhS98sorampq0ve+971U7A4A0AOlJIDuu+8+ffHFF3ruuedUV1enG2+8UVu3bj3rxgQAwJUr5Jxz1k38r1gspkgkYt0GAOAyRaNRZWdnn/d587vgAABXJgIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAm+lg3AKRCKBQKVOec65J9def9SNKNN97oXdPc3Oxd8+mnn3rX9EZBz9cggp4TqcAVEADABAEEADCR9AB6/vnnFQqFEtbYsWOTvRsAQA+XkveAbrjhBm3fvv2/O+nDW00AgEQpSYY+ffooPz8/FZ8aANBLpOQ9oEOHDqmwsFDFxcV66KGHdPjw4fNu29raqlgslrAAAL1f0gOopKRE69at09atW7VmzRrV1tbq1ltv1cmTJ8+5fXl5uSKRSHwNHz482S0BALqhkEvxTeENDQ0aOXKkVq1apYcffvis51tbW9Xa2hr/OBaLEUK4bPweUPD9SPweUFfrrb8HFI1GlZ2dfd7nU353wMCBA3Xdddepurr6nM+Hw2GFw+FUtwEA6GZS/ntAjY2NqqmpUUFBQap3BQDoQZIeQE899ZQqKyv1n//8Rx988IHuvfdepaen64EHHkj2rgAAPVjSfwT3+eef64EHHtCJEyc0ZMgQ3XLLLdq9e7eGDBmS7F0BAHqwlN+E4CsWiykSiVi3gRTpqjdbg57W3b2/rvLnP//Zu2bbtm3eNa+88op3Td++fb1rWlpavGu6Unp6undNR0dHCjpJrovdhMAsOACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACZS/gfpgMvVlYM7u/uQUF9r1qwJVPetb33Lu6apqcm7Jsgw0uuuu8675rPPPvOukaT29nbvmiB/GbYnDBZNBa6AAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmmIaNLtXbpk1LUigU8q4JchwWLVrkXTN79mzvGkmqqanxrmltbfWuufrqq71r6uvrvWvC4bB3jSRFo1HvmszMTO+a2267zbtmwYIF3jWS9Pjjj3vXBPneXgqugAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJhgGCm6va4a9hlUkH0NGTLEu2bo0KHeNfv37/eukaTx48d71+Tk5HjXbNmyxbumra3Nu6ajo8O7RpIyMjK8a4IchyADVhsaGrxrJKm9vT1QXSpwBQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEw0jR7XXlYNEg+vXr512zfv1675ogQy63b9/uXSNJkyZN8q4JMvDz+PHj3jV5eXneNa2trd41QR05csS7prGxMQWdnFtmZqZ3TUtLSwo64QoIAGCEAAIAmPAOoJ07d+qee+5RYWGhQqGQNm3alPC8c07PPfecCgoK1K9fP5WWlurQoUPJ6hcA0Et4B1BTU5MmTpyo1atXn/P5lStX6tVXX9Vrr72mPXv2aMCAAZo1a1bKfoYIAOiZvG9CmDNnjubMmXPO55xzeuWVV/TTn/5Uc+fOlSS9/vrrysvL06ZNm3T//fdfXrcAgF4jqe8B1dbWqq6uTqWlpfHHIpGISkpKtGvXrnPWtLa2KhaLJSwAQO+X1ACqq6uTdPZtknl5efHnvqm8vFyRSCS+hg8fnsyWAADdlPldcMuXL1c0Go2vIPfQAwB6nqQGUH5+viSpvr4+4fH6+vr4c98UDoeVnZ2dsAAAvV9SA6ioqEj5+fmqqKiIPxaLxbRnzx5NnTo1mbsCAPRw3nfBNTY2qrq6Ov5xbW2t9u/fr5ycHI0YMUJPPPGEfvGLX+jaa69VUVGRnn32WRUWFmrevHnJ7BsA0MN5B9DevXt1xx13xD9etmyZJGnhwoVat26dnnnmGTU1NemRRx5RQ0ODbrnlFm3dulV9+/ZNXtcAgB4v5LrZpMdYLKZIJGLdBrqRUCjkXdOVp/U777zjXfPJJ5941/zv//hdqjFjxnjXSMGGYwb5Ph04cMC7Jsj3tr293btGCvY1paene9cE+ZqC3jG8atUq75oNGzYE2lc0Gr3g+/rmd8EBAK5MBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAAT3n+Oobvq7hOTg+iNX1MQXfk13XXXXd41o0aN8q45evSod82+ffu8a4qLi71rJKmpqcm7JjMz07umo6PDu+bUqVPeNUEmVEtSbm6ud83VV1/tXdOnj/9LcdC/GjB//nzvmqDTsC+GKyAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmes0w0iADK9PSguVvkLqu6i/Ifk6fPu1d05WysrK8a1asWBFoXw8++KB3zcsvv+xd8/3vf9+7JshgzMbGRu8aSQqHw941QQZqjhkzxrsmIyPDuyZIb1LXHYevvvrKuybo93bChAmB6lKBKyAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmQi7I9MoUisViikQiCoVCCoVCl1wX5MvoZl/6FWHRokXeNb/5zW+8a1566SXvGkn69NNPvWtWrVrlXXP48GHvmkGDBnnXBB0021UDP4PUdHR0eNe0tbV510hSe3u7d43P69bXWltbvWtaWlq8ayRp3Lhx3jW+A4Gdc2ptbVU0GlV2dvZ5t+MKCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAn/SYBdxHdQaJDBokGGBkrSpEmTumxfvnyHBkrSlClTAu1r8eLF3jVVVVXeNXfeead3zTvvvONdI0m/+93vvGuOHj3qXRPkfG1sbPSuCTLsUwo28DNITZBhn0GEw+FAdUH6GzBggHdNkKGxQQfNBnktGjVqlNf2HR0dOnTo0EW34woIAGCCAAIAmPAOoJ07d+qee+5RYWGhQqGQNm3alPD8okWL4n/L5+s1e/bsZPULAOglvAOoqalJEydO1OrVq8+7zezZs3Xs2LH42rBhw2U1CQDofbzfoZwzZ47mzJlzwW3C4bDy8/MDNwUA6P1S8h7Qjh07lJubqzFjxujRRx/ViRMnzrtta2urYrFYwgIA9H5JD6DZs2fr9ddfV0VFhV566SVVVlZqzpw5571Fs7y8XJFIJL6GDx+e7JYAAN1Q0n8P6P7774//e/z48ZowYYJGjx6tHTt2aMaMGWdtv3z5ci1btiz+cSwWI4QA4AqQ8tuwi4uLNXjwYFVXV5/z+XA4rOzs7IQFAOj9Uh5An3/+uU6cOKGCgoJU7woA0IN4/wiusbEx4WqmtrZW+/fvV05OjnJycvTCCy9owYIFys/PV01NjZ555hldc801mjVrVlIbBwD0bN4BtHfvXt1xxx3xj79+/2bhwoVas2aNDhw4oN///vdqaGhQYWGhZs6cqZ///OeBZzEBAHqnkAsyFTGFYrGYIpGId12Qmg8++MC7RjrzY0Vf6enp3jVDhw71rsnIyPCuaW5u9q6RpMzMzC7Z1+7du71r5s6d610jBRs+2dDQ4F3T2dnpXdO/f3/vmiDnnRRsYGWQl5KuGsKZlhbs3YYg36cgxzzIINegX1NxcbF3za233uq1fUdHh/bt26doNHrB9/WZBQcAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMJH0P8ltZf78+d41QSbQBhVkuvDx48e9a4JMww7SmxRscnSQv3gbZLJ1LBbzrpGCTesOcsyDTEwOcryDTHMOKsg07CDnXlee40GOX5Bp3UEn0gcR5HXlH//4h9f2l3oucAUEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADARMgFmSCYQrFYTJFIRFlZWV4DBP/+979776uhocG7RpJaWlq8a4IMKOzfv3+X1AQdyhpkOGbfvn29a7pyoGaQ71Namv//xwX5moIM4ezXr593TdB9BRFkKGuQYxfkXJWC9Rfk2AU574J+TSNGjPCuufvuu722b29v1/bt2xWNRi84gJgrIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACb6WDdwPnfddZfXUL8gA0JPnTrlXSMFG/gZZKhhZmZml+wn6ODJrhpY2VWDO6VgxzzIMNIggvQWdGBlW1ubd02Q71OQIZxB9hN05nKQ/vr08X9ZDfLfbdDXr5ycHO+a/Px8r+0v9fzhCggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAICJbjuMNBqNeg31a21t9d5HcXGxd40kffXVV941QfpraGjwrgki6ODOIAMU+/Xr511z1VVXedcEFWR4Z5CaIMchyBDOkSNHetdIUkdHh3dNY2Ojd01zc7N3zRdffOFdE2S4qhRs0GyQwchBhqUGeU2RpJqaGu8a36/pUv+b4AoIAGCCAAIAmPAKoPLyct18883KyspSbm6u5s2bp4MHDyZs09LSorKyMg0aNEhXXXWVFixYoPr6+qQ2DQDo+bwCqLKyUmVlZdq9e7e2bdum9vZ2zZw5U01NTfFtnnzySb377rt6++23VVlZqaNHj2r+/PlJbxwA0LN53YSwdevWhI/XrVun3NxcVVVVafr06YpGo/rtb3+r9evX684775QkrV27Vtdff712796tb3/728nrHADQo13We0DRaFTSf//Ea1VVldrb21VaWhrfZuzYsRoxYoR27dp1zs/R2tqqWCyWsAAAvV/gAOrs7NQTTzyhadOmady4cZKkuro6ZWZmauDAgQnb5uXlqa6u7pyfp7y8XJFIJL6GDx8etCUAQA8SOIDKysr08ccf680337ysBpYvX65oNBpfR44cuazPBwDoGQL9IurSpUu1ZcsW7dy5U8OGDYs/np+fr7a2NjU0NCRcBdXX1ys/P/+cnyscDiscDgdpAwDQg3ldATnntHTpUm3cuFHvvfeeioqKEp6fPHmyMjIyVFFREX/s4MGDOnz4sKZOnZqcjgEAvYLXFVBZWZnWr1+vzZs3KysrK/6+TiQSUb9+/RSJRPTwww9r2bJlysnJUXZ2th577DFNnTqVO+AAAAm8AmjNmjWSpNtvvz3h8bVr12rRokWSpP/7v/9TWlqaFixYoNbWVs2aNUu/+tWvktIsAKD3CLkgU/BSKBaLKRKJeNcFGaj5/PPPe9dI0oMPPuhdE2Rg5YABA7xrumqIZNB9BRnuePr0ae+arhRkWOr//vL2paqurvau+ctf/uJdI0lbtmzxrjnfna4X8u9//9u7Jjs727vm1KlT3jVSsIG7Qf676Mr3wYcMGeJdM23aNK/tT58+rQ8//FDRaPSC3y9mwQEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATPSaadjdXZAJvpMmTfKuWbBggXfNTTfd5F0jSc3Nzd41BQUF3jUvvviid81PfvIT7xpJ2rRpk3fN9ddf713z3e9+17umN3r55Ze9a8aOHetdE2SqtRRs0nmQadhfffWVd01VVZV3jSRt3LjRu+aTTz4JtC+mYQMAuiUCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmes0w0rQ0/yzt7Oz0rgGs9OnTx7smyGDMoLrZSwm6AYaRAgC6JQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACb8pxt2UwwWRW93+vRp6xaApOIKCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJrwCqLy8XDfffLOysrKUm5urefPm6eDBgwnb3H777QqFQglryZIlSW0aANDzeQVQZWWlysrKtHv3bm3btk3t7e2aOXOmmpqaErZbvHixjh07Fl8rV65MatMAgJ7P6y+ibt26NeHjdevWKTc3V1VVVZo+fXr88f79+ys/Pz85HQIAeqXLeg8oGo1KknJychIef+ONNzR48GCNGzdOy5cvV3Nz83k/R2trq2KxWMICAFwBXEAdHR3uO9/5jps2bVrC47/+9a/d1q1b3YEDB9wf/vAHN3ToUHfvvfee9/OsWLHCSWKxWCxWL1vRaPSCORI4gJYsWeJGjhzpjhw5csHtKioqnCRXXV19zudbWlpcNBqNryNHjpgfNBaLxWJd/rpYAHm9B/S1pUuXasuWLdq5c6eGDRt2wW1LSkokSdXV1Ro9evRZz4fDYYXD4SBtAAB6MK8Acs7pscce08aNG7Vjxw4VFRVdtGb//v2SpIKCgkANAgB6J68AKisr0/r167V582ZlZWWprq5OkhSJRNSvXz/V1NRo/fr1uvvuuzVo0CAdOHBATz75pKZPn64JEyak5AsAAPRQPu/76Dw/51u7dq1zzrnDhw+76dOnu5ycHBcOh90111zjnn766Yv+HPB/RaNR859bslgsFuvy18Ve+0P/P1i6jVgspkgkYt0GAOAyRaNRZWdnn/d5ZsEBAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEx0uwByzlm3AABIgou9nne7ADp58qR1CwCAJLjY63nIdbNLjs7OTh09elRZWVkKhUIJz8ViMQ0fPlxHjhxRdna2UYf2OA5ncBzO4DicwXE4ozscB+ecTp48qcLCQqWlnf86p08X9nRJ0tLSNGzYsAtuk52dfUWfYF/jOJzBcTiD43AGx+EM6+MQiUQuuk23+xEcAODKQAABAEz0qAAKh8NasWKFwuGwdSumOA5ncBzO4DicwXE4oycdh253EwIA4MrQo66AAAC9BwEEADBBAAEATBBAAAATPSaAVq9erVGjRqlv374qKSnRhx9+aN1Sl3v++ecVCoUS1tixY63bSrmdO3fqnnvuUWFhoUKhkDZt2pTwvHNOzz33nAoKCtSvXz+Vlpbq0KFDNs2m0MWOw6JFi846P2bPnm3TbIqUl5fr5ptvVlZWlnJzczVv3jwdPHgwYZuWlhaVlZVp0KBBuuqqq7RgwQLV19cbdZwal3Icbr/99rPOhyVLlhh1fG49IoDeeustLVu2TCtWrNBHH32kiRMnatasWTp+/Lh1a13uhhtu0LFjx+Lrr3/9q3VLKdfU1KSJEydq9erV53x+5cqVevXVV/Xaa69pz549GjBggGbNmqWWlpYu7jS1LnYcJGn27NkJ58eGDRu6sMPUq6ysVFlZmXbv3q1t27apvb1dM2fOVFNTU3ybJ598Uu+++67efvttVVZW6ujRo5o/f75h18l3KcdBkhYvXpxwPqxcudKo4/NwPcCUKVNcWVlZ/OOOjg5XWFjoysvLDbvqeitWrHATJ060bsOUJLdx48b4x52dnS4/P9+9/PLL8ccaGhpcOBx2GzZsMOiwa3zzODjn3MKFC93cuXNN+rFy/PhxJ8lVVlY658587zMyMtzbb78d3+af//ynk+R27dpl1WbKffM4OOfcbbfd5h5//HG7pi5Bt78CamtrU1VVlUpLS+OPpaWlqbS0VLt27TLszMahQ4dUWFio4uJiPfTQQzp8+LB1S6Zqa2tVV1eXcH5EIhGVlJRckefHjh07lJubqzFjxujRRx/ViRMnrFtKqWg0KknKycmRJFVVVam9vT3hfBg7dqxGjBjRq8+Hbx6Hr73xxhsaPHiwxo0bp+XLl6u5udmivfPqdsNIv+nLL79UR0eH8vLyEh7Py8vTv/71L6OubJSUlGjdunUaM2aMjh07phdeeEG33nqrPv74Y2VlZVm3Z6Kurk6Sznl+fP3clWL27NmaP3++ioqKVFNTox//+MeaM2eOdu3apfT0dOv2kq6zs1NPPPGEpk2bpnHjxkk6cz5kZmZq4MCBCdv25vPhXMdBkh588EGNHDlShYWFOnDggH70ox/p4MGD+tOf/mTYbaJuH0D4rzlz5sT/PWHCBJWUlGjkyJH64x//qIcfftiwM3QH999/f/zf48eP14QJEzR69Gjt2LFDM2bMMOwsNcrKyvTxxx9fEe+DXsj5jsMjjzwS//f48eNVUFCgGTNmqKamRqNHj+7qNs+p2/8IbvDgwUpPTz/rLpb6+nrl5+cbddU9DBw4UNddd52qq6utWzHz9TnA+XG24uJiDR48uFeeH0uXLtWWLVv0/vvvJ/z5lvz8fLW1tamhoSFh+956PpzvOJxLSUmJJHWr86HbB1BmZqYmT56sioqK+GOdnZ2qqKjQ1KlTDTuz19jYqJqaGhUUFFi3YqaoqEj5+fkJ50csFtOePXuu+PPj888/14kTJ3rV+eGc09KlS7Vx40a99957KioqSnh+8uTJysjISDgfDh48qMOHD/eq8+Fix+Fc9u/fL0nd63ywvgviUrz55psuHA67devWuU8++cQ98sgjbuDAga6urs66tS71wx/+0O3YscPV1ta6v/3tb660tNQNHjzYHT9+3Lq1lDp58qTbt2+f27dvn5PkVq1a5fbt2+c+++wz55xzL774ohs4cKDbvHmzO3DggJs7d64rKipyp06dMu48uS50HE6ePOmeeuopt2vXLldbW+u2b9/uJk2a5K699lrX0tJi3XrSPProoy4SibgdO3a4Y8eOxVdzc3N8myVLlrgRI0a49957z+3du9dNnTrVTZ061bDr5LvYcaiurnY/+9nP3N69e11tba3bvHmzKy4udtOnTzfuPFGPCCDnnPvlL3/pRowY4TIzM92UKVPc7t27rVvqcvfdd58rKChwmZmZbujQoe6+++5z1dXV1m2l3Pvvv+8knbUWLlzonDtzK/azzz7r8vLyXDgcdjNmzHAHDx60bToFLnQcmpub3cyZM92QIUNcRkaGGzlypFu8eHGv+5+0c339ktzatWvj25w6dcr94Ac/cFdffbXr37+/u/fee92xY8fsmk6Bix2Hw4cPu+nTp7ucnBwXDofdNddc455++mkXjUZtG/8G/hwDAMBEt38PCADQOxFAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADDx/wAEBWvVQ6IR6AAAAABJRU5ErkJggg==\n"},"metadata":{}},{"output_type":"stream","name":"stdout","text":["Label: 9\n"]}]},{"cell_type":"markdown","metadata":{"id":"dHOu0NiW1JoE"},"source":["## Neural Network *Construction*\n","\n","Neural networks are usually defined by classes that inherit from the **nn.Module** class.\n","\n","For each network, we define a **`forward`** function that is run in the forward pass through the network.\n","\n","In Pytorch, **backpropagation** is done through automatic differentiation. It traverses backwards from the output, collecting gradients (derivatives of the error with respect to the parameters of the functions), and optimizes using gradient descent (or whichever optimization algorithm is chosen). Note that when you instantiate parameters that need to be updated, you should make sure that `requires_grad` is set to `True`. This signals to autograd that every operation on it should be tracked.\n","\n","Below are several neural net modules we can use in our final architecture:"]},{"cell_type":"code","metadata":{"id":"bkRU6SuC5Yoy"},"source":["from itertools import tee\n","import torch\n","import torch.nn as nn\n","from torch.autograd import Function\n","import numpy as np\n","\n","\n","# Basic multi-layer perceptron class\n","class MLP(nn.Module):\n","    def __init__(self, input_size, hidden_size, output_size):\n","        super(MLP, self).__init__()\n","        self.linear_layer = nn.Linear(input_size, hidden_size)\n","        self.activation = nn.Sigmoid()\n","        self.output_layer = nn.Linear(hidden_size, output_size)\n","\n","    def forward(self, x):\n","        return self.output_layer(self.activation(self.linear_layer(x)))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vH1QBFekLju4"},"source":["## Advanced: Under the hood of a Linear layer"]},{"cell_type":"markdown","metadata":{"id":"mpJZhsyt9Fpl"},"source":["Sometimes, we may want to add operations to `autograd` that is not already included in pytorch. To help you understand how we might do this, let's study how nn.Linear is implemented.\n","\n","![Screen Shot 2021-02-10 at 10.54.26 PM.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwAAADrCAYAAACLrkUlAAAMSGlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnltSSWiBCEgJvQlSpEsJoUUQkCrYCEkgocSQEETsLssquHYRARu6KqLoWgBZK+paF8HuWh4WVJR1sWBD5U0KrOt+773vne+be/+cOec/JXPvnQFAp5YnleahugDkSwplCZGhrIlp6SxSN0AADmhAF1jw+HIpOz4+BkAZuv9d3l6H1lCuuCi5/jn/X0VPIJTzAUDiIc4UyPn5EB8AAC/lS2WFABB9od56RqFUiSdDbCCDCUIsVeJsNS5V4kw1rlLZJCVwIN4FAJnG48myAdBugXpWET8b8mjfhNhNIhBLANAhQxzEF/EEEEdBPCo/f7oSQzvgkPkVT/bfODOHOXm87GGsrkUl5DCxXJrHm/l/tuN/S36eYiiGHRw0kSwqQVkz7NvN3OnRSkyDuFeSGRsHsT7E78UClT3EKFWkiEpW26OmfDkH9gwwIXYT8MKiITaFOEKSFxuj0WdmiSO4EMMVghaLC7lJGt9FQnl4ooazVjY9IW4IZ8k4bI1vI0+miqu0P6XITWZr+G+KhNwh/jcloqRUdc4YtUicEguxNsRMeW5itNoGsykRcWKHbGSKBGX+NhD7CyWRoWp+bGqWLCJBYy/Llw/Viy0SibmxGlxdKEqK0vDs4vNU+RtB3CKUsJOHeITyiTFDtQiEYeHq2rEOoSRZUy/WJS0MTdD4vpLmxWvscaowL1Kpt4LYVF6UqPHFgwrhglTz47HSwvgkdZ54Zg5vXLw6H7wYxAAOCAMsoIAjE0wHOUDc3tvcC3+pZyIAD8hANhACF41myCNVNSOB10RQAv6ASAjkw36hqlkhKIL6z8Na9dUFZKlmi1QeueAxxPkgGuTB3wqVl2Q4Wgp4BDXif0Tnw1zz4FDO/VPHhpoYjUYxxMvSGbIkhhPDiFHECKIjboIH4QF4DLyGwOGB++J+Q9n+ZU94TOgkPCBcI3QRbk0TL5R9Uw8LjAddMEKEpubMr2vG7SCrFx6KB0J+yI0zcRPggo+Bkdh4MIztBbUcTebK6r/l/lsNX3VdY0dxo6CUEZQQisO3ntpO2l7DLMqeft0hda6Zw33lDM98G5/zVacF8B79rSW2CNuPncFOYOeww1gzYGHHsBbsInZEiYdX0SPVKhqKlqDKJxfyiP8Rj6eJqeyk3K3Brcftk3quUFisfD8CznTpTJk4W1TIYsM3v5DFlfBdR7E83DzcAVB+R9SvqddM1fcBYZ7/S1dwHAC/cqjM/kvHswbg0GMAGG//0lm/go/HcgCOdPAVsiK1DldeCIAKdOATZQzMgTVwgPV4AG8QAEJAOBgH4kASSANTYZdFcD3LwAwwGywAZaACLAdrQDXYCLaAHWA32AeawWFwAvwKLoAOcA3chqunGzwHfeAtGEAQhITQEQZijFggtogz4oH4IkFIOBKDJCBpSAaSjUgQBTIb+Q6pQFYi1chmpB75GTmEnEDOIZ3ILeQ+0oO8Qj6iGEpDDVAz1A4djfqibDQaTUKnoNloAVqClqJL0Sq0Dt2FNqEn0AvoNbQLfY72YwDTwpiYJeaC+WIcLA5Lx7IwGTYXK8cqsTqsEWuF//MVrAvrxT7gRJyBs3AXuIKj8GScjxfgc/EleDW+A2/CT+FX8Pt4H/6FQCeYEpwJ/gQuYSIhmzCDUEaoJGwjHCSchk9TN+EtkUhkEu2JPvBpTCPmEGcRlxDXE/cQjxM7iQ+J/SQSyZjkTAokxZF4pEJSGWkdaRfpGOkyqZv0nqxFtiB7kCPI6WQJeSG5kryTfJR8mfyEPEDRpdhS/ClxFAFlJmUZZSullXKJ0k0ZoOpR7amB1CRqDnUBtYraSD1NvUN9raWlZaXlpzVBS6w1X6tKa6/WWa37Wh9o+jQnGoc2maagLaVtpx2n3aK9ptPpdvQQejq9kL6UXk8/Sb9Hf6/N0HbV5moLtOdp12g3aV/WfqFD0bHVYetM1SnRqdTZr3NJp1eXomuny9Hl6c7VrdE9pHtDt1+PoeeuF6eXr7dEb6feOb2n+iR9O/1wfYF+qf4W/ZP6DxkYw5rBYfAZ3zG2Mk4zug2IBvYGXIMcgwqD3QbtBn2G+oZjDFMMiw1rDI8YdjExph2Ty8xjLmPuY15nfhxhNoI9Qjhi8YjGEZdHvDMaaRRiJDQqN9pjdM3oozHLONw413iFcbPxXRPcxMlkgskMkw0mp016RxqMDBjJH1k+ct/I301RUyfTBNNZpltML5r2m5mbRZpJzdaZnTTrNWeah5jnmK82P2reY8GwCLIQW6y2OGbxjGXIYrPyWFWsU6w+S1PLKEuF5WbLdssBK3urZKuFVnus7lpTrX2ts6xXW7dZ99lY2Iy3mW3TYPO7LcXW11Zku9b2jO07O3u7VLsf7Jrtntob2XPtS+wb7O840B2CHQoc6hyuOhIdfR1zHdc7djihTl5OIqcap0vOqLO3s9h5vXPnKMIov1GSUXWjbrjQXNguRS4NLvddma4xrgtdm11fjLYZnT56xegzo7+4ebnluW11u+2u7z7OfaF7q/srDycPvkeNx1VPumeE5zzPFs+XY5zHCMdsGHPTi+E13usHrzavz94+3jLvRu8eHxufDJ9anxu+Br7xvkt8z/oR/EL95vkd9vvg7+1f6L/P/88Al4DcgJ0BT8fajxWO3Tr2YaBVIC9wc2BXECsoI2hTUFewZTAvuC74QYh1iCBkW8gTtiM7h72L/SLULVQWejD0HcefM4dzPAwLiwwrD2sP1w9PDq8OvxdhFZEd0RDRF+kVOSvyeBQhKjpqRdQNrhmXz63n9o3zGTdn3KloWnRidHX0gxinGFlM63h0/Ljxq8bfibWNlcQ2x4E4btyquLvx9vEF8b9MIE6In1Az4XGCe8LshDOJjMRpiTsT3yaFJi1Lup3skKxIbkvRSZmcUp/yLjUsdWVq18TRE+dMvJBmkiZOa0knpaekb0vvnxQ+ac2k7slek8smX59iP6V4yrmpJlPzph6ZpjONN21/BiEjNWNnxideHK+O15/JzazN7ONz+Gv5zwUhgtWCHmGgcKXwSVZg1sqsp9mB2auye0TBokpRr5gjrha/zInK2ZjzLjcud3vuYF5q3p58cn5G/iGJviRXcmq6+fTi6Z1SZ2mZtKvAv2BNQZ8sWrZNjsinyFsKDeCG/aLCQfG94n5RUFFN0fsZKTP2F+sVS4ovznSauXjmk5KIkp9m4bP4s9pmW85eMPv+HPaczXORuZlz2+ZZzyud1z0/cv6OBdQFuQt+W+i2cOXCN9+lftdaalY6v/Th95HfN5Rpl8nKbvwQ8MPGRfgi8aL2xZ6L1y3+Ui4oP1/hVlFZ8WkJf8n5H91/rPpxcGnW0vZl3ss2LCculyy/viJ4xY6VeitLVj5cNX5V02rW6vLVb9ZMW3OuckzlxrXUtYq1XVUxVS3rbNYtX/epWlR9rSa0Zk+tae3i2nfrBesvbwjZ0LjRbGPFxo+bxJtubo7c3FRnV1e5hbilaMvjrSlbz/zk+1P9NpNtFds+b5ds79qRsONUvU99/U7Tncsa0AZFQ8+uybs6doftbml0ady8h7mnYi/Yq9j77OeMn6/vi97Xtt93f+MB2wO1BxkHy5uQpplNfc2i5q6WtJbOQ+MOtbUGtB78xfWX7YctD9ccMTyy7Cj1aOnRwWMlx/qPS4/3nsg+8bBtWtvtkxNPXj014VT76ejTZ3+N+PXkGfaZY2cDzx4+53/u0Hnf880XvC80XfS6ePA3r98Otnu3N13yudTS4dfR2jm28+jl4MsnroRd+fUq9+qFa7HXOq8nX795Y/KNrpuCm09v5d16+XvR7wO3598h3Cm/q3u38p7pvbp/Of5rT5d315H7YfcvPkh8cPsh/+HzR/JHn7pLH9MfVz6xeFL/1OPp4Z6Ino5nk551P5c+H+gt+0Pvj9oXDi8O/Bny58W+iX3dL2UvB18teW38evubMW/a+uP7773Nfzvwrvy98fsdH3w/nPmY+vHJwIxPpE9Vnx0/t36J/nJnMH9wUMqT8VRbAQwONCsLgFfbAaCnwb1DBwDUSepznkoQ9dlUhcB/wuqzoEq8AdgeAkDyfABi4B5lAxy2ENPgXblVTwoBqKfn8NCIPMvTQ81FgycewvvBwddmAJBaAfgsGxwcWD84+HkrTPYWAMcL1OdLpRDh2WCTqxJ1dL8A38q/AYNxf0kQcKUZAAAACXBIWXMAABYlAAAWJQFJUiTwAAACBWlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iPgogICAgICAgICA8ZXhpZjpQaXhlbFlEaW1lbnNpb24+NTUyPC9leGlmOlBpeGVsWURpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6UGl4ZWxYRGltZW5zaW9uPjE2NDY8L2V4aWY6UGl4ZWxYRGltZW5zaW9uPgogICAgICAgICA8dGlmZjpPcmllbnRhdGlvbj4xPC90aWZmOk9yaWVudGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4K7yY9bwAAQABJREFUeAHsnQe0VFWW9w+iqChgQMGED1BAQYwEBRUzRkRQTCDmbNuru6fnm57VX6+eme75erodFbW1CSqCEUObQ6uoGFCSEkygklQQEEEEW8Svfhv+5aWseu9Weu/We3uz6t2qc0/Y538O6/7Pvvvs0+iHlAQXR8ARcAQcAUfAEXAEHAFHoJ4isGk97VfZu8U6wT4htV7IWDI0atQo6FN2RbyBxCKQniNZ1pSaH1xdHAFHwBFwBBwBR6C8CDRKPZQz6Fp5G6zk2oFq3bp11oXGjRvX2JVo/k022cRIcI2FPENFI8D8YNwhsox5TZJv/prq8/uOgCPgCDgCjoAj8FMEnPD+FJOfpEBg+GQSmDVr1oRVq1YFrmvXrjWS06RJk7DFFluErbbaKmy22WYb1QW5kWVvoxv+o+IRyDW2X3/9tc0PzRHmEPODT7NmzTZaBGme+Ryp+OngHXAEHAFHwBFIGAJOeKsZEFnqlOWbb74J8+bNCwsXLgxLly4NkBkIzNZbbx0guuQnDx/SITStWrUKu+22W2jTpk2aMJMP8dfZQrZyr5lzZNmyZWHu3Lk2R1asWBFWr15tc0MLoO+//97mB+mbb755aNGiRWjdunXYddddw0477ZQGwudIGgr/4gg4Ao6AI+AIFI2AE94sEGaSjc8++yxMmzYtfPzxx2G77bYLnTp1MoKClQ6iy0dCWVnzIDyzZ88O77//vpHhffbZJ3Tt2tUIMvlzWQVVl1+Ti0CU6PL9ww8/tDmyaNEiW+DstddettjByg+x3XTTH93lv1/3ffh2zbdh7Xdrw7Ivl9kcoTwLp3333Td07NjRytB75kjmm4XkouKaOQKOgCPgCDgCyUTACW/GuEQJBta6J598MixZsiQceeSRoUOHDvYqOqNIrJ9ffvllmDx5cpg0aVLo2bNn6NWrl7k8ZJLrWJXVUSaRPOmMGlipo7+jqtVXCzZWWvlwf/TRR+HRRx8N2267bTj88MNtIRQlt1E8avoOWX7ttdfCrFmzwgknnBD2339/KxJtr6Y6/L4j4Ag4Ao6AI+AI/BQBJ7wRTKLEYuLEiWHK5CnhwIMODN26dUu7H0DuRPwomo3URQkg36MWupUrV4bx48eHTz/9NBx11FFhjz32MA2iRDuiUuK+RvUUFvQvigl5wCUbNonrUJ4KaY7885//DE899ZS5LhxzzDFhzz33TNcUHX8Ss+EQzcP36Bz59LNPw7PPPGtpJ510kr1VULvpRvyLI+AIOAKOgCPgCMRGwAnvBqh4zdx4k8b2CvmOO+4ITZs2DQMGDEhvPINwQEqykZc4aEMCITayDH7++edh9J2jQ9f9uoa+x/W1KqJkMk6dtZkH3XP1PXov+r029auNtkQ6ly9fHu68887QpUsXW7TQto1vKj4dc6hQoQ5E5Hfq1KnhvvvuC5deemlo27atuzcUCqyXcwQcAUfAEWjwCDjhTU0BEU2sdsOHDzc/20MPPdQmB9EXIKm5yF6+M8iI0QbiS3u33nqr+XoOGjTIqpIu+dZbrvwisGzQe+655wJEHT9TNlg988wztjGPV/n4OPMqH0snr+XB6/jjjw/bb7/9RtbfculZ7no1LgsWLAhjx44N/fv3NxcX2mWOFOrGkE1vEWvu4T8+atSogKUX3KVHtnKe5gg4Ao6AI+AIOALZEag5UGj2cvUmFcsuFrXvvvvOrHaQN5FdiAdEplRkF9BoCwJN3Wx2u+aaa8wvGEue7kNqkiL0/Yd1P9iGKnxKr7jiCotQgcWRz7nnnhu+/fbbwAY9LJ6kvfXWW2GXXXYxsktfSolfXeDCWDFun3zySbj33nvDBRdcYGSXvvEpJdmlf8wPFhq0y8Li2muvDS+++KItKtCDOeviCDgCjoAj4Ag4AvERaPy7lMTPXr9yQlb0Cvruu++2zWSdO3c2skFPIRflEuoWGSTqw5yP5oR5c+eF9u3bp5tMGlFkY1a7du0s6gQLA/yPW7ZsGR544IFw5plnhqqqqrD8q+VhyRdLwsCBA60fshCnO1VhXyCdENDlqU2Hf01Z4y+55JKwww47GBll7jTapDwnpTH2Rm5T7RPl4YADDggPP/xw2HHHHcN2225n7ZdzflbYMLm6joAj4Ag4Ao5AtQiUj9FV22zd34SIiTCMGzfOyBqhpEhHaoNsivTSXr9T+oX58+ebFY+2pQf36lrQB+KHHHHEEeG3v/2tbbrjN8SXV+7EmUVmfzjbrLt851W/MOZ3pYktiFJkF9eTO1I+u2eddZaRXaWH8nDdjWCKvg0YPHhweOihh8LixYuNhKOHiyPgCDgCjoAj4AjUjECDJrzAg+8pxExuDKTVBtmlHSRKes8555zw6quvGqEhPUmkF+KFcIjG1VdfHV544QX7jYUX4bAFhLjDuDYgtYmjNVjCP9EF0WOPPmYWVizxZpUvk1U3l/oivRxSccrJJxvpRY+kzZFc+nu6I+AIOAKOgCNQ1wg0SMIrMoPfKf6m/U7tZ+OAFbMuSBrEBdLNQRZ9+/YNTz/9tOlTF7rkmpDoIisvG+yuu+4605n4wvi0vvLKK3aCGL7QnCqH0K9KFw4N+W7td+Gwww6zroAD/2pbRHr3SG0KxO2FhRGSpEVRbWPi7TkCjoAj4Ag4AnERqHxGErenkXwiLM8//3zg9LMtt9jSyJusmJGstfaVjU9Y7eTDyyEVSJJeW4vAsnmNI5PZwMWGtX//938Pb7/9tvmY4tcrcpwkwp7PQEIi0R1XBhYfIrt1tSCS7sITfYiEwYEobuUVOn51BBwBR8ARcARyI9DgCK9ZxFIGOvwgCbHVo0cPQ6cuyW7m8Bx77LHmMgDBShKhgXBBwIlRPHTo0IBPKcfhEpkBokvEBvygEW3mAu+oFTLzd2bfk/Bb+kLi2SS28847W2SEup4jzAXmBJvYcMEhcgMifZOAnevgCDgCjoAj4AgkEYGGSXhTIwGZYQMWJK6uLXeaGBAaCGXr1q0t7BWv05EkWXllZezeo7sR3F133dV0PDJ19PL/+bf/Y0fskrBJo/VTi/wqQ3rmb9KSJlpkvPPOO+Hoo4829fRWoK513aTxely7d+8evlr+lYWD07ypa928fUfAEXAEHAFHIKkINCjCiyUMcrBq1So7ElbH+pKWNOnatWt48803TS0RsCToKPK66y67hv/8z/8MW265pal14IEH2mY2fkQtjvj4fvTxR2nSzmv4efPmJaErWXXQ4uLjjz+2+LqEINO8yVqglhMh3uiIC0xV2yrbdIkKUcxrWSVvzhFwBBwBR8ARSDwCyWN6ZYRMZIYDBIgpK79ZkbiamoZURD/Kny1N9/K9inxz4ABk8csvl5tVdN0PyQpBtdlmm4Xdd9/dukf/CUu2U+udDB/wJA157733wk3DbrLv/Jk0aZKdVKYE5dPvpFynT58eOnbsaOrE1ZF80Q+Fo7/j1lMTBpqv+HvPmjXLsuNuUar6a2rf7zsCjoAj4Ag4ApWGQIMivCIKWBhFZvIZMMpHPyqbLU33Cr0SsaGqqirla7xofRXr+WOh1ZW8HORKCwj6r998j8p+++1nWBO9AeEABXxik0jO0EkLDny892i/h+mc2SdLzPInOg9UJltalqIFJbEoov7ly5dbeY1HQZV5IUfAEXAEHAFHoB4jkBjCC9ko5wcyINeAr776ykgX4ypiUtMY4+dLRAKJvhNOjHvI6tWr0yRQ+Qq5igzicrFw4UKrIq6ehbRXSBn0ETmkfOZv1Qnu6g9pYJVUYiY9cbvA+r/tdttaN+JiL1JPISI8IKSpXqXZjSL+SB8WRZBezRHaKcenCFW9qCPgCDgCjoAjkAgEEkN4eYiX8yNytnLlyrBmzRqLecsIiDzkGg2RFYjto48+Gjgc4je/+U36VfLXX38dRowYEc4444zw+muvp8mNyuWqN046Bw3IeoeepagzTrulyCNdua76ZlWaHPMb4lcT7qXQId86RMSx7jZr1sxOM1M/qqtLedhkeN5554Vf/epXYc6cOVYEt5Q//OEP4Ze//GXadzlFS6urLq976Em0EQSSXo7/Q3kp5JkdAUfAEXAEHIEEIrBpUnQSEZUVtpR6QQIgWdtss01YtGiRERl8UOOIiCahoE4//XTb8Hb++eeHf/3Xf7Xi1MkmuNNOOy0cedSRlgYBKgWhQ0dZjcFFhCyO3nWZR/3nio/pyhUrzQpZlXLR+Oijj8w3+dNPP02sawOLGEKvxRU2ktFXTphjjpycOg0N0osQ1gxSeuKJJ1roNsZQi6+49WfLJ4zxRed0OyzJkGvmXSnqp03aoC4WXnUdki0bBp7mCDgCjoAj4AjERaBR6qFWOnNT3FYj+fTgJiLBzJkzbfOTwoTZPchEnhYxkVRdsc5CTJctWxZuv/32cNVVV4WBAwfaAz0uMY0SlZ///OdGAn73u99ZLNQFCxZYTFq6Fc0X6WZeX4UJ+j755JNmVY6rZ14NeeasCLz++uv2FoCwdRqLrBkjidFxZ37hp/xv//ZvAasv1t4TTjihJHNDTUov/NEvvvji0Lt378CBICy+WNyxWCJPoQLRpR42IxIXmnjLarPQOr2cI+AIOAKOgCNQVwjUuYVXRI64onxKLXpIf/HFF+Hxxx8Pc+fOtcD9+bYDAYCIY+nCpYG4s7vttpsRXw5cQKKkJ9/6s+WHZEA6OLZXhL1JkyZFEZls7ZQrjbEFN41BGh94WGpvm3yfy9V+3HrRU4usli1bWqivnj17xi1u+aLzA+tuVcqaDcklIshxxx2XV135ZGY+ME/wS+f0u27dullxzdV86qopr/6v1pTP7zsCjoAj4Ag4AklDoM4Jb7kB4SE9bdo0i2kLIbjiiisKIrzoCdnl1TGk6NZbbw29evUKxGulDdLjuknE7XPjTRsHiPqFF14YnnrqqdC2bdv0QQNx66jrfFErIzhFf5car0L7CjmENELI77rrLvO55WCSfAXSSx2Ea+PYZSyuuBsQq7gcBBT9aO/ggw+2OfnBBx8YwR4wYIDN1fQCI9+OeH5HwBFwBBwBR6CeIZAYwhslQsVizINePoePPfZYwD/4pJNPCu+8/U44/PDDzdeRNvK1WFEvJA3ygi/w5ZdfHm677TYjSKSXmmB8s+obI4hYlNlIxWEUhYRTKxbPhlL+2WefNZ/bSy65JODHm68wn5gbEF8srbg17LLLLlaN5mO+ddaUf8WKFUas2bC23Xbb2Xz5y1/+EoYMGWIW3+/XfZ8ypq/fEFpTXbnu5/v/JFc9nu4IOAKOgCPgCNQVAvUqSgMgQpwhF2z2GjlipOF69tlnh88+/cwsbRBGdrVDTvMR6oXIII888oiFgxo2bFgYOWpkeOKJJyw904JpiUX8gXQRmgwLXlXqFfnTTz8dnnvuuXSN+Cajl3/yxwBiqjmAD/b1119v86NTp052nLDCzqXBzvMLc7DLPl1sbCjKGJVDWMzRVreDuoVXX33VFkWDBg0ySzUbBhtv0ngjtxLmaL6fcujtdToCjoAj4Ag4ArWJQGIIb7GdhrzwIIeU4qd74403hu49utuOefxgX3vtNfMRxhL73T+/C5BFJA4RIQ91I1gB+Y2PJ0TjhedfsDbefffddB50iVOvVVjNH0g7u/B79OgRIGVY7ejL6NGjbVOVToqjinxJTEPOz/gwdswV3F1wPwBbLKRs0sI6SyzeYoT59ewzz6ZJteoq1dzQfGRRRJi9pls1NYvuxIkTQ5s2bcLll10eXn75ZVsk0TZ9heS7OAKOgCPgCDgCDRGBekF49RqZAXzjjTcCbgxDhw4N++yzj43p5EmTzRUAX0qC9bfYpoVZgOMMuMguERPGjBljG5DYKARxQQgFhfz6178Ob0x8w4gu5AJCUizpxXpHXfgMI+z4J7zVnnvuGYYPH26Waogb7Ugfy+h/ciIAEQUzBMs8BPFnP/uZkd2XXnrJLKSEEmNcmVciljkrjNxgHFiE4HfNHEQeeughc6nRfCjV3FCzuDSweRLBXYejm7FOb7X1VuGyyy6zuTFq1Kiw8uv1lmD6VOy8VNt+dQQcAUfAEXAEKgWBiie8PMBF+h588MHw4YcfhksvvdSsXTzYsX5NnTbVwjYxKBAOrKafffaZjVE+D3+suhAhoknInQCXAw6HgIC23H49MWXzEGHW8iFL0QmjcsSs5SQthBBZWOxoFxeH/v3722vrKVOmWJ/ol5PeKIo//c5cgZBCEv/617/a5kXmClZ/iC+bzZo3b24xeBmDpUuXWiVx54jGbd0P68Ipp5xic4WxYswQ7jM33nrrrYLnBvVIH94AEM941113JdlIe/v27QNzAmE+HH/88ebjO2rkKDv4QmTf54pB5H8cAUfAEXAEGggCFUt4eejz0OYBvnTZUnNhIArD4MGDjcDw6h+CAZHZe++9jcSQhrCRCGtpHKEO2uJ1N+SW8GCtW7cOWHm5ByElMD9tcx/iCVF65plnrHrKiqDEaU95IOrEWBXh5comqHfeeceyQHKI94qf5t///ndLc9Ir9Da+agyYK8TExeJ52GGHma8uOcEa1waibkjAm8UTks/4kbfVjq1Cu3btbK4wJ1hgiWDiP/7iiy+qmaKukF3GnPknHenD5MmTzfWFe5B8okWcddZZYdy4cYEYw8xbnytFQe+FHQFHwBFwBCoMgYokvJAHPbTfe++9MPrO0faqn+D7CBY1COk333wTpk+fbv62pGPdQ6pSG8CwyhJKjAe/yILdzPJHpBfyoI/K6LeuFN9rr73CDjvsYDVRNh8RMVq4cGHYfvvtjcxQN3LooYcaYSEPH1w0iAHM9W9/+1tY9fX6I3y5J/3itE1efcgf53ucepOQBywYAz4TJkyw46FZFHXu3DlteSUEGX6vLGaYEwinpmGNRfIZQ/KCn+YD1yieHTp0sMWRVVzgH+pDIO/oiWjMOdWN+QexRcjLPdw0WCCxiHrggQcsTaQ3NfqW1/84Ao6AI+AIOAL1FYGKI7wQCB7UCJYyohYQpxZLmoiFCAoEB+sWfrs89PWAhyDi9ygrL/dqEurEQqiP2tBvrtJLbVGnyElN9Wfeh4RFD+KgHqzMbVOxeHklTlsiZ5yEhcXyhhtvsLjA3EO/OP2iXfLqE/1d3fekkyTw0lxhATR27FhzY8Ffl4UE91gAYfXH/YTFBMI4IlWpRRGCRRZsCO8VV8gfnRf8lqAXkRMKFcozvujNoRb77ruvVRVtg3nDJkoWfFrkMRdYBBLBgcUYcaRZ9NmcTfFd6nVxBBwBR8ARcATqKwIVQ3iNwKRIB0QC38U77rjDNgNdffXV5nfJA52HvhGKVB5OnsIH9sADD7SxEyHQFaIwfvx4Iz7UWcoHPjvned2MqD37UcMfEWWsuxxoweY0xEjJhrL4hOKjqYMuIGLoTjgt/FE5ihiir3IQu2yi/rLBCUshkS3QGYLHq/z58+fb5iuu4EgcYPJwDx/YFEXOVm0i0kT0GVc2kGH9rkoR2NNPP92wVN9Rlk2OWER18ARYqzxzRGHgStVf5sa8+fNs3hUClnR74/U3jLhG9db8x8rLfIDIS+gX/ebTp08f23yJHzOLPsrxyTVXVIdfHQFHwBFwBByBSkWgIgivHvJYxiCDI0aMsFfSbAxCuB8lhaTxSne//fYz6y4PeRFPkQKsfGxS0qvfUjzsVTcki1fNHE4RWyIGNkKfHXPMMaYzfaNe6Y8/KBuTsPKapMpxD/3pEzvzIam8tsayCen7fm120kt5TnObMWNGqEoRQnw/sYbfeeed9op/5YqVdrAGr8GRc845x6IPyGoYJY6WIQF/wIG5ACYzZ840f12O+MW3VYSPqxZO5vJy8PpjhIWx5hIHfbBwAk/StPks326qPcaS+ngbobmRD4aUl94QdYgrIr35rvoOOOAAW5wo0kd0HvGdOcSpg0SmeP755ylqda/9fv0GO0vwP46AI+AIOAKOQD1BoPHvUpLkvvBwFoHBsvn444+bpQ4LFvcQERTlJYQYm9UgjezAR6KkQL/Z+IVbBIdRQPRElqxAEX/YXMZGOWK60m5m29mqhmhAZrC4YU09+uijLVu0LGSG37g2QFTwQ1WINCO2KbLHlXBsECtCYkGutm62tb2Sx0qp+rgaEUstIsCSV+T4sBKDFnKIDkQswMqJuwTtYFW+6KKLbANgqbDKhkWhaehE/xFCjoHl+eefb/6r3KPPfDRPXnnlFduA2GHPDoaFsKG8YZOqi0WEDnRgnmkMyBNXovWyYZJxwzIrfeLWI73/8Y9/2JjxBkBpqkP9Yz5jicdiX5VazCDSgyt44NoDMWbDHosd/h802axJyf4fWKP+xxFwBBwBR8ARSAACibbwRh/mhBwjIsGVV15pURaiBAYcISIivlg/IRU80KlDD3rhLVIAIcVF4P7777dbkCXqKUbUVj7EiL5gNYWgPPXUU6Fv376mQqYu1E0aVl5IzNSpUy2f2kR/+osccsghFroM31WsmFjH1W/LkPrDb1ktzzjjjHDttdeam8jmm28eBg4caOUhzAjkkY1PTZs23Qhru5mAP/Sb/uO3evvtt1vfLrjgAiOW4Ms99Z/vWD5xz5DLS7YugDWWUMjpP/7x4wl3meOSrWy2NI0T5fU9W75saeoD/wc4hKRnj/VW6U0a5f4vjC8vmzpxo6C9qN5goP9Dp512mi3QbrjhBvNz5h55o/mz6eRpjoAj4Ag4Ao5ApSCQ+2lZhz3QwxYCCwnEBxPrIxEJZIkVgclUE+suvqZsVqtORDhwPcDnEaIpKfZBr/K6qt5sV3xwRT7YSES8XSy42Yg65VUnlljID/7MUTIjKyTlIcVYZDllDjcJhPsixfyWewIWaay5WPoQFgzIqlWr7ArJEvnNpZtlrOU/6AIm9GvhgoUWng6rJW4MiIgi39lop3FnUSTfXcornXxIFNOTTjopRfg/sJBw0fT1OfP7q/HTNU5p9YFwdxwvPWDAgJSC6115srlSgwVlWNBhteUwimzCvNNcADP+f7H4Yw7QTz66n628pzkCjoAj4Ag4ApWCQOIIr8gUD1s2bt100032Sv2oo44yTLnPgzqXsFGHV/oixjz8s0n0Yc5mJjZs4TJBOpIPIcmsnzr0ybwX/W1EJmV5RTjeFsLJJxsBUzkRVhYA+CDzOjpT1DZYYZ1kMxuWXBYOEGTVoT6SD5eFSy65JIwcOdKqwwKKQJYRXBzatW1n35PyB/zoC/3Fp3Xcg+MsYocwpH/RufLDuvXEdsVXK2yjHiQPEQ6Z/aJukc3zzjsvPPzwwzZPlJ6rXGY90d8aG65xRO1jueakP6IssCBS33PVofp5g8HCiIULaZk60xfSmAO44jBXmFO4DiG6n6sdT3cEHAFHwBFwBCoBgexssI40jz7E8VEl4gAba+Svy4OZB3CmkM7DHOsuJBnrJ6KHfmZ+/aYuHvTI0KFDzYLHw55yfNCnXKINZdTPgQBY44477rh0czXpTkbIDAdRZFp5uUf5aP+wePbo0cNI77y589JEkf6rLeIYv5ba7AcGECSiWOALS4g0fFmbt2hu5CjbGNBmbYqIIFcO3sA94fLLL7fIBeqT+iW99HvS5EmBeLjRCAfKk3mV9Z28F198cbj77rvNpYB0EcXMMqX4Td3qI6f73XzzzYHwc/gAK726dhgj8uGewubN115fv3DhFLhMic4VFoqE+aMcURzwBee+/p9klvXfjoAj4Ag4Ao5AJSCQmE1reoizeQp/XQjXkCFD0hukIBgiLJnAivCy2xwiQ4xd1ZeZN/O3HubUj2WYTWwQ56qUOwCkAWJKnlxtZ9ZX02+IAx9cCTjha/iI4WGn1jvZwRmUVV+qqwddRGawxHJMMtbebGXJSzofTg8jHxEcaB/yZHWl3CroK64d2B1//vOf24l1LBxYdBCaiwgNOjGMvNaPFHmSDyn6kEZ9fMol9IN2GC+IIBZpxrtfv35p1xDuZYqwwUUGIn/iiSfahkalZ+aP/lZ/2cSHWweWVr5z4h59ZY6Qp1QCltTH55NUrF2OrcbPFhcM9T1OWxoLxv25Z58z94Ytt8ju10595NcY4rsM+eX/lG1ma9Ik6/yKo4fncQQcAUfAEXAE6hqB0j2li+gJpAOSQjgvXBggMGyiIq0m4soDGmKAfyPhs7QJKR8CQl4RCYjdkiVLjNQocD9EAD3Qs1ChLP66tEW/0PXGG28M++27n1nuqDcO+VL76p+svOhKWjYd0Z979AHyw4lbWETx1yT/po03TR9iQYQGrOQclYwcf/zxYeqUqaFq9yr7TV2I9SPljqH26BMkmvtKs4wl/CMyRlu8picWM/rh94xoDKtrEpcXbWgUsawuv+4JPzbuYUkm1jGv/WmTftNnvhfTd8pH+0DIPMgufth6y4EecYWxkH5sYHv55ZetaHU6Uj/3+bAAZCEBTi6OgCPgCDgCjkAlI1DnFl4erDyYORmKV/sQXQiJHrqQm+pE5dnMg+8mhI6HfD7EgPpFDriygYs6Hn30UbPCYgkVmSMvbf5E4IFZkslHnXywhmI9fuyxx6y/p556arqvysc1jlAfRATfW6ziHA6BVa66vou00RfIDIsENuspEgHtEoWBKBFY9cjfsmXL0Kt3r1DVtmqjulkUYFnGnxSZPXu2fSCEjJnGxW6W4E+0X1jh8bdmkxVjwz0k15hLF6y7WCz7nbreGqxxiaue8MPyif8vixZI79ZbbW0LBNWXdX6kGuF+dfdUHqsuPt3//O6fFlaNMWCsa/q/kK0f1IkwLrgJtW3b1txnqiP7KoOuzAfcG4RhtjY8zRFwBBwBR8ARSDoCjVIPshw0rXZVZyc5D2N8RSEwevhXp4VIAEH88ePEx1IP6+rKVXcvSqwI5wSRJurD4YcfbiSQ1/75CnUu/mJxmDxpsoUI69Onj/nToqvgL0RvkRBOSxsxfEQYct4Qc0uI9iGbrtH7WHpZaLBxT1EYjLiv50k/Ka42sRZCPP/v//2/9tod8kfoMggwUQSibfykkjwSaI8PZJM4wHfddZe5VvTv399qEXGrDj/NE8g9+uHLrLQ8VElnjfaNg1CYeyw8iFfcpk0bcwVIZ475BbcIFhBsEoTwcqgKLgxItL2Y1W2UTWNGGDvGCatt3Drj5tuoQf/hCDgCjoAj4AgkDIFEEF49kMEmnwes8kLYIAeyDFdHfuLgjz6I6sESysYtrNBsLuP1MsSGOL8QHSx+sr5BythExpUNPxDKDz74wPIQKg3LKhYzRPrbjwL/CDuiFODTyiv+OGSOcnwgkvgBjx492gi9DrzIVYfaW758ebjvvvtsVz++yGBAvF9cBtj0VIq+ResgigbtsfBQdIXo/VzwKQ/6suGMKASlsECDA6I5wqJIhBLfXuYjbiHMD8YbfJSXBQqYMUew+GMdx40EMk7fOFBC1mrhnat/cdJVB+0xzoRZQ7dcYxynTs/jCDgCjoAj4AhUEgKJILwAlkkgagJRD2uIEJuqBg8ebCRBD/eayse5n6kTrgPEo+WoWUgwRBH3AF77QmzID9mVPy3WYMgP5JiQTxKRMP0u5qr+QqDuHH1nOOP0M8wCKnxqqlu6UA+WSnTHegqZpw6Il4gadak9+s/rfMJ1SfjdJ2W9jhP9QGVyXaP6s9jAFQErNP7d6IBE9aqpHtxTKMuiQ33OVSafdOGhMswJ5geWXxYg4AnhZY4wV2ibOcKHdDYC4oaDawZvNySZ9Sq90Kv6jD/2/AXzzYJMXaVup1D9vJwj4Ag4Ao6AI1BOBDYtZ+X51B2HvKg+HtKyqGLZJJoAxKzUD2/pRL0QBkhtu3bt7IMuWMyw1kGEFc2Bo4xl9ZWVTnqLQGam634hV3REPyyIbIBjoxMhyOLiQT6RIXyKOZKZAzDOOussI+vcQ4SFdCS8FcfWIuTBxQHLN7rgCgLpLXQ8wBJyiEDCIdecsAdBjBJhy1DNH+VlMyQfIjMgpcafOoUTixx8wPkgmh/MEY2/5giLiqhonjG3M/GO5ivm+z5d9wlTpk6xsWMxVugYFaODl3UEHAFHwBFwBGobgcQQ3nw6roc01lYsaPJ1zKeOfPJCPvQaXMQG0gRx4ZNLyIuuCPlF0nPlLzRdeLBpj5BZbChjoxPtxyFOIr20j3/rrrvtapumiNigmMbURT7VN3/efOs7bbEZDP9d9MAai1tFoUI7kF3cQdi4xet9fE4REdg4daML+iKvvvqq9UNjqD7EqSduHrWlMacN0iDpfHJJZv5yzRF0AT+wxcqN6wl+wtI7l36e7gg4Ao6AI+AI1AcEEuPSEBdMkTvyE68XSxqENx8yFLet6vKhBx9EV76LTHHVd9LLLSKkWGjxCxXpjOJVkw7kpR5IFwsJ8OVEt5NPPtmKqg3yYd2GPMlqSV9Jo6ysszW1F70f1fOT1KYtYgVjqdYRx/mOrx3ZnAqb9tnnn4Vnnn7GIjqgV7SdaPvl+E5bfBBd+a55UdtzxE6a22T9GwHCnbGQwJe3NjGh/y6OgCPgCDgCjkBtI7DeBFbbrZagPfwk8ZeUdbe2LVWQFdrkA8nTR2kiNXG6CpHUB/IR53uuerHeycc4V55c6ehMPyCL+JziF4118pZbbrGDQOgbuiG4bfAbCzdl+E7eQsgudQovXCM4wpfYs9qEyH3aiCtg2DhFdpGJb0w0iyZ6RduJW1cx+Uo5R4rRQ2UbpcguCwf06tmzp8US1j2/OgKOgCPgCDgC9RmBiiO8Ika8pubQBUT+s5U6UCLJXKMkqbrvUYsh/aYsZAYiylGy+DYXKpBF1c9xtkcddZQdbTtnzhxrB71oizzZPnHbpSz1oDs+rrgwLF68OFxzzTXpU92EQdw6o/kI84W1m8gYLusRAGuka9eu5vqCBRyMtZBZn8v/OgKOgCPgCDgC9QuBiiK8eigTS5Rd7hyOAGnKx/qXtOGD8LHJC3cA+sR33An4zZVNTxyzTDrfSSMfJCVTRGYOOuggi7fKZi/ygVG+onJgDs4XXHCBxSRmoYGAuSym5I1+4rQlokw9bCj729/+ZpEKBg4cmK5b/YlTXzQPuiCQfrCgHhHraL6G+B1swAIhbvD4F8fbd//jCDgCjoAj4AjUZwQqivCKyIwfP97isTIwIl2VPEgc4EB82WeffdY2a+FCQAguTk8jxjCflV+vDP/93/8d/vSnP1ke+ptJZEVmIHi8sn7llVcMlsx8cbGiPurCgs4mOI7UxV2CeLaQJsiqyFPcOsmnMaPud955J9yROiIYP+FDDz3U+sR97hUi6ivWXTbUYclECq2vEB2SXkYLRNyBWECx+RN8ChnLpPfV9XMEHAFHwBFwBECgMFZRB9jxMIaA8VodITwY5EYP7zpQqegmpT8bzIjDyut8rvjOcsLbkqVLwg477GCbt3ZqvZNZeSHChJMSacxUQnjgy0vYMKynkBnyFyr4v4I/9Zx55pl2FPGwYcMCJ9zRHnWLaNbURpTMcvLZ5MlTwmWXXRZ23333NOEqBTnFF/iQQw4xdTR3atKtId3XeGHl5chhBNyV3pCw8L46Ao6AI+AI1H8EKoLw8hAWCeLhjDUUqfSHMwQeAgih/M1vfhMeeeQRs7ixc56DEo7oc4QdlgDpxT0B1wJFLRAe2aaoyC0WU6zhpRARW+oidBk7/MeOHRtmzpxpY6O+5GqLsRLZxSVj5MiRRnDPP3+oHYcMKaUN6ilUqJ/yWCwJlWab3lLnJGsRUGi99bGccCbsG2NDZIyaxrA+4uB9cgQcAUfAEWgYCFQE4RWRee+99yx6ANZApDrSV2nDx6EN+OlOmzbNVNfxuSL19J0T2xARWvuR5Y9wgfARyYLX+6RBKosR6hBxbdu2rVlmsaRijUZytaHx4z7+19dff33o2aOnHXFLOe6XkpSay0ufDYuidfn7L6NTQxDNLdxfOLAEYRyU3hAw8D46Ao6AI+AINAwEEk94efiKDPFQ1mvq+vJQxqpGXzgp7ZJLLgn333+/zTwdaMEBAcjcuXPTp3fJOmc3cvwRPuA1YcIEyyXCmqNIrGTaph5IKqHL8OtFsNhC2Bkr7ql9uUKQhxjBnJzGBrjOXTpbHvJRX7GidrBU4nPcYc8OVmUp6i5Wt6SX32OPPUxFfMYRjZ398D+OgCPgCDgCjkA9QKB4plFmECBPyKxZs+y4Wlk5y9xsrVUffY18xBFHhBtuuCF8+umnRnDvvPPOMGnSJLOKbrXVVubPm69iuEEQ3QHLarStfOvJzC/SS3rfvn0DFmlILyewcY+2dBAFY/jQQw/ZaWz46+KyAUElD59iBYKmRREWZ1w5ECdu1SMbnQ/du3cPb731lhVg/By76rHzu46AI+AIOAKVhUCiCa+IDFeIH76jCASqFEQpKUMlK+TOO+8cfvvb39or/+XLl4ezzz7bDnb44x//aO4MOqI2W98zCYrIDFeOB8a6ipTylbWIEePBJjk2tI17cFz69ThWajbN3XzzzRblYciQIdYfyK4IainGQIui2bNnmz90+/btrdpsOJWivfpUhzACM8ZFVt761EfviyPgCDgCjoAjkGjCq+F59913w5Zbbpn2YdVDWvcr/Up/IBvIqaeeGv7nf/4n7LjjjkbesFbeeuutFsWA+yJ3ENwoyc2GiYi0rLy4RSDRcpZQxB/apR30x3J7xRVXmDX5H//4R/jwww/NheGYY46xmK80g/6lJLv0RfVB6nv0/HFRVES3GkzR6NzD/UW+vNnmU4MBxTvqCDgCjoAjUO8QSCzhhcjw0OWKH6tOVYNY1ceHsfqEPyWxduW6wcEJEN4WLVrY5FM+rvrOKWW4LSBRMrtmzRo7wYx06pGVV0SY9FIJpBMXBqzQgwYNClOnTg0dOnQIRx55ZGBDHkSXT6nbVn8h19Tdvp1bd/MdU3ADx6qqKiuKpRzR4sp++B9HwBFwBBwBR6CCEUgs4RWmWHfZHLXrrrvaQ7nUhEnt1PVV/WrWrFm4+uqrrc/otNtuu4WhQ4emrZjS84MPPggvvPCC/YTY/uEPf7AQYSLBkFsOiFC9kE4OGSiHlReyBDnChYGQY6NGjbKDL2jrscceM/9r9OBTShJFu+of/qdyeamviyKNfamvzBmNCyH/8INGhG2p2/P6HAFHwBFwBByB2kYgsYRXxI2jbAmOj/BQVnptA1Wb7RGxQUJ/5btLmiyaRESAmGBVbd68uR0Gce+996qYnV7WqWMnc4vAAoz07t07THhlfcSGUuEoogQ5WrhwYRgxYkTYZ599bOMYVurzzjvPTnx7/vnnTQfyyX3DEkrwRxZJDiNBnKjlDyoWesaShSULTBaaSKnHKn/NvIQj4Ag4Ao6AI1A8AokkvCJR06dPD9ttt136ZDH5ahbf7WTXIFIrLaO/9R13AV5BCytcISCxy5YtsyN1+b3uh/URLjjYgnKkfbf2u/TGJJVVO/leIUOQS9rFheGee+4J/fv3t01ytMf9bbbZxsKtsQmPgyqwRjOOhA4rVkTa8TuFzCO0qfRi62+o5bHy6lhqxkpzrqHi4f12BBwBR8ARqHwEEkd4ebjKQod1t0+fPpWPcp49yCRsmb+pTlZbxetlU98JJ5wQHnzwQYuHy2ECUVIp0kLoMx0lm63euKpCLLUAIeTYjBkzwlVXXWUWQpFO7uv7gAEDzJeXI4k5CAMSDuGWXnHbVT6RdVw7qAtrMmnSSfn8Gh8B/t8xHjvttFPYfvvtwzvvvGOFhXX8mjynI+AIOAKOgCOQLAQSR3ghSMjkyZPNf5UHLw9ckeBkwVd32uCPO2fOHCO1HKP79ttvh7322suOJyZmL8RPr/rRUmSGU+qaNGkSIIoQXuEdtycQIhFL2h0+fLhtqBs8eLAdnkF9UdLJd/JT7sADD7RQaw888ICFmUMndCiEUGk+4NbR54g+cdX3fDER6NWrl21yZNw0hjGLejZHwBFwBBwBRyBxCCSK8EJ8sNZhmSTurk5VK8YSmTjEi1BI5IPrV199ZZEXcGGAeLZs2dJwG3bTsMDGN0gtljp8fUV21XShr6wZH8aC+jjR7KabbjJf3aOOOsqq/n7dxmRX7YmcUn6XXXYJF198ccBd5fHHH7cs3M+H9Iqkz5g5w1wmdmq9k5VXO2rXr/kjoP9rrVu3ttB4OuqaOefiCDgCjoAj4AhUKgKNUg+yxDzJZB184403jNAdd9xxqV1aKWiLP4yrUsenbHpjZd17771D586dzcobtcpma1Rjwz1cInjdjVUXH118hVNB0mL5zkJsRUyJ1fv+++8HDqSApItQi3Rl04Ppqvu33Xab+QwTszianq2cp8VHQGP0+eefhyeffNLGRy4oGrv4tXlOR8ARcAQcAUeg7hFIjIWXhyyki3iyWP84HQxJHa9Q9yglUAMIHphx1XfDa8P6JZoWVZ90hLjGxDfmN7grPZpX30V28Ru+//77w6JFi+yACSO7ENhG610TlL+6K4RJbR199NEWp5cjiXG/4B5kln7lEt1jkxwWbCe7uZAqPF0LCqy8bHRcsmRJ4ZV5SUfAEXAEHAFHIAEIJIbwCospU6bYBiSiM7jkRgBSIoKo7+QWWYmmZauF8FOEM5s5c6bdFgmN5iVNCxEdEYxLwhlnnJH26yzE4oduqhu/43PPPTc89dRTYcKE9SHTqFNuC1F9pAv3cHlhY55L6RHQ+FDzAQccEPAJRwoZayvofxwBR8ARcAQcgTpGIBGEF/LDw5RDCyBgsu5CcETg6hinetM8eMpKykENbA7kN/hHSS/fyUs6fpxjxowJRFpgM5PIajEESHXjr43/8ZVXXhl4hT569Oi0i0U20stARK279WZgEtYR/b/beuutzd0kYeq5Oo6AI+AIOAKOQF4IJILwSmOIFZZHrLsiYbrn19IhAJnBVQSssd7pkAG1EF1oPP3000aKL7roonToL/IVQ3bVDld8QyG21Ddw4MDQsWPHcPPNN4fFixenXS0g2HzIg1sFESm6d+9u1UR1jdbr30uDALi7OAKOgCPgCDgClY5AnRNeHqgQMKy7WBvxLUWS/qBFP8hWdZ+k9gG8132/3k82Gn7K0jdYexkPNoVBRi+88ML0pjJIJ/lKKdGwV1idTznllHD77beHmbNmWlvSizbx78ZvtxJ8dyt5jmh8Sz3WqtevjoAj4Ag4Ao5AbSJQ54RXneU19X777WcxXXnNXVPUAJWrzSvkFgIoqyLkr7qPiBplkkZ+RTLZ+NWiRQsjkmBJf+bOnRtuueUWO73sxBNPNIiJxMC9cgl1iyBWpU6Qu+aaa8KrE161KAG0ib6c0oaPtxZFGody6ZRvvehPaDaNN+Nf3fzgXpLnSL799/yOgCPgCDgCjkBSESgfg4nZY1mQZs2aZQcXUAwiAJlJiqALH/SCeHFFOPxh1apVgWNzP1/0eWBj18oVKy0NcoaoDP0UEbIbCfrDsbwTJ040wslmsEceeSScc845FrKMfkPkiMRQbhFBBCdOjrvkkkuMEGLtXfPtmvDee++Flju0NJ9f9ErSogidkcabNDa96AsLt29WfWOxkL/88kvzUSbiAbGRmTc6LS/JcwScXRwBR8ARcAQcgUpHIDFxeD/++OPw/PPP22lhvGZHIBF1SWogekb2IgSXAxfwL+XgB8jtJo3XW+kgLZZ/XcrVIWUN5Tthu/hgRW3btm2aKNd1v6KT9rvvvgscT0xsXY5yJjYvRxRzcAWEDR/buhAtMGgb6z+EnJB1l156qS2M6lK3KB7oCbnlgyxdutQO5YDg8mHhwxwWmdecYg6A+7bbbmsW9t122y3svPPO6aqj/U8n1uIX9ETUL37rey2q4U05Ao6AI+AIOAIlQSARhFcPd8gXByJAIgedMcgesLpXkt7mUUm0XU4z41U6r/p5/d+uXbtAjFJ2sENaZKGDFEBkKAs5w/I7f/788NFHH1nLXbp0CV27djUySUJdkgja5oPukLS7777b9PzLX/5iaSLCeUBW8qyGZ8pFYNPGm4YXX3wxDBs2LPzmN7+xI4ppLDpGJW88RoXR9pkbbLpkMdSqVavQvl37sGOrHc1Sjf7MaYgvZWyOpHyoV69ZHZhblJ03b55tINx3331Dp06drHX6j9Q20Yz268MPPzSLOsQcfWpbFwPA/zgCjoAj4Ag4AkUikAjCSx+iD9nXX3/dLHrEZyVkVW0++KNtYZ179tlnw4L5C8Khhx1qVlpIbr5CnRAbTifDWtmnTx+Lb2r9zuOUsnzbzZU/ivUHH3xgLgxnnnmmWSa32GILi4CQBCu0CBbjcO+995rl+bXXXjNXACI6QNbpCySsNolYFD8I7t///vfQtGlTC9mGNX/zzTfPBX3OdCzWLDywZPO246STTgrt27e3/NH2clZQohvRtojQgU6DBg0yS7/Go0RNeTWOgCPgCDgCjkCtIZAYwkuPow/bOXPmhIcfftge/LJ4Re+XA6Fo/fgUjx8/3iyyPXv0DJtu9uOrfR78fKIiwhVNz0bEIL6QaPx/2RCGNdDqS4UJqw0/2SiR5Whf/GLPO+88i8Lw5bIvwz333hMIQYZLQxSPaF9r67vcFl5++WWL4mFHTacax/2C0GQcSYzLiGGeGo5Gm5Q2ekS2fkbxQw8iixxzzDFhn3322Si7jWmBcwSLL0f68hbh+OOPNxcOxiLbfNqo0SJ+SF8WEbydYIHB24xTTz3Vaq3ruVBE17yoI+AIOAKOgCMQEkV4GY/og5fNPXfccYe94uUYWmTt9ym/0tQr4lJL9IH++OOP2wYjrJ6y6Bqp2tCoyG1cHbKVxc2BvkF6CcWFWCSEMm0OQwfqZ1MVZHvcuHFGFgkBZm2nCBVk57nnnktbK6OYWKZa/KO2CY/GpjXILWNBP8Af/OgDllD8jpEoGS2HqqqfeckBGcQxZvzkZ65xznd+oKvK8l3lIfp8WJDg46s8uk/eUghYUyefBQsWhHvuuSfw/23//fe36oV5KdryOhwBR8ARcAQcgbpAIHGEVyCIXPD7iSeeMN9IXq3y6jh6T/mLuUbJ1dixYwPH52JZQ0rdFuSBD+RyxYoVRuY4XvfYY4+19qSL/SjRH+pEaPPTTz81P2lCe2Ue3gDhYaMVRJIoDWBdDn3idEu4Y2VHr8MPPzxNdnUPH+n7778/7LHHHuHII4+0ar9fm9rouGnjOE3klUdtLv5icRh952hrj2N3Ed3Lq8JqMoO5CCjuDcRDPuusswL+vdF71VQR+1ZU97feeiu88cbE1Il6p9kGulK3FVspz+gIOAKOgCPgCJQYgcQSXvpJTFMskgivjokicNppp5llrVQPYz3weY175513hp49e5obA23qHt9LLXpdT71sGOPEs379+lkzpbSogRNEF3lr0lvh5ZdeNvJERIAohrSJQLSiVt5S6mINxPgjnbGk8mpdlvaoLtGxwfWFxQOkEFeM6L0YzdWYRfWxWMCyO3jwYFsUoScifGusKI8M9JX6sR7TN94GHHrooWZ1jf6/yKPKn2RVv7iBHzKLHTDEBzl67ycFPcERcAQcAUfAEagwBBr/LiVJ1RmfVh76PPyxuu6+++5GDomMwOtkyBn3uRYiIhSU5TXuIYccYrFnaY9/ItuF1F1TGUgSpIIr/p8zZsywOL5VqUMXJIX2S+Xpn8gYsXU//+xzcw3g6Ga1rTa4ilBuv/32AcsqlmcIpNJVb21c0YcNajvssEPYc889f6JDFD/0RBhD5gi+p+iMqH/2o4A/In74XmNpxb+ZjWmGXyryQrn8rtGbPrIwIibxgQceGB566CHbxMn4FDPvgUFzn/B6f/vb3ww3/HUJQxddjBUAmRdxBBwBR8ARcAQSh0CiCS9oibBAYJo1a2YP/gkTJgSiC0B0IAX28A8p0psH76U+kUE2CLF5DFIhIlEuIhOdAVHShh/qCy+8YERN8VjV92iZON9F9qh/5cqV5jZB/yA0LBZE4jLr5zfWw6ZbNrXd+ezQb9OmjTWZmTeOHoXkQXfawqr5yiuvmKtHLtKdHvtUfjZ4QXYJa4eFElJKPRrPQnQRKcT6f/3114fTTz89VKUWJGn88plwhSiQKkMfRUA7dOhgC76OHTvaG4FC+qYyYDN79ux09AvmPqI+2w//4wg4Ao6AI+AI1BMEyn98VgmA4uEscgOZOfvss23DFZYpLFR2L7Uhi01ZcUWkkDBhkCteFyO0xae2hFfWECgEH2XizXIaFzpIx3x0iRIaIl3cdNNN5m9KJAGkJkKT6r3lw9pNXFk2jRWqi1VU4B9cWETszJqaIn7ZhLEHJ/rFW4ArrrjC3F94RY9wn/L5Ykl+yiIsiA477DDbPEk7SrebtfAHqyu+ycTCxaUHFw7pkU+/ojiymGBz5gUXXBAg0tQT7XMtdMubcAQcAUfAEXAEag2B7Cyi1prPryGRG0odddRR4Ygjjgh//etfzVLFPayyWChrEj3YsX6++eab4eSTTrYiEILaJLvSE9KLFQ8/XggN8U+RfHXJJDTPPPOMnUwGoYEgqd9qN9sVHKmHiAiEgwMfhLLlFtqgzyxA3n///fSmuppw4D5600eswbgd8DaAucHGNvDV6Xf59oE3CUS1YNOcSWo9UJM++bYRJz8b8RgXDj2pSlmZ8WePK+BKWXDgOOO77rorsPnu2muvtYUjuNGnuuhX3D54PkfAEXAEHAFHoBgEKorw0lEeynqAQ+Q4apYjiXEHQPC75eFenejBThza/fbbL2y19VZpQlBduXLew4oH8Wjbtq1dsTwjpMURCDOEhhPS7rvvvrBw4cJw2WWXhWz+ujXVB3lEsPJyWAaEj7Ryk17V/8Ybb5hfszZPSZ84esvKT9QGrLJsRMTSzbzAeh0HT/RgjkAOWXz06dPHmmZe1YarS65+at7yNoLDWfAr1v+HXGXoCx/mxmeffxaGDx9uPtEDThtgRcAjLr652vB0R8ARcAQcAUcg6QhUHOEFUB7yPMAhebzmxaLHDnOiHUD4uJeL9IpULVq0yEJ0KQZukh76uB/g2oCuNRFN8kBaIMyc+nXjjTeGqpQFkMgGlJVlL5+JCL7UyWYpFgSyJsYhi/m0E82rvuKiglVVIdPyHRcIKXXR786dO1tEBcLacUgE3hrUV1M/KI/g0sGmOVwlKMO8qkvReLIQ6NWrV3qRJ30zdUNnxpJyHNQx5q4x9jaDSCSU4X6++Ga24b8dAUfAEXAEHIFKQKAiCa+ANd/GDa9qBwwYYJYrNhdBZiEnPNAzyYB+z5w503xbqQtyBDGoaxEZY8NVVYq0vvvuu6YS/cgm9CVKaCD8A08f+ONBFqlyhZI0EaFu3boZAcX9Q5hm06XYNPUR31LIrjbXFTIulJGlHwv3lVdeadZQXuWzIBLOuXRW3zltT77PufLWdroih4ARizzcP7L1J0pm8WfmjcFVV10V2uzeJr0YVD9ruw/eniPgCDgCjoAjUNsIVDThBawoCYOcEUcU4od1jgc65EdkCoJIGq/o582bZ5t1qCOJD34OGeAggFz60SeRQV674wZw8cUXh93b7J4m+sX2C7y22GILi14hK68WDKZYif7QF8aRV/SMiyIGFErWTa3U+kVzg2v//v3tgAr8er/44gsbc/oiFwh1RXOFk9zAr2XLlsmyhKb6hY4sCNq1bRemTZ1mqmtcrE+p++i+6utVYeTIkTaGQ4cONYu9sNbcUb/96gg4Ao6AI+AI1GcEKp7wMjg83PWgJz4vfr2QRW3+4j4P+iiZwfInQhTn4U/9WIL1UV0bpcfYMFfTZEJXhNBkEEA2XaEf7SLqpxGaVassNizp9JlNb+jFvTh9olwcwa1h/vz5ZlEEM+kSp2w+eThGlxPgNC75lM2VFyzADKFuiO+I4SPC9OnTDSPb6LgB22gd+C4T/zcfAfvq5ofmTD51ZsursW3Xvl2YOWumZdG4cI8+s3AYdtMws5Yfd9xxlkdzI1udnuYIOAKOgCPgCNRnBOoF4fykbL4AAEAASURBVGWA9KDnoc6RuFg7EQ4L4MQuSIAIB+QtXzJD/ZAKfagP2Sh9w6lwdqPIP/jPVqXcGvDLlaC/+vnJJ5+EESNGGInr27evZYHYSS+VKeZKW9SJlbdr167piA20IRJZTP2UhSBSH/0kHBuHcCC0XSpRXeC32267hZ9d+7MwadIkO7KaNkQW+S780IcjixGVtx+5/qQ4NWWj8wOMKBtNy1W8kHRcXxAWRQjtIMSpfvTRR+3/AHjS71LPDWvI/zgCjoAj4Ag4AhWCQGIILw/kUnyiFjuIIJEG2Jn+4Ycf2mtgxgWCwEYkJBaZSeXD95PIB2yqwvLKa3FkVcrKynfq5OjZUlg/wQGBlC9YsMC+Q1pExiZOnGixWNmYBhEVoYnbF6sw5h/psu9++1r/40QGiFn1RiSMU9UOOugg23xHm+XoC/gxPiyIzj//fFPz9ttvtwURZJExRpYsXWJzhRPNkFi6pPg5PrWyyvOdciy2IPJ8uFcKkT4sRJjHLOCQNWvWhHHjxpl195JLLgnoz8ZO5S/0/1cpdPY6HAFHwBFwBByBukQgMYSXh3IpPuzEl6ULYLFwnXfeeYGYtLzGhoAQboqd7ojIgP2o5g/52ESGf+nPfvaz8OXyLy036ffff7+RNQivSGk1VcW+1bx58/DV8q8sPz6bEFuOz8W6ywYkTk+D0NBm3H7EbnxDRiOJKVeNLTbfInTp0qVav+J86xaxJVzWl8u+tKgK+daRb37mhhYlJ554oo3bzTffbCQRjJHFixYHsKfvIvzVtaM8kFwWIeeee66RX8rQ1v/+7/+G//iP/wic2IYov/0o8g/knY1rzOnhI4YbAeZgFuIRI2zsZG4U8ylSRS/uCDgCjoAj4AjUOQKb1rkGGxTAOhW1RpVCL4gFJLFFixZ2OhvhqSC+vNYWIYjTDvVAHI4++mhzI+DAiz//+c9WFCsbljsiAGChJC+fUhBQ2lz1zSprBzJFGwcccEDo3bu3pYEXeWivXGJEiVVESmh71KhRduww1kOwLaafWhxMfGNi2G///dKRGeISzUL7DOmV7iyIWDgQu5jvffr0MWLaZPP1hDFOG2DAGHAoxB//+EebB8OGDbOiEGf8sfv162fHHdOu+h2n7lx5NMeo+5FHHrHFXO9evS2MHJsyIdrRhV+ueqpLVxvM8VLoXF1bfs8RcAQcAUfAESgnAo1SD7XysaUYmuuhyittNgpxwhcPa5EIDgtAUjTSrvptP6r5Q36RMaxf1MuD+1/+5V/C73//e7PEqe1qqknfiua95ZZbAuGzsLY+9NBDRqg5+U1Qqt104Ty/qK2lS5faoRpYVsHn2GOPDW3atMmzttJmJ5wbbhbaCFVs7YSQ4+AQrJJ1LVjqcRHAEgvBg/xqLGrSLZrvuuuus1But956a5g8ZXJYs3qNxc0tBQmVHmoPdwbmBG3ic6wIFMUuGiDL4MD/GyzhXNWmdPCrI+AIOAKOgCNQKQjUuYVX5JAd9ATEL4dgVcMSyoEGJ510khHUfNtBTxEW/D+nTJlibgVY7kpJdqN6bbPNNhZT+OGHHw5DU2Gl8B8m/BhrAMOtlpYqtCUMIUHPPfecHd0rN4uozrG+p/T/Yd0P5ktLuDPIldxNNmmc8rKppX5JV/qGi0v79u3twA8O/fjTn/6k27Gu0fmBCw1xch977DEry5xDymElZQyuueYaI+oQU9rQgtEaLcEf6W1zrgT1eRWOgCPgCDgCjkBtI1DnhFcd5mFajgeqiBqb1t58881wwQUX2OYy2s23PYgZ1mIiKBAGDFID2UAgGZDqUgr6QXI5Qpkd+Vh8IR/56l0qnXA7wWeUo22JAnDqqaemraH5tIGlEJcSDnb4/PPPwymnnGJuIUas1uZTU+nyrl692kg8p7Ph3sC8yVeYH8wD3D2w7mKRx6+b8aI+Ecd8660uP+4LJ5xwgoWk44AJwq6hh4sj4Ag4Ao6AI+AI/IhAaRnaj/UW9M3cEDa4MFBBTb+z5bG0FKGKkgwiKOD/+utf/9oOnYBoFSKQGYgam4TmzJkTOMSADWwPP/Rw2GLLLdIW4ELqzlYGsltVVWX1YoHkFXtSZOWKlbaJjVfdhQrjgitDx44dC62ipOXwiYY0clQ1J8sVIpB5hAgbRAjh6GukHGSXelmA8cHFhANXnn32WSPaIthRN4SNvuPyk/qn/2O6UqckW5ru+dURcAQcAUfAEagkBFLvj5Mjmf65Nf1G88w8lhaxqEEGxowZE4YMGWJWMKyUpOVrwYPsynKGryeHMVx22WVGaEbdPspA5H6+9VrBHH8IeYY1+YwzzjCL6vvvv285CZ9FO3XxgRQibDKDHCKkgU8cfRT6C9/TuXPnGtmlXNzycdrIJw/tIpBBojXgL40FG2t6sQLJFQHWtdg6M8tjmVbdnDL48ccfmy88bdO36NuAjb5vWFjq/4+u0fqzpUXv+3dHwBFwBBwBR6BSEEgU4S0VaBAeWdSIKoClTYcIcBoZhFfELU6bEArILFesupzm1qlTJyv6hz/8IVx55ZXm+0mC2uW7iAjfCxHKQxAhvVgdcSOAiOG3ifWNtmr7I7cNXv3jz8sGNqXF0UWLBk5Vw/dZQnqc8qXMAwGUPg8++KCFRcPazPyBSBYjlOcACNWTORcyf+fblsgr0TvYjImQhn/5Cy+8YMSXvonQ51u/53cEHAFHwBFwBOoTAvWO8EIkRAaIoFCVcgkgnBYkBuFIYfxQsfQiNREP1Ycv5vXXXx8ef/xxc4uANEM6ic0LybjjjjvCk08+uVF96FFT/aZEjj+88m/VupXd5dU4r/9Hjhxp7TdOnepWV2RGmLDRkE1nCOSqpr6iL4SVOMLk1SKEtLoQ6YsbALoRdg7ZcccdLV4zaZpLcfSjPsg/lms2GmJxxc1AkROidRQ7N1QX7jWEJkNYHOFyQxxg2iVcnpNeIeVXR8ARcAQcgYaMQJ2HJSsl+BAOPhAoLIiEvDr99NOtieg9YvHuvvvuZqWNWoOr04XNQeTFZxWyC5GgHb7jX4vFGGseeYikoNBdWIPjtpHZPqSJV+z4g9IOZGbatGlGMrEqI4XWndlWPr/BUoQN3+jDDz/c8Pw+dUAFRDybqAz3cDHp1atXaNu2rY1XPqQyW92FpEFmGUOiQxDyjU2IiPTkSGo2gEF+lRa3HQ43oU+8TcAXmHnCd9I5JQ//bzaaMTfyrTuqI/ONk+JYCBHRQwSd9mjjqaeesn4pmgbpLo6AI+AIOAKOQENEoF49AUX+iOcLkRHZJV0EjUHm4AlClMUVSAlWYW3QgnhClqhTJ7Zh2WvWrJmRXkgqLg8KS0U71JGPQGYgzRAuhPqpA99h6ucVPAKJybduK1jEH/otTLt162bRL6gOsptLF/IjH87+0PoC2a0rWfv9Whs//F0JQaZjhtFd+hMVQ3NEaXH0xerPPIHgIswJfWcT4t57720LGN4+IPnUbQUifz777DMbf8guwlzgA9aEWMOVZ/To0el7xbRllfgfR8ARcAQcAUegQhGoN4SXhzwkFNcDLLjEQkWwesmypWtVys0BdwGssqTVRARElmmDvNH8+s2V+61bt7aYqFjV5OeLHtQRR6gDoR8QGU6Jo27pzj0sqgiv4iVRnZRWzqv6s9dee5mLBa/xcwm6MTbImxPftHBufKevqofftSHMh00bb2qv+3ntj3WUBUymLl27djUrKTrls3mLvPRXH+rV2OB6AJGmPRYthYrqw4qremTdNX03LEhw5aE9LY4Kbc/LOQKOgCPgCDgClY5AvSC8kAoIIf6MxFAdOnSoWdkgASJaDBTkirxYa7HyKuoBaTUJZWmDa5Sk6beu1LNkyZJAhAWOIDaJx3U3yjtl6pSA9RSRfrSh7wMGDAizZ88Ob7/9tukjErS+kvL/RRfwRXr06LH+QIzUd9IzRboRCxkMOYIXyZY3s2wpf4Md8wFf17Fjx4YzB50ZWrZsmXNRhN6LFy8OjTb5Efc4+tAvfTRnaJs0Flp8P/LII62qfDFAJ+rExQUL9b777vuTeqztFPFGcJ34Z6q/+FqTrvljN/2PI+AIOAKOgCPQQBCoeMLLAxwCgLV2+PDhdgzqDjvsYL+jZFfjKYJBlAFeZ0MgyCdSpnz5XilP3RAkwpa1atUqTJo0yUj4Jo3Wv2auqU4IJHkXLlwYFsxfkLYQ0z8J30U0L7zwwvDSSy+FefPmGQa1TWakF5ZsfJzRA4liyXflw39Vp+nRB42F+lbOa1QP/I579+4d2u/R3rDMnCfCEespp8oVK5qjRNh44IEH7GCKyZMnByIs5EtCpRt+4sxzXCZUf1TPKEk/68wzLVQZrj7R+RPN798dAUfAEXAEHIH6jMCPTKoCexl90PN6mqgBnEpGukJlZXZLJAvfWDYNvf7665ZFJDIzf76/CdXFq2QsvNRJSDFE7VZXn/LgqiDrMH1RuspC0Kgb/2F25EPg2JFf22QGvYQbsWvZKJhLsO6Sv+0G312R4Fz5S5muxQh18nqf1/z7779/VqJIHuGN9ZQIC8QMRt98QtlRj0T1sShgoaW50bx5c8ui+8qf68pcYOyJMMK87dOnT7Xl0ZkyyODBg8MTTzxhiynqKLQvVpn/cQQcAUfAEXAEKgyBiiW8URLD5jAsqmzSiVryco0FeRDiwE6ZMsV20kOQRd5ylasuHdJCvbhKYMXEHYFX/RYzd4P1t7ry8ifGTQGiwuY0JBcxFOklzNqgQYNst/7a79ZvxiqmH9XpmO0eetBvQozRLq/ZwUJES2SOmLSQYoR8Ss9WZ6nTNN681scV4Pjjj7cm0CGbHqRRhrHDJYC4tghzRP2yhJh/VB9zgznK3OBYarBDsumQrWr1Y/z48TbHCFWHPtWVF+llIx3HamNhJnJEsfM9m36e5gg4Ao6AI+AIJBWBiiS8PPj58KCHSPGqmKNV4wrlIAq8Doac4veLiLzFrSczn+qF0OlDnuoICffJCwHBBxnfUhEyERzyZBP0pR+QTXxCx4wdY9mUnq1MOdKkJxZHXCwQYcH39957z6zRbdq0MX1F9LhXbgFbSB9RO/gQGxfR/Kmpfd4YcLDDP/7xj5qyVntfeGhecM1HFFmCBdGc2XPMJYPyNc0t8oj0sig88cQTLZYzfsy1PU/QxcURcAQcAUfAEagLBCqW8PIQ5zU5cWnPPedcw64ma1cUYBGFgw46yIgvh0aUQtALIqFPTXWiswggh0pAdvOJ/Up7kDde0RMFAAseovSa2i/FfbVFbGPIodwXRIRxdeiTIsO1LZBKsCW8GweG8FofiTNPRFDJf8oppwSsqjNnzjRc8yWr1IHkOzfWl9qwIFJkibF3hwEDB1i/4vRDdeAbTpxkYjpjYcYFCNHYKZ9fHQFHwBFwBByB+ohAxRFeHvI8pIlBiivD0FREhk03W++OQHpciRKaM844I3z80cdhxowZacukyFrc+grJp75Qlnip++yzj73qLrTtY489Nnz7z2/N6i19UrZwfa2VK24L0dPX2CiF2wVkPNrfcisjsovVnEUAr/OJh0t63HlCPvLjh/3LX/7SNiMy7yDR9KU2hHZoD4vsnXfeGc4+52wLfQd5jdsP0zMVtAHSy9zi0A82vHEwhYsj4Ag4Ao6AI9AQEIjPEBOAhggTm344YQqiim8i/q+ykuajpggN5Peiiy8yoigrXpQQ51NnnLyQDvWF/LhUEG9XR9tyn/bjSlRXrN2zZs1Kk/cf1tUO4ZW+u+yyi40JbgwILg7y3VWeuP0qNJ9IIv66zJOTTz7ZfLxFgvOpl3lFOWIic7rdqFGjbOMXc0fjmE99cfOqbtrB5/bGG280txX5Suc60a66+qP445tMRJFXJ3i4suow83uOgCPgCDgC9QOBxr9LSSV0JUoQIR3HHnNseqMU/q+FCoSCutmgxK58rF5YBauqqox0QnaQKFkotC3KUR910S677bHaETUAAoJAdLiXr1AndVOWwyCwGNMHHTlbSJ356gCO6MEiBH9ZThYjkgTuFkipMKxOryh+RK/Aas4GQM2fQnQAO7DF55soC5BovnPIiHDnWkjdmX2JEl3q40CP//en/xdOO+00m5/F9ENjoHEi7BrHV2+//fYWj1gbJzN18t+OgCPgCDgCjkClI5A/s6qDHushT9NjxowJnILVuUtnIzGFWHYzuwChoQ1IL8fMEuKL8FWEkaJ+ER7yFCKQGAgT5akPIkOoqz//+c9GyAq17GbqImskB2vQj3vuuSd8/fXX1ibtl1tE+Nichg7EI5Z1FwxqU55++mnzheb1PW2jm/QrRA/DNuVGADm86qqrwrPPPGsn3UXHFMJYzByhPEJbCHGcR4wYEa684kqb89TNXCymH9Sr+c5C8aKLLrJwZYsWLfLIDYDj4gg4Ao6AI1AvEUi8hVdEiYe8NpYRkYF0HtylEuqHUHDVcblYe7HCYsmDDHOPdjNJDelRkc7kk54iKpzCBhnDdaJ///5mMVR9mfVE64z7nXYgt8R4Re9x48ZZXGDIjfoXt65886E/baMD/SSigCzX1FWK/lWnk9omcgcb1QYOHJjOXoq2beNXqn9szCPkHJvzOJgCH18iIGiMwVljKgU2aj/F/eVbrXyUVXlCu2F5xULOwoVxxGe3EDcGtZ95RR/api9sNuRNA5Zwfpt/cMrf18URcAQcAUfAEagvCDRKEbLaNb3liRwPZYgA8XLZADU0tUkNQe2NSISlFv+H9qiXD4QDQsPJZxxq0bFjR/O1LaQVNjvRhw8++MA2phEODUue9Q9ysTFnLqSJjcoINw4owKeXU9kQpW+UuUQ/NCYcvsHiZM6cOeHUU0811xPdK1FTP6kG6yiknuOiafvqq6+23+Xob5R84nLAAoYFETF2q1JuJJDGfAWyzjzjBDXI+jHHHGPWf+rhnqy++dZbU37VPf2d6WHCqxPC5ZdfbkXW/ZD6f+ektyb4/L4j4Ag4Ao5AhSCQaMIrsoIlDVLBw1iWSkhwuQRyxkdtcNoWFtl33303bN5k89CxU0cjNkQfQB/IiPJCIPhgGeY1MbrPnjM7NNu6mb2WxnrM635EZKMc/UB/BOJOSC50jMb3LcdiQaT2rUlvhZUrVprFEAszZLtchI0+CsfPP/883HvvvdZeriN3yV8KYW5qYUR9WGVZWHDF7YHFEQdNYGmn73yEuVwf2HzJQoi4unPnzTUrMT7H7du3NwJNveob38sh0blOdA3INgeZIBrPcrTrdToCjoAj4Ag4ArWJQGIJr8gur8Zvu+02I7sQzHITgCj40kFp7PpfsHBB+OzTz4zM4usLiYHM2KvgDUSXcqQR+olX3RxhzMY0EZ4oGVXd5bhG9SfualXK+ogVMppeqnZFjhQ+q1+/ftZ/COjee++9kQ9qqdqkHs0HIhnccsstYciQIYZ11ApbyvaidUXJotLRA/9sIiBAwPnNYoiFEa4PzCEs4IwBCx/cFZgnzBHmt6S25gjtRdvCdx09cUXRmEonvzoCjoAj4Ag4ApWKQCIJrwgZxACyC3nCz1DkpjbBFhmgTRFWtY+eWOuw1GlzGJZFCEPUoqf8qiuzHt0vx1XED+xuuOEGIzKdOnUqOZYas8mTJ4dFny8KJ5y4PuoECxbi4F566aVpK3ip+qk2wfWvf/2rbZDDQqr0UrVTUz3VjSu4swhgfrAJskmTJmb1JXoFcyRTqqsrM28pf9Ou5uXw4cMDB7IQXaMu/s+Vsl9elyPgCDgCjoAjAALl8wsoEN8oWeHksSOOOKLOyC5dgATow2+IATpyxXIHgdl2223t9TWHK0B4seZRhnzKG62L77UlbHQSaWFHPu4NuGhAtkgvlYAFxO7NN98MvQ/tbdWyGGjZsmUgNu/UqVMtrVRtCn8qJRoEIeVEdkXcrMFa+JNtfjDuCDhj/QcHoldg0cWyK7KbhDmCnpqvfGejHO4N+GCXep5Qv4sj4Ag4Ao6AI1DbCCSK8EZJDK/CO3TokCYxIgi1DVBmexADyB1X9OUj0iJya+mpffjkU97Memrzt0gL/qTnnnuu7ciXRboUBFR1EEarXbt2ZsEkTWPWu3fvMPGNiWniDT7FiupggxqLjlKFHytWL80Pxh2xubBhjmBtj84R7idljkgX9GPBxjzhQBT80BlH0l0cAUfAEXAEHIFKRSAxhFcEBiBffPFFIzGKTwuJSKKgV5TgiNxaeqnDLhQJgEgvFsa+ffsGrOdgXiyZUR24n+DOAPFEhAX38U1t265teOutt+xeseQJMk39tEdkA8K7IbSVtLkSnSNY24VL0vQ0AFN/0A98ObDkzDPPtPBo+B2TXuy4qQ2/OgKOgCPgCDgCtY1AoggvJAASQ/xWTpZCeMgmlRzU9mAV257ILSdsHXjggeYKQJ3FkBmRIMJpcQoZp6xlGzPCsL399tu2YUt6FNIfWY6ZI7Q5dEOYOtqkHy7FI8D4sHjAWk/MazbeIf7/sHhsvQZHwBFwBByBukEgEQxBZGXevHnhhRdeCOecc46hIUte3UBTP1sVacHNAF9SHeZBOiQnH2HcIEdsxiIkV7du3ay42uCHvuPn3LZtW1vQ5NNGNK/ILhEQ8NtlnhD/VvMnmte/lwYBxkzRIzSWpanZa3EEHAFHwBFwBGoPgTonvJAsLHNLly4NDz30kG2YgYiJ3NQeFA2jJUgLBBE5+eSTLe4qrgak50t4hRiWVkKPYd2ljkxipPYgxBDj1atX521VFrnGdYIQa2eddZa9dneyq1Eo7VVjmG08S9uS1+YIOAKOgCPgCJQfgTonvOria6+9Fo4/4Xjbzc7ufiyHLuVBIOrCgEsAG8o4MIF0NlbFERHNFStW2G7+Aw44IGcxkWmsvEQqmDZtmuUVqcpZcMMNLYq43nXXXeGoo44ya7F0qKm83y8cgbhjVHgLXtIRcAQcAUfAESg/AnVOeHmgQrLY7NSxQ0ezPrJL3KW8CBi5TW1OIsLB4CGDbUc+rgIKY1ZT6yJC+FwTTYNwbLmsgeTlHtKjR48wY8YMs/JG06trT20RuaMqdXhGXYUfq05Hv+cIOAKOgCPgCDgCyUWgzgkv0ECy8BOUJS+5cNUvzbCi4zrSokUL25F/++23pzeVkZ5LRGyx7nJ0MocUICK12cqJYNMWpFVW3mx5o2lYcZGXXnrJLNBE7lD7IsLR/P7dEXAEHAFHwBFwBByBTAQSQXhRSiQmU0H/XV4EIL2QSkjo8ccfH8aMGWMNKr261om727Fjx7DVVlsZcYbUViciqBxvzEEUa9asqdZ3GNJNneQlKsOgQYPS1auudIJ/cQQcAUfAEXAEHAFHIAcC1TOUHIXKkewEphyoxqsTUsmCo2vXrqF9+/bhnnvusYKkr/th4wMHtDDBuvvee++ZiwKZ44wf9UFiIchscps4caK1Iyuu/djwR37c+BY/++yz6cgd5I3TVrQu/+4IOAKOgCPgCDgCDRuBxBDehj0Myel9nz59jFASHs4E19tItDIIL0JkBo7z5dhcWWLtRg1/IL3IwQcfHKZPn24hzbAmq17uUR9+3F999ZVF7rjgggusHciuypPPxRFwBBwBR8ARcAQcgTgIOOGNg1IDyIPVVJZWTtj65JNPwjvvvGMEU1beH9atDyGHdfeDDz4I3bt3N2TyIaFqZ/PNNw9EdpgwYYLVoba5QoAJP3bHHXfYASQ77LCD6ZZPOw1gyLyLjoAj4Ag4Ao6AIxATASe8MYFqCNkglFhXEQ51ePrpp434kg4R/WGDqfeVl18xsqtDH/J1MRBx5bS3OXPmhBVfrTCSG7UUjx49OnA6GwcfRNMbwjh4Hx0BR8ARcAQcAUegtAg44S0tnhVfG9ZVCCYW2PPPP98OecCiC0nl8+WXX4b5C+abdZbOirzm23FcGGTlffW1V624iPNjjz0Wdmuzm/kHQ7QLbSNfnTy/I+AIOAKOgCPgCNRPBJzw1s9xLapXIr24Epx19lnmWkBEBeTFF180yys+tlG/20IbxA94wYIFYcmSJUZsn3/++bBq1arQ97i+ViUkWES40Da8nCPgCDgCjoAj4Ag0bASc8JZg/LFCYhUtBQEsgTolqQLSS7/aVrU1gvvggw+Gzz77LGDtJcJCsQKJBS82vVEfRw7jF8yhFGeccYZVz30nu8Ui7eUdAUfAEXAEHAFHoFGKVET24Dsg+SKQScoyf+dbX9Lyqz8c/XzrrbeG3//+96EqFbO3FK4GqoPriBEjjExfddVV9S4iA/1TX909I2kz3PVxBBwBR8ARaAgIuIW3iFEWGfzuu+/CokWLjNTIcllEtYkqSh+RLl26mBWbfpZacJfgEAvawOIrcljqduqiPvCD5OICwlV41oUu3qYj4Ag4Ao6AI9BQEXDCW+DIi+wSPmvYsGGhdevWYezYsfWK9IqsAdErr7wSfv7zn9uJZxwLDHmDmBYqIrXfrf0uEJHhsssusw1xixcvrjfEUHNk9erVdkgHvsn1bUFU6Ph7OUfAEXAEHAFHoDYRcMJbJNpstvrFL35htQwZMiQsW7asyBqTV/zzzz8P8+bNCwcddFDo379/eOqppwJpkF58l/OVKJEe98C40LlzZ4v6QAgynb6Wb51Jyy+yi/X/T3/6U9hrr73Cf/3Xf9lBG056kzZaro8j4Ag4Ao5AfUfACW+RI0wkg+uuu85qGTNmTNh+++2LrDF5xV999dXQu3dvU6xp06bh4osvDvSVDWyK6BBXa4ig5NFHHw3NmzcPhx56qCVxkMX8+fMtYkN9IYVff/11+N3vfmf9++Mf/xiWLl1q36M4WIL/cQQcAUfAEXAEHIGyIeCEt0BoRciaNGkSrrnmGotgwGENSudaySILJZEZIGn77LOPdQeLbsuWLcPJJ58chv9teFi7dm1epBdXBrDhaOIvvvginHjiiVYv9WAx7tWrV3j55ZctrZJJocZ/2223DRB7/JMfeuihsOuuu1rfdN9++B9HwBFwBBwBR8ARKCsCHqWhSHhFyiAw0e9FVlvnxeVje88994T9998/dOrUydwXFK4Mcvr666+Hjz76yE5lQ2GVyaU8ZJnyhB8j3u4ll1xiv0WuKcf34cOHh379+oVWrVql28xVZ5LTo/3Cf3errbYydaPpSdbfdXMEHAFHwBFwBOoLAm7hLcFIylqnawmqrNMqIKYQWtwLvvnmGyO7kDTIKiJyf/DBB5sLx+OPP27plCFfNhHZxWKMpfPcc8+1+mTxpYyIIFbel156yaqpZEyFE32E7HJVH7Nh5GmOgCPgCDgCjoAjUB4EnPAWgavIy1dffRWmT58e2I0vklNEtXValD5BXJHx48eHww8/3L6TLon2sW/fvmHR4kWBOL1INJ/yQ/Qgy5BnfH9x/WjWrJlZb9UWeUVu995rb8u7cOHCgjfGqe0kXNVHXZOgk+vgCDgCjoAj4Ag0JASc8BY42iK7bEr69a9/Hbp27Rr+/Oc/h3/+858VTXplccVVAdljjz3smknW+I3VFjlvyHlh8uTJFrKMdOqQiECTBtnFZ3e33XbL6apguG7SKPTo0cNcJqgHspyNSKuNpF41R/BVfu655yxWc3SxkFS9XS9HwBFwBBwBR6C+IeCEt8gRJVLBbbfdZrX89re/DStXriyyxrorDkGT2wIW20MOOcSUyUU2yQvp5VCFCy64wFwVouHKfgg/Hg1MjOJOHTvZMcKQ30wCndlrfIY5kAK3CiSXDpnlkvJbZBer/7XXXhuOPfZYizW8fPnyil4QJQVf18MRcAQcAUfAEcgHASe8+aCVJS8HTjz44IOhzxF9wgsvvJAOS6bX81mKJDZJltnZs2eHxps0Du3bt69RV5FefFTPPvvsMGrUKHNHIH3td2utPNbNLbfcMhx2+GFpH9Zc+JCOHlyJ+/vmm29aHRDkSiO9KI7F/+6777Y+PPLII+b2wo9K7It1wv84Ao6AI+AIOAIViIAT3gIHDUIGaYGInXbaaeHJJ54MRxxxhNVWiWQGnWXdhWR279Hd+iLyWR1MlCMfIbeIroDrApbfzTbbzAgrG9UGDhyYriIX2VUG3cfKi98vh15UmqgPLVq0MAw4pY6oFjvttJN1RfcrrV+uryPgCDgCjoAjUIkIeFiyIkcNoijiK3JYiWQG3SHvWHchvFhrEfoWtz+qgwgLbOTbe++9LQbtVVddFYhXrPtxIFdUB0KYTZ06NQwaNChOscTmieIY/Z5YhV0xR8ARcAQcAUegHiGQCAsv5IYjWBHIQCUJZFD+qFzjksOk9VF94ECInj17mnqMSz79IS/jR2QHyv7qV78yv1XILr/VRpy+k5e6OnToYG4Ac+fOtWKVNj/QVwshrvzOB9M4WHkeR8ARcAQcAUfAEagegUQQXggAYb2+X7ueYFUaqZk1a1a45ZZbQiWTMqYJ1l3Gol27dkbM8iGolBeZww0B39XDDjssLFiwgFt5kV3yowcEEaEexeW1hAr6Qz+EI1d+V9r8riC4XVVHwBFwBBwBRyArAnVOeHn4QwQWLVoUZr07y5TEGph0EWkh5FTnzp3DlVdeadZMojRUGqlBX2TChAlGLvkuqyTf44jGkbwjRowIe+21V8Bv9eGHH04T6XzHFd9g6oWAI3PmzDFsv1+X/PlhCm/4Q8i2X/ziF+kNeJU2P6J98e+OgCPgCDgCjkAlIlDnhFeg7X/A/oHX6QhhrkQodT+pV5FF6Z1UPXPpJSvq+++/H7bYYguLkUuaNrDlKhdNZ6yEAwQX313iEiMXXXRRuO+++8KyZcusznxJr+ZBnz597CAM6tykUfIjNkhvfJmJNnHddddZbGE28CG6bz/8jyPgCDgCjoAj4AiUFYE6J7wiSq1btQ6tWrWyDUr0WESsrL0vonLp3bJlyzBz5swwcuTIMGzYMDtBLEoAi2ii7EWJk6vX7S+//HKAVBYiIm+EZWPcjj76aKtm7fdrLUzb4MGDLXLDt99+mzfplX5t2rSx43nfe+89I9dJnx/CEf/lIUOG2M/TTz/dwrPxQ/NH+fzqCDgCjoAj4Ag4AuVDIBFRGkQQcWt4/PHHw3nnnWdWXkiNCE/5IGi4NWNtxZKL/zT+u/379w8/rEtZa1MnncUV1UEkBQ6rwLUD0djp/jvvvGMuE5dffnmasMYdW80PDqEgpi+HXCBKtx8J/CP9Fi9eHOh/ly5dAnGblZ5AlV0lR8ARcAQcAUegXiJQ5xbeKKpYeHfccUcjB9H0JH+HvEDudE2yrlHd0Fc+ssSH7d279/rb8bmuRV6gDojo+PHjw/nnn291iOzyQ23g4sAxxRzSgUB20SEf4Uji5s2bm0WdcrSTZJEVlzmN1buSyK7GBowVQSXJWLtujoAj4Ag4Ao5AdQgkgvBCDEReunfvbm4NPGQLIUXVdbYc99Bdlkpdy9FOqesU3lhmOQxhhx12sCZE0mpqT5ZbfHPx0eW1fdOmTdOW3WzlOV6X+se/ND59G7eKmiQ6P3r06BHeeustI8si0zWVr+v7kEd94uJb1zpLTzYKMsaISHBd6+btOwKOgCPgCDgC+SKQCMKL0jxgeaBi5cUvltfslSQiCJWgM2QXsrh27VpbXEAikbiERuUJPXb77bfb6Wrbb799zli7YCOCPWDAgPDurHet3Wh6TbgJX6y8W2+9dSAUXD4611R/Oe+juz7lbKdUdWusVq9eHSZOnJj2O447P0qlh9fjCDgCjoAj4AiUCoFEEV49aHv16hUmTZpkr1JFdErVYa/nRwTefvttW2Dwyj2uQHpkycaye9ihh4U999zTCC0kOpdQBqswgo82vrjE6G28SeN0eq6ypDMPVJ6DMbDyMl+o14lYdcjlf0//5/A7xvKPGwkYKz3/Gr2EI+AIOAKOgCNQtwgkhvACA4QJUoOFl9fsEDJERNh++J+iEBBhxTo7bdq0gAsJAsb5EBrCj2GN79a9W+yyGl9cHy688MJw//33h+XLl6fHvaaOidzuuuuuFrHh3XfframI388TARHbNWvWmC99z4PXn7qX7/zIs1nP7gg4Ao6AI+AIlBWBRBFeeirSdeihh9rrVF67i+iUFYkGVjkuI1jvsO6K5NQEAYsRxocQZitWrAj45CKkadxqqkOkFxeIfv36heHDh9upbEqvrjxtaPHD/GCzXVzdq6vX7/0UgSlTpoSqqqrQonkLW4QyPi6OgCPgCDgCjkClIpA4wqtX39tss42dsMWra0REp1KBToLeIodsCMRlBNcRJI71DrIL6cGqiv8ssXXjlrWMkT/UQ5vt27cPhx9+eBg7dqzdVXok60++Kg8RD7bddtswY8YMy+Pz4ydQ5Z2g+YF1l82MhxxyiNURdzGTd4NewBFwBBwBR8ARqCUEEkd46TekFyFUFsey8vpdRMdu+J+CEIDQILgy7LzzznYohIhsdRUqD+HHnnjiCYvIwBhBMjVW1ZXPdg8ShT64VLAR7e9//7tloz7pma1cNK1Pnz7h1VdfTe22KyzMWbQu//7jopKNahwNvdVWW5l1t9AxdkwdAUfAEXAEHIGkIJBIwisy1KxZM3vw6sjhuEQoKeAmSQ+wg7hg3YXQHHbYYaZeTdY7kd2VK1eaJfbss8+uMfxYnH5rjMlLjFpcJF566SUrWtM40w/INm4R+PNOmTrFyqGrS2EIgCeLym+++cZ8d2XddbJbGJ5eyhFwBBwBRyBZCCSS8EYhwgLIK3QexG7ljSKT33e98sdFhKgKLVq0qNFCC/EEc4jk3XffbSexYRkuxrIb1VrElTSI9PQZ0wNHB5NeE3kVUectwJtvvmn5N910UzspLtqGf4+HgBYZWMz333//sMUWW9g4C+d4tXguR8ARcAQcAUcgmQgklvDKAoiVt1OnThaGKpkQJl8rWe++/fbbwGYkWe+qIzMQIN2/6667AieldezYseSvuEVuIdYXnn9heOrJp8LChQvTRDsXupofP/H1/iHZp6/l6k9dpmt+YGXniOkDDzzQ1NH416Vu3rYj4Ag4Ao6AI1AKBBJLeOmcrE48gD/88MPw9ddfm/VP1spSANAQ6hCOuIbsvffegUVElNBWhwE+u5DKgw8+uGSW3cz2ZEXesumW4ayzz7LDLHChUHpm/szf3bt1t4NKIPSU8fmRiVC836+99lrYb7/97KCJuPMjXs2eyxFwBBwBR8ARqFsEEk149cobgrbHHnuYdRK43PIUf9LIeodLCBEWunXrVmNh3AnAGFeBL7/8Mpx66qlWhrRyYS9yS/SFgQMHhtGjRxspr47AogvEbJtttwlVqRBabHB0yQ8B5gf/zxhnNiVCeF0cAUfAEXAEHIH6hkBiCK+skLkAhqi9//77ZuUV0cmV19N/igDklZ33NVl3IbuQTMgxcW7xrUV+WPeji8NPay9NCu0yD3BhYbxxpUC08MnWiubNQQcdFGbOnGm+3uRXerYynrYeATDSAob50aVLF7fu+uRwBBwBR8ARqJcIJIbw5iKxkBdIGCGS8CElfqxLPARkvcM9AN/MAw44wArmIoMiu4sXLw6PPvqoHQEsstlok0bxGi1RLjYrsrHusccesxqrmx/0k5i8u+++u4VcK5EKDaIacF22bFmYN2+ebVaj07nmR4MAxDvpCDgCjoAjUC8RSAThhbDI2sT3TJEVqkePHmZ5hMDlIkCZZRvyb+HGRjUiM2DdFQnOxIV0LKyrVq0K99xzbzj33HPNdzdX/szypfqNzpoDnMTGq3YiS1Q33j8QiDclPXv2tJBauG9Ul79UulZ6PZof4It1l8gMLHpY5Lg4Ao6AI+AIOAL1CYFEPNl4wLJhBqJiFt11G8dTtbTUg3jzzTcP++67r71qZxDcEpV7KmoBweLggw8+qNZ3l7xgDNEcNWpUOOqoI8Muu+xSZ+RHutA7iDdxg3FniaZHe954k/Wh05o3b26nt0HwXapHQP93WFB89NFHAZcQRCS4+tJ+1xFwBBwBR8ARqCwE6pzw6sG75ZZbhnHjxhl6EBhZ+QQnZAfhVTcE7quvvspJgFTGr8FcQPCJbdq0aVYCK2IMVg888IBZ+rD21bWlj/GWDhxjjG64Wig9c2xF1Ai5Nm3qtLB69Wq38maCFPmt/3csNImCstlmm4W1a9cavpFsvqiMguHfHQFHwBFwBCoWgTonvCIq+Je2at0qPPzwwwYmPqN6KJNAPkgwhwvg2vDKK69YvmgeS/A/afcQrLtsPgMvRFgLIrATfs8995y90j7iiCMsDWKZmV/lauuKiwWkF1/eoUOHhjFjxxiRVXpUDxFhfL07d+kcdDpf5sIpWqahfgcT8MJ3d+7cuWnrLrhGhbmRngPrvUait/27I+AIOAKOgCNQMQjUOeEFKZGu4449ziy3nPaUCoC1Pj3yoOUhjXAS1IIFC8LSpUtjx2q1gg3kj/DEegdWTZo0SVtLoxCI+LBD/+OPPw74zErSREcJdXSFhKEnRwgf3/f4MHLkSPut9Khamh/EDJ4xY4b5I2fLFy3TEL9rfnCUc69evdJW8+iYgzm/iW2MX3fqv6OLI+AIOAKOgCNQsQgkgvDyYOUBi+CzOX36dHNbgMCsyzg5i4c1Vl5cGyZMmGBlog9qS2jAf0RicfngsA7F3RUZFDRYTiGDn3zySWCBMWTIELsFvknDE93Ri7BqWG/vuece01Xp6pPmEZuveGNAvxARPOVryFeNO+4hS5YssRP0wIO5INEcguw+8sgjRnq5p/+jyudXR8ARcAQcAUegUhBIBOEFLMgLD2PILD6b/7+9cw+2cuz7+EUHRHhLZz0VHQi94X3eMkM2hUKZkpJzksehZzTzGOMfPAZjjNODccixIjqIlyQdkIyX6CQijFPeqHZ02h7J6b0/v+233G1rr72OtdZe39/ste57Xfd9X4fvde25v9fv+l2/H6YNX3/9dXV6jU1s3H/YYYfZC3vt2rWJZ0kvd3Fyt2DBAvNaAJ4QlTiJddID4XnmmWfCyJEjzZyh5n3FiOVxFceFvfbaK7z88stJq+ftxC6VzVgQf8jcL0nGUNIM6nmi48P4QLvLbx8zNJ1znxxNnTo1tGrVKjRr1szS/dl6DpGaJwSEgBAQAvUQgaIhvGALMWHjDHaYaHonTJhgnhvim9j8Bc0mG5br2cHvz8Zf3JZYZl9xIstkwaNmOYEBDkgtOG/dutU8MgwePNjcj/Fs/L5ig45+dw0jphdopt1dmadTZx8faHlpP8EzLF1r8gmzltWrV5uZAhrz2gSbbiYWFRUViVtEeBNQ6EQICAEhIARKDIGiIrxgh0YS8oVbLMiYhZiNonxBxpzYOLFFy1tZWRnWrFljsHt6ifVB3qrrhARTD0w+wDKOCedOap988snI/Vjf0KlTJ8M7vqSdtwrlOSPqzthAmBC9+uqrRnzj6fEicWGHrTebs+LjJ35PuZzT997HTAIYHz458HHD/xfn2HQzYeL/D/H0csFK7RQCQkAICIH6h0DREV4g5sXMSxZ3Wj169AgTJk4w5J20OMFxLS8vaIT0chXX0GLiwWY+XIvFBcLjxAb3bxBdlv3BuZRwY2zQVnwyjx49OkyfPt0IrafTZtpJu3B1Bw5ogj09PgGwxDL58nZ/9dVXtmrC/1Zc3CUZEfnYzOYhpUttfMTbpHMhIASEgBAQAo5A0TJEJ2f4VcUt1axZs6zOpBt5+z3ULcvWa9asDRA9xF/s9qNAX5AASBcfzvlQrp/Hr++I+lCGa+9wx8WGLSYD1MNwjDxdeD2wfd22bVvA/RjCdce6QHDlPVsnt4QTHjZsWHjooYdqdVeG2QskDy3vjmqnj4X4+KhtjHh63kGKZUh9fFKDCRBBJsACu2Y7RuOY1QAmSs8991y46KKLEuPHn4tlp1MhIASEgBAQAiWHQFETXidpLK2ywWrRokX2grYX+C5/BCbo1eu/EzvyC9UDlOkEBhIA6eLDOR+Ig5/Hr5PuBNjbk+86er6YdkBaMPWIC8SGOuGqCw3eiBEj7DLPUb9SFLAH1/bt24d+/fqZ6Qvt8HTaS3+hCQYP1/IWsq3ez5ThY4Cjf5KNEa6RTl35FFIg/lVVVaF79+5WzK7R/xB1BjOiHGIzP3z48MBEgnTqJhECQkAICAEhUB8QaFjMjXAiwAsZknb33XcHwsd27do18aKm/thqotmE8LVu3bpaA5wnIgcpdFJAPRA0pGgMIZdE9OI3S8LUt1HDRqHxbo1tw0/z5s1th3ucOEBqPB/LLA9fnj/aOzSabgdNOV4em7zmzZsXLr30UqtnfSA0tJv+wTSDIBt49mBy5On0B4JG8+GHH7Y+K4THAbBEvB84r/q+ysrbuGGjbRD86aefrC+4B+07RJyVC+rD0ccE7eETz4v8chHHAb/MvXv3tqwYF5Th5YwfP960/vg7rg9jIxe89KwQEAJCQAjUPwR2iV6usdAOxdnAn3/5OTRs0DBs2LAhPPDAA+Hiiy8OkMk40SCi2PLly01Dla8XdjwffJKuWLHCfNu6JwmICzvZ991334RbL5z0U0+8IPBBo8Z1bJEPPPDABMDxvBOJWZzQfRAayP6MGTNsOZrfpPOB0GzcuDHcd999dq1ly5YJEpxFcUX3iLefik2eMjm02K+FbcbzdPqKCcCSJUsC3gkGDhyYN0JHGYgTSvodH9JMLiCwTZs2NXLLJA1Sy3iBaELO6RMmSj5OWrRoYRO3Nm3aWJ58eRsSCVmceB6rVq2yCc+FF15ouXg6P3BNh1YXMxfGJe3xNmVRpB4RAkJACAgBIVB0CJQE4QU111Su/r/VYeq0qeGSSy6xTUlx4kgUrv79+5uHB78/G8TjeeLHFc0Yu9bRHh9wwAGWPySmLqEOaIEhQJ9++qmRXzSwaCSdlJJHLuTCiQubt7p06WLEmnIhuuQL4Rs3blw48cQT7XouuNTV3p113fsLLCD2xx57rG1W83SvF5OloUOHhv322y9n0hvPG08QBLnALAANKRMbiCub5uoSSC/mOvgMJswv/UOwEHcZRjn0Y7ZjxMcH3k6wh+/cubOV4eMD92ObN28Op59+uhFs6pttWXW1VdeFgBAQAkJACOwsBEqG8AKQkzU0uewk//uYv0dv52CkDi0ekcWw1WSHeZyQZAJu/DnMJNgsB1E45phjQpMmTZJmBalwSUUW0MK+8MILgWXu04eebp4SeC5epueTztGf++abb0y7i+Yb8XTOmQRg8sGyvuNHen0TbxsaU4gtBA77XtuYFQ0SCB7a148++shIbxyjTLCgr/mQH2W9+OKLpj0eMmSI+f0lPZmkO0awscaLBqScPDF5QLKpr2MCmX799dfD+eefb3UnLzTQy5YtM6J++eWXZ12GPagvISAEhIAQEAJFjkCDf0ZS5HVMVA8yAXFA04rpwKLFi8Ihhxxi5IN0zBwgw2hfWaLlhV8bAUlkGjtxgoBWFo0YhIaQu5TBcnSctPCYk1uO/vHsat5LOkvceFDYr8V+Zm8KUcUe2W1tM6kr+VEG5c6cOTP06tXLSJK3mXRsWsGioqIiQZhIr48CdrS9cePGpsWE6LNZjUmKk0WihqGJbdu2reHiWKWLh/cpZUFMIdYdOnSwSHVodsHW7/E8HW+O/vFrHOP3cx2Ce/TRRwdsfidNmmS2vhB3rtEOjumIjw3uJzww5gqY1riJByYOL730krl2Y7LoGKWTt+4RAkJACAgBIVBqCCRXR5VAK44//ngjNx5i9tffqjcOHXXUUYnoWhDJOKFI1SwnuxCBW2+91ZaVzzzzTLO95FqcQEAi6iIefk/8CKngA8n9x5X/MIJ28803m02nk95UdYxfc7LGcvq/f/i3+Sy26xEfokyCC0DYTz311OrkNOocz78Uz8EQfNGO4m2ASQsEj3SOCOF0CauLQFzTHR/x/sfvM2QUjemAAQNsHNIfSLy/Oa9L4vdzL/VHmMCMHTs2zJkzJzz3/HOWRn39uiWk+OI+8kajTdQ5iDl1ZOLGhI7AI0zmuMa95C0RAkJACAgBIVBfESi5txwvcX/pE2L2448/DosXLw6EH0awseUebGaRdAgNRABShEnE5MmTwxVXXJEIyuDXyDNXgVTwgXzhEgpSVlFREW6//Xbb0Z8u6aVNTlAw7Ti2z7FWNSN3EQ64H1u6dGmAsCNO1uxHPf8CF8YH9syYoUBMEbSYCLaxTARw0RUfS3axlq84fti8Ll2yNFx11VVGIsGc8ui7fAj1pzzyRSN77bXXhu+rvk+4XfP2pSqL570+2J9D8hHS0BxPGD/BzDqyWQVJVa6uCQEhIASEgBAoVgRKjvACJC9916hdcMEFYf78+UZgHGTCpsajr0EAahMntGwsI58xY8bYhiPSIUROHGp7Ppt0X0KmXpCyc889N4yP3EJhpkF53rba8oZgUbfPP//cjmySIo182VyHnTB5OqHjWE5Ce8EW8xG8H+CFACGNa2wKQwOOgHeq8cE1vz579mzTjv7tkr/ZhjT6CcwZj/kU6ki+PgaxSceMZ8qUKVZMXaSXsYDguQQvIn/5y1/sN19PPfVUqDiuwsw+8knUEwXoRAgIASEgBIRAESKQ3zf1DmygE0N2wuNqCTLAUi2ClhfBzjKVQCjIp7Ky0jan4esXm09PLyRRdJIE6YCwsjTOEjybrKiTk5aa9Yd8cR2BtGHCgZAf7q6mTZtmS+3YC9MOL8duKpMvJ7w09+STT7aJBFh5f7qWF/MVxAmt/ajxRT+AIRrzlStXJrTmPkZq3J7XnzbGo/GA4GVjzz33DGiYkV2iSIPJ6u3jgyMbOJn8ueDJA+LMRIB2OR5+XUchIASEgBAQAvUVgZIlvHSIEYKI1LE0y472hx952FxDcQ0tHoEYkGQvdl74PM/yNsve2LqiDdsRRMYq9Xu9qBt1gYQRAeu5//nDXrM2QsPzkHnq36lTJ8uOpepHH33U3LLhEmtHtsMqUGRfkFRwRc4555zw7rvvhg8++MB+g3l8fHBvMqx9jLgP29GjR9vznm4/CvyFqY63gzGKKQarF5EVedI6ezsg50wGsd1FsFvGbRrEGeH5ZP8XdlFfQkAICAEhIATqGQIlTXjpC0gfhACtbr++/cwGl3T8jWIH6VpeJwJcQ1zzyaY33I6xE35HEpnqWlSTcSce+I+lzniaSCa0weuN9g5XYy7YHuPjt1u3bgktsV8r1yNYQfzBFxMPvFawyQ8BJwiga3lrYuRY0x94M8Ae2rX/3gc1nynUb28H+bMKAeEl0h/p8XHtdebI+IDUI4wnCDDEH2GcoyGWCAEhIASEgBAoFwRKnvDSUf7iJ6BDx44dE6QXV0zY5SJOKjl3koB/0m/Xf5tY9o3fw307SigXYoZg2kCdsecl3esarwsknnSP3IavYNxZ9enTx9LZECepRsBXATAHwPRl6tSpFmiBMcOEgU1dSG19j2aUyZP59I36iPx2hlAu5But7fGRhxJIOJKs3nhmoH38L3y56ktzWxe36eaaRAgIASEgBIRAOSFQ7958kAGIAZpbNutAENi8g2AfiziRZBPSsRXVHg5cE2g37IQvJ2bY3uL3F3dUSJzwOrnBHtVtMyFka9euDaecckqi1n5fIqHMTwzbqO8x9SC08IQJE8xbATgTZYwIZ4ibDoA5GK6vXG+aUreT3tm4spGNumH6QmjiFR+s2K7eXj8mTJg/YOYyccJE8xPM/wHjX2TXINOXEBACQkAIlBkC9Ybw8rJ3wsKudpZw0eBCcF599VXr1rg9JC7IWrZsaTaOO8OUIdk4czJC4AFIGGSMNEiOE1/qDaFH64hLtkWLFoWzzz7bsrOl6ggHyZ8R8L7HXRmmH3jFQLBpfeWVV+zcCaNj/c6id8zcxcjiz8VBFr1ubMb73zeqtdOMEV8hwFYZbTQuze6//34zgWCjGtfddZ81Vl9CQAgIASEgBMoIgXpDeOkzf/FzHDVqVGBX+g8//GDkkF32iJNiwsziEqyYxEn7brvjtUoxAAARG0lEQVTtZvaXkBckTmTnRZrrQYMGGRkmgtZFF11kwQS4h3ZLakfACS0TCkggZgEdo2V/NMD4beY6kwlwxL6XSYdr0ndtUBzYUjdIL/XGTMNtkKk76dj34ncXm25MfJgYaWzUPiZ0RQgIASEgBMoDgeJ4i+cRazcNIIIUm3R48bNTHY0vS7wsC2MCUFVVZRpeii4mouikDE0kJD1OVggo0TZalt9nn33CPffcE9BkEzoY7V0xtSGP3ZnXrMAWPBE0/2vWrAnLli0zc5CaWl7MYPDhy+SDZ7xf8lqhLDNzLS823EuWLLFc6H8CsBDFj8kc9Yb4ci91L6b6Z9lsPSYEhIAQEAJCIGsE6h3hBQknvdhs2tJvtDGpcePGYcWKaptHtHnujN8JUF0IQhzQ/vnHl5Dj6Z5WV16prjsxITwudYack0be7Lwn5Cwknnbtv//+1UvVO2kjVap2FOs1iKH3OROiefPmmbac8cD4YOwgH638yGxlM2kH+fr48DKSpWWSZ7J7fYx06tTJgo8wBhkfbFbDF/W6devC4MGDE4/6/YkEnQgBISAEhIAQKDME6iXhpQ8hLpANNKW4Z2JzF5ovBFLAUi+SDhlwLRnaYf+Qf810J0uWcY5fEDO0de5GC/OGVq1aGemFnGGHSvvyWWaOVS6Zx8EWgkhfYhIyb+680Lx5c1sFoBFc27xls9nC8jvdMUK+Pj44Z3zUTAu1B/2jqIyESREa/k2bNplpA5sZcVfmZJfxkU7dMypUNwsBISAEhIAQKEEEGhZLnSEHfPItkJdDDz3UNKLXXHNN6NGjhxFF3Hgh6RAC7vnxxx/Npy92k05kMJWAbBCpDVdg237aZkQa0pOLkD9losH9Igp5jLBBDW0vy9jufiydutvD+voTAkwUGBts7jp14Knh6aefNnL6zTffmE00JjEEIkHSwZl7COuM7S/9j3a+bdu2YcOGDTbBgnySH2m5itcHMs3kh4AaTOZoD/6CSUfTTBspN1chP4kQEAJCQAgIgVJGoGjeZLzEebHm++Ma0JNOOincdNNN4Y477jCSCqHJRBo1amR2v52iZeThw4cnSBAEeMaMGeHAzgdadrmS3XidsMOExHz++edWBhHlcDeF+OSAoz7ZYcBYA19MXxgfzz//vNlNQ1zR+MZxth+1fHlfcHnkyJGBMUK+CH2It4Rx48aZizzS4vfzOxvxPLAzJn/clN12220Jks44zNf/lNUv/3PRbJqtZ4SAEBACQkAIZIVAbqrIrIpM/hDBFNCuoRnLh1bKS+GlD/mAeKBh+yLSmLJ5zYmw35fqCLkgH+xn0abhBxUNGrJt2zYLEoFdMNHe/N5U+aV7jche69evN5tdvAWcd955iUeldUtAkdOJjwPMR5gQTZw40fqXPk5XGBuMWbS3TzzxROgYeVDApRzCRIm8mCQx/rgvn33HBkbKfOyxx2wFgoAl+cjf/2/4f2SFhAliPsd2utjqPiEgBISAEBAC+UCgaAgvL1P/5KNhngd5QmpwTwY5uPLKKxPaN7+nriMvf/KBrBx88MEWrYswwGwSmjZtmkVHg+xCrJ1A1ZVnOtexz8SUgYkA2jsEsp4PQpNO+eVyD31LHxN5bf2368OA/gPMv3Em7fcxgpkLgS3Gjh0b5s6dG956661wxBFHGNllkpTPFQCv313/uiv07NnTViBIy8eEkfaQTz7y8nrqKASEgBAQAkJgZyFQNISXzWV8Ci1s9HFXTpmU5YSGZ4YOHWobnCoqKsLdd99t/k7zTXYpB+0aZgxo1wiBu2XLFpFdgMmz0Lf0H2QU8nvGGWeYZjaTYpwgcmR84AP63nvvNZtu/D2TbyHILuV16NjBCG8m9c30Xp8UZPqc7hcCQkAICAEhUAwIFA3hRZPES7VQAqGBQKLpzVZrhWYVDSvL1P379w/XXnutkVHqXIi6swGKaHCES2Y5HPMJaXcLNUKqvTMQ2hlTErDPVOgbxhnPX3311RalDZMUpFCEkfLcvIbzQpRDuyDWEiEgBISAEBACpYpA0RDeHUXksOXF40I2ArGA7LKpiRC/M2fONPdguDnD6wOEI58mDZAuCEw+dvZn095yfYYNa3jfyEZ84sNGuIpoBQDyixRqfDOWGdNehoipQaEvISAEhIAQEALbIVA0hHe7WhXwB4SVjT2Z2lNCZFiSRjvMbn4idbVr1y5cf/314c477ww33HCDkV2uO+nwY7ZaN8gM9UUKrQEvIOQlkTV9xZhgFQDMt/64Nad6M/H55dc/IuD5GHBCHB8bFOS/My2UzXFOpr2MTPPQ/UJACAgBISAE6jsCZUN4nVCwZN2wQcNEmOF0OtiJBFq/6667zjapQXaRc88917wzdOvWLYwYMWI7Da8/52X773TK5B7MJzBlQMjDiY0l6CvvCLh2Ht+8bEhEvO/SLczvZxXg9QWvW/hiNrJ53/t18vO0dPOO3+f5QHjZ3CgRAkJACAgBISAEakegaPzw1l7F/FyBLEIwWGLec689EzvaSUslTkqIePb444/bJjJMF7Zu3RoFzfrNXEGh4cUtGbvyV61aZR/yhJRgM8w1AhDwu67y4nXBOwPR1ZBs7Y7j+ek8NQJOIjFHqKqqsj62J1IPkUSm9C2k+bPPPgsLFy4Mt9xyS5g9e7aRXsYfEyZ3i8dDlMc4Ig377HQlPoaIrOZhsuPp6eal+4SAEBACQkAIlAMCZaPhpTMhjRASNoJBSnDaD0lwopOsw/1aq5atwmWXXZbQshq5iIjQCSecYB/P/8svvwx//a+/hilTp4S+fftamQSmGDJkiJHlZGXUTPMy8U183HHH1bys3wVCwCdFe+yxR8JWG3dzv/4W+c6NIunVJd5vRMgbM2aMjZX4pjK8bdx4443hrLPOso2I5Ddr1iwLGkHACqSu8Wg3/f7FJGrdunU2nkny8uP36FwICAEhIASEgBCI9tKUIwhoxCC8SLokoVHjRkZgXNPKc/6s29dCmDpFUbbuu/++cNddd5nrMiKxHX744QF3aOmIa+mInIX2DnKOeFnp5KF7skfA+5cQzitXrswqI+yAGQs+wWJTGecciej2wgsvWL7YaE+dOtXGBwnck0k/s5rAuIJI8yxlSoSAEBACQkAICIE/I1BWb0i30dy//f6GBPaPEAzIQl0CEeXjWkC/39P4zTlL0yyJQ3hvv/32UFlZmXAp5s+kOpIHQjhh7IIhSWgJRWZSoZa/a44z0cW++uqrpH1eV2n0IR/GFkcnoz//9HM45JBDAhrgLyIzBlyWHXnkkYnxlwnZpQ6sABDUAvFxYz/0JQSEgBAQAkJACGyHQFkRXlpuWrSwSzjooIPC22+/bWCkQxYgI05I/MjDfu7XIUwsM6PpJdrWzTffbMQJLVw64oSLukGGEC8jned1T24IgDXjgYAfaE/fe+89yzCdSZGX7GMhfuQaXhvw9DFgwIDw4IMPmv1u7969M3KTZ+M3qmPl+sqwrnKdTYrI28cN5xIhIASEgBAQAkJgewTKjvA6MUDT9v777wd86KL5RYuaizhpRmu8fPnysGnzJtPmEaBiypQpaRESDyCwePFic4+FOQP5ep1zqZ+ezRwBQg3PmzfP+oAx4n2caU5OltesWRPmz58f2rdvby7QINT4Wib8MOJkO1X+Xoc5s+eEnv/Z055h7PKsRAgIASEgBISAEEiOQIN/RpL8Uv1NhTSgacNcgDDD3bt3TzQ2F+LAs3hlwEa46V5NA5ufsAWFOO3ddG8j1rXlDyky4v3zL2Ha9Glh6OlDA/a/SG3PJCqtk7wi4Hjjngz3YpBSgn9ka1pCfnxwM0eefIieR//i8xcTB9zPce5lJ2uQ+47GYwia59NOO83u14QoGVpKEwJCQAgIASHwBwK7ROQvTadLfzxU6mc02YnF5MmTzXSgS5cuOUdKi+cLRjV/14Yb9/GBuMyZM8c2IfXp0yft52vLV+nZI+B9h8b+ySefNM8K+LuF9LoteCa5e36pnkl1DxMiJ7aPPPKIeQZhYhVPT5W3rgkBISAEhIAQKGcEys6kgc6G7Poyc79+/cK0adPMTy5ExtOzGRTkC2lBE+fkhWNdeTppwRSCzWpHH310NsXrmTwi4GMEkoudLaQXyda0wfPzseFjArteT+OeZOJjiWtE+UPbDNkl3UlwsueUJgSEgBAQAkJACFQjUJaEl6ZDFCAb2FEOGjQoTJo0KUEgnIxkM0ggLZhLOHnhmIqUuMaQIBPY+g4bNszupw6eRzb10DO5IwD+9EPPnj1tEyKrAQjpkM1MhXHgY8PHRINdGyTSkuXn5VDmO++8YwEx2PQmEQJCQAgIASEgBNJHoGwJLxARTABCgwsq3DtBOC09Iia5kF7LpI4viAxloDHcsmVLGD9+fLj44osTkdycENWRjS4XEIH4hAP/udhnv/baa1aik+ECFh9++7WaVFPWihUrjPAOP3O4FcnYidevkPVQ3kJACAgBISAESh2BsrThjXdaXIP2xhtvWFjg4cOHV2uA8X8bkeJddk2+1BzPJ5NzNMuWb0Rk0OwSsnjw4MEhH3bEmdRD96aHAOSSCQhH+qp169YWQIKnMUdg0pJv8umaf8rAi8OCBQvC5Zdfbhvd4te4LhECQkAICAEhIARSI1D2hNfhgfhCWvDasHDhQjMtaN68uV3OF6mBMFEOBAlBazd9+vQwatSo0K5du6w3RFlm+iooAk56KYQ+I0raGWecYZ4V4tr6XCtRc4w8++yzNikaPXq0lRWvR65l6XkhIASEgBAQAuWCgAhvrKedTHz44Ydh7ty5AU8J2G+6oFmDFGdibgAZ8o8T3e+//96WxtmgNmLEiNCsWTORXQe5iI8+PqgiGlf8JQ8cODB07tzZau1klfGRicbXCPNvUbS/yILBx8jatWvDjBkzQpMmTcxDBAVIs2sw60sICAEhIASEQMYIiPDGIHNiCmHBrpagA6tXrw59+/YNBx98cOzO6ohtJNRGbMiLa/HraAXffPNN0yL36tUrHHXUUQnTCSc62xWiH0WHQJz04g935syZ5s+ZACOYOrj4WOJ3fAz4dY7JxsimTZts3K1atcrMJtxHtMhuHDmdCwEhIASEgBDIDAER3iR4xUnNJ598ErDthZB269bNduu3aNEiyVPJkwg2AGn++OOPw6effmoupY455hjT6vJEvKzkOSi12BAwMhupY33T46JFi2wSw7jo2rVr6NChQ8CdWbqydevWAHlmjEB0CVaCazoClyAaI+kiqfuEgBAQAkJACCRHQIQ3OS5/IhmEhV26dKmR1qZNm1p4WKJjcU7kLKK2QUyIyrVx48aA2cJ3330XvvjiC9PwHX744eYNgvsRSBNSm/bPLuqreBGIuu/XyAzBzVvoe0xUMHNgrLRp08Y0vk32aBL23mdvI8BEUkNTW1VVZWMErw+EtmZCtPvuu1sAFCZVjRs3tnaTZ81VguIFRDUTAkJACAgBIVC8CIjwpuqbiNREFrh/IqWVlZUBG0s+ROLCVMHtexs0bGDkpdl/NAutWrYKrdu03k7bJ6KbCvDSu5asP9nkCOldt26dfZj8bNu2rdrNWOTwA+KLbS4+oDGD4MOEySVZnn5NRyEgBISAEBACQiBzBER408QMEsLHNXppPlZ92+/aQGnrMkKt5G5GI4tkM0ZyGl8lh5QqLASEgBAQAkJgxyIgwpsF3hAb08JF2rpoW1pi2dnSovycvJA1JDcbApRFtfRIkSBA/zv5dZMVn+wkGyOMD7+vSJqgaggBISAEhIAQqFcIiPDmqTudyIi45AnQepYNpjH8IRoj1TjoWwgIASEgBITAjkJAhHdHIa1yhIAQEAJCQAgIASEgBHYKAv8PeYUzZPoX3sQAAAAASUVORK5CYII=)\n","\n","`Function`'s are what `autograd` uses to compute the results and gradients, and encode the operation history. When defining a new `Function`, we must define static methods `forward()` which performs the operation and `backward()` which computes the gradient formulas with respect to the arguments to the forward function.\n","\n","**Forward pass**: For a linear layer, given an input $X \\in \\mathbb{R}^{N \\times D}$ and weights $W \\in \\mathbb{R}^{D \\times M}$, the forward pass computes the output $Y = XW$, where $Y \\in \\mathbb{R}^{N \\times M}$. Note that a bias can be represented by adding an additional dimension to $X$ containing 1's, and a corresponding additional dimension to the weights.\n","\n","<!-- For a linear layer, given an input $x \\in \\mathbb{R}^{D \\times 1}$, weights $W \\in \\mathbb{R}^{D \\times M}$, and bias $b \\in \\mathbb{R}^{M \\times 1}$, the forward pass computes the output $y = W^Tx + b$, where $y \\in \\mathbb{R}^{M \\times 1}$. -->\n","\n","**Backward pass**: During the backward pass, we assume that the upstream gradient $\\frac{\\partial L}{\\partial Y}$ has already been computed. Given this upstream gradient, we aim to calculate $\\frac{\\partial L}{\\partial X}$ and $\\frac{\\partial L}{\\partial W}$. Applying the chain rule, we get:\n","\n","$$\\frac{\\partial L}{\\partial X} = \\frac{\\partial L}{\\partial Y}W^{T}$$\n","\n","$$\\frac{\\partial L}{\\partial W} = X^{T}\\frac{\\partial L}{\\partial Y}$$\n","\n","These are the gradients returned by the `backward()` function (note that we can compute the gradient with respect to the bias by plugging in a vector of 1's for X in the last expression above).\n"]},{"cell_type":"code","metadata":{"id":"Cif0hFWQgpCL"},"source":["# Inherit from Function\n","class LinearFunction(Function):\n","    @staticmethod\n","    def forward(ctx, input, weight, bias=None):\n","        ctx.save_for_backward(input, weight, bias)  # save to help compute grad\n","        output = input.mm(weight.t())               # multiply w/ weight matrix\n","        if bias is not None:\n","            output += bias.unsqueeze(0).expand_as(output)\n","        return output\n","\n","    @staticmethod\n","    def backward(ctx, grad_output):\n","        input, weight, bias = ctx.saved_tensors\n","        grad_input = grad_weight = grad_bias = None\n","\n","        # These needs_input_grad checks are optional.\n","        # Returning gradients for inputs that don't require it is not an error.\n","        if ctx.needs_input_grad[0]:\n","            grad_input = grad_output.mm(weight)\n","        if ctx.needs_input_grad[1]:\n","            grad_weight = grad_output.t().mm(input)\n","        if bias is not None and ctx.needs_input_grad[2]:\n","            grad_bias = grad_output.sum(0)\n","\n","        return grad_input, grad_weight, grad_bias"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FjDZQvPIIRT1"},"source":["Given the `LinearFunction`, we can now define the `Linear` `nn.Module` that uses it."]},{"cell_type":"code","metadata":{"id":"PvYS11QOis4O"},"source":["class Linear(nn.Module):\n","    def __init__(self, input_features, output_features, bias=True):\n","        super(Linear, self).__init__()\n","        self.input_features = input_features\n","        self.output_features = output_features\n","\n","        # nn.Parameter is a special kind of Tensor, that will get\n","        # automatically registered as Module's parameter once it's assigned\n","        # as an attribute. Parameters and buffers need to be registered, or\n","        # they won't appear in .parameters() (doesn't apply to buffers), and\n","        # won't be converted when e.g. .cuda() is called. You can use\n","        # .register_buffer() to register buffers.\n","        # nn.Parameters require gradients by default.\n","        self.weight = nn.Parameter(torch.Tensor(output_features, input_features))\n","        if bias:\n","            self.bias = nn.Parameter(torch.Tensor(output_features))\n","        else:\n","            # You should always register all possible parameters, but the\n","            # optional ones can be None if you want.\n","            self.register_parameter('bias', None)\n","\n","        # Not a very smart way to initialize weights\n","        self.weight.data.uniform_(-0.1, 0.1)\n","        if self.bias is not None:\n","            self.bias.data.uniform_(-0.1, 0.1)\n","\n","    def forward(self, input):\n","        # See the autograd section for explanation of what happens here.\n","        return LinearFunction.apply(input, self.weight, self.bias)\n","\n","    def extra_repr(self):\n","        # (Optional)Set the extra information about this module. You can test\n","        # it by printing an object of this class.\n","        return 'input_features={}, output_features={}, bias={}'.format(\n","            self.input_features, self.output_features, self.bias is not None\n","        )"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ZBzNg800KeDe"},"source":["Composing linear layers together, we can form an MLP."]},{"cell_type":"code","metadata":{"id":"3e1fKgOWg7vQ"},"source":["# Multi-layer perceptron class using custom Linear module\n","class MLP(nn.Module):\n","    def __init__(self, input_size, hidden_size, output_size):\n","        super(MLP, self).__init__()\n","        self.linear_layer = Linear(input_size, hidden_size)\n","        self.activation = nn.Sigmoid()\n","        self.output_layer = Linear(hidden_size, output_size)\n","\n","    def forward(self, x):\n","        return self.output_layer(self.activation(self.linear_layer(x)))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"iB0QYby8FsCu"},"source":["## Evaluation metric\n","\n","Since we have binary labels, we can compute accuracy as our metric of interest."]},{"cell_type":"code","metadata":{"id":"3L-I2eHA5Yox"},"source":["# Define evaluation metric\n","def binary_accuracy(preds, y):\n","    \"\"\"\n","    Returns accuracy per batch\n","    \"\"\"\n","    rounded_preds = torch.argmax(nn.functional.softmax(preds, dim=1), dim=1)\n","    correct = (rounded_preds == y).float()\n","    acc = correct.sum() / len(correct)\n","    return acc"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"01M2gpL0MDdx"},"source":["## Advanced: Custom loss function"]},{"cell_type":"markdown","metadata":{"id":"4V1Dqjww3gpQ"},"source":["Sometimes, the loss function you want to use may not already be implemented in Pytorch. To implement such as loss based on a function that does not already exist in pytorch, you can first implement a `Function` and then define a `nn.Module` loss based on that Function. Here's an example implementation for the binary cross-entropy loss function.\n","\n","**Forward pass:** Given a set of $N$ datapoints, with corresponding binary class indicators, $\\{y_i\\}_{i=1}^N$, where each $y_i \\in \\{0,1\\}$, and a set of predictions $p_i = P(y_i=1)$ that represent the probability that the $i^\\text{th}$ sample belongs to class $y=1$, the binary cross entropy is given by,\n","\n","$$ L = \\frac{1}{N} \\sum_{i=1}^N \\left( y_i\\log(p_i) + (1-y_i) \\log (1-p_i) \\right)$$\n","\n","**Backward pass:** We would like to compute the gradients of the loss with respect to the inputs to the loss functions. Generally, there are no upstream gradients for the loss, but in keeping with convention, these are passed to the `backward` method as `grad_output`. The gradient of the loss with respect to the $p_i$ (`input`) is given by\n","\n","$$ \\frac{\\partial L}{\\partial p_i} = \\frac{y_i - p_i}{p_i(1-p_i)} $$\n","The binary class indicators (`target`) are discrete variables, and the gradient of the loss with respect to these indicators is not defined. Therefore, we return a `None` object for this gradient."]},{"cell_type":"code","metadata":{"id":"nDC4It9hHKJG"},"source":["class CrossEntropyFunction(Function):\n","    @staticmethod\n","    def forward(ctx, input, target):\n","        ctx.save_for_backward(input, target)      # save to help compute grad\n","        return -(target*torch.log(input)).sum(axis=1).mean()\n","\n","    @staticmethod\n","    def backward(ctx, grad_output):\n","        input_y, target_y = ctx.saved_tensors\n","        grad_input = grad_target = None\n","\n","        grad_input = grad_output * (-target_y/input_y)\n","        return grad_input, grad_target\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"kBKkHy6rGX48"},"source":["class CrossEntropyLoss(nn.Module):\n","    def __init__(self):\n","        super(Linear, self).__init__()\n","\n","    def forward(self, prediction, target):\n","        softmax_preds = nn.Softmax(dim=1)(prediction)\n","        return CrossEntropyFunction.apply(softmax_preds,target)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Initiate the model"],"metadata":{"id":"Zo6ubKwY3Uqk"}},{"cell_type":"code","source":["INPUT_SIZE = 784\n","HIDDEN_SIZE = 200\n","OUTPUT_SIZE = 10\n","EPOCHS = 10\n","LEARNING_RATE = 0.01\n","\n","device = 'cuda' if torch.cuda.is_available() else 'cpu'\n","model = MLP(INPUT_SIZE, HIDDEN_SIZE, OUTPUT_SIZE).to(device)"],"metadata":{"id":"Uvw_h3-g2ntj"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"OzVp0zAE3rBX"},"source":["##Training the Classifier\n","\n","Now, let's put everything together and train the classifier."]},{"cell_type":"code","metadata":{"id":"xGOt4YSY5Yoz"},"source":["# Procedures for training and evaluation\n","\n","def train(model, dataloader, n_epochs, learning_rate):\n","    # Tells the model we are in training mode.\n","    # Only needed for certain layers (e.g. batchnorm, dropout)\n","    model.train()\n","\n","    # You can pick from a variety of loss functions, or write your own.\n","    loss_fn = nn.CrossEntropyLoss()\n","\n","    # Defining the optimizer here. Other options include Adam, etc.\n","    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n","\n","    best_acc = 0.0\n","\n","    for epoch in range(n_epochs):\n","        epoch_loss = 0.0\n","        epoch_acc = 0.0\n","\n","        num_batches = len(dataloader)\n","\n","        for i, (feature, label) in enumerate(dataloader):\n","            # Clear the gradients from previous iteration.\n","            # (Forgetting this is a common mistake!)\n","            optimizer.zero_grad()\n","\n","            # Move data to the corresponding device\n","            feature, label = feature.to(device), label.to(device)\n","\n","            # Forward pass of the network. It will return the predictions.\n","            predictions = model(feature.view(-1, 784))\n","\n","            # Calculate the loss of the predictions with respect to the actual labels.\n","            loss = loss_fn(predictions, label)\n","            acc = binary_accuracy(predictions, label)\n","\n","            # This step is the backward pass.\n","            loss.backward()\n","\n","            # This step is calling the optimizer to do autodiff.\n","            optimizer.step()\n","\n","            # Adding the loss of current batch to the sum of loss of this epoch.\n","            epoch_loss += loss.item()\n","            epoch_acc += acc.item()\n","\n","        # Divide over batch (since we added every batch)\n","        epoch_loss /= num_batches\n","        epoch_acc /= num_batches\n","\n","        print(\"Epoch {}: Loss: {}, Accuracy: {}\".format(epoch,\n","                                                        epoch_loss,\n","                                                        epoch_acc))\n","\n","\n","def evaluate(model, dataloader):\n","    total_acc = 0\n","    model.eval()\n","\n","    num_batches = len(dataloader)\n","\n","    with torch.no_grad():\n","        for i, (feature, label) in enumerate(dataloader):\n","            feature, label = feature.to(device), label.to(device)\n","            predictions = model(feature.view(-1, 784))\n","            acc = binary_accuracy(predictions, label)\n","            total_acc += acc.item()\n","\n","    return total_acc / num_batches"],"execution_count":null,"outputs":[]},{"cell_type":"code","source":["train(model, train_dataloader, EPOCHS, LEARNING_RATE)\n","test_acc = evaluate(model, test_dataloader)\n","print('Test Set Accuracy {}'.format(test_acc))"],"metadata":{"id":"E4w7MH28Vfej"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Counting model parameters\n","\n","Suppose we want to know how many trainable parameters are in our model. PyTorch doesn't have a default function to do so, but we can achieve this with the one-liner below:"],"metadata":{"id":"T_sBdSGkyycw"}},{"cell_type":"code","source":["total_params = sum(\n","\tparam.numel() for param in model.parameters()\n",")\n","\n","print(f\"Total number of model parameters: {total_params}\")"],"metadata":{"id":"unA1YI0My9G6"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["If you want just the trainable parameters, we can do the following (in this case it is the same):"],"metadata":{"id":"kLGFLjiHzU2C"}},{"cell_type":"code","source":["trainable_params = sum(\n","\tp.numel() for p in model.parameters() if p.requires_grad\n",")\n","\n","print(f\"Total number of trainable model parameters: {trainable_params}\")"],"metadata":{"id":"Pq6WW-MvzYk1"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["[Source](https://wandb.ai/wandb_fc/tips/reports/How-To-Calculate-Number-of-Model-Parameters-for-PyTorch-and-TensorFlow-Models--VmlldzoyMDYyNzIx#pytorch)"],"metadata":{"id":"RDvJdeYyzdnK"}},{"cell_type":"markdown","source":["# LSTM Basics"],"metadata":{"id":"Aac8QM01Rhqh"}},{"cell_type":"code","source":["import torch\n","import torch.nn as nn\n","\n","# Define an LSTM model\n","\n","'''\n","Input Shape: [batch_size, sequence_length, input_size]\n","batch_size: The number of sequences in a batch.\n","sequence_length: The number of time steps in each sequence.\n","input_size: The number of features at each time step.\n","'''\n","\n","class LSTMModel(nn.Module):\n","    def __init__(self, input_size, hidden_size, output_size):\n","        '''\n","        input_size: The number of features in the input at each time step.\n","        hidden_size: The number of features in the hidden state of the LSTM.\n","        output_size: The number of features in the output (e.g., the number of classes for classification).\n","        '''\n","        super(LSTMModel, self).__init__()\n","        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)\n","        self.fc = nn.Linear(hidden_size, output_size)\n","\n","    def forward(self, x):\n","        # x: input sequence\n","        '''\n","        Hidden and Cell Shape: [num_layers, batch_size, hidden_size]\n","        num_layers: The number of LSTM layers (here it's 1).\n","        batch_size: The number of sequences in a batch (same as x.size(0)).\n","        hidden_size: The number of features in the hidden state.\n","        '''\n","        h0 = torch.zeros(1, x.size(0), hidden_size).to(x.device)  # Initial hidden state\n","        c0 = torch.zeros(1, x.size(0), hidden_size).to(x.device)  # Initial cell state\n","\n","\n","        ''' for lstm\n","        Input Shape: [batch_size, sequence_length, input_size]\n","        Output Shape (out): [batch_size, sequence_length, hidden_size]\n","        The LSTM processes each sequence step-by-step, and for each time step, it produces a hidden state with hidden_size features.\n","        '''\n","        out, _ = self.lstm(x, (h0, c0))  # Forward pass through LSTM\n","\n","\n","        '''\n","        out[:, -1, :] Shape: [batch_size, hidden_size]\n","        out[:, -1, :] selects the hidden state from the last time step of the sequence for each batch.\n","        '''\n","        ''' for fc\n","        Input Shape: [batch_size, hidden_size]\n","        Output Shape: [batch_size, output_size]\n","        This layer maps the hidden_size features from the LSTM's last hidden state to output_size features, which is the final prediction.\n","        '''\n","        out = self.fc(out[:, -1, :])     # Take output of last time step\n","\n","\n","        return out\n","\n","# Hyperparameters\n","input_size = 10\n","hidden_size = 20\n","output_size = 1\n","seq_length = 5\n","batch_size = 3\n","\n","# Create LSTM model\n","model = LSTMModel(input_size, hidden_size, output_size)\n","\n","# Dummy input\n","x = torch.randn(batch_size, seq_length, input_size)\n","\n","# Forward pass\n","output = model(x)\n","print(output)\n"],"metadata":{"id":"NFlKAsEk9sfn","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1725040797355,"user_tz":240,"elapsed":14683,"user":{"displayName":"Ketan Ramaneti","userId":"08035029517799240380"}},"outputId":"44cca428-5336-4779-8574-914644b33830"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["tensor([[-0.2099],\n","        [-0.2826],\n","        [-0.1841]], grad_fn=<AddmmBackward0>)\n"]}]},{"cell_type":"markdown","source":["# Weights and Biases\n"],"metadata":{"id":"VmZFpLLbRkcZ"}},{"cell_type":"markdown","source":["## WandB Basics"],"metadata":{"id":"kksIt1stzkWj"}},{"cell_type":"code","source":["# Install wandb\n","!pip install wandb -q"],"metadata":{"id":"cBpkDKdttcr2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["import wandb\n","\n","# Log in to your W&B account\n","# You can find your API key here: https://wandb.ai/authorize\n","wandb_api = \"a9a931737a1301f6a050fef1b00d44bb659b9b3b\"\n","wandb.login(key=wandb_api)\n","\n","# Start a new run with a custom name\n","wandb.init(project=\"GenAI_HW0_test\", name=\"run1\")\n","# wandb.log({\"learning_rate\":0.001})\n","\n","# Your training code here\n","\n","wandb.finish()"],"metadata":{"id":"F8lRIMNLW-XI","colab":{"base_uri":"https://localhost:8080/","height":225},"executionInfo":{"status":"ok","timestamp":1756488343998,"user_tz":240,"elapsed":3105,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"6e21c827-2a9f-49b7-ed3e-b5922e6a7885"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m If you're specifying your api key in code, ensure this code is not shared publicly.\n","\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Consider setting the WANDB_API_KEY environment variable, or running `wandb login` from the command line.\n","\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n"]},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_172541-cy5y83n8</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/GenAI_HW0_test/runs/cy5y83n8' target=\"_blank\">run1</a></strong> to <a href='https://wandb.ai/gaminghero812/GenAI_HW0_test' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/GenAI_HW0_test' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0_test</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/GenAI_HW0_test/runs/cy5y83n8' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0_test/runs/cy5y83n8</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">run1</strong> at: <a href='https://wandb.ai/gaminghero812/GenAI_HW0_test/runs/cy5y83n8' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0_test/runs/cy5y83n8</a><br> View project at: <a href='https://wandb.ai/gaminghero812/GenAI_HW0_test' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0_test</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_172541-cy5y83n8/logs</code>"]},"metadata":{}}]},{"cell_type":"code","source":["# Changing the run name after initialization\n","\n","# Start a new run without a custom name\n","run = wandb.init(project=\"GenAI_HW0\")\n","\n","# Change the name of the run\n","run.name = \"run2\"\n","\n","# Your training code here\n","\n","wandb.finish()"],"metadata":{"id":"xq1k9Sz5W-U3","colab":{"base_uri":"https://localhost:8080/","height":173},"executionInfo":{"status":"ok","timestamp":1756488589996,"user_tz":240,"elapsed":1505,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"fdbb8e3e-90ae-46d7-f147-3736f9a3fdad"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_172948-l6ocmi6b</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/l6ocmi6b' target=\"_blank\">feasible-snowball-10</a></strong> to <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/l6ocmi6b' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0/runs/l6ocmi6b</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">run2</strong> at: <a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/l6ocmi6b' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0/runs/l6ocmi6b</a><br> View project at: <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_172948-l6ocmi6b/logs</code>"]},"metadata":{}}]},{"cell_type":"code","source":["# Multiple runs\n","\n","# List of different configurations or hyperparameters\n","# Useful to have this in a dictionary passed into init so you can access run-specific hyperparameters\n","configs = [\n","    {\"learning_rate\": 0.01, \"batch_size\": 32, \"run_name\": \"run_lr_0.01_bs_32\"},\n","    {\"learning_rate\": 0.001, \"batch_size\": 64, \"run_name\": \"run_lr_0.001_bs_64\"},\n","    # Add more configurations as needed\n","]\n","\n","for config in configs:\n","    # Start a new run with a custom name for each configuration\n","    wandb.init(project=\"GenAI_HW0\", name=config[\"run_name\"], config=config)\n","\n","    # Simulate training code\n","    # e.g., train your model with the specified config['learning_rate'] and config['batch_size']\n","\n","    wandb.log({\"accuracy\": 0.9, \"loss\": 0.02})  # Example of logging a metric\n","\n","    wandb.finish()  # Finish the current run"],"metadata":{"id":"QIGC90UyW-R2","colab":{"base_uri":"https://localhost:8080/","height":875},"executionInfo":{"status":"ok","timestamp":1756488784533,"user_tz":240,"elapsed":3120,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"8ad74539-2978-4983-f79f-3bbb372ef2a0"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_173301-qkg5syx2</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/qkg5syx2' target=\"_blank\">run_lr_0.01_bs_32</a></strong> to <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/qkg5syx2' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0/runs/qkg5syx2</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["<br>    <style><br>        .wandb-row {<br>            display: flex;<br>            flex-direction: row;<br>            flex-wrap: wrap;<br>            justify-content: flex-start;<br>            width: 100%;<br>        }<br>        .wandb-col {<br>            display: flex;<br>            flex-direction: column;<br>            flex-basis: 100%;<br>            flex: 1;<br>            padding: 10px;<br>        }<br>    </style><br><div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>accuracy</td><td>▁</td></tr><tr><td>loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>accuracy</td><td>0.9</td></tr><tr><td>loss</td><td>0.02</td></tr></table><br/></div></div>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">run_lr_0.01_bs_32</strong> at: <a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/qkg5syx2' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0/runs/qkg5syx2</a><br> View project at: <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_173301-qkg5syx2/logs</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_173302-drhqwd7t</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/drhqwd7t' target=\"_blank\">run_lr_0.001_bs_64</a></strong> to <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/drhqwd7t' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0/runs/drhqwd7t</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["<br>    <style><br>        .wandb-row {<br>            display: flex;<br>            flex-direction: row;<br>            flex-wrap: wrap;<br>            justify-content: flex-start;<br>            width: 100%;<br>        }<br>        .wandb-col {<br>            display: flex;<br>            flex-direction: column;<br>            flex-basis: 100%;<br>            flex: 1;<br>            padding: 10px;<br>        }<br>    </style><br><div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>accuracy</td><td>▁</td></tr><tr><td>loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>accuracy</td><td>0.9</td></tr><tr><td>loss</td><td>0.02</td></tr></table><br/></div></div>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">run_lr_0.001_bs_64</strong> at: <a href='https://wandb.ai/gaminghero812/GenAI_HW0/runs/drhqwd7t' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0/runs/drhqwd7t</a><br> View project at: <a href='https://wandb.ai/gaminghero812/GenAI_HW0' target=\"_blank\">https://wandb.ai/gaminghero812/GenAI_HW0</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_173302-drhqwd7t/logs</code>"]},"metadata":{}}]},{"cell_type":"markdown","source":["## WandB Basics Examples"],"metadata":{"id":"dYp-5hPctiUj"}},{"cell_type":"markdown","source":["### Training loop with wandb"],"metadata":{"id":"ifjO6bcWuA7A"}},{"cell_type":"code","source":["# Install wandb\n","!pip install wandb -q\n","import wandb"],"metadata":{"id":"L1Q0YF_ttp5w"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Log in to your W&B account\n","# You can find your API key here: https://wandb.ai/authorize\n","wandb_api = \"\"\n","wandb.login(key=wandb_api)"],"metadata":{"id":"XEgOlqatt1On","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1756486900385,"user_tz":240,"elapsed":1008,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"4cb914b9-d5c0-49d9-aa3c-2a18eb765a17"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["/usr/local/lib/python3.12/dist-packages/notebook/notebookapp.py:191: SyntaxWarning: invalid escape sequence '\\/'\n","  | |_| | '_ \\/ _` / _` |  _/ -_)\n","\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m If you're specifying your api key in code, ensure this code is not shared publicly.\n","\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Consider setting the WANDB_API_KEY environment variable, or running `wandb login` from the command line.\n","\u001b[34m\u001b[1mwandb\u001b[0m: No netrc file found, creating one.\n","\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n","\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mgaminghero812\u001b[0m to \u001b[32mhttps://api.wandb.ai\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"]},{"output_type":"execute_result","data":{"text/plain":["True"]},"metadata":{},"execution_count":4}]},{"cell_type":"code","source":["# Define a train function which uses \"wandb.log\"\n","\n","def train(model, dataloader, n_epochs, learning_rate):\n","    model.train()\n","    loss_fn = nn.CrossEntropyLoss()\n","    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n","\n","    best_acc = 0.0\n","\n","    for epoch in range(n_epochs):\n","        epoch_loss = 0.0\n","        epoch_acc = 0.0\n","\n","        num_batches = len(dataloader)\n","\n","        for i, (feature, label) in enumerate(dataloader):\n","            optimizer.zero_grad()\n","            feature, label = feature.to(device), label.to(device)\n","            predictions = model(feature.view(-1, 784))\n","            loss = loss_fn(predictions, label)\n","            acc = binary_accuracy(predictions, label)\n","            loss.backward()\n","            optimizer.step()\n","\n","            epoch_loss += loss.item()\n","            epoch_acc += acc.item()\n","\n","        epoch_loss /= num_batches\n","        epoch_acc /= num_batches\n","\n","        print(\"Epoch {}: Loss: {}, Accuracy: {}\".format(epoch,\n","                                                        epoch_loss,\n","                                                        epoch_acc))\n","\n","        # Note: this is the only line of code which has changed!\n","        # Each call to wandb.log increments the global \"step\" variable\n","        # Metrics are tracked as Loss vs step and Accuracy vs step\n","        wandb.log({\"Loss\": epoch_loss, \"Accuracy\": epoch_acc})"],"metadata":{"id":"wRR15bP_ttES"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["wandb.init(\n","    name = \"model training example\",\n","    reinit = True,\n","    project = \"10-423-623 Recitation 0\",\n",")\n","# Use the model and dataloader we defined above\n","model = MLP(INPUT_SIZE, HIDDEN_SIZE, OUTPUT_SIZE).to(device)\n","\n","train(model, train_dataloader, EPOCHS, LEARNING_RATE)\n","wandb.finish()"],"metadata":{"id":"CpZqBNmItugL","colab":{"base_uri":"https://localhost:8080/","height":620},"executionInfo":{"status":"ok","timestamp":1756489109085,"user_tz":240,"elapsed":105250,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"475c24de-dea5-4413-8dd1-ce2a27290c47"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_173643-dijej21j</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/dijej21j' target=\"_blank\">model training example</a></strong> to <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/dijej21j' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/dijej21j</a>"]},"metadata":{}},{"output_type":"stream","name":"stdout","text":["Epoch 0: Loss: 1.7087802430714116, Accuracy: 0.5697461353944563\n","Epoch 1: Loss: 1.0631483935598116, Accuracy: 0.7144189765458422\n","Epoch 2: Loss: 0.8540157215681665, Accuracy: 0.7374067164179104\n","Epoch 3: Loss: 0.7550320436260594, Accuracy: 0.7507496002132196\n","Epoch 4: Loss: 0.6970862852675574, Accuracy: 0.7625099946695096\n","Epoch 5: Loss: 0.6574420073329766, Accuracy: 0.7716717750533049\n","Epoch 6: Loss: 0.6279020991597348, Accuracy: 0.7804171108742004\n","Epoch 7: Loss: 0.6041985510318264, Accuracy: 0.7872634594882729\n","Epoch 8: Loss: 0.5845946877686454, Accuracy: 0.7954590884861408\n","Epoch 9: Loss: 0.5678035661355773, Accuracy: 0.8011560501066098\n"]},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["<br>    <style><br>        .wandb-row {<br>            display: flex;<br>            flex-direction: row;<br>            flex-wrap: wrap;<br>            justify-content: flex-start;<br>            width: 100%;<br>        }<br>        .wandb-col {<br>            display: flex;<br>            flex-direction: column;<br>            flex-basis: 100%;<br>            flex: 1;<br>            padding: 10px;<br>        }<br>    </style><br><div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>Accuracy</td><td>▁▅▆▆▇▇▇███</td></tr><tr><td>Loss</td><td>█▄▃▂▂▂▁▁▁▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>Accuracy</td><td>0.80116</td></tr><tr><td>Loss</td><td>0.5678</td></tr></table><br/></div></div>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">model training example</strong> at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/dijej21j' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/dijej21j</a><br> View project at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_173643-dijej21j/logs</code>"]},"metadata":{}}]},{"cell_type":"markdown","source":["The result will look as below on the wandb web platform\n","\n","![wandb interface](https://drive.usercontent.google.com/download?id=1u_QBxmJDKHa74CFgprSOS31BLnfA7sZG)"],"metadata":{"id":"smym2UNbt6Us"}},{"cell_type":"markdown","source":["### Logging Hyperparameters with wandb\n","As seen in the example above (multiple runs) you can also save the hyperparameters used in a training run to wandb for later reference. This is optional but highly recommended"],"metadata":{"id":"aYYqYLLZuEnm"}},{"cell_type":"code","source":["wandb.init(\n","    #name = \"\", # If we don't enter a name, a random (useless) name will be assigned\n","    config={\n","        \"learning_rate\": LEARNING_RATE,\n","        \"epochs\": EPOCHS,\n","    }, # We can use the config dict to store hyperparamter details\n","    reinit = True,\n","    project = \"10-423-623 Recitation 0\",\n",")\n","\n","# Same training loop here\n","wandb.log({\"accuracy\": 0.9, \"loss\": 0.02}) # Example of logging during training\n","\n","wandb.finish()"],"metadata":{"id":"0Ej_D5-iuOUC","colab":{"base_uri":"https://localhost:8080/","height":533},"executionInfo":{"status":"ok","timestamp":1756489002444,"user_tz":240,"elapsed":2033,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"ce269fed-c8cb-4f83-cdda-1f157451aa72"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Finishing previous runs because reinit is set to True."]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">model training example</strong> at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/1ebh6x6i' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/1ebh6x6i</a><br> View project at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_173557-1ebh6x6i/logs</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_173640-upw3sw4y</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/upw3sw4y' target=\"_blank\">fast-dream-6</a></strong> to <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/upw3sw4y' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/upw3sw4y</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["<br>    <style><br>        .wandb-row {<br>            display: flex;<br>            flex-direction: row;<br>            flex-wrap: wrap;<br>            justify-content: flex-start;<br>            width: 100%;<br>        }<br>        .wandb-col {<br>            display: flex;<br>            flex-direction: column;<br>            flex-basis: 100%;<br>            flex: 1;<br>            padding: 10px;<br>        }<br>    </style><br><div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>accuracy</td><td>▁</td></tr><tr><td>loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>accuracy</td><td>0.9</td></tr><tr><td>loss</td><td>0.02</td></tr></table><br/></div></div>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">fast-dream-6</strong> at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/upw3sw4y' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/upw3sw4y</a><br> View project at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_173640-upw3sw4y/logs</code>"]},"metadata":{}}]},{"cell_type":"markdown","source":["![File in cloud](https://drive.usercontent.google.com/download?id=1n_yETpPj1Dhdc-feNnFa_q_BFB8yBJ6Y)"],"metadata":{"id":"vi6O-eNWuVGG"}},{"cell_type":"markdown","source":["### Displaying a plot with a legend in wandb\n","\n","After running multiple training runs with different hyperparameters, you may wish to visualize the results."],"metadata":{"id":"3Ku36AMqunqJ"}},{"cell_type":"code","source":["configs = [\n","    {\"learning_rate\": LEARNING_RATE, \"run_name\": f\"{LEARNING_RATE}\", \"epochs\": EPOCHS},\n","    {\"learning_rate\": LEARNING_RATE * 2, \"run_name\": f\"{LEARNING_RATE * 2}\", \"epochs\": EPOCHS},\n","    # Add more configurations as needed\n","]\n","\n","for config in configs:\n","    # Start a new run with a custom name for each configuration\n","    wandb.init(project=\"10-423-623 Recitation 0\", name=config[\"run_name\"], config=config)\n","\n","    model = MLP(INPUT_SIZE, HIDDEN_SIZE, OUTPUT_SIZE).to(device)\n","\n","    train(model, train_dataloader, config['epochs'], config['learning_rate'])\n","    wandb.finish()"],"metadata":{"id":"_3vquTcOu5mA","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"error","timestamp":1756489397150,"user_tz":240,"elapsed":181657,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"f169a11f-b96d-471b-9f62-36d005f6e066"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_174015-gdoe4frw</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/gdoe4frw' target=\"_blank\">0.01</a></strong> to <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/gdoe4frw' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/gdoe4frw</a>"]},"metadata":{}},{"output_type":"stream","name":"stdout","text":["Epoch 0: Loss: 1.6742552121056677, Accuracy: 0.5894356343283582\n","Epoch 1: Loss: 1.04148855703726, Accuracy: 0.7194662846481876\n","Epoch 2: Loss: 0.8392756873610686, Accuracy: 0.7381896321961621\n","Epoch 3: Loss: 0.7467035868846531, Accuracy: 0.7516824360341151\n","Epoch 4: Loss: 0.6926768926669286, Accuracy: 0.7613272921108742\n","Epoch 5: Loss: 0.656218036183162, Accuracy: 0.7707222814498934\n","Epoch 6: Loss: 0.6285045695965732, Accuracy: 0.7790011993603412\n","Epoch 7: Loss: 0.6061207796020041, Accuracy: 0.7855477078891258\n","Epoch 8: Loss: 0.5873620561254558, Accuracy: 0.7932436034115139\n","Epoch 9: Loss: 0.571652545476519, Accuracy: 0.7994236407249466\n"]},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["<br>    <style><br>        .wandb-row {<br>            display: flex;<br>            flex-direction: row;<br>            flex-wrap: wrap;<br>            justify-content: flex-start;<br>            width: 100%;<br>        }<br>        .wandb-col {<br>            display: flex;<br>            flex-direction: column;<br>            flex-basis: 100%;<br>            flex: 1;<br>            padding: 10px;<br>        }<br>    </style><br><div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>Accuracy</td><td>▁▅▆▆▇▇▇███</td></tr><tr><td>Loss</td><td>█▄▃▂▂▂▁▁▁▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>Accuracy</td><td>0.79942</td></tr><tr><td>Loss</td><td>0.57165</td></tr></table><br/></div></div>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">0.01</strong> at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/gdoe4frw' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/gdoe4frw</a><br> View project at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_174015-gdoe4frw/logs</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_174201-jcqyqivz</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/jcqyqivz' target=\"_blank\">0.02</a></strong> to <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/jcqyqivz' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/jcqyqivz</a>"]},"metadata":{}},{"output_type":"stream","name":"stdout","text":["Epoch 0: Loss: 1.3869858713927807, Accuracy: 0.6338952558635395\n","Epoch 1: Loss: 0.8019711408279598, Accuracy: 0.7419043176972282\n","Epoch 2: Loss: 0.6773381477225818, Accuracy: 0.7667910447761194\n","Epoch 3: Loss: 0.6167526525983424, Accuracy: 0.7847481343283582\n","Epoch 4: Loss: 0.5774095805405554, Accuracy: 0.7980077292110874\n","Epoch 5: Loss: 0.549505368637632, Accuracy: 0.8083855277185501\n","Epoch 6: Loss: 0.5282550311641399, Accuracy: 0.8160814232409381\n"]},{"output_type":"error","ename":"KeyboardInterrupt","evalue":"","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)","\u001b[0;32m/tmp/ipython-input-3426039710.py\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mMLP\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mINPUT_SIZE\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mHIDDEN_SIZE\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mOUTPUT_SIZE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m     \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_dataloader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'epochs'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'learning_rate'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m     \u001b[0mwandb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfinish\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/tmp/ipython-input-2258995059.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(model, dataloader, n_epochs, learning_rate)\u001b[0m\n\u001b[1;32m     14\u001b[0m         \u001b[0mnum_batches\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataloader\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mfeature\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataloader\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m             \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m             \u001b[0mfeature\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeature\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py\u001b[0m in \u001b[0;36m__next__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    732\u001b[0m                 \u001b[0;31m# TODO(https://github.com/pytorch/pytorch/issues/76750)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    733\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[call-arg]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 734\u001b[0;31m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_next_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    735\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_yielded\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    736\u001b[0m             if (\n","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py\u001b[0m in \u001b[0;36m_next_data\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    788\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_next_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    789\u001b[0m         \u001b[0mindex\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_next_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# may raise StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 790\u001b[0;31m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dataset_fetcher\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfetch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# may raise StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    791\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_pin_memory\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    792\u001b[0m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpin_memory\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpin_memory\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_pin_memory_device\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/fetch.py\u001b[0m in \u001b[0;36mfetch\u001b[0;34m(self, possibly_batched_index)\u001b[0m\n\u001b[1;32m     50\u001b[0m                 \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getitems__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     51\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 52\u001b[0;31m                 \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midx\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0midx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     53\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/torchvision/datasets/mnist.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, index)\u001b[0m\n\u001b[1;32m    144\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    145\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m             \u001b[0mimg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mimg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtarget_transform\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/torchvision/transforms/transforms.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, pic)\u001b[0m\n\u001b[1;32m    135\u001b[0m             \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mConverted\u001b[0m \u001b[0mimage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m         \"\"\"\n\u001b[0;32m--> 137\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpic\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    138\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    139\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__repr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/torchvision/transforms/functional.py\u001b[0m in \u001b[0;36mto_tensor\u001b[0;34m(pic)\u001b[0m\n\u001b[1;32m    174\u001b[0m     \u001b[0mimg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mimg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpermute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontiguous\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    175\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mimg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mByteTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 176\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mimg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdefault_float_dtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdiv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m255\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    177\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    178\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mimg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mKeyboardInterrupt\u001b[0m: "]}]},{"cell_type":"markdown","source":["Now that the project has multiple runs (with different hyperparameters) we can enable/disable visualizing individual training curves in wanb from the project page\n","\n","![Result](https://drive.usercontent.google.com/download?id=1UDpmubhPCnzatekHdhC9_7TGaQQ881fy)"],"metadata":{"id":"9rayXjo-vxoV"}},{"cell_type":"markdown","source":["## WandB Additional Features"],"metadata":{"id":"IIvtyPi7tmfM"}},{"cell_type":"markdown","source":["### Logging your code to wandb\n","\n","Wandb allows academic users to save up to 100GB of free files. We recommend you use this space to automatically save the code used for every training run, allowing you to recreate any hyperparamter configuration exactly OR to recover files in case of disaster.\n","\n","Follow the following steps to enable automatic code backup:\n","\n","(or check out the documentation: https://docs.wandb.ai/guides/app/features/panels/code):"],"metadata":{"id":"9rLSetwUv8_x"}},{"cell_type":"markdown","source":["Step 1: On your wandb homepage, click on your team name on the left sidebar\n","\n","![Step 1](https://drive.usercontent.google.com/download?id=1m_a7KWrA3asePNG9eUQhKuLgUA-M2ikH)"],"metadata":{"id":"EHd80NOowE6S"}},{"cell_type":"markdown","source":["Step 2: Click on \"Team Settings\"\n","\n","\n","![Step 2](https://drive.usercontent.google.com/download?id=1lUjlrELVbNs-p8G8-tyLKCoqpPkVqdD4)"],"metadata":{"id":"Rn6lkq0awFWa"}},{"cell_type":"markdown","source":["Step 3: Toggle on \"Enable code saving by default\"\n","\n","\n","![Step 3](https://drive.usercontent.google.com/download?id=1wyzHxQ74zCluLZDvINeF8BPr-FRf33Bf)"],"metadata":{"id":"SGZCbuSYwHmJ"}},{"cell_type":"markdown","source":["Result: Now any wandb run (including those from a .jupyter notebook) will automatically save the code used to generate the data shown\n","\n","\n","![Result](https://drive.usercontent.google.com/download?id=1vn8HfullIyZrTlLi2niB9E4TiEla1FcZ)"],"metadata":{"id":"4J5IrI_jwSNa"}},{"cell_type":"markdown","source":["### Logging files to wandb\n","\n","WandB allows for you to log any file you would like to the cloud. For more details beyond this simple example, please see the documentation: https://docs.wandb.ai/ref/python/save"],"metadata":{"id":"HDSttWMYwXCQ"}},{"cell_type":"code","source":["run = wandb.init(\n","    name = \"save file demo\",\n","    reinit = True,\n","    project = \"10-423-623 Recitation 0\",\n",")"],"metadata":{"id":"0N121GYTwaTF","colab":{"base_uri":"https://localhost:8080/","height":464},"executionInfo":{"status":"ok","timestamp":1756489564578,"user_tz":240,"elapsed":1412,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"3a594b0a-e4d3-4ca4-9611-2a10f8cd1e0e"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Finishing previous runs because reinit is set to True."]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["<br>    <style><br>        .wandb-row {<br>            display: flex;<br>            flex-direction: row;<br>            flex-wrap: wrap;<br>            justify-content: flex-start;<br>            width: 100%;<br>        }<br>        .wandb-col {<br>            display: flex;<br>            flex-direction: column;<br>            flex-basis: 100%;<br>            flex: 1;<br>            padding: 10px;<br>        }<br>    </style><br><div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>Accuracy</td><td>▁▅▆▇▇██</td></tr><tr><td>Loss</td><td>█▃▂▂▁▁▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>Accuracy</td><td>0.81608</td></tr><tr><td>Loss</td><td>0.52826</td></tr></table><br/></div></div>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">0.02</strong> at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/jcqyqivz' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/jcqyqivz</a><br> View project at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_174201-jcqyqivz/logs</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_174603-agtbkom1</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/agtbkom1' target=\"_blank\">save file demo</a></strong> to <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/agtbkom1' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/agtbkom1</a>"]},"metadata":{}}]},{"cell_type":"code","source":["# Save your network architecture as txt\n","with open(\"network_architecture.txt\", \"w\") as f:\n","    f.write(str(model))\n","\n","# Log this file into wandb (or any other files you want to save in a similar manner)\n","wandb.save('network_architecture.txt')\n","wandb.finish()"],"metadata":{"id":"dq57BX1Jwb-s","colab":{"base_uri":"https://localhost:8080/","height":86},"executionInfo":{"status":"ok","timestamp":1756489577300,"user_tz":240,"elapsed":913,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"7e91ea4b-65b1-4696-95a4-656d1eab3d9f"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">save file demo</strong> at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/agtbkom1' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/agtbkom1</a><br> View project at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a><br>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 1 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_174603-agtbkom1/logs</code>"]},"metadata":{}}]},{"cell_type":"markdown","source":["![File in cloud](https://drive.usercontent.google.com/download?id=1gmiWcPMxZ89PxDO8RfwXpBuFHaTh5eC6)"],"metadata":{"id":"HkvjjuTBwcen"}},{"cell_type":"markdown","source":["### Displaying images in wandb\n","\n","It can be helpful to save images assocated with a training run to WandB. For more details beyond this simple example, plese see the documentation: https://docs.wandb.ai/guides/track/log/media"],"metadata":{"id":"UZolLN2SwhcK"}},{"cell_type":"code","source":["# Step 1: Gather the data we want to log in wandb\n","train_features, train_labels = next(iter(train_dataloader))\n","\n","# Step 2: Wrap each image (or optionally an entire array) with wandb.Image\n","images = []\n","for i in range(len(train_features)):\n","    item = train_features[i]\n","    images.append(wandb.Image(item, caption=f\"Label: {train_labels[i]}\")) # Optional caption is helpful!"],"metadata":{"id":"FL_nGSbSwjM2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["wandb.init(\n","    name = \"images demo\",\n","    reinit = True,\n","    project = \"10-423-623 Recitation 0\",\n",")\n","wandb.log({\"examples\": images})\n","wandb.finish()"],"metadata":{"id":"tNMwr8vTwk6v","colab":{"base_uri":"https://localhost:8080/","height":173},"executionInfo":{"status":"ok","timestamp":1756490047396,"user_tz":240,"elapsed":1863,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"a6ec2662-6c3b-4fa9-934c-79c0c30490b7"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_175405-vvgh89g2</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/vvgh89g2' target=\"_blank\">images demo</a></strong> to <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/vvgh89g2' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/vvgh89g2</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">images demo</strong> at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/vvgh89g2' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/vvgh89g2</a><br> View project at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a><br>Synced 5 W&B file(s), 64 media file(s), 0 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_175405-vvgh89g2/logs</code>"]},"metadata":{}}]},{"cell_type":"markdown","source":["![Result](https://drive.usercontent.google.com/download?id=1Ji4sMbmVucX7vtzS-fpDHJVQ0AGv2nY3)"],"metadata":{"id":"sJenz70nwmvq"}},{"cell_type":"markdown","source":["### Displaying tables in wandb\n","\n","Wandb allows for visualization of data in table format. For details beyond this simple example, please see the documentation!\n","\n","https://docs.wandb.ai/guides/track/log/log-tables"],"metadata":{"id":"63jkK9T7woeR"}},{"cell_type":"code","source":["# Step 1: Generate the data we want to store in wandb (here, we get the prediction for each image)\n","train_features, train_labels = next(iter(train_dataloader))\n","no_items = len(train_features)\n","model.eval()\n","\n","with torch.no_grad():\n","    preds = model(train_features.view(-1, 784))\n","    rounded_preds = torch.argmax(nn.functional.softmax(preds, dim=1), dim=1)"],"metadata":{"id":"FFLOkIfRwsSm"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Step 2: Create each row in the table iteratively.\n","my_table = []\n","\n","for i in range(no_items):\n","    img = train_features[i]\n","    truth = train_labels[i]\n","    prediction = rounded_preds[i]\n","    row = [i, wandb.Image(img), truth, prediction]\n","    my_table.append(row)\n","\n","# create a wandb.Table() with corresponding columns\n","columns = [\"id\", \"image\", \"truth\", \"prediction\"]\n","test_table = wandb.Table(data=my_table, columns=columns)"],"metadata":{"id":"1r5Xey7owtCk"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Step 3: Log to wandb\n","\n","wandb.init(\n","    name = \"table_demo\",\n","    reinit = True,\n","    project = \"10-423-623 Recitation 0\",\n",")\n","wandb.log({\"data_table\": test_table})\n","wandb.finish()"],"metadata":{"id":"Q54QR7NVwuUu","colab":{"base_uri":"https://localhost:8080/","height":173},"executionInfo":{"status":"ok","timestamp":1756490166229,"user_tz":240,"elapsed":3010,"user":{"displayName":"Abishek Anand","userId":"04137973513850597275"}},"outputId":"56620b39-d7e9-44e4-c5f9-2bc7a02aa4fc"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Tracking run with wandb version 0.21.1"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Run data is saved locally in <code>/content/wandb/run-20250829_175603-0xwe9bhm</code>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Syncing run <strong><a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/0xwe9bhm' target=\"_blank\">table_demo</a></strong> to <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View project at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run at <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/0xwe9bhm' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/0xwe9bhm</a>"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":[" View run <strong style=\"color:#cdcd00\">table_demo</strong> at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/0xwe9bhm' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200/runs/0xwe9bhm</a><br> View project at: <a href='https://wandb.ai/gaminghero812/10-423-623%20Recitation%200' target=\"_blank\">https://wandb.ai/gaminghero812/10-423-623%20Recitation%200</a><br>Synced 5 W&B file(s), 1 media file(s), 66 artifact file(s) and 0 other file(s)"]},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["<IPython.core.display.HTML object>"],"text/html":["Find logs at: <code>./wandb/run-20250829_175603-0xwe9bhm/logs</code>"]},"metadata":{}}]},{"cell_type":"markdown","source":["![Result](https://drive.usercontent.google.com/download?id=1-68vsS051YnvMKci2U03w-HTrNIVJwZ7)"],"metadata":{"id":"YZqEjtsFw5LC"}},{"cell_type":"markdown","source":["### Displaying a histogram in wandb\n","\n","It is also possible to use wandb to geneate histograms. For details beyond this simple example, please see the documentation!\n","\n","https://docs.wandb.ai/guides/track/log/media#histograms\n","\n"],"metadata":{"id":"kvGF1dR0w9Hq"}},{"cell_type":"code","source":["!wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n","with open('input.txt', 'r', encoding='utf-8') as f:\n","  text = f.read()\n","\n","words = text.split()\n","word_lengths = [len(word) for word in words]\n","\n","wandb.init()\n","wandb.log({\"word_lengths\": wandb.Histogram(word_lengths)})\n","wandb.finish()"],"metadata":{"id":"daoPZao2w_37"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["By default, wandb will plot a heatmap representation of multiple histograms \"with the training step on the x-axis, the metric value on the y-axis, and the count represented by color, to ease comparison of histograms logged throughout training.\" (see below)\n","\n","![Histogram](https://drive.usercontent.google.com/download?id=1x9e1buvPw5HsRvKBL-a83FpP7nKvfkxq)\n","\n","In this case there is only one timepoint so a heatmap is not helpful. One-off histograms can be found in the \"Overview\" section\n","\n","![Histogram](https://drive.usercontent.google.com/download?id=1aart34dLyB2rGulkUD_SQYdqx3K0ky-m)"],"metadata":{"id":"26WCNG8YxCK-"}},{"cell_type":"markdown","source":["We can also simply plot the data using matplotlib to get the same result"],"metadata":{"id":"mbCmOPYBxGAt"}},{"cell_type":"code","source":["import matplotlib.pyplot as plt\n","\n","plt.hist(word_lengths, bins=64)\n","plt.xlabel('Word Length')\n","plt.ylabel('Frequency')\n","plt.title('Distribution of Word Lengths')\n","plt.show()\n"],"metadata":{"id":"83w8WpqJxG--"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## More WandB!!\n","\n","WandB allows support for many more interesting features than can be covered in one recitation. Please check the documentation for information on things such as:\n","\n","\n","*   Hyperparameter sweeps\n","*   In-line jupyter %%wandb magic\n","*   Enabling/Disabling wandb for testing w/o modifying your code\n","*   Dataset and model weight tracking\n","\n","And much more!\n","\n","https://docs.wandb.ai/\n","\n"],"metadata":{"id":"wZ49c41FxMY9"}},{"cell_type":"markdown","source":["#Additional Learning: Einops Operations"],"metadata":{"id":"paPbNu_7fJ1S"}},{"cell_type":"markdown","source":["###1. Rearrange The rearrange operation is used to reorder dimensions, reshape tensors, or combine or split dimensions.\n","\n"],"metadata":{"id":"yZAPslPtfSVu"}},{"cell_type":"code","source":["import torch\n","from einops import rearrange\n","\n","# Reordering\n","\n","# Original tensor with shape (batch, height, width, channels)\n","x = torch.randn(10, 28, 28, 3)\n","\n","# Reorder to (batch, channels, height, width)\n","y = rearrange(x, 'b h w c -> b c h w')\n","\n","# Equivalent to\n","# y = x.permute(0, 3, 1, 2)\n","\n","print(y.shape)"],"metadata":{"id":"mn5I7facfJXN","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1737078460975,"user_tz":300,"elapsed":12527,"user":{"displayName":"Jingxuan Chen","userId":"11801539904368741364"}},"outputId":"36718c5d-bd7c-4a29-a1f6-a40d50f59d04"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([10, 3, 28, 28])\n"]}]},{"cell_type":"code","source":["# Reshaping\n","\n","# Original tensor with shape (batch, height, width, channels)\n","x = torch.randn(10, 28, 28, 3)\n","\n","# Flatten height and width into one dimension\n","y = rearrange(x, 'b h w c -> b (h w) c')\n","\n","# Equivalent to\n","# y = x.reshape((x.size(0), -1, x.size(3)))\n","\n","print(y.shape)"],"metadata":{"id":"9F_je1z8fOlC"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Original tensor with shape (batch, sequence_length, embedding_dim)\n","x = torch.randn(10, 16, 64)\n","\n","# Split embedding_dim into two dimensions\n","y = rearrange(x, 'b s (h d) -> b s h d', h=8)\n","\n","# Equivalent to\n","# y = x.reshape((x.size(0), x.size(1), 8, -1))\n","\n","print(y.shape)"],"metadata":{"id":"STMZ2iaOfgq8"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### 2. Reduce: The reduce operation is used to apply a reduction operation (like sum, mean, max) over specified dimensions."],"metadata":{"id":"WR8zEDe7gLMk"}},{"cell_type":"code","source":["import torch\n","from einops import reduce\n","\n","# Original tensor with shape (batch, height, width, channels)\n","x = torch.randn(10, 28, 28, 3)\n","\n","# Sum over the height and width dimensions\n","y = reduce(x, 'b h w c -> b c', 'sum')\n","\n","# Equivalent to\n","# y = x.sum((1, 2))\n","\n","print(y.shape)"],"metadata":{"id":"FjVLnCgxgMHo"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### 3. Einsum: Allows for complex tensor contractions using Einstein summation notation."],"metadata":{"id":"moMsFBqNgkM7"}},{"cell_type":"code","source":["# Matrix multiplication\n","\n","import torch\n","from einops import einsum\n","\n","# Matrices with shapes (batch, dim1, dim2) and (batch, dim2, dim3)\n","A = torch.randn(10, 32, 64)\n","B = torch.randn(10, 64, 16)\n","\n","# Perform batch matrix multiplication\n","C = einsum(A, B, 'b i j, b j k -> b i k')\n","\n","# Equivalent to\n","# C = A @ B\n","\n","print(C.shape)"],"metadata":{"id":"b_tInxX5gguU"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Outer product\n","\n","# Vectors with shape (batch, dim)\n","v1 = torch.randn(10, 32)\n","v2 = torch.randn(10, 64)\n","\n","# Outer product to produce a matrix\n","outer = einsum(v1, v2, 'b i, b j -> b i j')\n","\n","# no staightforward way to do in pytorch without using torch.einsum\n","# there is torch.outer, but that only works with 1D tensors\n","\n","\n","print(outer.shape)"],"metadata":{"id":"odJGNUE0gwn8","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1737078820060,"user_tz":300,"elapsed":86,"user":{"displayName":"Jingxuan Chen","userId":"11801539904368741364"}},"outputId":"0606d247-913b-4925-f1ce-8bf0360b45b1"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([10, 32, 64])\n"]}]},{"cell_type":"code","source":["print(v1.unsqueeze(1).shape)\n","print(v2.unsqueeze(2).shape)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Q0EryEhM90Vw","executionInfo":{"status":"ok","timestamp":1737078995436,"user_tz":300,"elapsed":86,"user":{"displayName":"Jingxuan Chen","userId":"11801539904368741364"}},"outputId":"1b9aed81-8041-44e0-98d1-aa4bbd550188"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([10, 1, 32])\n","torch.Size([10, 64, 1])\n"]}]}]}