{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"machine_shape":"hm","collapsed_sections":["k_yKKFNPZzeW","gBjDQlUeZ1Hj","yWjh3DouZ-58"]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["# Recitation 0: Introduction to Google Colab"],"metadata":{"id":"NJFOWKUFYoYk"}},{"cell_type":"markdown","source":["# What's in this video?\n","\n","- Basics of Google Colab and it's Overview\n","- Bash and Magic Commands\n","- Runtime\n","- Managing your files using Google Drive\n","- Saving and Loading Model Checkpoints\n","- Managing Dataset\n","- Colab Pro or Colab Pro+"],"metadata":{"id":"C6ASwJPOmB2_"}},{"cell_type":"markdown","source":["# Basics\n","\n","#### Google Colab\n","\n","- Colab is developed by Google Research and provides a Jupyter Notebook-style Python execution environment accessible directly through a web browser.\n","- Main benefit is its computing resources.\n","- For free, you get access to CPU and Tesla T4 GPU.\n","- To access more powerful GPUs like L4 and A100, you can choose to pay for Google Colab Pro or Pro+ (https://colab.research.google.com/signup)\n","\n","#### Accessing Colab\n","- Go to https://colab.research.google.com/ to create and access your notebooks\n","- Directly from Google Drive\n","- From your GitHub repository\n","- Upload from local system\n","\n","This recitation assumes basic knowledge of using Jupyter Notebooks, so please familiarize yourself with it if you haven't already.\n"],"metadata":{"id":"8x-Ov_cqYtZh"}},{"cell_type":"markdown","source":["# Bash and Magic Commands\n","\n","Colab runs in a linux environemnt and you can access the terminal with `!`"],"metadata":{"id":"aVcR53o6Yw16"}},{"cell_type":"markdown","source":["#### Bash Commands"],"metadata":{"id":"OmIYabvTZg8O"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"Nau1JGAEYZk3"},"outputs":[],"source":["!nvidia-smi"]},{"cell_type":"code","source":["!pip install torch\n","import gc\n","import os\n","import torch\n","import torch.nn as nn\n","import torch.optim as optim"],"metadata":{"id":"ByRjEG-CY31w"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["!ls\n","# !cd ..\n","# !mkdir"],"metadata":{"id":"6UmUuQlsZXlw"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["#### Magic commands"],"metadata":{"id":"NmIqsLadZbvo"}},{"cell_type":"code","source":["%time\n","# %env\n","# %matplotlib inline\n","# %debug"],"metadata":{"id":"yfRtKSPCZeO-"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Runtime"],"metadata":{"id":"1oqX33rSZkwM"}},{"cell_type":"code","source":["!nvidia-smi\n","import torch\n","DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n","print(\"Device: \", DEVICE)"],"metadata":{"id":"9z-2NXx1ZrH_"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Utilizing Free GPU/TPU Resources\n","\n","#### Changing Runtime\n","- Runtime > Change runtime type\n","- Select GPU/TPU and High-RAM option\n","\n","\n","\n","#### GPUs: Training Time of ResNet50\n","- T4: 1x Speedup (Baseline)\n","- V100: 3.6x Speedup (Comparing to T4)\n","- A100: 10x Speedup (Comparing to T4)\n","- TPU: TPU is a completely different architecture and require many training constraints"],"metadata":{"id":"sOFlPFPmZqYT"}},{"cell_type":"markdown","source":["### Restart Session vs Restart Runtime\n","\n","\n","Restart session\n","- Runtime > Restart session\n","- Clears all session variables\n","\n","Restart runtime\n","- Runtime > Disconnect and delete runtime\n","- Deletes session\n","- Lose files in content folder\n","- Switching GPUs will also delete current runtime"],"metadata":{"id":"SxyqXtzqmn8x"}},{"cell_type":"code","source":["torch.cuda.empty_cache()  # Clear unused GPU memory cached by PyTorch to free up space.\n","gc.collect()  # Call the Python's garbage collector to release unused memory."],"metadata":{"id":"MSyEXKxDmrEB"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Sample Helpful Code Snippets"],"metadata":{"id":"efc0e4GeZvoQ"}},{"cell_type":"markdown","source":["### Mounting to Google Drive\n","\n","Very useful as you lose all files after the runtime ends"],"metadata":{"id":"k_yKKFNPZzeW"}},{"cell_type":"code","source":["from google.colab import drive\n","drive.mount(\"/content/drive\")"],"metadata":{"id":"d2rPwLUyZx8r"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Saving/Loading files - Model checkpoints\n","\n","Moving files to and from Google Drive"],"metadata":{"id":"gBjDQlUeZ1Hj"}},{"cell_type":"code","source":["class MLP(nn.Module):\n","    def __init__(self, size):\n","        super(MLP, self).__init__()\n","        self.layers = []\n","        for in_dim, out_dim in zip(size[:-2], size[1:-1]):\n","          self.layers.extend([\n","              nn.Linear(in_dim, out_dim),\n","              nn.ReLU(),\n","              nn.BatchNorm1d(out_dim),\n","              nn.Dropout(0.5),\n","        ])\n","        self.layers.append(nn.Linear(size[-2], size[-1]))\n","        self.model = nn.Sequential(*self.layers)\n","        self.model.apply(self.init_param)\n","\n","    def init_param(self, param):\n","      if type(param) == nn.Linear:\n","        nn.init.xavier_uniform_(param.weight)\n","\n","    def forward(self, x):\n","      return self.model(x)\n","\n","\n","# Define your model\n","model = MLP([40, 2048, 512, 256, 71])\n","\n","# Define optimizer, scheduler, loss function, epochs, and other metrics\n","epoch = 5\n","metrics = {'accuracy': 0.85}\n","optimizer = optim.Adam(model.parameters(), lr=0.001)\n","scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)"],"metadata":{"id":"TqVTDvWxZ2Yo"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["#### Checkpoint Saving and Loading Model"],"metadata":{"id":"S6nh1LaocNbS"}},{"cell_type":"code","source":["def save_model(model, optimizer, scheduler, metrics, epoch, path):\n","    \"\"\"\n","    Saves the model and other related states to a checkpoint file.\n","\n","    Functionality:\n","    - Saves the model's state dictionary, optimizer state, scheduler state,\n","      metrics, and epoch to the specified file checkpoint path.\n","    \"\"\"\n","    torch.save(\n","        {'model_state_dict'         : model.state_dict(),\n","         'optimizer_state_dict'     : optimizer.state_dict(),\n","         'scheduler_state_dict'     : scheduler.state_dict(),\n","         'metric'                   : metrics,\n","         'epoch'                    : epoch},\n","         path)\n","\n","def load_model(model, optimizer=None, scheduler=None, path='./checkpoint.pth'):\n","    \"\"\"\n","    Loads the model and other related states from a checkpoint file.\n","\n","    Functionality:\n","    - Loads the checkpoint from the specified file path using `torch.load`.\n","    - Restores the model's state dictionary from the checkpoint.\n","    - Optionally restores the optimizer and scheduler states if they are provided.\n","    \"\"\"\n","    checkpoint = torch.load(path, weights_only=False)\n","    model.load_state_dict(checkpoint['model_state_dict'])\n","    if optimizer is not None:\n","        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n","    else:\n","        optimizer = None\n","    if scheduler is not None:\n","        scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n","    else:\n","        scheduler = None\n","    epoch = checkpoint['epoch']\n","    metrics = checkpoint['metric']\n","    return model, optimizer, scheduler, epoch, metrics"],"metadata":{"id":"7X5vhmWocLmX"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["#### Saving Model"],"metadata":{"id":"xDegbEHWgVj7"}},{"cell_type":"code","source":["# Define the directory and checkpoint's file path.\n","CHECKPOINT_DIR = '/content/drive/MyDrive/Checkpoints'\n","MODEL_SAVE_PATH = os.path.join(CHECKPOINT_DIR, '11785_s25_rec0_google_colab_checkpoint.pt')\n","os.makedirs(CHECKPOINT_DIR, exist_ok=True)\n","\n","# Save the model.\n","save_model(model, optimizer, scheduler, metrics, epoch, MODEL_SAVE_PATH)\n","print(f\"Model saved to {MODEL_SAVE_PATH}\")"],"metadata":{"id":"-thnl4qAchh_"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["#### Loading Model"],"metadata":{"id":"YB5DzvIBgav3"}},{"cell_type":"code","source":["# Create a new instance of the same model architecture.\n","loaded_model = MLP([40, 2048, 512, 256, 71])\n","\n","# Load the model, optimizer, and other saved states.\n","loaded_model, loaded_optimizer, loaded_scheduler, loaded_epoch, loaded_metrics = load_model(\n","    loaded_model, optimizer, scheduler, MODEL_SAVE_PATH\n",")\n","\n","# Verify the loaded states.\n","print(f\"Model loaded. Resumed at epoch {loaded_epoch} with metrics: {loaded_metrics}\")"],"metadata":{"id":"sDhYkd7LceqZ"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Managing dataset\n","\n","Obtaining dataset\n","- Kaggle Command\n","- Manually uploading\n","\n","DO NOT directly store dataset in Google Drive, instead...\n","- Download/uploading dataset every time\n","- Move dataset from Google Drive into content folder\n","- Connect to GCP or AWS"],"metadata":{"id":"yWjh3DouZ-58"}},{"cell_type":"code","source":["# Downloads dataset from kaggle\n","!pip install --upgrade --force-reinstall --no-deps kaggle==1.5.8\n","!mkdir /root/.kaggle\n","\n","# Retrieve the Kaggle Username and API key\n","from google.colab import userdata\n","kaggle_username = 'shubhamkachroo'\n","kaggle_api_key = userdata.get('KAGGLE_API_KEY')\n","\n","# Creates Kaggle config file with retrieved username and API key\n","with open(\"/root/.kaggle/kaggle.json\", \"w+\") as f:\n","    f.write(f'{{\"username\":\"{kaggle_username}\",\"key\":\"{kaggle_api_key}\"}}')\n","# Sets appropriate permissions for Kaggle config file\n","!chmod 600 /root/.kaggle/kaggle.json\n","\n","# Make sure to join the competition on Kaggle before running this command!!\n","# Downloads dataset of the competition using Kaggle API.\n","!kaggle competitions download -c 11785-fall23-kaggle-setup\n","\n","# Unzips downloaded dataset into given directory folder.\n","!unzip -qo /content/11785-fall23-kaggle-setup.zip -d '/content/11785-fall23-kaggle-setup'"],"metadata":{"id":"glGuFriJZ_4n"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Important Considerations for Students\n","\n","- Session Timeout: Google Colab sessions may time out after a certain period of inactivity. To prevent this, remember to save your work frequently and consider using Colab Pro to extend session runtimes.\n","\n","- Limited Persistent Storage: While Google Colab saves your notebooks on Google Drive, the storage space is limited. Make sure to clean up unnecessary files or download your work to your local machine to free up space.\n","\n","- Resource Limits: Free Google Colab accounts have some resource limitations, such as GPU availability and maximum session runtimes. For resource-intensive projects, consider upgrading to Colab Pro for improved performance."],"metadata":{"id":"FVTCHiaZofKD"}},{"cell_type":"markdown","source":["# Colab Pro\n","\n","- Longer session runtime, reducing risk of timeout\n","- Priority access to GPU\n","- Increased storage\n","- Background Execution (Google Colab Pro+)"],"metadata":{"id":"0p59ke10aB0z"}}]}