{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[{"file_id":"1uaQc4_QCbtdKCmyKVPrP6CartoxFh9DT","timestamp":1772752598039}]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"kPt5q27L5557"},"source":["# CS246 - Colab 9 (Winter 2025)\n","## Studying COVID-19"]},{"cell_type":"code","source":["from IPython.display import Image\n","print(\"Colab 9 Mascot saying Bye-Bye :( \\n\")\n","Image(url='https://media.giphy.com/media/tuvMgAPzxaQBq/giphy.gif',width=200)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":210},"id":"mittXosPIUOV","executionInfo":{"status":"ok","timestamp":1772751681630,"user_tz":480,"elapsed":53,"user":{"displayName":"Rishabh Ranjan","userId":"14643331064616738605"}},"outputId":"25f686ee-70cb-4a78-8682-022c886b7195"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Colab 9 Mascot saying Bye-Bye :( \n","\n"]},{"output_type":"execute_result","data":{"text/html":["<img src=\"https://media.giphy.com/media/tuvMgAPzxaQBq/giphy.gif\" width=\"200\"/>"],"text/plain":["<IPython.core.display.Image object>"]},"metadata":{},"execution_count":8}]},{"cell_type":"markdown","metadata":{"id":"p0-YhEpP_Ds-"},"source":["### Setup"]},{"cell_type":"markdown","metadata":{"id":"VwxKjA7kO2IY"},"source":["Let's set up Spark on your Colab environment.  Run the cell below!"]},{"cell_type":"code","source":["!pip install pyspark\n","!pip install -U -q PyDrive2\n","\n","import os\n","os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-17-openjdk-amd64\"\n","os.environ[\"PATH\"] = \"/usr/lib/jvm/java-17-openjdk-amd64/bin:\" + os.environ[\"PATH\"]\n","\n","!java -version  # Should show Java 17"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2DDYOPrDOvVy","executionInfo":{"status":"ok","timestamp":1772751692594,"user_tz":480,"elapsed":10967,"user":{"displayName":"Rishabh Ranjan","userId":"14643331064616738605"}},"outputId":"bb1cffde-664d-48f4-ea6a-4e25168c60ee"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Requirement already satisfied: pyspark in /usr/local/lib/python3.12/dist-packages (4.0.2)\n","Requirement already satisfied: py4j<0.10.9.10,>=0.10.9.7 in /usr/local/lib/python3.12/dist-packages (from pyspark) (0.10.9.9)\n","openjdk version \"17.0.17\" 2025-10-21\n","OpenJDK Runtime Environment (build 17.0.17+10-Ubuntu-122.04)\n","OpenJDK 64-Bit Server VM (build 17.0.17+10-Ubuntu-122.04, mixed mode, sharing)\n"]}]},{"cell_type":"markdown","metadata":{"id":"PUUjUvXe3Sjk"},"source":["Now we authenticate a Google Drive client to download the files we will be processing in our Spark job.\n","\n","**Make sure to follow the interactive instructions.**"]},{"cell_type":"code","metadata":{"id":"lRElWs_x2mGh"},"source":["from pydrive2.auth import GoogleAuth\n","from pydrive2.drive import GoogleDrive\n","from google.colab import auth\n","from oauth2client.client import GoogleCredentials\n","\n","# Authenticate and create the PyDrive client\n","auth.authenticate_user()\n","gauth = GoogleAuth()\n","gauth.credentials = GoogleCredentials.get_application_default()\n","drive = GoogleDrive(gauth)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"QHsFTGUy2n1c"},"source":["id='1YT7ttUAafCjbVdm6obeHp1TWAK0rEtoR'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('time_series_covid19_confirmed_global.csv')\n","\n","id='1YxEA5UQ2EFJ_9oLssM__Gs1ncVNufGNA'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('time_series_covid19_deaths_global.csv')\n","\n","id='1CNxszuZTeIw-5cF5yrzKMZdb1qV0hSoy'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('time_series_covid19_recovered_global.csv')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qwtlO4_m_LbQ"},"source":["If you executed the cells above, you should be able to see the dataset we will use for this Colab under the \"Files\" tab on the left panel.\n","\n","Next, we import some of the common libraries needed for our task."]},{"cell_type":"code","metadata":{"id":"twk-K-jilWK7"},"source":["import pandas as pd\n","import numpy as np\n","import matplotlib.pyplot as plt\n","%matplotlib inline\n","\n","import pyspark\n","from pyspark.sql import *\n","from pyspark.sql.types import *\n","from pyspark.sql.functions import *\n","from pyspark import SparkContext, SparkConf"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"BtrJlMBt1Ela"},"source":["Let's initialize the Spark context."]},{"cell_type":"code","metadata":{"id":"Vm3sAVeK1EDZ"},"source":["# create the session\n","conf = SparkConf().set(\"spark.ui.port\", \"4050\")\n","\n","# create the context\n","sc = pyspark.SparkContext(conf=conf)\n","spark = SparkSession.builder.getOrCreate()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"kAYRX2PMm0L6"},"source":["### Data Loading"]},{"cell_type":"markdown","metadata":{"id":"7hXdMR6wnEIM"},"source":["In this Colab, we will be analyzing the time series data of the Coronavirus COVID-19 Global Cases, collected by Johns Hopkins CSSE.\n","\n","Here you can check a realtime dashboard based on this dataset (data as of 2023): [https://www.arcgis.com/apps/opsdashboard/index.html?fbclid=IwAR2hQKsEZ3D38wVtXGryUhP9CG0Z6MYbUM_boPEaV8FBe71wUvDPc65ZG78#/bda7594740fd40299423467b48e9ecf6](https://www.arcgis.com/apps/opsdashboard/index.html?fbclid=IwAR2hQKsEZ3D38wVtXGryUhP9CG0Z6MYbUM_boPEaV8FBe71wUvDPc65ZG78#/bda7594740fd40299423467b48e9ecf6)\n","\n","---\n","\n","\n","\n","*   ```confirmed```: dataframe containing the cumulative number of confirmed COVID-19 cases, divided by geographical area\n","*   ```deaths```: dataframe containing the cumulative number of deaths due to COVID-19, divided by geographical area\n","*   ```recovered```: dataframe containing the cumulative number of recoevered patients, divided by geographical area\n","\n","The data sets contain data entries for each day, representing the cumulative totals as of that day.\n","\n","\n","\n","\n","\n"]},{"cell_type":"code","metadata":{"id":"EUSeIQlNqv6J"},"source":["confirmed = spark.read.csv('time_series_covid19_confirmed_global.csv', header=True)\n","deaths = spark.read.csv('time_series_covid19_deaths_global.csv', header=True)\n","recovered = spark.read.csv('time_series_covid19_recovered_global.csv', header=True)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"NvyFsE-fLXpg","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1772751723824,"user_tz":480,"elapsed":114,"user":{"displayName":"Rishabh Ranjan","userId":"14643331064616738605"}},"outputId":"cc31795d-c8a6-4ae7-cabf-3cf141f7ad97"},"source":["confirmed.printSchema()"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["root\n"," |-- Province/State: string (nullable = true)\n"," |-- Country/Region: string (nullable = true)\n"," |-- Lat: string (nullable = true)\n"," |-- Long: string (nullable = true)\n"," |-- 1/22/20: string (nullable = true)\n"," |-- 1/23/20: string (nullable = true)\n"," |-- 1/24/20: string (nullable = true)\n"," |-- 1/25/20: string (nullable = true)\n"," |-- 1/26/20: string (nullable = true)\n"," |-- 1/27/20: string (nullable = true)\n"," |-- 1/28/20: string (nullable = true)\n"," |-- 1/29/20: string (nullable = true)\n"," |-- 1/30/20: string (nullable = true)\n"," |-- 1/31/20: string (nullable = true)\n"," |-- 2/1/20: string (nullable = true)\n"," |-- 2/2/20: string (nullable = true)\n"," |-- 2/3/20: string (nullable = true)\n"," |-- 2/4/20: string (nullable = true)\n"," |-- 2/5/20: string (nullable = true)\n"," |-- 2/6/20: string (nullable = true)\n"," |-- 2/7/20: string (nullable = true)\n"," |-- 2/8/20: string (nullable = true)\n"," |-- 2/9/20: string (nullable = true)\n"," |-- 2/10/20: string (nullable = true)\n"," |-- 2/11/20: string (nullable = true)\n"," |-- 2/12/20: string (nullable = true)\n"," |-- 2/13/20: string (nullable = true)\n"," |-- 2/14/20: string (nullable = true)\n"," |-- 2/15/20: string (nullable = true)\n"," |-- 2/16/20: string (nullable = true)\n"," |-- 2/17/20: string (nullable = true)\n"," |-- 2/18/20: string (nullable = true)\n"," |-- 2/19/20: string (nullable = true)\n"," |-- 2/20/20: string (nullable = true)\n"," |-- 2/21/20: string (nullable = true)\n"," |-- 2/22/20: string (nullable = true)\n"," |-- 2/23/20: string (nullable = true)\n"," |-- 2/24/20: string (nullable = true)\n"," |-- 2/25/20: string (nullable = true)\n"," |-- 2/26/20: string (nullable = true)\n"," |-- 2/27/20: string (nullable = true)\n"," |-- 2/28/20: string (nullable = true)\n"," |-- 2/29/20: string (nullable = true)\n"," |-- 3/1/20: string (nullable = true)\n"," |-- 3/2/20: string (nullable = true)\n"," |-- 3/3/20: string (nullable = true)\n"," |-- 3/4/20: string (nullable = true)\n"," |-- 3/5/20: string (nullable = true)\n"," |-- 3/6/20: string (nullable = true)\n"," |-- 3/7/20: string (nullable = true)\n"," |-- 3/8/20: string (nullable = true)\n"," |-- 3/9/20: string (nullable = true)\n"," |-- 3/10/20: string (nullable = true)\n"," |-- 3/11/20: string (nullable = true)\n"," |-- 3/12/20: string (nullable = true)\n"," |-- 3/13/20: string (nullable = true)\n"," |-- 3/14/20: string (nullable = true)\n"," |-- 3/15/20: string (nullable = true)\n"," |-- 3/16/20: string (nullable = true)\n"," |-- 3/17/20: string (nullable = true)\n"," |-- 3/18/20: string (nullable = true)\n"," |-- 3/19/20: string (nullable = true)\n"," |-- 3/20/20: string (nullable = true)\n"," |-- 3/21/20: string (nullable = true)\n"," |-- 3/22/20: string (nullable = true)\n"," |-- 3/23/20: string (nullable = true)\n"," |-- 3/24/20: string (nullable = true)\n"," |-- 3/25/20: string (nullable = true)\n"," |-- 3/26/20: string (nullable = true)\n"," |-- 3/27/20: string (nullable = true)\n"," |-- 3/28/20: string (nullable = true)\n"," |-- 3/29/20: string (nullable = true)\n"," |-- 3/30/20: string (nullable = true)\n"," |-- 3/31/20: string (nullable = true)\n"," |-- 4/1/20: string (nullable = true)\n"," |-- 4/2/20: string (nullable = true)\n"," |-- 4/3/20: string (nullable = true)\n"," |-- 4/4/20: string (nullable = true)\n"," |-- 4/5/20: string (nullable = true)\n"," |-- 4/6/20: string (nullable = true)\n"," |-- 4/7/20: string (nullable = true)\n"," |-- 4/8/20: string (nullable = true)\n"," |-- 4/9/20: string (nullable = true)\n"," |-- 4/10/20: string (nullable = true)\n"," |-- 4/11/20: string (nullable = true)\n"," |-- 4/12/20: string (nullable = true)\n"," |-- 4/13/20: string (nullable = true)\n"," |-- 4/14/20: string (nullable = true)\n"," |-- 4/15/20: string (nullable = true)\n"," |-- 4/16/20: string (nullable = true)\n"," |-- 4/17/20: string (nullable = true)\n"," |-- 4/18/20: string (nullable = true)\n"," |-- 4/19/20: string (nullable = true)\n"," |-- 4/20/20: string (nullable = true)\n"," |-- 4/21/20: string (nullable = true)\n"," |-- 4/22/20: string (nullable = true)\n"," |-- 4/23/20: string (nullable = true)\n"," |-- 4/24/20: string (nullable = true)\n"," |-- 4/25/20: string (nullable = true)\n"," |-- 4/26/20: string (nullable = true)\n"," |-- 4/27/20: string (nullable = true)\n"," |-- 4/28/20: string (nullable = true)\n"," |-- 4/29/20: string (nullable = true)\n"," |-- 4/30/20: string (nullable = true)\n"," |-- 5/1/20: string (nullable = true)\n"," |-- 5/2/20: string (nullable = true)\n"," |-- 5/3/20: string (nullable = true)\n"," |-- 5/4/20: string (nullable = true)\n"," |-- 5/5/20: string (nullable = true)\n"," |-- 5/6/20: string (nullable = true)\n"," |-- 5/7/20: string (nullable = true)\n"," |-- 5/8/20: string (nullable = true)\n"," |-- 5/9/20: string (nullable = true)\n"," |-- 5/10/20: string (nullable = true)\n"," |-- 5/11/20: string (nullable = true)\n"," |-- 5/12/20: string (nullable = true)\n"," |-- 5/13/20: string (nullable = true)\n"," |-- 5/14/20: string (nullable = true)\n"," |-- 5/15/20: string (nullable = true)\n"," |-- 5/16/20: string (nullable = true)\n"," |-- 5/17/20: string (nullable = true)\n"," |-- 5/18/20: string (nullable = true)\n"," |-- 5/19/20: string (nullable = true)\n"," |-- 5/20/20: string (nullable = true)\n"," |-- 5/21/20: string (nullable = true)\n"," |-- 5/22/20: string (nullable = true)\n"," |-- 5/23/20: string (nullable = true)\n"," |-- 5/24/20: string (nullable = true)\n"," |-- 5/25/20: string (nullable = true)\n"," |-- 5/26/20: string (nullable = true)\n"," |-- 5/27/20: string (nullable = true)\n"," |-- 5/28/20: string (nullable = true)\n"," |-- 5/29/20: string (nullable = true)\n"," |-- 5/30/20: string (nullable = true)\n"," |-- 5/31/20: string (nullable = true)\n"," |-- 6/1/20: string (nullable = true)\n"," |-- 6/2/20: string (nullable = true)\n"," |-- 6/3/20: string (nullable = true)\n"," |-- 6/4/20: string (nullable = true)\n"," |-- 6/5/20: string (nullable = true)\n"," |-- 6/6/20: string (nullable = true)\n"," |-- 6/7/20: string (nullable = true)\n"," |-- 6/8/20: string (nullable = true)\n"," |-- 6/9/20: string (nullable = true)\n"," |-- 6/10/20: string (nullable = true)\n"," |-- 6/11/20: string (nullable = true)\n"," |-- 6/12/20: string (nullable = true)\n"," |-- 6/13/20: string (nullable = true)\n"," |-- 6/14/20: string (nullable = true)\n"," |-- 6/15/20: string (nullable = true)\n"," |-- 6/16/20: string (nullable = true)\n"," |-- 6/17/20: string (nullable = true)\n"," |-- 6/18/20: string (nullable = true)\n"," |-- 6/19/20: string (nullable = true)\n"," |-- 6/20/20: string (nullable = true)\n"," |-- 6/21/20: string (nullable = true)\n"," |-- 6/22/20: string (nullable = true)\n"," |-- 6/23/20: string (nullable = true)\n"," |-- 6/24/20: string (nullable = true)\n"," |-- 6/25/20: string (nullable = true)\n"," |-- 6/26/20: string (nullable = true)\n"," |-- 6/27/20: string (nullable = true)\n"," |-- 6/28/20: string (nullable = true)\n"," |-- 6/29/20: string (nullable = true)\n"," |-- 6/30/20: string (nullable = true)\n"," |-- 7/1/20: string (nullable = true)\n"," |-- 7/2/20: string (nullable = true)\n"," |-- 7/3/20: string (nullable = true)\n"," |-- 7/4/20: string (nullable = true)\n"," |-- 7/5/20: string (nullable = true)\n"," |-- 7/6/20: string (nullable = true)\n"," |-- 7/7/20: string (nullable = true)\n"," |-- 7/8/20: string (nullable = true)\n"," |-- 7/9/20: string (nullable = true)\n"," |-- 7/10/20: string (nullable = true)\n"," |-- 7/11/20: string (nullable = true)\n"," |-- 7/12/20: string (nullable = true)\n"," |-- 7/13/20: string (nullable = true)\n"," |-- 7/14/20: string (nullable = true)\n"," |-- 7/15/20: string (nullable = true)\n"," |-- 7/16/20: string (nullable = true)\n"," |-- 7/17/20: string (nullable = true)\n"," |-- 7/18/20: string (nullable = true)\n"," |-- 7/19/20: string (nullable = true)\n"," |-- 7/20/20: string (nullable = true)\n"," |-- 7/21/20: string (nullable = true)\n"," |-- 7/22/20: string (nullable = true)\n"," |-- 7/23/20: string (nullable = true)\n"," |-- 7/24/20: string (nullable = true)\n"," |-- 7/25/20: string (nullable = true)\n"," |-- 7/26/20: string (nullable = true)\n"," |-- 7/27/20: string (nullable = true)\n"," |-- 7/28/20: string (nullable = true)\n"," |-- 7/29/20: string (nullable = true)\n"," |-- 7/30/20: string (nullable = true)\n"," |-- 7/31/20: string (nullable = true)\n"," |-- 8/1/20: string (nullable = true)\n"," |-- 8/2/20: string (nullable = true)\n"," |-- 8/3/20: string (nullable = true)\n"," |-- 8/4/20: string (nullable = true)\n"," |-- 8/5/20: string (nullable = true)\n"," |-- 8/6/20: string (nullable = true)\n"," |-- 8/7/20: string (nullable = true)\n"," |-- 8/8/20: string (nullable = true)\n"," |-- 8/9/20: string (nullable = true)\n"," |-- 8/10/20: string (nullable = true)\n"," |-- 8/11/20: string (nullable = true)\n"," |-- 8/12/20: string (nullable = true)\n"," |-- 8/13/20: string (nullable = true)\n"," |-- 8/14/20: string (nullable = true)\n"," |-- 8/15/20: string (nullable = true)\n"," |-- 8/16/20: string (nullable = true)\n"," |-- 8/17/20: string (nullable = true)\n"," |-- 8/18/20: string (nullable = true)\n"," |-- 8/19/20: string (nullable = true)\n"," |-- 8/20/20: string (nullable = true)\n"," |-- 8/21/20: string (nullable = true)\n"," |-- 8/22/20: string (nullable = true)\n"," |-- 8/23/20: string (nullable = true)\n"," |-- 8/24/20: string (nullable = true)\n"," |-- 8/25/20: string (nullable = true)\n"," |-- 8/26/20: string (nullable = true)\n"," |-- 8/27/20: string (nullable = true)\n"," |-- 8/28/20: string (nullable = true)\n"," |-- 8/29/20: string (nullable = true)\n"," |-- 8/30/20: string (nullable = true)\n"," |-- 8/31/20: string (nullable = true)\n"," |-- 9/1/20: string (nullable = true)\n"," |-- 9/2/20: string (nullable = true)\n"," |-- 9/3/20: string (nullable = true)\n"," |-- 9/4/20: string (nullable = true)\n"," |-- 9/5/20: string (nullable = true)\n"," |-- 9/6/20: string (nullable = true)\n"," |-- 9/7/20: string (nullable = true)\n"," |-- 9/8/20: string (nullable = true)\n"," |-- 9/9/20: string (nullable = true)\n"," |-- 9/10/20: string (nullable = true)\n"," |-- 9/11/20: string (nullable = true)\n"," |-- 9/12/20: string (nullable = true)\n"," |-- 9/13/20: string (nullable = true)\n"," |-- 9/14/20: string (nullable = true)\n"," |-- 9/15/20: string (nullable = true)\n"," |-- 9/16/20: string (nullable = true)\n"," |-- 9/17/20: string (nullable = true)\n"," |-- 9/18/20: string (nullable = true)\n"," |-- 9/19/20: string (nullable = true)\n"," |-- 9/20/20: string (nullable = true)\n"," |-- 9/21/20: string (nullable = true)\n"," |-- 9/22/20: string (nullable = true)\n"," |-- 9/23/20: string (nullable = true)\n"," |-- 9/24/20: string (nullable = true)\n"," |-- 9/25/20: string (nullable = true)\n"," |-- 9/26/20: string (nullable = true)\n"," |-- 9/27/20: string (nullable = true)\n"," |-- 9/28/20: string (nullable = true)\n"," |-- 9/29/20: string (nullable = true)\n"," |-- 9/30/20: string (nullable = true)\n"," |-- 10/1/20: string (nullable = true)\n"," |-- 10/2/20: string (nullable = true)\n"," |-- 10/3/20: string (nullable = true)\n"," |-- 10/4/20: string (nullable = true)\n"," |-- 10/5/20: string (nullable = true)\n"," |-- 10/6/20: string (nullable = true)\n"," |-- 10/7/20: string (nullable = true)\n"," |-- 10/8/20: string (nullable = true)\n"," |-- 10/9/20: string (nullable = true)\n"," |-- 10/10/20: string (nullable = true)\n"," |-- 10/11/20: string (nullable = true)\n"," |-- 10/12/20: string (nullable = true)\n"," |-- 10/13/20: string (nullable = true)\n"," |-- 10/14/20: string (nullable = true)\n"," |-- 10/15/20: string (nullable = true)\n"," |-- 10/16/20: string (nullable = true)\n"," |-- 10/17/20: string (nullable = true)\n"," |-- 10/18/20: string (nullable = true)\n"," |-- 10/19/20: string (nullable = true)\n"," |-- 10/20/20: string (nullable = true)\n"," |-- 10/21/20: string (nullable = true)\n"," |-- 10/22/20: string (nullable = true)\n"," |-- 10/23/20: string (nullable = true)\n"," |-- 10/24/20: string (nullable = true)\n"," |-- 10/25/20: string (nullable = true)\n"," |-- 10/26/20: string (nullable = true)\n"," |-- 10/27/20: string (nullable = true)\n"," |-- 10/28/20: string (nullable = true)\n"," |-- 10/29/20: string (nullable = true)\n"," |-- 10/30/20: string (nullable = true)\n"," |-- 10/31/20: string (nullable = true)\n"," |-- 11/1/20: string (nullable = true)\n"," |-- 11/2/20: string (nullable = true)\n"," |-- 11/3/20: string (nullable = true)\n"," |-- 11/4/20: string (nullable = true)\n"," |-- 11/5/20: string (nullable = true)\n"," |-- 11/6/20: string (nullable = true)\n"," |-- 11/7/20: string (nullable = true)\n"," |-- 11/8/20: string (nullable = true)\n"," |-- 11/9/20: string (nullable = true)\n"," |-- 11/10/20: string (nullable = true)\n"," |-- 11/11/20: string (nullable = true)\n"," |-- 11/12/20: string (nullable = true)\n"," |-- 11/13/20: string (nullable = true)\n"," |-- 11/14/20: string (nullable = true)\n"," |-- 11/15/20: string (nullable = true)\n"," |-- 11/16/20: string (nullable = true)\n"," |-- 11/17/20: string (nullable = true)\n"," |-- 11/18/20: string (nullable = true)\n"," |-- 11/19/20: string (nullable = true)\n"," |-- 11/20/20: string (nullable = true)\n"," |-- 11/21/20: string (nullable = true)\n"," |-- 11/22/20: string (nullable = true)\n"," |-- 11/23/20: string (nullable = true)\n"," |-- 11/24/20: string (nullable = true)\n"," |-- 11/25/20: string (nullable = true)\n"," |-- 11/26/20: string (nullable = true)\n"," |-- 11/27/20: string (nullable = true)\n"," |-- 11/28/20: string (nullable = true)\n"," |-- 11/29/20: string (nullable = true)\n"," |-- 11/30/20: string (nullable = true)\n"," |-- 12/1/20: string (nullable = true)\n"," |-- 12/2/20: string (nullable = true)\n"," |-- 12/3/20: string (nullable = true)\n"," |-- 12/4/20: string (nullable = true)\n"," |-- 12/5/20: string (nullable = true)\n"," |-- 12/6/20: string (nullable = true)\n"," |-- 12/7/20: string (nullable = true)\n"," |-- 12/8/20: string (nullable = true)\n"," |-- 12/9/20: string (nullable = true)\n"," |-- 12/10/20: string (nullable = true)\n"," |-- 12/11/20: string (nullable = true)\n"," |-- 12/12/20: string (nullable = true)\n"," |-- 12/13/20: string (nullable = true)\n"," |-- 12/14/20: string (nullable = true)\n"," |-- 12/15/20: string (nullable = true)\n"," |-- 12/16/20: string (nullable = true)\n"," |-- 12/17/20: string (nullable = true)\n"," |-- 12/18/20: string (nullable = true)\n"," |-- 12/19/20: string (nullable = true)\n"," |-- 12/20/20: string (nullable = true)\n"," |-- 12/21/20: string (nullable = true)\n"," |-- 12/22/20: string (nullable = true)\n"," |-- 12/23/20: string (nullable = true)\n"," |-- 12/24/20: string (nullable = true)\n"," |-- 12/25/20: string (nullable = true)\n"," |-- 12/26/20: string (nullable = true)\n"," |-- 12/27/20: string (nullable = true)\n"," |-- 12/28/20: string (nullable = true)\n"," |-- 12/29/20: string (nullable = true)\n"," |-- 12/30/20: string (nullable = true)\n"," |-- 12/31/20: string (nullable = true)\n"," |-- 1/1/21: string (nullable = true)\n"," |-- 1/2/21: string (nullable = true)\n"," |-- 1/3/21: string (nullable = true)\n"," |-- 1/4/21: string (nullable = true)\n"," |-- 1/5/21: string (nullable = true)\n"," |-- 1/6/21: string (nullable = true)\n"," |-- 1/7/21: string (nullable = true)\n"," |-- 1/8/21: string (nullable = true)\n"," |-- 1/9/21: string (nullable = true)\n"," |-- 1/10/21: string (nullable = true)\n"," |-- 1/11/21: string (nullable = true)\n"," |-- 1/12/21: string (nullable = true)\n"," |-- 1/13/21: string (nullable = true)\n"," |-- 1/14/21: string (nullable = true)\n"," |-- 1/15/21: string (nullable = true)\n"," |-- 1/16/21: string (nullable = true)\n"," |-- 1/17/21: string (nullable = true)\n"," |-- 1/18/21: string (nullable = true)\n"," |-- 1/19/21: string (nullable = true)\n"," |-- 1/20/21: string (nullable = true)\n"," |-- 1/21/21: string (nullable = true)\n"," |-- 1/22/21: string (nullable = true)\n"," |-- 1/23/21: string (nullable = true)\n"," |-- 1/24/21: string (nullable = true)\n"," |-- 1/25/21: string (nullable = true)\n"," |-- 1/26/21: string (nullable = true)\n"," |-- 1/27/21: string (nullable = true)\n"," |-- 1/28/21: string (nullable = true)\n"," |-- 1/29/21: string (nullable = true)\n"," |-- 1/30/21: string (nullable = true)\n"," |-- 1/31/21: string (nullable = true)\n"," |-- 2/1/21: string (nullable = true)\n"," |-- 2/2/21: string (nullable = true)\n"," |-- 2/3/21: string (nullable = true)\n"," |-- 2/4/21: string (nullable = true)\n"," |-- 2/5/21: string (nullable = true)\n"," |-- 2/6/21: string (nullable = true)\n"," |-- 2/7/21: string (nullable = true)\n"," |-- 2/8/21: string (nullable = true)\n"," |-- 2/9/21: string (nullable = true)\n"," |-- 2/10/21: string (nullable = true)\n"," |-- 2/11/21: string (nullable = true)\n"," |-- 2/12/21: string (nullable = true)\n"," |-- 2/13/21: string (nullable = true)\n"," |-- 2/14/21: string (nullable = true)\n"," |-- 2/15/21: string (nullable = true)\n"," |-- 2/16/21: string (nullable = true)\n"," |-- 2/17/21: string (nullable = true)\n"," |-- 2/18/21: string (nullable = true)\n"," |-- 2/19/21: string (nullable = true)\n"," |-- 2/20/21: string (nullable = true)\n"," |-- 2/21/21: string (nullable = true)\n"," |-- 2/22/21: string (nullable = true)\n"," |-- 2/23/21: string (nullable = true)\n"," |-- 2/24/21: string (nullable = true)\n"," |-- 2/25/21: string (nullable = true)\n"," |-- 2/26/21: string (nullable = true)\n"," |-- 2/27/21: string (nullable = true)\n"," |-- 2/28/21: string (nullable = true)\n"," |-- 3/1/21: string (nullable = true)\n"," |-- 3/2/21: string (nullable = true)\n"," |-- 3/3/21: string (nullable = true)\n"," |-- 3/4/21: string (nullable = true)\n"," |-- 3/5/21: string (nullable = true)\n"," |-- 3/6/21: string (nullable = true)\n"," |-- 3/7/21: string (nullable = true)\n"," |-- 3/8/21: string (nullable = true)\n"," |-- 3/9/21: string (nullable = true)\n"," |-- 3/10/21: string (nullable = true)\n"," |-- 3/11/21: string (nullable = true)\n"," |-- 3/12/21: string (nullable = true)\n"," |-- 3/13/21: string (nullable = true)\n"," |-- 3/14/21: string (nullable = true)\n"," |-- 3/15/21: string (nullable = true)\n"," |-- 3/16/21: string (nullable = true)\n"," |-- 3/17/21: string (nullable = true)\n"," |-- 3/18/21: string (nullable = true)\n"," |-- 3/19/21: string (nullable = true)\n"," |-- 3/20/21: string (nullable = true)\n"," |-- 3/21/21: string (nullable = true)\n"," |-- 3/22/21: string (nullable = true)\n"," |-- 3/23/21: string (nullable = true)\n"," |-- 3/24/21: string (nullable = true)\n"," |-- 3/25/21: string (nullable = true)\n"," |-- 3/26/21: string (nullable = true)\n"," |-- 3/27/21: string (nullable = true)\n"," |-- 3/28/21: string (nullable = true)\n"," |-- 3/29/21: string (nullable = true)\n"," |-- 3/30/21: string (nullable = true)\n"," |-- 3/31/21: string (nullable = true)\n"," |-- 4/1/21: string (nullable = true)\n"," |-- 4/2/21: string (nullable = true)\n"," |-- 4/3/21: string (nullable = true)\n"," |-- 4/4/21: string (nullable = true)\n"," |-- 4/5/21: string (nullable = true)\n"," |-- 4/6/21: string (nullable = true)\n"," |-- 4/7/21: string (nullable = true)\n"," |-- 4/8/21: string (nullable = true)\n"," |-- 4/9/21: string (nullable = true)\n"," |-- 4/10/21: string (nullable = true)\n"," |-- 4/11/21: string (nullable = true)\n"," |-- 4/12/21: string (nullable = true)\n"," |-- 4/13/21: string (nullable = true)\n"," |-- 4/14/21: string (nullable = true)\n"," |-- 4/15/21: string (nullable = true)\n"," |-- 4/16/21: string (nullable = true)\n"," |-- 4/17/21: string (nullable = true)\n"," |-- 4/18/21: string (nullable = true)\n"," |-- 4/19/21: string (nullable = true)\n"," |-- 4/20/21: string (nullable = true)\n"," |-- 4/21/21: string (nullable = true)\n"," |-- 4/22/21: string (nullable = true)\n"," |-- 4/23/21: string (nullable = true)\n"," |-- 4/24/21: string (nullable = true)\n"," |-- 4/25/21: string (nullable = true)\n"," |-- 4/26/21: string (nullable = true)\n"," |-- 4/27/21: string (nullable = true)\n"," |-- 4/28/21: string (nullable = true)\n"," |-- 4/29/21: string (nullable = true)\n"," |-- 4/30/21: string (nullable = true)\n"," |-- 5/1/21: string (nullable = true)\n"," |-- 5/2/21: string (nullable = true)\n"," |-- 5/3/21: string (nullable = true)\n"," |-- 5/4/21: string (nullable = true)\n"," |-- 5/5/21: string (nullable = true)\n"," |-- 5/6/21: string (nullable = true)\n"," |-- 5/7/21: string (nullable = true)\n"," |-- 5/8/21: string (nullable = true)\n"," |-- 5/9/21: string (nullable = true)\n"," |-- 5/10/21: string (nullable = true)\n"," |-- 5/11/21: string (nullable = true)\n"," |-- 5/12/21: string (nullable = true)\n"," |-- 5/13/21: string (nullable = true)\n"," |-- 5/14/21: string (nullable = true)\n"," |-- 5/15/21: string (nullable = true)\n"," |-- 5/16/21: string (nullable = true)\n"," |-- 5/17/21: string (nullable = true)\n"," |-- 5/18/21: string (nullable = true)\n","\n"]}]},{"cell_type":"markdown","metadata":{"id":"WV45mAMfrmxA"},"source":["### Your Task"]},{"cell_type":"markdown","metadata":{"id":"iLLbq024rr9l"},"source":["We are aware of the stress we are all experiencing because of the fact that many of you have projects and exams due this week. As such, we decided to conclude our series of Colabs with a **lightweight task** -- given everything you have learned about Spark during the quarter, this Colab should take you just a few minutes to complete."]},{"cell_type":"markdown","metadata":{"id":"4wYiCEv_zhVf"},"source":["Consider the entries for May 1, 2021, in the timeseries, and compute:\n","\n","\n","*   number of confirmed COVID-19 cases across the globe\n","*   number of deaths due to COVID-19 (across the globe)\n","*   number of recovered patients across the globe\n","\n"]},{"cell_type":"code","metadata":{"id":"w-Hoy-9Xzf8r","colab":{"base_uri":"https://localhost:8080/","height":35},"executionInfo":{"status":"ok","timestamp":1772751723827,"user_tz":480,"elapsed":3,"user":{"displayName":"Rishabh Ranjan","userId":"14643331064616738605"}},"outputId":"9c9248f1-66a1-41e7-b05d-776ea163ca05"},"source":["''' 6 lines of code in total expected. '''\n","\n","# YOUR CODE HERE\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["' 6 lines of code in total expected. '"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":16}]},{"cell_type":"markdown","metadata":{"id":"yahkAHoS0HuU"},"source":["Consider the data points for March 1, 2020, and March 1, 2021, and filter out the geographical locations where less than 50 cases have been confirmed.\n","For the areas still taken into consideration after the filtering step, compute the ratio between number of deaths and number of confirmed cases.\n","\n","(Hint:\n","1. For Q2.5, you don't need to sum over the country(combine Province/State having the same Country/Region). In other words, the column Province/State should exist in the final dataframe for this question.\n","2. For Q2.6, the computed death rate should be in the range [0.2, 0.3] and your reported country should be a country in the middle east.)"]},{"cell_type":"code","metadata":{"id":"8FwzsH4l1VCZ","colab":{"base_uri":"https://localhost:8080/","height":35},"executionInfo":{"status":"ok","timestamp":1772751723829,"user_tz":480,"elapsed":2,"user":{"displayName":"Rishabh Ranjan","userId":"14643331064616738605"}},"outputId":"983f12e8-729e-4a25-9bd8-c4576cd7472a"},"source":["''' 12-25 lines of code in total expected but can differ based on your style. '''\n","''' For sub-parts of the question (if any), creating different cells of code would be recommended.'''\n","\n","# YOUR CODE HERE\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["' For sub-parts of the question (if any), creating different cells of code would be recommended.'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":17}]},{"cell_type":"markdown","metadata":{"id":"MLBtZyFqPKkI"},"source":["Consider the data points for March 1, 2021, and May 1, 2021, in the timeseries, and filter out the geographical locations where less than 50 deaths have been confirmed (as of March 1, 2021).\n","For the areas still taken into consideration after the filtering step, compute **the percent increase in cumulative deaths**(not difference in death rate) between the two dates."]},{"cell_type":"code","metadata":{"id":"FUWisUyMPI9A","colab":{"base_uri":"https://localhost:8080/","height":35},"executionInfo":{"status":"ok","timestamp":1772751723831,"user_tz":480,"elapsed":1,"user":{"displayName":"Rishabh Ranjan","userId":"14643331064616738605"}},"outputId":"cc3db0e1-a0d7-44b0-9cee-86aa607c1cbb"},"source":["''' 4-5 lines of code in total expected. '''\n","\n","# YOUR CODE HERE\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["' 4-5 lines of code in total expected. '"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":18}]},{"cell_type":"markdown","metadata":{"id":"SIrXJyVNP2AI"},"source":["Once you have working code for each cell above, **head over to Gradescope, read the questions carefully, and submit your solution for this Colab**!"]}]}