{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[{"file_id":"1NUy6HpAixV0JER8kJDeWDNlI0hodv2wX","timestamp":1704769604288}]},"kernelspec":{"name":"python3","display_name":"Python 3"},"gpuClass":"standard"},"cells":[{"cell_type":"markdown","metadata":{"id":"kPt5q27L5557"},"source":["# CS246 - Colab 1\n","## Word Count in Spark"]},{"cell_type":"markdown","metadata":{"id":"p0-YhEpP_Ds-"},"source":["### Setup"]},{"cell_type":"markdown","metadata":{"id":"Zsj5WYpR9QId"},"source":["Let's set up Spark on your Colab environment.  Run the cell below!"]},{"cell_type":"code","metadata":{"id":"k-qHai2252mI"},"source":["!pip install pyspark\n","!pip install -U -q PyDrive2\n","#the output 'xxx is not a symbolic link' will not affect your implementation or execution\n","#to fix 'xxx is not a symbolic link', you can comment out the lines starting from !mv xxxx\n","#you may need to replace xxx.11 with the correct version if other errors come up after colab update\n","#to get the correct version, use !ls /usr/local/lib to find out\n","!mv /usr/local/lib/libtbbmalloc_proxy.so.2 /usr/local/lib/libtbbmalloc_proxy.so.2.backup\n","!mv /usr/local/lib/libtbbmalloc.so.2 /usr/local/lib/libtbbmalloc.so.2.backup\n","!mv /usr/local/lib/libtbbbind_2_5.so.3 /usr/local/lib/libtbbbind_2_5.so.3.backup\n","!mv /usr/local/lib/libtbb.so.12 /usr/local/lib/libtbb.so.12.backup\n","!mv /usr/local/lib/libtbbbind_2_0.so.3 /usr/local/lib/libtbbbind_2_0.so.3.backup\n","!mv /usr/local/lib/libtbbbind.so.3 /usr/local/lib/libtbbbind.so.3.backup\n","!ln -s /usr/local/lib/libtbbmalloc_proxy.so.2.11 /usr/local/lib/libtbbmalloc_proxy.so.2\n","!ln -s /usr/local/lib/libtbbmalloc.so.2.11 /usr/local/lib/libtbbmalloc.so.2\n","!ln -s /usr/local/lib/libtbbbind_2_5.so.3.11 /usr/local/lib/libtbbbind_2_5.so.3\n","!ln -s /usr/local/lib/libtbb.so.12.11 /usr/local/lib/libtbb.so.12\n","!ln -s /usr/local/lib/libtbbbind_2_0.so.3.11 /usr/local/lib/libtbbbind_2_0.so.3\n","!ln -s /usr/local/lib/libtbbbind.so.3.11 /usr/local/lib/libtbbbind.so.3\n","# !sudo ldconfig\n","#If error related to the above execution occurs, you can try commenting out the above 12 lines under pip install PyDrive2 (not included)\n","!apt install openjdk-8-jdk-headless -qq\n","import os\n","os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n","#the output 'xxx is not a symbolic link' will not affect your implementation or execution\n","#to fix 'xxx is not a symbolic link', you can comment out the lines starting from !mv xxxx\n","#you may need to replace xxx.11 with the correct version if other errors come up after colab update\n","#to get the correct version, use !ls /usr/local/lib to find out\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-CJ71AKe91eh"},"source":["Now we authenticate a Google Drive client to download the file we will be processing in our Spark job.\n","\n","**Make sure to follow the interactive instructions.**"]},{"cell_type":"code","metadata":{"id":"5K93ABEy9Zlo"},"source":["from pydrive2.auth import GoogleAuth\n","from pydrive2.drive import GoogleDrive\n","from google.colab import auth\n","from oauth2client.client import GoogleCredentials\n","\n","# Authenticate and create the PyDrive client\n","auth.authenticate_user()\n","gauth = GoogleAuth()\n","gauth.credentials = GoogleCredentials.get_application_default()\n","drive = GoogleDrive(gauth)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"0orRvrc1-545"},"source":["id='1SE6k_0YukzGd5wK-E4i6mG83nydlfvSa'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('pg100.txt')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qwtlO4_m_LbQ"},"source":["If you executed the cells above, you should be able to see the file *pg100.txt* under the \"Files\" tab on the left panel."]},{"cell_type":"markdown","metadata":{"id":"CRaF2A_j_nC7"},"source":["### Your task"]},{"cell_type":"markdown","metadata":{"id":"ebLNUxP0_8x3"},"source":["If you successfully run the setup stage, you are ready to work on the *pg100.txt* file which contains a copy of the complete works of Shakespeare.\n","\n","Write a Spark application which outputs the number of words that start with each letter. This means that for every letter, we want to count the total number of (non-unique) words that start with a specific letter. (If a specific (aka unique) word that starts with letter 'a' appears N times, it should be counted in words starting with 'a' N times.)\n","\n","In your implementation, **ignore the letter case**, i.e., consider all words as lower case. Also, you can ignore all words that **start** with a non-alphabetic character. You should output word counts for the **entire document**, inclusive of the title, author, and the main texts. If you encounter words broken as a result of new lines, e.g. \"pro-ject\" where the segment after the dash sign is on a new line, no special processing is needed and you can safely consider it as two words (\"pro\" and \"ject\").\n","\n","Your outputs will be graded on a range -- if your differences from the ground-truths are within an error threshold of 5, you'll be considered correct.\n","\n","**Hint:**\n","1. split only on space (' ') but not hyphen/dash ('-') or other symbols.\n","2. you may find spark functions explode (https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.explode.html) and split (https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.split.html) helpful, but you don't need to restrict to them as long as you can satisfy your goal.\n","\n","Clarification:\n","\n","1. If a word 'project' is separated into two lines in the form of 'pro-' in the first line and 'ject' in the second line, it should be treated as two words (when you import the text using spark.read.text, it treats each newline as a new row in the DataFrame). However, for the word 'self-love' that appears in a single line, it should be treated as one word starting with letter 's'.\n","\n","2. As a sanity check, the number of words that start with \"b\" should be 45455. Note that you will be considered correct as long as your answer is within the error threshold of 5."]},{"cell_type":"code","metadata":{"id":"xu-e7Ph2_ruG"},"source":["from pyspark.sql import *\n","from pyspark.sql.functions import *\n","from pyspark import SparkContext\n","import pandas as pd\n","\n","# create the Spark Session\n","spark = SparkSession.builder.getOrCreate()\n","\n","# create the Spark Context\n","sc = spark.sparkContext"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"AuAxGFPFB43Y"},"source":["# YOUR\n","txt = spark.read.text(\"\")"],"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# CODE"],"metadata":{"id":"fU-euN6nX9-a"},"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"7jDCs412ZZcF"},"source":["# HERE"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SIrXJyVNP2AI"},"source":["Once you obtained the desired results, **head over to Gradescope and submit your solution for this Colab**!"]}]}