{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[{"file_id":"1Db3fj_3jlDXgdrFWH6fUe03QF6ZT5EL5","timestamp":1681321624702}]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"kPt5q27L5557"},"source":["# CS246 - Colab 2\n","## Frequent Pattern Mining in Spark"]},{"cell_type":"code","source":["from IPython.display import Image\n","print(\"Colab 2 Mascot\")\n","Image(url='https://cdn.dribbble.com/users/222579/screenshots/1654898/stubby-ben-rex-roll.gif',width=150)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":151},"id":"kcxD8L9ORHZx","executionInfo":{"status":"ok","timestamp":1736559927207,"user_tz":480,"elapsed":102,"user":{"displayName":"Zhemin Huang","userId":"17477919057836646557"}},"outputId":"c33f88af-aa06-4ddb-cf07-b7484c1ffb8f"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Colab 2 Mascot\n"]},{"output_type":"execute_result","data":{"text/html":["<img src=\"https://cdn.dribbble.com/users/222579/screenshots/1654898/stubby-ben-rex-roll.gif\" width=\"150\"/>"],"text/plain":["<IPython.core.display.Image object>"]},"metadata":{},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"p0-YhEpP_Ds-"},"source":["### Setup"]},{"cell_type":"markdown","metadata":{"id":"Zsj5WYpR9QId"},"source":["Let's set up Spark on your Colab environment.  Run the cell below!"]},{"cell_type":"code","metadata":{"id":"k-qHai2252mI"},"source":["!pip install pyspark\n","!pip install -U -q PyDrive2\n","!apt install openjdk-8-jdk-headless -qq\n","import os\n","os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\""],"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from google.colab import drive\n","drive.mount('/content/drive')"],"metadata":{"id":"anafrcL39KZZ"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-CJ71AKe91eh"},"source":["Now we authenticate a Google Drive client to download the file we will be processing in our Spark job.\n","\n","**Make sure to follow the interactive instructions.**"]},{"cell_type":"code","metadata":{"id":"5K93ABEy9Zlo"},"source":["from pydrive2.auth import GoogleAuth\n","from pydrive2.drive import GoogleDrive\n","from google.colab import auth\n","from oauth2client.client import GoogleCredentials\n","\n","# Authenticate and create the PyDrive client\n","auth.authenticate_user()\n","gauth = GoogleAuth()\n","gauth.credentials = GoogleCredentials.get_application_default()\n","drive = GoogleDrive(gauth)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"0orRvrc1-545"},"source":["id='1dhi1F78ssqR8gE6U-AgB80ZW7V_9snX4'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('products.csv')\n","\n","id='1KZBNEaIyMTcsRV817us6uLZgm-Mii8oU'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('order_products__train.csv')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qwtlO4_m_LbQ"},"source":["If you executed the cells above, you should be able to see the dataset we will need for this Colab under the \"Files\" tab on the left panel."]},{"cell_type":"code","metadata":{"id":"twk-K-jilWK7"},"source":["import pandas as pd\n","import numpy as np\n","import matplotlib.pyplot as plt\n","%matplotlib inline\n","\n","import pyspark\n","from pyspark.sql import *\n","from pyspark.sql.functions import *\n","from pyspark import SparkContext, SparkConf"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"dr-8fK-1lmY0"},"source":["Let's initialize the Spark context."]},{"cell_type":"code","metadata":{"id":"UOwtm2l7lePt"},"source":["# create the session\n","conf = SparkConf().set(\"spark.ui.port\", \"4050\")\n","\n","# create the context\n","sc = pyspark.SparkContext(conf=conf)\n","spark = SparkSession.builder.getOrCreate()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"CRaF2A_j_nC7"},"source":["### Your task"]},{"cell_type":"markdown","metadata":{"id":"ebLNUxP0_8x3"},"source":["If you run successfully the setup stage, you are ready to work with the **3 Million Instacart Orders** dataset. In case you want to read more about it, check the [official Instacart blog post](https://tech.instacart.com/3-million-instacart-orders-open-sourced-d40d29ead6f2) about it, a concise [schema description](https://gist.github.com/jeremystan/c3b39d947d9b88b3ccff3147dbcf6c6b) of the dataset, and the [download page](https://www.kaggle.com/c/instacart-market-basket-analysis/data).\n","\n","In this Colab, we will be working only with a small training dataset (~131K orders) to perform fast Frequent Pattern Mining with the FP-Growth algorithm."]},{"cell_type":"code","metadata":{"id":"xu-e7Ph2_ruG"},"source":["products = spark.read.csv('products.csv', header=True, inferSchema=True)\n","orders = spark.read.csv('order_products__train.csv', header=True, inferSchema=True)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"hhxZZRT9syUO"},"source":["products.printSchema()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"8VeRYRz2s1pm"},"source":["orders.printSchema()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"h5muD_Io59CG"},"source":["Use the Spark Dataframe API to join 'products' and 'orders', so that you will be able to see the product names in each transaction (and not only their ids).  Then, group by the orders by 'order_id' to obtain one row per basket (i.e., set of products purchased together by one customer)."]},{"cell_type":"code","metadata":{"id":"zRH4o4p7s7V6"},"source":["''' 2 lines of code expected '''\n","# YOUR CODE HERE"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"EfHoTLAg6qnM"},"source":["In this Colab we will explore [MLlib](https://spark.apache.org/mllib/), Apache Spark's scalable machine learning library. Specifically, you can use its implementation of the [FP-Growth](https://spark.apache.org/docs/latest/ml-frequent-pattern-mining.html#fp-growth) algorithm to perform efficiently Frequent Pattern Mining in Spark.\n","Use the Python example in the documentation, and train a model with\n","\n","```minSupport=0.01``` and ```minConfidence=0.5```\n","\n"]},{"cell_type":"code","metadata":{"id":"boWgxXNns089"},"source":["''' 3 lines of code expected '''\n","# YOUR CODE HERE"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"6kpTVdfD8UiO"},"source":["Compute how many frequent itemsets and association rules were generated by running FP-growth alongside showing the table of top frequent itemsets and association rules.\n"]},{"cell_type":"code","metadata":{"id":"6KYgQ_URunvA"},"source":["''' 5 lines of code in total expected but can differ based on your style. for sub-parts of the question, creating different cells of code would be recommended.'''\n","# YOUR CODE HERE"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qT8Lwm1VAPoN"},"source":["Now retrain the FP-growth model changing only\n","```minsupport=0.001```\n","and compute how many frequent itemsets and association rules were generated.\n"]},{"cell_type":"code","metadata":{"id":"F4LTM9beApYn"},"source":["''' 5 lines of code in total expected but can differ based on your style. for sub-parts of the question, creating different cells of code would be recommended.'''\n","# YOUR CODE HERE"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"k9FOt5jRNFGt"},"source":["To conclude, go to Gradescope and read the remaining questions. We will ask you to inspect the resulting dataframes, and report a few results.\n","\n"]},{"cell_type":"code","metadata":{"id":"wEqWxzTCNS87"},"source":["# YOUR CODE HERE"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"2WSXPflUN76-"},"source":["Once you obtained the desired results, **head over to Gradescope and submit your solution for this Colab**! As a sanity check, for `minSupport=0.01` you should obtain `freq([Limes]) = 6033`"]}]}