{"cells":[{"cell_type":"markdown","metadata":{"id":"kPt5q27L5557"},"source":["# CS246 - Colab 7 (Winter 2026)\n","## Decision Trees in Spark"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZbeOUh3Lgrzp"},"outputs":[],"source":["from IPython.display import Image\n","print(\"Colab 7 Mascot\")\n","Image(url='https://media2.giphy.com/media/kudIERso2pFiE/giphy.gif?cid=ecf05e47bipq7szjsc8vmss2uxmt1zha9gksgdrn1z8mmadj&ep=v1_gifs_search&rid=giphy.gif&ct=g',width=200)"]},{"cell_type":"markdown","metadata":{"id":"p0-YhEpP_Ds-"},"source":["### Setup"]},{"cell_type":"markdown","metadata":{"id":"Zsj5WYpR9QId"},"source":["Let's set up Spark on your Colab environment.  Run the cell below!"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"k-qHai2252mI"},"outputs":[],"source":["!pip install pyspark\n","!pip install -U -q PyDrive2\n","\n","import os\n","os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-17-openjdk-amd64\"\n","os.environ[\"PATH\"] = \"/usr/lib/jvm/java-17-openjdk-amd64/bin:\" + os.environ[\"PATH\"]\n","\n","!java -version  # Should show Java 17"]},{"cell_type":"markdown","metadata":{"id":"PUUjUvXe3Sjk"},"source":["Now we authenticate a Google Drive client to download the files we will be processing in our Spark job.\n","\n","**Make sure to follow the interactive instructions.**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lRElWs_x2mGh"},"outputs":[],"source":["from pydrive2.auth import GoogleAuth\n","from pydrive2.drive import GoogleDrive\n","from google.colab import auth\n","from oauth2client.client import GoogleCredentials\n","\n","# Authenticate and create the PyDrive client\n","auth.authenticate_user()\n","gauth = GoogleAuth()\n","gauth.credentials = GoogleCredentials.get_application_default()\n","drive = GoogleDrive(gauth)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"QHsFTGUy2n1c"},"outputs":[],"source":["id='1aJrdYMVmmnUKYhLTlXtyB0FQ9gYJqCrs'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('mnist-digits-train.txt')\n","\n","id='1yLwxRaJIyrC03yxqbTKpedMmHEF86AAq'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('mnist-digits-test.txt')"]},{"cell_type":"markdown","metadata":{"id":"qwtlO4_m_LbQ"},"source":["If you executed the cells above, you should be able to see the dataset we will use for this Colab under the \"Files\" tab on the left panel.\n","\n","Next, we import some of the common libraries needed for our task."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"twk-K-jilWK7"},"outputs":[],"source":["import pandas as pd\n","import numpy as np\n","import matplotlib.pyplot as plt\n","%matplotlib inline\n","\n","import pyspark\n","from pyspark.sql import *\n","from pyspark.sql.types import *\n","from pyspark.sql.functions import *\n","from pyspark import SparkContext, SparkConf"]},{"cell_type":"markdown","metadata":{"id":"BtrJlMBt1Ela"},"source":["Let's initialize the Spark context."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Vm3sAVeK1EDZ"},"outputs":[],"source":["# create the session\n","conf = SparkConf() \\\n","    .set(\"spark.ui.port\", \"4050\") \\\n","    .set(\"spark.driver.cores\", \"4\") \\\n","    .set(\"spark.driver.memory\", \"16g\") \\\n","# create the context\n","sc = pyspark.SparkContext(conf=conf)\n","spark = SparkSession.builder.getOrCreate()"]},{"cell_type":"markdown","metadata":{"id":"kAYRX2PMm0L6"},"source":["### Data Loading"]},{"cell_type":"markdown","metadata":{"id":"7hXdMR6wnEIM"},"source":["![MNIST](https://upload.wikimedia.org/wikipedia/commons/thumb/2/27/MnistExamples.png/220px-MnistExamples.png)\n","\n","In this Colab, we will be using the famous [MNIST database](https://en.wikipedia.org/wiki/MNIST_database), a large collection of handwritten digits that is widely used for training and testing in the field of machine learning.\n","\n","For your convenience, the dataset has already been converted to the popular LibSVM format, where each digit is represented as a sparse vector of grayscale pixel values."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"XqlePNxzCL4K"},"outputs":[],"source":["training = spark.read.format(\"libsvm\").load(\"mnist-digits-train.txt\")\n","test = spark.read.format(\"libsvm\").load(\"mnist-digits-test.txt\")\n","\n","# Cache data for multiple uses\n","training.cache()\n","test.cache()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5K93ABEy9Zlo"},"outputs":[],"source":["training.show(truncate=False)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MC_m1oygCoEm"},"outputs":[],"source":["training.printSchema()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"81Vgo4ovCqtQ"},"outputs":[],"source":["test.printSchema()"]},{"cell_type":"markdown","metadata":{"id":"CRaF2A_j_nC7"},"source":["### Your task"]},{"cell_type":"markdown","metadata":{"id":"mrHJptH3Tb-3"},"source":["First of all, find out how many instances(samples) we have in our training / test split."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ulXLzPdsTn9W"},"outputs":[],"source":["''' 2 lines of code in total expected. '''\n","\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"AvFaqrp_T1k8"},"source":["Now train a Decision Tree on the training dataset using Spark MLlib.\n","\n","You can refer to the Python example on this documentation page: [https://spark.apache.org/docs/latest/ml-classification-regression.html#decision-tree-classifier](https://spark.apache.org/docs/latest/ml-classification-regression.html#decision-tree-classifier)\n","\n","(Note : You **don't** need to and should **NOT** use Indexer as shown by the example, so that your answers can match our solution answers for Gradescope problems.)\n","\n","Use **seed of 7** (colab 7) for training purposes throughout the colab."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lLUGeyfIUSW-"},"outputs":[],"source":["''' 4-8 lines (this number includes package import) of code in total expected but can differ based on your style.  '''\n","''' For sub-parts of the question (if any), creating different cells of code would be recommended.'''\n","\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"ZobpPmpXVgWm"},"source":["With the Decision Tree you just induced on the training data, predict the labels of the test set.\n","Print the predictions for the first 10 digits, and compare them with the labels."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Un-IC4U7Vp6O"},"outputs":[],"source":["''' 2 lines of code in total expected. '''\n","\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"bqVi3PigZD82"},"source":["The small sample above looks good, but not great!\n","\n","Let's dig deeper. Compute the accuracy of our model, using the ```MulticlassClassificationEvaluator``` from MLlib."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"XljroElIZds2"},"outputs":[],"source":["''' 4-5 lines of code in total expected. '''\n","\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"z66_fSucZ1Tc"},"source":["Find out the max depth of the trained Decision Tree, and its total number of nodes."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"B_eruDchaNQv"},"outputs":[],"source":["''' 1 line of code in total expected. '''\n","\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"9jMDSsX-al5Q"},"source":["It appears that the default settings of the Decision Tree implemented in MLlib did not allow us to train a very powerful model!\n","\n","Before starting to train a Decision Tree, you can tune the max depth it can reach using the [setMaxDepth()](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.classification.DecisionTreeClassifier.html#pyspark.ml.classification.DecisionTreeClassifier.setMaxDepth) method. Train 21 different DTs, varying the max depth from 0 to 20, endpoints included (i.e., [0, 20]). For each value of the parameter, print the accuracy achieved on the test set, and the number of nodes contained in the given DT.\n","\n","For this problem, we say that the decision trees overfit the dataset if the classification accuracy starts to drop.\n","\n","**IMPORTANT:** this parameter sweep can take 30 minutes or more, depending on how busy is your Colab instance. Notice how the induction time grows super-linearly!"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"tOir3f6RbUDe"},"outputs":[],"source":["''' 6-7 lines of code in total expected. '''\n","''' For sub-parts of the question (if any), creating different cells of code would be recommended.'''\n","\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"SIrXJyVNP2AI"},"source":["Once you have working code for each cell above, **head over to Gradescope, read carefully the questions, and submit your solution for this Colab**!"]}],"metadata":{"colab":{"provenance":[{"file_id":"17NdZpsmUkT9kfZukMiM0y0Q6TJ8b9L9i","timestamp":1771531766726}]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0}