{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[{"file_id":"1Tu7xeYM0qBB-RkqLSnYGrIyzb3q7bYo1","timestamp":1681987493112}]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"kPt5q27L5557"},"source":["# CS246 - Colab 3\n","## K-Means & PCA"]},{"cell_type":"markdown","metadata":{"id":"p0-YhEpP_Ds-"},"source":["### Setup"]},{"cell_type":"markdown","metadata":{"id":"Zsj5WYpR9QId"},"source":["Let's set up Spark on your Colab environment.  Run the cell below!"]},{"cell_type":"code","metadata":{"id":"k-qHai2252mI","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1737968023756,"user_tz":480,"elapsed":32255,"user":{"displayName":"Zhemin Huang","userId":"17477919057836646557"}},"outputId":"23e9010f-491c-4c3e-b4ca-03d741c42ec0"},"source":["!pip install pyspark\n","!pip install -U -q PyDrive2\n","!apt install openjdk-8-jdk-headless -qq\n","import os\n","os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\""],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Requirement already satisfied: pyspark in /usr/local/lib/python3.11/dist-packages (3.5.4)\n","Requirement already satisfied: py4j==0.10.9.7 in /usr/local/lib/python3.11/dist-packages (from pyspark) (0.10.9.7)\n","The following additional packages will be installed:\n","  libxtst6 openjdk-8-jre-headless\n","Suggested packages:\n","  openjdk-8-demo openjdk-8-source libnss-mdns fonts-dejavu-extra fonts-nanum fonts-ipafont-gothic\n","  fonts-ipafont-mincho fonts-wqy-microhei fonts-wqy-zenhei fonts-indic\n","The following NEW packages will be installed:\n","  libxtst6 openjdk-8-jdk-headless openjdk-8-jre-headless\n","0 upgraded, 3 newly installed, 0 to remove and 49 not upgraded.\n","Need to get 39.7 MB of archives.\n","After this operation, 144 MB of additional disk space will be used.\n","Selecting previously unselected package libxtst6:amd64.\n","(Reading database ... 124574 files and directories currently installed.)\n","Preparing to unpack .../libxtst6_2%3a1.2.3-1build4_amd64.deb ...\n","Unpacking libxtst6:amd64 (2:1.2.3-1build4) ...\n","Selecting previously unselected package openjdk-8-jre-headless:amd64.\n","Preparing to unpack .../openjdk-8-jre-headless_8u432-ga~us1-0ubuntu2~22.04_amd64.deb ...\n","Unpacking openjdk-8-jre-headless:amd64 (8u432-ga~us1-0ubuntu2~22.04) ...\n","Selecting previously unselected package openjdk-8-jdk-headless:amd64.\n","Preparing to unpack .../openjdk-8-jdk-headless_8u432-ga~us1-0ubuntu2~22.04_amd64.deb ...\n","Unpacking openjdk-8-jdk-headless:amd64 (8u432-ga~us1-0ubuntu2~22.04) ...\n","Setting up libxtst6:amd64 (2:1.2.3-1build4) ...\n","Setting up openjdk-8-jre-headless:amd64 (8u432-ga~us1-0ubuntu2~22.04) ...\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode\n","Setting up openjdk-8-jdk-headless:amd64 (8u432-ga~us1-0ubuntu2~22.04) ...\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/clhsdb to provide /usr/bin/clhsdb (clhsdb) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/hsdb to provide /usr/bin/hsdb (hsdb) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javah to provide /usr/bin/javah (javah) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode\n","update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode\n","Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\n","/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libtcm.so.1 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libumf.so.0 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libhwloc.so.15 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libur_adapter_level_zero.so.0 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libur_loader.so.0 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libur_adapter_opencl.so.0 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libtcm_debug.so.1 is not a symbolic link\n","\n","/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n","\n"]}]},{"cell_type":"markdown","metadata":{"id":"qwtlO4_m_LbQ"},"source":["Now we import some of the libraries usually needed by our workload.\n","\n","\n","\n"]},{"cell_type":"code","metadata":{"id":"twk-K-jilWK7"},"source":["import pandas as pd\n","import numpy as np\n","import matplotlib.pyplot as plt\n","%matplotlib inline\n","\n","import pyspark\n","from pyspark.sql import *\n","from pyspark.sql.types import *\n","from pyspark.sql.functions import *\n","from pyspark import SparkContext, SparkConf"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"BtrJlMBt1Ela"},"source":["Let's initialize the Spark context."]},{"cell_type":"code","source":["# create the session\n","conf = SparkConf().set(\"spark.ui.port\", \"4050\")\n","\n","# create the context\n","sc = pyspark.SparkContext(conf=conf)\n","spark = SparkSession.builder.getOrCreate()\n","print(spark.sparkContext.master)"],"metadata":{"id":"KH91tEik9zZ3","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1737968050667,"user_tz":480,"elapsed":12111,"user":{"displayName":"Zhemin Huang","userId":"17477919057836646557"}},"outputId":"cad020b5-8c5a-4cb5-dd84-e0339a43e424"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["local[*]\n"]}]},{"cell_type":"markdown","metadata":{"id":"kAYRX2PMm0L6"},"source":["### Data Preprocessing"]},{"cell_type":"markdown","metadata":{"id":"7hXdMR6wnEIM"},"source":["In this Colab, rather than downloading a file from Google Drive, we will load a famous machine learning dataset, the [Breast Cancer Wisconsin dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html), using the ```scikit-learn``` datasets loader."]},{"cell_type":"code","metadata":{"id":"5K93ABEy9Zlo"},"source":["from sklearn.datasets import load_breast_cancer\n","breast_cancer = load_breast_cancer()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"wpsaYOqRxar2"},"source":["For convenience, given that the dataset is small, we first\n","\n","*   construct a Pandas dataframe\n","*   tune the schema\n","*   and convert it into a Spark dataframe."]},{"cell_type":"code","metadata":{"id":"Oitav_xhQD9w","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1681990460890,"user_tz":420,"elapsed":7584,"user":{"displayName":"Nirali Sharad Parekh","userId":"02249563440626634370"}},"outputId":"6ae9b9ff-629d-45c5-f490-ea0e750ffc0e"},"source":["pd_df = pd.DataFrame(breast_cancer.data, columns=breast_cancer.feature_names)\n","df = spark.createDataFrame(pd_df)\n","\n","def set_df_columns_nullable(spark, df, column_list, nullable=False):\n","    for struct_field in df.schema:\n","        if struct_field.name in column_list:\n","            struct_field.nullable = nullable\n","    df_mod = spark.createDataFrame(df.rdd, df.schema)\n","    return df_mod\n","\n","df = set_df_columns_nullable(spark, df, df.columns)\n","df = df.withColumn('features', array(df.columns))\n","vectors = df.rdd.map(lambda row: Vectors.dense(row.features))\n","\n","df.printSchema()"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["root\n"," |-- mean radius: double (nullable = false)\n"," |-- mean texture: double (nullable = false)\n"," |-- mean perimeter: double (nullable = false)\n"," |-- mean area: double (nullable = false)\n"," |-- mean smoothness: double (nullable = false)\n"," |-- mean compactness: double (nullable = false)\n"," |-- mean concavity: double (nullable = false)\n"," |-- mean concave points: double (nullable = false)\n"," |-- mean symmetry: double (nullable = false)\n"," |-- mean fractal dimension: double (nullable = false)\n"," |-- radius error: double (nullable = false)\n"," |-- texture error: double (nullable = false)\n"," |-- perimeter error: double (nullable = false)\n"," |-- area error: double (nullable = false)\n"," |-- smoothness error: double (nullable = false)\n"," |-- compactness error: double (nullable = false)\n"," |-- concavity error: double (nullable = false)\n"," |-- concave points error: double (nullable = false)\n"," |-- symmetry error: double (nullable = false)\n"," |-- fractal dimension error: double (nullable = false)\n"," |-- worst radius: double (nullable = false)\n"," |-- worst texture: double (nullable = false)\n"," |-- worst perimeter: double (nullable = false)\n"," |-- worst area: double (nullable = false)\n"," |-- worst smoothness: double (nullable = false)\n"," |-- worst compactness: double (nullable = false)\n"," |-- worst concavity: double (nullable = false)\n"," |-- worst concave points: double (nullable = false)\n"," |-- worst symmetry: double (nullable = false)\n"," |-- worst fractal dimension: double (nullable = false)\n"," |-- features: array (nullable = false)\n"," |    |-- element: double (containsNull = false)\n","\n"]}]},{"cell_type":"markdown","metadata":{"id":"TtR1xRvonxiO"},"source":["With the next cell, we build the two data structures that we will be using throughout this Colab:\n","\n","\n","*   ```features```, a dataframe of Dense vectors, containing all the original features in the dataset;\n","*   ```labels```, a series of binary labels indicating if the corresponding set of features belongs to a subject with breast cancer, or not.\n","\n"]},{"cell_type":"code","metadata":{"id":"GP23Xkgwi0SD"},"source":["from pyspark.ml.linalg import Vectors\n","features = spark.createDataFrame(vectors.map(Row), [\"features\"])\n","labels = pd.Series(breast_cancer.target)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"CRaF2A_j_nC7"},"source":["### Your task"]},{"cell_type":"markdown","metadata":{"id":"ebLNUxP0_8x3"},"source":["If you run successfully the Setup and Data Preprocessing stages, you are now ready to cluster the data with the [K-means](https://spark.apache.org/docs/latest/ml-clustering.html) algorithm included in pyspark.ml (Spark's Machine Learning library).\n","Set the ```k``` parameter to **2** and seed to **1**, fit the model, and the compute the [Silhouette score](https://en.wikipedia.org/wiki/Silhouette_(clustering)) (i.e., a measure of quality of the obtained clustering, here we use squared euclidean distance).  \n","\n","**IMPORTANT:** use the MLlib implementation of the Silhouette score (via ```ClusteringEvaluator```)."]},{"cell_type":"code","metadata":{"id":"0xVIfPHZwWaE"},"source":["''' 8-9 lines of code in total expected but can differ based on your style.\n","for sub-parts of the question, creating different cells of code would be recommended.\n","The running time should be less than 1 minute'''\n","# YOUR CODE HERE\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5GB09n7sqTO6"},"source":["Take the predictions produced by K-means, and compare them with the ```labels``` variable (i.e., the ground truth from our dataset).  \n","\n","Compute how many data points in the dataset have been clustered correctly (i.e., positive cases in one cluster, negative cases in the other), please use the best case scenario since the output cluster ids can be a permutation of labels.\n","\n","*HINT*: you can use ```np.count_nonzero(series_a == series_b)``` to quickly compute the element-wise comparison of two series.\n","\n","**IMPORTANT**: K-means is a clustering algorithm, so it will not output a label for each data point, but just a cluster identifier!  As such, label ```0``` does not necessarily match the cluster identifier ```0```.\n"]},{"cell_type":"code","metadata":{"id":"WQhC3APIPPM5"},"source":["''' 4 lines of code in total expected but can differ based on your style. for sub-parts of the question, creating different cells of code would be recommended.'''\n","# YOUR CODE HERE\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"fLIprM1JsdTU"},"source":["Now perform dimensionality reduction on the ```features``` using the [PCA](https://spark.apache.org/docs/latest/ml-features.html#pca) statistical procedure, available as well in MLlib.\n","\n","Set the ```k``` parameter to **2**, effectively reducing the dataset size of a **15X** factor."]},{"cell_type":"code","metadata":{"id":"p4J8JMDkSb24"},"source":["''' 6 lines of code in total expected but can differ based on your style. for sub-parts of the question, creating different cells of code would be recommended.\n","The running time should be less than 30 seconds. Sanity check: the fourth row in the result should be [-692.6905100570509,38.57692259208171]'''\n","# YOUR CODE HERE\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"8leQR4-atMAl"},"source":["Now run K-means with the same parameters as above, but on the ```pcaFeatures``` produced by the PCA reduction you just executed.\n","\n","Compute the Silhouette score, as well as the number of data points that have been clustered correctly (also the best case scenario)."]},{"cell_type":"code","metadata":{"id":"U_snSSj5k2y5"},"source":["''' 11-13 lines of code in total expected but can differ based on your style. for sub-parts of the question, creating different cells of code would be recommended.'''\n","# YOUR CODE HERE\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SIrXJyVNP2AI"},"source":["Once you obtained the desired results, **head over to Gradescope and submit the `.ipynb` file for this Colab**!"]}]}