Unsupervised Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the third assignment for the Coursera course \"Advanced Machine Learning and Signal Processing\"\n",
    "\n",
    "Just execute all cells one after the other and you are done - just note that in the last one you must update your email address (the one you've used for coursera) and obtain a submission token, you get this from the programming assignment directly on coursera.\n",
    "\n",
    "Please fill in the sections labelled with \"###YOUR_CODE_GOES_HERE###\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for a Spark session to start...\n",
      "Spark Initialization Done! ApplicationId = app-20190901225604-0000\n",
      "KERNEL_ID = fb770cca-016e-4aa4-9418-cd7075667a81\n",
      "--2019-09-01 22:56:06--  https://github.com/IBM/coursera/raw/master/coursera_ml/a2.parquet\n",
      "Resolving github.com (github.com)... 140.82.118.4\n",
      "Connecting to github.com (github.com)|140.82.118.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/IBM/coursera/master/coursera_ml/a2.parquet [following]\n",
      "--2019-09-01 22:56:07--  https://raw.githubusercontent.com/IBM/coursera/master/coursera_ml/a2.parquet\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 59032 (58K) [application/octet-stream]\n",
      "Saving to: 'a2.parquet'\n",
      "\n",
      "a2.parquet          100%[===================>]  57.65K  --.-KB/s    in 0.003s  \n",
      "\n",
      "2019-09-01 22:56:07 (16.9 MB/s) - 'a2.parquet' saved [59032/59032]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://github.com/IBM/coursera/raw/master/coursera_ml/a2.parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it’s time to have a look at the recorded sensor data. You should see data similar to the one exemplified below….\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "|CLASS|   SENSORID|                  X|                  Y|                  Z|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "|    0|         26| 380.66434005495194| -139.3470983812975|-247.93697521077704|\n",
      "|    0|         29| 104.74324299209692| -32.27421440203938|-25.105013725863852|\n",
      "|    0| 8589934658| 118.11469236129976| 45.916682927433534| -87.97203782706572|\n",
      "|    0|34359738398| 246.55394030642543|-0.6122810693132044|-398.18662513951506|\n",
      "|    0|17179869241|-190.32584900181487|  234.7849657520335|-206.34483804019288|\n",
      "|    0|25769803830| 178.62396382387422| -47.07529438881511|  84.38310769821979|\n",
      "|    0|25769803831|  85.03128805189493|-4.3024316644854546|-1.1841857567516714|\n",
      "|    0|34359738411| 26.786262674736566| -46.33193951911338| 20.880756008396055|\n",
      "|    0| 8589934592|-16.203752396859194| 51.080957032176954| -96.80526656416971|\n",
      "|    0|25769803852|   47.2048142440404|  -78.2950899652916| 181.99604091494786|\n",
      "|    0|34359738369| 15.608872398939273| -79.90322809181754|  69.62150711098005|\n",
      "|    0|         19|-4.8281721129789315| -67.38050508399905| 221.24876396496404|\n",
      "|    0|         54| -98.40725712852762|-19.989364074314732|  -302.695196085276|\n",
      "|    0|17179869313| 22.835845394816594|   17.1633660118843| 32.877914832011385|\n",
      "|    0|34359738454|  84.20178070080324| -32.81572075916947| -48.63517643958031|\n",
      "|    0|          0|  56.54732521345129| -7.980106018032676|  95.05162719436447|\n",
      "|    0|17179869201|  -57.6008655247749|  5.135393798773895| 236.99158698947267|\n",
      "|    0|17179869308| -65.59264738389012| -48.92660057215126| -61.58970715383383|\n",
      "|    0|25769803790|  34.82337351291005|  9.483542084393937|  197.6066372962772|\n",
      "|    0|25769803825|  39.80573823439121|-0.7955236412785212| -79.66652640650325|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df=spark.read.load('a2.parquet')\n",
    "\n",
    "df.createOrReplaceTempView(\"df\")\n",
    "spark.sql(\"SELECT * from df\").show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s check if we have balanced classes – this means that we have roughly the same number of examples for each class we want to predict. This is important for classification but also helpful for clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+-----+\n",
      "|count(class)|class|\n",
      "+------------+-----+\n",
      "|        1416|    1|\n",
      "|        1626|    0|\n",
      "+------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"SELECT count(class), class from df group by class\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a VectorAssembler which consumes columns X, Y and Z and produces a column “features”\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorAssembler\n",
    "vectorAssembler = VectorAssembler(inputCols=[\"X\",\"Y\",\"Z\"],\n",
    "                                  outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please insatiate a clustering algorithm from the SparkML package and assign it to the clust variable. Here we don’t need to take care of the “CLASS” column since we are in unsupervised learning mode – so let’s pretend to not even have the “CLASS” column for now – but it will become very handy later in assessing the clustering performance. PLEASE NOTE – IN REAL-WORLD SCENARIOS THERE IS NO CLASS COLUMN – THEREFORE YOU CAN’T ASSESS CLASSIFICATION PERFORMANCE USING THIS COLUMN \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.clustering import KMeans\n",
    "\n",
    "clust = KMeans().setK(12).setSeed(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s train...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "pipeline = Pipeline(stages=[vectorAssembler, clust])\n",
    "model = pipeline.fit(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and evaluate..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+----------+\n",
      "|CLASS|   SENSORID|                  X|                  Y|                  Z|            features|prediction|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+----------+\n",
      "|    0|         26| 380.66434005495194| -139.3470983812975|-247.93697521077704|[380.664340054951...|        10|\n",
      "|    0|         29| 104.74324299209692| -32.27421440203938|-25.105013725863852|[104.743242992096...|         2|\n",
      "|    0| 8589934658| 118.11469236129976| 45.916682927433534| -87.97203782706572|[118.114692361299...|         9|\n",
      "|    0|34359738398| 246.55394030642543|-0.6122810693132044|-398.18662513951506|[246.553940306425...|         1|\n",
      "|    0|17179869241|-190.32584900181487|  234.7849657520335|-206.34483804019288|[-190.32584900181...|         7|\n",
      "|    0|25769803830| 178.62396382387422| -47.07529438881511|  84.38310769821979|[178.623963823874...|         5|\n",
      "|    0|25769803831|  85.03128805189493|-4.3024316644854546|-1.1841857567516714|[85.0312880518949...|         2|\n",
      "|    0|34359738411| 26.786262674736566| -46.33193951911338| 20.880756008396055|[26.7862626747365...|         0|\n",
      "|    0| 8589934592|-16.203752396859194| 51.080957032176954| -96.80526656416971|[-16.203752396859...|         9|\n",
      "|    0|25769803852|   47.2048142440404|  -78.2950899652916| 181.99604091494786|[47.2048142440404...|        11|\n",
      "|    0|34359738369| 15.608872398939273| -79.90322809181754|  69.62150711098005|[15.6088723989392...|         5|\n",
      "|    0|         19|-4.8281721129789315| -67.38050508399905| 221.24876396496404|[-4.8281721129789...|        11|\n",
      "|    0|         54| -98.40725712852762|-19.989364074314732|  -302.695196085276|[-98.407257128527...|         4|\n",
      "|    0|17179869313| 22.835845394816594|   17.1633660118843| 32.877914832011385|[22.8358453948165...|         0|\n",
      "|    0|34359738454|  84.20178070080324| -32.81572075916947| -48.63517643958031|[84.2017807008032...|         2|\n",
      "|    0|          0|  56.54732521345129| -7.980106018032676|  95.05162719436447|[56.5473252134512...|         5|\n",
      "|    0|17179869201|  -57.6008655247749|  5.135393798773895| 236.99158698947267|[-57.600865524774...|         6|\n",
      "|    0|17179869308| -65.59264738389012| -48.92660057215126| -61.58970715383383|[-65.592647383890...|         0|\n",
      "|    0|25769803790|  34.82337351291005|  9.483542084393937|  197.6066372962772|[34.8233735129100...|        11|\n",
      "|    0|25769803825|  39.80573823439121|-0.7955236412785212| -79.66652640650325|[39.8057382343912...|         9|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prediction = model.transform(df)\n",
    "prediction.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9316239316239316"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction.createOrReplaceTempView('prediction')\n",
    "spark.sql('''\n",
    "select max(correct)/max(total) as accuracy from (\n",
    "\n",
    "    select sum(correct) as correct, count(correct) as total from (\n",
    "        select case when class != prediction then 1 else 0 end as correct from prediction \n",
    "    ) \n",
    "    \n",
    "    union\n",
    "    \n",
    "    select sum(correct) as correct, count(correct) as total from (\n",
    "        select case when class = prediction then 1 else 0 end as correct from prediction \n",
    "    ) \n",
    ")\n",
    "''').rdd.map(lambda row: row.accuracy).collect()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you reached at least 55% of accuracy you are fine to submit your predictions to the grader. Otherwise please experiment with parameters setting to your clustering algorithm, use a different algorithm or just re-record your data and try to obtain. In case you are stuck, please use the Coursera Discussion Forum. Please note again – in a real-world scenario there is no way in doing this – since there is no class label in your data. Please have a look at this further reading on clustering performance evaluation https://en.wikipedia.org/wiki/Cluster_analysis#Evaluation_and_assessment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-09-01 22:57:45--  https://raw.githubusercontent.com/IBM/coursera/master/rklib.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2540 (2.5K) [text/plain]\n",
      "Saving to: 'rklib.py'\n",
      "\n",
      "rklib.py            100%[===================>]   2.48K  --.-KB/s    in 0s      \n",
      "\n",
      "2019-09-01 22:57:45 (53.0 MB/s) - 'rklib.py' saved [2540/2540]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -Rf a2_m3.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction= prediction.repartition(1)\n",
    "prediction.write.json('a2_m3.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "\n",
    "def zipdir(path, ziph):\n",
    "    for root, dirs, files in os.walk(path):\n",
    "        for file in files:\n",
    "            ziph.write(os.path.join(root, file))\n",
    "\n",
    "zipf = zipfile.ZipFile('a2_m3.json.zip', 'w', zipfile.ZIP_DEFLATED)\n",
    "zipdir('a2_m3.json', zipf)\n",
    "zipf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "!base64 a2_m3.json.zip > a2_m3.json.zip.base64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"Cu6KW\",\"id\":\"f_F-qCtuEei_fRLwaVDk3g~Cu6KW~AYbkvM0MEemTUBLueEAsDA\",\"courseId\":\"f_F-qCtuEei_fRLwaVDk3g\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "from rklib import submit\n",
    "key = \"pPfm62VXEeiJOBL0dhxPkA\"\n",
    "part = \"EOTMs\"\n",
    "email = \"dennislamcv@gmail.com\"\n",
    "secret = \"EPSyr304ucXcdHm7\"\n",
    "\n",
    "\n",
    "with open('a2_m3.json.zip.base64', 'r') as myfile:\n",
    "    data=myfile.read()\n",
    "submit(email, secret, key, part, [part], data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 with Spark",
   "language": "python3",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}