Programming Assignment Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the second assignment for the Coursera course \"Advanced Machine Learning and Signal Processing\"\n",
    "\n",
    "\n",
    "Just execute all cells one after the other and you are done - just note that in the last one you have to update your email address (the one you've used for coursera) and obtain a submission token, you get this from the programming assignment directly on coursera.\n",
    "\n",
    "Please fill in the sections labelled with \"###YOUR_CODE_GOES_HERE###\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for a Spark session to start...\n",
      "Spark Initialization Done! ApplicationId = app-20190901102723-0000\n",
      "KERNEL_ID = fe6ef039-37c0-4409-8075-b758ea2c7e15\n",
      "--2019-09-01 10:27:25--  https://github.com/IBM/coursera/raw/master/coursera_ml/a2.parquet\n",
      "Resolving github.com (github.com)... 140.82.113.4\n",
      "Connecting to github.com (github.com)|140.82.113.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/IBM/coursera/master/coursera_ml/a2.parquet [following]\n",
      "--2019-09-01 10:27:25--  https://raw.githubusercontent.com/IBM/coursera/master/coursera_ml/a2.parquet\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 59032 (58K) [application/octet-stream]\n",
      "Saving to: 'a2.parquet'\n",
      "\n",
      "a2.parquet          100%[===================>]  57.65K  --.-KB/s    in 0.002s  \n",
      "\n",
      "2019-09-01 10:27:25 (24.4 MB/s) - 'a2.parquet' saved [59032/59032]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://github.com/IBM/coursera/raw/master/coursera_ml/a2.parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it’s time to have a look at the recorded sensor data. You should see data similar to the one exemplified below….\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "|CLASS|   SENSORID|                  X|                  Y|                  Z|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "|    0|         26| 380.66434005495194| -139.3470983812975|-247.93697521077704|\n",
      "|    0|         29| 104.74324299209692| -32.27421440203938|-25.105013725863852|\n",
      "|    0| 8589934658| 118.11469236129976| 45.916682927433534| -87.97203782706572|\n",
      "|    0|34359738398| 246.55394030642543|-0.6122810693132044|-398.18662513951506|\n",
      "|    0|17179869241|-190.32584900181487|  234.7849657520335|-206.34483804019288|\n",
      "|    0|25769803830| 178.62396382387422| -47.07529438881511|  84.38310769821979|\n",
      "|    0|25769803831|  85.03128805189493|-4.3024316644854546|-1.1841857567516714|\n",
      "|    0|34359738411| 26.786262674736566| -46.33193951911338| 20.880756008396055|\n",
      "|    0| 8589934592|-16.203752396859194| 51.080957032176954| -96.80526656416971|\n",
      "|    0|25769803852|   47.2048142440404|  -78.2950899652916| 181.99604091494786|\n",
      "|    0|34359738369| 15.608872398939273| -79.90322809181754|  69.62150711098005|\n",
      "|    0|         19|-4.8281721129789315| -67.38050508399905| 221.24876396496404|\n",
      "|    0|         54| -98.40725712852762|-19.989364074314732|  -302.695196085276|\n",
      "|    0|17179869313| 22.835845394816594|   17.1633660118843| 32.877914832011385|\n",
      "|    0|34359738454|  84.20178070080324| -32.81572075916947| -48.63517643958031|\n",
      "|    0|          0|  56.54732521345129| -7.980106018032676|  95.05162719436447|\n",
      "|    0|17179869201|  -57.6008655247749|  5.135393798773895| 236.99158698947267|\n",
      "|    0|17179869308| -65.59264738389012| -48.92660057215126| -61.58970715383383|\n",
      "|    0|25769803790|  34.82337351291005|  9.483542084393937|  197.6066372962772|\n",
      "|    0|25769803825|  39.80573823439121|-0.7955236412785212| -79.66652640650325|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df=spark.read.load('a2.parquet')\n",
    "\n",
    "df.createOrReplaceTempView(\"df\")\n",
    "spark.sql(\"SELECT * from df\").show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please create a VectorAssembler which consumes columns X, Y and Z and produces a column “features”\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorAssembler\n",
    "vectorAssembler = VectorAssembler(inputCols=[\"X\",\"Y\",\"Z\"],outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please instantiate a classifier from the SparkML package and assign it to the classifier variable. Make sure to either\n",
    "1.\tRename the “CLASS” column to “label” or\n",
    "2.\tSpecify the label-column correctly to be “CLASS”\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import GBTClassifier\n",
    "\n",
    "classifier = GBTClassifier(labelCol='CLASS',featuresCol='features', maxIter=10)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s train and evaluate…\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "pipeline = Pipeline(stages=[vectorAssembler, classifier])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = pipeline.fit(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = model.transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+----------+\n",
      "|CLASS|   SENSORID|                  X|                  Y|                  Z|            features|       rawPrediction|         probability|prediction|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+----------+\n",
      "|    0|         26| 380.66434005495194| -139.3470983812975|-247.93697521077704|[380.664340054951...|[1.32589766213010...|[0.93412155816733...|       0.0|\n",
      "|    0|         29| 104.74324299209692| -32.27421440203938|-25.105013725863852|[104.743242992096...|[1.32589766213010...|[0.93412155816733...|       0.0|\n",
      "|    0| 8589934658| 118.11469236129976| 45.916682927433534| -87.97203782706572|[118.114692361299...|[1.32680169324328...|[0.93423273625928...|       0.0|\n",
      "|    0|34359738398| 246.55394030642543|-0.6122810693132044|-398.18662513951506|[246.553940306425...|[1.32711813715030...|[0.93427161142354...|       0.0|\n",
      "|    0|17179869241|-190.32584900181487|  234.7849657520335|-206.34483804019288|[-190.32584900181...|[1.32590267922033...|[0.93412217565278...|       0.0|\n",
      "|    0|25769803830| 178.62396382387422| -47.07529438881511|  84.38310769821979|[178.623963823874...|[1.32589766213010...|[0.93412155816733...|       0.0|\n",
      "|    0|25769803831|  85.03128805189493|-4.3024316644854546|-1.1841857567516714|[85.0312880518949...|[1.32621410603712...|[0.93416049441777...|       0.0|\n",
      "|    0|34359738411| 26.786262674736566| -46.33193951911338| 20.880756008396055|[26.7862626747365...|[1.32589766213010...|[0.93412155816733...|       0.0|\n",
      "|    0| 8589934592|-16.203752396859194| 51.080957032176954| -96.80526656416971|[-16.203752396859...|[1.32590267922033...|[0.93412217565278...|       0.0|\n",
      "|    0|25769803852|   47.2048142440404|  -78.2950899652916| 181.99604091494786|[47.2048142440404...|[1.32589955669978...|[0.93412179134479...|       0.0|\n",
      "|    0|34359738369| 15.608872398939273| -79.90322809181754|  69.62150711098005|[15.6088723989392...|[1.32590267922033...|[0.93412217565278...|       0.0|\n",
      "|    0|         19|-4.8281721129789315| -67.38050508399905| 221.24876396496404|[-4.8281721129789...|[1.32396579485090...|[0.93388339065422...|       0.0|\n",
      "|    0|         54| -98.40725712852762|-19.989364074314732|  -302.695196085276|[-98.407257128527...|[1.32590267922033...|[0.93412217565278...|       0.0|\n",
      "|    0|17179869313| 22.835845394816594|   17.1633660118843| 32.877914832011385|[22.8358453948165...|[1.32590267922033...|[0.93412217565278...|       0.0|\n",
      "|    0|34359738454|  84.20178070080324| -32.81572075916947| -48.63517643958031|[84.2017807008032...|[1.32589766213010...|[0.93412155816733...|       0.0|\n",
      "|    0|          0|  56.54732521345129| -7.980106018032676|  95.05162719436447|[56.5473252134512...|[1.32589766213010...|[0.93412155816733...|       0.0|\n",
      "|    0|17179869201|  -57.6008655247749|  5.135393798773895| 236.99158698947267|[-57.600865524774...|[1.33122144395126...|[0.93477377205806...|       0.0|\n",
      "|    0|17179869308| -65.59264738389012| -48.92660057215126| -61.58970715383383|[-65.592647383890...|[1.32590267922033...|[0.93412217565278...|       0.0|\n",
      "|    0|25769803790|  34.82337351291005|  9.483542084393937|  197.6066372962772|[34.8233735129100...|[1.51228073545265...|[0.95367147883022...|       0.0|\n",
      "|    0|25769803825|  39.80573823439121|-0.7955236412785212| -79.66652640650325|[39.8057382343912...|[1.32711813715030...|[0.93427161142354...|       0.0|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prediction.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9986850756081526"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
    "binEval = MulticlassClassificationEvaluator().setMetricName(\"accuracy\") .setPredictionCol(\"prediction\").setLabelCol(\"CLASS\")\n",
    "    \n",
    "binEval.evaluate(prediction) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are happy with the result (I’m happy with > 0.55) please submit your solution to the grader by executing the following cells, please don’t forget to obtain an assignment submission token (secret) from the Coursera’s graders web page and paste it to the “secret” variable below, including your email address you’ve used for Coursera. (0.55 means that you are performing better than random guesses)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -Rf a2_m2.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = prediction.repartition(1)\n",
    "prediction.write.json('a2_m2.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-09-01 10:30:59--  https://raw.githubusercontent.com/IBM/coursera/master/rklib.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2540 (2.5K) [text/plain]\n",
      "Saving to: 'rklib.py'\n",
      "\n",
      "rklib.py            100%[===================>]   2.48K  --.-KB/s    in 0s      \n",
      "\n",
      "2019-09-01 10:31:05 (74.2 MB/s) - 'rklib.py' saved [2540/2540]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "\n",
    "def zipdir(path, ziph):\n",
    "    for root, dirs, files in os.walk(path):\n",
    "        for file in files:\n",
    "            ziph.write(os.path.join(root, file))\n",
    "\n",
    "zipf = zipfile.ZipFile('a2_m2.json.zip', 'w', zipfile.ZIP_DEFLATED)\n",
    "zipdir('a2_m2.json', zipf)\n",
    "zipf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "!base64 a2_m2.json.zip > a2_m2.json.zip.base64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"LTL4F\",\"id\":\"f_F-qCtuEei_fRLwaVDk3g~LTL4F~yIQmA8yjEem5hRKzMl4oag\",\"courseId\":\"f_F-qCtuEei_fRLwaVDk3g\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "from rklib import submit\n",
    "key = \"J3sDL2J8EeiaXhILFWw2-g\"\n",
    "part = \"G4P6f\"\n",
    "email = \"dennislamcv@gmail.com\"\n",
    "secret = \"fcBVsiKx6itWJq73\"\n",
    "\n",
    "with open('a2_m2.json.zip.base64', 'r') as myfile:\n",
    "    data=myfile.read()\n",
    "submit(email, secret, key, part, [part], data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 with Spark",
   "language": "python3",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}