05. Linear Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression Example\n",
    "\n",
    "Let's walk through the steps of the official documentation example. Doing this will help your ability to read from the documentation, understand it, and then apply it to your own problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import findspark\n",
    "findspark.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName('Linear Regression').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.regression import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load training data\n",
    "training = spark.read.format(\"libsvm\").load(\"data/sample_linear_regression_data.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting! We haven't seen libsvm formats before. In fact the aren't very popular when working with datasets in Python, but the Spark Documentation makes use of them a lot because of their formatting. Let's see what the training data looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+--------------------+\n",
      "|              label|            features|\n",
      "+-------------------+--------------------+\n",
      "| -9.490009878824548|(10,[0,1,2,3,4,5,...|\n",
      "| 0.2577820163584905|(10,[0,1,2,3,4,5,...|\n",
      "| -4.438869807456516|(10,[0,1,2,3,4,5,...|\n",
      "|-19.782762789614537|(10,[0,1,2,3,4,5,...|\n",
      "| -7.966593841555266|(10,[0,1,2,3,4,5,...|\n",
      "| -7.896274316726144|(10,[0,1,2,3,4,5,...|\n",
      "| -8.464803554195287|(10,[0,1,2,3,4,5,...|\n",
      "| 2.1214592666251364|(10,[0,1,2,3,4,5,...|\n",
      "| 1.0720117616524107|(10,[0,1,2,3,4,5,...|\n",
      "|-13.772441561702871|(10,[0,1,2,3,4,5,...|\n",
      "| -5.082010756207233|(10,[0,1,2,3,4,5,...|\n",
      "|  7.887786536531237|(10,[0,1,2,3,4,5,...|\n",
      "| 14.323146365332388|(10,[0,1,2,3,4,5,...|\n",
      "|-20.057482615789212|(10,[0,1,2,3,4,5,...|\n",
      "|-0.8995693247765151|(10,[0,1,2,3,4,5,...|\n",
      "| -19.16829262296376|(10,[0,1,2,3,4,5,...|\n",
      "|  5.601801561245534|(10,[0,1,2,3,4,5,...|\n",
      "|-3.2256352187273354|(10,[0,1,2,3,4,5,...|\n",
      "| 1.5299675726687754|(10,[0,1,2,3,4,5,...|\n",
      "| -0.250102447941961|(10,[0,1,2,3,4,5,...|\n",
      "+-------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "training.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the format that Spark expects. Two columns with the names \"label\" and \"features\". \n",
    "\n",
    "The \"label\" column then needs to have the numerical label, either a regression numerical value, or a numerical value that matches to a classification grouping. Later on we will talk about unsupervised learning algorithms that by their nature do not use or require a label.\n",
    "\n",
    "The feature column has inside of it a vector of all the features that belong to that row. Usually what we end up doing is combining the various feature columns we have into a single 'features' column using the data transformations we've learned about.\n",
    "\n",
    "Let's continue working through this simple example!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# These are the default values for the featuresCol, labelCol, predictionCol\n",
    "lr = LinearRegression(featuresCol='features', labelCol='label', predictionCol='prediction')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model\n",
    "lrModel = lr.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coefficients: [0.0073350710225801715,0.8313757584337543,-0.8095307954684084,2.441191686884721,0.5191713795290003,1.1534591903547016,-0.2989124112808717,-0.5128514186201779,-0.619712827067017,0.6956151804322931]\n",
      "\n",
      "\n",
      "Intercept:0.14228558260358093\n"
     ]
    }
   ],
   "source": [
    "# Print the coefficients and intercept for linear regression\n",
    "print(\"Coefficients: {}\".format(str(lrModel.coefficients))) # For each feature...\n",
    "print('\\n')\n",
    "print(\"Intercept:{}\".format(str(lrModel.intercept)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a summary attribute that contains even more info!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Summarize the model over the training set and print out some metrics\n",
    "trainingSummary = lrModel.summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lots of info, here are a few examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|          residuals|\n",
      "+-------------------+\n",
      "|-11.011130022096554|\n",
      "| 0.9236590911176538|\n",
      "|-4.5957401897776675|\n",
      "|  -20.4201774575836|\n",
      "|-10.339160314788181|\n",
      "|-5.9552091439610555|\n",
      "|-10.726906349283922|\n",
      "|  2.122807193191233|\n",
      "|  4.077122222293811|\n",
      "|-17.316168071241652|\n",
      "| -4.593044343959059|\n",
      "|  6.380476690746936|\n",
      "| 11.320566035059846|\n",
      "|-20.721971774534094|\n",
      "| -2.736692773777401|\n",
      "| -16.66886934252847|\n",
      "|  8.242186378876315|\n",
      "|-1.3723486332690233|\n",
      "|-0.7060332131264666|\n",
      "|-1.1591135969994064|\n",
      "+-------------------+\n",
      "only showing top 20 rows\n",
      "\n",
      "RMSE: 10.16309157133015\n",
      "r2: 0.027839179518600154\n"
     ]
    }
   ],
   "source": [
    "trainingSummary.residuals.show()\n",
    "print(\"RMSE: {}\".format(trainingSummary.rootMeanSquaredError))\n",
    "print(\"r2: {}\".format(trainingSummary.r2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train/Test Splits\n",
    "\n",
    "But wait! We've commited a big mistake, we never separated our data set into a training and test set. Instead we trained on ALL of the data, something we generally want to avoid doing. Read ISLR and check out the theory lecture for more info on this, but remember we won't get a fair evaluation of our model by judging how well it does again on the same data it was trained on!\n",
    "\n",
    "Luckily Spark DataFrames have an almost too convienent method of splitting the data! Let's see it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data = spark.read.format(\"libsvm\").load(\"data/sample_linear_regression_data.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pass in the split between training/test as a list.\n",
    "# No correct, but generally 70/30 or 60/40 splits are used. \n",
    "# Depending on how much data you have and how unbalanced it is.\n",
    "train_data,test_data = all_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+--------------------+\n",
      "|              label|            features|\n",
      "+-------------------+--------------------+\n",
      "|-28.571478869743427|(10,[0,1,2,3,4,5,...|\n",
      "|-28.046018037776633|(10,[0,1,2,3,4,5,...|\n",
      "|-26.805483428483072|(10,[0,1,2,3,4,5,...|\n",
      "|-26.736207182601724|(10,[0,1,2,3,4,5,...|\n",
      "| -23.51088409032297|(10,[0,1,2,3,4,5,...|\n",
      "|-22.837460416919342|(10,[0,1,2,3,4,5,...|\n",
      "|-20.212077258958672|(10,[0,1,2,3,4,5,...|\n",
      "|-20.057482615789212|(10,[0,1,2,3,4,5,...|\n",
      "|-19.884560774273424|(10,[0,1,2,3,4,5,...|\n",
      "|-19.872991038068406|(10,[0,1,2,3,4,5,...|\n",
      "|-19.782762789614537|(10,[0,1,2,3,4,5,...|\n",
      "|-19.402336030214553|(10,[0,1,2,3,4,5,...|\n",
      "| -19.16829262296376|(10,[0,1,2,3,4,5,...|\n",
      "|-18.845922472898582|(10,[0,1,2,3,4,5,...|\n",
      "| -18.27521356600463|(10,[0,1,2,3,4,5,...|\n",
      "|-17.803626188664516|(10,[0,1,2,3,4,5,...|\n",
      "|-17.494200356883344|(10,[0,1,2,3,4,5,...|\n",
      "|-17.428674570939506|(10,[0,1,2,3,4,5,...|\n",
      "| -17.32672073267595|(10,[0,1,2,3,4,5,...|\n",
      "|-17.065399625876015|(10,[0,1,2,3,4,5,...|\n",
      "+-------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+--------------------+\n",
      "|              label|            features|\n",
      "+-------------------+--------------------+\n",
      "|-23.487440120936512|(10,[0,1,2,3,4,5,...|\n",
      "|-22.949825936196074|(10,[0,1,2,3,4,5,...|\n",
      "|-21.432387764165806|(10,[0,1,2,3,4,5,...|\n",
      "| -19.66731861537172|(10,[0,1,2,3,4,5,...|\n",
      "|-16.692207021311106|(10,[0,1,2,3,4,5,...|\n",
      "|-15.780685032623301|(10,[0,1,2,3,4,5,...|\n",
      "|-15.375857723312297|(10,[0,1,2,3,4,5,...|\n",
      "|-15.348871155379253|(10,[0,1,2,3,4,5,...|\n",
      "|-14.328978509075442|(10,[0,1,2,3,4,5,...|\n",
      "|-13.420594775890757|(10,[0,1,2,3,4,5,...|\n",
      "|-13.039928064104615|(10,[0,1,2,3,4,5,...|\n",
      "|-12.773226999251197|(10,[0,1,2,3,4,5,...|\n",
      "|-12.558575788856189|(10,[0,1,2,3,4,5,...|\n",
      "|-12.500773785355054|(10,[0,1,2,3,4,5,...|\n",
      "| -12.46765638103286|(10,[0,1,2,3,4,5,...|\n",
      "|-11.904986902675114|(10,[0,1,2,3,4,5,...|\n",
      "|-11.827072996392571|(10,[0,1,2,3,4,5,...|\n",
      "| -11.43180236554046|(10,[0,1,2,3,4,5,...|\n",
      "|-10.600130341909033|(10,[0,1,2,3,4,5,...|\n",
      "| -10.58331129986813|(10,[0,1,2,3,4,5,...|\n",
      "+-------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "unlabeled_data = test_data.select('features')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            features|\n",
      "+--------------------+\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unlabeled_data.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we only train on the train_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_model = lr.fit(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can directly get a .summary object using the evaluate method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_results = correct_model.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|          residuals|\n",
      "+-------------------+\n",
      "|-22.913414037206493|\n",
      "|-27.056978988040907|\n",
      "|-22.409676776826654|\n",
      "|-20.329810686781038|\n",
      "| -17.71072147527376|\n",
      "| -19.22119026999788|\n",
      "|-16.073665709803368|\n",
      "| -12.94413678930788|\n",
      "| -17.63981679784643|\n",
      "|-12.221197260585644|\n",
      "| -13.46900004724097|\n",
      "| -9.821554232373689|\n",
      "|-12.819409028045213|\n",
      "|-13.579136938806736|\n",
      "|-10.593220202192882|\n",
      "|-11.969705338569307|\n",
      "| -15.39504931911258|\n",
      "| -15.54331443971452|\n",
      "| -11.21303292301279|\n",
      "|-13.302398673988074|\n",
      "+-------------------+\n",
      "only showing top 20 rows\n",
      "\n",
      "RMSE: 9.847385948926645\n"
     ]
    }
   ],
   "source": [
    "test_results.residuals.show()\n",
    "print(\"RMSE: {}\".format(test_results.rootMeanSquaredError))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well that is nice, but realistically we will eventually want to test this model against unlabeled data, after all, that is the whole point of building the model in the first place. We can again do this with a convenient method call, in this case, transform(). Which was actually being called within the evaluate() method. Let's see it in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = correct_model.transform(unlabeled_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------------------+\n",
      "|            features|         prediction|\n",
      "+--------------------+-------------------+\n",
      "|(10,[0,1,2,3,4,5,...| -0.574026083730019|\n",
      "|(10,[0,1,2,3,4,5,...|  4.107153051844833|\n",
      "|(10,[0,1,2,3,4,5,...| 0.9772890126608487|\n",
      "|(10,[0,1,2,3,4,5,...| 0.6624920714093178|\n",
      "|(10,[0,1,2,3,4,5,...| 1.0185144539626556|\n",
      "|(10,[0,1,2,3,4,5,...|  3.440505237374579|\n",
      "|(10,[0,1,2,3,4,5,...| 0.6978079864910695|\n",
      "|(10,[0,1,2,3,4,5,...|-2.4047343660713727|\n",
      "|(10,[0,1,2,3,4,5,...| 3.3108382887709853|\n",
      "|(10,[0,1,2,3,4,5,...|-1.1993975153051144|\n",
      "|(10,[0,1,2,3,4,5,...|0.42907198313635486|\n",
      "|(10,[0,1,2,3,4,5,...|-2.9516727668775085|\n",
      "|(10,[0,1,2,3,4,5,...|0.26083323918902335|\n",
      "|(10,[0,1,2,3,4,5,...| 1.0783631534516807|\n",
      "|(10,[0,1,2,3,4,5,...| -1.874436178839979|\n",
      "|(10,[0,1,2,3,4,5,...|0.06471843589419363|\n",
      "|(10,[0,1,2,3,4,5,...| 3.5679763227200083|\n",
      "|(10,[0,1,2,3,4,5,...|   4.11151207417406|\n",
      "|(10,[0,1,2,3,4,5,...| 0.6129025811037566|\n",
      "|(10,[0,1,2,3,4,5,...|  2.719087374119944|\n",
      "+--------------------+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so this data is a bit meaningless, so let's explore this same process with some data that actually makes a little more intuitive sense!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression Code Along"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically what we do here is examine a dataset with Ecommerce Customer Data for a company's website and mobile app. Then we want to see if we can build a regression model that will predict the customer's yearly spend on the company's product."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing to do is start a Spark Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use Spark to read in the Ecommerce Customers csv file.\n",
    "data = spark.read.csv(\"data/Ecommerce_Customers.csv\",inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Email: string (nullable = true)\n",
      " |-- Address: string (nullable = true)\n",
      " |-- Avatar: string (nullable = true)\n",
      " |-- Avg Session Length: double (nullable = true)\n",
      " |-- Time on App: double (nullable = true)\n",
      " |-- Time on Website: double (nullable = true)\n",
      " |-- Length of Membership: double (nullable = true)\n",
      " |-- Yearly Amount Spent: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print the Schema of the DataFrame\n",
    "data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+\n",
      "|               Email|             Address|          Avatar|Avg Session Length|       Time on App|   Time on Website|Length of Membership|Yearly Amount Spent|\n",
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+\n",
      "|mstephenson@ferna...|835 Frank TunnelW...|          Violet| 34.49726772511229| 12.65565114916675| 39.57766801952616|  4.0826206329529615|  587.9510539684005|\n",
      "|   hduke@hotmail.com|4547 Archer Commo...|       DarkGreen| 31.92627202636016|11.109460728682564|37.268958868297744|    2.66403418213262|  392.2049334443264|\n",
      "|    pallen@yahoo.com|24645 Valerie Uni...|          Bisque|33.000914755642675|11.330278057777512|37.110597442120856|   4.104543202376424| 487.54750486747207|\n",
      "|riverarebecca@gma...|1414 David Throug...|     SaddleBrown| 34.30555662975554|13.717513665142507| 36.72128267790313|   3.120178782748092|  581.8523440352177|\n",
      "|mstephens@davidso...|14023 Rodriguez P...|MediumAquaMarine| 33.33067252364639|12.795188551078114| 37.53665330059473|   4.446308318351434|  599.4060920457634|\n",
      "|alvareznancy@luca...|645 Martha Park A...|     FloralWhite|33.871037879341976|12.026925339755056| 34.47687762925054|   5.493507201364199|   637.102447915074|\n",
      "|katherine20@yahoo...|68388 Reyes Light...|   DarkSlateBlue| 32.02159550138701|11.366348309710526| 36.68377615286961|   4.685017246570912|  521.5721747578274|\n",
      "|  awatkins@yahoo.com|Unit 6538 Box 898...|            Aqua|32.739142938380326| 12.35195897300293| 37.37335885854755|  4.4342734348999375|  549.9041461052942|\n",
      "|vchurch@walter-ma...|860 Lee KeyWest D...|          Salmon| 33.98777289568564|13.386235275676436|37.534497341555735|  3.2734335777477144|  570.2004089636196|\n",
      "|    bonnie69@lin.biz|PSC 2734, Box 525...|           Brown|31.936548618448917|11.814128294972196| 37.14516822352819|   3.202806071553459|  427.1993848953282|\n",
      "|andrew06@peterson...|26104 Alexander G...|          Tomato|33.992572774953736|13.338975447662113| 37.22580613162114|   2.482607770510596|  492.6060127179966|\n",
      "|ryanwerner@freema...|Unit 2413 Box 034...|          Tomato| 33.87936082480498|11.584782999535266| 37.08792607098381|    3.71320920294043|  522.3374046069357|\n",
      "|   knelson@gmail.com|6705 Miller Orcha...|       RoyalBlue|29.532428967057943|10.961298400154098| 37.42021557502538|   4.046423164299585|  408.6403510726275|\n",
      "|wrightpeter@yahoo...|05302 Dunlap Ferr...|          Bisque| 33.19033404372265|12.959226091609382|36.144666700041924|   3.918541839158999|  573.4158673313865|\n",
      "|taylormason@gmail...|7773 Powell Sprin...|        DarkBlue|32.387975853153876|13.148725692056516| 36.61995708279922|   2.494543646659249|  470.4527333009554|\n",
      "| jstark@anderson.com|49558 Ramirez Roa...|            Peru|30.737720372628182|12.636606052000127|36.213763093698624|  3.3578468423262944|  461.7807421962299|\n",
      "| wjennings@gmail.com|6362 Wilson Mount...|      PowderBlue| 32.12538689728784|11.733861690857394|  34.8940927514398|  3.1361327164897803| 457.84769594494855|\n",
      "|rebecca45@hale-ba...|8982 Burton RowWi...|       OliveDrab|32.338899323067196|12.013194694014402| 38.38513659413844|   2.420806160901484| 407.70454754954415|\n",
      "|alejandro75@hotma...|64475 Andre Club ...|            Cyan|32.187812045932155|  14.7153875441565| 38.24411459434352|   1.516575580831944|  452.3156754800354|\n",
      "|samuel46@love-wes...|544 Alexander Hei...|   LightSeaGreen| 32.61785606282345|13.989592555825254|37.190503800397956|   4.064548550437977|   605.061038804892|\n",
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(Email='mstephenson@fernandez.com', Address='835 Frank TunnelWrightmouth, MI 82180-9605', Avatar='Violet', Avg Session Length=34.49726772511229, Time on App=12.65565114916675, Time on Website=39.57766801952616, Length of Membership=4.0826206329529615, Yearly Amount Spent=587.9510539684005)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mstephenson@fernandez.com\n",
      "835 Frank TunnelWrightmouth, MI 82180-9605\n",
      "Violet\n",
      "34.49726772511229\n",
      "12.65565114916675\n",
      "39.57766801952616\n",
      "4.0826206329529615\n",
      "587.9510539684005\n"
     ]
    }
   ],
   "source": [
    "for item in data.head():\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting Up DataFrame for Machine Learning "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A few things we need to do before Spark can accept the data!\n",
    "# It needs to be in the form of two columns\n",
    "# (\"label\",\"features\")\n",
    "\n",
    "# Import VectorAssembler and Vectors\n",
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Email',\n",
       " 'Address',\n",
       " 'Avatar',\n",
       " 'Avg Session Length',\n",
       " 'Time on App',\n",
       " 'Time on Website',\n",
       " 'Length of Membership',\n",
       " 'Yearly Amount Spent']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(\n",
    "    inputCols=[\"Avg Session Length\", \"Time on App\", \n",
    "               \"Time on Website\",'Length of Membership'],\n",
    "    outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = assembler.transform(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            features|\n",
      "+--------------------+\n",
      "|[34.4972677251122...|\n",
      "|[31.9262720263601...|\n",
      "|[33.0009147556426...|\n",
      "|[34.3055566297555...|\n",
      "|[33.3306725236463...|\n",
      "|[33.8710378793419...|\n",
      "|[32.0215955013870...|\n",
      "|[32.7391429383803...|\n",
      "|[33.9877728956856...|\n",
      "|[31.9365486184489...|\n",
      "|[33.9925727749537...|\n",
      "|[33.8793608248049...|\n",
      "|[29.5324289670579...|\n",
      "|[33.1903340437226...|\n",
      "|[32.3879758531538...|\n",
      "|[30.7377203726281...|\n",
      "|[32.1253868972878...|\n",
      "|[32.3388993230671...|\n",
      "|[32.1878120459321...|\n",
      "|[32.6178560628234...|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.select(\"features\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+--------------------+\n",
      "|               Email|             Address|          Avatar|Avg Session Length|       Time on App|   Time on Website|Length of Membership|Yearly Amount Spent|            features|\n",
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+--------------------+\n",
      "|mstephenson@ferna...|835 Frank TunnelW...|          Violet| 34.49726772511229| 12.65565114916675| 39.57766801952616|  4.0826206329529615|  587.9510539684005|[34.4972677251122...|\n",
      "|   hduke@hotmail.com|4547 Archer Commo...|       DarkGreen| 31.92627202636016|11.109460728682564|37.268958868297744|    2.66403418213262|  392.2049334443264|[31.9262720263601...|\n",
      "|    pallen@yahoo.com|24645 Valerie Uni...|          Bisque|33.000914755642675|11.330278057777512|37.110597442120856|   4.104543202376424| 487.54750486747207|[33.0009147556426...|\n",
      "|riverarebecca@gma...|1414 David Throug...|     SaddleBrown| 34.30555662975554|13.717513665142507| 36.72128267790313|   3.120178782748092|  581.8523440352177|[34.3055566297555...|\n",
      "|mstephens@davidso...|14023 Rodriguez P...|MediumAquaMarine| 33.33067252364639|12.795188551078114| 37.53665330059473|   4.446308318351434|  599.4060920457634|[33.3306725236463...|\n",
      "|alvareznancy@luca...|645 Martha Park A...|     FloralWhite|33.871037879341976|12.026925339755056| 34.47687762925054|   5.493507201364199|   637.102447915074|[33.8710378793419...|\n",
      "|katherine20@yahoo...|68388 Reyes Light...|   DarkSlateBlue| 32.02159550138701|11.366348309710526| 36.68377615286961|   4.685017246570912|  521.5721747578274|[32.0215955013870...|\n",
      "|  awatkins@yahoo.com|Unit 6538 Box 898...|            Aqua|32.739142938380326| 12.35195897300293| 37.37335885854755|  4.4342734348999375|  549.9041461052942|[32.7391429383803...|\n",
      "|vchurch@walter-ma...|860 Lee KeyWest D...|          Salmon| 33.98777289568564|13.386235275676436|37.534497341555735|  3.2734335777477144|  570.2004089636196|[33.9877728956856...|\n",
      "|    bonnie69@lin.biz|PSC 2734, Box 525...|           Brown|31.936548618448917|11.814128294972196| 37.14516822352819|   3.202806071553459|  427.1993848953282|[31.9365486184489...|\n",
      "|andrew06@peterson...|26104 Alexander G...|          Tomato|33.992572774953736|13.338975447662113| 37.22580613162114|   2.482607770510596|  492.6060127179966|[33.9925727749537...|\n",
      "|ryanwerner@freema...|Unit 2413 Box 034...|          Tomato| 33.87936082480498|11.584782999535266| 37.08792607098381|    3.71320920294043|  522.3374046069357|[33.8793608248049...|\n",
      "|   knelson@gmail.com|6705 Miller Orcha...|       RoyalBlue|29.532428967057943|10.961298400154098| 37.42021557502538|   4.046423164299585|  408.6403510726275|[29.5324289670579...|\n",
      "|wrightpeter@yahoo...|05302 Dunlap Ferr...|          Bisque| 33.19033404372265|12.959226091609382|36.144666700041924|   3.918541839158999|  573.4158673313865|[33.1903340437226...|\n",
      "|taylormason@gmail...|7773 Powell Sprin...|        DarkBlue|32.387975853153876|13.148725692056516| 36.61995708279922|   2.494543646659249|  470.4527333009554|[32.3879758531538...|\n",
      "| jstark@anderson.com|49558 Ramirez Roa...|            Peru|30.737720372628182|12.636606052000127|36.213763093698624|  3.3578468423262944|  461.7807421962299|[30.7377203726281...|\n",
      "| wjennings@gmail.com|6362 Wilson Mount...|      PowderBlue| 32.12538689728784|11.733861690857394|  34.8940927514398|  3.1361327164897803| 457.84769594494855|[32.1253868972878...|\n",
      "|rebecca45@hale-ba...|8982 Burton RowWi...|       OliveDrab|32.338899323067196|12.013194694014402| 38.38513659413844|   2.420806160901484| 407.70454754954415|[32.3388993230671...|\n",
      "|alejandro75@hotma...|64475 Andre Club ...|            Cyan|32.187812045932155|  14.7153875441565| 38.24411459434352|   1.516575580831944|  452.3156754800354|[32.1878120459321...|\n",
      "|samuel46@love-wes...|544 Alexander Hei...|   LightSeaGreen| 32.61785606282345|13.989592555825254|37.190503800397956|   4.064548550437977|   605.061038804892|[32.6178560628234...|\n",
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_data = output.select(\"features\",'Yearly Amount Spent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data,test_data = final_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                363|\n",
      "|   mean| 501.40787593834455|\n",
      "| stddev|  78.80903721898501|\n",
      "|    min|  275.9184206503857|\n",
      "|    max|  765.5184619388373|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                137|\n",
      "|   mean|  493.7661325815803|\n",
      "| stddev|  80.66749795769643|\n",
      "|    min| 256.67058229005585|\n",
      "|    max|  712.3963268096637|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Linear Regression Model object\n",
    "lr = LinearRegression(labelCol='Yearly Amount Spent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model to the data and call this model lrModel\n",
    "lrModel = lr.fit(train_data,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coefficients: [25.634084354454366,38.89277735480858,0.6741835808988965,61.53276311131052] Intercept: -1058.8950162650913\n"
     ]
    }
   ],
   "source": [
    "# Print the coefficients and intercept for linear regression\n",
    "print(\"Coefficients: {} Intercept: {}\".format(lrModel.coefficients,lrModel.intercept))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_results = lrModel.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|          residuals|\n",
      "+-------------------+\n",
      "|-12.034804552204264|\n",
      "| 10.237417071612185|\n",
      "|-17.529330626595424|\n",
      "|-4.6241749545394555|\n",
      "|-13.212443891400824|\n",
      "| -21.64355837957612|\n",
      "|-1.5331663819712276|\n",
      "|  3.534174116877807|\n",
      "|  2.065186824669979|\n",
      "| -5.806709219396112|\n",
      "| -9.819527937013106|\n",
      "|-1.1287514621923265|\n",
      "| -7.686452401576048|\n",
      "| -18.47414943273526|\n",
      "|  7.282723738195671|\n",
      "| -9.909760515347614|\n",
      "|-2.6115973703316513|\n",
      "| -17.57843602129475|\n",
      "| 4.6032560979255095|\n",
      "| -5.180112946922804|\n",
      "+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Interesting results....\n",
    "test_results.residuals.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "unlabeled_data = test_data.select('features')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = lrModel.transform(unlabeled_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+------------------+\n",
      "|            features|        prediction|\n",
      "+--------------------+------------------+\n",
      "|[30.3931845423455...|331.96367435539787|\n",
      "|[30.7377203726281...| 451.5433251246177|\n",
      "|[30.8162006488763...| 283.6156715750644|\n",
      "|[30.8364326747734...|472.12607538152906|\n",
      "|[31.0662181616375...| 462.1457370990752|\n",
      "|[31.1239743499119...| 508.5906122193419|\n",
      "|[31.2606468698795...| 422.8597976389226|\n",
      "|[31.3584771924370...| 491.6417763325976|\n",
      "|[31.4459724827577...| 482.8117781104586|\n",
      "|[31.5257524169682...|  449.772336029278|\n",
      "|[31.5261978982398...| 418.9140541293509|\n",
      "|[31.6610498227460...|417.48710504209316|\n",
      "|[31.7207699002873...|  546.461385879599|\n",
      "|[31.8164283341993...| 519.5966409363916|\n",
      "|[31.8209982016720...| 417.3925572750177|\n",
      "|[31.8279790554652...|449.91250806228913|\n",
      "|[31.8627411090001...| 558.9097385443783|\n",
      "|[31.9048571310136...| 491.5282934441109|\n",
      "|[31.9480174211613...| 457.3176207949723|\n",
      "|[31.9673209478824...|450.92995418657506|\n",
      "+--------------------+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 9.814729847300555\n",
      "MSE: 96.32892197549238\n"
     ]
    }
   ],
   "source": [
    "print(\"RMSE: {}\".format(test_results.rootMeanSquaredError))\n",
    "print(\"MSE: {}\".format(test_results.meanSquaredError))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent results! Let's see how you handle some more realistically modeled data in the Consulting Project!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Transformations\n",
    "\n",
    "You won't always get data in a convienent format, often you will have to deal with data that is non-numerical, such as customer names, or zipcodes, country names, etc...\n",
    "\n",
    "A big part of working with data is using your own domain knowledge to build an intuition of how to deal with the data, sometimes the best course of action is to drop the data, other times feature-engineering is a good way to go, or you could try to transform the data into something the Machine Learning Algorithms will understand.\n",
    "\n",
    "Spark has several built in methods of dealing with thse transformations, check them all out here: http://spark.apache.org/docs/latest/ml-features.html\n",
    "\n",
    "Let's see some examples of all of this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.csv('data/fake_customers.csv',inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+----------+-----+\n",
      "|   Name|     Phone|Group|\n",
      "+-------+----------+-----+\n",
      "|   John|4085552424|    A|\n",
      "|   Mike|3105552738|    B|\n",
      "| Cassie|4085552424|    B|\n",
      "|  Laura|3105552438|    B|\n",
      "|  Sarah|4085551234|    A|\n",
      "|  David|3105557463|    C|\n",
      "|   Zach|4085553987|    C|\n",
      "|  Kiera|3105552938|    A|\n",
      "|  Alexa|4085559467|    C|\n",
      "|Karissa|3105553475|    A|\n",
      "+-------+----------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Features\n",
    "\n",
    "### StringIndexer\n",
    "\n",
    "We often have to convert string information into numerical information as a categorical feature. This is easily done with the StringIndexer Method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------+-------------+\n",
      "|user_id|category|categoryIndex|\n",
      "+-------+--------+-------------+\n",
      "|      0|       a|          0.0|\n",
      "|      1|       b|          2.0|\n",
      "|      2|       c|          1.0|\n",
      "|      3|       a|          0.0|\n",
      "|      4|       a|          0.0|\n",
      "|      5|       c|          1.0|\n",
      "+-------+--------+-------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.feature import StringIndexer\n",
    "\n",
    "df = spark.createDataFrame(\n",
    "    [(0, \"a\"), (1, \"b\"), (2, \"c\"), (3, \"a\"), (4, \"a\"), (5, \"c\")],\n",
    "    [\"user_id\", \"category\"])\n",
    "\n",
    "indexer = StringIndexer(inputCol=\"category\", outputCol=\"categoryIndex\")\n",
    "indexed = indexer.fit(df).transform(df)\n",
    "indexed.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step would be to encode these categories into \"dummy\" variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VectorIndexer\n",
    "\n",
    "VectorAssembler is a transformer that combines a given list of columns into a single vector column. It is useful for combining raw features and features generated by different feature transformers into a single feature vector, in order to train ML models like logistic regression and decision trees. VectorAssembler accepts the following input column types: all numeric types, boolean type, and vector type. In each row, the values of the input columns will be concatenated into a vector in the specified order. \n",
    "\n",
    "Assume that we have a DataFrame with the columns id, hour, mobile, userFeatures, and clicked:\n",
    "\n",
    "     id | hour | mobile | userFeatures     | clicked\n",
    "    ----|------|--------|------------------|---------\n",
    "     0  | 18   | 1.0    | [0.0, 10.0, 0.5] | 1.0\n",
    "     \n",
    "userFeatures is a vector column that contains three user features. We want to combine hour, mobile, and userFeatures into a single feature vector called features and use it to predict clicked or not. If we set VectorAssembler’s input columns to hour, mobile, and userFeatures and output column to features, after transformation we should get the following DataFrame:\n",
    "\n",
    "     id | hour | mobile | userFeatures     | clicked | features\n",
    "    ----|------|--------|------------------|---------|-----------------------------\n",
    "     0  | 18   | 1.0    | [0.0, 10.0, 0.5] | 1.0     | [18.0, 1.0, 0.0, 10.0, 0.5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+----+------+--------------+-------+\n",
      "| id|hour|mobile|  userFeatures|clicked|\n",
      "+---+----+------+--------------+-------+\n",
      "|  0|  18|   1.0|[0.0,10.0,0.5]|    1.0|\n",
      "+---+----+------+--------------+-------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "\n",
    "dataset = spark.createDataFrame(\n",
    "    [(0, 18, 1.0, Vectors.dense([0.0, 10.0, 0.5]), 1.0)],\n",
    "    [\"id\", \"hour\", \"mobile\", \"userFeatures\", \"clicked\"])\n",
    "dataset.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assembled columns 'hour', 'mobile', 'userFeatures' to vector column 'features'\n",
      "+--------------------+-------+\n",
      "|            features|clicked|\n",
      "+--------------------+-------+\n",
      "|[18.0,1.0,0.0,10....|    1.0|\n",
      "+--------------------+-------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "assembler = VectorAssembler(\n",
    "    inputCols=[\"hour\", \"mobile\", \"userFeatures\"],\n",
    "    outputCol=\"features\")\n",
    "\n",
    "output = assembler.transform(dataset)\n",
    "print(\"Assembled columns 'hour', 'mobile', 'userFeatures' to vector column 'features'\")\n",
    "output.select(\"features\", \"clicked\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There ar emany more data transformations available, we will cover them once we encounter a need for them, for now these were the most important ones.\n",
    "\n",
    "Let's continue on to Linear Regression!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Congratulations! You've been contracted by Hyundai Heavy Industries to help them build a predictive model for some ships. [Hyundai Heavy Industries](http://www.hyundai.eu/en) is one of the world's largest ship manufacturing companies and builds cruise liners.\n",
    "\n",
    "You've been flown to their headquarters in Ulsan, South Korea to help them give accurate estimates of how many crew members a ship will require.\n",
    "\n",
    "They are currently building new ships for some customers and want you to create a model and use it to predict how many crew members the ships will need.\n",
    "\n",
    "Here is what the data looks like so far:\n",
    "\n",
    "    Description: Measurements of ship size, capacity, crew, and age for 158 cruise\n",
    "    ships.\n",
    "\n",
    "\n",
    "    Variables/Columns\n",
    "    Ship Name     1-20\n",
    "    Cruise Line   21-40\n",
    "    Age (as of 2013)   46-48\n",
    "    Tonnage (1000s of tons)   50-56\n",
    "    passengers (100s)   58-64\n",
    "    Length (100s of feet)  66-72\n",
    "    Cabins  (100s)   74-80\n",
    "    Passenger Density   82-88\n",
    "    Crew  (100s)   90-96\n",
    "    \n",
    "It is saved in a csv file for you called \"cruise_ship_info.csv\". Your job is to create a regression model that will help predict how many crew members will be needed for future ships. The client also mentioned that they have found that particular cruise lines will differ in acceptable crew counts, so it is most likely an important feature to include in your analysis! \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.csv('data/cruise_ship_info.csv',inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Ship_name: string (nullable = true)\n",
      " |-- Cruise_line: string (nullable = true)\n",
      " |-- Age: integer (nullable = true)\n",
      " |-- Tonnage: double (nullable = true)\n",
      " |-- passengers: double (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- cabins: double (nullable = true)\n",
      " |-- passenger_density: double (nullable = true)\n",
      " |-- crew: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "|  Ship_name|Cruise_line|Age|           Tonnage|passengers|length|cabins|passenger_density|crew|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "|    Journey|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|\n",
      "|      Quest|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|\n",
      "|Celebration|   Carnival| 26|            47.262|     14.86|  7.22|  7.43|             31.8| 6.7|\n",
      "|   Conquest|   Carnival| 11|             110.0|     29.74|  9.53| 14.88|            36.99|19.1|\n",
      "|    Destiny|   Carnival| 17|           101.353|     26.42|  8.92| 13.21|            38.36|10.0|\n",
      "|    Ecstasy|   Carnival| 22|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Elation|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Fantasy|   Carnival| 23|            70.367|     20.56|  8.55| 10.22|            34.23| 9.2|\n",
      "|Fascination|   Carnival| 19|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Freedom|   Carnival|  6|110.23899999999999|      37.0|  9.51| 14.87|            29.79|11.5|\n",
      "|      Glory|   Carnival| 10|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|\n",
      "|    Holiday|   Carnival| 28|            46.052|     14.52|  7.27|  7.26|            31.72| 6.6|\n",
      "|Imagination|   Carnival| 18|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|Inspiration|   Carnival| 17|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|     Legend|   Carnival| 11|              86.0|     21.24|  9.63| 10.62|            40.49| 9.3|\n",
      "|   Liberty*|   Carnival|  8|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|\n",
      "|    Miracle|   Carnival|  9|              88.5|     21.24|  9.63| 10.62|            41.67|10.3|\n",
      "|   Paradise|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|      Pride|   Carnival| 12|              88.5|     21.24|  9.63| 11.62|            41.67| 9.3|\n",
      "|  Sensation|   Carnival| 20|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "|summary|Ship_name|Cruise_line|               Age|           Tonnage|       passengers|           length|            cabins|passenger_density|             crew|\n",
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "|  count|      158|        158|               158|               158|              158|              158|               158|              158|              158|\n",
      "|   mean| Infinity|       null|15.689873417721518| 71.28467088607599|18.45740506329114|8.130632911392404| 8.830000000000005|39.90094936708861|7.794177215189873|\n",
      "| stddev|      NaN|       null| 7.615691058751413|37.229540025907866|9.677094775143416|1.793473548054825|4.4714172221480615| 8.63921711391542|3.503486564627034|\n",
      "|    min|Adventure|    Azamara|                 4|             2.329|             0.66|             2.79|              0.33|             17.7|             0.59|\n",
      "|    max|Zuiderdam|   Windstar|                48|             220.0|             54.0|            11.82|              27.0|            71.43|             21.0|\n",
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.describe().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with the Cruise_line categorical variable\n",
    "Ship Name is a useless arbitrary string, but the cruise_line itself may be useful. Let's make it into a categorical variable!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupBy('Cruise_line').count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Ship_name='Journey', Cruise_line='Azamara', Age=6, Tonnage=30.276999999999997, passengers=6.94, length=5.94, cabins=3.55, passenger_density=42.64, crew=3.55, cruise_cat=16.0),\n",
       " Row(Ship_name='Quest', Cruise_line='Azamara', Age=6, Tonnage=30.276999999999997, passengers=6.94, length=5.94, cabins=3.55, passenger_density=42.64, crew=3.55, cruise_cat=16.0),\n",
       " Row(Ship_name='Celebration', Cruise_line='Carnival', Age=26, Tonnage=47.262, passengers=14.86, length=7.22, cabins=7.43, passenger_density=31.8, crew=6.7, cruise_cat=1.0),\n",
       " Row(Ship_name='Conquest', Cruise_line='Carnival', Age=11, Tonnage=110.0, passengers=29.74, length=9.53, cabins=14.88, passenger_density=36.99, crew=19.1, cruise_cat=1.0),\n",
       " Row(Ship_name='Destiny', Cruise_line='Carnival', Age=17, Tonnage=101.353, passengers=26.42, length=8.92, cabins=13.21, passenger_density=38.36, crew=10.0, cruise_cat=1.0)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.ml.feature import StringIndexer\n",
    "indexer = StringIndexer(inputCol=\"Cruise_line\", outputCol=\"cruise_cat\")\n",
    "indexed = indexer.fit(df).transform(df)\n",
    "indexed.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Ship_name',\n",
       " 'Cruise_line',\n",
       " 'Age',\n",
       " 'Tonnage',\n",
       " 'passengers',\n",
       " 'length',\n",
       " 'cabins',\n",
       " 'passenger_density',\n",
       " 'crew',\n",
       " 'cruise_cat']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexed.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(\n",
    "  inputCols=['Age',\n",
    "             'Tonnage',\n",
    "             'passengers',\n",
    "             'length',\n",
    "             'cabins',\n",
    "             'passenger_density',\n",
    "             'cruise_cat'],\n",
    "    outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = assembler.transform(indexed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----+\n",
      "|            features|crew|\n",
      "+--------------------+----+\n",
      "|[6.0,30.276999999...|3.55|\n",
      "|[6.0,30.276999999...|3.55|\n",
      "|[26.0,47.262,14.8...| 6.7|\n",
      "|[11.0,110.0,29.74...|19.1|\n",
      "|[17.0,101.353,26....|10.0|\n",
      "|[22.0,70.367,20.5...| 9.2|\n",
      "|[15.0,70.367,20.5...| 9.2|\n",
      "|[23.0,70.367,20.5...| 9.2|\n",
      "|[19.0,70.367,20.5...| 9.2|\n",
      "|[6.0,110.23899999...|11.5|\n",
      "|[10.0,110.0,29.74...|11.6|\n",
      "|[28.0,46.052,14.5...| 6.6|\n",
      "|[18.0,70.367,20.5...| 9.2|\n",
      "|[17.0,70.367,20.5...| 9.2|\n",
      "|[11.0,86.0,21.24,...| 9.3|\n",
      "|[8.0,110.0,29.74,...|11.6|\n",
      "|[9.0,88.5,21.24,9...|10.3|\n",
      "|[15.0,70.367,20.5...| 9.2|\n",
      "|[12.0,88.5,21.24,...| 9.3|\n",
      "|[20.0,70.367,20.5...| 9.2|\n",
      "+--------------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.select(\"features\", \"crew\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "final_data = output.select(\"features\", \"crew\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data,test_data = final_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.regression import LinearRegression\n",
    "# Create a Linear Regression Model object\n",
    "lr = LinearRegression(labelCol='crew')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Fit the model to the data and call this model lrModel\n",
    "lrModel = lr.fit(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coefficients: [-0.0145423814068,0.0137445818936,-0.111000735058,0.422234330769,0.705574105078,-0.00631202648669,0.0306212943631] Intercept: -0.5598623529951635\n"
     ]
    }
   ],
   "source": [
    "# Print the coefficients and intercept for linear regression\n",
    "print(\"Coefficients: {} Intercept: {}\".format(lrModel.coefficients,lrModel.intercept))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_results = lrModel.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.3174339720092743\n",
      "MSE: 1.7356322706041332\n",
      "R2: 0.8671622449217978\n"
     ]
    }
   ],
   "source": [
    "print(\"RMSE: {}\".format(test_results.rootMeanSquaredError))\n",
    "print(\"MSE: {}\".format(test_results.meanSquaredError))\n",
    "print(\"R2: {}\".format(test_results.r2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# R2 of 0.86 is pretty good, let's check the data a little closer\n",
    "from pyspark.sql.functions import corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------+\n",
      "|corr(crew, passengers)|\n",
      "+----------------------+\n",
      "|    0.9152341306065384|\n",
      "+----------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(corr('crew','passengers')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+\n",
      "|corr(crew, cabins)|\n",
      "+------------------+\n",
      "|0.9508226063578497|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(corr('crew','cabins')).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so maybe it does make sense! Well that is good news for us, this is information we can bring to the company!\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}