Movie-Recommendations-ALS.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:43:44.924008Z",
     "start_time": "2020-06-28T09:43:44.905537Z"
    }
   },
   "outputs": [],
   "source": [
    "import findspark\n",
    "findspark.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>`findspark` is a  python library which is used to find the location of the Spark installed on the machine.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:43:58.304623Z",
     "start_time": "2020-06-28T09:43:44.929336Z"
    }
   },
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "\n",
    "spark = SparkSession.builder \\\n",
    "                    .master('local[*]') \\\n",
    "                    .appName('movielens') \\\n",
    "                    .getOrCreate()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>the master node 'local[*]' tells Spark to run locally with as many worker threads as logical cores on your machine.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:02.562757Z",
     "start_time": "2020-06-28T09:43:58.310590Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- userId: integer (nullable = true)\n",
      " |-- movieId: integer (nullable = true)\n",
      " |-- rating: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.types import StructType, StructField, IntegerType, DoubleType,StringType\n",
    "\n",
    "schema = StructType([\n",
    "    StructField(\"userId\", IntegerType()),\n",
    "    StructField(\"movieId\", IntegerType()),\n",
    "    StructField(\"rating\", DoubleType())\n",
    "])\n",
    "\n",
    "ratings = spark.read.csv(\"rating.csv\", header=True, schema=schema).limit(10000)\n",
    "ratings.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> A common issue with spark is that it thinks numbers are strings, and vice versa. So I defined the schema on read with the desired data types.</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:08.518358Z",
     "start_time": "2020-06-28T09:44:02.566848Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+------+\n",
      "|userId|movieId|rating|\n",
      "+------+-------+------+\n",
      "|     1|      2|   3.5|\n",
      "|     1|     29|   3.5|\n",
      "|     1|     32|   3.5|\n",
      "|     1|     47|   3.5|\n",
      "|     1|     50|   3.5|\n",
      "|     1|    112|   3.5|\n",
      "|     1|    151|   4.0|\n",
      "|     1|    223|   4.0|\n",
      "|     1|    253|   4.0|\n",
      "|     1|    260|   4.0|\n",
      "|     1|    293|   4.0|\n",
      "|     1|    296|   4.0|\n",
      "|     1|    318|   4.0|\n",
      "|     1|    337|   3.5|\n",
      "|     1|    367|   3.5|\n",
      "|     1|    541|   4.0|\n",
      "|     1|    589|   3.5|\n",
      "|     1|    593|   3.5|\n",
      "|     1|    653|   3.0|\n",
      "|     1|    919|   3.5|\n",
      "+------+-------+------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ratings.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:10.371228Z",
     "start_time": "2020-06-28T09:44:08.522694Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+\n",
      "|userId|count|\n",
      "+------+-----+\n",
      "|     1|  175|\n",
      "|     2|   61|\n",
      "|     3|  187|\n",
      "|     4|   28|\n",
      "|     5|   66|\n",
      "|     6|   24|\n",
      "|     7|  276|\n",
      "|     8|   70|\n",
      "|     9|   35|\n",
      "|    10|   38|\n",
      "|    11|  504|\n",
      "|    12|   36|\n",
      "|    13|   62|\n",
      "|    14|  243|\n",
      "|    15|   49|\n",
      "|    16|   60|\n",
      "|    17|   26|\n",
      "|    18|  121|\n",
      "|    19|   50|\n",
      "|    20|   28|\n",
      "+------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# To check number of movies each user has rated\n",
    "ratings.groupBy(\"userId\").count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:11.547500Z",
     "start_time": "2020-06-28T09:44:10.378675Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movie with the fewest ratings: \n",
      "+----------+\n",
      "|min(count)|\n",
      "+----------+\n",
      "|         1|\n",
      "+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import rank, col, avg, min\n",
    "# Min number of ratings for movies\n",
    "print(\"Movie with the fewest ratings: \")\n",
    "ratings.groupBy(\"movieId\").count().select(min(\"count\")).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:12.633108Z",
     "start_time": "2020-06-28T09:44:11.555268Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Avg num ratings per movie: \n",
      "+-----------------+\n",
      "|       avg(count)|\n",
      "+-----------------+\n",
      "|3.461405330564209|\n",
      "+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Avg number of ratings per movie\n",
    "print(\"Avg num ratings per movie: \")\n",
    "ratings.groupBy(\"movieId\").count().select(avg(\"count\")).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:13.587215Z",
     "start_time": "2020-06-28T09:44:12.638301Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User with the fewest ratings: \n",
      "+----------+\n",
      "|min(count)|\n",
      "+----------+\n",
      "|        20|\n",
      "+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Min number of ratings for user\n",
    "print(\"User with the fewest ratings: \")\n",
    "ratings.groupBy(\"userId\").count().select(min(\"count\")).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T08:38:15.246546Z",
     "start_time": "2020-06-28T08:38:15.236084Z"
    }
   },
   "source": [
    "<b>Each user has given ratings for atleast 20 movies</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:14.493833Z",
     "start_time": "2020-06-28T09:44:13.595617Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Avg num ratings per user: \n",
      "+------------------+\n",
      "|        avg(count)|\n",
      "+------------------+\n",
      "|109.89010989010988|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Avg number of ratings per users\n",
    "print(\"Avg num ratings per user: \")\n",
    "ratings.groupBy(\"userId\").count().select(avg(\"count\")).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Model Building and Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:14.900637Z",
     "start_time": "2020-06-28T09:44:14.498635Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import the required functions\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "from pyspark.ml.recommendation import ALS\n",
    "from pyspark.ml.tuning import ParamGridBuilder,CrossValidator \n",
    "\n",
    "# Create test and train set\n",
    "(train, test) = ratings.randomSplit([0.8, 0.2], seed = 1234)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:27.034912Z",
     "start_time": "2020-06-28T09:44:14.905578Z"
    }
   },
   "outputs": [],
   "source": [
    "# Creating and training a basic model\n",
    "als = ALS(rank=5, maxIter=10, seed=0, userCol= 'userId', itemCol='movieId', ratingCol=\"rating\",coldStartStrategy=\"drop\",nonnegative=True)\n",
    "model = als.fit(train.select([\"userId\", \"movieId\", \"rating\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:56.836693Z",
     "start_time": "2020-06-28T09:44:27.038916Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE=1.118327786291745\n",
      "+------+-------+------+----------+\n",
      "|userId|movieId|rating|prediction|\n",
      "+------+-------+------+----------+\n",
      "|    54|   1088|   3.0| 2.8224826|\n",
      "|    17|   1580|   4.0| 3.5979238|\n",
      "|    23|   1580|   5.0|  4.184193|\n",
      "|    25|   1580|   1.5| 3.4518485|\n",
      "|    46|   1580|   4.0|  3.137214|\n",
      "|    76|   1591|   1.0| 3.0296664|\n",
      "|    58|   1645|   4.0|  3.402373|\n",
      "|    58|   1959|   4.0| 3.9246402|\n",
      "|     3|   2366|   4.0| 4.0371027|\n",
      "|    77|   3175|   3.0| 3.5477808|\n",
      "|    73|   3175|   4.0| 3.1020522|\n",
      "|    14|   3175|   4.5| 2.3943448|\n",
      "|    69|    540|   3.0| 1.8369429|\n",
      "|    58|    540|   2.0| 2.4598627|\n",
      "|    44|    858|   5.0|  4.484232|\n",
      "|    35|    858|   5.0|  4.887832|\n",
      "|    58|    858|   5.0|  5.212182|\n",
      "|    11|    858|   2.5|  4.303579|\n",
      "|    74|    858|   5.0| 4.4629736|\n",
      "|    85|   1025|   4.5| 2.1133268|\n",
      "+------+-------+------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Evaluating predictions\n",
    "evaluator=RegressionEvaluator(metricName=\"rmse\",labelCol=\"rating\",predictionCol=\"prediction\")\n",
    "predictions=model.transform(test)\n",
    "rmse=evaluator.evaluate(predictions)\n",
    "print(\"RMSE=\"+str(rmse))\n",
    "predictions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>With the baseline model we achieved a RMSE of 1.1.</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning and Cross Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:44:56.863463Z",
     "start_time": "2020-06-28T09:44:56.843200Z"
    }
   },
   "outputs": [],
   "source": [
    "paramGrid = ParamGridBuilder().addGrid(als.rank, [1, 5, 10])\\\n",
    "                              .addGrid(als.maxIter, [20])\\\n",
    "                              .addGrid(als.regParam, [0.05, 0.1, 0.5])\\\n",
    "                              .build()\n",
    "crossval = CrossValidator(estimator=als, estimatorParamMaps=paramGrid, evaluator=evaluator, numFolds=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:04:29.461179Z",
     "start_time": "2020-06-28T09:44:56.868576Z"
    }
   },
   "outputs": [],
   "source": [
    "cvModel = crossval.fit(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:04:29.513007Z",
     "start_time": "2020-06-28T10:04:29.467426Z"
    }
   },
   "outputs": [],
   "source": [
    "predictions = cvModel.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:04:43.299834Z",
     "start_time": "2020-06-28T10:04:29.519593Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The root mean squared error for our model is: 0.9771839049156706\n"
     ]
    }
   ],
   "source": [
    "print('The root mean squared error for our model is: {}'.format(evaluator.evaluate(predictions.na.drop())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T08:54:58.660029Z",
     "start_time": "2020-06-28T08:54:58.647692Z"
    }
   },
   "source": [
    "<b>This time we got an RMSE of 0.98 which is good enough to get decent recommendations, keep in mind that we have only taken ten thousand ratings and there is scope for more hyper parameter tuning.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:04:55.863426Z",
     "start_time": "2020-06-28T10:04:43.303824Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+------+----------+\n",
      "|userId|movieId|rating|prediction|\n",
      "+------+-------+------+----------+\n",
      "|    54|   1088|   3.0| 3.0709338|\n",
      "|    17|   1580|   4.0|  4.146607|\n",
      "|    23|   1580|   5.0| 4.0154433|\n",
      "|    25|   1580|   1.5| 3.4749508|\n",
      "|    46|   1580|   4.0| 3.7126157|\n",
      "|    76|   1591|   1.0| 3.7709544|\n",
      "|    58|   1645|   4.0|  3.096809|\n",
      "|    58|   1959|   4.0| 3.7447603|\n",
      "|     3|   2366|   4.0| 4.9220014|\n",
      "|    77|   3175|   3.0|  3.516558|\n",
      "|    73|   3175|   4.0|    3.2938|\n",
      "|    14|   3175|   4.5| 3.4732938|\n",
      "|    69|    540|   3.0| 3.1697989|\n",
      "|    58|    540|   2.0| 3.8704958|\n",
      "|    44|    858|   5.0|  4.800685|\n",
      "|    35|    858|   5.0| 4.9648337|\n",
      "|    58|    858|   5.0|  5.177353|\n",
      "|    11|    858|   2.5| 5.2545037|\n",
      "|    74|    858|   5.0|  5.044103|\n",
      "|    85|   1025|   4.5| 3.2344704|\n",
      "+------+-------+------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Next I read the movie dataset which has the title and genres for those movies.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:04:55.914852Z",
     "start_time": "2020-06-28T10:04:55.873629Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- userId: integer (nullable = true)\n",
      " |-- movieId: integer (nullable = true)\n",
      " |-- rating: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "schema = StructType([\n",
    "    StructField(\"movieId\", IntegerType()),\n",
    "    StructField(\"title\", StringType()),\n",
    "    StructField(\"genres\", StringType())\n",
    "])\n",
    "\n",
    "movies = spark.read.csv(\"movie.csv\", header=True, schema=schema)\n",
    "ratings.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:04:55.949154Z",
     "start_time": "2020-06-28T10:04:55.922516Z"
    }
   },
   "outputs": [],
   "source": [
    "original = ratings.join(movies,on=['movieId'], how='inner')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:04:55.977347Z",
     "start_time": "2020-06-28T10:04:55.953253Z"
    }
   },
   "outputs": [],
   "source": [
    "preds = predictions.join(movies,on=['movieId'], how='inner')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:07:22.520518Z",
     "start_time": "2020-06-28T09:07:22.502941Z"
    }
   },
   "source": [
    "<b>I perform `INNER JOIN` on movies dataframe using ratings and predictions dataframe.`INNER JOIN` is basically the intersection between the two tables in this case with respect to the column movieId.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:04:57.077291Z",
     "start_time": "2020-06-28T10:04:55.982712Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+------+----------------+--------------------+\n",
      "|movieId|userId|rating|           title|              genres|\n",
      "+-------+------+------+----------------+--------------------+\n",
      "|      1|    91|   4.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    90|   3.5|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    84|   5.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    82|   5.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    80|   3.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    69|   4.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    66|   4.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    59|   4.5|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    58|   5.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    54|   4.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    53|   4.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    47|   1.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    39|   5.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    34|   5.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    31|   3.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    24|   4.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    23|   4.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    22|   3.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    19|   5.0|Toy Story (1995)|Adventure|Animati...|\n",
      "|      1|    16|   3.0|Toy Story (1995)|Adventure|Animati...|\n",
      "+-------+------+------+----------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "original.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:07:07.003356Z",
     "start_time": "2020-06-28T10:06:55.624543Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+------+----------+--------------------+--------------------+\n",
      "|movieId|userId|rating|prediction|               title|              genres|\n",
      "+-------+------+------+----------+--------------------+--------------------+\n",
      "|   1088|    54|   3.0| 3.0709338|Dirty Dancing (1987)|Drama|Musical|Rom...|\n",
      "|   1580|    17|   4.0|  4.146607|Men in Black (a.k...|Action|Comedy|Sci-Fi|\n",
      "|   1580|    23|   5.0| 4.0154433|Men in Black (a.k...|Action|Comedy|Sci-Fi|\n",
      "|   1580|    25|   1.5| 3.4749508|Men in Black (a.k...|Action|Comedy|Sci-Fi|\n",
      "|   1580|    46|   4.0| 3.7126157|Men in Black (a.k...|Action|Comedy|Sci-Fi|\n",
      "|   1591|    76|   1.0| 3.7709544|        Spawn (1997)|Action|Adventure|...|\n",
      "|   1645|    58|   4.0|  3.096809|Devil's Advocate,...|Drama|Mystery|Thr...|\n",
      "|   1959|    58|   4.0| 3.7447603|Out of Africa (1985)|       Drama|Romance|\n",
      "|   2366|     3|   4.0| 4.9220014|    King Kong (1933)|Action|Adventure|...|\n",
      "|   3175|    77|   3.0|  3.516558| Galaxy Quest (1999)|Adventure|Comedy|...|\n",
      "|   3175|    73|   4.0|    3.2938| Galaxy Quest (1999)|Adventure|Comedy|...|\n",
      "|   3175|    14|   4.5| 3.4732938| Galaxy Quest (1999)|Adventure|Comedy|...|\n",
      "|    540|    69|   3.0| 3.1697989|       Sliver (1993)|            Thriller|\n",
      "|    540|    58|   2.0| 3.8704958|       Sliver (1993)|            Thriller|\n",
      "|    858|    44|   5.0|  4.800685|Godfather, The (1...|         Crime|Drama|\n",
      "|    858|    35|   5.0| 4.9648337|Godfather, The (1...|         Crime|Drama|\n",
      "|    858|    58|   5.0|  5.177353|Godfather, The (1...|         Crime|Drama|\n",
      "|    858|    11|   2.5| 5.2545037|Godfather, The (1...|         Crime|Drama|\n",
      "|    858|    74|   5.0|  5.044103|Godfather, The (1...|         Crime|Drama|\n",
      "|   1025|    85|   4.5| 3.2344704|Sword in the Ston...|Animation|Childre...|\n",
      "+-------+------+------+----------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "preds.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:20:43.947032Z",
     "start_time": "2020-06-28T09:20:43.937675Z"
    }
   },
   "source": [
    "<b>Now we will compare the original ratings and the recommendations provided to a particular user.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:06:07.483084Z",
     "start_time": "2020-06-28T10:06:06.358963Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+------+--------------------+--------------------+\n",
      "|movieId|userId|rating|               title|              genres|\n",
      "+-------+------+------+--------------------+--------------------+\n",
      "|    551|    11|   5.0|Nightmare Before ...|Animation|Childre...|\n",
      "|   1591|    11|   5.0|        Spawn (1997)|Action|Adventure|...|\n",
      "|    588|    11|   5.0|      Aladdin (1992)|Adventure|Animati...|\n",
      "|     32|    11|   5.0|Twelve Monkeys (a...|Mystery|Sci-Fi|Th...|\n",
      "|    593|    11|   5.0|Silence of the La...|Crime|Horror|Thri...|\n",
      "|    150|    11|   5.0|    Apollo 13 (1995)|Adventure|Drama|IMAX|\n",
      "|    673|    11|   5.0|    Space Jam (1996)|Adventure|Animati...|\n",
      "|    172|    11|   5.0|Johnny Mnemonic (...|Action|Sci-Fi|Thr...|\n",
      "|    736|    11|   5.0|      Twister (1996)|Action|Adventure|...|\n",
      "|   1073|    11|   5.0|Willy Wonka & the...|Children|Comedy|F...|\n",
      "|   1580|    11|   5.0|Men in Black (a.k...|Action|Comedy|Sci-Fi|\n",
      "|   1196|    11|   5.0|Star Wars: Episod...|Action|Adventure|...|\n",
      "|    260|    11|   5.0|Star Wars: Episod...|Action|Adventure|...|\n",
      "|   1200|    11|   5.0|       Aliens (1986)|Action|Adventure|...|\n",
      "|    356|    11|   5.0| Forrest Gump (1994)|Comedy|Drama|Roma...|\n",
      "|   1214|    11|   5.0|        Alien (1979)|       Horror|Sci-Fi|\n",
      "|    480|    11|   5.0|Jurassic Park (1993)|Action|Adventure|...|\n",
      "|   1225|    11|   5.0|      Amadeus (1984)|               Drama|\n",
      "|    170|    11|   5.0|      Hackers (1995)|Action|Adventure|...|\n",
      "|   1240|    11|   5.0|Terminator, The (...|Action|Sci-Fi|Thr...|\n",
      "+-------+------+------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "original.filter(col('userId')==11).filter(col('rating')>=4.5).sort('rating',ascending=False).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T09:34:09.318926Z",
     "start_time": "2020-06-28T09:34:09.304242Z"
    }
   },
   "source": [
    "<b>The highest rated movies by user 11 belong to Action,Adventure and Sci-Fi.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:06:20.509863Z",
     "start_time": "2020-06-28T10:06:12.387690Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+------+----------+--------------------+--------------------+\n",
      "|movieId|userId|rating|prediction|               title|              genres|\n",
      "+-------+------+------+----------+--------------------+--------------------+\n",
      "|   8371|    11|   4.5| 6.0230994|Chronicles of Rid...|Action|Sci-Fi|Thr...|\n",
      "|  54001|    11|   5.0|  5.576076|Harry Potter and ...|Adventure|Drama|F...|\n",
      "|    858|    11|   2.5| 5.2545037|Godfather, The (1...|         Crime|Drama|\n",
      "|   4006|    11|   2.0|  5.180258|Transformers: The...|Adventure|Animati...|\n",
      "|  71282|    11|   5.0|  5.096739|   Food, Inc. (2008)|         Documentary|\n",
      "|  33493|    11|   5.0|  5.013171|Star Wars: Episod...|Action|Adventure|...|\n",
      "|   5378|    11|   4.5|  5.013171|Star Wars: Episod...|Action|Adventure|...|\n",
      "|    318|    11|   5.0| 4.9973865|Shawshank Redempt...|         Crime|Drama|\n",
      "|   2762|    11|   5.0| 4.9828043|Sixth Sense, The ...|Drama|Horror|Mystery|\n",
      "|   4232|    11|   2.5| 4.8184795|     Spy Kids (2001)|Action|Adventure|...|\n",
      "|   8907|    11|   4.0| 4.8184795|   Shark Tale (2004)|Animation|Childre...|\n",
      "|   7153|    11|   5.0|  4.719417|Lord of the Rings...|Action|Adventure|...|\n",
      "|    541|    11|   4.5| 4.7148547| Blade Runner (1982)|Action|Sci-Fi|Thr...|\n",
      "|   1240|    11|   5.0| 4.7070827|Terminator, The (...|Action|Sci-Fi|Thr...|\n",
      "|  48394|    11|   4.5| 4.6798334|Pan's Labyrinth (...|Drama|Fantasy|Thr...|\n",
      "|    356|    11|   5.0| 4.6780057| Forrest Gump (1994)|Comedy|Drama|Roma...|\n",
      "|  52281|    11|   4.0| 4.6532893|   Grindhouse (2007)|Action|Crime|Horr...|\n",
      "|   2313|    11|   4.5| 4.5598598|Elephant Man, The...|               Drama|\n",
      "|   1198|    11|   4.0| 4.5563555|Raiders of the Lo...|    Action|Adventure|\n",
      "|    110|    11|   4.0| 4.5481143|   Braveheart (1995)|    Action|Drama|War|\n",
      "+-------+------+------+----------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "preds.filter(col('userId')==11).filter(col('prediction')>=4.0).sort('prediction',ascending=False).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Wow!! The recommendations are pretty accurate.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:06:26.200037Z",
     "start_time": "2020-06-28T10:06:24.945597Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+------+--------------------+--------------------+\n",
      "|movieId|userId|rating|               title|              genres|\n",
      "+-------+------+------+--------------------+--------------------+\n",
      "|    527|    63|   5.0|Schindler's List ...|           Drama|War|\n",
      "|    750|    63|   5.0|Dr. Strangelove o...|          Comedy|War|\n",
      "|    912|    63|   5.0|   Casablanca (1942)|       Drama|Romance|\n",
      "|    920|    63|   5.0|Gone with the Win...|   Drama|Romance|War|\n",
      "|    969|    63|   5.0|African Queen, Th...|Adventure|Comedy|...|\n",
      "|   1050|    63|   5.0|Looking for Richa...|   Documentary|Drama|\n",
      "|   1177|    63|   5.0|Enchanted April (...|       Drama|Romance|\n",
      "|   1183|    63|   5.0|English Patient, ...|   Drama|Romance|War|\n",
      "|   1196|    63|   5.0|Star Wars: Episod...|Action|Adventure|...|\n",
      "|   1204|    63|   5.0|Lawrence of Arabi...| Adventure|Drama|War|\n",
      "|   1210|    63|   5.0|Star Wars: Episod...|Action|Adventure|...|\n",
      "|   1224|    63|   5.0|      Henry V (1989)|Action|Drama|Roma...|\n",
      "|   1233|    63|   5.0|Boot, Das (Boat, ...|    Action|Drama|War|\n",
      "|   1234|    63|   5.0|   Sting, The (1973)|        Comedy|Crime|\n",
      "|   1242|    63|   5.0|        Glory (1989)|           Drama|War|\n",
      "|   1299|    63|   5.0|Killing Fields, T...|           Drama|War|\n",
      "|   1304|    63|   5.0|Butch Cassidy and...|      Action|Western|\n",
      "|   2028|    63|   5.0|Saving Private Ry...|    Action|Drama|War|\n",
      "|   2609|    63|   5.0|King of Masks, Th...|               Drama|\n",
      "|   2677|    63|   5.0|Buena Vista Socia...| Documentary|Musical|\n",
      "+-------+------+------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "original.filter(col('userId')==63).filter(col('rating')>=4.5).sort('rating',ascending=False).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>The highest rated movies by user 63 belong to Action,drama and war.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-28T10:06:36.376111Z",
     "start_time": "2020-06-28T10:06:29.503426Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+------+----------+--------------------+-----------------+\n",
      "|movieId|userId|rating|prediction|               title|           genres|\n",
      "+-------+------+------+----------+--------------------+-----------------+\n",
      "|   1050|    63|   5.0|  5.068888|Looking for Richa...|Documentary|Drama|\n",
      "|   5060|    63|   5.0|  4.994673|M*A*S*H (a.k.a. M...| Comedy|Drama|War|\n",
      "|   1208|    63|   4.0| 4.6689067|Apocalypse Now (1...| Action|Drama|War|\n",
      "|   1234|    63|   5.0| 4.5235553|   Sting, The (1973)|     Comedy|Crime|\n",
      "|   2028|    63|   5.0| 4.3869686|Saving Private Ry...| Action|Drama|War|\n",
      "|   2858|    63|   4.0|  4.350927|American Beauty (...|     Comedy|Drama|\n",
      "+-------+------+------+----------+--------------------+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "preds.filter(col('userId')==63).filter(col('prediction')>=4.0).sort('prediction',ascending=False).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>The ALS recommendations are pretty consistent with their original preferences.Ofcourse lot more can be done like adding more data and better hyperparameter tuning to improve the model's performance.</b>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}