Movie Recommender.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bc688d86",
   "metadata": {},
   "source": [
    "# Second Exercise: Cosine Similarity for movie comparison\n",
    "\n",
    "In this exercise you have to implement in a python notebook using the spark framework:\n",
    "\n",
    "1. The distributed (map/reduce) algorithm of slide \"3.7\" (in notebook \"8-Item-to-Items-globalfiltering-recommenders-py3-sshow.ipynb\")\n",
    "for computing the cosine similarity of a set of products with negative and positive ratings, using as input information an RDD (or spark dataframe that is also distributed) with ratings with this format:\n",
    "\n",
    "     (userID,movieID,rating)\n",
    "\n",
    "2. The computation of the Cosine Similarity (with the previous algorithm) of all the pairs of movies from the different files you have with this exercise:\n",
    "  filtered50movies.csv filtered100movies.csv  filtered150movies.csv   filtered200movies.csv\n",
    "\n",
    "Each file contains ratings for a different set of movies, but the ones in a smaller file\n",
    "are always a subset of a file with bigger size. We provide files with different size\n",
    "in case you have some memory issues in your computer, so use the biggest file you are able to use, although during \"testing\" of your code you can of course use the smallest file, or even any smaller subset of the file filtered50movies.csv.\n",
    "\n",
    "3. Show on the screen the information for the \"top 10\" most similar pairs, but using the\n",
    "name of the movies you can find in the file movies.\n",
    "\n",
    "All the steps should be implemented always with map/reduce operations with spark RDDs/dataframes. Except the last step, when you have to find the name of the movies in the top-ten recommendations.\n",
    "\n",
    "Present your notebook with plenty of comments in all your functions.\n",
    "\n",
    "NOTE: The ratings for movies come from a dataset obtained from the smallest dataset from:\n",
    "https://grouplens.org/datasets/movielens/\n",
    "But the ratings have been re-scaled from the range [0,5] to the range [-3,2.5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cd5b92d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Libraries\n",
    "import pyspark\n",
    "import os\n",
    "import math\n",
    "import sys\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql.functions import col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ecedfbcc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - in-memory</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://b560m.lan:4041\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v3.1.2</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>MovieRecommender</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x7fb74c6ea340>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Spark Session\n",
    "spark = SparkSession.builder.appName('MovieRecommender').getOrCreate()\n",
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cc990193",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------------------+--------------------+\n",
      "|movieId|               title|              genres|\n",
      "+-------+--------------------+--------------------+\n",
      "|      1|    Toy Story (1995)|Adventure|Animati...|\n",
      "|      2|      Jumanji (1995)|Adventure|Childre...|\n",
      "|      3|Grumpier Old Men ...|      Comedy|Romance|\n",
      "|      4|Waiting to Exhale...|Comedy|Drama|Romance|\n",
      "|      5|Father of the Bri...|              Comedy|\n",
      "+-------+--------------------+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Movies Information\n",
    "moviesDF = spark.read.csv('inputs/movies.csv',header=True)\n",
    "\n",
    "# Cast\n",
    "moviesDF = moviesDF.withColumn(\"movieId\",moviesDF.movieId.cast('int'))\n",
    "moviesDF.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "68722b0d",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+------+\n",
      "|UserID|MovieID|Rating|\n",
      "+------+-------+------+\n",
      "|     1|      1|   1.5|\n",
      "|     5|      1|   1.5|\n",
      "|     7|      1|   2.0|\n",
      "|    15|      1|  -0.5|\n",
      "|    17|      1|   2.0|\n",
      "+------+-------+------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# User Qualifications\n",
    "userMoviesDF = spark.read.csv('inputs/filtered50movies.csv',header=True)\n",
    "\n",
    "# Cast\n",
    "userMoviesDF = userMoviesDF.withColumn(\"UserID\",userMoviesDF.UserID.cast('int'))\n",
    "userMoviesDF = userMoviesDF.withColumn(\"MovieID\",userMoviesDF.MovieID.cast('int'))\n",
    "userMoviesDF = userMoviesDF.withColumn(\"Rating\",userMoviesDF.Rating.cast('float'))\n",
    "\n",
    "# Sort By MoviesID\n",
    "userMoviesDF = userMoviesDF.orderBy('MovieID')\n",
    "\n",
    "userMoviesDF.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a42bcd99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cartesian product\n",
    "userMoviesRDD = userMoviesDF.rdd.cartesian(userMoviesDF.rdd)\n",
    "\n",
    "# Filter the same user and different movie\n",
    "userMoviesRDD = userMoviesRDD.filter(lambda x: x[0][0] == x[1][0] and x[0][1] < x[1][1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9dc1f948",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(Row(UserID=1, MovieID=1, Rating=1.5), Row(UserID=1, MovieID=3, Rating=1.5)),\n",
       " (Row(UserID=19, MovieID=1, Rating=1.5),\n",
       "  Row(UserID=19, MovieID=3, Rating=0.5)),\n",
       " (Row(UserID=32, MovieID=1, Rating=0.5),\n",
       "  Row(UserID=32, MovieID=3, Rating=0.5)),\n",
       " (Row(UserID=43, MovieID=1, Rating=2.5),\n",
       "  Row(UserID=43, MovieID=3, Rating=2.5)),\n",
       " (Row(UserID=44, MovieID=1, Rating=0.5),\n",
       "  Row(UserID=44, MovieID=3, Rating=0.5))]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "userMoviesRDD.take(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "eb4cff5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Map\n",
    "userMoviesRDD = userMoviesRDD.map(lambda i:((i[0][1],i[1][1]), (i[0][2]*i[1][2],i[0][2]**2,i[1][2]**2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d5147596",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[((1, 3), (2.25, 2.25, 2.25)),\n",
       " ((1, 3), (0.75, 2.25, 0.25)),\n",
       " ((1, 3), (0.25, 0.25, 0.25)),\n",
       " ((1, 3), (6.25, 6.25, 6.25)),\n",
       " ((1, 3), (0.25, 0.25, 0.25))]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "userMoviesRDD.take(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "36a543e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reduce by Key\n",
    "userMoviesRDD = userMoviesRDD.reduceByKey(lambda x,y:(x[0]+y[0],x[1]+y[1],x[2]+y[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "64e549f8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[((1, 3), (48.0, 88.25, 51.5)),\n",
       " ((1, 6), (119.0, 150.25, 160.5)),\n",
       " ((1, 47), (190.0, 243.5, 291.25)),\n",
       " ((1, 50), (240.25, 255.0, 348.0)),\n",
       " ((1, 70), (31.75, 68.75, 61.75))]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "userMoviesRDD.sortByKey().take(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7667f1b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate Cosine Distance\n",
    "def cosineDistance(val):\n",
    "    return val[0]/(math.sqrt(val[1])*math.sqrt(val[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6a20e69f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Cosine for each pair of movies\n",
    "userMoviesRDD = userMoviesRDD.map(lambda x: (x[0][0],x[0][1],cosineDistance(x[1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8d017c55",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get top 10 relations\n",
    "top10 = userMoviesRDD.sortBy(lambda x: -x[2]).take(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "53e40d2f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(151, 441, 1.0),\n",
       " (661, 923, 0.9476070829586856),\n",
       " (151, 457, 0.9178523316578322),\n",
       " (50, 296, 0.9117100850753502),\n",
       " (151, 919, 0.9101665113610138),\n",
       " (3, 151, 0.9052317076000181),\n",
       " (163, 661, 0.9030616159415418),\n",
       " (441, 923, 0.901674573007367),\n",
       " (151, 608, 0.8979133729352984),\n",
       " (608, 923, 0.8908671638779693)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "91e7d5df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Movie TItle By Id\n",
    "def getMovieTitleById(id):\n",
    "    return moviesDF.filter(moviesDF.movieId == id).first().title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "056f582d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Dataframe with top 10 relations\n",
    "top10DF = spark.createDataFrame(top10,[\"Movie_1_ID\",\"Movie_2_ID\",\"Cosine Distance\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "84106392",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+----------+------------------+\n",
      "|Movie_1_ID|Movie_2_ID|   Cosine Distance|\n",
      "+----------+----------+------------------+\n",
      "|       151|       441|               1.0|\n",
      "|       661|       923|0.9476070829586856|\n",
      "|       151|       457|0.9178523316578322|\n",
      "|        50|       296|0.9117100850753502|\n",
      "|       151|       919|0.9101665113610138|\n",
      "|         3|       151|0.9052317076000181|\n",
      "|       163|       661|0.9030616159415418|\n",
      "|       441|       923| 0.901674573007367|\n",
      "|       151|       608|0.8979133729352984|\n",
      "|       608|       923|0.8908671638779693|\n",
      "+----------+----------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "top10DF.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "23ca0458",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+------------------+\n",
      "|             Movie_1|             Movie_2|   Cosine Distance|\n",
      "+--------------------+--------------------+------------------+\n",
      "|      Rob Roy (1995)|Dazed and Confuse...|               1.0|\n",
      "|James and the Gia...| Citizen Kane (1941)|0.9476070829586856|\n",
      "|      Rob Roy (1995)|Fugitive, The (1993)|0.9178523316578322|\n",
      "|Usual Suspects, T...| Pulp Fiction (1994)|0.9117100850753502|\n",
      "|      Rob Roy (1995)|Wizard of Oz, The...|0.9101665113610138|\n",
      "|Grumpier Old Men ...|      Rob Roy (1995)|0.9052317076000181|\n",
      "|    Desperado (1995)|James and the Gia...|0.9030616159415418|\n",
      "|Dazed and Confuse...| Citizen Kane (1941)| 0.901674573007367|\n",
      "|      Rob Roy (1995)|        Fargo (1996)|0.8979133729352984|\n",
      "|        Fargo (1996)| Citizen Kane (1941)|0.8908671638779693|\n",
      "+--------------------+--------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Join Movies Id with Titles\n",
    "subset = top10DF.join(moviesDF,top10DF.Movie_1_ID == moviesDF.movieId,\"inner\")\n",
    "subset = subset.withColumnRenamed(\"title\", \"Movie_1\")\n",
    "subset = subset.select(col(\"Movie_1\"), col(\"Movie_2_ID\"), col(\"Cosine Distance\"))\n",
    "\n",
    "subset = subset.join(moviesDF,subset.Movie_2_ID == moviesDF.movieId,\"inner\")\n",
    "subset = subset.withColumnRenamed(\"title\", \"Movie_2\")\n",
    "subset = subset.select(col(\"Movie_1\"), col(\"Movie_2\"), col(\"Cosine Distance\"))\n",
    "subset.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5dea103f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def getCosineSimilarity(dataframe):\n",
    "    \n",
    "    \"\"\"\n",
    "    Function/Algorithm that calculate the cosine similarity using as input a dataframe \n",
    "    like (UserID,Product,Rating).\n",
    "    \n",
    "    Return RDD; format (Product1,Product2,CosineDistance)\n",
    "    \"\"\"\n",
    "    \n",
    "     # Cartesian product\n",
    "    resultRDD = dataframe.rdd.cartesian(dataframe.rdd)\n",
    "\n",
    "    # Filter the same user and different movie\n",
    "    resultRDD = resultRDD.filter(lambda x: x[0][0] == x[1][0] and x[0][1] < x[1][1])\n",
    "    \n",
    "    # Map\n",
    "    resultRDD = resultRDD.map(lambda i:((i[0][1],i[1][1]), (i[0][2]*i[1][2],i[0][2]**2,i[1][2]**2)))\n",
    "    \n",
    "    # Reduce by Key\n",
    "    resultRDD = resultRDD.reduceByKey(lambda x,y:(x[0]+y[0],x[1]+y[1],x[2]+y[2]))\n",
    "    \n",
    "    # Get Cosine for each pair of movies\n",
    "    resultRDD = resultRDD.map(lambda x: (x[0][0],x[0][1],cosineDistance(x[1])))\n",
    "    \n",
    "    return resultRDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9a906d98",
   "metadata": {},
   "outputs": [],
   "source": [
    "def getRecommendations(csv_file,n=10):\n",
    "    \n",
    "    \"\"\"\n",
    "    Function that calculates cosine similarity top pairs for a movies dataset using an input as CSV.\n",
    "    \n",
    "    Return Dataframe; format [Movie1,Movie2,Cosine Distance]\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    # Movies Information\n",
    "    moviesDF = spark.read.csv('inputs/movies.csv',header=True)\n",
    "    moviesDF = moviesDF.withColumn(\"movieId\",moviesDF.movieId.cast('int'))\n",
    "    \n",
    "    # User Qualifications\n",
    "    userMoviesDF = spark.read.csv('inputs/'+csv_file,header=True)\n",
    "\n",
    "    # Cast\n",
    "    userMoviesDF = userMoviesDF.withColumn(\"UserID\",userMoviesDF.UserID.cast('int'))\n",
    "    userMoviesDF = userMoviesDF.withColumn(\"MovieID\",userMoviesDF.MovieID.cast('int'))\n",
    "    userMoviesDF = userMoviesDF.withColumn(\"Rating\",userMoviesDF.Rating.cast('float'))\n",
    "\n",
    "    # Sort By MoviesID\n",
    "    userMoviesDF = userMoviesDF.orderBy('MovieID')\n",
    "    \n",
    "    # Calc Cosine Similarity\n",
    "    userMoviesRDD = getCosineSimilarity(userMoviesDF)\n",
    "    \n",
    "    # Top 10 similar pairs, using their name\n",
    "    top10 = userMoviesRDD.sortBy(lambda x: -x[2]).take(n)\n",
    "    top10DF = spark.createDataFrame(top10,[\"Movie_1_ID\",\"Movie_2_ID\",\"Cosine Distance\"])\n",
    "    subset = top10DF.join(moviesDF,top10DF.Movie_1_ID == moviesDF.movieId,\"inner\")\n",
    "    \n",
    "    subset = subset.withColumnRenamed(\"title\", \"Movie_1\")\n",
    "    subset = subset.select(col(\"Movie_1\"), col(\"Movie_2_ID\"), col(\"Cosine Distance\"))\n",
    "\n",
    "    subset = subset.join(moviesDF,subset.Movie_2_ID == moviesDF.movieId,\"inner\")\n",
    "    subset = subset.withColumnRenamed(\"title\", \"Movie_2\")\n",
    "    subset = subset.select(col(\"Movie_1\"), col(\"Movie_2\"), col(\"Cosine Distance\"))\n",
    "    return subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6ee4d690",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+------------------+\n",
      "|             Movie_1|             Movie_2|   Cosine Distance|\n",
      "+--------------------+--------------------+------------------+\n",
      "|      Rob Roy (1995)|Dazed and Confuse...|               1.0|\n",
      "|James and the Gia...| Citizen Kane (1941)|0.9476070829586856|\n",
      "|      Rob Roy (1995)|Fugitive, The (1993)|0.9178523316578322|\n",
      "|Usual Suspects, T...| Pulp Fiction (1994)|0.9117100850753502|\n",
      "|      Rob Roy (1995)|Wizard of Oz, The...|0.9101665113610138|\n",
      "|Grumpier Old Men ...|      Rob Roy (1995)|0.9052317076000181|\n",
      "|    Desperado (1995)|James and the Gia...|0.9030616159415418|\n",
      "|Dazed and Confuse...| Citizen Kane (1941)| 0.901674573007367|\n",
      "|      Rob Roy (1995)|        Fargo (1996)|0.8979133729352984|\n",
      "|        Fargo (1996)| Citizen Kane (1941)|0.8908671638779693|\n",
      "+--------------------+--------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "getRecommendations('filtered50movies.csv').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "78bf4cfc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+------------------+\n",
      "|             Movie_1|             Movie_2|   Cosine Distance|\n",
      "+--------------------+--------------------+------------------+\n",
      "|      Rob Roy (1995)|Dazed and Confuse...|               1.0|\n",
      "|Star Wars: Episod...|Star Wars: Episod...|0.9560939845339547|\n",
      "|      Platoon (1986)|   Goodfellas (1990)|0.9504490201227582|\n",
      "|James and the Gia...| Citizen Kane (1941)|0.9476070829586856|\n",
      "|      Rob Roy (1995)|Monty Python's Li...|0.9472110029417574|\n",
      "|Raiders of the Lo...|Indiana Jones and...| 0.942606433250396|\n",
      "| Pulp Fiction (1994)|      Platoon (1986)|0.9396928763131717|\n",
      "|        Fargo (1996)|      Platoon (1986)|0.9374347353342417|\n",
      "|Reservoir Dogs (1...|      Platoon (1986)|0.9364700594629586|\n",
      "|Star Wars: Episod...|Star Wars: Episod...|0.9342673969280977|\n",
      "+--------------------+--------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "getRecommendations('filtered100movies.csv').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f7fa80dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+------------------+\n",
      "|             Movie_1|             Movie_2|   Cosine Distance|\n",
      "+--------------------+--------------------+------------------+\n",
      "|      Rob Roy (1995)|Dazed and Confuse...|               1.0|\n",
      "|Star Wars: Episod...|Star Wars: Episod...|0.9560939845339547|\n",
      "|      Platoon (1986)|   Goodfellas (1990)|0.9504490201227582|\n",
      "|James and the Gia...| Citizen Kane (1941)|0.9476070829586856|\n",
      "|      Rob Roy (1995)|Monty Python's Li...|0.9472110029417574|\n",
      "| Citizen Kane (1941)|Conan the Barbari...|0.9435183924675248|\n",
      "|      Rob Roy (1995)|American History ...|0.9434864536402258|\n",
      "|Raiders of the Lo...|Indiana Jones and...| 0.942606433250396|\n",
      "|Jungle Book, The ...|Conan the Barbari...|0.9417419115948374|\n",
      "| Citizen Kane (1941)|L.A. Confidential...| 0.941241407642987|\n",
      "+--------------------+--------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "getRecommendations('filtered150movies.csv').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a82bef80",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+------------------+\n",
      "|             Movie_1|             Movie_2|   Cosine Distance|\n",
      "+--------------------+--------------------+------------------+\n",
      "|      Rob Roy (1995)|Dazed and Confuse...|               1.0|\n",
      "|Jungle Book, The ...|   Dick Tracy (1990)|0.9922778767136677|\n",
      "|Dirty Dozen, The ...|   Robin Hood (1973)|0.9855258295520649|\n",
      "|Jungle Book, The ...|       Dr. No (1962)|0.9832820049844603|\n",
      "|Jungle Book, The ...|¡Three Amigos! (1...|0.9785497849867492|\n",
      "|      Rob Roy (1995)|           Go (1999)|0.9782797401561579|\n",
      "|  Wild Things (1998)|   Robin Hood (1973)| 0.977897823397447|\n",
      "|      Rob Roy (1995)|Crocodile Dundee ...|0.9734503756241593|\n",
      "|        Bambi (1942)|Dirty Dozen, The ...|0.9689627902499088|\n",
      "|Jungle Book, The ...|           Go (1999)|0.9666666666666666|\n",
      "+--------------------+--------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "getRecommendations('filtered200movies.csv').show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cf3444b",
   "metadata": {},
   "source": [
    "The value range is [-1,1]:\n",
    "- -1 means that the pair of movies are not related. \n",
    "- 0 means that the pair of movies are neutral.\n",
    "- 1 means that the pair of movies are related."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ef836ee",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    "The datasets provided have a maximum of 200 movies, whose main factor (rating) is the user's opinion.\n",
    "The users' opinion will be the variable that will join the films in pairs and will help us to make the recommendation according to their criteria.\n",
    "\n",
    "Sometimes the use of the users' opinion can cause bad results, because it could happen that two movies are similar but this concrete dataset doesn't relate them. In the current case study, we believe that it would be better to use as well the genres of the movies to find the similarity between them, as a second criteria.\n",
    "\n",
    "As we can see in the different results, it seems that the recommendation algorithm based on the user's opinion works fairly well. For instance, we observe that movies of sagas such as Star Wars or Indiana Jones follow a coherence of high relationship with their respective ones. In addition, movies that share the same scope,actors or references in them are related.\n",
    "\n",
    "In conlusion the recommendation that has been developed uses user feedback to recommend movies that has similar opinions.\n",
    "\n",
    "By Francesc Contreras & Albert Pérez"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}