movie_recommendation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mini project: simple movie recommendation\n",
    "\n",
    "> We will create a simple movie recommendation system based on the movie reviews from users. We will use a known dataset that contains movie reviews from many users. Each user rates more than one movie. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas to play with the dataset\n",
    "import pandas as pd\n",
    "\n",
    "# We will use the MovieLens dataset which contains user ratings for movies.\n",
    "# We have downloaded the dataset locally in the data folder.\n",
    "ratings_filename = './data/u.data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (100000, 4)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  MovieID  Rating   Datetime\n",
       "0     196      242       3  881250949\n",
       "1     186      302       3  891717742\n",
       "2      22      377       1  878887116\n",
       "3     244       51       2  880606923\n",
       "4     166      346       1  886397596"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The data is provided in a TSV form.\n",
    "# This is like the CSV, but the columns instead of being comma seperated, they are tab seperated.\n",
    "# Use the read_csv() function of pandas and specify the delimiter\n",
    "all_ratings = pd.read_csv(ratings_filename, delimiter=\"\\t\", header=None, \n",
    "                          names = [\"UserID\", \"MovieID\", \"Rating\", \"Datetime\"])\n",
    "\n",
    "# The full dataset: 100000 ratings by 943 users on 1682 movies.\n",
    "# Each user has rated many movies (at least 20)\n",
    "# Users and items are numbered consecutively from 1.  \n",
    "# The data is randomly ordered. \n",
    "# This is a tab separated list of: \n",
    "# user id | item id | rating | timestamp. \n",
    "\n",
    "print(f'Shape: {all_ratings.shape}')\n",
    "\n",
    "# Sample:\n",
    "all_ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-12-04 15:55:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>1998-04-04 19:22:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>1997-11-07 07:18:36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>1997-11-27 05:02:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>1998-02-02 05:33:16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  MovieID  Rating            Datetime\n",
       "0     196      242       3 1997-12-04 15:55:49\n",
       "1     186      302       3 1998-04-04 19:22:22\n",
       "2      22      377       1 1997-11-07 07:18:36\n",
       "3     244       51       2 1997-11-27 05:02:03\n",
       "4     166      346       1 1998-02-02 05:33:16"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The last attribute seems to be a unix datetime.\n",
    "# Although we will not need it, let's transform it to datetime object.\n",
    "all_ratings[\"Datetime\"] = pd.to_datetime(all_ratings['Datetime'], unit='s')\n",
    "all_ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We need to transform the dataset into a transactions-like format that we have seen in the class.\n",
    "\n",
    "> Also: this is a movie recommendation project based on a dataset with movie reviews. We need to incorporate the reviews somehow. The review is the \"Rating\" attribute which is categorical. How are we going to handle it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Datetime</th>\n",
       "      <th>Favorable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-12-04 15:55:49</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>1998-04-04 19:22:22</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>1997-11-07 07:18:36</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>1997-11-27 05:02:03</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>1998-02-02 05:33:16</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  MovieID  Rating            Datetime  Favorable\n",
       "0     196      242       3 1997-12-04 15:55:49      False\n",
       "1     186      302       3 1998-04-04 19:22:22      False\n",
       "2      22      377       1 1997-11-07 07:18:36      False\n",
       "3     244       51       2 1997-11-27 05:02:03      False\n",
       "4     166      346       1 1998-02-02 05:33:16      False"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Attribute construction: \n",
    "# Create an additional attribute that we will call \"Favorable\"\n",
    "# This will be true if the user gave a rating above 3\n",
    "\n",
    "# Select all rows with rating > 3 and add the attribute Favorable = True. \n",
    "# Otherwise, Favorable = False\n",
    "all_ratings[\"Favorable\"] = all_ratings[\"Rating\"] > 3\n",
    "all_ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Datetime</th>\n",
       "      <th>Favorable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-12-04 15:55:49</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>1998-04-04 19:22:22</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>1997-11-07 07:18:36</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>1998-02-02 05:33:16</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>115</td>\n",
       "      <td>265</td>\n",
       "      <td>2</td>\n",
       "      <td>1997-12-03 17:51:28</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99951</th>\n",
       "      <td>130</td>\n",
       "      <td>121</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-10-07 18:59:06</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99959</th>\n",
       "      <td>193</td>\n",
       "      <td>690</td>\n",
       "      <td>4</td>\n",
       "      <td>1998-03-05 18:40:21</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99978</th>\n",
       "      <td>113</td>\n",
       "      <td>975</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-10-04 03:40:24</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99998</th>\n",
       "      <td>13</td>\n",
       "      <td>225</td>\n",
       "      <td>2</td>\n",
       "      <td>1997-12-17 22:52:36</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99999</th>\n",
       "      <td>12</td>\n",
       "      <td>203</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-11-19 17:13:03</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19531 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       UserID  MovieID  Rating            Datetime  Favorable\n",
       "0         196      242       3 1997-12-04 15:55:49      False\n",
       "1         186      302       3 1998-04-04 19:22:22      False\n",
       "2          22      377       1 1997-11-07 07:18:36      False\n",
       "4         166      346       1 1998-02-02 05:33:16      False\n",
       "6         115      265       2 1997-12-03 17:51:28      False\n",
       "...       ...      ...     ...                 ...        ...\n",
       "99951     130      121       5 1997-10-07 18:59:06       True\n",
       "99959     193      690       4 1998-03-05 18:40:21       True\n",
       "99978     113      975       5 1997-10-04 03:40:24       True\n",
       "99998      13      225       2 1997-12-17 22:52:36      False\n",
       "99999      12      203       3 1997-11-19 17:13:03      False\n",
       "\n",
       "[19531 rows x 5 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's use only a small sample of 200 users\n",
    "# I.e. we will use the ratings of those users (not 200 samples)\n",
    "\n",
    "ratings = all_ratings[all_ratings['UserID'].isin(range(200))]\n",
    "ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Datetime</th>\n",
       "      <th>Favorable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>122</td>\n",
       "      <td>387</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-11-11 17:47:39</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>119</td>\n",
       "      <td>392</td>\n",
       "      <td>4</td>\n",
       "      <td>1998-01-30 16:13:34</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>167</td>\n",
       "      <td>486</td>\n",
       "      <td>4</td>\n",
       "      <td>1998-04-16 14:54:12</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>38</td>\n",
       "      <td>95</td>\n",
       "      <td>5</td>\n",
       "      <td>1998-04-13 01:14:54</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>63</td>\n",
       "      <td>277</td>\n",
       "      <td>4</td>\n",
       "      <td>1997-10-01 23:10:01</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99848</th>\n",
       "      <td>5</td>\n",
       "      <td>174</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-09-30 16:15:30</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99950</th>\n",
       "      <td>130</td>\n",
       "      <td>93</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-09-22 18:41:05</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99951</th>\n",
       "      <td>130</td>\n",
       "      <td>121</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-10-07 18:59:06</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99959</th>\n",
       "      <td>193</td>\n",
       "      <td>690</td>\n",
       "      <td>4</td>\n",
       "      <td>1998-03-05 18:40:21</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99978</th>\n",
       "      <td>113</td>\n",
       "      <td>975</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-10-04 03:40:24</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11043 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       UserID  MovieID  Rating            Datetime  Favorable\n",
       "16        122      387       5 1997-11-11 17:47:39       True\n",
       "20        119      392       4 1998-01-30 16:13:34       True\n",
       "21        167      486       4 1998-04-16 14:54:12       True\n",
       "26         38       95       5 1998-04-13 01:14:54       True\n",
       "28         63      277       4 1997-10-01 23:10:01       True\n",
       "...       ...      ...     ...                 ...        ...\n",
       "99848       5      174       5 1997-09-30 16:15:30       True\n",
       "99950     130       93       5 1997-09-22 18:41:05       True\n",
       "99951     130      121       5 1997-10-07 18:59:06       True\n",
       "99959     193      690       4 1998-03-05 18:40:21       True\n",
       "99978     113      975       5 1997-10-04 03:40:24       True\n",
       "\n",
       "[11043 rows x 5 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Our movie recommendation system will rely only on favorable movies.\n",
    "# Out of the above sample, get the Favorable ratings.\n",
    "\n",
    "# Filter the dataframe and get the rows that have Favorable = True\n",
    "favorable_ratings = ratings[ratings[\"Favorable\"]]\n",
    "favorable_ratings\n",
    "# about half (more or less..)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transactions-like data\n",
    "> Let's create sequential patterns for the users. Each user did not watch all the movies that has rated at once! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movies from the 1st user:\n",
      "[61, 33, 160, 20, 202, 171, 265, 47, 222, 253, 113, 227, 90, 64, 228, 121, 114, 132, 134, 98, 186, 221, 84, 60, 177, 174, 82, 56, 80, 229, 235, 6, 206, 76, 72, 185, 96, 258, 81, 212, 151, 51, 175, 107, 209, 108, 12, 14, 44, 163, 210, 184, 157, 150, 183, 248, 208, 128, 242, 193, 236, 250, 91, 129, 241, 267, 86, 196, 39, 230, 23, 224, 65, 190, 100, 154, 214, 161, 170, 9, 246, 22, 187, 135, 68, 146, 176, 166, 89, 249, 269, 32, 270, 133, 239, 194, 256, 93, 234, 1, 197, 173, 75, 268, 144, 119, 181, 257, 109, 182, 223, 46, 169, 162, 66, 77, 199, 57, 50, 192, 178, 87, 238, 156, 106, 115, 137, 127, 16, 79, 45, 48, 25, 251, 195, 168, 123, 191, 203, 55, 42, 7, 43, 165, 198, 124, 95, 58, 216, 204, 3, 207, 19, 18, 59, 15, 111, 52, 88, 13, 28, 172, 152]\n",
      "\n",
      "Movies from the 2nd user:\n",
      "[292, 251, 50, 297, 13, 303, 257, 316, 301, 313, 279, 299, 277, 282, 111, 295, 242, 283, 276, 1, 14, 293, 310, 306, 25, 273, 311, 269, 255, 284, 237, 300, 100, 127, 285, 304, 272, 286, 275, 302]\n"
     ]
    }
   ],
   "source": [
    "# Now, for each user create a list with their favorite movies\n",
    "# And put those lists in a bigger list in order to create an \"itemset list\"\n",
    "# e.g. [\n",
    "#.       [id1, id2, id3, ...],               # Each row is a list of movies for a specific user\n",
    "#.       [id10, id1, id4, id3, ...],\n",
    "#.       ...\n",
    "#     ]\n",
    "\n",
    "# Does the above remind the HORIZONTAL DATA FORMAT? \n",
    "\n",
    "# Group favorable ratings by user id and movie id and get the list of movies for each user\n",
    "favorable_movies_by_user = [list(v.values) for k,v in favorable_ratings.groupby(\"UserID\")[\"MovieID\"]]\n",
    "# print a couple of examples\n",
    "print('Movies from the 1st user:')\n",
    "print(favorable_movies_by_user[0])\n",
    "print('')\n",
    "print('Movies from the 2nd user:')\n",
    "print(favorable_movies_by_user[1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We have an itemset list that contains the favorite movies for each user\n",
    "# Run the apriori algorithm to get the frequent itemsets and the association rules\n",
    "from apriori_algorithm import *\n",
    "\n",
    "freqItemSet, rules = apriori(favorable_movies_by_user, minSup=0.2, minConf=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: [[127], [56], [238], [89], [294], [28], [25], [181], [1], [258], [204], [79], [237], [117], [96], [300], [121], [216], [269], [69], [172], [135], [22], [195], [423], [9], [276], [183], [318], [286], [196], [15], [12], [100], [50], [302], [210], [191], [176], [64], [174], [288], [7], [98], [357], [168], [222], [173], [313]]\n",
      "\n",
      "2: [[98, 100], [96, 174], [56, 100], [50, 127], [64, 50], [89, 50], [50, 181], [174, 79], [1, 174], [50, 98], [50, 79], [181, 174], [89, 174], [56, 174], [64, 98], [50, 173], [50, 7], [50, 195], [258, 50], [1, 50], [98, 174], [64, 174], [56, 98], [50, 174], [100, 174], [100, 7], [1, 100], [195, 174], [100, 127], [50, 172], [50, 100], [56, 50], [172, 174], [172, 181]]\n",
      "\n",
      "3: [[50, 172, 181], [50, 181, 174], [50, 174, 98], [172, 181, 174], [50, 195, 174], [56, 50, 174], [50, 172, 174], [50, 174, 79]]\n",
      "\n",
      "4: [[50, 181, 172, 174]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's print the frequent k-itemsets\n",
    "for key in freqItemSet:\n",
    "    values = [list(x) for x in freqItemSet[key]]\n",
    "    print(f'{key}: {values}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{50, 195} --> {174} [conf: 1.0]\n",
      "{172, 181} --> {50} [conf: 0.9782608695652174]\n",
      "{172, 181, 174} --> {50} [conf: 0.9761904761904762]\n",
      "{56, 50} --> {174} [conf: 0.9565217391304348]\n",
      "{195, 174} --> {50} [conf: 0.9545454545454546]\n",
      "{50, 79} --> {174} [conf: 0.9523809523809523]\n",
      "{172, 174} --> {50} [conf: 0.9423076923076923]\n",
      "{195} --> {174} [conf: 0.9361702127659575]\n",
      "{181, 174} --> {50} [conf: 0.9230769230769231]\n",
      "{172} --> {50} [conf: 0.9152542372881356]\n",
      "{172, 181} --> {174} [conf: 0.9130434782608695]\n",
      "{50, 172, 181} --> {174} [conf: 0.9111111111111111]\n",
      "{50, 172} --> {174} [conf: 0.9074074074074074]\n"
     ]
    }
   ],
   "source": [
    "# Store the rules and print them in descending order of confidence\n",
    "\n",
    "ordered_rules = []\n",
    "for rule in rules:\n",
    "    ordered_rules.append(rule)\n",
    "ordered_rules.reverse()\n",
    "\n",
    "for rule in ordered_rules:\n",
    "    print(f'{rule[0]} --> {rule[1]} [conf: {rule[2]}]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rule #1\n",
      "If a person recommends {50, 195} they will also recommend {174}\n",
      " - Confidence: 1.0\n",
      "\n",
      "Rule #2\n",
      "If a person recommends {172, 181} they will also recommend {50}\n",
      " - Confidence: 0.9782608695652174\n",
      "\n",
      "Rule #3\n",
      "If a person recommends {172, 181, 174} they will also recommend {50}\n",
      " - Confidence: 0.9761904761904762\n",
      "\n",
      "Rule #4\n",
      "If a person recommends {56, 50} they will also recommend {174}\n",
      " - Confidence: 0.9565217391304348\n",
      "\n",
      "Rule #5\n",
      "If a person recommends {195, 174} they will also recommend {50}\n",
      " - Confidence: 0.9545454545454546\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's print a few top rules in a more friendly way\n",
    "for index in range(5):\n",
    "    print(f'Rule #{index+1}')\n",
    "    print(f'If a person recommends {ordered_rules[index][0]} they will also recommend {ordered_rules[index][1]}')\n",
    "    print(f' - Confidence: {ordered_rules[index][2]}')\n",
    "    print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's just make our recommendation system a bit better by providing the movie title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Release Date</th>\n",
       "      <th>Video Release</th>\n",
       "      <th>IMDB</th>\n",
       "      <th>&lt;UNK&gt;</th>\n",
       "      <th>Action</th>\n",
       "      <th>Adventure</th>\n",
       "      <th>Animation</th>\n",
       "      <th>Children's</th>\n",
       "      <th>...</th>\n",
       "      <th>Fantasy</th>\n",
       "      <th>Film-Noir</th>\n",
       "      <th>Horror</th>\n",
       "      <th>Musical</th>\n",
       "      <th>Mystery</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Sci-Fi</th>\n",
       "      <th>Thriller</th>\n",
       "      <th>War</th>\n",
       "      <th>Western</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Toy%20Story%2...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>GoldenEye (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?GoldenEye%20(...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Four Rooms (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Four%20Rooms%...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Get Shorty (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Get%20Shorty%...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Copycat (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Copycat%20(1995)</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieID              Title Release Date  Video Release  \\\n",
       "0        1   Toy Story (1995)  01-Jan-1995            NaN   \n",
       "1        2   GoldenEye (1995)  01-Jan-1995            NaN   \n",
       "2        3  Four Rooms (1995)  01-Jan-1995            NaN   \n",
       "3        4  Get Shorty (1995)  01-Jan-1995            NaN   \n",
       "4        5     Copycat (1995)  01-Jan-1995            NaN   \n",
       "\n",
       "                                                IMDB  <UNK>  Action  \\\n",
       "0  http://us.imdb.com/M/title-exact?Toy%20Story%2...      0       0   \n",
       "1  http://us.imdb.com/M/title-exact?GoldenEye%20(...      0       1   \n",
       "2  http://us.imdb.com/M/title-exact?Four%20Rooms%...      0       0   \n",
       "3  http://us.imdb.com/M/title-exact?Get%20Shorty%...      0       1   \n",
       "4  http://us.imdb.com/M/title-exact?Copycat%20(1995)      0       0   \n",
       "\n",
       "   Adventure  Animation  Children's  ...  Fantasy  Film-Noir  Horror  Musical  \\\n",
       "0          0          1           1  ...        0          0       0        0   \n",
       "1          1          0           0  ...        0          0       0        0   \n",
       "2          0          0           0  ...        0          0       0        0   \n",
       "3          0          0           0  ...        0          0       0        0   \n",
       "4          0          0           0  ...        0          0       0        0   \n",
       "\n",
       "   Mystery  Romance  Sci-Fi  Thriller  War  Western  \n",
       "0        0        0       0         0    0        0  \n",
       "1        0        0       0         1    0        0  \n",
       "2        0        0       0         1    0        0  \n",
       "3        0        0       0         0    0        0  \n",
       "4        0        0       0         1    0        0  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# There is an additional file in the dataset that provides the movie info\n",
    "\n",
    "# -- Information about the (movies): \"|\" separated list of:\n",
    "#    movie id | movie title | release date | video release date | IMDb URL | unknown |\n",
    "#    Action | Adventure | Animation | Children's | Comedy | Crime | Documentary | Drama | Fantasy |\n",
    "#    Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi | Thriller | War | Western |\n",
    "#\n",
    "# -- The last 19 fields are the genres, a 1 indicates the movie\n",
    "#    is of that genre, a 0 indicates it is not; movies can be in\n",
    "#    several genres at once. The movie ids are the ones used in the u.data dataset that we used above.\n",
    "\n",
    "movie_name_filename = './data/u.item'\n",
    "movie_name_data = pd.read_csv(movie_name_filename, delimiter=\"|\", header=None, encoding = \"mac-roman\")\n",
    "movie_name_data.columns = [\"MovieID\", \"Title\", \"Release Date\", \"Video Release\", \"IMDB\", \"<UNK>\", \"Action\", \"Adventure\",\n",
    "                           \"Animation\", \"Children's\", \"Comedy\", \"Crime\", \"Documentary\", \"Drama\", \"Fantasy\", \"Film-Noir\",\n",
    "                           \"Horror\", \"Musical\", \"Mystery\", \"Romance\", \"Sci-Fi\", \"Thriller\", \"War\", \"Western\"]\n",
    "\n",
    "# Sample\n",
    "movie_name_data.head()\n",
    "\n",
    "# Note how the genre of the movie is encoded in the dataset.\n",
    "# Instead of having a single feature of nominal values, it is one hot - encoded. \n",
    "# We have a separate feature for each genre that take value either 0 or 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rule #1\n",
      "If a person recommends Star Wars (1977), Terminator, The (1984) they will also recommend Raiders of the Lost Ark (1981)\n",
      " - Confidence: 1.0\n",
      "\n",
      "Rule #2\n",
      "If a person recommends Empire Strikes Back, The (1980), Return of the Jedi (1983) they will also recommend Star Wars (1977)\n",
      " - Confidence: 0.9782608695652174\n",
      "\n",
      "Rule #3\n",
      "If a person recommends Empire Strikes Back, The (1980), Return of the Jedi (1983), Raiders of the Lost Ark (1981) they will also recommend Star Wars (1977)\n",
      " - Confidence: 0.9761904761904762\n",
      "\n",
      "Rule #4\n",
      "If a person recommends Pulp Fiction (1994), Star Wars (1977) they will also recommend Raiders of the Lost Ark (1981)\n",
      " - Confidence: 0.9565217391304348\n",
      "\n",
      "Rule #5\n",
      "If a person recommends Terminator, The (1984), Raiders of the Lost Ark (1981) they will also recommend Star Wars (1977)\n",
      " - Confidence: 0.9545454545454546\n",
      "\n",
      "Rule #6\n",
      "If a person recommends Star Wars (1977), Fugitive, The (1993) they will also recommend Raiders of the Lost Ark (1981)\n",
      " - Confidence: 0.9523809523809523\n",
      "\n",
      "Rule #7\n",
      "If a person recommends Empire Strikes Back, The (1980), Raiders of the Lost Ark (1981) they will also recommend Star Wars (1977)\n",
      " - Confidence: 0.9423076923076923\n",
      "\n",
      "Rule #8\n",
      "If a person recommends Terminator, The (1984) they will also recommend Raiders of the Lost Ark (1981)\n",
      " - Confidence: 0.9361702127659575\n",
      "\n",
      "Rule #9\n",
      "If a person recommends Return of the Jedi (1983), Raiders of the Lost Ark (1981) they will also recommend Star Wars (1977)\n",
      " - Confidence: 0.9230769230769231\n",
      "\n",
      "Rule #10\n",
      "If a person recommends Empire Strikes Back, The (1980) they will also recommend Star Wars (1977)\n",
      " - Confidence: 0.9152542372881356\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's print the same rules by using the movie name\n",
    "import copy\n",
    "rules = copy.deepcopy(ordered_rules)\n",
    "\n",
    "# Define a function to get movie name\n",
    "def get_movie_name(movie_id):\n",
    "    title_object = movie_name_data[movie_name_data[\"MovieID\"] == movie_id][\"Title\"]\n",
    "    title = title_object.values[0]\n",
    "    return title\n",
    "\n",
    "# Print the rules again using the movie title this time\n",
    "for index in range(10):\n",
    "    print(f'Rule #{index+1}')\n",
    "    premise = rules[index][0]\n",
    "    conclusion = rules[index][1].pop()\n",
    "    premise_names = \", \".join(get_movie_name(idx) for idx in premise)\n",
    "    conclusion_name = get_movie_name(conclusion)\n",
    "    print(f'If a person recommends {premise_names} they will also recommend {conclusion_name}')\n",
    "    print(f' - Confidence: {rules[index][2]}')\n",
    "    print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How would you use such a system?\n",
    "> Example:\n",
    "You browse movies in your streaming platform and you select to watch \"The Terminator\". \n",
    "Given rule 8, a recommendation could be \"Raiders of the lost Ark\", in a way such as:\n",
    "\"Users also liked Raiders of the lost Ark!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* This is a simple recommendation engine that is based on \"market basket analysis\", and frequent itemsets. It does not consider meta-data and other characteristics or features that are based on user preferences. It relies on items that appear in the same \"transaction\". The transaction here (if we were to simulate the market basket analysis) is the list of favorable movies of each user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "local-venv",
   "language": "python",
   "name": "local-venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}