Talk-Copy3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Boilerplate code below..\n",
    "\n",
    "Setting a default seed for RNG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "\n",
    "my_seed = 1337\n",
    "random.seed(my_seed)\n",
    "np.random.seed(my_seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from typing import *\n",
    "from IPython.display import display, HTML, Markdown\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "\n",
    "def display_best_and_worse_recommendations(recommendations: pd.DataFrame):\n",
    "    recommendations.sort_values('Estimated Prediction', ascending=False, inplace=True)\n",
    "\n",
    "    top_recommendations = recommendations.iloc[:10]\n",
    "    top_recommendations.columns = ['Prediction (sorted by best)', 'Movie Title']\n",
    "\n",
    "    worse_recommendations = recommendations.iloc[-10:]\n",
    "    worse_recommendations.columns = ['Prediction (sorted by worse)', 'Movie Title']\n",
    "\n",
    "    display(HTML(\"<h1>Recommendations your user will love</h1>\"))\n",
    "    display(top_recommendations)\n",
    "\n",
    "    display(HTML(\"<h1>Recommendations your user will hate</h1>\"))\n",
    "    display(worse_recommendations)\n",
    "    \n",
    "\n",
    "def load_movies_dataset() -> pd.DataFrame:\n",
    "    movie_data_columns = [\n",
    "    'movie_id', 'title', 'release_date', 'video_release_date', 'url',\n",
    "    'unknown', 'Action', 'Adventure', 'Animation', \"Children's\",\n",
    "    'Comedy', 'Crime', 'Documentary', 'Drama', 'Fantasy', 'Film-Noir',\n",
    "    'Horror', 'Musical', 'Mystery', 'Romance', 'Sci-Fi', 'Thriller',\n",
    "    'War', 'Western'\n",
    "    ]\n",
    "\n",
    "    movie_data = pd.read_csv(\n",
    "        'datasets/ml-100k/u.item', \n",
    "        sep = '|', \n",
    "        encoding = \"ISO-8859-1\", \n",
    "        header = None, \n",
    "        names = movie_data_columns,\n",
    "        index_col = 'movie_id'\n",
    "    )\n",
    "    movie_data['release_date'] = pd.to_datetime(movie_data['release_date'])\n",
    "    return movie_data\n",
    "\n",
    "def load_ratings() -> pd.DataFrame:\n",
    "    ratings_data = pd.read_csv(\n",
    "        'datasets/ml-100k/u.data',\n",
    "        sep = '\\t',\n",
    "        encoding = \"ISO-8859-1\",\n",
    "        header = None,\n",
    "        names=['user_id', 'movie_id', 'rating', 'timestamp']\n",
    "    )\n",
    "    return ratings_data[['user_id', 'movie_id', 'rating']]\n",
    "\n",
    "def load_ratings_with_name() -> pd.DataFrame:\n",
    "    ratings_data = load_ratings()\n",
    "    movies_data = load_movies_dataset()\n",
    "    ratings_data['user_id'] = ratings_data['user_id'].map(lambda k: f\"User {k}\")\n",
    "    \n",
    "    ratings_and_movies = ratings_data \\\n",
    "        .set_index('movie_id') \\\n",
    "        .join(movies_data['title']) \\\n",
    "        .reset_index()\n",
    "    \n",
    "    ratings_and_movies['movie_title'] = ratings_and_movies['title']\n",
    "    return ratings_and_movies[['user_id', 'movie_title', 'rating']].sample(frac=1)\n",
    "    \n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# A practical guide to Singular Value Decomposition in Python\n",
    "\n",
    "Recommender systems have become increasingly popular in recent years, and are used by some of the largest websites in the world to predict the likelihood of a user taking an action on an item. In the world of Netflix, this means recommending similar movies to the ones you have seen. In the world of dating, this means suggesting matches similar to people you already showed interest in!\n",
    "\n",
    "My path to recommenders has been an unusual one: from a Software Engineer to working on matching algorithms at a dating company, with a little background on machine learning. With my knowledge of Python and the use of basic SVD (Singular Value Decomposition) frameworks, I was able to understand SVDs from a practical standpoint of what you can do with them, instead of focusing on the science.\n",
    "\n",
    "In my talk, you will learn 2 practical ways of generating recommendations using SVDs: matrix factorization and item similarity. We will be learning the high-level components of SVD the \"doer way\": we will be implementing a simple movie recommendation engine with the help of Jupiter notebooks, the MovieLens database, and the Surprise recommendation package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    " - Recommendations via Matrix Factorization: Performing predict() manually"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Table of contents\n",
    "\n",
    " - Downloading and exploring the MovieLens dataset\n",
    " - Training a SVD using Surprise in 4 simple steps\n",
    " - Using the predict() API inside of Surprise\n",
    " - Recommendations via Product based CF: Finding similarity between vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Downloading and exploring the MovieLens dataset\n",
    "\n",
    "<p><img src=\"https://static1.squarespace.com/static/51cdafc4e4b09eb676a64e68/t/579282fabebafbb6c366252c/1469219594863/\" alt=\"Drawing\" style=\"width: 400px; float: left\"/></p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Exploring the Dataset\n",
    "\n",
    "\n",
    "\n",
    "- Open Source dataset\n",
    "- 20 million ratings\n",
    "- 27,000 movies\n",
    "- 138,000 users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "movie_data = load_movies_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'ratings_data' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-bfd5b9699953>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# EXPLORATION\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mratings_with_movies\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mratings_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'movie_title'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmovie_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'title'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhow\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'inner'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m ratings_with_genres = ratings_with_movies.reset_index(drop=True)[['user_id', 'Action', 'Adventure', 'Animation', \"Children's\",\n\u001b[1;32m      5\u001b[0m     \u001b[0;34m'Comedy'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Crime'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Documentary'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Drama'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Fantasy'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Film-Noir'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'ratings_data' is not defined"
     ]
    }
   ],
   "source": [
    "# EXPLORATION\n",
    "\n",
    "ratings_with_movies = ratings_data.set_index('movie_title').join(movie_data.set_index('title'), how='inner')\n",
    "ratings_with_genres = ratings_with_movies.reset_index(drop=True)[['user_id', 'Action', 'Adventure', 'Animation', \"Children's\",\n",
    "    'Comedy', 'Crime', 'Documentary', 'Drama', 'Fantasy', 'Film-Noir',\n",
    "    'Horror', 'Musical', 'Mystery', 'Romance', 'Sci-Fi', 'Thriller',\n",
    "    'War', 'Western']]\n",
    "\n",
    "user_and_preferences = ratings_with_genres.groupby('user_id').sum()\n",
    "\n",
    "total_user_rating = user_and_preferences.sum(axis=1)\n",
    "user_and_preferences.std(axis=1).sort_values(ascending=False)\n",
    "\n",
    "\n",
    "user_and_preferences.loc['User 450']\n",
    "\n",
    "\n",
    "user_and_preferences.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>36649</th>\n",
       "      <td>User 742</td>\n",
       "      <td>Jerry Maguire (1996)</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2478</th>\n",
       "      <td>User 908</td>\n",
       "      <td>Usual Suspects, The (1995)</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82838</th>\n",
       "      <td>User 758</td>\n",
       "      <td>Real Genius (1985)</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69729</th>\n",
       "      <td>User 393</td>\n",
       "      <td>Things to Do in Denver when You're Dead (1995)</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36560</th>\n",
       "      <td>User 66</td>\n",
       "      <td>Jerry Maguire (1996)</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                                     movie_title  rating\n",
       "36649  User 742                            Jerry Maguire (1996)       4\n",
       "2478   User 908                      Usual Suspects, The (1995)       3\n",
       "82838  User 758                              Real Genius (1985)       4\n",
       "69729  User 393  Things to Do in Denver when You're Dead (1995)       3\n",
       "36560   User 66                            Jerry Maguire (1996)       4"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_data = load_ratings_with_name()\n",
    "ratings_data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64719</th>\n",
       "      <td>User 730</td>\n",
       "      <td>Picture Perfect (1997)</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10385</th>\n",
       "      <td>User 399</td>\n",
       "      <td>Black Beauty (1994)</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50679</th>\n",
       "      <td>User 534</td>\n",
       "      <td>Liar Liar (1997)</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90771</th>\n",
       "      <td>User 741</td>\n",
       "      <td>Twister (1996)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55889</th>\n",
       "      <td>User 176</td>\n",
       "      <td>Mighty Aphrodite (1995)</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id              movie_title  rating\n",
       "64719  User 730   Picture Perfect (1997)       2\n",
       "10385  User 399      Black Beauty (1994)       3\n",
       "50679  User 534         Liar Liar (1997)       5\n",
       "90771  User 741           Twister (1996)       1\n",
       "55889  User 176  Mighty Aphrodite (1995)       4"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove movies with few ratings\n",
    "movie_ratings = ratings_data.groupby('movie_title').size()\n",
    "valid_movies = movie_ratings[movie_ratings > 15]\n",
    "movie_ratings = ratings_data.set_index('movie_title', drop=False).join(valid_movies.to_frame(), how='inner').reset_index(drop=True)\n",
    "\n",
    "del movie_ratings[0]\n",
    "\n",
    "movie_ratings = movie_ratings.sample(frac=1)\n",
    "movie_ratings.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Training a SVD using Surprise in 4 simple steps\n",
    "\n",
    "Let's take the **interactions** between the Users and Movies, and generate **latent features**  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from surprise import SVD, NMF, accuracy\n",
    "from surprise import Dataset, Reader\n",
    "from surprise.model_selection import cross_validate, train_test_split\n",
    "\n",
    "# Step 1: create a Reader.\n",
    "# A reader tells our SVD what the lower and upper bound of our ratings is.\n",
    "# MovieLens ratings are from 1 to 5\n",
    "reader = Reader(rating_scale=(1, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Step 2: create a new Dataset instance with a DataFrame and the reader\n",
    "# The DataFrame needs to have 3 columns in this specific order: [user_id, product_id, rating]\n",
    "data = Dataset.load_from_df(ratings_data, reader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Step 3: keep 25% of your trainset for testing\n",
    "trainset, testset = train_test_split(data, test_size=.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "reader = Reader(rating_scale=(1, 5))\n",
    "data = Dataset.load_from_df(ratings_data, reader)\n",
    "trainset, testset = train_test_split(data, test_size=.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<surprise.prediction_algorithms.matrix_factorization.SVD at 0x111c24f28>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 4: train a new SVD\n",
    "model = SVD(n_factors=100, biased=False)\n",
    "model.fit(trainset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.9366\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9365724030417191"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Optionally, validate the RMSE (root-mean-square error) to ensure model training was effectiv \n",
    "predictions = model.test(testset)\n",
    "accuracy.rmse(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0000000000000002"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TEST mormalization\n",
    "pd.DataFrame(model.qi).iloc[0].pow(2).sum()\n",
    "model.qi /= np.linalg.norm(model.qi, ord=2, axis=1).reshape(-1, 1)\n",
    "pd.DataFrame(model.qi).iloc[0].pow(2).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Generating predictions with simplicity\n",
    "\n",
    "Coomputes the rating prediction for given user and movie with `model.predict()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(uid='User 196', iid='Toy Story (1995)', r_ui=None, est=1.5455571744427679, details={'was_impossible': False})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "daniel = \"User 196\"\n",
    "toy_story = \"Toy Story (1995)\"\n",
    "model.predict(daniel, toy_story)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Inspecting our Product Matrix\n",
    "\n",
    "Surprise SVD stores the product matrix under the `model.qi` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1664, 100)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.qi.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The matrix has `n_factors` columns (we chose 100). Every row represents a movie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Mapping every vector back to it's movie\n",
    "\n",
    "Every row is mapped to a movie. How do we map every row to it's vector?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Movie name</th>\n",
       "      <th>model.qi row idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Restoration (1995)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Monty Python and the Holy Grail (1974)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Starship Troopers (1997)</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Swingers (1996)</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>As Good As It Gets (1997)</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               Movie name  model.qi row idx\n",
       "0                      Restoration (1995)                 0\n",
       "1  Monty Python and the Holy Grail (1974)                 1\n",
       "2                Starship Troopers (1997)                 2\n",
       "3                         Swingers (1996)                 3\n",
       "4               As Good As It Gets (1997)                 4"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_to_row_idx: Dict[Any, int] = model.trainset._raw2inner_id_items\n",
    "\n",
    "# Unpacking a dict into a DataFrame for readibility\n",
    "item_to_row_idx_df = pd.DataFrame(\n",
    "    list(item_to_row_idx.items()),\n",
    "    columns=['Movie name', 'model.qi row idx'],\n",
    ")\n",
    "item_to_row_idx_df.head(5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Identifying Toy Story\n",
    "\n",
    "It's very easy to fetch the latent features of a product. Once we have the **row index** of the product, we can fetch the **row** from the `model.qi` matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "toy_story_row_idx : int = item_to_row_idx['Toy Story (1995)']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00974943,  0.03265917,  0.03659085, -0.06292274, -0.06491639,\n",
       "        0.09442259, -0.02551203, -0.02027142, -0.02335466, -0.01492428,\n",
       "       -0.06290319,  0.01435514, -0.00619643,  0.01799988,  0.02820319,\n",
       "       -0.21907753, -0.05845834,  0.08035567,  0.05981695, -0.09046247,\n",
       "        0.22661292,  0.03933661, -0.10093707, -0.15839697, -0.01699003,\n",
       "        0.07315269, -0.11690985, -0.08911202, -0.09714794, -0.0814975 ,\n",
       "       -0.01107214,  0.12426467, -0.19211571,  0.08327614, -0.07947047,\n",
       "        0.06593131,  0.0065287 ,  0.01774967,  0.04007586,  0.16557257,\n",
       "       -0.01542594, -0.19190932, -0.0248609 ,  0.13164055,  0.07578262,\n",
       "       -0.0357468 , -0.16622227, -0.18182406,  0.06855151, -0.03599035,\n",
       "        0.07501229,  0.01276342, -0.0289171 , -0.01152304, -0.09080726,\n",
       "       -0.1464064 , -0.17687792, -0.09365408,  0.0740219 , -0.10258863,\n",
       "        0.08704034, -0.05372785,  0.09271249,  0.04955725,  0.17063114,\n",
       "        0.18700405, -0.13139778, -0.07092693,  0.05760666,  0.10850156,\n",
       "       -0.03169866, -0.11674676,  0.0719402 , -0.04987593, -0.03947938,\n",
       "       -0.05978007,  0.05884171, -0.14392439,  0.09604145, -0.04881222,\n",
       "        0.00098721, -0.00282168, -0.03589193,  0.15316648, -0.28675625,\n",
       "       -0.09344255,  0.10528893,  0.06598872,  0.19359322,  0.0092357 ,\n",
       "       -0.14618867, -0.03667336, -0.08024489,  0.11369858, -0.07232714,\n",
       "       -0.16099911,  0.11624157, -0.07265093, -0.05405805, -0.03981238])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.qi[toy_story_row_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Every product has 100 features\n"
     ]
    }
   ],
   "source": [
    "print(f\"Every product has {model.qi[toy_story_row_idx].shape[0]} features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Recommendations via Product based CF: Finding similarity between vectors\n",
    "\n",
    "2 products are \"similar\" when the cosine distance of their latent features is <= 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from scipy.spatial.distance import cosine\n",
    "\n",
    "\n",
    "def cosine_distance(vector_a: np.array, vector_b: np.array) -> float:\n",
    "    return cosine(vector_a, vector_b)\n",
    "\n",
    "def get_vector_by_movie_title(movie_title: str, trained_model: SVD) -> np.array:\n",
    "    movie_row_idx = trained_model.trainset._raw2inner_id_items[movie_title]\n",
    "    return trained_model.qi[movie_row_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.32043956574320265"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fetch the vectors of \"Toy Story\" and \"Wizard of Oz\"\n",
    "toy_story_vector: np.array = get_vector_by_movie_title('Toy Story (1995)', model)\n",
    "wizard_of_oz: np.array = get_vector_by_movie_title('Wizard of Oz, The (1939)', model)\n",
    "\n",
    "# Calculate the distance between the vectors. The smaller the number,\n",
    "# the more similar the two movies are\n",
    "similarity_score = cosine_distance(toy_story_vector, wizard_of_oz)\n",
    "similarity_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "def get_top_similarities(movie_title: str, model: SVD) -> pd.DataFrame:\n",
    "    movie_vector: np.array = get_vector_by_movie_title(movie_title, model)\n",
    "    similarity_table = []\n",
    "    for other_movie_title in model.trainset._raw2inner_id_items.keys():\n",
    "        other_movie_vector = get_vector_by_movie_title(other_movie_title, model)\n",
    "        similarity_score = cosine_distance(other_movie_vector, movie_vector)\n",
    "        similarity_table.append((similarity_score, other_movie_title))\n",
    "        \n",
    "    similarity_table = pd.DataFrame(\n",
    "        similarity_table,\n",
    "        columns=['similarity', 'movie title']\n",
    "    ).sort_values('similarity', ascending=True)\n",
    "    \n",
    "    return similarity_table.iloc[:10]\n",
    "    \n",
    "    return similarity_table.set_index('movie title').join(movie_data).iloc[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>similarity</th>\n",
       "      <th>movie title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1094</th>\n",
       "      <td>0.235412</td>\n",
       "      <td>Affair to Remember, An (1957)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>0.258220</td>\n",
       "      <td>Shawshank Redemption, The (1994)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>291</th>\n",
       "      <td>0.271727</td>\n",
       "      <td>Grand Day Out, A (1992)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>0.282496</td>\n",
       "      <td>Wrong Trousers, The (1993)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>867</th>\n",
       "      <td>0.283445</td>\n",
       "      <td>Mr. Smith Goes to Washington (1939)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>0.287266</td>\n",
       "      <td>Abyss, The (1989)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>0.290202</td>\n",
       "      <td>E.T. the Extra-Terrestrial (1982)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466</th>\n",
       "      <td>0.291383</td>\n",
       "      <td>Schindler's List (1993)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>0.292125</td>\n",
       "      <td>Stand by Me (1986)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      similarity                          movie title\n",
       "209     0.000000                     Toy Story (1995)\n",
       "1094    0.235412        Affair to Remember, An (1957)\n",
       "90      0.258220     Shawshank Redemption, The (1994)\n",
       "291     0.271727              Grand Day Out, A (1992)\n",
       "192     0.282496           Wrong Trousers, The (1993)\n",
       "867     0.283445  Mr. Smith Goes to Washington (1939)\n",
       "36      0.287266                    Abyss, The (1989)\n",
       "145     0.290202    E.T. the Extra-Terrestrial (1982)\n",
       "466     0.291383              Schindler's List (1993)\n",
       "120     0.292125                   Stand by Me (1986)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_top_similarities('Toy Story (1995)', model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>similarity</th>\n",
       "      <th>movie title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>657</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>Monty Python's Life of Brian (1979)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.191159</td>\n",
       "      <td>Monty Python and the Holy Grail (1974)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>0.233888</td>\n",
       "      <td>Ran (1985)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>0.245717</td>\n",
       "      <td>Wrong Trousers, The (1993)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>555</th>\n",
       "      <td>0.246895</td>\n",
       "      <td>Philadelphia Story, The (1940)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>0.254793</td>\n",
       "      <td>Up in Smoke (1978)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187</th>\n",
       "      <td>0.256729</td>\n",
       "      <td>Delicatessen (1991)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1116</th>\n",
       "      <td>0.257891</td>\n",
       "      <td>Body Snatcher, The (1945)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>0.260078</td>\n",
       "      <td>Red Rock West (1992)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>0.260469</td>\n",
       "      <td>Lost Horizon (1937)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      similarity                             movie title\n",
       "657     0.000000     Monty Python's Life of Brian (1979)\n",
       "1       0.191159  Monty Python and the Holy Grail (1974)\n",
       "142     0.233888                              Ran (1985)\n",
       "192     0.245717              Wrong Trousers, The (1993)\n",
       "555     0.246895          Philadelphia Story, The (1940)\n",
       "234     0.254793                      Up in Smoke (1978)\n",
       "187     0.256729                     Delicatessen (1991)\n",
       "1116    0.257891               Body Snatcher, The (1945)\n",
       "682     0.260078                    Red Rock West (1992)\n",
       "390     0.260469                     Lost Horizon (1937)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_top_similarities('Monty Python\\'s Life of Brian (1979)', model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Recommendations via Matrix Reconstruction: Reconstructing the score between vectors\n",
    "\n",
    "So far we have seen how the `predict()` API works in surface. But how does it **really** work inside of surprise. It's, surprisingly, simple! (get the pun?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def dot_product(vector_a: np.array, vector_b: np.array) -> float:\n",
    "    return vector_a.dot(vector_b)\n",
    "\n",
    "\n",
    "def get_vector_by_user_name(user_name: str, trained_model: SVD) -> np.array:\n",
    "    movie_row_idx = trained_model.trainset._raw2inner_id_users[user_name]\n",
    "    return trained_model.pu[movie_row_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.4846220818359699"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_908_vector = get_vector_by_user_name(\"User 908\", model)\n",
    "dot_product(user_908_vector, toy_story_vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}