Month 2 Part II - Item Item Collaborative Filtering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the [last notebook](https://github.com/caiomiyashiro/RecommenderSystemsNotebooks/blob/master/Month%202%20Part%20I%20-%20User%20User%20Collaborative%20Filtering.ipynb), we took a look at the Nearest Neighboor User User CF, a form of recommender system that looked at the similarity between users to define which item should be suggested to a new customer.\n",
    "\n",
    "We saw the benefits from User User CF when comparing it to non personalised and content based recommendations, but we also saw that it comes with one difficulty.\n",
    "\n",
    "- It doesn't scale well. Even in a big e-commerce dataset, the amount of intersecting items between 2 users is not as big as it could be when User User CF was created. Because of this, when a user bought an item that intersected now with another different customer, the new similarity could drastically change, causing the system's owners to recalculate the new similarities very often.\n",
    "  \n",
    "- If they don't update the matrix often, they can also lose profits over it because it wouldn't map the user's short term interests which, on the internet, can vary quite a lot.\n",
    "\n",
    "All in all, short term interest and sparse mutual interest space make the User User CF inapt for high scale companies.\n",
    "\n",
    "# Item Item Collaborative Filtering (CF)\n",
    "\n",
    "[Item Item CF](https://en.wikipedia.org/wiki/Item-item_collaborative_filtering) was created by ([Sarwar et all, 1998](https://patentimages.storage.googleapis.com/41/80/fb/07d4d9e61e7431/US6266649.pdf)) in partneship with Amazon in order to fix the problems with the User User CF. In Item Item perspective, as the name suggests, changes the perspective from User centered to a Item centered view, *i.e.*, instead of having a User User similarity matrix, they started to use a item item similarity matrix. Then, when a user $u$ bought and liked an item $i_{1}$ and $i_{1}$ was similar to item $i_{2}$, then we predicted that $u$ would also like $i_{2}$. Take a look at the image below:\n",
    "\n",
    "<img src=\"images/notebook5_image1.jpeg\" width=\"500\">\n",
    "\n",
    "Why this simple change in perspective helped to solve the inneficiency problems present in the User User CF?\n",
    "\n",
    "By considering an enviroment of a big e-commerce company, we end with the number of users >> number of items.   \n",
    "In this case, even if a single user hasn't given many reviews, the chances are that many users have given a review to a specific item.\n",
    "By having a big number of reviews, an item relationship to other items doesn't change too much by receiving a few more reviews, *i.e.*, item item relationship are more stable. Therefore, by being more stable, the similarity matrix doesn't have to be recalculated often, as in the User User CF.\n",
    "  \n",
    "An extra perfomance improvement comes also from the prediction calculation. In the Item Item CF, a new prediction for a user $u$ for a product $p$ is made by retrieving the items similarities and calculating a weighted average. The number of neighboors for this calculation is only the item that $u$ has liked or bought in the past and this number is often small enought. Therefore, we don't need to search the big user user similarity matrix to find the best $k$ neighboors.\n",
    "\n",
    "# Item Item Steps\n",
    "\n",
    "As always, we're going to work with one of the datasets from the [Coursera's Specialization on Recommender Systems](https://www.coursera.org/specializations/recommender-systems). This dataset is from the last week in the course of [Nearest Neighboors CF](https://www.coursera.org/learn/collaborative-filtering) for Item Item CF. Dataset is [here](https://d396qusza40orc.cloudfront.net/umntestsite/on-demand_files/A5/Assignment%205.xls) (Coursera's page) and [here](https://github.com/caiomiyashiro/RecommenderSystemsNotebooks/blob/master/data/Item%20Item%20Collaborative%20Filtering%20-%20Ratings.csv) (personal Github account).\n",
    "  \n",
    "The steps taken to evaluate and recommend are similar to User User CF, with some different calculations in the prediction step, as we've said.\n",
    "\n",
    "- Load traditional input - User Item Review dataset\n",
    "- Create similarity matrix\n",
    "- Make predictions\n",
    "\n",
    "Lets go!\n",
    "\n",
    "## Example Dataset\n",
    "\n",
    "The dataset is a matrix with size 25 users x 25 movies and each cell $c_{u,m}$ contains the rating user $u$ gave to movie $m$. If user $u$ didn't rate movie $m$, the cell is empty. As the float values were stored with commas and consequently were being casted as strings, I had to process it a little bit to replace the commas for dots and then convert the column to floats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset shape: (20, 20)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1: Toy Story (1995)</th>\n",
       "      <th>1210: Star Wars: Episode VI - Return of the Jedi (1983)</th>\n",
       "      <th>356: Forrest Gump (1994)</th>\n",
       "      <th>318: Shawshank Redemption, The (1994)</th>\n",
       "      <th>593: Silence of the Lambs, The (1991)</th>\n",
       "      <th>3578: Gladiator (2000)</th>\n",
       "      <th>260: Star Wars: Episode IV - A New Hope (1977)</th>\n",
       "      <th>2028: Saving Private Ryan (1998)</th>\n",
       "      <th>296: Pulp Fiction (1994)</th>\n",
       "      <th>1259: Stand by Me (1986)</th>\n",
       "      <th>2396: Shakespeare in Love (1998)</th>\n",
       "      <th>2916: Total Recall (1990)</th>\n",
       "      <th>780: Independence Day (ID4) (1996)</th>\n",
       "      <th>541: Blade Runner (1982)</th>\n",
       "      <th>1265: Groundhog Day (1993)</th>\n",
       "      <th>2571: Matrix, The (1999)</th>\n",
       "      <th>527: Schindler's List (1993)</th>\n",
       "      <th>2762: Sixth Sense, The (1999)</th>\n",
       "      <th>1198: Raiders of the Lost Ark (1981)</th>\n",
       "      <th>34: Babe (1995)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>User</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>755</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5277</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1577</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4388</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1202</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      1: Toy Story (1995)  \\\n",
       "User                        \n",
       "755                   2.0   \n",
       "5277                  1.0   \n",
       "1577                  NaN   \n",
       "4388                  2.0   \n",
       "1202                  NaN   \n",
       "\n",
       "      1210: Star Wars: Episode VI - Return of the Jedi (1983)  \\\n",
       "User                                                            \n",
       "755                                                 5.0         \n",
       "5277                                                NaN         \n",
       "1577                                                NaN         \n",
       "4388                                                3.0         \n",
       "1202                                                3.0         \n",
       "\n",
       "      356: Forrest Gump (1994)  318: Shawshank Redemption, The (1994)  \\\n",
       "User                                                                    \n",
       "755                        2.0                                    NaN   \n",
       "5277                       NaN                                    2.0   \n",
       "1577                       NaN                                    5.0   \n",
       "4388                       NaN                                    NaN   \n",
       "1202                       4.0                                    1.0   \n",
       "\n",
       "      593: Silence of the Lambs, The (1991)  3578: Gladiator (2000)  \\\n",
       "User                                                                  \n",
       "755                                     4.0                     4.0   \n",
       "5277                                    4.0                     2.0   \n",
       "1577                                    2.0                     NaN   \n",
       "4388                                    NaN                     1.0   \n",
       "1202                                    4.0                     1.0   \n",
       "\n",
       "      260: Star Wars: Episode IV - A New Hope (1977)  \\\n",
       "User                                                   \n",
       "755                                              1.0   \n",
       "5277                                             5.0   \n",
       "1577                                             NaN   \n",
       "4388                                             NaN   \n",
       "1202                                             4.0   \n",
       "\n",
       "      2028: Saving Private Ryan (1998)  296: Pulp Fiction (1994)  \\\n",
       "User                                                               \n",
       "755                                2.0                       NaN   \n",
       "5277                               NaN                       NaN   \n",
       "1577                               NaN                       NaN   \n",
       "4388                               3.0                       4.0   \n",
       "1202                               4.0                       NaN   \n",
       "\n",
       "      1259: Stand by Me (1986)  2396: Shakespeare in Love (1998)  \\\n",
       "User                                                               \n",
       "755                        3.0                               2.0   \n",
       "5277                       4.0                               3.0   \n",
       "1577                       1.0                               NaN   \n",
       "4388                       NaN                               NaN   \n",
       "1202                       1.0                               5.0   \n",
       "\n",
       "      2916: Total Recall (1990)  780: Independence Day (ID4) (1996)  \\\n",
       "User                                                                  \n",
       "755                         NaN                                 5.0   \n",
       "5277                        2.0                                 2.0   \n",
       "1577                        1.0                                 4.0   \n",
       "4388                        4.0                                 NaN   \n",
       "1202                        1.0                                 NaN   \n",
       "\n",
       "      541: Blade Runner (1982)  1265: Groundhog Day (1993)  \\\n",
       "User                                                         \n",
       "755                        2.0                         5.0   \n",
       "5277                       NaN                         2.0   \n",
       "1577                       4.0                         1.0   \n",
       "4388                       3.0                         5.0   \n",
       "1202                       4.0                         NaN   \n",
       "\n",
       "      2571: Matrix, The (1999)  527: Schindler's List (1993)  \\\n",
       "User                                                           \n",
       "755                        4.0                           2.0   \n",
       "5277                       NaN                           5.0   \n",
       "1577                       1.0                           2.0   \n",
       "4388                       NaN                           5.0   \n",
       "1202                       3.0                           5.0   \n",
       "\n",
       "      2762: Sixth Sense, The (1999)  1198: Raiders of the Lost Ark (1981)  \\\n",
       "User                                                                        \n",
       "755                             5.0                                   NaN   \n",
       "5277                            1.0                                   3.0   \n",
       "1577                            3.0                                   1.0   \n",
       "4388                            1.0                                   1.0   \n",
       "1202                            5.0                                   NaN   \n",
       "\n",
       "      34: Babe (1995)  \n",
       "User                   \n",
       "755               NaN  \n",
       "5277              NaN  \n",
       "1577              3.0  \n",
       "4388              2.0  \n",
       "1202              NaN  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/Item Item Collaborative Filtering - Ratings.csv', index_col=0, nrows=20)\n",
    "\n",
    "df.drop('Mean', axis=1, inplace=True) # remove mean column that comes at the end\n",
    "\n",
    "# replace commas for dots and convert previous string column into float\n",
    "def processCol(col):\n",
    "    return col.astype(str).apply(lambda val: val.replace(',','.')).astype(float)\n",
    "df = df.apply(processCol)\n",
    "\n",
    "print('Dataset shape: ' + str(df.shape))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Similarity Matrix\n",
    "\n",
    "\n",
    "### Similarity Function\n",
    "\n",
    "As for the User User CF, we have a few possibilities to choose from when deciding how we're going to define if an item is similar to another item. Again, ([Herlocker et all, 2002](https://grouplens.org/site-content/uploads/evaluating-TOIS-20041.pdf)) did an analysis on the performance of these metrics on Item Item CF and realised that, for this case, the [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) was the best performant metric. So we're going with them this time. On the **next notebook**, we try to analyse these metrics and see why it performs better in certain cases and others not.\n",
    "\n",
    "\n",
    "## Calculating User User Similarity with Cosine Similarity:\n",
    "\n",
    "One important point here is on the calculation of the denominator of the cosine similarity. Even though we make the dot product only with values existing in both arrays, the norm of the individual vectors are considering all values, and not the intersections between array1 and array2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1: Toy Story (1995)</th>\n",
       "      <th>1210: Star Wars: Episode VI - Return of the Jedi (1983)</th>\n",
       "      <th>356: Forrest Gump (1994)</th>\n",
       "      <th>318: Shawshank Redemption, The (1994)</th>\n",
       "      <th>593: Silence of the Lambs, The (1991)</th>\n",
       "      <th>3578: Gladiator (2000)</th>\n",
       "      <th>260: Star Wars: Episode IV - A New Hope (1977)</th>\n",
       "      <th>2028: Saving Private Ryan (1998)</th>\n",
       "      <th>296: Pulp Fiction (1994)</th>\n",
       "      <th>1259: Stand by Me (1986)</th>\n",
       "      <th>2396: Shakespeare in Love (1998)</th>\n",
       "      <th>2916: Total Recall (1990)</th>\n",
       "      <th>780: Independence Day (ID4) (1996)</th>\n",
       "      <th>541: Blade Runner (1982)</th>\n",
       "      <th>1265: Groundhog Day (1993)</th>\n",
       "      <th>2571: Matrix, The (1999)</th>\n",
       "      <th>527: Schindler's List (1993)</th>\n",
       "      <th>2762: Sixth Sense, The (1999)</th>\n",
       "      <th>1198: Raiders of the Lost Ark (1981)</th>\n",
       "      <th>34: Babe (1995)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1: Toy Story (1995)</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.644995</td>\n",
       "      <td>0.580540</td>\n",
       "      <td>0.667424</td>\n",
       "      <td>0.570229</td>\n",
       "      <td>0.587852</td>\n",
       "      <td>0.747409</td>\n",
       "      <td>0.534579</td>\n",
       "      <td>0.667846</td>\n",
       "      <td>0.492659</td>\n",
       "      <td>0.376659</td>\n",
       "      <td>0.623056</td>\n",
       "      <td>0.690665</td>\n",
       "      <td>0.383067</td>\n",
       "      <td>0.661016</td>\n",
       "      <td>0.505010</td>\n",
       "      <td>0.463817</td>\n",
       "      <td>0.421637</td>\n",
       "      <td>0.466817</td>\n",
       "      <td>0.618070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1210: Star Wars: Episode VI - Return of the Jedi (1983)</th>\n",
       "      <td>0.644995</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.563029</td>\n",
       "      <td>0.456052</td>\n",
       "      <td>0.516566</td>\n",
       "      <td>0.483187</td>\n",
       "      <td>0.589805</td>\n",
       "      <td>0.408752</td>\n",
       "      <td>0.685662</td>\n",
       "      <td>0.534324</td>\n",
       "      <td>0.533429</td>\n",
       "      <td>0.391934</td>\n",
       "      <td>0.605856</td>\n",
       "      <td>0.515397</td>\n",
       "      <td>0.526952</td>\n",
       "      <td>0.535673</td>\n",
       "      <td>0.573529</td>\n",
       "      <td>0.565297</td>\n",
       "      <td>0.252604</td>\n",
       "      <td>0.511576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>356: Forrest Gump (1994)</th>\n",
       "      <td>0.580540</td>\n",
       "      <td>0.563029</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.293041</td>\n",
       "      <td>0.381346</td>\n",
       "      <td>0.569209</td>\n",
       "      <td>0.595550</td>\n",
       "      <td>0.463003</td>\n",
       "      <td>0.399114</td>\n",
       "      <td>0.527926</td>\n",
       "      <td>0.647153</td>\n",
       "      <td>0.491498</td>\n",
       "      <td>0.498741</td>\n",
       "      <td>0.487713</td>\n",
       "      <td>0.298290</td>\n",
       "      <td>0.631039</td>\n",
       "      <td>0.320494</td>\n",
       "      <td>0.602943</td>\n",
       "      <td>0.288275</td>\n",
       "      <td>0.456849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318: Shawshank Redemption, The (1994)</th>\n",
       "      <td>0.667424</td>\n",
       "      <td>0.456052</td>\n",
       "      <td>0.293041</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.589000</td>\n",
       "      <td>0.212846</td>\n",
       "      <td>0.565577</td>\n",
       "      <td>0.598344</td>\n",
       "      <td>0.538219</td>\n",
       "      <td>0.340151</td>\n",
       "      <td>0.329203</td>\n",
       "      <td>0.332674</td>\n",
       "      <td>0.617366</td>\n",
       "      <td>0.531981</td>\n",
       "      <td>0.437319</td>\n",
       "      <td>0.255345</td>\n",
       "      <td>0.497511</td>\n",
       "      <td>0.459446</td>\n",
       "      <td>0.467347</td>\n",
       "      <td>0.542782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>593: Silence of the Lambs, The (1991)</th>\n",
       "      <td>0.570229</td>\n",
       "      <td>0.516566</td>\n",
       "      <td>0.381346</td>\n",
       "      <td>0.589000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.551612</td>\n",
       "      <td>0.682137</td>\n",
       "      <td>0.640590</td>\n",
       "      <td>0.400471</td>\n",
       "      <td>0.661958</td>\n",
       "      <td>0.484751</td>\n",
       "      <td>0.414499</td>\n",
       "      <td>0.738445</td>\n",
       "      <td>0.585662</td>\n",
       "      <td>0.673091</td>\n",
       "      <td>0.530856</td>\n",
       "      <td>0.757630</td>\n",
       "      <td>0.715565</td>\n",
       "      <td>0.702452</td>\n",
       "      <td>0.309159</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    1: Toy Story (1995)  \\\n",
       "1: Toy Story (1995)                                            1.000000   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...             0.644995   \n",
       "356: Forrest Gump (1994)                                       0.580540   \n",
       "318: Shawshank Redemption, The (1994)                          0.667424   \n",
       "593: Silence of the Lambs, The (1991)                          0.570229   \n",
       "\n",
       "                                                    1210: Star Wars: Episode VI - Return of the Jedi (1983)  \\\n",
       "1: Toy Story (1995)                                                                          0.644995         \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                                           1.000000         \n",
       "356: Forrest Gump (1994)                                                                     0.563029         \n",
       "318: Shawshank Redemption, The (1994)                                                        0.456052         \n",
       "593: Silence of the Lambs, The (1991)                                                        0.516566         \n",
       "\n",
       "                                                    356: Forrest Gump (1994)  \\\n",
       "1: Toy Story (1995)                                                 0.580540   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                  0.563029   \n",
       "356: Forrest Gump (1994)                                            1.000000   \n",
       "318: Shawshank Redemption, The (1994)                               0.293041   \n",
       "593: Silence of the Lambs, The (1991)                               0.381346   \n",
       "\n",
       "                                                    318: Shawshank Redemption, The (1994)  \\\n",
       "1: Toy Story (1995)                                                              0.667424   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                               0.456052   \n",
       "356: Forrest Gump (1994)                                                         0.293041   \n",
       "318: Shawshank Redemption, The (1994)                                            1.000000   \n",
       "593: Silence of the Lambs, The (1991)                                            0.589000   \n",
       "\n",
       "                                                    593: Silence of the Lambs, The (1991)  \\\n",
       "1: Toy Story (1995)                                                              0.570229   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                               0.516566   \n",
       "356: Forrest Gump (1994)                                                         0.381346   \n",
       "318: Shawshank Redemption, The (1994)                                            0.589000   \n",
       "593: Silence of the Lambs, The (1991)                                            1.000000   \n",
       "\n",
       "                                                    3578: Gladiator (2000)  \\\n",
       "1: Toy Story (1995)                                               0.587852   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                0.483187   \n",
       "356: Forrest Gump (1994)                                          0.569209   \n",
       "318: Shawshank Redemption, The (1994)                             0.212846   \n",
       "593: Silence of the Lambs, The (1991)                             0.551612   \n",
       "\n",
       "                                                    260: Star Wars: Episode IV - A New Hope (1977)  \\\n",
       "1: Toy Story (1995)                                                                       0.747409   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                                        0.589805   \n",
       "356: Forrest Gump (1994)                                                                  0.595550   \n",
       "318: Shawshank Redemption, The (1994)                                                     0.565577   \n",
       "593: Silence of the Lambs, The (1991)                                                     0.682137   \n",
       "\n",
       "                                                    2028: Saving Private Ryan (1998)  \\\n",
       "1: Toy Story (1995)                                                         0.534579   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                          0.408752   \n",
       "356: Forrest Gump (1994)                                                    0.463003   \n",
       "318: Shawshank Redemption, The (1994)                                       0.598344   \n",
       "593: Silence of the Lambs, The (1991)                                       0.640590   \n",
       "\n",
       "                                                    296: Pulp Fiction (1994)  \\\n",
       "1: Toy Story (1995)                                                 0.667846   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                  0.685662   \n",
       "356: Forrest Gump (1994)                                            0.399114   \n",
       "318: Shawshank Redemption, The (1994)                               0.538219   \n",
       "593: Silence of the Lambs, The (1991)                               0.400471   \n",
       "\n",
       "                                                    1259: Stand by Me (1986)  \\\n",
       "1: Toy Story (1995)                                                 0.492659   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                  0.534324   \n",
       "356: Forrest Gump (1994)                                            0.527926   \n",
       "318: Shawshank Redemption, The (1994)                               0.340151   \n",
       "593: Silence of the Lambs, The (1991)                               0.661958   \n",
       "\n",
       "                                                    2396: Shakespeare in Love (1998)  \\\n",
       "1: Toy Story (1995)                                                         0.376659   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                          0.533429   \n",
       "356: Forrest Gump (1994)                                                    0.647153   \n",
       "318: Shawshank Redemption, The (1994)                                       0.329203   \n",
       "593: Silence of the Lambs, The (1991)                                       0.484751   \n",
       "\n",
       "                                                    2916: Total Recall (1990)  \\\n",
       "1: Toy Story (1995)                                                  0.623056   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                   0.391934   \n",
       "356: Forrest Gump (1994)                                             0.491498   \n",
       "318: Shawshank Redemption, The (1994)                                0.332674   \n",
       "593: Silence of the Lambs, The (1991)                                0.414499   \n",
       "\n",
       "                                                    780: Independence Day (ID4) (1996)  \\\n",
       "1: Toy Story (1995)                                                           0.690665   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                            0.605856   \n",
       "356: Forrest Gump (1994)                                                      0.498741   \n",
       "318: Shawshank Redemption, The (1994)                                         0.617366   \n",
       "593: Silence of the Lambs, The (1991)                                         0.738445   \n",
       "\n",
       "                                                    541: Blade Runner (1982)  \\\n",
       "1: Toy Story (1995)                                                 0.383067   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                  0.515397   \n",
       "356: Forrest Gump (1994)                                            0.487713   \n",
       "318: Shawshank Redemption, The (1994)                               0.531981   \n",
       "593: Silence of the Lambs, The (1991)                               0.585662   \n",
       "\n",
       "                                                    1265: Groundhog Day (1993)  \\\n",
       "1: Toy Story (1995)                                                   0.661016   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                    0.526952   \n",
       "356: Forrest Gump (1994)                                              0.298290   \n",
       "318: Shawshank Redemption, The (1994)                                 0.437319   \n",
       "593: Silence of the Lambs, The (1991)                                 0.673091   \n",
       "\n",
       "                                                    2571: Matrix, The (1999)  \\\n",
       "1: Toy Story (1995)                                                 0.505010   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                  0.535673   \n",
       "356: Forrest Gump (1994)                                            0.631039   \n",
       "318: Shawshank Redemption, The (1994)                               0.255345   \n",
       "593: Silence of the Lambs, The (1991)                               0.530856   \n",
       "\n",
       "                                                    527: Schindler's List (1993)  \\\n",
       "1: Toy Story (1995)                                                     0.463817   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                      0.573529   \n",
       "356: Forrest Gump (1994)                                                0.320494   \n",
       "318: Shawshank Redemption, The (1994)                                   0.497511   \n",
       "593: Silence of the Lambs, The (1991)                                   0.757630   \n",
       "\n",
       "                                                    2762: Sixth Sense, The (1999)  \\\n",
       "1: Toy Story (1995)                                                      0.421637   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                       0.565297   \n",
       "356: Forrest Gump (1994)                                                 0.602943   \n",
       "318: Shawshank Redemption, The (1994)                                    0.459446   \n",
       "593: Silence of the Lambs, The (1991)                                    0.715565   \n",
       "\n",
       "                                                    1198: Raiders of the Lost Ark (1981)  \\\n",
       "1: Toy Story (1995)                                                             0.466817   \n",
       "1210: Star Wars: Episode VI - Return of the Jed...                              0.252604   \n",
       "356: Forrest Gump (1994)                                                        0.288275   \n",
       "318: Shawshank Redemption, The (1994)                                           0.467347   \n",
       "593: Silence of the Lambs, The (1991)                                           0.702452   \n",
       "\n",
       "                                                    34: Babe (1995)  \n",
       "1: Toy Story (1995)                                        0.618070  \n",
       "1210: Star Wars: Episode VI - Return of the Jed...         0.511576  \n",
       "356: Forrest Gump (1994)                                   0.456849  \n",
       "318: Shawshank Redemption, The (1994)                      0.542782  \n",
       "593: Silence of the Lambs, The (1991)                      0.309159  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def cos_similarity(item1, item2):\n",
    "    item1Values = ~np.isnan(item1)\n",
    "    item2Values = ~np.isnan(item2)\n",
    "    allValues = np.logical_and(item1Values,item2Values) # get only existent elements of both vectors\n",
    "    return np.dot(item1[allValues], item2[allValues])/(np.linalg.norm(item1[item1Values]) * np.linalg.norm(item2[item2Values]))\n",
    "\n",
    "def pre_cos_similarity(item1, df):\n",
    "    return df.apply(lambda item2: cos_similarity(item1, item2))\n",
    "\n",
    "df_corr = df.apply(lambda item1: pre_cos_similarity(item1, df))\n",
    "df_corr.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predictions Calculation\n",
    "\n",
    "By now we already know which items are more similar to each other. This will help us when predicting a new rating, by giving higher weights for more similar items than other the user has bough. \n",
    "  \n",
    "The way we're going to calculate the new predictions is the same we used for User User CF, *i.e.*, a weighted average:\n",
    "\n",
    "$$\\frac{\\sum_{n=1}^{k} r_{n}w_{n}}{\\sum_{n=1}^{k} w_{n}}$$\n",
    "  \n",
    "The difference is that we don't have the neighboors anymore, so the $n$ in the summation is considering **all** the items user $u$ has rated and $w$ is still the similarities, but now *item similarity*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1: Toy Story (1995)</th>\n",
       "      <th>1210: Star Wars: Episode VI - Return of the Jedi (1983)</th>\n",
       "      <th>356: Forrest Gump (1994)</th>\n",
       "      <th>318: Shawshank Redemption, The (1994)</th>\n",
       "      <th>593: Silence of the Lambs, The (1991)</th>\n",
       "      <th>3578: Gladiator (2000)</th>\n",
       "      <th>260: Star Wars: Episode IV - A New Hope (1977)</th>\n",
       "      <th>2028: Saving Private Ryan (1998)</th>\n",
       "      <th>296: Pulp Fiction (1994)</th>\n",
       "      <th>1259: Stand by Me (1986)</th>\n",
       "      <th>2396: Shakespeare in Love (1998)</th>\n",
       "      <th>2916: Total Recall (1990)</th>\n",
       "      <th>780: Independence Day (ID4) (1996)</th>\n",
       "      <th>541: Blade Runner (1982)</th>\n",
       "      <th>1265: Groundhog Day (1993)</th>\n",
       "      <th>2571: Matrix, The (1999)</th>\n",
       "      <th>527: Schindler's List (1993)</th>\n",
       "      <th>2762: Sixth Sense, The (1999)</th>\n",
       "      <th>1198: Raiders of the Lost Ark (1981)</th>\n",
       "      <th>34: Babe (1995)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>User</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>755</th>\n",
       "      <td>3.192403</td>\n",
       "      <td>3.292306</td>\n",
       "      <td>3.100111</td>\n",
       "      <td>3.138513</td>\n",
       "      <td>3.272413</td>\n",
       "      <td>3.169078</td>\n",
       "      <td>3.040837</td>\n",
       "      <td>3.088188</td>\n",
       "      <td>3.121772</td>\n",
       "      <td>3.259814</td>\n",
       "      <td>3.154181</td>\n",
       "      <td>3.146678</td>\n",
       "      <td>3.409717</td>\n",
       "      <td>3.202508</td>\n",
       "      <td>3.391725</td>\n",
       "      <td>3.309275</td>\n",
       "      <td>3.173982</td>\n",
       "      <td>3.326625</td>\n",
       "      <td>3.169455</td>\n",
       "      <td>3.131759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5277</th>\n",
       "      <td>2.666898</td>\n",
       "      <td>2.774215</td>\n",
       "      <td>2.708550</td>\n",
       "      <td>2.723963</td>\n",
       "      <td>2.883304</td>\n",
       "      <td>2.809767</td>\n",
       "      <td>2.922240</td>\n",
       "      <td>2.780218</td>\n",
       "      <td>2.802797</td>\n",
       "      <td>2.928801</td>\n",
       "      <td>2.852131</td>\n",
       "      <td>2.701826</td>\n",
       "      <td>2.674951</td>\n",
       "      <td>2.784252</td>\n",
       "      <td>2.690720</td>\n",
       "      <td>2.770176</td>\n",
       "      <td>2.973883</td>\n",
       "      <td>2.716267</td>\n",
       "      <td>2.818155</td>\n",
       "      <td>2.610945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1577</th>\n",
       "      <td>2.371890</td>\n",
       "      <td>2.408644</td>\n",
       "      <td>2.317307</td>\n",
       "      <td>2.735981</td>\n",
       "      <td>2.320003</td>\n",
       "      <td>2.144546</td>\n",
       "      <td>2.283465</td>\n",
       "      <td>2.427900</td>\n",
       "      <td>2.397125</td>\n",
       "      <td>2.151301</td>\n",
       "      <td>2.307571</td>\n",
       "      <td>2.146637</td>\n",
       "      <td>2.379497</td>\n",
       "      <td>2.512756</td>\n",
       "      <td>2.227885</td>\n",
       "      <td>2.264895</td>\n",
       "      <td>2.327398</td>\n",
       "      <td>2.440141</td>\n",
       "      <td>2.280008</td>\n",
       "      <td>2.612319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4388</th>\n",
       "      <td>2.854355</td>\n",
       "      <td>2.926799</td>\n",
       "      <td>2.668524</td>\n",
       "      <td>2.869568</td>\n",
       "      <td>2.832946</td>\n",
       "      <td>2.600812</td>\n",
       "      <td>2.803925</td>\n",
       "      <td>2.827416</td>\n",
       "      <td>2.980781</td>\n",
       "      <td>2.890292</td>\n",
       "      <td>2.755777</td>\n",
       "      <td>2.974583</td>\n",
       "      <td>2.834529</td>\n",
       "      <td>2.904316</td>\n",
       "      <td>2.999510</td>\n",
       "      <td>2.911118</td>\n",
       "      <td>3.065207</td>\n",
       "      <td>2.784864</td>\n",
       "      <td>2.678815</td>\n",
       "      <td>2.626968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1202</th>\n",
       "      <td>3.078704</td>\n",
       "      <td>3.287867</td>\n",
       "      <td>3.304391</td>\n",
       "      <td>3.213417</td>\n",
       "      <td>3.325591</td>\n",
       "      <td>3.081952</td>\n",
       "      <td>3.191459</td>\n",
       "      <td>3.384223</td>\n",
       "      <td>3.175650</td>\n",
       "      <td>3.112196</td>\n",
       "      <td>3.457590</td>\n",
       "      <td>3.038464</td>\n",
       "      <td>3.209556</td>\n",
       "      <td>3.447542</td>\n",
       "      <td>3.204562</td>\n",
       "      <td>3.465700</td>\n",
       "      <td>3.396677</td>\n",
       "      <td>3.488553</td>\n",
       "      <td>3.205717</td>\n",
       "      <td>3.146035</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      1: Toy Story (1995)  \\\n",
       "User                        \n",
       "755              3.192403   \n",
       "5277             2.666898   \n",
       "1577             2.371890   \n",
       "4388             2.854355   \n",
       "1202             3.078704   \n",
       "\n",
       "      1210: Star Wars: Episode VI - Return of the Jedi (1983)  \\\n",
       "User                                                            \n",
       "755                                            3.292306         \n",
       "5277                                           2.774215         \n",
       "1577                                           2.408644         \n",
       "4388                                           2.926799         \n",
       "1202                                           3.287867         \n",
       "\n",
       "      356: Forrest Gump (1994)  318: Shawshank Redemption, The (1994)  \\\n",
       "User                                                                    \n",
       "755                   3.100111                               3.138513   \n",
       "5277                  2.708550                               2.723963   \n",
       "1577                  2.317307                               2.735981   \n",
       "4388                  2.668524                               2.869568   \n",
       "1202                  3.304391                               3.213417   \n",
       "\n",
       "      593: Silence of the Lambs, The (1991)  3578: Gladiator (2000)  \\\n",
       "User                                                                  \n",
       "755                                3.272413                3.169078   \n",
       "5277                               2.883304                2.809767   \n",
       "1577                               2.320003                2.144546   \n",
       "4388                               2.832946                2.600812   \n",
       "1202                               3.325591                3.081952   \n",
       "\n",
       "      260: Star Wars: Episode IV - A New Hope (1977)  \\\n",
       "User                                                   \n",
       "755                                         3.040837   \n",
       "5277                                        2.922240   \n",
       "1577                                        2.283465   \n",
       "4388                                        2.803925   \n",
       "1202                                        3.191459   \n",
       "\n",
       "      2028: Saving Private Ryan (1998)  296: Pulp Fiction (1994)  \\\n",
       "User                                                               \n",
       "755                           3.088188                  3.121772   \n",
       "5277                          2.780218                  2.802797   \n",
       "1577                          2.427900                  2.397125   \n",
       "4388                          2.827416                  2.980781   \n",
       "1202                          3.384223                  3.175650   \n",
       "\n",
       "      1259: Stand by Me (1986)  2396: Shakespeare in Love (1998)  \\\n",
       "User                                                               \n",
       "755                   3.259814                          3.154181   \n",
       "5277                  2.928801                          2.852131   \n",
       "1577                  2.151301                          2.307571   \n",
       "4388                  2.890292                          2.755777   \n",
       "1202                  3.112196                          3.457590   \n",
       "\n",
       "      2916: Total Recall (1990)  780: Independence Day (ID4) (1996)  \\\n",
       "User                                                                  \n",
       "755                    3.146678                            3.409717   \n",
       "5277                   2.701826                            2.674951   \n",
       "1577                   2.146637                            2.379497   \n",
       "4388                   2.974583                            2.834529   \n",
       "1202                   3.038464                            3.209556   \n",
       "\n",
       "      541: Blade Runner (1982)  1265: Groundhog Day (1993)  \\\n",
       "User                                                         \n",
       "755                   3.202508                    3.391725   \n",
       "5277                  2.784252                    2.690720   \n",
       "1577                  2.512756                    2.227885   \n",
       "4388                  2.904316                    2.999510   \n",
       "1202                  3.447542                    3.204562   \n",
       "\n",
       "      2571: Matrix, The (1999)  527: Schindler's List (1993)  \\\n",
       "User                                                           \n",
       "755                   3.309275                      3.173982   \n",
       "5277                  2.770176                      2.973883   \n",
       "1577                  2.264895                      2.327398   \n",
       "4388                  2.911118                      3.065207   \n",
       "1202                  3.465700                      3.396677   \n",
       "\n",
       "      2762: Sixth Sense, The (1999)  1198: Raiders of the Lost Ark (1981)  \\\n",
       "User                                                                        \n",
       "755                        3.326625                              3.169455   \n",
       "5277                       2.716267                              2.818155   \n",
       "1577                       2.440141                              2.280008   \n",
       "4388                       2.784864                              2.678815   \n",
       "1202                       3.488553                              3.205717   \n",
       "\n",
       "      34: Babe (1995)  \n",
       "User                   \n",
       "755          3.131759  \n",
       "5277         2.610945  \n",
       "1577         2.612319  \n",
       "4388         2.626968  \n",
       "1202         3.146035  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def predictRating(userRatings, itemSimilarity):\n",
    "    userHasRating = ~np.isnan(userRatings)\n",
    "    return np.dot(userRatings[userHasRating], itemSimilarity[userHasRating])/np.sum(itemSimilarity[userHasRating])\n",
    "\n",
    "def pre_predictRating(userRatings, df_corr):\n",
    "    return df_corr.apply(lambda itemSimilarity: predictRating(userRatings, itemSimilarity))\n",
    "\n",
    "predictions = df.apply(lambda userRatings: pre_predictRating(userRatings, df_corr), axis=1)\n",
    "predictions.head()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean Normalised Weighted Average\n",
    "\n",
    "As in the same way of the User User CF, we can calculate predictions using the absolute value of the reviews or from the mean centralised values of it. The advantages are the same: consider the scale variability of reviewers when attributing a final score for a item of interest:\n",
    "\n",
    "$$\\bar{r_{u}} + \\frac{\\sum_{n=1}^{k} (r_{n} - \\bar{r_{n}})w_{n}}{\\sum_{n=1}^{k} w_{n}}$$\n",
    "\n",
    "We took the same function as above, but added two extra parameters:\n",
    "- $userMeanRating$: mean average ratings for a specific user\n",
    "- $neighboorsMeanRating$: mean average rating for all the nearest neighboors for a specific user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mean normalise\n",
    "def subtractFromMean(col, meanCol):\n",
    "    result = np.array([np.nan] * col.shape[0])\n",
    "    isValidValue = ~np.isnan(col)\n",
    "    result[isValidValue] = col.values[isValidValue] - meanCol.values[isValidValue]\n",
    "    return result\n",
    "userMeanRatings = df.apply(np.mean, axis=1)\n",
    "df_ratings_norm = df.apply(lambda col: subtractFromMean(col, userMeanRatings))\n",
    "\n",
    "# similarity matrix\n",
    "df_corr_norm = df_ratings_norm.apply(lambda item1: pre_cos_similarity(item1, df_ratings_norm))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove negative correlations\n",
    "\n",
    "In this example, we are replacing the negative correlations by 0, as we can interpret as a maximum weight for unwanted items:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def replaceNegative(col):\n",
    "    col[col < 0] = 0\n",
    "    return col\n",
    "df_corr_norm2 = df_corr_norm.apply(replaceNegative)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1: Toy Story (1995)</th>\n",
       "      <th>1210: Star Wars: Episode VI - Return of the Jedi (1983)</th>\n",
       "      <th>356: Forrest Gump (1994)</th>\n",
       "      <th>318: Shawshank Redemption, The (1994)</th>\n",
       "      <th>593: Silence of the Lambs, The (1991)</th>\n",
       "      <th>3578: Gladiator (2000)</th>\n",
       "      <th>260: Star Wars: Episode IV - A New Hope (1977)</th>\n",
       "      <th>2028: Saving Private Ryan (1998)</th>\n",
       "      <th>296: Pulp Fiction (1994)</th>\n",
       "      <th>1259: Stand by Me (1986)</th>\n",
       "      <th>2396: Shakespeare in Love (1998)</th>\n",
       "      <th>2916: Total Recall (1990)</th>\n",
       "      <th>780: Independence Day (ID4) (1996)</th>\n",
       "      <th>541: Blade Runner (1982)</th>\n",
       "      <th>1265: Groundhog Day (1993)</th>\n",
       "      <th>2571: Matrix, The (1999)</th>\n",
       "      <th>527: Schindler's List (1993)</th>\n",
       "      <th>2762: Sixth Sense, The (1999)</th>\n",
       "      <th>1198: Raiders of the Lost Ark (1981)</th>\n",
       "      <th>34: Babe (1995)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>User</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>755</th>\n",
       "      <td>2.152600</td>\n",
       "      <td>4.365913</td>\n",
       "      <td>2.362540</td>\n",
       "      <td>2.597612</td>\n",
       "      <td>3.900120</td>\n",
       "      <td>3.404381</td>\n",
       "      <td>1.456120</td>\n",
       "      <td>2.120407</td>\n",
       "      <td>2.025099</td>\n",
       "      <td>3.439884</td>\n",
       "      <td>2.286692</td>\n",
       "      <td>3.716030</td>\n",
       "      <td>4.516014</td>\n",
       "      <td>2.615461</td>\n",
       "      <td>4.592515</td>\n",
       "      <td>4.308328</td>\n",
       "      <td>2.098549</td>\n",
       "      <td>4.584935</td>\n",
       "      <td>3.060120</td>\n",
       "      <td>2.611896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5277</th>\n",
       "      <td>1.466851</td>\n",
       "      <td>2.789617</td>\n",
       "      <td>2.904472</td>\n",
       "      <td>1.840136</td>\n",
       "      <td>3.220890</td>\n",
       "      <td>2.324026</td>\n",
       "      <td>4.565822</td>\n",
       "      <td>3.095917</td>\n",
       "      <td>2.804427</td>\n",
       "      <td>3.602141</td>\n",
       "      <td>3.297938</td>\n",
       "      <td>1.996194</td>\n",
       "      <td>2.043163</td>\n",
       "      <td>2.641633</td>\n",
       "      <td>1.922817</td>\n",
       "      <td>2.681204</td>\n",
       "      <td>4.562483</td>\n",
       "      <td>2.032737</td>\n",
       "      <td>2.861378</td>\n",
       "      <td>1.822288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1577</th>\n",
       "      <td>3.084638</td>\n",
       "      <td>2.173100</td>\n",
       "      <td>1.603240</td>\n",
       "      <td>4.375861</td>\n",
       "      <td>2.250719</td>\n",
       "      <td>1.586648</td>\n",
       "      <td>1.725537</td>\n",
       "      <td>1.958046</td>\n",
       "      <td>3.382771</td>\n",
       "      <td>1.161933</td>\n",
       "      <td>2.030983</td>\n",
       "      <td>1.279478</td>\n",
       "      <td>3.472474</td>\n",
       "      <td>3.798850</td>\n",
       "      <td>1.422635</td>\n",
       "      <td>1.487751</td>\n",
       "      <td>2.258958</td>\n",
       "      <td>2.541503</td>\n",
       "      <td>1.401180</td>\n",
       "      <td>2.940755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4388</th>\n",
       "      <td>2.432925</td>\n",
       "      <td>2.910112</td>\n",
       "      <td>2.455503</td>\n",
       "      <td>2.840470</td>\n",
       "      <td>2.335598</td>\n",
       "      <td>1.101522</td>\n",
       "      <td>3.214281</td>\n",
       "      <td>2.863773</td>\n",
       "      <td>3.392348</td>\n",
       "      <td>2.290185</td>\n",
       "      <td>3.688964</td>\n",
       "      <td>3.803343</td>\n",
       "      <td>2.320929</td>\n",
       "      <td>3.045505</td>\n",
       "      <td>3.944862</td>\n",
       "      <td>2.873867</td>\n",
       "      <td>4.401957</td>\n",
       "      <td>1.465993</td>\n",
       "      <td>1.423875</td>\n",
       "      <td>2.785818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1202</th>\n",
       "      <td>2.773346</td>\n",
       "      <td>2.847472</td>\n",
       "      <td>3.383074</td>\n",
       "      <td>1.766545</td>\n",
       "      <td>3.969441</td>\n",
       "      <td>2.040090</td>\n",
       "      <td>4.087295</td>\n",
       "      <td>4.013910</td>\n",
       "      <td>3.085791</td>\n",
       "      <td>2.003364</td>\n",
       "      <td>4.603075</td>\n",
       "      <td>1.338091</td>\n",
       "      <td>3.550230</td>\n",
       "      <td>3.620304</td>\n",
       "      <td>2.597993</td>\n",
       "      <td>3.069085</td>\n",
       "      <td>4.421799</td>\n",
       "      <td>4.215749</td>\n",
       "      <td>3.736910</td>\n",
       "      <td>2.225889</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      1: Toy Story (1995)  \\\n",
       "User                        \n",
       "755              2.152600   \n",
       "5277             1.466851   \n",
       "1577             3.084638   \n",
       "4388             2.432925   \n",
       "1202             2.773346   \n",
       "\n",
       "      1210: Star Wars: Episode VI - Return of the Jedi (1983)  \\\n",
       "User                                                            \n",
       "755                                            4.365913         \n",
       "5277                                           2.789617         \n",
       "1577                                           2.173100         \n",
       "4388                                           2.910112         \n",
       "1202                                           2.847472         \n",
       "\n",
       "      356: Forrest Gump (1994)  318: Shawshank Redemption, The (1994)  \\\n",
       "User                                                                    \n",
       "755                   2.362540                               2.597612   \n",
       "5277                  2.904472                               1.840136   \n",
       "1577                  1.603240                               4.375861   \n",
       "4388                  2.455503                               2.840470   \n",
       "1202                  3.383074                               1.766545   \n",
       "\n",
       "      593: Silence of the Lambs, The (1991)  3578: Gladiator (2000)  \\\n",
       "User                                                                  \n",
       "755                                3.900120                3.404381   \n",
       "5277                               3.220890                2.324026   \n",
       "1577                               2.250719                1.586648   \n",
       "4388                               2.335598                1.101522   \n",
       "1202                               3.969441                2.040090   \n",
       "\n",
       "      260: Star Wars: Episode IV - A New Hope (1977)  \\\n",
       "User                                                   \n",
       "755                                         1.456120   \n",
       "5277                                        4.565822   \n",
       "1577                                        1.725537   \n",
       "4388                                        3.214281   \n",
       "1202                                        4.087295   \n",
       "\n",
       "      2028: Saving Private Ryan (1998)  296: Pulp Fiction (1994)  \\\n",
       "User                                                               \n",
       "755                           2.120407                  2.025099   \n",
       "5277                          3.095917                  2.804427   \n",
       "1577                          1.958046                  3.382771   \n",
       "4388                          2.863773                  3.392348   \n",
       "1202                          4.013910                  3.085791   \n",
       "\n",
       "      1259: Stand by Me (1986)  2396: Shakespeare in Love (1998)  \\\n",
       "User                                                               \n",
       "755                   3.439884                          2.286692   \n",
       "5277                  3.602141                          3.297938   \n",
       "1577                  1.161933                          2.030983   \n",
       "4388                  2.290185                          3.688964   \n",
       "1202                  2.003364                          4.603075   \n",
       "\n",
       "      2916: Total Recall (1990)  780: Independence Day (ID4) (1996)  \\\n",
       "User                                                                  \n",
       "755                    3.716030                            4.516014   \n",
       "5277                   1.996194                            2.043163   \n",
       "1577                   1.279478                            3.472474   \n",
       "4388                   3.803343                            2.320929   \n",
       "1202                   1.338091                            3.550230   \n",
       "\n",
       "      541: Blade Runner (1982)  1265: Groundhog Day (1993)  \\\n",
       "User                                                         \n",
       "755                   2.615461                    4.592515   \n",
       "5277                  2.641633                    1.922817   \n",
       "1577                  3.798850                    1.422635   \n",
       "4388                  3.045505                    3.944862   \n",
       "1202                  3.620304                    2.597993   \n",
       "\n",
       "      2571: Matrix, The (1999)  527: Schindler's List (1993)  \\\n",
       "User                                                           \n",
       "755                   4.308328                      2.098549   \n",
       "5277                  2.681204                      4.562483   \n",
       "1577                  1.487751                      2.258958   \n",
       "4388                  2.873867                      4.401957   \n",
       "1202                  3.069085                      4.421799   \n",
       "\n",
       "      2762: Sixth Sense, The (1999)  1198: Raiders of the Lost Ark (1981)  \\\n",
       "User                                                                        \n",
       "755                        4.584935                              3.060120   \n",
       "5277                       2.032737                              2.861378   \n",
       "1577                       2.541503                              1.401180   \n",
       "4388                       1.465993                              1.423875   \n",
       "1202                       4.215749                              3.736910   \n",
       "\n",
       "      34: Babe (1995)  \n",
       "User                   \n",
       "755          2.611896  \n",
       "5277         1.822288  \n",
       "1577         2.940755  \n",
       "4388         2.785818  \n",
       "1202         2.225889  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_norm = df.apply(lambda userRatings: pre_predictRating(userRatings, df_corr_norm2), axis=1)\n",
    "predictions_norm.head() \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* I didn't quite understand why we didn't use the not normalised item's ratings in this calculation. Ideally, we would use the mean centered user ratings as well.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison Between Approaches\n",
    "\n",
    "The comparison follows the same guidelines we used when evaluating the User User CF. \n",
    "\n",
    "\n",
    "# Final Considerations on User User CF\n",
    "\n",
    "When we mean centered the user's rating for the User User CF, the objective was clear, we wanted to take into account that users rate in different parts of the scale. But what about mean centering for the Item Item CF? This evaluation I'll leave it to the next notebook, where we evaluate and compare the main metrics used for similarity calculation in a CF system.\n",
    "\n",
    "-- \n",
    "\n",
    "Item Item CF brings efficiencies steps forward from the User User CF schema. With it, we bring personalised recommendations and in a way that is computationally efficient to scale for giant e-commerce companies, such as Amazon or Netflix. But Item Item CF isn't a gold system, where we can implement it and always get good results. It has a few premisses:\n",
    "\n",
    "- First, it has the premisse that number of users >> number of items. This is a prerequisite to have stable entities, items in this case, and doesn't need to recalculate the similarity matrix often, as in the User User CF.\n",
    "  \n",
    "  \n",
    "- Secondly, and this is an interesting feature, Item Item CF is better when the item ratings are stable, *i.e.*, they have lots of evaluations. This means that the user's items are probably going to have a lot of influence from these popular items and, at the end, receiving popular items recommendations. This is good when you want to be safe about your recommendations, such as expensive services or products or rarely bought, such as houses or cars. However, this lack of '*serendipity*' is missed when we want to enable users to find that particular rare item and amazingly matched with your tastes. As an example, If we take Spotify, we don't want to receive recommendations such as 'Hey, as you listened to Mozart, here is what we think you'd like: Bach'. Spotify greatness works on the premisse of finding the bands and songs that can surprise you, so they wouldn't be effective by working on the Item Item CF schema. Of course, we are going to see more advance techniques in the future where these companies apply modern algorithms to have good recommendations and still be performatic, but the idea now was to show how we can't rely on one algorithm as the best of them all.\n",
    "\n",
    "<img src=\"images/notebook5_image2.png\" width=\"500\">\n",
    "  \n",
    "  \n",
    "In the [next notebook](https://github.com/caiomiyashiro/RecommenderSystemsNotebooks/blob/master/Month%202%20Part%20III%20-%20Notes%20on%20Similarity%20Metrics%20for%20CF.ipynb), we finalise the discussion over Collaborative Filtering by investigating a little more on how the similarity metrics work and try to find out some of its features such as:  \n",
    "  \n",
    "* Why pearson end up being better for User User CF and Cosine Similarity better for Item Item CF?\n",
    "* What are the strenghts and weakness when thinking on using one of the evaluated metrics?\n",
    "* Some filosophies on what they represent and how we can think about them geometrically\n",
    "  \n",
    "Stay tuned :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}