Collaborative Filtering Amazon Watch Reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href=\"#Data-Preprocessing\" data-toc-modified-id=\"Data-Preprocessing-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Data Preprocessing</a></span><ul class=\"toc-item\"><li><span><a href=\"#Loading-Data\" data-toc-modified-id=\"Loading-Data-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Loading Data</a></span></li><li><span><a href=\"#Removing-Duplicates\" data-toc-modified-id=\"Removing-Duplicates-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Removing Duplicates</a></span></li><li><span><a href=\"#Checking-Total-Reviews\" data-toc-modified-id=\"Checking-Total-Reviews-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Checking Total Reviews</a></span></li></ul></li><li><span><a href=\"#Model-Selection\" data-toc-modified-id=\"Model-Selection-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Model Selection</a></span><ul class=\"toc-item\"><li><span><a href=\"#Baseline-Estimate\" data-toc-modified-id=\"Baseline-Estimate-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Baseline Estimate</a></span></li><li><span><a href=\"#Single-Value-Decomposition\" data-toc-modified-id=\"Single-Value-Decomposition-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Single Value Decomposition</a></span></li></ul></li><li><span><a href=\"#Serving-Recommendations\" data-toc-modified-id=\"Serving-Recommendations-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Serving Recommendations</a></span><ul class=\"toc-item\"><li><span><a href=\"#Finding-Popular-Classes\" data-toc-modified-id=\"Finding-Popular-Classes-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>Finding Popular Classes</a></span></li></ul></li><li><span><a href=\"#Wrapping-Up\" data-toc-modified-id=\"Wrapping-Up-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Wrapping Up</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Recommender System Using Amazon Reviews \n",
    "\n",
    "## Introduction\n",
    "Recommendation systems are in tons of things you interact with every day. Amazon, Spotify, and Facebook are some of the biggest players, and they're using all the data they can to suggest products that they think you'll love.\n",
    "\n",
    "![Spotify recommendations](https://raw.githubusercontent.com/smwitkowski/Amazon-Recommender-System/master/images/spotify_recommendations.PNG)\n",
    "\n",
    "Some companies have teams of people collection, cleaning, and building models around this data. However, with a few useful Python packages and some great data from [Amazon's customer review dataset](https://s3.amazonaws.com/amazon-reviews-pds/readme.html), I'm going to build a recommendation system by myself.\n",
    "\n",
    "## Data Preprocessing\n",
    "\n",
    "### Loading Data\n",
    "There is **a lot** of data in Amazon's full customer review dataset, so I'm only going to be using data on the 'Watches' category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>marketplace</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>review_id</th>\n",
       "      <th>product_id</th>\n",
       "      <th>product_parent</th>\n",
       "      <th>product_title</th>\n",
       "      <th>product_category</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>helpful_votes</th>\n",
       "      <th>total_votes</th>\n",
       "      <th>vine</th>\n",
       "      <th>verified_purchase</th>\n",
       "      <th>review_headline</th>\n",
       "      <th>review_body</th>\n",
       "      <th>review_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>US</td>\n",
       "      <td>3653882</td>\n",
       "      <td>R3O9SGZBVQBV76</td>\n",
       "      <td>B00FALQ1ZC</td>\n",
       "      <td>937001370</td>\n",
       "      <td>Invicta Women's 15150 \"Angel\" 18k Yellow Gold ...</td>\n",
       "      <td>Watches</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Five Stars</td>\n",
       "      <td>Absolutely love this watch! Get compliments al...</td>\n",
       "      <td>2015-08-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>US</td>\n",
       "      <td>14661224</td>\n",
       "      <td>RKH8BNC3L5DLF</td>\n",
       "      <td>B00D3RGO20</td>\n",
       "      <td>484010722</td>\n",
       "      <td>Kenneth Cole New York Women's KC4944 Automatic...</td>\n",
       "      <td>Watches</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>I love thiswatch it keeps time wonderfully</td>\n",
       "      <td>I love this watch it keeps time wonderfully.</td>\n",
       "      <td>2015-08-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>US</td>\n",
       "      <td>27324930</td>\n",
       "      <td>R2HLE8WKZSU3NL</td>\n",
       "      <td>B00DKYC7TK</td>\n",
       "      <td>361166390</td>\n",
       "      <td>Ritche 22mm Black Stainless Steel Bracelet Wat...</td>\n",
       "      <td>Watches</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Two Stars</td>\n",
       "      <td>Scratches</td>\n",
       "      <td>2015-08-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  marketplace  customer_id       review_id  product_id  product_parent  \\\n",
       "0          US      3653882  R3O9SGZBVQBV76  B00FALQ1ZC       937001370   \n",
       "1          US     14661224   RKH8BNC3L5DLF  B00D3RGO20       484010722   \n",
       "2          US     27324930  R2HLE8WKZSU3NL  B00DKYC7TK       361166390   \n",
       "\n",
       "                                       product_title product_category  \\\n",
       "0  Invicta Women's 15150 \"Angel\" 18k Yellow Gold ...          Watches   \n",
       "1  Kenneth Cole New York Women's KC4944 Automatic...          Watches   \n",
       "2  Ritche 22mm Black Stainless Steel Bracelet Wat...          Watches   \n",
       "\n",
       "   star_rating  helpful_votes  total_votes vine verified_purchase  \\\n",
       "0            5              0            0    N                 Y   \n",
       "1            5              0            0    N                 Y   \n",
       "2            2              1            1    N                 Y   \n",
       "\n",
       "                              review_headline  \\\n",
       "0                                  Five Stars   \n",
       "1  I love thiswatch it keeps time wonderfully   \n",
       "2                                   Two Stars   \n",
       "\n",
       "                                         review_body review_date  \n",
       "0  Absolutely love this watch! Get compliments al...  2015-08-31  \n",
       "1       I love this watch it keeps time wonderfully.  2015-08-31  \n",
       "2                                          Scratches  2015-08-31  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "# Read in the data, skip any lines that return an error\n",
    "reviews = pd.read_csv(\n",
    "    'data\\Watch Reviews.tsv',\n",
    "    sep='\\t',\n",
    "    error_bad_lines=False,\n",
    "    warn_bad_lines=False)\n",
    "reviews.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Theres are 15 columns in the data. Amazon details what each columns containes, which I've included below:\n",
    "\n",
    "* marketplace       - 2 letter country code of the marketplace where the review was written.\n",
    "* customer_id       - Random identifier that can be used to aggregate reviews written by a single author.\n",
    "* review_id         - The unique ID of the review.\n",
    "* product_id        - The unique Product ID the review pertains to. In the multilingual dataset the reviews for the same product in different countries can be grouped by the same product_id.\n",
    "* product_parent    - Random identifier that can be used to aggregate reviews for the same product.\n",
    "* product_title     - Title of the product.\n",
    "* product_category  - Broad product category that can be used to group reviews (also used to group the dataset into coherent parts).\n",
    "* star_rating       - The 1-5 star rating of the review.\n",
    "* helpful_votes     - Number of helpful votes.\n",
    "* total_votes       - Number of total votes the review received.\n",
    "* vine              - Review was written as part of the Vine program.\n",
    "* verified_purchase - The review is on a verified purchase.\n",
    "* review_headline   - The title of the review.\n",
    "* review_body       - The review text.\n",
    "* review_date       - The date the review was written."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 960204 entries, 0 to 960203\n",
      "Data columns (total 15 columns):\n",
      "marketplace          960204 non-null object\n",
      "customer_id          960204 non-null int64\n",
      "review_id            960204 non-null object\n",
      "product_id           960204 non-null object\n",
      "product_parent       960204 non-null int64\n",
      "product_title        960202 non-null object\n",
      "product_category     960204 non-null object\n",
      "star_rating          960204 non-null int64\n",
      "helpful_votes        960204 non-null int64\n",
      "total_votes          960204 non-null int64\n",
      "vine                 960204 non-null object\n",
      "verified_purchase    960204 non-null object\n",
      "review_headline      960197 non-null object\n",
      "review_body          960056 non-null object\n",
      "review_date          960200 non-null object\n",
      "dtypes: int64(5), object(10)\n",
      "memory usage: 109.9+ MB\n"
     ]
    }
   ],
   "source": [
    "reviews.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Duplicates\n",
    "All-in-all, this data is pretty clean! There are some records with missing data, but it won't cause any issues in our analysis.\n",
    "\n",
    "I'm not certain how this data was collected or if it processed before it was published; there could be some duplicated reviews in here. To check, I'll see if there are any review_id duplicated in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(reviews.review_id.duplicated())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There aren't any duplicated `review_id` values, but that doesn't mean that there aren't any duplicative reviews. Hypothetically, someone could submit the same review twice, and we wouldn't know only by looking at `review_id`.\n",
    "\n",
    "Instead of looking at `review_id`, I'm going to see if a customer posted two reviews on the same product. My assumption is that is `customer_id` and `product_id` show up more than once, then it's a duplicated review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>marketplace</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>review_id</th>\n",
       "      <th>product_id</th>\n",
       "      <th>product_parent</th>\n",
       "      <th>product_title</th>\n",
       "      <th>product_category</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>helpful_votes</th>\n",
       "      <th>total_votes</th>\n",
       "      <th>vine</th>\n",
       "      <th>verified_purchase</th>\n",
       "      <th>review_headline</th>\n",
       "      <th>review_body</th>\n",
       "      <th>review_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>180410</th>\n",
       "      <td>US</td>\n",
       "      <td>685318</td>\n",
       "      <td>R1J1TVEYBP3A7M</td>\n",
       "      <td>B003QG1SO2</td>\n",
       "      <td>47470030</td>\n",
       "      <td>Timex Men's Easy Reader Date Leather Strap Watch</td>\n",
       "      <td>Watches</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Three Stars</td>\n",
       "      <td>it's ok, but there's one part below the the nu...</td>\n",
       "      <td>2015-03-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227516</th>\n",
       "      <td>US</td>\n",
       "      <td>685318</td>\n",
       "      <td>R1DJUOOB0RFMRP</td>\n",
       "      <td>B003QG1SO2</td>\n",
       "      <td>47470030</td>\n",
       "      <td>Timex Men's Easy Reader Date Leather Strap Watch</td>\n",
       "      <td>Watches</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Four Stars</td>\n",
       "      <td>it's nice.</td>\n",
       "      <td>2015-02-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45889</th>\n",
       "      <td>US</td>\n",
       "      <td>817344</td>\n",
       "      <td>R3R845UJQPL7DP</td>\n",
       "      <td>B004VW55NA</td>\n",
       "      <td>921527551</td>\n",
       "      <td>Women Geneva Rhinestone Leather Band Quartz Wr...</td>\n",
       "      <td>Watches</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Three Stars</td>\n",
       "      <td>nice &amp; shiny</td>\n",
       "      <td>2015-07-20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46029</th>\n",
       "      <td>US</td>\n",
       "      <td>817344</td>\n",
       "      <td>R39DJJ1LM1K14N</td>\n",
       "      <td>B004VW55NA</td>\n",
       "      <td>921527551</td>\n",
       "      <td>Women Geneva Rhinestone Leather Band Quartz Wr...</td>\n",
       "      <td>Watches</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Three Stars</td>\n",
       "      <td>Nice</td>\n",
       "      <td>2015-07-20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       marketplace  customer_id       review_id  product_id  product_parent  \\\n",
       "180410          US       685318  R1J1TVEYBP3A7M  B003QG1SO2        47470030   \n",
       "227516          US       685318  R1DJUOOB0RFMRP  B003QG1SO2        47470030   \n",
       "45889           US       817344  R3R845UJQPL7DP  B004VW55NA       921527551   \n",
       "46029           US       817344  R39DJJ1LM1K14N  B004VW55NA       921527551   \n",
       "\n",
       "                                            product_title product_category  \\\n",
       "180410   Timex Men's Easy Reader Date Leather Strap Watch          Watches   \n",
       "227516   Timex Men's Easy Reader Date Leather Strap Watch          Watches   \n",
       "45889   Women Geneva Rhinestone Leather Band Quartz Wr...          Watches   \n",
       "46029   Women Geneva Rhinestone Leather Band Quartz Wr...          Watches   \n",
       "\n",
       "        star_rating  helpful_votes  total_votes vine verified_purchase  \\\n",
       "180410            3              0            1    N                 Y   \n",
       "227516            4              0            0    N                 Y   \n",
       "45889             3              0            0    N                 Y   \n",
       "46029             3              0            0    N                 Y   \n",
       "\n",
       "       review_headline                                        review_body  \\\n",
       "180410     Three Stars  it's ok, but there's one part below the the nu...   \n",
       "227516      Four Stars                                         it's nice.   \n",
       "45889      Three Stars                                       nice & shiny   \n",
       "46029      Three Stars                                               Nice   \n",
       "\n",
       "       review_date  \n",
       "180410  2015-03-10  \n",
       "227516  2015-02-01  \n",
       "45889   2015-07-20  \n",
       "46029   2015-07-20  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "purchase_ids = ['customer_id', 'product_id']\n",
    "\n",
    "# Get a dataframe consisting only of reviews that are duplicated\n",
    "duplicates = reviews[reviews.duplicated(subset=purchase_ids,\n",
    "                                        keep=False)].sort_values(purchase_ids)\n",
    "duplicates.head(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it looks like there are some reviews that are exactly the same, but some people have *updated* their review by submitting a new one. With that in mind, I'm only going to keep the most recent review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews = (reviews\n",
    "           # Sort the values so we'll keep the most recent review.\n",
    "           .sort_values(['customer_id', 'product_id', 'review_date'], ascending=[False, False, True])\n",
    "           .drop_duplicates(subset=purchase_ids, keep='last'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I want to look at the products that have been reviewed in this data. I expect that some products have been reviewed many times, while others might have only been reviewed once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SE JT6216 21-Piece Watch Repair Kit</th>\n",
       "      <td>4390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Timex Unisex Weekender Analog Quartz Watch</th>\n",
       "      <td>3229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bling Jewelry Plated Classic Round CZ Ladies Watch</th>\n",
       "      <td>3050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Casio Men's Sport Analog Dive Watch</th>\n",
       "      <td>2047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Casio Women's LRW-200H-2BVCF Stainless Steel Watch Resin Band</th>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    product_title\n",
       "SE JT6216 21-Piece Watch Repair Kit                          4390\n",
       "Timex Unisex Weekender Analog Quartz Watch                   3229\n",
       "Bling Jewelry Plated Classic Round CZ Ladies Watch           3050\n",
       "Casio Men's Sport Analog Dive Watch                          2047\n",
       "Casio Women's LRW-200H-2BVCF Stainless Steel Wa...           2008"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Burberry Men's BU7721 Sport Diving Blue Diving Dial Watch</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TAG Heuer Women's WJ1316.BA0573 Link Watch</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Akribos XXIV Bravura Mens Watch AK480SS</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ebel Women's Quartz Watch 9157F13-9925</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K&amp;BROS Women's 9149-1 Steel Flower Stainless Steel Black Dial Watch</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    product_title\n",
       "Burberry Men's BU7721 Sport Diving Blue Diving ...              1\n",
       "TAG Heuer Women's WJ1316.BA0573 Link Watch                      1\n",
       "Akribos XXIV Bravura Mens Watch AK480SS                         1\n",
       "Ebel Women's Quartz Watch 9157F13-9925                          1\n",
       "K&BROS Women's 9149-1 Steel Flower Stainless St...              1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.product_title.value_counts().to_frame().head(5)\n",
    "reviews.product_title.value_counts().to_frame().tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, the top reviewed item is a repair kit! While this isn't a watch, I'm not going to remove it because it's in the \"Watch\" category on Amazon.\n",
    "\n",
    "Now, something worth noting is that each product comes with a `parent_product` value. I expect that this is main product in an Amazon listing, and child products are those of different sizes or different colors. I'm going to do some digging and see if that's the case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "297568235    48\n",
       "315669547    48\n",
       "324241921    37\n",
       "802598355    35\n",
       "995353619    31\n",
       "Name: product_parent, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[['product_parent',\n",
    "         'product_id']].drop_duplicates().product_parent.value_counts().head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Theres a lot of different products that share `product_parent`. I'm going to look at the titles of one of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_parent</th>\n",
       "      <th>product_id</th>\n",
       "      <th>product_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>47769</th>\n",
       "      <td>297568235</td>\n",
       "      <td>B005DM0PHS</td>\n",
       "      <td>Clockwork Synergy Classic Nylon Nato watch str...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>644527</th>\n",
       "      <td>297568235</td>\n",
       "      <td>B005BDZFBA</td>\n",
       "      <td>Clockwork Synergy Classic Nylon Nato watch str...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>690385</th>\n",
       "      <td>297568235</td>\n",
       "      <td>B005BKNGCI</td>\n",
       "      <td>Clockwork Synergy Classic Nylon Nato watch str...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>849420</th>\n",
       "      <td>297568235</td>\n",
       "      <td>B005CHHTE6</td>\n",
       "      <td>Clockwork Synergy Classic Nylon Nato watch str...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>849105</th>\n",
       "      <td>297568235</td>\n",
       "      <td>B005CHHKUE</td>\n",
       "      <td>Clockwork Synergy Classic Nylon Nato watch str...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        product_parent  product_id  \\\n",
       "47769        297568235  B005DM0PHS   \n",
       "644527       297568235  B005BDZFBA   \n",
       "690385       297568235  B005BKNGCI   \n",
       "849420       297568235  B005CHHTE6   \n",
       "849105       297568235  B005CHHKUE   \n",
       "\n",
       "                                            product_title  \n",
       "47769   Clockwork Synergy Classic Nylon Nato watch str...  \n",
       "644527  Clockwork Synergy Classic Nylon Nato watch str...  \n",
       "690385  Clockwork Synergy Classic Nylon Nato watch str...  \n",
       "849420  Clockwork Synergy Classic Nylon Nato watch str...  \n",
       "849105  Clockwork Synergy Classic Nylon Nato watch str...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[reviews.product_parent == 297568235][[\n",
    "    'product_parent', 'product_id', 'product_title'\n",
    "]].drop_duplicates().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a lot of products for some Clockwork Synergy product. I wonder if they're different products or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>47769</th>\n",
       "      <td>Clockwork Synergy Classic Nylon Nato watch str...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           product_title\n",
       "47769  Clockwork Synergy Classic Nylon Nato watch str..."
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[reviews.product_parent == 297568235][['product_title'\n",
    "                                              ]].drop_duplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it seems that `product_parent` distinguishes between different products since there are different values for `product_id`. However, all of these products share the same value for `product_title`. It seems that the different `product_id` values represent differences in things colors or sizes, like in the product example below.\n",
    "\n",
    "![Synergy Nylon Product](https://raw.githubusercontent.com/smwitkowski/Amazon-Recommender-System/master/images/nylon_selections.PNG)\n",
    "\n",
    "This could cause a problem in the future; a user theoretically could be recommended three products with unique `product_id` values, but they're really all the same product. I want to recommend three different products instead.\n",
    "\n",
    "The main issue here is that there are many different values of `product_id` for each value of `product_title`. I don't want to remove any products or reviews, so instead I'm going to alter the data a bit to make sure that there is only one `product_id` per `product_title`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Ice-Watch Ice-Glow Glow Yellow - Big Men's watch #GL.GY.B.S.11</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Victorinox Swiss Army Men's 241261 Classic Chronograph Black Dial Watch</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nautica Unisex N09920G BFD 101 Classic Analog with Enamel Bezel Watch</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Swiss Legend Men's 30465-01-RDA \"Cyclone\" Stainless Steel Watch with Black Silicone Strap</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Marc Ecko Men's E95042G2 Rhino Logo Silver Stainless Steel Watch</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    product_title\n",
       "Ice-Watch Ice-Glow Glow Yellow - Big Men's watc...              1\n",
       "Victorinox Swiss Army Men's 241261 Classic Chro...              1\n",
       "Nautica Unisex N09920G BFD 101 Classic Analog w...              1\n",
       "Swiss Legend Men's 30465-01-RDA \"Cyclone\" Stain...              1\n",
       "Marc Ecko Men's E95042G2 Rhino Logo Silver Stai...              1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "products = reviews[['product_id', 'product_title']].drop_duplicates(\n",
    "    subset='product_title', keep='first')\n",
    "column_order = reviews.columns\n",
    "reviews = reviews.drop(\n",
    "    'product_id', axis=1).merge(\n",
    "        products, on='product_title')[column_order]\n",
    "\n",
    "reviews[['product_id', 'product_title'\n",
    "         ]].drop_duplicates().product_title.value_counts().head(5).to_frame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we only have one `product_id` per `product_title`. I'm going to check and see if there are still multiple `product_id` values for one `product_parent` value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "544812143    23\n",
       "193303569     9\n",
       "86574288      9\n",
       "286396053     7\n",
       "459113922     7\n",
       "Name: product_parent, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[['product_parent',\n",
    "         'product_id']].drop_duplicates().product_parent.value_counts().head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seem so, but since we changed the `product_id` value for items with matching `product_title` values, I expect these are different products."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>353378</th>\n",
       "      <td>Game Time Men's NFL Retro Series Watch - India...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>569449</th>\n",
       "      <td>Game Time Men's NFL Retro Series Watch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>581501</th>\n",
       "      <td>Game Time Men's NFL Retro Series Watch - Phila...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>697318</th>\n",
       "      <td>Game Time Men's NFL Retro Series Watch - New E...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>712317</th>\n",
       "      <td>Game Time Men's NFL Retro Series Watch - Houst...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            product_title\n",
       "353378  Game Time Men's NFL Retro Series Watch - India...\n",
       "569449             Game Time Men's NFL Retro Series Watch\n",
       "581501  Game Time Men's NFL Retro Series Watch - Phila...\n",
       "697318  Game Time Men's NFL Retro Series Watch - New E...\n",
       "712317  Game Time Men's NFL Retro Series Watch - Houst..."
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>312536</th>\n",
       "      <td>Seiko Men's SKZ211K1 Five Sports Stainless Ste...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>861913</th>\n",
       "      <td>Seiko 5 Sport Black Dial Mens Watch SKZ211K1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>948609</th>\n",
       "      <td>Seiko 5 Wrist Watches-Seiko 5 Sports Automatic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>953379</th>\n",
       "      <td>SKZ211 Seiko 5 Sports Automatic Atlas Diver Ye...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>955967</th>\n",
       "      <td>Seiko 5 Men's Sports Automatic 200M Watch</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            product_title\n",
       "312536  Seiko Men's SKZ211K1 Five Sports Stainless Ste...\n",
       "861913       Seiko 5 Sport Black Dial Mens Watch SKZ211K1\n",
       "948609  Seiko 5 Wrist Watches-Seiko 5 Sports Automatic...\n",
       "953379  SKZ211 Seiko 5 Sports Automatic Atlas Diver Ye...\n",
       "955967          Seiko 5 Men's Sports Automatic 200M Watch"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[reviews.product_parent == 544812143][['product_title'\n",
    "                                              ]].drop_duplicates().head()\n",
    "\n",
    "reviews[reviews.product_parent == 286396053][['product_title'\n",
    "                                              ]].drop_duplicates().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it looks like there are some products that are still very similar and share a `parent_product` value. The first set of parent id's showed different watches that highlighted different NFL teams.\n",
    "\n",
    "<table><tr><td><img src='https://raw.githubusercontent.com/smwitkowski/Amazon-Recommender-System/master/images/Arizona%20Retro.PNG'></td><td><img src='https://raw.githubusercontent.com/smwitkowski/Amazon-Recommender-System/master/images/San%20Diego%20Retro.PNG'></td></tr></table>\n",
    "\n",
    "These *could* be considered as the same product, but if you talk to a fan, an Arizona Caridinals watch **is not** the same as a New England Patriots watch.\n",
    "\n",
    "The second set of `parent_product` values shows very different products.\n",
    "\n",
    "<table><tr><td><img src='https://raw.githubusercontent.com/smwitkowski/Amazon-Recommender-System/master/images/Seiko%201.PNG'></td><td><img src='https://raw.githubusercontent.com/smwitkowski/Amazon-Recommender-System/master/images/Seiko%202.PNG'></td></tr></table>\n",
    "\n",
    "I don't think it would ever make sense to group these watches together, so I'm going to keep the data as is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking Total Reviews\n",
    "Many recommender systems run into a problem called the Cold-Start problem. Essentially, a user can't be recommended anything because they haven't rated anything! Additionally, if you introduce a new product, nobody has rated it and it can't be recommended.  Since we have rating data, we're not going to run into either of these problems, although we could face a similar one. \n",
    "\n",
    "If a user rated *one* item, how well do you think a recommender system could work? If you have one point of a line, you have no idea which direction the line is going. In the same vein, if you have a user with only one review, you *might* be able to rule out some items, but it would be very difficult to be confident in your recommendations to that user.\n",
    "\n",
    "For this reason, I'm going to explore how many reviews have been submitted by user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>frequency</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>reviews</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>593745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>81301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>22899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>482</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         frequency\n",
       "reviews           \n",
       "1           593745\n",
       "2            81301\n",
       "3            22899\n",
       "4             9093\n",
       "5             4481\n",
       "6             2479\n",
       "7             1520\n",
       "8              986\n",
       "9              712\n",
       "10             482"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(reviews.customer_id.value_counts().rename_axis('id').reset_index(\n",
    "    name='frequency').frequency.value_counts(\n",
    "        normalize=False).rename_axis('reviews').to_frame().head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that over 80% of our review dataset contains users that have only reviewed a single product. While we would like a lot of review data for every individual, data is data and we have to work with what we have. The good news is that over 80,000 people have reviewed at least 2 products, and we should be able to build a decent foundational model with the data we have in front of us.\n",
    "\n",
    "## Model Selection\n",
    "\n",
    "There are a lot of different packages that can help you build a recommender system. For this one, I'm going to be using the   [Surprise package](http://surpriselib.com/). `Surprise` has a few different algorithms built in. The author has even included some datasets, so you could get started building something similar without collecting or cleaning the data yourself. \n",
    "\n",
    "In this case, I need to load in a custom dataset to use with `Surprise`. It's not as easy as pointing to the dataframe, but it's pretty close. According to the [documentation](https://surprise.readthedocs.io/en/stable/getting_started.html#use-a-custom-dataset), we need to make sure our dataframe has three columns: the user ids, the item ids, and the ratings. Additionally, we'll need to specify the rating scale. In our case, users can rate a product discretely from 1 to 5. \n",
    "\n",
    "I'm also going to split the data into training and testing data using the `Surprise` package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from surprise import accuracy\n",
    "from surprise import Dataset\n",
    "from surprise import Reader\n",
    "from surprise.model_selection import train_test_split\n",
    "\n",
    "# Set the rating scale\n",
    "reader = Reader(rating_scale=(1, 5))\n",
    "\n",
    "# Load data from the review data frame and create train and test sets\n",
    "data = Dataset.load_from_df(\n",
    "    reviews[['customer_id', 'product_id', 'star_rating']], reader)\n",
    "trainset, testset = train_test_split(data, test_size=.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline Estimate\n",
    "\n",
    "First, I'm going to build a recommendation system using baseline estimates. If you'd like to learn more about baseline estimates, you can read the [Surprise documentation](https://surprise.readthedocs.io/en/stable/basic_algorithms.html#surprise.prediction_algorithms.baseline_only.BaselineOnly) or [Yehuda Koren. Factor in the neighbors: scalable and accurate collaborative filtering. 2010](http://courses.ischool.berkeley.edu/i290-dm/s11/SECURE/a1-koren.pdf) which explains the concept in much more detail.\n",
    "\n",
    "Briefly, baseline estimates look at the average rating an item earned in the entire dataset, and the average rating a user usually gives. Sometimes, some items are rated higher on average than others. In a similar way, some users rate items more critically than others. These deviations are used in baseline estimates to predict a score.\n",
    "\n",
    "Baseline estimates can be calculated using Stochastic Gradient Descent (SGD) or Alternating Least Squares (ALS). In this example, I'll be using ALS.\n",
    "\n",
    "I've already used `GridSearchCV` to find some optimal parameters, although it could be improved with more computational power and/or time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimating biases using als...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.2383393223557766"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from surprise.model_selection import GridSearchCV\n",
    "from surprise.prediction_algorithms.baseline_only import BaselineOnly\n",
    "\n",
    "# param_grid = {'bsl_options': {'method': ['als'],\n",
    "#                              'n_epochs': random.sample(range(10, 20), 5),\n",
    "#                              'reg_u': random.sample(range(10, 30), 5),\n",
    "#                              'reg_i': random.sample(range(10, 30), 5)}}\n",
    "\n",
    "# gs = GridSearchCV(BaselineOnly, param_grid, measures=['RMSE', 'MAE'], cv=5, n_jobs = -1)\n",
    "# gs.fit(data)\n",
    "# print(gs.best_score['rmse'], gs.best_params['rmse'])\n",
    "\n",
    "bsl_options = {'method': 'als', 'n_epochs': 18, 'reg_u': 11, 'reg_i': 11}\n",
    "\n",
    "algo = BaselineOnly(bsl_options=bsl_options)\n",
    "fit = algo.fit(trainset)\n",
    "predictions = fit.test(testset)\n",
    "accuracy.rmse(predictions, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single Value Decomposition\n",
    "\n",
    "Using baseline estimates are a great start, but are pretty unsophicated compared to the other models provided in the `Suprise` package.\n",
    "\n",
    "One of the algorithms available through the `Surprise` package is a single-value decompostion (SVD) algorithm, famously used in the [Netflix competition](https://netflixprize.com/). [Popularized by Simon Funk](http://sifter.org/~simon/journal/20061211.html), this algorithm essentially boils down all a user's rating to one value, and uses that value along with a baseline estimate to predict a rating. If you want to learn more, [here's a good article](https://medium.com/@m_n_malaeb/singular-value-decomposition-svd-in-recommender-systems-for-non-math-statistics-programming-4a622de653e9)\n",
    "\n",
    "As with the baseline estimates, I ran some cross validation to find a good combination of parameters (this took at least a full day to finish running on my laptop - I would recommend just choosing some parameters if you're just learning the `Suprise` package).\n",
    "\n",
    "Hopefully, we'll see an improved RMSE when using this aglorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from surprise import SVD\n",
    "\n",
    "# param_grid = {'n_epochs': range(5, 20),\n",
    "#               'lr_all': np.linspace(0.1,0.020,9),\n",
    "#               'reg_all': np.linspace(0.1,1,10)}\n",
    "\n",
    "# gs = GridSearchCV(SVD, param_grid, measures=['rmse'], cv=3, n_jobs = -1)\n",
    "\n",
    "# gs.fit(data)\n",
    "\n",
    "# best RMSE score\n",
    "# print(gs.best_score['rmse'])\n",
    "\n",
    "# combination of parameters that gave the best RMSE score\n",
    "# print(gs.best_params['rmse'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2313336546972184"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "algo = SVD(n_epochs=19, lr_all=0.02, reg_all=0.2)\n",
    "fit = algo.fit(trainset)\n",
    "predictions = algo.test(testset)\n",
    "accuracy.rmse(predictions, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We lowered the RMSE about by about 0.01. It's not a lot, but any improvement is good. However, if we wanted to re-train this model frequently, using the baseline estimate might be the more favorable option when considering the [cost of using cloud resources](https://aws.amazon.com/aml/pricing/).\n",
    "\n",
    "## Serving Recommendations\n",
    "\n",
    "Recommendations are no good unless you're able to serve them to customers. Depending on a business's current technology stack and business model, these recommendations can come through an e-commerce store, through an email, or even through a sales rep. Each delivery method will require different skills and tools to implement, but no matter what you'll need to get the recommendations out of your fitted algorithm and to the customer.\n",
    "\n",
    "Luckily, there author of `Surprise` has written a function we can use to get the top-N recommendations for each user in our dataset. To avoid reinventing the wheel, I'll be using [that function](https://surprise.readthedocs.io/en/stable/FAQ.html#how-to-get-the-top-n-recommendations-for-each-user)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "\n",
    "def get_top_n(predictions, n=10):\n",
    "\n",
    "    # First map the predictions to each user.\n",
    "    top_n = defaultdict(list)\n",
    "    for uid, iid, true_r, est, _ in predictions:\n",
    "        top_n[uid].append((iid, est))\n",
    "\n",
    "    # Then sort the predictions for each user and retrieve the k highest ones.\n",
    "    for uid, user_ratings in top_n.items():\n",
    "        user_ratings.sort(key=lambda x: x[1], reverse=True)\n",
    "        top_n[uid] = user_ratings[:n]\n",
    "\n",
    "    return top_n\n",
    "\n",
    "\n",
    "top_n = get_top_n(predictions, n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('B00CQ7IDZ4', 4.1219223126882296),\n",
       " ('B00HUCIOOK', 4.0721615059252416),\n",
       " ('B00INXSUKS', 3.9437460503852786),\n",
       " ('B00C7I62S2', 3.8137130798781897),\n",
       " ('B008MVVBWU', 3.6817009584164819),\n",
       " ('B008D902Q2', 3.4667934012274104)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_n[reviews.customer_id[8]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding Popular Classes\n",
    "\n",
    "Unfortunately, this function doesn't always recommend ten items. I suspect this is because the data is very sparse, and there is a high number of customers who have only rated one product, as well as many products that have only one rating. Regardless, we need to come up with a way to fill in the rest of the gaps.\n",
    "\n",
    "A simple yet effective way to generate recommendations is to recommend the most popular products. This is what I'll do to generate ten unique recommendations for each user.\n",
    "\n",
    "First, I need to build a dataset of popular products. Each person could define *popular* in a different way, but in this scenario I'm only going to consider products that have received at least 100 reviews. Then, I'll find the average rating for each product and sort them in descending order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_count = reviews.product_id.value_counts()\n",
    "review_count_ten = review_count[review_count >= 100]\n",
    "hundred_reviews = reviews[reviews.product_id.isin(review_count_ten.index)]\n",
    "items = (hundred_reviews[['product_id', 'star_rating'\n",
    "                          ]].groupby('product_id').agg('mean').sort_values(\n",
    "                              'star_rating', ascending=False).index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, I'm going to go through each user's recommendations and add the most popular products that aren't already recommended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def recommendation_list(user_list, user_predictions, item_list):\n",
    "    recommendations = {}\n",
    "    for i in range(100):\n",
    "        user = user_list[i]\n",
    "        if user in user_predictions:\n",
    "            user_recs = [\n",
    "                user_predictions[user][i][0]\n",
    "                for i in range(len(user_predictions[user]))\n",
    "            ]\n",
    "            if user_recs:\n",
    "                num_items = len(user_recs)\n",
    "            else:\n",
    "                num_items = 0\n",
    "\n",
    "            idx = 0\n",
    "            while num_items < 10:\n",
    "                product = item_list[idx]\n",
    "                if product not in user_recs:\n",
    "                    user_recs.append(product)\n",
    "                    num_items = len(user_recs)\n",
    "                idx += 1\n",
    "            recommendations.update({user: user_recs})\n",
    "    return recommendations\n",
    "\n",
    "\n",
    "recs = recommendation_list(reviews.customer_id.unique().tolist(), top_n, items)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B004TB226Q',\n",
       " 'B00843L74S',\n",
       " 'B00EXTZ34C',\n",
       " 'B001QFYKMW',\n",
       " 'B005MKGPC0',\n",
       " 'B0043ZWQWI',\n",
       " 'B004TB2DWY',\n",
       " 'B00AHAFFTO',\n",
       " 'B009DRP9RU',\n",
       " 'B0021AEDSM']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_user = reviews.customer_id.unique().tolist()[1]\n",
    "recs[example_user]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping Up\n",
    "\n",
    "Now, we have a list of recommendations that we can use to recommend new products to all of our customers! Each list of recommendations are unique to each customer, and have been generated from *real* data.\n",
    "\n",
    "There are a few other ways to build or otherwise improve this recommender system that I avoided:\n",
    "\n",
    "* **Use k-NN algorithms from the `Surprise` package.**\n",
    "\n",
    "There was just too much data here. I could use a small sample of the data and build it using that, but I elected not to use it at all.\n",
    "\n",
    "* **Use content filtering to build a hybrid recommender system.**\n",
    "\n",
    "I've done this throughout my career, but I had access to detailed product descriptions. In this data, I only have the product title. While informative, it's not very helpful if we use TFIDF to build a content filtering algorithm.\n",
    "\n",
    "* **Use more tuned parameters**\n",
    "\n",
    "I did some basic cross validation to select the best parameters. If I needed to improve the recommendations, I would spend much more time tuning these parameters and make sure that the algorithm is serving the best recommendations possible with the data available.\n",
    "\n",
    "I had a lot of fun working on the project, and learned more about the math behind recommendation system than I would have expected. Thanks for reading it all the way through, I appreciate it!\n",
    "\n",
    "I'd love to hear way you think! Feel free to connect with me via [LinkedIn](https://www.linkedin.com/in/witkowskism/) and let me know if this was helpful!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "gist": {
   "data": {
    "description": "Collaborative Filtering Amazon Reviews",
    "public": true
   },
   "id": ""
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}