kernelnf.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "kernelnf.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "k1XEVxvv6RsT",
        "F2MpRAP66Rsa",
        "bqil3LIB6Rsa",
        "N_vFr2Mf6Rsb",
        "exm_QK486Rsh",
        "51PLkEY96Rsp",
        "FP18bnrZ6Rss",
        "ESTQDq0K6Rsu",
        "DeMxK5CT6Rsz",
        "vz47ZM-Wcd-d",
        "VdwW9LTH6RtI",
        "rFfPks_T6RtJ",
        "-MdgYJWO6RtS",
        "ReLBfYsC6RtS",
        "39PIXMP96RtV",
        "iKb9hXAm6Rta"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/gaop123/SuperList/blob/master/kernelnf.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KXYcBAg18N-Q",
        "colab_type": "text"
      },
      "source": [
        "Hey Yoram!\n",
        "\n",
        "Please note that the Netflix dataset is too big to upload to github so i have a duplicate kaggle notebook - there the dataset is allready preloaded.\n",
        "link: \n",
        "\n",
        "Thanks for the greate class,\n",
        "It was very much inspiring! **bold text**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "daOedV1A-JLP",
        "colab_type": "text"
      },
      "source": [
        "# My objectives are to show different approaches for recommendation system on the huge Netfix dataset:\n",
        "\n",
        "# 0. Netflix dataset statistics\n",
        "# 1. Model-based CF - matrix factorization methods\n",
        "# 2. Model-based CF - clustering models methods\n",
        "# 3. Memory-based CF - statistic correlation coefficient methods\n",
        "> # 4. Future - I will try to compare all the different method, ATM i still did not find any way to do it effectively\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HpqsSwsG_D-f",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hGZjt42G_oqk",
        "colab_type": "text"
      },
      "source": [
        "# First i will analyze the dataset\n",
        "Data loading\n",
        "\n",
        "Each data file (there are 4 of them) contains below columns:\n",
        "\n",
        "Movie ID (as first line of each new movie record / file)\n",
        "\n",
        "Customer ID\n",
        "\n",
        "Rating (1 to 5)\n",
        "\n",
        "Date they gave the ratings\n",
        "\n",
        "There is another file contains the mapping of Movie ID to the movie background like name, year of release, etc\n",
        "\n",
        "Let's import the library we needed before we get started:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ySHpherb6fOj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "f08d0758-bc5c-4b2d-e19d-07d899ffeecb"
      },
      "source": [
        "pip install surprise"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: surprise in /usr/local/lib/python3.6/dist-packages (0.1)\n",
            "Requirement already satisfied: scikit-surprise in /usr/local/lib/python3.6/dist-packages (from surprise) (1.1.0)\n",
            "Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from scikit-surprise->surprise) (1.4.1)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-surprise->surprise) (0.14.1)\n",
            "Requirement already satisfied: numpy>=1.11.2 in /usr/local/lib/python3.6/dist-packages (from scikit-surprise->surprise) (1.18.2)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from scikit-surprise->surprise) (1.12.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YRcOHwt4ccCs",
        "trusted": true,
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import math\n",
        "import re\n",
        "from scipy.sparse import csr_matrix\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from surprise import Dataset, SVD,  SVDpp, SlopeOne, NMF, NormalPredictor, KNNBaseline, KNNBasic, KNNWithMeans, KNNWithZScore, BaselineOnly, CoClustering, accuracy \n",
        "from surprise.reader import Reader\n",
        "from surprise.model_selection.validation import cross_validate as cross_validate\n",
        "sns.set_style(\"darkgrid\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jK_DosW6Rr_",
        "colab_type": "text"
      },
      "source": [
        "Next let's load first data file and get a feeling of how huge the dataset is:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "Uen427UJ6Rr_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "outputId": "b87f59a2-2b23-4d99-8e76-d3cec03eab48"
      },
      "source": [
        "# Skip date\n",
        "df1 = pd.read_csv('../input/netflix-prize-data/combined_data_1.txt', header = None, names = ['CustomerID','Rating'], usecols = [0,1])\n",
        "df1['Rating'] = df1['Rating'].astype(float)\n",
        "\n",
        "df = df1"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "error",
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-8-7c1cb9ae82c2>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdf1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'../input/netflix-prize-data/combined_data_1.txt'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnames\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'CustomerID'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Rating'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0musecols\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mdf1\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Rating'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf1\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Rating'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfloat\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[1;32m    674\u001b[0m         )\n\u001b[1;32m    675\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 676\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    677\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    678\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    446\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    447\u001b[0m     \u001b[0;31m# Create the parser.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 448\u001b[0;31m     \u001b[0mparser\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfp_or_buf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    449\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    450\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[1;32m    878\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    879\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 880\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    881\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    882\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[0;34m(self, engine)\u001b[0m\n\u001b[1;32m   1112\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"c\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1113\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"c\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1114\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCParserWrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1115\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1116\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"python\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, src, **kwds)\u001b[0m\n\u001b[1;32m   1889\u001b[0m         \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"usecols\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0musecols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1890\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1891\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTextReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1892\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munnamed_cols\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munnamed_cols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1893\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.__cinit__\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._setup_parser_source\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] File ../input/netflix-prize-data/combined_data_1.txt does not exist: '../input/netflix-prize-data/combined_data_1.txt'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wjhPlnVl6RsC",
        "colab_type": "text"
      },
      "source": [
        " Let's also load the 3 remaining dataset as well<br>\n",
        "****it's on a seperate commented block because it is too heavy to load all datases on every test run - though it needs to be uncommented when the best accuracies are needed****:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "TobHWWbS6RsC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# df2 = pd.read_csv('../input/netflix-prize-data/combined_data_2.txt', header = None, names = ['CustomerID', 'Rating'], usecols = [0,1])\n",
        "# df3 = pd.read_csv('../input/netflix-prize-data/combined_data_3.txt', header = None, names = ['CustomerID', 'Rating'], usecols = [0,1])\n",
        "# df4 = pd.read_csv('../input/netflix-prize-data/combined_data_4.txt', header = None, names = ['CustomerID', 'Rating'], usecols = [0,1])\n",
        "\n",
        "\n",
        "# df2['Rating'] = df2['Rating'].astype(float)\n",
        "# df3['Rating'] = df3['Rating'].astype(float)\n",
        "# df4['Rating'] = df4['Rating'].astype(float)\n",
        "\n",
        "# print('Dataset 2 shape: {}'.format(df2.shape))\n",
        "# print('Dataset 3 shape: {}'.format(df3.shape))\n",
        "# print('Dataset 4 shape: {}'.format(df4.shape))\n",
        "\n",
        "# df = df1.append(df2)\n",
        "# df = df.append(df3)\n",
        "# df = df.append(df4)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjSl4DXl6RsF",
        "colab_type": "text"
      },
      "source": [
        "![](http://)lets peek at the data set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "sXClgAhR6RsF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.index = np.arange(0,len(df))\n",
        "print('Full dataset shape: {}'.format(df.shape))\n",
        "print('-Dataset examples-')\n",
        "print(df.iloc[::5000000, :])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "njz2u5rP6RsK",
        "colab_type": "text"
      },
      "source": [
        "Now we load the movie mapping file:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "0CxN0x3J6RsK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_title = pd.read_csv('../input/netflix-prize-data/movie_titles.csv', encoding = \"ISO-8859-1\", header = None, names = ['Movie_Id', 'Year', 'Name'])\n",
        "df_title.set_index('Movie_Id', inplace = True)\n",
        "print (df_title.head(10))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wu0ByvCg6RsN",
        "colab_type": "text"
      },
      "source": [
        "Let's give a first look on how the data spread:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "NN6L1EiI6RsO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "p = df.groupby('Rating')['Rating'].agg(['count'])\n",
        "\n",
        "# get movie count\n",
        "movie_count = df.isnull().sum()[1]\n",
        "\n",
        "# get customer count\n",
        "cust_count = df['CustomerID'].nunique() - movie_count\n",
        "\n",
        "# get rating count\n",
        "rating_count = df['CustomerID'].count() - movie_count\n",
        "\n",
        "ax = p.plot(kind = 'barh', legend = False, figsize = (15,10))\n",
        "plt.title('Total pool: {:,} Movies, {:,} customers, {:,} ratings given'.format(movie_count, cust_count, rating_count), fontsize=20)\n",
        "plt.axis('off')\n",
        "\n",
        "for i in range(1,6):\n",
        "    ax.text(p.iloc[i-1][0]/4, i-1, 'Rating {}: {:.0f}%'.format(i, p.iloc[i-1][0]*100 / p.sum()[0]), color = 'white', weight = 'bold')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uiUFHso96RsQ",
        "colab_type": "text"
      },
      "source": [
        "We can see that the rating tends to be relatively positive (>3). This may be due to the fact that unhappy customers tend to just leave instead of making efforts to rate. so low rating movies mean they are generally really bad.."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cxzVNiSy6RsR",
        "colab_type": "text"
      },
      "source": [
        "Now lets add the movies column:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "EeoICZCe6RsR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_nan = pd.DataFrame(pd.isnull(df.Rating))\n",
        "df_nan = df_nan[df_nan['Rating'] == True]\n",
        "df_nan = df_nan.reset_index()\n",
        "\n",
        "movie_np = []\n",
        "movie_id = 1\n",
        "\n",
        "for i,j in zip(df_nan['index'][1:],df_nan['index'][:-1]):\n",
        "\n",
        "    temp = np.full((1,i-j-1), movie_id)\n",
        "    movie_np = np.append(movie_np, temp)\n",
        "    movie_id += 1\n",
        "\n",
        "last_record = np.full((1,len(df) - df_nan.iloc[-1, 0] - 1),movie_id)\n",
        "movie_np = np.append(movie_np, last_record)\n",
        "\n",
        "df = df[pd.notnull(df['Rating'])]\n",
        "\n",
        "df['Movie_Id'] = movie_np.astype(int)\n",
        "df['CustomerID'] = df['CustomerID'].astype(int)\n",
        "print('-Dataset examples-')\n",
        "print(df.iloc[::5000000, :])\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k1XEVxvv6RsT",
        "colab_type": "text"
      },
      "source": [
        "# Data slicing\n",
        "The data set now is super huge and i cant work with it in the current form, so i will reduce the data volumn by improving the data quality below:\n",
        "\n",
        "Remove movie with too less reviews (they are relatively not popular)\n",
        "Remove customer who give too less reviews (they are relatively less active)\n",
        "Having above benchmark will have significant improvement on efficiency, since those unpopular movies and non-active customers still occupy same volumn as those popular movies and active customers in the view of matrix (NaN still occupy space). This should help improve the statistical signifiance too."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "CBqIK7Sh6RsU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "movies_percentile = 0.7\n",
        "#Movies rate count percentile\n",
        "#I will leave only movies on the (1-movies_percentile) percentile with respect to movies rating count\n",
        "\n",
        "customers_percentile = 0.7\n",
        "#Customers rate count percentile\n",
        "#I will leave only customers on the (1-customers_percentile) percentile with respect to customers rating count\n",
        "\n",
        "\n",
        "df_movie_summary = df.groupby('Movie_Id')['Rating'].agg(['count'])\n",
        "df_movie_summary.index = df_movie_summary.index.map(int)\n",
        "movie_benchmark = round(df_movie_summary['count'].quantile(movies_percentile),0)\n",
        "drop_movie_list = df_movie_summary[df_movie_summary['count'] < movie_benchmark].index\n",
        "\n",
        "df_cust_summary = df.groupby('CustomerID')['Rating'].agg(['count'])\n",
        "df_cust_summary.index = df_cust_summary.index.map(int)\n",
        "cust_benchmark = round(df_cust_summary['count'].quantile(customers_percentile),0)\n",
        "drop_cust_list = df_cust_summary[df_cust_summary['count'] < cust_benchmark].index\n",
        "\n",
        "print('Movies minimum rating count: {}'.format(movie_benchmark))\n",
        "print('Customers minimum rating count: {}'.format(cust_benchmark))\n",
        "\n",
        "print('Original Shape: {}'.format(df.shape))\n",
        "df = df[~df['Movie_Id'].isin(drop_movie_list)]\n",
        "df = df[~df['CustomerID'].isin(drop_cust_list)]\n",
        "\n",
        "print('After Trim Shape: {}'.format(df.shape))\n",
        "\n",
        "print('unique movies left:')\n",
        "print(df['Movie_Id'].unique().size)\n",
        "print('unique customers left:')\n",
        "print(df['CustomerID'].unique().size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gx2zG0_K6RsW",
        "colab_type": "text"
      },
      "source": [
        "Now i will pivot the dataset and convert it into a matrix M, \n",
        "where Mi,j is the rating the ith customer gave to the jth movie\n",
        "\n",
        "I will also replace all NaN values with zeros - and should keep in mind that there is no zero rating - the rating ranges from 1 to 5,\n",
        "so the value '0' will state that this movie was not being reviewed and not that it's given rating is zero."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "wOPjfrE16RsW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_p = pd.pivot_table(df,values='Rating',index='CustomerID',columns='Movie_Id')\n",
        "df_p = df_p.fillna(0)\n",
        "print(df_p.head(10))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4tu2XsQM6Rsa",
        "colab_type": "text"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F2MpRAP66Rsa",
        "colab_type": "text"
      },
      "source": [
        "# A  recommendation system is a subclass of information filtering system that seeks to predict the \"rating\" a user would give to an item"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqil3LIB6Rsa",
        "colab_type": "text"
      },
      "source": [
        "# Task1 - Model-based CF - matrix factorization methods"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QBLjRQx56Rsb",
        "colab_type": "text"
      },
      "source": [
        "1. [Surprise](http://surpriselib.com/) is a Python scikit building and analyzing recommender systems that deal with explicit rating data.<br>\n",
        "I will use this library for the perpose of trying to recommend Netflix movies to Netflix users"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_vFr2Mf6Rsb",
        "colab_type": "text"
      },
      "source": [
        "# I will try the following models:\n",
        "\n",
        "> SVD, SVDpp, NMF, NormalPredictor CoClustering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fZK3r5Fz6Rsc",
        "colab_type": "text"
      },
      "source": [
        "Matrix Factorization-based algorithms\n",
        "> SVD<br>\n",
        "> SVD algorithm is equivalent to Probabilistic Matrix Factorization<br>\n",
        "> SVDpp<br>\n",
        "> The SVDpp algorithm is an extension of SVD that takes into account implicit ratings.<br>\n",
        "> NMF<br>\n",
        "> NMF is a collaborative filtering algorithm based on Non-negative Matrix Factorization. It is very similar with SVD.<br>\n",
        "> Slope One<br>\n",
        "> SlopeOne is a straightforward implementation of the SlopeOne algorithm.<br>\n",
        "> Co-clustering<br>\n",
        "> Coclustering is a collaborative filtering algorithm based on co-clustering.<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q6aLY08o6Rsc",
        "colab_type": "text"
      },
      "source": [
        "Step1 - evaluate:\n",
        "\n",
        "Perform 3-folds cross validation in order to determine the best predictor.<br>\n",
        "For the accuracy metric i use “rmse” - root squared error."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nE8PJK8q6Rsc",
        "colab_type": "text"
      },
      "source": [
        "**All the algorithms below excpects a dataset with the following scheme 'CustomerID', 'Movie_Id', 'Rating': <br>\n",
        "The return is a function F: CustomerID -> Rating **"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "6Cxjm4VH6Rsd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cross_validate_cf_algorithms(rows):\n",
        "\n",
        "    reader = Reader()\n",
        "\n",
        "    data = Dataset.load_from_df(df[['CustomerID', 'Movie_Id', 'Rating']][:rows], reader)\n",
        "\n",
        "    benchmark = []\n",
        "    # Iterate over all algorithms\n",
        "    for algorithm in [SVD(), SVDpp(), NMF(), NormalPredictor(),  CoClustering()]:\n",
        "        # Perform cross validation\n",
        "        results = cross_validate(algorithm, data, measures=['RMSE'], cv=3, verbose=False)\n",
        "\n",
        "        # Get results & append algorithm name\n",
        "        tmp = pd.DataFrame.from_dict(results).mean(axis=0)\n",
        "        tmp = tmp.append(pd.Series([str(algorithm).split(' ')[0].split('.')[-1]], index=['Algorithm']))\n",
        "        benchmark.append(tmp)\n",
        "\n",
        "    print(pd.DataFrame(benchmark).set_index('Algorithm').sort_values('test_rmse'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "bBg_F8nY6Rsf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# get just top 100K rows for faster run time\n",
        "cross_validate_cf_algorithms(100000)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vFrphJxc6Rsg",
        "colab_type": "text"
      },
      "source": [
        "SVD performed best"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "exm_QK486Rsh",
        "colab_type": "text"
      },
      "source": [
        "# Step2 - train the best model - SVD:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8EK6fGcY6Rsh",
        "colab_type": "text"
      },
      "source": [
        "Return the top-N recommendation for each user from a set of predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "I4IX2DNU6Rsi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "from collections import defaultdict\n",
        "def get_top_n(predictions, n=10):\n",
        "\n",
        "    # First map the predictions to each user.\n",
        "    top_n = defaultdict(list)\n",
        "    for uid, iid, true_r, est, _ in predictions:\n",
        "        top_n[uid].append((iid, est))\n",
        "\n",
        "    # Then sort the predictions for each user and retrieve the k highest ones.\n",
        "    for uid, user_ratings in top_n.items():\n",
        "        user_ratings.sort(key=lambda x: x[1], reverse=True)\n",
        "        top_n[uid] = user_ratings[:n]\n",
        "\n",
        "    return top_n\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l4OdmSrQ6Rsj",
        "colab_type": "text"
      },
      "source": [
        "A simple function to retrieve the movie name from the movie ID"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "Qy2hDTw96Rsk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_m = df.set_index('Movie_Id')\n",
        "names_movie_mapping = df_title.join(df_m)\n",
        "print(names_movie_mapping)\n",
        "\n",
        "def get_movie_name(id):\n",
        "    \n",
        "    return names_movie_mapping.loc[names_movie_mapping.index == id, 'Name'].unique()[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "5wv7AZKa6Rsn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "get_movie_name(id=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "51PLkEY96Rsp",
        "colab_type": "text"
      },
      "source": [
        "# Train SVD model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "Egp8cfCp6Rsq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "reader = Reader()\n",
        "data = Dataset.load_from_df(df[['CustomerID', 'Movie_Id', 'Rating']][:10000], reader)\n",
        "trainset = data.build_full_trainset()\n",
        "\n",
        "algo = SVD()\n",
        "predictions = algo.fit(trainset)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FP18bnrZ6Rss",
        "colab_type": "text"
      },
      "source": [
        "# Print the recommanded movie for each customer:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "id": "iDJaAVPM6Rss",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "\n",
        "testset = trainset.build_anti_testset()\n",
        "predictions = algo.test(testset)\n",
        "\n",
        "top_n = get_top_n(predictions, n=10)\n",
        "\n",
        "# Print the recommended movies for each user\n",
        "recommanded_movies = {}\n",
        "for uid, user_ratings in top_n.items():\n",
        "\n",
        "    recommanded_movies[uid] = [get_movie_name(movie_id) for (movie_id, _) in user_ratings]\n",
        "    print(uid, recommanded_movies[uid])\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ESTQDq0K6Rsu",
        "colab_type": "text"
      },
      "source": [
        "# Now we have a recommendation function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMIr3qor6Rsu",
        "colab_type": "text"
      },
      "source": [
        "The function's returns the recommanded movie for the customer:<br>\n",
        "The prediction rule is to take each movie that the customer loved (rating = 5) , than for each movie predict using SVD and finaly take most frequent movie:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "mcStlpz-6Rsv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def recommand_SVD(CustomerID):\n",
        "\n",
        "    res = []\n",
        "\n",
        "    for x in df[(df['CustomerID'] == CustomerID) & (df['Rating'] == 5)]['Movie_Id']:\n",
        "        \n",
        "        p = algo.predict(CustomerID, x)[1]      \n",
        "        res.append(p) \n",
        "\n",
        "        \n",
        "    return get_movie_name(np.bincount(res).argmax())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zW-SQtU96Rsx",
        "colab_type": "text"
      },
      "source": [
        "Let's predict which movies a specific user would love to watch:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "-W4W7TUL6Rsx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"The recommanded movie for customer 1333 using clustering CF is: \", recommand_SVD(1333))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_AqhZjG6Rsz",
        "colab_type": "text"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DeMxK5CT6Rsz",
        "colab_type": "text"
      },
      "source": [
        "# Task2 - Model-based CF- clustering models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vz47ZM-Wcd-d",
        "colab_type": "text"
      },
      "source": [
        "# Clusterize Netflix movies\n",
        "\n",
        "I will try to clusterize the movies based on the rating recieved from the users.\n",
        "\n",
        "\n",
        "The steps:\n",
        "1. prepare the dataset to sklearn\n",
        "\n",
        "2.  Compare few clustering algorithms\n",
        "\n",
        "    *   DBSCAN\n",
        "    *   K-means\n",
        "    *   XXX\n",
        "\n",
        "3.   Create a function F that will map the movies to their corresponding clusters (F: movieID -> movieClusterId)\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "48-GpC0_6Rs1",
        "colab_type": "text"
      },
      "source": [
        "import the required SKlearn libraries:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "UIsShsij6Rs1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn_pandas import DataFrameMapper, cross_val_score\n",
        "import numpy as np\n",
        "import sklearn.preprocessing, sklearn.decomposition, sklearn.linear_model, sklearn.pipeline, sklearn.metrics\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.cluster import DBSCAN, KMeans\n",
        "from sklearn import metrics\n",
        "from sklearn.decomposition import PCA,SparsePCA, TruncatedSVD, NMF\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "import matplotlib.cm as cm\n",
        "from sklearn.metrics import silhouette_samples, silhouette_score\n",
        "from sklearn.impute import SimpleImputer\n",
        "import random"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GfRZhjqt6Rs3",
        "colab_type": "text"
      },
      "source": [
        "Preparing the dataset - i would like each movie to be represented by it's column, that is, all rating recieved by all users,\n",
        "The problem is the sparsness, that is, the zero ratings,\n",
        "So, my solution for this will be to replace all zeroes rating with the mean of all non zeroes ratings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "mmP6mGMd6Rs4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X = df_p.as_matrix(columns=df_p.columns[:]).transpose()\n",
        "\n",
        "imp_mean = SimpleImputer(missing_values=0, strategy='mean')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lwZDOY9k6Rs5",
        "colab_type": "text"
      },
      "source": [
        "Lets reduce the dimensions first- for that i will use PCA - lets find the best size for the new dimension - n_component parameter in sklearn"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "KmMdbtLY6Rs6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pca = PCA().fit(X)\n",
        "\n",
        "#Plotting the Cumulative Summation of the Explained Variance\n",
        "plt.figure()\n",
        "plt.plot(np.cumsum(pca.explained_variance_ratio_))\n",
        "plt.xlabel('Number of Components')\n",
        "plt.ylabel('Variance (%)') #for each component\n",
        "plt.title('Movies Dataset Explained Variance')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "emW996WW6Rs7",
        "colab_type": "text"
      },
      "source": [
        "We can see that we can reduce the size to 600 (reducing it by more than 1 / 2) with a minimal loss of 0.1 of the variance\n",
        "So now i will use PCA with n_component = 600 to reduce X to 600 dimensions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "H7a0KDu06Rs8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pca = PCA(n_components=200)\n",
        "X_reduced = pca.fit_transform(X)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q5Y0j2KU6Rs-",
        "colab_type": "text"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fILHtoCw6Rs-",
        "colab_type": "text"
      },
      "source": [
        "Now i will try K-means on the reduced data with K in [5, 10, 15, 20, 25, 30, 35, 40, 45, 50]<br>\n",
        "Than, for each result of k ill plot the **silhouette score** and than i will pick the best value for K"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "5b042ZFo6Rs-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "range_n_clusters = [5, 10, 15, 20, 25, 30, 35, 40, 45, 50]\n",
        "data = X_reduced\n",
        "\n",
        "for n_clusters in range_n_clusters:\n",
        "    # Create a subplot with 1 row and 2 columns\n",
        "    fig, (ax1, ax2) = plt.subplots(1, 2)\n",
        "    fig.set_size_inches(18, 7)\n",
        "\n",
        "    # The 1st subplot is the silhouette plot\n",
        "    # The silhouette coefficient can range from -1, 1 but in this example all\n",
        "    # lie within [-0.1, 1]\n",
        "    ax1.set_xlim([-0.1, 1])\n",
        "    # The (n_clusters+1)*10 is for inserting blank space between silhouette\n",
        "    # plots of individual clusters, to demarcate them clearly.\n",
        "    ax1.set_ylim([0, len(data) + (n_clusters + 1) * 10])\n",
        "\n",
        "    # Initialize the clusterer with n_clusters value and a random generator\n",
        "    # seed of 10 for reproducibility.\n",
        "    clusterer = KMeans(n_clusters=n_clusters, random_state=10)\n",
        "    cluster_labels = clusterer.fit_predict(data)\n",
        "\n",
        "    # The silhouette_score gives the average value for all the samples.\n",
        "    # This gives a perspective into the density and separation of the formed\n",
        "    # clusters\n",
        "    silhouette_avg = silhouette_score(data, cluster_labels)\n",
        "    print(\"For n_clusters =\", n_clusters,\n",
        "          \"The average silhouette_score is :\", silhouette_avg)\n",
        "\n",
        "    # Compute the silhouette scores for each sample\n",
        "    sample_silhouette_values = silhouette_samples(data, cluster_labels)\n",
        "\n",
        "    y_lower = 10\n",
        "    for i in range(n_clusters):\n",
        "        # Aggregate the silhouette scores for samples belonging to\n",
        "        # cluster i, and sort them\n",
        "        ith_cluster_silhouette_values = \\\n",
        "            sample_silhouette_values[cluster_labels == i]\n",
        "\n",
        "        ith_cluster_silhouette_values.sort()\n",
        "\n",
        "        size_cluster_i = ith_cluster_silhouette_values.shape[0]\n",
        "        y_upper = y_lower + size_cluster_i\n",
        "\n",
        "        color = cm.nipy_spectral(float(i) / n_clusters)\n",
        "        ax1.fill_betweenx(np.arange(y_lower, y_upper),\n",
        "                          0, ith_cluster_silhouette_values,\n",
        "                          facecolor=color, edgecolor=color, alpha=0.7)\n",
        "\n",
        "        # Label the silhouette plots with their cluster numbers at the middle\n",
        "        ax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))\n",
        "\n",
        "        # Compute the new y_lower for next plot\n",
        "        y_lower = y_upper + 10  # 10 for the 0 samples\n",
        "\n",
        "    ax1.set_title(\"The silhouette plot for the various clusters.\")\n",
        "    ax1.set_xlabel(\"The silhouette coefficient values\")\n",
        "    ax1.set_ylabel(\"Cluster label\")\n",
        "\n",
        "    # The vertical line for average silhouette score of all the values\n",
        "    ax1.axvline(x=silhouette_avg, color=\"red\", linestyle=\"--\")\n",
        "\n",
        "    ax1.set_yticks([])  # Clear the yaxis labels / ticks\n",
        "    ax1.set_xticks([-0.1, 0, 0.2, 0.4, 0.6, 0.8, 1])\n",
        "\n",
        "    # 2nd Plot showing the actual clusters formed\n",
        "    colors = cm.nipy_spectral(cluster_labels.astype(float) / n_clusters)\n",
        "    ax2.scatter(data[:, 0], data[:, 1], marker='.', s=30, lw=0, alpha=0.7,\n",
        "                c=colors, edgecolor='k')\n",
        "\n",
        "    # Labeling the clusters\n",
        "    centers = clusterer.cluster_centers_\n",
        "    # Draw white circles at cluster centers\n",
        "    ax2.scatter(centers[:, 0], centers[:, 1], marker='o',\n",
        "                c=\"white\", alpha=1, s=200, edgecolor='k')\n",
        "\n",
        "    for i, c in enumerate(centers):\n",
        "        ax2.scatter(c[0], c[1], marker='$%d$' % i, alpha=1,\n",
        "                    s=50, edgecolor='k')\n",
        "\n",
        "    ax2.set_title(\"The visualization of the clustered data.\")\n",
        "    ax2.set_xlabel(\"Feature space for the 1st feature\")\n",
        "    ax2.set_ylabel(\"Feature space for the 2nd feature\")\n",
        "\n",
        "    plt.suptitle((\"Silhouette analysis for KMeans clustering on sample data \"\n",
        "                  \"with n_clusters = %d\" % n_clusters),\n",
        "                 fontsize=14, fontweight='bold')\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8_v5g3z26RtB",
        "colab_type": "text"
      },
      "source": [
        "So as we can see, the best K will be 35 with a Silhouette value of 0.86,<br>\n",
        "Lets train the model again with 35 clusters:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "anJNvIkY6RtB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "    clusterer = KMeans(n_clusters=35, random_state=10)\n",
        "    cluster_labels = clusterer.fit_predict(X_reduced)\n",
        "\n",
        "    silhouette = silhouette_score(X_reduced, cluster_labels)\n",
        "    print(\"For n_clusters =\", 35,\n",
        "          \"The silhouette_score is :\", silhouette)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1EkKyYg06RtD",
        "colab_type": "text"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0rfHCJFs6RtD",
        "colab_type": "text"
      },
      "source": [
        "Next i will try DBScan algorithm"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "id": "4zdoIElh6RtE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "db = DBSCAN().fit(X_reduced)\n",
        "core_samples_mask = np.zeros_like(db.labels_, dtype=bool)\n",
        "core_samples_mask[db.core_sample_indices_] = True\n",
        "labels = db.labels_\n",
        "\n",
        "# Number of clusters in labels, ignoring noise if present.\n",
        "n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)\n",
        "n_noise_ = list(labels).count(-1)\n",
        "\n",
        "print('Estimated number of clusters: %d' % n_clusters_)\n",
        "print('Estimated number of noise points: %d' % n_noise_)\n",
        "\n",
        "print(\"Silhouette Coefficient: %0.3f\"\n",
        "      % metrics.silhouette_score(X, labels))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MAWKLvt66RtG",
        "colab_type": "text"
      },
      "source": [
        "Infortunatly DBscan did not manage to find clusters in our data.."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pKYXCkN_6RtG",
        "colab_type": "text"
      },
      "source": [
        "Now i will try some more clustering algorithms:<br>\n",
        "'MiniBatchKMeans', 'AffinityPropagation', 'SpectralClustering', 'Ward', 'AgglomerativeClustering', 'OPTICS', 'Birch', 'GaussianMixture'<br>\n",
        "i will also plot the resulting clusters and print the scores (only if more than 10 clusters found)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "kZ9IZEF_6RtG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "import time\n",
        "import warnings\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from sklearn import cluster, datasets, mixture\n",
        "from sklearn.neighbors import kneighbors_graph\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from itertools import cycle, islice\n",
        "\n",
        "np.random.seed(0)\n",
        "\n",
        "# ============\n",
        "# Generate datasets. We choose the size big enough to see the scalability\n",
        "# of the algorithms, but not too big to avoid too long running times\n",
        "# ============\n",
        "n_samples = len(X_reduced[0])\n",
        "\n",
        "# ============\n",
        "# Set up cluster parameters\n",
        "# ============\n",
        "plt.figure(figsize=(9 * 2 + 3, 12.5))\n",
        "plt.subplots_adjust(left=.02, right=.98, bottom=.001, top=.96, wspace=.05,\n",
        "                    hspace=.01)\n",
        "\n",
        "plot_num = 1\n",
        "\n",
        "default_base = {'quantile': .3,\n",
        "                'eps': .3,\n",
        "                'damping': .9,\n",
        "                'preference': -200,\n",
        "                'n_neighbors': 10,\n",
        "                'n_clusters': 35,\n",
        "                'min_samples': 20,\n",
        "                'xi': 0.05,\n",
        "                'min_cluster_size': 0.01}\n",
        "\n",
        "datasets = [\n",
        "    (X_reduced, {'damping': .77, 'preference': -240,\n",
        "                     'quantile': .2, 'n_clusters': 35,\n",
        "                     'min_samples': 20, 'xi': 0.25})]\n",
        "\n",
        "for i_dataset, (dataset, algo_params) in enumerate(datasets):\n",
        "    # update parameters with dataset-specific values\n",
        "    params = default_base.copy()\n",
        "    params.update(algo_params)\n",
        "\n",
        "    X = dataset\n",
        "\n",
        "    # normalize dataset for easier parameter selection\n",
        "    X = StandardScaler().fit_transform(X)\n",
        "\n",
        "    # estimate bandwidth for mean shift\n",
        "    bandwidth = cluster.estimate_bandwidth(X, quantile=params['quantile'])\n",
        "\n",
        "    # connectivity matrix for structured Ward\n",
        "    connectivity = kneighbors_graph(\n",
        "        X, n_neighbors=params['n_neighbors'], include_self=False)\n",
        "    # make connectivity symmetric\n",
        "    connectivity = 0.5 * (connectivity + connectivity.T)\n",
        "\n",
        "    # ============\n",
        "    # Create cluster objects\n",
        "    # ============\n",
        "    ms = cluster.MeanShift(bandwidth=bandwidth, bin_seeding=True)\n",
        "    two_means = cluster.MiniBatchKMeans(n_clusters=params['n_clusters'])\n",
        "    ward = cluster.AgglomerativeClustering(\n",
        "        n_clusters=params['n_clusters'], linkage='ward',\n",
        "        connectivity=connectivity)\n",
        "    spectral = cluster.SpectralClustering(\n",
        "        n_clusters=params['n_clusters'], eigen_solver='arpack',\n",
        "        affinity=\"nearest_neighbors\")\n",
        "    dbscan = cluster.DBSCAN(eps=params['eps'])\n",
        "    optics = cluster.OPTICS(min_samples=params['min_samples'],\n",
        "                            xi=params['xi'],\n",
        "                            min_cluster_size=params['min_cluster_size'])\n",
        "    affinity_propagation = cluster.AffinityPropagation(\n",
        "        damping=params['damping'], preference=params['preference'])\n",
        "    average_linkage = cluster.AgglomerativeClustering(\n",
        "        linkage=\"average\", affinity=\"cityblock\",\n",
        "        n_clusters=params['n_clusters'], connectivity=connectivity)\n",
        "    birch = cluster.Birch(n_clusters=params['n_clusters'])\n",
        "    gmm = mixture.GaussianMixture(\n",
        "        n_components=params['n_clusters'], covariance_type='full')\n",
        "\n",
        "    clustering_algorithms = (\n",
        "        ('MiniBatchKMeans', two_means),\n",
        "        ('AffinityPropagation', affinity_propagation),\n",
        "        ('MeanShift', ms),\n",
        "        ('SpectralClustering', spectral),\n",
        "        ('Ward', ward),\n",
        "        ('AgglomerativeClustering', average_linkage),\n",
        "        ('DBSCAN', dbscan),\n",
        "        ('OPTICS', optics),\n",
        "        ('Birch', birch),\n",
        "        ('GaussianMixture', gmm)\n",
        "    )\n",
        "\n",
        "    for name, algorithm in clustering_algorithms:\n",
        "        t0 = time.time()\n",
        "\n",
        "        # catch warnings related to kneighbors_graph\n",
        "        with warnings.catch_warnings():\n",
        "            warnings.filterwarnings(\n",
        "                \"ignore\",\n",
        "                message=\"the number of connected components of the \" +\n",
        "                \"connectivity matrix is [0-9]{1,2}\" +\n",
        "                \" > 1. Completing it to avoid stopping the tree early.\",\n",
        "                category=UserWarning)\n",
        "            warnings.filterwarnings(\n",
        "                \"ignore\",\n",
        "                message=\"Graph is not fully connected, spectral embedding\" +\n",
        "                \" may not work as expected.\",\n",
        "                category=UserWarning)\n",
        "            algorithm.fit(X)\n",
        "        \n",
        "        t1 = time.time()\n",
        "        if hasattr(algorithm, 'labels_'):\n",
        "            y_pred = algorithm.labels_.astype(np.int)\n",
        "            \n",
        "            if (len(np.unique(algorithm.labels_)) > 1):\n",
        "                print(name + \" silhouette_avg: \", silhouette_score(X, algorithm.labels_))\n",
        "        else:\n",
        "            y_pred = algorithm.predict(X)\n",
        "\n",
        "        plt.subplot(len(datasets), len(clustering_algorithms), plot_num)\n",
        "        if i_dataset == 0:\n",
        "            plt.title(name, size=18)\n",
        "\n",
        "        colors = np.array(list(islice(cycle(['#377eb8', '#ff7f00', '#4daf4a',\n",
        "                                             '#f781bf', '#a65628', '#984ea3',\n",
        "                                             '#999999', '#e41a1c', '#dede00']),\n",
        "                                      int(max(y_pred) + 1))))\n",
        "        # add black color for outliers (if any)\n",
        "        colors = np.append(colors, [\"#000000\"])\n",
        "        plt.scatter(X[:, 0], X[:, 1], s=10, color=colors[y_pred])\n",
        "\n",
        "        plt.xlim(-2.5, 2.5)\n",
        "        plt.ylim(-2.5, 2.5)\n",
        "        plt.xticks(())\n",
        "        plt.yticks(())\n",
        "        plt.text(.99, .01, ('%.2fs' % (t1 - t0)).lstrip('0'),\n",
        "                 transform=plt.gca().transAxes, size=15,\n",
        "                 horizontalalignment='right')\n",
        "        plot_num += 1\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zW51MGFt6RtI",
        "colab_type": "text"
      },
      "source": [
        "As we can see 'Ward' performed best with a score of 0.61, it is a good result but still less than KMeans."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VdwW9LTH6RtI",
        "colab_type": "text"
      },
      "source": [
        "# So the best clustering algorithm for our task is KMeans with K=35"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rFfPks_T6RtJ",
        "colab_type": "text"
      },
      "source": [
        "# The recommandation function:<br>\n",
        "Returns the recommanded movie for the customer:<br>\n",
        "The prediction rule is to take each movie that the customer loved (rating = 5)<br>\n",
        "Take it's corresponding row from the dataset - (all customers are the features in every row)<br>\n",
        "Reduce the dimension with PCA<br>\n",
        "Predict using KMeans , K=35<br>\n",
        "Take a random movie from this cluster<br>\n",
        "Finaly take most frequent movie:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N1lW3T4X6RtJ",
        "colab_type": "text"
      },
      "source": [
        "Below is a simple function that returns a random movie that is inside a given cluster"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "uJB8_xdQ6RtJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_movie(cluster):\n",
        "     return random.choice(np.squeeze(np.argwhere(cluster_labels==cluster)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "97DmMGuQ6RtM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "get_movie(1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "bv5CEu-W6RtN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def recommand_KMeans(CustomerID):\n",
        "    \n",
        "    res = []\n",
        "\n",
        "    for movieID in df[(df['CustomerID'] == CustomerID) & (df['Rating'] == 5)]['Movie_Id']:\n",
        "        \n",
        "        x = np.squeeze(df_p.as_matrix(columns=[movieID])).reshape(1, -1)\n",
        "        \n",
        "        transformed = pca.transform(x)\n",
        "\n",
        "        p = np.squeeze(clusterer.predict(transformed))\n",
        "\n",
        "        res.append(get_movie(p))   \n",
        "        \n",
        "        \n",
        "    return get_movie_name(np.bincount(res).argmax())\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "8rz0lTQ76RtP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"The recommanded movie(ID) for customer 1333 using clustering CF is: \", recommand_KMeans(1333))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D1dU-7HB6RtR",
        "colab_type": "text"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-MdgYJWO6RtS",
        "colab_type": "text"
      },
      "source": [
        "# Task3 - Memory-based CF"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ReLBfYsC6RtS",
        "colab_type": "text"
      },
      "source": [
        "# I will try the following correlation coefficients:\n",
        "\n",
        "> Pearson, Kendall, Spearman "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x8lFoV9m6RtT",
        "colab_type": "text"
      },
      "source": [
        "The idea here is to measure the linear correlation between rating of all pairs of movies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "sOl6Qkza6RtT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_similar_movies(method_name, movie_title, n_movies, min_count=0):\n",
        "\n",
        "    i = int(df_title.index[df_title['Name'] == movie_title][0])\n",
        "    target = df_p[i]\n",
        "    similar_to_target = df_p.corrwith(target, method=method_name)\n",
        "    corr_target = pd.DataFrame(similar_to_target, columns = [method_name])\n",
        "    corr_target.dropna(inplace = True)\n",
        "    corr_target = corr_target.sort_values(method_name, ascending = False)\n",
        "    corr_target.index = corr_target.index.map(int)\n",
        "    corr_target = corr_target.join(df_title).join(df_movie_summary)[[method_name, 'Name', 'count']]\n",
        "    return [name for name in corr_target[corr_target['count']>min_count][:n_movies]['Name']]\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39PIXMP96RtV",
        "colab_type": "text"
      },
      "source": [
        "# The recommandation function:<br>\n",
        "Returns the recommanded movie for the customer:<br>\n",
        "The prediction rule is to take all the movies that the customer loved (rating = 5), than take the corresponding movieID column from the dataset - find the highest correlated column of another movie and return it, do it for each of the movies that the customer loved."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "3Grz8z0E6RtV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# n_movies - is the maximum number of movies to return\n",
        "# as_names - the names of the movies / only the id \n",
        "\n",
        "def recommand_corr(method_name, CustomerID):\n",
        "\n",
        "    res = []\n",
        "\n",
        "    for x in df[(df['CustomerID'] == CustomerID) & (df['Rating'] == 5)]['Movie_Id']:\n",
        "        p = get_similar_movies(method_name, get_movie_name(x), 1)[0]\n",
        "        res.append(p)    \n",
        "\n",
        "        \n",
        "    return res"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "J6ArSGWN6RtY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "for method in ['pearson', 'kendall', 'spearman']:\n",
        "    print(method, recommand_corr(method, 1333))\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IQhK6_U06Rta",
        "colab_type": "text"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iKb9hXAm6Rta",
        "colab_type": "text"
      },
      "source": [
        "# My future work..\n",
        "\n",
        "My next step will be to try and comapre all ther different approaches."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dt08bu5T6Rtb",
        "colab_type": "text"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hH4QRGzi6Rtb",
        "colab_type": "text"
      },
      "source": [
        "**Thanks alot for the greate class, (Yoram you were greate!)**"
      ]
    }
  ]
}