Content_Based_Filtering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8fa940b4",
   "metadata": {},
   "source": [
    "### Content Based Filtering\n",
    "They suggest similar items based on a particular item. This system uses item metadata, such as genre, director, description, actors, etc. for movies, to make these recommendations. The general idea behind these recommender systems is that if a person liked a particular item, he or she will also like an item that is similar to it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19cca487",
   "metadata": {},
   "source": [
    "### Plot description based Recommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "89899023",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# import packages and data\n",
    "\n",
    "dframe_1 = pd.read_csv('data/tmdb_5000_credits.csv')\n",
    "dframe_2 = pd.read_csv('data/tmdb_5000_movies.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "09f21e2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "dframe_1.columns = ['id', 'title', 'cast', 'crew']\n",
    "dframe_2 = dframe_2.merge(dframe_1, on='id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "921b6f61",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    In the 22nd century, a paraplegic Marine is di...\n",
       "1    Captain Barbossa, long believed to be dead, ha...\n",
       "2    A cryptic message from Bond’s past sends him o...\n",
       "Name: overview, dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dframe_2['overview'].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe28c15c",
   "metadata": {},
   "source": [
    "### TF-IDF\n",
    "we're using the TF-IDF approach \n",
    "the overall importance of each word to the document in which they appear is TF * IDF\n",
    "TF(term frequency) = (term instances / total instances)\n",
    "IDF(inverse document frequency) = log(number of documents / documents with term)\n",
    "\n",
    "this will give a matrix where each column represents a word that appear at least once in the document,\n",
    "and each row represents a movie\n",
    "\n",
    "This is done to reduce the importance of words that occur frequently in plot overviews and therefore their importance in computing the final similarity score.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7d420e50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4803, 20978)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import TfIdfVectorizer from scikit-learn\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# Define a TF-IDF Vectorizer Object. Remove all english stop words such as 'the', 'a'\n",
    "tfidf = TfidfVectorizer(stop_words='english')\n",
    "\n",
    "# Replace NaN with an empty string\n",
    "dframe_2['overview'] = dframe_2['overview'].fillna('')\n",
    "\n",
    "# Construct the required TF-IDF matrix bu fitting and transforming the data\n",
    "tfidf_matrix = tfidf.fit_transform(dframe_2['overview'])\n",
    "\n",
    "# Output the shape of tfidf_matrix\n",
    "tfidf_matrix.shape"
   ]
  },
  {
   "attachments": {
    "download.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAB1CAMAAADKkk7zAAAAh1BMVEX///8AAAD5+fnFxcWtra3z8/P8/Pzw8PBycnL09PTo6Oj39/e9vb3g4ODBwcGXl5ednZ3Pz8/c3NyIiIiOjo5FRUXS0tJ+fn64uLimpqZPT0/d3d1tbW1jY2NXV1c9PT02NjZ3d3cvLy9TU1NdXV0YGBggICBISEg5OTkqKioLCwsTExMdHR2pbwthAAARwklEQVR4nO1d2YKiOhBNARJAIIGETVkVtbX7/7/vJiCILTPdLr3ckfPQ4xoYDlU5VamUCE2YMGHChAkTng2+YS1Z8tNnMeFzyMhBm716P30aEz4DnGQ1UjbxT5/HhM8BYmSBqf/0aUz4DDDMEQusyRX+L5CskF5HxPrp85jwGcRiyvIi/6dPY8KECRN+BKZ1DvzTJzThL/DyrGIsapFmteG8+wA+f0GZmeZMQPm+U5zQA2/BCI9wSfoC7Pz9GVmbg6d6SAWShGr+/FvPc0IDF2CgBdX1y/nb3ls+NC7FS2EfUR4FhT3F0j8ABuVgsnJ29vBNzGARnn1c3XPpBeNNNgn+n8AaqpOZ6PYwFa+7xSJQzz6tQfMBe7WYMh8/AecNtD+9VSXG6izZa5KtfG7yzZT5+BloZ1PXAHON42h7tu7l1zsXW0uaTWT9FAxYjMZZoeHM+IHKh/pRArr7BUlouqJy8tKVSWl8P3AAbOS6Y5IgJQEuH/taK+Q1qGJN04zME1/w4snAfgDeAS6vu26XNI6jNgLzeGN9x2kLxQdDfMEm72PpCd+BVXbpCp2UUEIjiMRj5ZiS8uu6EfT2Pg+RaZkXX5rw9bDT5cVrcyppQiFUwifGSRt1qRsmGZoTiDBWkynk+gGo+eVlN+M2LA4hnSMvSknzahtt4aTcqchO6j8p/wlfhzAfBLp64/IUjwcrTRdGxSEgqqVltrAoj2aQCo3Bc8NThPBI3Z865ecFzgfxr642TxQ10RJNQVj81aiKNAMraK6SNqer2Y0moZE/Kfhvhl7RwTO/HrMXp+L+hfyzKs17Vk2oO+NLRthp8WXrhHywRqK4xsuYufhpGl8c30mrp4234nR0ytZplhpGmmVplKhfsQRID5rqNoiThAdSAV5i5tmXVjR/4uB4B/uxu1pXkxyaqZ1B8AWV6j7sF7sW5dsGDiv14+9MUN8AwvG3GDSJVSeA8vGX0rGHcG11WhL+BDIAMEbfUY5szSp44997UhPGYW3TA8Bs7K2erQy2UzT6K0DAWQCMzkuCrTfu+CF9W9Ev0IVsuLKvhyp5/CH+OQQB0l5hNfaWYOsQpFm9h+wPE9tdSIcL9grl+y84xj8GF8QdvhrqDMw6ryfY2rDQ8+JildrjX78H1jAs0J1k9I6ZMEQGVIsLodT7V1TYHh918xayAbJHR1whfVc/GE9sfQRr9VoUhbCtk87AaZe869myAILLhY37UKkgAmSaJIlMAOpIn9j6EOTg+b7v7ABOc3zvoXq2fIDdg5MHikdeRMzVAT0HW8ogLzO7XrgF6+Yf7QBvI2lS3ip3R4Rko2mhuwAi4O4qq8NfwJbuf3k2C/NWEihujJFDriuLxKnQ7oWJkPEiXGERnCcTdL7YAOzz9bqA8vEKXjhCguIjpIaxv4MtnbPLF+cxWyJvV4yFEBZJHvZfn1GjKVBw0rRiWLev22SoLIUXlAtFS797dHaifo8vKIPQDiREsw5IITnwr19inMPm0oc4iw0VlL1GI1+I9ztpAvEjQphw0dBjZQZWdxqa07Fyr98JUzt3BNYXLs2coEfR5RUyNXmX++UYWw5v3Ap7wI2Eq0Z4m3QXIj/gsszh/7MCPv+RG2s+ljtuCkz9lzG20FxqrvGV0ivhrJpZy38Txwn3wiM7xpR9/QM0TjwPhZ46Q47n+cjVhKgINalvLPn8yJZDiI2b11TLTxzF9zyMMDtQz5Gfk39Dz/9r8BkTu7klQkob/6knPPGWSIkDKWNm5E2Q5slaST3JnnaZ7u8Ispi+ErTYlo7O9mW0plVAI0a2DGGyLZIjW95LGkdBLATBvlznK2Yb2zy0og2sXhhOiv2eYb46/O0iWy+LJFoJLx/tE20nhrSCSGNljHCUy/eXgdyWZjd1yHYwremNwQVZBhKJuQMcZHpQWIoDpaZbmQgkFRXokS0bIrQ0ah/p2iGNs8rCUeEhhUNsmiJaqgthLbWGWyfu5kGPstN3Jei6IbigYCm6syGIFIKxNEFWncn3Q9gEu2C/l17RLbskn704DZV/RVK2ga4Vp8NUv7ckM4ZFgrEvF1gdmZtJ5U7anfgTyesfNmwVgi3sCbOhhbjl7QNtJjki2BKXvjUCBsKZGl1aRzFnJ+jdgZiULQoCudVTWZSYQxZjyxI2JlcRdRuYYnpBU0eplt3ahzIYqW+jNPPUx6DnZfQw1qMOcyfck1oXAf6brOE+siWJ2a/P2WrmLb8qV2+B0BTxa3slectWqzLUIprx+G+zVtXsSNMRhoW8OtnGwzsRtTLlyJaZbJtpq1E0akn/MhRSA3gMRuVTj2T/oMPci0EGxqv2UPdssRG2hG3pMWS+SWRdQ7x5x5YjlXy1CaveUeHBTekqHVttlhVDINmqNyrS3fT1wJC1lgIeS1udNdYr6Fh1tmUN7jGvS9nqyoPQq2/LPR2m32z/sMPci/40E8lbejiy5YyyJeItf7HwRNRa8Bjbb+/Y4mpzH+aslxheZPRIj95GPW5jQiCX65Ria3ExNl7UIiiW85ai7Xxhxm1G5TRvqdVpqK/ro6Tb6ekwv3SXjrSFRKgMxMVUwg/W0bbMfS4T14UU73Lt3CmZYEvMJ05WVAS7m/ZKkiBs5i2dy2h+lsEH2b1SHokwMYGp7W0RiZtCMSoRBjSa0Cs9ROs2N2QXkyZ8ByZrhCgsqgyW6BVgQyLhI4tY/qFbgEMqn0eq+LNQbShTI90AI+JpaSN1If61kZ/DSmt8HQ8+uvNXUGVCBopDrnNhaPMKcmNRWDI6kHezla2yqPV1SlL/zvv7B1GBuNstjNTPXRlHxcgP3+dZdC9s3TzXPiykw3YrGvVQmqFuzRW1cZ5Oo9pF/NC5UisdSTG/gytmS3W0xskSduk2NqrJ1cGRT+hOiHA4+uVfDHjQOCp3UHV7MIRZev6Cuv7YEVaejrLTfcbd/j6KxWhZM0AgQsZS3lHuEV77oZlG0ZLf1K7C0a6DfZniteLrhkja/+WyuOV8R8BAJfcsHS2LMy+K+SfShNEZW+4r9OGDLRx82rAlA/xCytATUnmaki3nNrbqK8V3cHkU7VoB36q610flBvx0m96T2NPt3ZDr2PgE85E6ZEtcRK0zroatJgyUbDVtl/QSmkVKcaGk/ryDrd39hdPaTSWofvrxZ74JijYoiv5cx9MztiwRx64644ov2cIrKHD7Oalk72Arv3vxyqJfUCb323HGFhMStm8Hc8bWvJQPera2IGOD2+etsLpbm4Tky9KdvxdnbJXR+gBZl5FMh7ZVygeSLUvXFRUO0ohvZ4vSuz2hS56wPeiQLQ18ddvXEjaa8MK2Nqlh5PAayat9O1uGd/dScfzXFOg/iiFbeTlHAcDxMoyztY0Yi4rXppHq7Ww9oB5Re3K2nC3FOn+FfaszztjqPWEgLQ/vYHXXvNU339JnM3OGb6jTxrSThLppiiGeY3PegC12EF4ue+32LPxh3pLXWdGaUvmb2fLTjq04K6JqfAfSB0OQLvL30jytSv4Us9iAraLmhJP60MRSR03YqOSerdklWzcpeI13FYmVB0tk1NcPofZ9CW1eaygZaTn0D6JjS9G1Q3PdwzcALJeC7J6thWBLls4qltSEiqIsM9jwO9iK+s7FilxB2NwwBfUiQxdMhYg9l23ZSQDE11GoCVdoUP89W6W0KLKBLUkSksNeNpq7ma1B++mCIgz4+tBpIDIImy2z2Pq/ZZdvQcdWtMjznaagZJ3n+Vr2WLiwLXMtPpMvBPIqvitPuD5JQrCQ9ur9vVRgBErSz3W40lBcM+0ZivHOs7pDuEO25mObCMwb2cKnJoSOmLLUNL26sN4hfbnsknjIY8ZTZDbOs7pDXHjCC9xqWza722t5/PfWv30h/mxbH7N1q21x7YoAaz66e8EmV5ijbv0rXpIJti5+4aWBGskNpvJRjZGyG/mEGSfCJV3vgtqeADN8jvFcFLYJHelnF8vdVfr83RDjN4HiUa7+G71cXXHbXbYlk3DEzND2/xOmoI8tJim+15azXomF/IpJidxmnCSUCIhI78JG5R7aGUlD9nJxQ+ha0/oz5schmjEIH2835C487bRsN+E6zGrJFt7BtrnWhDEWGWVTzTrEUpZmW4sKOZuLNXCLyDoRK4It0/ohAsj7EHnmqW0xpodRtDJREEw//XQb1KixZXUPg9IcN33Xr2heHTTpbGPkby82lYa87bWbAzmZUxjt+zoUh+1ldZmRAcNqoqBgPbF1G0jSXmF6GP4Qhrk4tyBPmF7zYJ6sLzyhe0xduMV24OJwNFjzYk0JupJBItv7r+LJE96GtFvciiAbTP7Ls9ZtuF4fGvp01bis3IqP1qYkr/WAcZ+e9JIBzWMGhoks4ws6Kj4J+kwGXp9l34fKVNdSrdkMoYfURxexVZ93ElMXG2ik08KJuS6bw2TCxDC10f3Ln88Jpe5VpLr9U+dKZ+24Ms2vhwZRtfcqw+oXt1BYv47KPbWMTFm2WEQmZpGr1tO8dRO86mQMFNaj0YPCKQrllp0ZeVns6vfVTf5A7tvleiw+J6+VFsfZns6RvVvsCjLZ1k1IyEn8KSmwsZvez3VkSU0/l52R1feMqvwU5JkExqL7FDillO2oiUIxxEjz5AmfQaUObnOngJGyQCWt7TiBsfxJg3joGq0KLiuo8Hp1nLYmMXgXzn5bFREY6T3hFlFVGW0NyBi0swAsLulFXsletYXKEUT/Rs7pm+Ed3ZmeD1NVijGybmzlnuVY4eZFpiaw1itC6yhJMB1KyRnhl8kvDu1+YCEJxb+xOhnYVbC6zZ3+Wak/H5m25u1ui/mL7DMhvtiJCEyPJuXzQQSma2xk8SSFZfPeprk3qmRi6yrQbs+nNqygSNILSTijDCJLSHfyBoaYdPS+1Yu3P25H89jAmbrsQhLOYloCT5KE7atmp5z/FEUbj4NVwzHfFA1aFsTZoFJJaWMoXFWRYENXIwGqO3ZnR5h1v4ds85OpeGzg5WbttkNMxVdlopc2P5mB1SkBfx1o9la1HnCQKVLrYWxM2jo1xRIQV3cm/8WY0uz4fhisjhtzB5LQZ8N+FVrk9EMchxFI7O0zVNg8DpjTYNFe6pPICPNhAlAbk/JCY/Bj+hDHLGsbIZmnCpolG278dYtkbIHL1l4mXXgNTMepN41ROX0Pq2XNl85y6cv9y37C3mDUXS2D47LK3PFZ29fX4R2vFos8McTSRLrlxzzYjFcd5HzyhFcibZdH4ug445sZyGK4fF1FRr4GgGzsW7q9sLqATNHa5sxeJwIxgbKWY6SRUa+3YojRjIW1da4pBJmAZJzayAGeHL2SorquLbecN78JJDBaxGTSIO4NRm1/8thlR383D9tN63Y/xCgp6opOCY0rQYDKuT67rkRK95PTWocHhuTJvrJHJk7iad66Etq+CYvza9Ors5NZhKXsDDe/eufWbPKD18ILCtmv7NQo03HP3dPHPyfoZC9xV0HTwHpXcPXJBjMTPoRZS5nhRv31pWdJp5huP/SRmMkugSHrozSXD/cBeWT3hNv8vwiNKCT09AspZ3ZhWfBh4xw9lg3k1FNdtmkN4yvsLSY18ShEMi3eV9Ag7f1q7sdsCVEom932zas8er7FDmcTW48COXCM6u76JuEqRDZpweV881GrPyRFYYr1fnHL03iCQnocQtwGgq2vOvmnQ7yqLbOPXzVrYwlneIR0aJ+wLb9YhJh0FTRhHCXCGR5hoomtByLcrSyv6iYrhb6vnv0EW066ipes/1ycnpvj5AkfB6UGJzkV0xZugmLegn3SE2J+SMJeVeqEeH7YDRFOtvVQpOBEdmdQ1ptKFPO0lcf34OO9RUIUErUXGU4VxY5yGsKKC+ZNWYsHQYjC9SkZSM534Wtxwj/efSxE4SnvpNv8jF9V0zif4uMHgW5YcLqYyrkVzBVl/vGaofe2YKcKGuX8h4UUMYQ5TVwPgv2yz+/cUuovDvXUhvtb4CzOdhjcAquCu7qjTvg8Mvjrj358AjMKH7fhnvAQGHB32wQXph+A+yaw7d2KTd1P++a+Ccm9IkOIwkf8ZOOEz8C/P3a1LnYHTfjH8R9wYiZCyjFLIwAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "id": "e950f6e2",
   "metadata": {},
   "source": [
    "we will be using the cosine similarity to calculate a numeric quantity that denotes the similarity between two movies.\n",
    "\n",
    "we use the cosine similarity score since it is independent of magnitude and is relatively easy and fast to calculate mathematically.\n",
    "\n",
    "![download.png](attachment:download.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3fdfec26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import linear_kernel\n",
    "from sklearn.metrics.pairwise import linear_kernel\n",
    "\n",
    "# Compute the cosine similarity matrix\n",
    "cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e963c383",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We are going to define a function that takes in a movie title as an input and outputs a list of the 10 most similar movies.\n",
    "# Firstly, for this, we need a reverse mapping of movie titles and DataFrame indices.\n",
    "# In other words, we need a mechanism to identify the index of a movie in our metadata DataFrame, given its title.\n",
    "\n",
    "# Construct a reverse map of indices and movie titles\n",
    "indices = pd.Series(dframe_2.index, index=dframe_2['title_y']).drop_duplicates()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "66af863f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function takes in movie title as input and outputs most similar movies\n",
    "\n",
    "def get_recommendations(title, cosine_sim=cosine_sim):\n",
    "    # Get the index of the movie that matches the title\n",
    "    idx = indices[title]\n",
    "    \n",
    "    # Get the pairwise similarity scores of all movies with that movei\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    \n",
    "    # sort the movie based on similartiy scores\n",
    "    sim_scores = sorted(sim_scores, key = lambda x: x[1], reverse=True)\n",
    "    \n",
    "    # Get the scores of the 10 most similar movies\n",
    "    sim_scores = sim_scores[1:11]\n",
    "    \n",
    "    # Get the movie indices\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    \n",
    "    # Return the top 10 most similar movies\n",
    "    return dframe_2['title_y'].iloc[movie_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "39714ef4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65                              The Dark Knight\n",
       "299                              Batman Forever\n",
       "428                              Batman Returns\n",
       "1359                                     Batman\n",
       "3854    Batman: The Dark Knight Returns, Part 2\n",
       "119                               Batman Begins\n",
       "2507                                  Slow Burn\n",
       "9            Batman v Superman: Dawn of Justice\n",
       "1181                                        JFK\n",
       "210                              Batman & Robin\n",
       "Name: title_y, dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight Rises')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e6bb10ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3604                       Apollo 18\n",
       "2130                    The American\n",
       "634                       The Matrix\n",
       "1341            The Inhabited Island\n",
       "529                 Tears of the Sun\n",
       "1610                           Hanna\n",
       "311     The Adventures of Pluto Nash\n",
       "847                         Semi-Pro\n",
       "775                        Supernova\n",
       "2628             Blood and Chocolate\n",
       "Name: title_y, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('Avatar')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b045be2",
   "metadata": {},
   "source": [
    "While our system has done a decent job of finding movies with similar plot descriptions, the quality of recommendations is not that great. \"The Dark Knight Rises\" returns all Batman movies while it is more likely that the people who liked that movie are more inclined to enjoy other Christopher Nolan movies. This is something that cannot be captured by the present system."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f0d1126",
   "metadata": {},
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d38d3e3a",
   "metadata": {},
   "source": [
    "### Credits, Genres and Keywords Based Recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1230e9a",
   "metadata": {},
   "source": [
    "We are going to build a recommender based on the following metadata: the 3 top actors, the director, related genres and the movie plot keywords.\n",
    "\n",
    "From the cast, crew and keywords features, we need to extract the three most important actors, the director and the keywords associated with that movie. Right now, our data is present in the form of \"stringified\" lists , we need to convert it into a safe and usable structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "da0e6737",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ast import literal_eval\n",
    "\n",
    "features = ['cast', 'crew', 'keywords', 'genres']\n",
    "\n",
    "for feature in features:\n",
    "    dframe_2[feature] = dframe_2[feature].apply(literal_eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "23866d14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the director's name from the crew feature. If the director is not listed, return NaN\n",
    "\n",
    "def get_director(x):\n",
    "    for i in x:\n",
    "        if i['job'] == 'Director':\n",
    "            return i['name']\n",
    "        \n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2cc80936",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return the list top 3 elements or entire list; whichever is more.\n",
    "\n",
    "def get_list(x):\n",
    "    if isinstance(x, list):\n",
    "        names = [i['name'] for i in x]\n",
    "        # Check if more than 3 elements exist.\n",
    "        #If yes, return only first three.\n",
    "        #If no, return entire list.\n",
    "        \n",
    "        if len(names) > 3:\n",
    "            names = names[:3]\n",
    "            \n",
    "        return names\n",
    "    # Return empty list in case of missing/malformed data\n",
    "    return []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cb1c249a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define new director, cast, genres and keywords features that are in a suitable form\n",
    "dframe_2['director'] = dframe_2['crew'].apply(get_director)\n",
    "\n",
    "features = ['cast', 'keywords', 'genres']\n",
    "\n",
    "for feature in features:\n",
    "    dframe_2[feature] = dframe_2[feature].apply(get_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "93adae36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title_y</th>\n",
       "      <th>cast</th>\n",
       "      <th>director</th>\n",
       "      <th>keywords</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Sam Worthington, Zoe Saldana, Sigourney Weaver]</td>\n",
       "      <td>James Cameron</td>\n",
       "      <td>[culture clash, future, space war]</td>\n",
       "      <td>[Action, Adventure, Fantasy]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Johnny Depp, Orlando Bloom, Keira Knightley]</td>\n",
       "      <td>Gore Verbinski</td>\n",
       "      <td>[ocean, drug abuse, exotic island]</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Daniel Craig, Christoph Waltz, Léa Seydoux]</td>\n",
       "      <td>Sam Mendes</td>\n",
       "      <td>[spy, based on novel, secret agent]</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    title_y  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "\n",
       "                                               cast        director  \\\n",
       "0  [Sam Worthington, Zoe Saldana, Sigourney Weaver]   James Cameron   \n",
       "1     [Johnny Depp, Orlando Bloom, Keira Knightley]  Gore Verbinski   \n",
       "2      [Daniel Craig, Christoph Waltz, Léa Seydoux]      Sam Mendes   \n",
       "\n",
       "                              keywords                        genres  \n",
       "0   [culture clash, future, space war]  [Action, Adventure, Fantasy]  \n",
       "1   [ocean, drug abuse, exotic island]  [Adventure, Fantasy, Action]  \n",
       "2  [spy, based on novel, secret agent]    [Action, Adventure, Crime]  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dframe_2[['title_y', 'cast', 'director', 'keywords', \n",
    "         'genres']].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3f65888a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to convert all strings to lower case and strip names of space\n",
    "\n",
    "def clean_data(x):\n",
    "    if isinstance(x, list):\n",
    "        return[str.lower(i.replace(\" \", \"\")) for i in x]\n",
    "    else:\n",
    "        # Check if director exists.\n",
    "        #If not, return empty string\n",
    "        if isinstance(x, str):\n",
    "            return str.lower(x.replace(\" \", \"\"))\n",
    "        else:\n",
    "            return ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7d001c31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply clean_data function to feature\n",
    "features = ['cast', 'keywords', 'director', 'genres']\n",
    "\n",
    "for feature in features:\n",
    "    dframe_2[feature] = dframe_2[feature].apply(clean_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "03fded0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_soup(x):\n",
    "    return ' '.join(x['keywords']) + ' ' + ' '.join(x['cast']) + ' ' + x['director'] + ' ' + ' '.join(x['genres'])\n",
    "\n",
    "dframe_2['soup'] = dframe_2.apply(create_soup, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e5fc6d4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import CountVectorizer and create count matrix\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer(stop_words='english')\n",
    "count_matrix = count.fit_transform(dframe_2['soup'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3fb2b135",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute the Cosine Similarity matrix based on the count_matrix\n",
    "\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "cosine_sim2 = cosine_similarity(count_matrix, count_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ed7a6008",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset index of our main DataFrame and construct reverse mapping as before\n",
    "dframe_2 = dframe_2.reset_index()\n",
    "indices = pd.Series(dframe_2.index, index = dframe_2['title_y'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "09941af6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65               The Dark Knight\n",
       "119                Batman Begins\n",
       "4638    Amidst the Devil's Wings\n",
       "1196                The Prestige\n",
       "3073           Romeo Is Bleeding\n",
       "3326              Black November\n",
       "1503                      Takers\n",
       "1986                      Faster\n",
       "303                     Catwoman\n",
       "747               Gangster Squad\n",
       "Name: title_y, dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight Rises', cosine_sim2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6f2f4cad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "867      The Godfather: Part III\n",
       "2731      The Godfather: Part II\n",
       "4638    Amidst the Devil's Wings\n",
       "2649           The Son of No One\n",
       "1525              Apocalypse Now\n",
       "1018             The Cotton Club\n",
       "1170     The Talented Mr. Ripley\n",
       "1209               The Rainmaker\n",
       "1394               Donnie Brasco\n",
       "1850                    Scarface\n",
       "Name: title_y, dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Godfather', cosine_sim2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "962e9132",
   "metadata": {},
   "source": [
    "We see that our recommender has been successful in capturing more information due to more metadata and has given us (arguably) better recommendations. It is more likely that Marvels or DC comics fans will like the movies of the same production house. Therefore, to our features above we can add production_company . We can also increase the weight of the director , by adding the feature multiple times in the soup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74edbead",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}