TMDB_Movie_Recommender_System.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6ba83a37",
   "metadata": {},
   "source": [
    "# MOVIE RECOMMENDER SYSTEM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91d06afd",
   "metadata": {},
   "source": [
    "We are using the dataset from TMDB (https://www.themoviedb.org/about). **The Movie Database** (TMDB) is a community built movie and TV database. The platform consists of around 5000 movies and associated details. The dataset is available on Kaggle - https://www.kaggle.com/datasets/tmdb/tmdb-movie-metadata.\n",
    "\n",
    "### TASK:\n",
    "> Users will be requested to type/select a movie based on their interest. We would be developing a recommender system to help suggest back a list of movies similar to the one selected by the user. We will be sharing atmost 6 recommended movies based on user's choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4db12b57",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "color_pal = sns.color_palette()\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ce2e4d76",
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = pd.read_csv('tmdb_5000_movies.csv')\n",
    "credits = pd.read_csv('tmdb_5000_credits.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4f76fb1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of movies dataframe: (4803, 20)\n",
      "Shape of credits dataframe: (4803, 4)\n"
     ]
    }
   ],
   "source": [
    "print('Shape of movies dataframe:', movies.shape)\n",
    "print('Shape of credits dataframe:', credits.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "752d80dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>keywords</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>237000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.avatarmovie.com/</td>\n",
       "      <td>19995</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>150.437577</td>\n",
       "      <td>[{\"name\": \"Ingenious Film Partners\", \"id\": 289...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2009-12-10</td>\n",
       "      <td>2787965087</td>\n",
       "      <td>162.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Enter the World of Pandora.</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>11800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>300000000</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>http://disney.go.com/disneypictures/pirates/</td>\n",
       "      <td>285</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>en</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>139.082615</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2007-05-19</td>\n",
       "      <td>961000000</td>\n",
       "      <td>169.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>At the end of the world, the adventure begins.</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>4500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      budget                                             genres  \\\n",
       "0  237000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "1  300000000  [{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...   \n",
       "\n",
       "                                       homepage     id  \\\n",
       "0                   http://www.avatarmovie.com/  19995   \n",
       "1  http://disney.go.com/disneypictures/pirates/    285   \n",
       "\n",
       "                                            keywords original_language  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...                en   \n",
       "1  [{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...                en   \n",
       "\n",
       "                             original_title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "\n",
       "                                            overview  popularity  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...  150.437577   \n",
       "1  Captain Barbossa, long believed to be dead, ha...  139.082615   \n",
       "\n",
       "                                production_companies  \\\n",
       "0  [{\"name\": \"Ingenious Film Partners\", \"id\": 289...   \n",
       "1  [{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...   \n",
       "\n",
       "                                production_countries release_date     revenue  \\\n",
       "0  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2009-12-10  2787965087   \n",
       "1  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2007-05-19   961000000   \n",
       "\n",
       "   runtime                                   spoken_languages    status  \\\n",
       "0    162.0  [{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...  Released   \n",
       "1    169.0           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "\n",
       "                                          tagline  \\\n",
       "0                     Enter the World of Pandora.   \n",
       "1  At the end of the world, the adventure begins.   \n",
       "\n",
       "                                      title  vote_average  vote_count  \n",
       "0                                    Avatar           7.2       11800  \n",
       "1  Pirates of the Caribbean: At World's End           6.9        4500  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "878eeec2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credits.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8abb89b6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4803 entries, 0 to 4802\n",
      "Data columns (total 20 columns):\n",
      " #   Column                Non-Null Count  Dtype  \n",
      "---  ------                --------------  -----  \n",
      " 0   budget                4803 non-null   int64  \n",
      " 1   genres                4803 non-null   object \n",
      " 2   homepage              1712 non-null   object \n",
      " 3   id                    4803 non-null   int64  \n",
      " 4   keywords              4803 non-null   object \n",
      " 5   original_language     4803 non-null   object \n",
      " 6   original_title        4803 non-null   object \n",
      " 7   overview              4800 non-null   object \n",
      " 8   popularity            4803 non-null   float64\n",
      " 9   production_companies  4803 non-null   object \n",
      " 10  production_countries  4803 non-null   object \n",
      " 11  release_date          4802 non-null   object \n",
      " 12  revenue               4803 non-null   int64  \n",
      " 13  runtime               4801 non-null   float64\n",
      " 14  spoken_languages      4803 non-null   object \n",
      " 15  status                4803 non-null   object \n",
      " 16  tagline               3959 non-null   object \n",
      " 17  title                 4803 non-null   object \n",
      " 18  vote_average          4803 non-null   float64\n",
      " 19  vote_count            4803 non-null   int64  \n",
      "dtypes: float64(3), int64(4), object(13)\n",
      "memory usage: 750.6+ KB\n"
     ]
    }
   ],
   "source": [
    "movies.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1411e403",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4803 entries, 0 to 4802\n",
      "Data columns (total 4 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   movie_id  4803 non-null   int64 \n",
      " 1   title     4803 non-null   object\n",
      " 2   cast      4803 non-null   object\n",
      " 3   crew      4803 non-null   object\n",
      "dtypes: int64(1), object(3)\n",
      "memory usage: 150.2+ KB\n"
     ]
    }
   ],
   "source": [
    "credits.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f6439288",
   "metadata": {},
   "outputs": [],
   "source": [
    "# analyzing both dataframes, there is a common feature named 'title'\n",
    "# we will be merging both dataframes using this common feature\n",
    "\n",
    "merged_df = movies.merge(credits, on='title')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "526e375e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of merged dataframe: (4809, 23)\n"
     ]
    }
   ],
   "source": [
    "print('Shape of merged dataframe:', merged_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f600129a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>keywords</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>...</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>237000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.avatarmovie.com/</td>\n",
       "      <td>19995</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>150.437577</td>\n",
       "      <td>[{\"name\": \"Ingenious Film Partners\", \"id\": 289...</td>\n",
       "      <td>...</td>\n",
       "      <td>162.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Enter the World of Pandora.</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>11800</td>\n",
       "      <td>19995</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>300000000</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>http://disney.go.com/disneypictures/pirates/</td>\n",
       "      <td>285</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>en</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>139.082615</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...</td>\n",
       "      <td>...</td>\n",
       "      <td>169.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>At the end of the world, the adventure begins.</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>4500</td>\n",
       "      <td>285</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      budget                                             genres  \\\n",
       "0  237000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "1  300000000  [{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...   \n",
       "\n",
       "                                       homepage     id  \\\n",
       "0                   http://www.avatarmovie.com/  19995   \n",
       "1  http://disney.go.com/disneypictures/pirates/    285   \n",
       "\n",
       "                                            keywords original_language  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...                en   \n",
       "1  [{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...                en   \n",
       "\n",
       "                             original_title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "\n",
       "                                            overview  popularity  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...  150.437577   \n",
       "1  Captain Barbossa, long believed to be dead, ha...  139.082615   \n",
       "\n",
       "                                production_companies  ... runtime  \\\n",
       "0  [{\"name\": \"Ingenious Film Partners\", \"id\": 289...  ...   162.0   \n",
       "1  [{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...  ...   169.0   \n",
       "\n",
       "                                    spoken_languages    status  \\\n",
       "0  [{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...  Released   \n",
       "1           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "\n",
       "                                          tagline  \\\n",
       "0                     Enter the World of Pandora.   \n",
       "1  At the end of the world, the adventure begins.   \n",
       "\n",
       "                                      title vote_average vote_count movie_id  \\\n",
       "0                                    Avatar          7.2      11800    19995   \n",
       "1  Pirates of the Caribbean: At World's End          6.9       4500      285   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "\n",
       "[2 rows x 23 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "681ed8ef",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 4809 entries, 0 to 4808\n",
      "Data columns (total 23 columns):\n",
      " #   Column                Non-Null Count  Dtype  \n",
      "---  ------                --------------  -----  \n",
      " 0   budget                4809 non-null   int64  \n",
      " 1   genres                4809 non-null   object \n",
      " 2   homepage              1713 non-null   object \n",
      " 3   id                    4809 non-null   int64  \n",
      " 4   keywords              4809 non-null   object \n",
      " 5   original_language     4809 non-null   object \n",
      " 6   original_title        4809 non-null   object \n",
      " 7   overview              4806 non-null   object \n",
      " 8   popularity            4809 non-null   float64\n",
      " 9   production_companies  4809 non-null   object \n",
      " 10  production_countries  4809 non-null   object \n",
      " 11  release_date          4808 non-null   object \n",
      " 12  revenue               4809 non-null   int64  \n",
      " 13  runtime               4807 non-null   float64\n",
      " 14  spoken_languages      4809 non-null   object \n",
      " 15  status                4809 non-null   object \n",
      " 16  tagline               3965 non-null   object \n",
      " 17  title                 4809 non-null   object \n",
      " 18  vote_average          4809 non-null   float64\n",
      " 19  vote_count            4809 non-null   int64  \n",
      " 20  movie_id              4809 non-null   int64  \n",
      " 21  cast                  4809 non-null   object \n",
      " 22  crew                  4809 non-null   object \n",
      "dtypes: float64(3), int64(5), object(15)\n",
      "memory usage: 901.7+ KB\n"
     ]
    }
   ],
   "source": [
    "merged_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f2116cb6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"credit_id\": \"5602a8a7c3a3685532001c9a\", \"gender\": 2, \"id\": 65731, \"name\": \"Sam Worthington\", \"order\": 0}, {\"cast_id\": 3, \"character\": \"Neytiri\", \"credit_id\": \"52fe48009251416c750ac9cb\", \"gender\": 1, \"id\": 8691, \"name\": \"Zoe Saldana\", \"order\": 1}, {\"cast_id\": 25, \"character\": \"Dr. Grace Augustine\", \"credit_id\": \"52fe48009251416c750aca39\", \"gender\": 1, \"id\": 10205, \"name\": \"Sigourney Weaver\", \"order\": 2}, {\"cast_id\": 4, \"character\": \"Col. Quaritch\", \"credit_id\": \"52fe48009251416c750ac9cf\", \"gender\": 2, \"id\": 32747, \"name\": \"Stephen Lang\", \"order\": 3}, {\"cast_id\": 5, \"character\": \"Trudy Chacon\", \"credit_id\": \"52fe48009251416c750ac9d3\", \"gender\": 1, \"id\": 17647, \"name\": \"Michelle Rodriguez\", \"order\": 4}, {\"cast_id\": 8, \"character\": \"Selfridge\", \"credit_id\": \"52fe48009251416c750ac9e1\", \"gender\": 2, \"id\": 1771, \"name\": \"Giovanni Ribisi\", \"order\": 5}, {\"cast_id\": 7, \"character\": \"Norm Spellman\", \"credit_id\": \"52fe48009251416c750ac9dd\", \"gender\": 2, \"id\": 59231, \"name\": \"Joel David Moore\", \"order\": 6}, {\"cast_id\": 9, \"character\": \"Moat\", \"credit_id\": \"52fe48009251416c750ac9e5\", \"gender\": 1, \"id\": 30485, \"name\": \"CCH Pounder\", \"order\": 7}, {\"cast_id\": 11, \"character\": \"Eytukan\", \"credit_id\": \"52fe48009251416c750ac9ed\", \"gender\": 2, \"id\": 15853, \"name\": \"Wes Studi\", \"order\": 8}, {\"cast_id\": 10, \"character\": \"Tsu\\'Tey\", \"credit_id\": \"52fe48009251416c750ac9e9\", \"gender\": 2, \"id\": 10964, \"name\": \"Laz Alonso\", \"order\": 9}, {\"cast_id\": 12, \"character\": \"Dr. Max Patel\", \"credit_id\": \"52fe48009251416c750ac9f1\", \"gender\": 2, \"id\": 95697, \"name\": \"Dileep Rao\", \"order\": 10}, {\"cast_id\": 13, \"character\": \"Lyle Wainfleet\", \"credit_id\": \"52fe48009251416c750ac9f5\", \"gender\": 2, \"id\": 98215, \"name\": \"Matt Gerald\", \"order\": 11}, {\"cast_id\": 32, \"character\": \"Private Fike\", \"credit_id\": \"52fe48009251416c750aca5b\", \"gender\": 2, \"id\": 154153, \"name\": \"Sean Anthony Moran\", \"order\": 12}, {\"cast_id\": 33, \"character\": \"Cryo Vault Med Tech\", \"credit_id\": \"52fe48009251416c750aca5f\", \"gender\": 2, \"id\": 397312, \"name\": \"Jason Whyte\", \"order\": 13}, {\"cast_id\": 34, \"character\": \"Venture Star Crew Chief\", \"credit_id\": \"52fe48009251416c750aca63\", \"gender\": 2, \"id\": 42317, \"name\": \"Scott Lawrence\", \"order\": 14}, {\"cast_id\": 35, \"character\": \"Lock Up Trooper\", \"credit_id\": \"52fe48009251416c750aca67\", \"gender\": 2, \"id\": 986734, \"name\": \"Kelly Kilgour\", \"order\": 15}, {\"cast_id\": 36, \"character\": \"Shuttle Pilot\", \"credit_id\": \"52fe48009251416c750aca6b\", \"gender\": 0, \"id\": 1207227, \"name\": \"James Patrick Pitt\", \"order\": 16}, {\"cast_id\": 37, \"character\": \"Shuttle Co-Pilot\", \"credit_id\": \"52fe48009251416c750aca6f\", \"gender\": 0, \"id\": 1180936, \"name\": \"Sean Patrick Murphy\", \"order\": 17}, {\"cast_id\": 38, \"character\": \"Shuttle Crew Chief\", \"credit_id\": \"52fe48009251416c750aca73\", \"gender\": 2, \"id\": 1019578, \"name\": \"Peter Dillon\", \"order\": 18}, {\"cast_id\": 39, \"character\": \"Tractor Operator / Troupe\", \"credit_id\": \"52fe48009251416c750aca77\", \"gender\": 0, \"id\": 91443, \"name\": \"Kevin Dorman\", \"order\": 19}, {\"cast_id\": 40, \"character\": \"Dragon Gunship Pilot\", \"credit_id\": \"52fe48009251416c750aca7b\", \"gender\": 2, \"id\": 173391, \"name\": \"Kelson Henderson\", \"order\": 20}, {\"cast_id\": 41, \"character\": \"Dragon Gunship Gunner\", \"credit_id\": \"52fe48009251416c750aca7f\", \"gender\": 0, \"id\": 1207236, \"name\": \"David Van Horn\", \"order\": 21}, {\"cast_id\": 42, \"character\": \"Dragon Gunship Navigator\", \"credit_id\": \"52fe48009251416c750aca83\", \"gender\": 0, \"id\": 215913, \"name\": \"Jacob Tomuri\", \"order\": 22}, {\"cast_id\": 43, \"character\": \"Suit #1\", \"credit_id\": \"52fe48009251416c750aca87\", \"gender\": 0, \"id\": 143206, \"name\": \"Michael Blain-Rozgay\", \"order\": 23}, {\"cast_id\": 44, \"character\": \"Suit #2\", \"credit_id\": \"52fe48009251416c750aca8b\", \"gender\": 2, \"id\": 169676, \"name\": \"Jon Curry\", \"order\": 24}, {\"cast_id\": 46, \"character\": \"Ambient Room Tech\", \"credit_id\": \"52fe48009251416c750aca8f\", \"gender\": 0, \"id\": 1048610, \"name\": \"Luke Hawker\", \"order\": 25}, {\"cast_id\": 47, \"character\": \"Ambient Room Tech / Troupe\", \"credit_id\": \"52fe48009251416c750aca93\", \"gender\": 0, \"id\": 42288, \"name\": \"Woody Schultz\", \"order\": 26}, {\"cast_id\": 48, \"character\": \"Horse Clan Leader\", \"credit_id\": \"52fe48009251416c750aca97\", \"gender\": 2, \"id\": 68278, \"name\": \"Peter Mensah\", \"order\": 27}, {\"cast_id\": 49, \"character\": \"Link Room Tech\", \"credit_id\": \"52fe48009251416c750aca9b\", \"gender\": 0, \"id\": 1207247, \"name\": \"Sonia Yee\", \"order\": 28}, {\"cast_id\": 50, \"character\": \"Basketball Avatar / Troupe\", \"credit_id\": \"52fe48009251416c750aca9f\", \"gender\": 1, \"id\": 1207248, \"name\": \"Jahnel Curfman\", \"order\": 29}, {\"cast_id\": 51, \"character\": \"Basketball Avatar\", \"credit_id\": \"52fe48009251416c750acaa3\", \"gender\": 0, \"id\": 89714, \"name\": \"Ilram Choi\", \"order\": 30}, {\"cast_id\": 52, \"character\": \"Na\\'vi Child\", \"credit_id\": \"52fe48009251416c750acaa7\", \"gender\": 0, \"id\": 1207249, \"name\": \"Kyla Warren\", \"order\": 31}, {\"cast_id\": 53, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acaab\", \"gender\": 0, \"id\": 1207250, \"name\": \"Lisa Roumain\", \"order\": 32}, {\"cast_id\": 54, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acaaf\", \"gender\": 1, \"id\": 83105, \"name\": \"Debra Wilson\", \"order\": 33}, {\"cast_id\": 57, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acabb\", \"gender\": 0, \"id\": 1207253, \"name\": \"Chris Mala\", \"order\": 34}, {\"cast_id\": 55, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acab3\", \"gender\": 0, \"id\": 1207251, \"name\": \"Taylor Kibby\", \"order\": 35}, {\"cast_id\": 56, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acab7\", \"gender\": 0, \"id\": 1207252, \"name\": \"Jodie Landau\", \"order\": 36}, {\"cast_id\": 58, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acabf\", \"gender\": 0, \"id\": 1207254, \"name\": \"Julie Lamm\", \"order\": 37}, {\"cast_id\": 59, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acac3\", \"gender\": 0, \"id\": 1207257, \"name\": \"Cullen B. Madden\", \"order\": 38}, {\"cast_id\": 60, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acac7\", \"gender\": 0, \"id\": 1207259, \"name\": \"Joseph Brady Madden\", \"order\": 39}, {\"cast_id\": 61, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acacb\", \"gender\": 0, \"id\": 1207262, \"name\": \"Frankie Torres\", \"order\": 40}, {\"cast_id\": 62, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acacf\", \"gender\": 1, \"id\": 1158600, \"name\": \"Austin Wilson\", \"order\": 41}, {\"cast_id\": 63, \"character\": \"Troupe\", \"credit_id\": \"52fe48019251416c750acad3\", \"gender\": 1, \"id\": 983705, \"name\": \"Sara Wilson\", \"order\": 42}, {\"cast_id\": 64, \"character\": \"Troupe\", \"credit_id\": \"52fe48019251416c750acad7\", \"gender\": 0, \"id\": 1207263, \"name\": \"Tamica Washington-Miller\", \"order\": 43}, {\"cast_id\": 65, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acadb\", \"gender\": 1, \"id\": 1145098, \"name\": \"Lucy Briant\", \"order\": 44}, {\"cast_id\": 66, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acadf\", \"gender\": 2, \"id\": 33305, \"name\": \"Nathan Meister\", \"order\": 45}, {\"cast_id\": 67, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acae3\", \"gender\": 0, \"id\": 1207264, \"name\": \"Gerry Blair\", \"order\": 46}, {\"cast_id\": 68, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acae7\", \"gender\": 2, \"id\": 33311, \"name\": \"Matthew Chamberlain\", \"order\": 47}, {\"cast_id\": 69, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acaeb\", \"gender\": 0, \"id\": 1207265, \"name\": \"Paul Yates\", \"order\": 48}, {\"cast_id\": 70, \"character\": \"Op Center Duty Officer\", \"credit_id\": \"52fe48019251416c750acaef\", \"gender\": 0, \"id\": 1207266, \"name\": \"Wray Wilson\", \"order\": 49}, {\"cast_id\": 71, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acaf3\", \"gender\": 2, \"id\": 54492, \"name\": \"James Gaylyn\", \"order\": 50}, {\"cast_id\": 72, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acaf7\", \"gender\": 0, \"id\": 1207267, \"name\": \"Melvin Leno Clark III\", \"order\": 51}, {\"cast_id\": 73, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acafb\", \"gender\": 0, \"id\": 1207268, \"name\": \"Carvon Futrell\", \"order\": 52}, {\"cast_id\": 74, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acaff\", \"gender\": 0, \"id\": 1207269, \"name\": \"Brandon Jelkes\", \"order\": 53}, {\"cast_id\": 75, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb03\", \"gender\": 0, \"id\": 1207270, \"name\": \"Micah Moch\", \"order\": 54}, {\"cast_id\": 76, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb07\", \"gender\": 0, \"id\": 1207271, \"name\": \"Hanniyah Muhammad\", \"order\": 55}, {\"cast_id\": 77, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb0b\", \"gender\": 0, \"id\": 1207272, \"name\": \"Christopher Nolen\", \"order\": 56}, {\"cast_id\": 78, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb0f\", \"gender\": 0, \"id\": 1207273, \"name\": \"Christa Oliver\", \"order\": 57}, {\"cast_id\": 79, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb13\", \"gender\": 0, \"id\": 1207274, \"name\": \"April Marie Thomas\", \"order\": 58}, {\"cast_id\": 80, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb17\", \"gender\": 0, \"id\": 1207275, \"name\": \"Bravita A. Threatt\", \"order\": 59}, {\"cast_id\": 81, \"character\": \"Mining Chief (uncredited)\", \"credit_id\": \"52fe48019251416c750acb1b\", \"gender\": 0, \"id\": 1207276, \"name\": \"Colin Bleasdale\", \"order\": 60}, {\"cast_id\": 82, \"character\": \"Veteran Miner (uncredited)\", \"credit_id\": \"52fe48019251416c750acb1f\", \"gender\": 0, \"id\": 107969, \"name\": \"Mike Bodnar\", \"order\": 61}, {\"cast_id\": 83, \"character\": \"Richard (uncredited)\", \"credit_id\": \"52fe48019251416c750acb23\", \"gender\": 0, \"id\": 1207278, \"name\": \"Matt Clayton\", \"order\": 62}, {\"cast_id\": 84, \"character\": \"Nav\\'i (uncredited)\", \"credit_id\": \"52fe48019251416c750acb27\", \"gender\": 1, \"id\": 147898, \"name\": \"Nicole Dionne\", \"order\": 63}, {\"cast_id\": 85, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb2b\", \"gender\": 0, \"id\": 1207280, \"name\": \"Jamie Harrison\", \"order\": 64}, {\"cast_id\": 86, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb2f\", \"gender\": 0, \"id\": 1207281, \"name\": \"Allan Henry\", \"order\": 65}, {\"cast_id\": 87, \"character\": \"Ground Technician (uncredited)\", \"credit_id\": \"52fe48019251416c750acb33\", \"gender\": 2, \"id\": 1207282, \"name\": \"Anthony Ingruber\", \"order\": 66}, {\"cast_id\": 88, \"character\": \"Flight Crew Mechanic (uncredited)\", \"credit_id\": \"52fe48019251416c750acb37\", \"gender\": 0, \"id\": 1207283, \"name\": \"Ashley Jeffery\", \"order\": 67}, {\"cast_id\": 14, \"character\": \"Samson Pilot\", \"credit_id\": \"52fe48009251416c750ac9f9\", \"gender\": 0, \"id\": 98216, \"name\": \"Dean Knowsley\", \"order\": 68}, {\"cast_id\": 89, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb3b\", \"gender\": 0, \"id\": 1201399, \"name\": \"Joseph Mika-Hunt\", \"order\": 69}, {\"cast_id\": 90, \"character\": \"Banshee (uncredited)\", \"credit_id\": \"52fe48019251416c750acb3f\", \"gender\": 0, \"id\": 236696, \"name\": \"Terry Notary\", \"order\": 70}, {\"cast_id\": 91, \"character\": \"Soldier (uncredited)\", \"credit_id\": \"52fe48019251416c750acb43\", \"gender\": 0, \"id\": 1207287, \"name\": \"Kai Pantano\", \"order\": 71}, {\"cast_id\": 92, \"character\": \"Blast Technician (uncredited)\", \"credit_id\": \"52fe48019251416c750acb47\", \"gender\": 0, \"id\": 1207288, \"name\": \"Logan Pithyou\", \"order\": 72}, {\"cast_id\": 93, \"character\": \"Vindum Raah (uncredited)\", \"credit_id\": \"52fe48019251416c750acb4b\", \"gender\": 0, \"id\": 1207289, \"name\": \"Stuart Pollock\", \"order\": 73}, {\"cast_id\": 94, \"character\": \"Hero (uncredited)\", \"credit_id\": \"52fe48019251416c750acb4f\", \"gender\": 0, \"id\": 584868, \"name\": \"Raja\", \"order\": 74}, {\"cast_id\": 95, \"character\": \"Ops Centreworker (uncredited)\", \"credit_id\": \"52fe48019251416c750acb53\", \"gender\": 0, \"id\": 1207290, \"name\": \"Gareth Ruck\", \"order\": 75}, {\"cast_id\": 96, \"character\": \"Engineer (uncredited)\", \"credit_id\": \"52fe48019251416c750acb57\", \"gender\": 0, \"id\": 1062463, \"name\": \"Rhian Sheehan\", \"order\": 76}, {\"cast_id\": 97, \"character\": \"Col. Quaritch\\'s Mech Suit (uncredited)\", \"credit_id\": \"52fe48019251416c750acb5b\", \"gender\": 0, \"id\": 60656, \"name\": \"T. J. Storm\", \"order\": 77}, {\"cast_id\": 98, \"character\": \"Female Marine (uncredited)\", \"credit_id\": \"52fe48019251416c750acb5f\", \"gender\": 0, \"id\": 1207291, \"name\": \"Jodie Taylor\", \"order\": 78}, {\"cast_id\": 99, \"character\": \"Ikran Clan Leader (uncredited)\", \"credit_id\": \"52fe48019251416c750acb63\", \"gender\": 1, \"id\": 1186027, \"name\": \"Alicia Vela-Bailey\", \"order\": 79}, {\"cast_id\": 100, \"character\": \"Geologist (uncredited)\", \"credit_id\": \"52fe48019251416c750acb67\", \"gender\": 0, \"id\": 1207292, \"name\": \"Richard Whiteside\", \"order\": 80}, {\"cast_id\": 101, \"character\": \"Na\\'vi (uncredited)\", \"credit_id\": \"52fe48019251416c750acb6b\", \"gender\": 0, \"id\": 103259, \"name\": \"Nikie Zambo\", \"order\": 81}, {\"cast_id\": 102, \"character\": \"Ambient Room Tech / Troupe\", \"credit_id\": \"52fe48019251416c750acb6f\", \"gender\": 1, \"id\": 42286, \"name\": \"Julene Renee\", \"order\": 82}]'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_df['cast'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5647eaf2",
   "metadata": {},
   "source": [
    "- {\"cast_id\": 242, \"character\": \"Jake Sully\", \"credit_id\": \"5602a8a7c3a3685532001c9a\", \"gender\": 2, \"id\": 65731, \"name\": \"Sam Worthington\", \"order\": 0}\n",
    "\n",
    "On extracting the very first record under 'Cast' feature we could infer that it basically consists of all the cast details including *Cast ID*, *Gender*, *Name of the character played by actor/actress*, *Real life-name of actor/actress*, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c4d2df8b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"credit_id\": \"52fe48009251416c750aca23\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1721, \"job\": \"Editor\", \"name\": \"Stephen E. Rivkin\"}, {\"credit_id\": \"539c47ecc3a36810e3001f87\", \"department\": \"Art\", \"gender\": 2, \"id\": 496, \"job\": \"Production Design\", \"name\": \"Rick Carter\"}, {\"credit_id\": \"54491c89c3a3680fb4001cf7\", \"department\": \"Sound\", \"gender\": 0, \"id\": 900, \"job\": \"Sound Designer\", \"name\": \"Christopher Boyes\"}, {\"credit_id\": \"54491cb70e0a267480001bd0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 900, \"job\": \"Supervising Sound Editor\", \"name\": \"Christopher Boyes\"}, {\"credit_id\": \"539c4a4cc3a36810c9002101\", \"department\": \"Production\", \"gender\": 1, \"id\": 1262, \"job\": \"Casting\", \"name\": \"Mali Finn\"}, {\"credit_id\": \"5544ee3b925141499f0008fc\", \"department\": \"Sound\", \"gender\": 2, \"id\": 1729, \"job\": \"Original Music Composer\", \"name\": \"James Horner\"}, {\"credit_id\": \"52fe48009251416c750ac9c3\", \"department\": \"Directing\", \"gender\": 2, \"id\": 2710, \"job\": \"Director\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750ac9d9\", \"department\": \"Writing\", \"gender\": 2, \"id\": 2710, \"job\": \"Writer\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca17\", \"department\": \"Editing\", \"gender\": 2, \"id\": 2710, \"job\": \"Editor\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca29\", \"department\": \"Production\", \"gender\": 2, \"id\": 2710, \"job\": \"Producer\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca3f\", \"department\": \"Writing\", \"gender\": 2, \"id\": 2710, \"job\": \"Screenplay\", \"name\": \"James Cameron\"}, {\"credit_id\": \"539c4987c3a36810ba0021a4\", \"department\": \"Art\", \"gender\": 2, \"id\": 7236, \"job\": \"Art Direction\", \"name\": \"Andrew Menzies\"}, {\"credit_id\": \"549598c3c3a3686ae9004383\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 6690, \"job\": \"Visual Effects Producer\", \"name\": \"Jill Brooks\"}, {\"credit_id\": \"52fe48009251416c750aca4b\", \"department\": \"Production\", \"gender\": 1, \"id\": 6347, \"job\": \"Casting\", \"name\": \"Margery Simkin\"}, {\"credit_id\": \"570b6f419251417da70032fe\", \"department\": \"Art\", \"gender\": 2, \"id\": 6878, \"job\": \"Supervising Art Director\", \"name\": \"Kevin Ishioka\"}, {\"credit_id\": \"5495a0fac3a3686ae9004468\", \"department\": \"Sound\", \"gender\": 0, \"id\": 6883, \"job\": \"Music Editor\", \"name\": \"Dick Bernstein\"}, {\"credit_id\": \"54959706c3a3686af3003e81\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8159, \"job\": \"Sound Effects Editor\", \"name\": \"Shannon Mills\"}, {\"credit_id\": \"54491d58c3a3680fb1001ccb\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8160, \"job\": \"Foley\", \"name\": \"Dennie Thorpe\"}, {\"credit_id\": \"54491d6cc3a3680fa5001b2c\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8163, \"job\": \"Foley\", \"name\": \"Jana Vance\"}, {\"credit_id\": \"52fe48009251416c750aca57\", \"department\": \"Costume & Make-Up\", \"gender\": 1, \"id\": 8527, \"job\": \"Costume Design\", \"name\": \"Deborah Lynn Scott\"}, {\"credit_id\": \"52fe48009251416c750aca2f\", \"department\": \"Production\", \"gender\": 2, \"id\": 8529, \"job\": \"Producer\", \"name\": \"Jon Landau\"}, {\"credit_id\": \"539c4937c3a36810ba002194\", \"department\": \"Art\", \"gender\": 0, \"id\": 9618, \"job\": \"Art Direction\", \"name\": \"Sean Haworth\"}, {\"credit_id\": \"539c49b6c3a36810c10020e6\", \"department\": \"Art\", \"gender\": 1, \"id\": 12653, \"job\": \"Set Decoration\", \"name\": \"Kim Sinclair\"}, {\"credit_id\": \"570b6f2f9251413a0e00020d\", \"department\": \"Art\", \"gender\": 1, \"id\": 12653, \"job\": \"Supervising Art Director\", \"name\": \"Kim Sinclair\"}, {\"credit_id\": \"54491a6c0e0a26748c001b19\", \"department\": \"Art\", \"gender\": 2, \"id\": 14350, \"job\": \"Set Designer\", \"name\": \"Richard F. Mays\"}, {\"credit_id\": \"56928cf4c3a3684cff0025c4\", \"department\": \"Production\", \"gender\": 1, \"id\": 20294, \"job\": \"Executive Producer\", \"name\": \"Laeta Kalogridis\"}, {\"credit_id\": \"52fe48009251416c750aca51\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 17675, \"job\": \"Costume Design\", \"name\": \"Mayes C. Rubeo\"}, {\"credit_id\": \"52fe48009251416c750aca11\", \"department\": \"Camera\", \"gender\": 2, \"id\": 18265, \"job\": \"Director of Photography\", \"name\": \"Mauro Fiore\"}, {\"credit_id\": \"5449194d0e0a26748f001b39\", \"department\": \"Art\", \"gender\": 0, \"id\": 42281, \"job\": \"Set Designer\", \"name\": \"Scott Herbertson\"}, {\"credit_id\": \"52fe48009251416c750aca05\", \"department\": \"Crew\", \"gender\": 0, \"id\": 42288, \"job\": \"Stunts\", \"name\": \"Woody Schultz\"}, {\"credit_id\": \"5592aefb92514152de0010f5\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 29067, \"job\": \"Makeup Artist\", \"name\": \"Linda DeVetta\"}, {\"credit_id\": \"5592afa492514152de00112c\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 29067, \"job\": \"Hairstylist\", \"name\": \"Linda DeVetta\"}, {\"credit_id\": \"54959ed592514130fc002e5d\", \"department\": \"Camera\", \"gender\": 2, \"id\": 33302, \"job\": \"Camera Operator\", \"name\": \"Richard Bluck\"}, {\"credit_id\": \"539c4891c3a36810ba002147\", \"department\": \"Art\", \"gender\": 2, \"id\": 33303, \"job\": \"Art Direction\", \"name\": \"Simon Bright\"}, {\"credit_id\": \"54959c069251417a81001f3a\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 113145, \"job\": \"Visual Effects Supervisor\", \"name\": \"Richard Martin\"}, {\"credit_id\": \"54959a0dc3a3680ff5002c8d\", \"department\": \"Crew\", \"gender\": 2, \"id\": 58188, \"job\": \"Visual Effects Editor\", \"name\": \"Steve R. Moore\"}, {\"credit_id\": \"52fe48009251416c750aca1d\", \"department\": \"Editing\", \"gender\": 2, \"id\": 58871, \"job\": \"Editor\", \"name\": \"John Refoua\"}, {\"credit_id\": \"54491a4dc3a3680fc30018ca\", \"department\": \"Art\", \"gender\": 0, \"id\": 92359, \"job\": \"Set Designer\", \"name\": \"Karl J. Martin\"}, {\"credit_id\": \"52fe48009251416c750aca35\", \"department\": \"Camera\", \"gender\": 1, \"id\": 72201, \"job\": \"Director of Photography\", \"name\": \"Chiling Lin\"}, {\"credit_id\": \"52fe48009251416c750ac9ff\", \"department\": \"Crew\", \"gender\": 0, \"id\": 89714, \"job\": \"Stunts\", \"name\": \"Ilram Choi\"}, {\"credit_id\": \"54959c529251416e2b004394\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 93214, \"job\": \"Visual Effects Supervisor\", \"name\": \"Steven Quale\"}, {\"credit_id\": \"54491edf0e0a267489001c37\", \"department\": \"Crew\", \"gender\": 1, \"id\": 122607, \"job\": \"Dialect Coach\", \"name\": \"Carla Meyer\"}, {\"credit_id\": \"539c485bc3a368653d001a3a\", \"department\": \"Art\", \"gender\": 2, \"id\": 132585, \"job\": \"Art Direction\", \"name\": \"Nick Bassett\"}, {\"credit_id\": \"539c4903c3a368653d001a74\", \"department\": \"Art\", \"gender\": 0, \"id\": 132596, \"job\": \"Art Direction\", \"name\": \"Jill Cormack\"}, {\"credit_id\": \"539c4967c3a368653d001a94\", \"department\": \"Art\", \"gender\": 0, \"id\": 132604, \"job\": \"Art Direction\", \"name\": \"Andy McLaren\"}, {\"credit_id\": \"52fe48009251416c750aca45\", \"department\": \"Crew\", \"gender\": 0, \"id\": 236696, \"job\": \"Motion Capture Artist\", \"name\": \"Terry Notary\"}, {\"credit_id\": \"54959e02c3a3680fc60027d2\", \"department\": \"Crew\", \"gender\": 2, \"id\": 956198, \"job\": \"Stunt Coordinator\", \"name\": \"Garrett Warren\"}, {\"credit_id\": \"54959ca3c3a3686ae300438c\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 957874, \"job\": \"Visual Effects Supervisor\", \"name\": \"Jonathan Rothbart\"}, {\"credit_id\": \"570b6f519251412c74001b2f\", \"department\": \"Art\", \"gender\": 0, \"id\": 957889, \"job\": \"Supervising Art Director\", \"name\": \"Stefan Dechant\"}, {\"credit_id\": \"570b6f62c3a3680b77007460\", \"department\": \"Art\", \"gender\": 2, \"id\": 959555, \"job\": \"Supervising Art Director\", \"name\": \"Todd Cherniawsky\"}, {\"credit_id\": \"539c4a3ac3a36810da0021cc\", \"department\": \"Production\", \"gender\": 0, \"id\": 1016177, \"job\": \"Casting\", \"name\": \"Miranda Rivers\"}, {\"credit_id\": \"539c482cc3a36810c1002062\", \"department\": \"Art\", \"gender\": 0, \"id\": 1032536, \"job\": \"Production Design\", \"name\": \"Robert Stromberg\"}, {\"credit_id\": \"539c4b65c3a36810c9002125\", \"department\": \"Costume & Make-Up\", \"gender\": 2, \"id\": 1071680, \"job\": \"Costume Design\", \"name\": \"John Harding\"}, {\"credit_id\": \"54959e6692514130fc002e4e\", \"department\": \"Camera\", \"gender\": 0, \"id\": 1177364, \"job\": \"Steadicam Operator\", \"name\": \"Roberto De Angelis\"}, {\"credit_id\": \"539c49f1c3a368653d001aac\", \"department\": \"Costume & Make-Up\", \"gender\": 2, \"id\": 1202850, \"job\": \"Makeup Department Head\", \"name\": \"Mike Smithson\"}, {\"credit_id\": \"5495999ec3a3686ae100460c\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1204668, \"job\": \"Visual Effects Producer\", \"name\": \"Alain Lalanne\"}, {\"credit_id\": \"54959cdfc3a3681153002729\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1206410, \"job\": \"Visual Effects Supervisor\", \"name\": \"Lucas Salton\"}, {\"credit_id\": \"549596239251417a81001eae\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1234266, \"job\": \"Post Production Supervisor\", \"name\": \"Janace Tashjian\"}, {\"credit_id\": \"54959c859251416e1e003efe\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1271932, \"job\": \"Visual Effects Supervisor\", \"name\": \"Stephen Rosenbaum\"}, {\"credit_id\": \"5592af28c3a368775a00105f\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1310064, \"job\": \"Makeup Artist\", \"name\": \"Frankie Karena\"}, {\"credit_id\": \"539c4adfc3a36810e300203b\", \"department\": \"Costume & Make-Up\", \"gender\": 1, \"id\": 1319844, \"job\": \"Costume Supervisor\", \"name\": \"Lisa Lovaas\"}, {\"credit_id\": \"54959b579251416e2b004371\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1327028, \"job\": \"Visual Effects Supervisor\", \"name\": \"Jonathan Fawkner\"}, {\"credit_id\": \"539c48a7c3a36810b5001fa7\", \"department\": \"Art\", \"gender\": 0, \"id\": 1330561, \"job\": \"Art Direction\", \"name\": \"Robert Bavin\"}, {\"credit_id\": \"539c4a71c3a36810da0021e0\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330567, \"job\": \"Costume Supervisor\", \"name\": \"Anthony Almaraz\"}, {\"credit_id\": \"539c4a8ac3a36810ba0021e4\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330570, \"job\": \"Costume Supervisor\", \"name\": \"Carolyn M. Fenton\"}, {\"credit_id\": \"539c4ab6c3a36810da0021f0\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330574, \"job\": \"Costume Supervisor\", \"name\": \"Beth Koenigsberg\"}, {\"credit_id\": \"54491ab70e0a267480001ba2\", \"department\": \"Art\", \"gender\": 0, \"id\": 1336191, \"job\": \"Set Designer\", \"name\": \"Sam Page\"}, {\"credit_id\": \"544919d9c3a3680fc30018bd\", \"department\": \"Art\", \"gender\": 0, \"id\": 1339441, \"job\": \"Set Designer\", \"name\": \"Tex Kadonaga\"}, {\"credit_id\": \"54491cf50e0a267483001b0c\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1352422, \"job\": \"Dialogue Editor\", \"name\": \"Kim Foscato\"}, {\"credit_id\": \"544919f40e0a26748c001b09\", \"department\": \"Art\", \"gender\": 0, \"id\": 1352962, \"job\": \"Set Designer\", \"name\": \"Tammy S. Lee\"}, {\"credit_id\": \"5495a115c3a3680ff5002d71\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1357070, \"job\": \"Transportation Coordinator\", \"name\": \"Denny Caira\"}, {\"credit_id\": \"5495a12f92514130fc002e94\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1357071, \"job\": \"Transportation Coordinator\", \"name\": \"James Waitkus\"}, {\"credit_id\": \"5495976fc3a36811530026b0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1360103, \"job\": \"Supervising Sound Editor\", \"name\": \"Addison Teague\"}, {\"credit_id\": \"54491837c3a3680fb1001c5a\", \"department\": \"Art\", \"gender\": 2, \"id\": 1376887, \"job\": \"Set Designer\", \"name\": \"C. Scott Baker\"}, {\"credit_id\": \"54491878c3a3680fb4001c9d\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376888, \"job\": \"Set Designer\", \"name\": \"Luke Caska\"}, {\"credit_id\": \"544918dac3a3680fa5001ae0\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376889, \"job\": \"Set Designer\", \"name\": \"David Chow\"}, {\"credit_id\": \"544919110e0a267486001b68\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376890, \"job\": \"Set Designer\", \"name\": \"Jonathan Dyer\"}, {\"credit_id\": \"54491967c3a3680faa001b5e\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376891, \"job\": \"Set Designer\", \"name\": \"Joseph Hiura\"}, {\"credit_id\": \"54491997c3a3680fb1001c8a\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376892, \"job\": \"Art Department Coordinator\", \"name\": \"Rebecca Jellie\"}, {\"credit_id\": \"544919ba0e0a26748f001b42\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376893, \"job\": \"Set Designer\", \"name\": \"Robert Andrew Johnson\"}, {\"credit_id\": \"54491b1dc3a3680faa001b8c\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376895, \"job\": \"Assistant Art Director\", \"name\": \"Mike Stassi\"}, {\"credit_id\": \"54491b79c3a3680fbb001826\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376897, \"job\": \"Construction Coordinator\", \"name\": \"John Villarino\"}, {\"credit_id\": \"54491baec3a3680fb4001ce6\", \"department\": \"Art\", \"gender\": 2, \"id\": 1376898, \"job\": \"Assistant Art Director\", \"name\": \"Jeffrey Wisniewski\"}, {\"credit_id\": \"54491d2fc3a3680fb4001d07\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1376899, \"job\": \"Dialogue Editor\", \"name\": \"Cheryl Nardi\"}, {\"credit_id\": \"54491d86c3a3680fa5001b2f\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1376901, \"job\": \"Dialogue Editor\", \"name\": \"Marshall Winn\"}, {\"credit_id\": \"54491d9dc3a3680faa001bb0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1376902, \"job\": \"Supervising Sound Editor\", \"name\": \"Gwendolyn Yates Whittle\"}, {\"credit_id\": \"54491dc10e0a267486001bce\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1376903, \"job\": \"Sound Re-Recording Mixer\", \"name\": \"William Stein\"}, {\"credit_id\": \"54491f500e0a26747c001c07\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1376909, \"job\": \"Choreographer\", \"name\": \"Lula Washington\"}, {\"credit_id\": \"549599239251412c4e002a2e\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1391692, \"job\": \"Visual Effects Producer\", \"name\": \"Chris Del Conte\"}, {\"credit_id\": \"54959d54c3a36831b8001d9a\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 1391695, \"job\": \"Visual Effects Supervisor\", \"name\": \"R. Christopher White\"}, {\"credit_id\": \"54959bdf9251412c4e002a66\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1394070, \"job\": \"Visual Effects Supervisor\", \"name\": \"Dan Lemmon\"}, {\"credit_id\": \"5495971d92514132ed002922\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1394129, \"job\": \"Sound Effects Editor\", \"name\": \"Tim Nielsen\"}, {\"credit_id\": \"5592b25792514152cc0011aa\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1394286, \"job\": \"CG Supervisor\", \"name\": \"Michael Mulholland\"}, {\"credit_id\": \"54959a329251416e2b004355\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1394750, \"job\": \"Visual Effects Editor\", \"name\": \"Thomas Nittmann\"}, {\"credit_id\": \"54959d6dc3a3686ae9004401\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1394755, \"job\": \"Visual Effects Supervisor\", \"name\": \"Edson Williams\"}, {\"credit_id\": \"5495a08fc3a3686ae300441c\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1394953, \"job\": \"Digital Intermediate\", \"name\": \"Christine Carr\"}, {\"credit_id\": \"55402d659251413d6d000249\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1395269, \"job\": \"Visual Effects Supervisor\", \"name\": \"John Bruno\"}, {\"credit_id\": \"54959e7b9251416e1e003f3e\", \"department\": \"Camera\", \"gender\": 0, \"id\": 1398970, \"job\": \"Steadicam Operator\", \"name\": \"David Emmerichs\"}, {\"credit_id\": \"54959734c3a3686ae10045e0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1400906, \"job\": \"Sound Effects Editor\", \"name\": \"Christopher Scarabosio\"}, {\"credit_id\": \"549595dd92514130fc002d79\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401784, \"job\": \"Production Supervisor\", \"name\": \"Jennifer Teves\"}, {\"credit_id\": \"549596009251413af70028cc\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401785, \"job\": \"Production Manager\", \"name\": \"Brigitte Yorke\"}, {\"credit_id\": \"549596e892514130fc002d99\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1401786, \"job\": \"Sound Effects Editor\", \"name\": \"Ken Fischer\"}, {\"credit_id\": \"549598229251412c4e002a1c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401787, \"job\": \"Special Effects Coordinator\", \"name\": \"Iain Hutton\"}, {\"credit_id\": \"549598349251416e2b00432b\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401788, \"job\": \"Special Effects Coordinator\", \"name\": \"Steve Ingram\"}, {\"credit_id\": \"54959905c3a3686ae3004324\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401789, \"job\": \"Visual Effects Producer\", \"name\": \"Joyce Cox\"}, {\"credit_id\": \"5495994b92514132ed002951\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401790, \"job\": \"Visual Effects Producer\", \"name\": \"Jenny Foster\"}, {\"credit_id\": \"549599cbc3a3686ae1004613\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401791, \"job\": \"Visual Effects Editor\", \"name\": \"Christopher Marino\"}, {\"credit_id\": \"549599f2c3a3686ae100461e\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401792, \"job\": \"Visual Effects Editor\", \"name\": \"Jim Milton\"}, {\"credit_id\": \"54959a51c3a3686af3003eb5\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401793, \"job\": \"Visual Effects Producer\", \"name\": \"Cyndi Ochs\"}, {\"credit_id\": \"54959a7cc3a36811530026f4\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401794, \"job\": \"Visual Effects Editor\", \"name\": \"Lucas Putnam\"}, {\"credit_id\": \"54959b91c3a3680ff5002cb4\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401795, \"job\": \"Visual Effects Supervisor\", \"name\": \"Anthony \\'Max\\' Ivins\"}, {\"credit_id\": \"54959bb69251412c4e002a5f\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401796, \"job\": \"Visual Effects Supervisor\", \"name\": \"John Knoll\"}, {\"credit_id\": \"54959cbbc3a3686ae3004391\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 1401799, \"job\": \"Visual Effects Supervisor\", \"name\": \"Eric Saindon\"}, {\"credit_id\": \"54959d06c3a3686ae90043f6\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401800, \"job\": \"Visual Effects Supervisor\", \"name\": \"Wayne Stables\"}, {\"credit_id\": \"54959d259251416e1e003f11\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401801, \"job\": \"Visual Effects Supervisor\", \"name\": \"David Stinnett\"}, {\"credit_id\": \"54959db49251413af7002975\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401803, \"job\": \"Visual Effects Supervisor\", \"name\": \"Guy Williams\"}, {\"credit_id\": \"54959de4c3a3681153002750\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401804, \"job\": \"Stunt Coordinator\", \"name\": \"Stuart Thorp\"}, {\"credit_id\": \"54959ef2c3a3680fc60027f2\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1401805, \"job\": \"Best Boy Electric\", \"name\": \"Giles Coburn\"}, {\"credit_id\": \"54959f07c3a3680fc60027f9\", \"department\": \"Camera\", \"gender\": 2, \"id\": 1401806, \"job\": \"Still Photographer\", \"name\": \"Mark Fellman\"}, {\"credit_id\": \"54959f47c3a3681153002774\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1401807, \"job\": \"Lighting Technician\", \"name\": \"Scott Sprague\"}, {\"credit_id\": \"54959f8cc3a36831b8001df2\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401808, \"job\": \"Animation Director\", \"name\": \"Jeremy Hollobon\"}, {\"credit_id\": \"54959fa0c3a36831b8001dfb\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401809, \"job\": \"Animation Director\", \"name\": \"Orlando Meunier\"}, {\"credit_id\": \"54959fb6c3a3686af3003f54\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401810, \"job\": \"Animation Director\", \"name\": \"Taisuke Tanimura\"}, {\"credit_id\": \"54959fd2c3a36831b8001e02\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1401812, \"job\": \"Set Costumer\", \"name\": \"Lilia Mishel Acevedo\"}, {\"credit_id\": \"54959ff9c3a3686ae300440c\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1401814, \"job\": \"Set Costumer\", \"name\": \"Alejandro M. Hernandez\"}, {\"credit_id\": \"5495a0ddc3a3686ae10046fe\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1401815, \"job\": \"Digital Intermediate\", \"name\": \"Marvin Hall\"}, {\"credit_id\": \"5495a1f7c3a3686ae3004443\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401816, \"job\": \"Publicist\", \"name\": \"Judy Alley\"}, {\"credit_id\": \"5592b29fc3a36869d100002f\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1418381, \"job\": \"CG Supervisor\", \"name\": \"Mike Perry\"}, {\"credit_id\": \"5592b23a9251415df8001081\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1426854, \"job\": \"CG Supervisor\", \"name\": \"Andrew Morley\"}, {\"credit_id\": \"55491e1192514104c40002d8\", \"department\": \"Art\", \"gender\": 0, \"id\": 1438901, \"job\": \"Conceptual Design\", \"name\": \"Seth Engstrom\"}, {\"credit_id\": \"5525d5809251417276002b06\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1447362, \"job\": \"Visual Effects Art Director\", \"name\": \"Eric Oliver\"}, {\"credit_id\": \"554427ca925141586500312a\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1447503, \"job\": \"Modeling\", \"name\": \"Matsune Suzuki\"}, {\"credit_id\": \"551906889251415aab001c88\", \"department\": \"Art\", \"gender\": 0, \"id\": 1447524, \"job\": \"Art Department Manager\", \"name\": \"Paul Tobin\"}, {\"credit_id\": \"5592af8492514152cc0010de\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1452643, \"job\": \"Hairstylist\", \"name\": \"Roxane Griffin\"}, {\"credit_id\": \"553d3c109251415852001318\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1453938, \"job\": \"Lighting Artist\", \"name\": \"Arun Ram-Mohan\"}, {\"credit_id\": \"5592af4692514152d5001355\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1457305, \"job\": \"Makeup Artist\", \"name\": \"Georgia Lockhart-Adams\"}, {\"credit_id\": \"5592b2eac3a36877470012a5\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1466035, \"job\": \"CG Supervisor\", \"name\": \"Thrain Shadbolt\"}, {\"credit_id\": \"5592b032c3a36877450015f1\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483220, \"job\": \"CG Supervisor\", \"name\": \"Brad Alexander\"}, {\"credit_id\": \"5592b05592514152d80012f6\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483221, \"job\": \"CG Supervisor\", \"name\": \"Shadi Almassizadeh\"}, {\"credit_id\": \"5592b090c3a36877570010b5\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483222, \"job\": \"CG Supervisor\", \"name\": \"Simon Clutterbuck\"}, {\"credit_id\": \"5592b0dbc3a368774b00112c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483223, \"job\": \"CG Supervisor\", \"name\": \"Graeme Demmocks\"}, {\"credit_id\": \"5592b0fe92514152db0010c1\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483224, \"job\": \"CG Supervisor\", \"name\": \"Adrian Fernandes\"}, {\"credit_id\": \"5592b11f9251415df8001059\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483225, \"job\": \"CG Supervisor\", \"name\": \"Mitch Gates\"}, {\"credit_id\": \"5592b15dc3a3687745001645\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483226, \"job\": \"CG Supervisor\", \"name\": \"Jerry Kung\"}, {\"credit_id\": \"5592b18e925141645a0004ae\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483227, \"job\": \"CG Supervisor\", \"name\": \"Andy Lomas\"}, {\"credit_id\": \"5592b1bfc3a368775d0010e7\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483228, \"job\": \"CG Supervisor\", \"name\": \"Sebastian Marino\"}, {\"credit_id\": \"5592b2049251415df8001078\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483229, \"job\": \"CG Supervisor\", \"name\": \"Matthias Menz\"}, {\"credit_id\": \"5592b27b92514152d800136a\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483230, \"job\": \"CG Supervisor\", \"name\": \"Sergei Nevshupov\"}, {\"credit_id\": \"5592b2c3c3a36869e800003c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483231, \"job\": \"CG Supervisor\", \"name\": \"Philippe Rebours\"}, {\"credit_id\": \"5592b317c3a36877470012af\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483232, \"job\": \"CG Supervisor\", \"name\": \"Michael Takarangi\"}, {\"credit_id\": \"5592b345c3a36877470012bb\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483233, \"job\": \"CG Supervisor\", \"name\": \"David Weitzberg\"}, {\"credit_id\": \"5592b37cc3a368775100113b\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483234, \"job\": \"CG Supervisor\", \"name\": \"Ben White\"}, {\"credit_id\": \"573c8e2f9251413f5d000094\", \"department\": \"Crew\", \"gender\": 1, \"id\": 1621932, \"job\": \"Stunts\", \"name\": \"Min Windle\"}]'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_df['crew'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b39fa0e2",
   "metadata": {},
   "source": [
    "- {\"credit_id\": \"52fe48009251416c750aca23\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1721, \"job\": \"Editor\", \"name\": \"Stephen E. Rivkin\"}\n",
    "\n",
    "On extracting the very first record under 'Crew' feature we could infer that it basically consists of all the crew details including *Name of the person working as a crew member*, *His/Her department (editing/sound/director)*, *Gender*, *job of that person (say, if that person is working as Production Designer/ Sound Designer/ Director of Photography/ Art Director/ Visual Effects Producer etc.)*, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7e14c47",
   "metadata": {},
   "source": [
    "Since we will be building a *content-based recommender system*, we need to select specific features that would be helpful in deriving *tags* which would form the base of our recommendation model. Based on these tags, the recommendations will be shown for the users. Listing the features below that we would be using to formulate tags for our model:\n",
    "- genre\n",
    "- id\n",
    "- keywords\n",
    "- title\n",
    "- overview\n",
    "- cast\n",
    "- crew"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "95ac174d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'budget', 'popularity', 'revenue', 'runtime', 'vote_average', 'vote_count'\n",
    "# these are numeric features & they won't do any help in deriving tags for the movies\n",
    "\n",
    "# suppose let us consider an example accounting for budget\n",
    "# say there is an user named 'Mr. XYZ' who liked one of the movies with a high budget as $300 million\n",
    "# now it is not guaranteed that this user might like other movies ranging around the same budget\n",
    "# same is the issue with other numerical features as stated above\n",
    "# so we would be dropping these features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "207d668d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "en    4510\n",
       "fr      70\n",
       "es      32\n",
       "zh      27\n",
       "de      27\n",
       "hi      19\n",
       "ja      16\n",
       "it      14\n",
       "ko      12\n",
       "cn      12\n",
       "ru      11\n",
       "pt       9\n",
       "da       7\n",
       "sv       5\n",
       "nl       4\n",
       "fa       4\n",
       "th       3\n",
       "he       3\n",
       "ta       2\n",
       "cs       2\n",
       "ro       2\n",
       "id       2\n",
       "ar       2\n",
       "vi       1\n",
       "sl       1\n",
       "ps       1\n",
       "no       1\n",
       "ky       1\n",
       "hu       1\n",
       "pl       1\n",
       "af       1\n",
       "nb       1\n",
       "tr       1\n",
       "is       1\n",
       "xx       1\n",
       "te       1\n",
       "el       1\n",
       "Name: original_language, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_df['original_language'].value_counts()\n",
    "\n",
    "# let's analyze the table below\n",
    "# roughly 95% of our dataset is having movies listed in 'English' as their official language\n",
    "# since it's a very unbalanced data & it won't help us deriving any relevant tags\n",
    "# we could consider dropping this feature from our dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "10183e87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'homepage', 'production_companies', 'production_countries'\n",
    "# these features do not have any relevance in deriving tags for the movies\n",
    "# considering our daily life, we never go around recommending movies to our friends\n",
    "# based on the respective production housenames or the country wherein these productions are situated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "48a8421f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# original_title feature might contain name of a movie in native/regional language\n",
    "# so we prefer keeping 'title' feature in our dataframe instead of 'originakl_title' feature\n",
    "# since the 'title' feature consists of the english names of all the movies in our dataset\n",
    "\n",
    "# 'overview' feature includes a short description regarding the storyline\n",
    "# in our daily lives we do recommend movies to our family/friends highlighting a brief storyline of a movie\n",
    "# we could derive some useful tags from the 'overview' feature which would account for helpful recommendation\n",
    "# since 'overview' feature is serving our purpose of deriving tags much better compared to 'tagline' feature\n",
    "# so we could prefer having 'overview' feature instead of 'tagline' feature in our dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ad96849f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'cast', 'crew' - we consider to include these features in our dataframe\n",
    "# comparing our daily lives, we do share our recommendation to our family/friends\n",
    "# based on actors/actresses, directors, etc.\n",
    "\n",
    "# let us assume an example: say there is an user named 'Mrs. ABC'\n",
    "# She does like movies directed by some person named 'Tom Holland' based on here reviews\n",
    "# Now there is a strong correlation wherein she would like other movies as well that have been directed by this person"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f621cc9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_df = merged_df[['movie_id', 'title', 'genres', 'keywords', 'overview', 'cast', 'crew']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1b2044f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>[{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...</td>\n",
       "      <td>[{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>[{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>[{\"cast_id\": 5, \"character\": \"John Carter\", \"c...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                              genres  \\\n",
       "0  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "1  [{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...   \n",
       "2  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "3  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...   \n",
       "4  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...   \n",
       "1  [{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...   \n",
       "2  [{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...   \n",
       "3  [{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...   \n",
       "4  [{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...   \n",
       "1  Captain Barbossa, long believed to be dead, ha...   \n",
       "2  A cryptic message from Bond’s past sends him o...   \n",
       "3  Following the death of District Attorney Harve...   \n",
       "4  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "4  [{\"cast_id\": 5, \"character\": \"John Carter\", \"c...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ecb31d19",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "movie_id    0\n",
       "title       0\n",
       "genres      0\n",
       "keywords    0\n",
       "overview    3\n",
       "cast        0\n",
       "crew        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking for missing values\n",
    "feat_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d29e871f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# as the count for missing values is very less we can consider dropping these null values\n",
    "feat_df = feat_df.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4b613092",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking for duplicated values\n",
    "feat_df.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "90c7106e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# our motive is to derive a final dataframe with 3 features: 'movie_id', 'title', 'tags'\n",
    "# so for 'tags' feature we need to merge all the 'genre', 'overview', 'keywords', 'cast', 'crew' features together\n",
    "# so that all the relevants words could be encapsulated under a single feature\n",
    "# this would help us correlate b/w the tags of two and more moview and finally share the recommendations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c48f2a0",
   "metadata": {},
   "source": [
    "## PRE-PROCESSING STEP:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "087a18f2",
   "metadata": {},
   "source": [
    "#### I. TRANSFORMING THE \"GENRES\" FEATURE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "082c1ba6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"name\": \"Fantasy\"}, {\"id\": 878, \"name\": \"Science Fiction\"}]'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df['genres'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8d53a7a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# analyzing above results we could infer that for a single record:\n",
    "# the 'genres' feature consists of data bundled as multiple dictionaries under a list\n",
    "\n",
    "# [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"name\": \"Fantasy\"}, {\"id\": 878, \"name\": \"Science Fiction\"}]\n",
    "# for above data, we need to retrieve results as a list of tags (as shared below):\n",
    "    # - ['Action', 'Adventure', 'Fantasy', 'Science Fiction']\n",
    "\n",
    "# however there is a small problem: if we see below the type returned is not a list but a string\n",
    "# as a result it is very diffcult to split this string object and retrieve the required data\n",
    "type(feat_df['genres'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "4a34123d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# literal_eval() function under ast module helps transform a string object into a list type\n",
    "# so at first, we are importing the ast module\n",
    "\n",
    "import ast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "eef3a971",
   "metadata": {},
   "outputs": [],
   "source": [
    "# let us create a function(func_1) where we would be splitting the bundled dictionaries\n",
    "# so basically the object is a (string of list) and we need to convert it into a list using ast.literal_eval() function\n",
    "# finally we would be appending the names from these bundled dictionaries as tags to an empty list to formulate a list of tags\n",
    "\n",
    "def func_1(obj):\n",
    "    list = []\n",
    "    for i in ast.literal_eval(obj):\n",
    "        list.append(i['name'])\n",
    "    return list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "1b53d74e",
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_df['genres'] = feat_df['genres'].apply(func_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "978d9a24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, Science Fiction]</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, Science Fiction]</td>\n",
       "      <td>[{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>[{\"cast_id\": 5, \"character\": \"John Carter\", \"c...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                          genres  \\\n",
       "0  [Action, Adventure, Fantasy, Science Fiction]   \n",
       "1                   [Adventure, Fantasy, Action]   \n",
       "2                     [Action, Adventure, Crime]   \n",
       "3               [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, Science Fiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...   \n",
       "1  [{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...   \n",
       "2  [{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...   \n",
       "3  [{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...   \n",
       "4  [{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...   \n",
       "1  Captain Barbossa, long believed to be dead, ha...   \n",
       "2  A cryptic message from Bond’s past sends him o...   \n",
       "3  Following the death of District Attorney Harve...   \n",
       "4  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "4  [{\"cast_id\": 5, \"character\": \"John Carter\", \"c...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3f40692",
   "metadata": {},
   "source": [
    "#### II. TRANSFORMING THE \"KEYWORDS\" FEATURE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "59a9373c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\": 2964, \"name\": \"future\"}, {\"id\": 3386, \"name\": \"space war\"}, {\"id\": 3388, \"name\": \"space colony\"}, {\"id\": 3679, \"name\": \"society\"}, {\"id\": 3801, \"name\": \"space travel\"}, {\"id\": 9685, \"name\": \"futuristic\"}, {\"id\": 9840, \"name\": \"romance\"}, {\"id\": 9882, \"name\": \"space\"}, {\"id\": 9951, \"name\": \"alien\"}, {\"id\": 10148, \"name\": \"tribe\"}, {\"id\": 10158, \"name\": \"alien planet\"}, {\"id\": 10987, \"name\": \"cgi\"}, {\"id\": 11399, \"name\": \"marine\"}, {\"id\": 13065, \"name\": \"soldier\"}, {\"id\": 14643, \"name\": \"battle\"}, {\"id\": 14720, \"name\": \"love affair\"}, {\"id\": 165431, \"name\": \"anti war\"}, {\"id\": 193554, \"name\": \"power relations\"}, {\"id\": 206690, \"name\": \"mind and soul\"}, {\"id\": 209714, \"name\": \"3d\"}]'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df['keywords'][0]\n",
    "\n",
    "# 'keyword' feature also had same format data bundled as a dictionary under a list\n",
    "# so we can apply the same func_1() function to extract the tags out of this (str) object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6c48755f",
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_df['keywords'] = feat_df['keywords'].apply(func_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "3a283073",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, Science Fiction]</td>\n",
       "      <td>[culture clash, future, space war, space colon...</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drug abuse, exotic island, east india ...</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, based on novel, secret agent, sequel, mi...</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dc comics, crime fighter, terrorist, secret i...</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, Science Fiction]</td>\n",
       "      <td>[based on novel, mars, medallion, space travel...</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>[{\"cast_id\": 5, \"character\": \"John Carter\", \"c...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                          genres  \\\n",
       "0  [Action, Adventure, Fantasy, Science Fiction]   \n",
       "1                   [Adventure, Fantasy, Action]   \n",
       "2                     [Action, Adventure, Crime]   \n",
       "3               [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, Science Fiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [culture clash, future, space war, space colon...   \n",
       "1  [ocean, drug abuse, exotic island, east india ...   \n",
       "2  [spy, based on novel, secret agent, sequel, mi...   \n",
       "3  [dc comics, crime fighter, terrorist, secret i...   \n",
       "4  [based on novel, mars, medallion, space travel...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...   \n",
       "1  Captain Barbossa, long believed to be dead, ha...   \n",
       "2  A cryptic message from Bond’s past sends him o...   \n",
       "3  Following the death of District Attorney Harve...   \n",
       "4  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "4  [{\"cast_id\": 5, \"character\": \"John Carter\", \"c...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ae7c9d43",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now let us analyze the 'cast' & 'crew' features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99698d7b",
   "metadata": {},
   "source": [
    "#### III. TRANSFORMING THE \"CAST\" FEATURE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f9e48b8a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"credit_id\": \"5602a8a7c3a3685532001c9a\", \"gender\": 2, \"id\": 65731, \"name\": \"Sam Worthington\", \"order\": 0}, {\"cast_id\": 3, \"character\": \"Neytiri\", \"credit_id\": \"52fe48009251416c750ac9cb\", \"gender\": 1, \"id\": 8691, \"name\": \"Zoe Saldana\", \"order\": 1}, {\"cast_id\": 25, \"character\": \"Dr. Grace Augustine\", \"credit_id\": \"52fe48009251416c750aca39\", \"gender\": 1, \"id\": 10205, \"name\": \"Sigourney Weaver\", \"order\": 2}, {\"cast_id\": 4, \"character\": \"Col. Quaritch\", \"credit_id\": \"52fe48009251416c750ac9cf\", \"gender\": 2, \"id\": 32747, \"name\": \"Stephen Lang\", \"order\": 3}, {\"cast_id\": 5, \"character\": \"Trudy Chacon\", \"credit_id\": \"52fe48009251416c750ac9d3\", \"gender\": 1, \"id\": 17647, \"name\": \"Michelle Rodriguez\", \"order\": 4}, {\"cast_id\": 8, \"character\": \"Selfridge\", \"credit_id\": \"52fe48009251416c750ac9e1\", \"gender\": 2, \"id\": 1771, \"name\": \"Giovanni Ribisi\", \"order\": 5}, {\"cast_id\": 7, \"character\": \"Norm Spellman\", \"credit_id\": \"52fe48009251416c750ac9dd\", \"gender\": 2, \"id\": 59231, \"name\": \"Joel David Moore\", \"order\": 6}, {\"cast_id\": 9, \"character\": \"Moat\", \"credit_id\": \"52fe48009251416c750ac9e5\", \"gender\": 1, \"id\": 30485, \"name\": \"CCH Pounder\", \"order\": 7}, {\"cast_id\": 11, \"character\": \"Eytukan\", \"credit_id\": \"52fe48009251416c750ac9ed\", \"gender\": 2, \"id\": 15853, \"name\": \"Wes Studi\", \"order\": 8}, {\"cast_id\": 10, \"character\": \"Tsu\\'Tey\", \"credit_id\": \"52fe48009251416c750ac9e9\", \"gender\": 2, \"id\": 10964, \"name\": \"Laz Alonso\", \"order\": 9}, {\"cast_id\": 12, \"character\": \"Dr. Max Patel\", \"credit_id\": \"52fe48009251416c750ac9f1\", \"gender\": 2, \"id\": 95697, \"name\": \"Dileep Rao\", \"order\": 10}, {\"cast_id\": 13, \"character\": \"Lyle Wainfleet\", \"credit_id\": \"52fe48009251416c750ac9f5\", \"gender\": 2, \"id\": 98215, \"name\": \"Matt Gerald\", \"order\": 11}, {\"cast_id\": 32, \"character\": \"Private Fike\", \"credit_id\": \"52fe48009251416c750aca5b\", \"gender\": 2, \"id\": 154153, \"name\": \"Sean Anthony Moran\", \"order\": 12}, {\"cast_id\": 33, \"character\": \"Cryo Vault Med Tech\", \"credit_id\": \"52fe48009251416c750aca5f\", \"gender\": 2, \"id\": 397312, \"name\": \"Jason Whyte\", \"order\": 13}, {\"cast_id\": 34, \"character\": \"Venture Star Crew Chief\", \"credit_id\": \"52fe48009251416c750aca63\", \"gender\": 2, \"id\": 42317, \"name\": \"Scott Lawrence\", \"order\": 14}, {\"cast_id\": 35, \"character\": \"Lock Up Trooper\", \"credit_id\": \"52fe48009251416c750aca67\", \"gender\": 2, \"id\": 986734, \"name\": \"Kelly Kilgour\", \"order\": 15}, {\"cast_id\": 36, \"character\": \"Shuttle Pilot\", \"credit_id\": \"52fe48009251416c750aca6b\", \"gender\": 0, \"id\": 1207227, \"name\": \"James Patrick Pitt\", \"order\": 16}, {\"cast_id\": 37, \"character\": \"Shuttle Co-Pilot\", \"credit_id\": \"52fe48009251416c750aca6f\", \"gender\": 0, \"id\": 1180936, \"name\": \"Sean Patrick Murphy\", \"order\": 17}, {\"cast_id\": 38, \"character\": \"Shuttle Crew Chief\", \"credit_id\": \"52fe48009251416c750aca73\", \"gender\": 2, \"id\": 1019578, \"name\": \"Peter Dillon\", \"order\": 18}, {\"cast_id\": 39, \"character\": \"Tractor Operator / Troupe\", \"credit_id\": \"52fe48009251416c750aca77\", \"gender\": 0, \"id\": 91443, \"name\": \"Kevin Dorman\", \"order\": 19}, {\"cast_id\": 40, \"character\": \"Dragon Gunship Pilot\", \"credit_id\": \"52fe48009251416c750aca7b\", \"gender\": 2, \"id\": 173391, \"name\": \"Kelson Henderson\", \"order\": 20}, {\"cast_id\": 41, \"character\": \"Dragon Gunship Gunner\", \"credit_id\": \"52fe48009251416c750aca7f\", \"gender\": 0, \"id\": 1207236, \"name\": \"David Van Horn\", \"order\": 21}, {\"cast_id\": 42, \"character\": \"Dragon Gunship Navigator\", \"credit_id\": \"52fe48009251416c750aca83\", \"gender\": 0, \"id\": 215913, \"name\": \"Jacob Tomuri\", \"order\": 22}, {\"cast_id\": 43, \"character\": \"Suit #1\", \"credit_id\": \"52fe48009251416c750aca87\", \"gender\": 0, \"id\": 143206, \"name\": \"Michael Blain-Rozgay\", \"order\": 23}, {\"cast_id\": 44, \"character\": \"Suit #2\", \"credit_id\": \"52fe48009251416c750aca8b\", \"gender\": 2, \"id\": 169676, \"name\": \"Jon Curry\", \"order\": 24}, {\"cast_id\": 46, \"character\": \"Ambient Room Tech\", \"credit_id\": \"52fe48009251416c750aca8f\", \"gender\": 0, \"id\": 1048610, \"name\": \"Luke Hawker\", \"order\": 25}, {\"cast_id\": 47, \"character\": \"Ambient Room Tech / Troupe\", \"credit_id\": \"52fe48009251416c750aca93\", \"gender\": 0, \"id\": 42288, \"name\": \"Woody Schultz\", \"order\": 26}, {\"cast_id\": 48, \"character\": \"Horse Clan Leader\", \"credit_id\": \"52fe48009251416c750aca97\", \"gender\": 2, \"id\": 68278, \"name\": \"Peter Mensah\", \"order\": 27}, {\"cast_id\": 49, \"character\": \"Link Room Tech\", \"credit_id\": \"52fe48009251416c750aca9b\", \"gender\": 0, \"id\": 1207247, \"name\": \"Sonia Yee\", \"order\": 28}, {\"cast_id\": 50, \"character\": \"Basketball Avatar / Troupe\", \"credit_id\": \"52fe48009251416c750aca9f\", \"gender\": 1, \"id\": 1207248, \"name\": \"Jahnel Curfman\", \"order\": 29}, {\"cast_id\": 51, \"character\": \"Basketball Avatar\", \"credit_id\": \"52fe48009251416c750acaa3\", \"gender\": 0, \"id\": 89714, \"name\": \"Ilram Choi\", \"order\": 30}, {\"cast_id\": 52, \"character\": \"Na\\'vi Child\", \"credit_id\": \"52fe48009251416c750acaa7\", \"gender\": 0, \"id\": 1207249, \"name\": \"Kyla Warren\", \"order\": 31}, {\"cast_id\": 53, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acaab\", \"gender\": 0, \"id\": 1207250, \"name\": \"Lisa Roumain\", \"order\": 32}, {\"cast_id\": 54, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acaaf\", \"gender\": 1, \"id\": 83105, \"name\": \"Debra Wilson\", \"order\": 33}, {\"cast_id\": 57, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acabb\", \"gender\": 0, \"id\": 1207253, \"name\": \"Chris Mala\", \"order\": 34}, {\"cast_id\": 55, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acab3\", \"gender\": 0, \"id\": 1207251, \"name\": \"Taylor Kibby\", \"order\": 35}, {\"cast_id\": 56, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acab7\", \"gender\": 0, \"id\": 1207252, \"name\": \"Jodie Landau\", \"order\": 36}, {\"cast_id\": 58, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acabf\", \"gender\": 0, \"id\": 1207254, \"name\": \"Julie Lamm\", \"order\": 37}, {\"cast_id\": 59, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acac3\", \"gender\": 0, \"id\": 1207257, \"name\": \"Cullen B. Madden\", \"order\": 38}, {\"cast_id\": 60, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acac7\", \"gender\": 0, \"id\": 1207259, \"name\": \"Joseph Brady Madden\", \"order\": 39}, {\"cast_id\": 61, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acacb\", \"gender\": 0, \"id\": 1207262, \"name\": \"Frankie Torres\", \"order\": 40}, {\"cast_id\": 62, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acacf\", \"gender\": 1, \"id\": 1158600, \"name\": \"Austin Wilson\", \"order\": 41}, {\"cast_id\": 63, \"character\": \"Troupe\", \"credit_id\": \"52fe48019251416c750acad3\", \"gender\": 1, \"id\": 983705, \"name\": \"Sara Wilson\", \"order\": 42}, {\"cast_id\": 64, \"character\": \"Troupe\", \"credit_id\": \"52fe48019251416c750acad7\", \"gender\": 0, \"id\": 1207263, \"name\": \"Tamica Washington-Miller\", \"order\": 43}, {\"cast_id\": 65, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acadb\", \"gender\": 1, \"id\": 1145098, \"name\": \"Lucy Briant\", \"order\": 44}, {\"cast_id\": 66, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acadf\", \"gender\": 2, \"id\": 33305, \"name\": \"Nathan Meister\", \"order\": 45}, {\"cast_id\": 67, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acae3\", \"gender\": 0, \"id\": 1207264, \"name\": \"Gerry Blair\", \"order\": 46}, {\"cast_id\": 68, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acae7\", \"gender\": 2, \"id\": 33311, \"name\": \"Matthew Chamberlain\", \"order\": 47}, {\"cast_id\": 69, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acaeb\", \"gender\": 0, \"id\": 1207265, \"name\": \"Paul Yates\", \"order\": 48}, {\"cast_id\": 70, \"character\": \"Op Center Duty Officer\", \"credit_id\": \"52fe48019251416c750acaef\", \"gender\": 0, \"id\": 1207266, \"name\": \"Wray Wilson\", \"order\": 49}, {\"cast_id\": 71, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acaf3\", \"gender\": 2, \"id\": 54492, \"name\": \"James Gaylyn\", \"order\": 50}, {\"cast_id\": 72, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acaf7\", \"gender\": 0, \"id\": 1207267, \"name\": \"Melvin Leno Clark III\", \"order\": 51}, {\"cast_id\": 73, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acafb\", \"gender\": 0, \"id\": 1207268, \"name\": \"Carvon Futrell\", \"order\": 52}, {\"cast_id\": 74, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acaff\", \"gender\": 0, \"id\": 1207269, \"name\": \"Brandon Jelkes\", \"order\": 53}, {\"cast_id\": 75, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb03\", \"gender\": 0, \"id\": 1207270, \"name\": \"Micah Moch\", \"order\": 54}, {\"cast_id\": 76, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb07\", \"gender\": 0, \"id\": 1207271, \"name\": \"Hanniyah Muhammad\", \"order\": 55}, {\"cast_id\": 77, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb0b\", \"gender\": 0, \"id\": 1207272, \"name\": \"Christopher Nolen\", \"order\": 56}, {\"cast_id\": 78, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb0f\", \"gender\": 0, \"id\": 1207273, \"name\": \"Christa Oliver\", \"order\": 57}, {\"cast_id\": 79, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb13\", \"gender\": 0, \"id\": 1207274, \"name\": \"April Marie Thomas\", \"order\": 58}, {\"cast_id\": 80, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb17\", \"gender\": 0, \"id\": 1207275, \"name\": \"Bravita A. Threatt\", \"order\": 59}, {\"cast_id\": 81, \"character\": \"Mining Chief (uncredited)\", \"credit_id\": \"52fe48019251416c750acb1b\", \"gender\": 0, \"id\": 1207276, \"name\": \"Colin Bleasdale\", \"order\": 60}, {\"cast_id\": 82, \"character\": \"Veteran Miner (uncredited)\", \"credit_id\": \"52fe48019251416c750acb1f\", \"gender\": 0, \"id\": 107969, \"name\": \"Mike Bodnar\", \"order\": 61}, {\"cast_id\": 83, \"character\": \"Richard (uncredited)\", \"credit_id\": \"52fe48019251416c750acb23\", \"gender\": 0, \"id\": 1207278, \"name\": \"Matt Clayton\", \"order\": 62}, {\"cast_id\": 84, \"character\": \"Nav\\'i (uncredited)\", \"credit_id\": \"52fe48019251416c750acb27\", \"gender\": 1, \"id\": 147898, \"name\": \"Nicole Dionne\", \"order\": 63}, {\"cast_id\": 85, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb2b\", \"gender\": 0, \"id\": 1207280, \"name\": \"Jamie Harrison\", \"order\": 64}, {\"cast_id\": 86, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb2f\", \"gender\": 0, \"id\": 1207281, \"name\": \"Allan Henry\", \"order\": 65}, {\"cast_id\": 87, \"character\": \"Ground Technician (uncredited)\", \"credit_id\": \"52fe48019251416c750acb33\", \"gender\": 2, \"id\": 1207282, \"name\": \"Anthony Ingruber\", \"order\": 66}, {\"cast_id\": 88, \"character\": \"Flight Crew Mechanic (uncredited)\", \"credit_id\": \"52fe48019251416c750acb37\", \"gender\": 0, \"id\": 1207283, \"name\": \"Ashley Jeffery\", \"order\": 67}, {\"cast_id\": 14, \"character\": \"Samson Pilot\", \"credit_id\": \"52fe48009251416c750ac9f9\", \"gender\": 0, \"id\": 98216, \"name\": \"Dean Knowsley\", \"order\": 68}, {\"cast_id\": 89, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb3b\", \"gender\": 0, \"id\": 1201399, \"name\": \"Joseph Mika-Hunt\", \"order\": 69}, {\"cast_id\": 90, \"character\": \"Banshee (uncredited)\", \"credit_id\": \"52fe48019251416c750acb3f\", \"gender\": 0, \"id\": 236696, \"name\": \"Terry Notary\", \"order\": 70}, {\"cast_id\": 91, \"character\": \"Soldier (uncredited)\", \"credit_id\": \"52fe48019251416c750acb43\", \"gender\": 0, \"id\": 1207287, \"name\": \"Kai Pantano\", \"order\": 71}, {\"cast_id\": 92, \"character\": \"Blast Technician (uncredited)\", \"credit_id\": \"52fe48019251416c750acb47\", \"gender\": 0, \"id\": 1207288, \"name\": \"Logan Pithyou\", \"order\": 72}, {\"cast_id\": 93, \"character\": \"Vindum Raah (uncredited)\", \"credit_id\": \"52fe48019251416c750acb4b\", \"gender\": 0, \"id\": 1207289, \"name\": \"Stuart Pollock\", \"order\": 73}, {\"cast_id\": 94, \"character\": \"Hero (uncredited)\", \"credit_id\": \"52fe48019251416c750acb4f\", \"gender\": 0, \"id\": 584868, \"name\": \"Raja\", \"order\": 74}, {\"cast_id\": 95, \"character\": \"Ops Centreworker (uncredited)\", \"credit_id\": \"52fe48019251416c750acb53\", \"gender\": 0, \"id\": 1207290, \"name\": \"Gareth Ruck\", \"order\": 75}, {\"cast_id\": 96, \"character\": \"Engineer (uncredited)\", \"credit_id\": \"52fe48019251416c750acb57\", \"gender\": 0, \"id\": 1062463, \"name\": \"Rhian Sheehan\", \"order\": 76}, {\"cast_id\": 97, \"character\": \"Col. Quaritch\\'s Mech Suit (uncredited)\", \"credit_id\": \"52fe48019251416c750acb5b\", \"gender\": 0, \"id\": 60656, \"name\": \"T. J. Storm\", \"order\": 77}, {\"cast_id\": 98, \"character\": \"Female Marine (uncredited)\", \"credit_id\": \"52fe48019251416c750acb5f\", \"gender\": 0, \"id\": 1207291, \"name\": \"Jodie Taylor\", \"order\": 78}, {\"cast_id\": 99, \"character\": \"Ikran Clan Leader (uncredited)\", \"credit_id\": \"52fe48019251416c750acb63\", \"gender\": 1, \"id\": 1186027, \"name\": \"Alicia Vela-Bailey\", \"order\": 79}, {\"cast_id\": 100, \"character\": \"Geologist (uncredited)\", \"credit_id\": \"52fe48019251416c750acb67\", \"gender\": 0, \"id\": 1207292, \"name\": \"Richard Whiteside\", \"order\": 80}, {\"cast_id\": 101, \"character\": \"Na\\'vi (uncredited)\", \"credit_id\": \"52fe48019251416c750acb6b\", \"gender\": 0, \"id\": 103259, \"name\": \"Nikie Zambo\", \"order\": 81}, {\"cast_id\": 102, \"character\": \"Ambient Room Tech / Troupe\", \"credit_id\": \"52fe48019251416c750acb6f\", \"gender\": 1, \"id\": 42286, \"name\": \"Julene Renee\", \"order\": 82}]'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df['cast'][0]\n",
    "\n",
    "# we could see a long list of actors/actresses under 'cast' feature for a single movie\n",
    "# now let us consider an assumption here (say, we want only the top 4 actors/actresses for a particular movie)\n",
    "# first dictionary - {\"cast_id\": 242, \"character\": \"Jake Sully\", \"credit_id\": \"5602a8a7c3a3685532001c9a\", \"gender\": 2, \"id\": 65731, \"name\": \"Sam Worthington\", \"order\": 0}\n",
    "# second dictionary - {\"cast_id\": 3, \"character\": \"Neytiri\", \"credit_id\": \"52fe48009251416c750ac9cb\", \"gender\": 1, \"id\": 8691, \"name\": \"Zoe Saldana\", \"order\": 1}\n",
    "# third dictionary - {\"cast_id\": 25, \"character\": \"Dr. Grace Augustine\", \"credit_id\": \"52fe48009251416c750aca39\", \"gender\": 1, \"id\": 10205, \"name\": \"Sigourney Weaver\", \"order\": 2\n",
    "# fourth dictionary - {\"cast_id\": 4, \"character\": \"Col. Quaritch\", \"credit_id\": \"52fe48009251416c750ac9cf\", \"gender\": 2, \"id\": 32747, \"name\": \"Stephen Lang\", \"order\": 3}\n",
    "\n",
    "# now here we want only the names of the top 4 actors/actresses\n",
    "# so we only need the 'name' attribute from each of these dictionary objects"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1260773",
   "metadata": {},
   "source": [
    "Let us create a function(func_2) where we would be splitting the bundled dictionaries for 'cast' feature. So basically the object is a (string of list) and we need to convert it into a list using ast.literal_eval() function. Finally we would be appending the names of these actors/actresses as tags to an empty list to formulate a list of actor/actress names.\n",
    "\n",
    "## FOR SIMPLIFICATION:\n",
    "- \"character\": \"Jake Sully\" ---- \"name\": \"Sam Worthington\"\n",
    "- This indicates Sam Worthington had played the character named 'Jake Sully' in movie titled 'Avatar'\n",
    "- So here we need the real life name (i.e. Sam Worthington) in our list\n",
    "- This will help recommend relevant movies associated with this particular actor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "62dca897",
   "metadata": {},
   "outputs": [],
   "source": [
    "def func_2(obj):\n",
    "    list = []\n",
    "    counter=0\n",
    "    for i in ast.literal_eval(obj):\n",
    "        if counter != 4:\n",
    "            list.append(i['name'])\n",
    "            counter = counter + 1\n",
    "        else:\n",
    "            break\n",
    "    return list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "889e77e0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0       [Sam Worthington, Zoe Saldana, Sigourney Weave...\n",
      "1       [Johnny Depp, Orlando Bloom, Keira Knightley, ...\n",
      "2       [Daniel Craig, Christoph Waltz, Léa Seydoux, R...\n",
      "3       [Christian Bale, Michael Caine, Gary Oldman, A...\n",
      "4       [Taylor Kitsch, Lynn Collins, Samantha Morton,...\n",
      "                              ...                        \n",
      "4804    [Carlos Gallardo, Jaime de Hoyos, Peter Marqua...\n",
      "4805    [Edward Burns, Kerry Bishé, Marsha Dietlein, C...\n",
      "4806    [Eric Mabius, Kristin Booth, Crystal Lowe, Geo...\n",
      "4807    [Daniel Henney, Eliza Coupe, Bill Paxton, Alan...\n",
      "4808    [Drew Barrymore, Brian Herzlinger, Corey Feldm...\n",
      "Name: cast, Length: 4806, dtype: object\n"
     ]
    }
   ],
   "source": [
    "feat_df['cast'] = feat_df['cast'].apply(func_2)\n",
    "print(feat_df['cast'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "78501bdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we have successfully transformed top 4 actors/actresses for each movie in the form of a list\n",
    "# so we are getting the lead actors/actress for each movie which we can use in our 'tag' feature\n",
    "# to correlate with other movies relevant to that actor/actress and help recommend same back to the users "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "d8883a79",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, Science Fiction]</td>\n",
       "      <td>[culture clash, future, space war, space colon...</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>[Sam Worthington, Zoe Saldana, Sigourney Weave...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drug abuse, exotic island, east india ...</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>[Johnny Depp, Orlando Bloom, Keira Knightley, ...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, based on novel, secret agent, sequel, mi...</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>[Daniel Craig, Christoph Waltz, Léa Seydoux, R...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dc comics, crime fighter, terrorist, secret i...</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>[Christian Bale, Michael Caine, Gary Oldman, A...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, Science Fiction]</td>\n",
       "      <td>[based on novel, mars, medallion, space travel...</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>[Taylor Kitsch, Lynn Collins, Samantha Morton,...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                          genres  \\\n",
       "0  [Action, Adventure, Fantasy, Science Fiction]   \n",
       "1                   [Adventure, Fantasy, Action]   \n",
       "2                     [Action, Adventure, Crime]   \n",
       "3               [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, Science Fiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [culture clash, future, space war, space colon...   \n",
       "1  [ocean, drug abuse, exotic island, east india ...   \n",
       "2  [spy, based on novel, secret agent, sequel, mi...   \n",
       "3  [dc comics, crime fighter, terrorist, secret i...   \n",
       "4  [based on novel, mars, medallion, space travel...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...   \n",
       "1  Captain Barbossa, long believed to be dead, ha...   \n",
       "2  A cryptic message from Bond’s past sends him o...   \n",
       "3  Following the death of District Attorney Harve...   \n",
       "4  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [Sam Worthington, Zoe Saldana, Sigourney Weave...   \n",
       "1  [Johnny Depp, Orlando Bloom, Keira Knightley, ...   \n",
       "2  [Daniel Craig, Christoph Waltz, Léa Seydoux, R...   \n",
       "3  [Christian Bale, Michael Caine, Gary Oldman, A...   \n",
       "4  [Taylor Kitsch, Lynn Collins, Samantha Morton,...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c59ec7e",
   "metadata": {},
   "source": [
    "#### IV. TRANSFORMING THE \"CREW\" FEATURE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "ad64da80",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"credit_id\": \"52fe48009251416c750aca23\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1721, \"job\": \"Editor\", \"name\": \"Stephen E. Rivkin\"}, {\"credit_id\": \"539c47ecc3a36810e3001f87\", \"department\": \"Art\", \"gender\": 2, \"id\": 496, \"job\": \"Production Design\", \"name\": \"Rick Carter\"}, {\"credit_id\": \"54491c89c3a3680fb4001cf7\", \"department\": \"Sound\", \"gender\": 0, \"id\": 900, \"job\": \"Sound Designer\", \"name\": \"Christopher Boyes\"}, {\"credit_id\": \"54491cb70e0a267480001bd0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 900, \"job\": \"Supervising Sound Editor\", \"name\": \"Christopher Boyes\"}, {\"credit_id\": \"539c4a4cc3a36810c9002101\", \"department\": \"Production\", \"gender\": 1, \"id\": 1262, \"job\": \"Casting\", \"name\": \"Mali Finn\"}, {\"credit_id\": \"5544ee3b925141499f0008fc\", \"department\": \"Sound\", \"gender\": 2, \"id\": 1729, \"job\": \"Original Music Composer\", \"name\": \"James Horner\"}, {\"credit_id\": \"52fe48009251416c750ac9c3\", \"department\": \"Directing\", \"gender\": 2, \"id\": 2710, \"job\": \"Director\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750ac9d9\", \"department\": \"Writing\", \"gender\": 2, \"id\": 2710, \"job\": \"Writer\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca17\", \"department\": \"Editing\", \"gender\": 2, \"id\": 2710, \"job\": \"Editor\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca29\", \"department\": \"Production\", \"gender\": 2, \"id\": 2710, \"job\": \"Producer\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca3f\", \"department\": \"Writing\", \"gender\": 2, \"id\": 2710, \"job\": \"Screenplay\", \"name\": \"James Cameron\"}, {\"credit_id\": \"539c4987c3a36810ba0021a4\", \"department\": \"Art\", \"gender\": 2, \"id\": 7236, \"job\": \"Art Direction\", \"name\": \"Andrew Menzies\"}, {\"credit_id\": \"549598c3c3a3686ae9004383\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 6690, \"job\": \"Visual Effects Producer\", \"name\": \"Jill Brooks\"}, {\"credit_id\": \"52fe48009251416c750aca4b\", \"department\": \"Production\", \"gender\": 1, \"id\": 6347, \"job\": \"Casting\", \"name\": \"Margery Simkin\"}, {\"credit_id\": \"570b6f419251417da70032fe\", \"department\": \"Art\", \"gender\": 2, \"id\": 6878, \"job\": \"Supervising Art Director\", \"name\": \"Kevin Ishioka\"}, {\"credit_id\": \"5495a0fac3a3686ae9004468\", \"department\": \"Sound\", \"gender\": 0, \"id\": 6883, \"job\": \"Music Editor\", \"name\": \"Dick Bernstein\"}, {\"credit_id\": \"54959706c3a3686af3003e81\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8159, \"job\": \"Sound Effects Editor\", \"name\": \"Shannon Mills\"}, {\"credit_id\": \"54491d58c3a3680fb1001ccb\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8160, \"job\": \"Foley\", \"name\": \"Dennie Thorpe\"}, {\"credit_id\": \"54491d6cc3a3680fa5001b2c\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8163, \"job\": \"Foley\", \"name\": \"Jana Vance\"}, {\"credit_id\": \"52fe48009251416c750aca57\", \"department\": \"Costume & Make-Up\", \"gender\": 1, \"id\": 8527, \"job\": \"Costume Design\", \"name\": \"Deborah Lynn Scott\"}, {\"credit_id\": \"52fe48009251416c750aca2f\", \"department\": \"Production\", \"gender\": 2, \"id\": 8529, \"job\": \"Producer\", \"name\": \"Jon Landau\"}, {\"credit_id\": \"539c4937c3a36810ba002194\", \"department\": \"Art\", \"gender\": 0, \"id\": 9618, \"job\": \"Art Direction\", \"name\": \"Sean Haworth\"}, {\"credit_id\": \"539c49b6c3a36810c10020e6\", \"department\": \"Art\", \"gender\": 1, \"id\": 12653, \"job\": \"Set Decoration\", \"name\": \"Kim Sinclair\"}, {\"credit_id\": \"570b6f2f9251413a0e00020d\", \"department\": \"Art\", \"gender\": 1, \"id\": 12653, \"job\": \"Supervising Art Director\", \"name\": \"Kim Sinclair\"}, {\"credit_id\": \"54491a6c0e0a26748c001b19\", \"department\": \"Art\", \"gender\": 2, \"id\": 14350, \"job\": \"Set Designer\", \"name\": \"Richard F. Mays\"}, {\"credit_id\": \"56928cf4c3a3684cff0025c4\", \"department\": \"Production\", \"gender\": 1, \"id\": 20294, \"job\": \"Executive Producer\", \"name\": \"Laeta Kalogridis\"}, {\"credit_id\": \"52fe48009251416c750aca51\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 17675, \"job\": \"Costume Design\", \"name\": \"Mayes C. Rubeo\"}, {\"credit_id\": \"52fe48009251416c750aca11\", \"department\": \"Camera\", \"gender\": 2, \"id\": 18265, \"job\": \"Director of Photography\", \"name\": \"Mauro Fiore\"}, {\"credit_id\": \"5449194d0e0a26748f001b39\", \"department\": \"Art\", \"gender\": 0, \"id\": 42281, \"job\": \"Set Designer\", \"name\": \"Scott Herbertson\"}, {\"credit_id\": \"52fe48009251416c750aca05\", \"department\": \"Crew\", \"gender\": 0, \"id\": 42288, \"job\": \"Stunts\", \"name\": \"Woody Schultz\"}, {\"credit_id\": \"5592aefb92514152de0010f5\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 29067, \"job\": \"Makeup Artist\", \"name\": \"Linda DeVetta\"}, {\"credit_id\": \"5592afa492514152de00112c\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 29067, \"job\": \"Hairstylist\", \"name\": \"Linda DeVetta\"}, {\"credit_id\": \"54959ed592514130fc002e5d\", \"department\": \"Camera\", \"gender\": 2, \"id\": 33302, \"job\": \"Camera Operator\", \"name\": \"Richard Bluck\"}, {\"credit_id\": \"539c4891c3a36810ba002147\", \"department\": \"Art\", \"gender\": 2, \"id\": 33303, \"job\": \"Art Direction\", \"name\": \"Simon Bright\"}, {\"credit_id\": \"54959c069251417a81001f3a\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 113145, \"job\": \"Visual Effects Supervisor\", \"name\": \"Richard Martin\"}, {\"credit_id\": \"54959a0dc3a3680ff5002c8d\", \"department\": \"Crew\", \"gender\": 2, \"id\": 58188, \"job\": \"Visual Effects Editor\", \"name\": \"Steve R. Moore\"}, {\"credit_id\": \"52fe48009251416c750aca1d\", \"department\": \"Editing\", \"gender\": 2, \"id\": 58871, \"job\": \"Editor\", \"name\": \"John Refoua\"}, {\"credit_id\": \"54491a4dc3a3680fc30018ca\", \"department\": \"Art\", \"gender\": 0, \"id\": 92359, \"job\": \"Set Designer\", \"name\": \"Karl J. Martin\"}, {\"credit_id\": \"52fe48009251416c750aca35\", \"department\": \"Camera\", \"gender\": 1, \"id\": 72201, \"job\": \"Director of Photography\", \"name\": \"Chiling Lin\"}, {\"credit_id\": \"52fe48009251416c750ac9ff\", \"department\": \"Crew\", \"gender\": 0, \"id\": 89714, \"job\": \"Stunts\", \"name\": \"Ilram Choi\"}, {\"credit_id\": \"54959c529251416e2b004394\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 93214, \"job\": \"Visual Effects Supervisor\", \"name\": \"Steven Quale\"}, {\"credit_id\": \"54491edf0e0a267489001c37\", \"department\": \"Crew\", \"gender\": 1, \"id\": 122607, \"job\": \"Dialect Coach\", \"name\": \"Carla Meyer\"}, {\"credit_id\": \"539c485bc3a368653d001a3a\", \"department\": \"Art\", \"gender\": 2, \"id\": 132585, \"job\": \"Art Direction\", \"name\": \"Nick Bassett\"}, {\"credit_id\": \"539c4903c3a368653d001a74\", \"department\": \"Art\", \"gender\": 0, \"id\": 132596, \"job\": \"Art Direction\", \"name\": \"Jill Cormack\"}, {\"credit_id\": \"539c4967c3a368653d001a94\", \"department\": \"Art\", \"gender\": 0, \"id\": 132604, \"job\": \"Art Direction\", \"name\": \"Andy McLaren\"}, {\"credit_id\": \"52fe48009251416c750aca45\", \"department\": \"Crew\", \"gender\": 0, \"id\": 236696, \"job\": \"Motion Capture Artist\", \"name\": \"Terry Notary\"}, {\"credit_id\": \"54959e02c3a3680fc60027d2\", \"department\": \"Crew\", \"gender\": 2, \"id\": 956198, \"job\": \"Stunt Coordinator\", \"name\": \"Garrett Warren\"}, {\"credit_id\": \"54959ca3c3a3686ae300438c\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 957874, \"job\": \"Visual Effects Supervisor\", \"name\": \"Jonathan Rothbart\"}, {\"credit_id\": \"570b6f519251412c74001b2f\", \"department\": \"Art\", \"gender\": 0, \"id\": 957889, \"job\": \"Supervising Art Director\", \"name\": \"Stefan Dechant\"}, {\"credit_id\": \"570b6f62c3a3680b77007460\", \"department\": \"Art\", \"gender\": 2, \"id\": 959555, \"job\": \"Supervising Art Director\", \"name\": \"Todd Cherniawsky\"}, {\"credit_id\": \"539c4a3ac3a36810da0021cc\", \"department\": \"Production\", \"gender\": 0, \"id\": 1016177, \"job\": \"Casting\", \"name\": \"Miranda Rivers\"}, {\"credit_id\": \"539c482cc3a36810c1002062\", \"department\": \"Art\", \"gender\": 0, \"id\": 1032536, \"job\": \"Production Design\", \"name\": \"Robert Stromberg\"}, {\"credit_id\": \"539c4b65c3a36810c9002125\", \"department\": \"Costume & Make-Up\", \"gender\": 2, \"id\": 1071680, \"job\": \"Costume Design\", \"name\": \"John Harding\"}, {\"credit_id\": \"54959e6692514130fc002e4e\", \"department\": \"Camera\", \"gender\": 0, \"id\": 1177364, \"job\": \"Steadicam Operator\", \"name\": \"Roberto De Angelis\"}, {\"credit_id\": \"539c49f1c3a368653d001aac\", \"department\": \"Costume & Make-Up\", \"gender\": 2, \"id\": 1202850, \"job\": \"Makeup Department Head\", \"name\": \"Mike Smithson\"}, {\"credit_id\": \"5495999ec3a3686ae100460c\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1204668, \"job\": \"Visual Effects Producer\", \"name\": \"Alain Lalanne\"}, {\"credit_id\": \"54959cdfc3a3681153002729\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1206410, \"job\": \"Visual Effects Supervisor\", \"name\": \"Lucas Salton\"}, {\"credit_id\": \"549596239251417a81001eae\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1234266, \"job\": \"Post Production Supervisor\", \"name\": \"Janace Tashjian\"}, {\"credit_id\": \"54959c859251416e1e003efe\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1271932, \"job\": \"Visual Effects Supervisor\", \"name\": \"Stephen Rosenbaum\"}, {\"credit_id\": \"5592af28c3a368775a00105f\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1310064, \"job\": \"Makeup Artist\", \"name\": \"Frankie Karena\"}, {\"credit_id\": \"539c4adfc3a36810e300203b\", \"department\": \"Costume & Make-Up\", \"gender\": 1, \"id\": 1319844, \"job\": \"Costume Supervisor\", \"name\": \"Lisa Lovaas\"}, {\"credit_id\": \"54959b579251416e2b004371\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1327028, \"job\": \"Visual Effects Supervisor\", \"name\": \"Jonathan Fawkner\"}, {\"credit_id\": \"539c48a7c3a36810b5001fa7\", \"department\": \"Art\", \"gender\": 0, \"id\": 1330561, \"job\": \"Art Direction\", \"name\": \"Robert Bavin\"}, {\"credit_id\": \"539c4a71c3a36810da0021e0\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330567, \"job\": \"Costume Supervisor\", \"name\": \"Anthony Almaraz\"}, {\"credit_id\": \"539c4a8ac3a36810ba0021e4\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330570, \"job\": \"Costume Supervisor\", \"name\": \"Carolyn M. Fenton\"}, {\"credit_id\": \"539c4ab6c3a36810da0021f0\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330574, \"job\": \"Costume Supervisor\", \"name\": \"Beth Koenigsberg\"}, {\"credit_id\": \"54491ab70e0a267480001ba2\", \"department\": \"Art\", \"gender\": 0, \"id\": 1336191, \"job\": \"Set Designer\", \"name\": \"Sam Page\"}, {\"credit_id\": \"544919d9c3a3680fc30018bd\", \"department\": \"Art\", \"gender\": 0, \"id\": 1339441, \"job\": \"Set Designer\", \"name\": \"Tex Kadonaga\"}, {\"credit_id\": \"54491cf50e0a267483001b0c\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1352422, \"job\": \"Dialogue Editor\", \"name\": \"Kim Foscato\"}, {\"credit_id\": \"544919f40e0a26748c001b09\", \"department\": \"Art\", \"gender\": 0, \"id\": 1352962, \"job\": \"Set Designer\", \"name\": \"Tammy S. Lee\"}, {\"credit_id\": \"5495a115c3a3680ff5002d71\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1357070, \"job\": \"Transportation Coordinator\", \"name\": \"Denny Caira\"}, {\"credit_id\": \"5495a12f92514130fc002e94\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1357071, \"job\": \"Transportation Coordinator\", \"name\": \"James Waitkus\"}, {\"credit_id\": \"5495976fc3a36811530026b0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1360103, \"job\": \"Supervising Sound Editor\", \"name\": \"Addison Teague\"}, {\"credit_id\": \"54491837c3a3680fb1001c5a\", \"department\": \"Art\", \"gender\": 2, \"id\": 1376887, \"job\": \"Set Designer\", \"name\": \"C. Scott Baker\"}, {\"credit_id\": \"54491878c3a3680fb4001c9d\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376888, \"job\": \"Set Designer\", \"name\": \"Luke Caska\"}, {\"credit_id\": \"544918dac3a3680fa5001ae0\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376889, \"job\": \"Set Designer\", \"name\": \"David Chow\"}, {\"credit_id\": \"544919110e0a267486001b68\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376890, \"job\": \"Set Designer\", \"name\": \"Jonathan Dyer\"}, {\"credit_id\": \"54491967c3a3680faa001b5e\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376891, \"job\": \"Set Designer\", \"name\": \"Joseph Hiura\"}, {\"credit_id\": \"54491997c3a3680fb1001c8a\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376892, \"job\": \"Art Department Coordinator\", \"name\": \"Rebecca Jellie\"}, {\"credit_id\": \"544919ba0e0a26748f001b42\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376893, \"job\": \"Set Designer\", \"name\": \"Robert Andrew Johnson\"}, {\"credit_id\": \"54491b1dc3a3680faa001b8c\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376895, \"job\": \"Assistant Art Director\", \"name\": \"Mike Stassi\"}, {\"credit_id\": \"54491b79c3a3680fbb001826\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376897, \"job\": \"Construction Coordinator\", \"name\": \"John Villarino\"}, {\"credit_id\": \"54491baec3a3680fb4001ce6\", \"department\": \"Art\", \"gender\": 2, \"id\": 1376898, \"job\": \"Assistant Art Director\", \"name\": \"Jeffrey Wisniewski\"}, {\"credit_id\": \"54491d2fc3a3680fb4001d07\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1376899, \"job\": \"Dialogue Editor\", \"name\": \"Cheryl Nardi\"}, {\"credit_id\": \"54491d86c3a3680fa5001b2f\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1376901, \"job\": \"Dialogue Editor\", \"name\": \"Marshall Winn\"}, {\"credit_id\": \"54491d9dc3a3680faa001bb0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1376902, \"job\": \"Supervising Sound Editor\", \"name\": \"Gwendolyn Yates Whittle\"}, {\"credit_id\": \"54491dc10e0a267486001bce\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1376903, \"job\": \"Sound Re-Recording Mixer\", \"name\": \"William Stein\"}, {\"credit_id\": \"54491f500e0a26747c001c07\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1376909, \"job\": \"Choreographer\", \"name\": \"Lula Washington\"}, {\"credit_id\": \"549599239251412c4e002a2e\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1391692, \"job\": \"Visual Effects Producer\", \"name\": \"Chris Del Conte\"}, {\"credit_id\": \"54959d54c3a36831b8001d9a\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 1391695, \"job\": \"Visual Effects Supervisor\", \"name\": \"R. Christopher White\"}, {\"credit_id\": \"54959bdf9251412c4e002a66\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1394070, \"job\": \"Visual Effects Supervisor\", \"name\": \"Dan Lemmon\"}, {\"credit_id\": \"5495971d92514132ed002922\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1394129, \"job\": \"Sound Effects Editor\", \"name\": \"Tim Nielsen\"}, {\"credit_id\": \"5592b25792514152cc0011aa\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1394286, \"job\": \"CG Supervisor\", \"name\": \"Michael Mulholland\"}, {\"credit_id\": \"54959a329251416e2b004355\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1394750, \"job\": \"Visual Effects Editor\", \"name\": \"Thomas Nittmann\"}, {\"credit_id\": \"54959d6dc3a3686ae9004401\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1394755, \"job\": \"Visual Effects Supervisor\", \"name\": \"Edson Williams\"}, {\"credit_id\": \"5495a08fc3a3686ae300441c\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1394953, \"job\": \"Digital Intermediate\", \"name\": \"Christine Carr\"}, {\"credit_id\": \"55402d659251413d6d000249\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1395269, \"job\": \"Visual Effects Supervisor\", \"name\": \"John Bruno\"}, {\"credit_id\": \"54959e7b9251416e1e003f3e\", \"department\": \"Camera\", \"gender\": 0, \"id\": 1398970, \"job\": \"Steadicam Operator\", \"name\": \"David Emmerichs\"}, {\"credit_id\": \"54959734c3a3686ae10045e0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1400906, \"job\": \"Sound Effects Editor\", \"name\": \"Christopher Scarabosio\"}, {\"credit_id\": \"549595dd92514130fc002d79\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401784, \"job\": \"Production Supervisor\", \"name\": \"Jennifer Teves\"}, {\"credit_id\": \"549596009251413af70028cc\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401785, \"job\": \"Production Manager\", \"name\": \"Brigitte Yorke\"}, {\"credit_id\": \"549596e892514130fc002d99\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1401786, \"job\": \"Sound Effects Editor\", \"name\": \"Ken Fischer\"}, {\"credit_id\": \"549598229251412c4e002a1c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401787, \"job\": \"Special Effects Coordinator\", \"name\": \"Iain Hutton\"}, {\"credit_id\": \"549598349251416e2b00432b\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401788, \"job\": \"Special Effects Coordinator\", \"name\": \"Steve Ingram\"}, {\"credit_id\": \"54959905c3a3686ae3004324\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401789, \"job\": \"Visual Effects Producer\", \"name\": \"Joyce Cox\"}, {\"credit_id\": \"5495994b92514132ed002951\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401790, \"job\": \"Visual Effects Producer\", \"name\": \"Jenny Foster\"}, {\"credit_id\": \"549599cbc3a3686ae1004613\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401791, \"job\": \"Visual Effects Editor\", \"name\": \"Christopher Marino\"}, {\"credit_id\": \"549599f2c3a3686ae100461e\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401792, \"job\": \"Visual Effects Editor\", \"name\": \"Jim Milton\"}, {\"credit_id\": \"54959a51c3a3686af3003eb5\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401793, \"job\": \"Visual Effects Producer\", \"name\": \"Cyndi Ochs\"}, {\"credit_id\": \"54959a7cc3a36811530026f4\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401794, \"job\": \"Visual Effects Editor\", \"name\": \"Lucas Putnam\"}, {\"credit_id\": \"54959b91c3a3680ff5002cb4\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401795, \"job\": \"Visual Effects Supervisor\", \"name\": \"Anthony \\'Max\\' Ivins\"}, {\"credit_id\": \"54959bb69251412c4e002a5f\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401796, \"job\": \"Visual Effects Supervisor\", \"name\": \"John Knoll\"}, {\"credit_id\": \"54959cbbc3a3686ae3004391\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 1401799, \"job\": \"Visual Effects Supervisor\", \"name\": \"Eric Saindon\"}, {\"credit_id\": \"54959d06c3a3686ae90043f6\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401800, \"job\": \"Visual Effects Supervisor\", \"name\": \"Wayne Stables\"}, {\"credit_id\": \"54959d259251416e1e003f11\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401801, \"job\": \"Visual Effects Supervisor\", \"name\": \"David Stinnett\"}, {\"credit_id\": \"54959db49251413af7002975\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401803, \"job\": \"Visual Effects Supervisor\", \"name\": \"Guy Williams\"}, {\"credit_id\": \"54959de4c3a3681153002750\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401804, \"job\": \"Stunt Coordinator\", \"name\": \"Stuart Thorp\"}, {\"credit_id\": \"54959ef2c3a3680fc60027f2\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1401805, \"job\": \"Best Boy Electric\", \"name\": \"Giles Coburn\"}, {\"credit_id\": \"54959f07c3a3680fc60027f9\", \"department\": \"Camera\", \"gender\": 2, \"id\": 1401806, \"job\": \"Still Photographer\", \"name\": \"Mark Fellman\"}, {\"credit_id\": \"54959f47c3a3681153002774\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1401807, \"job\": \"Lighting Technician\", \"name\": \"Scott Sprague\"}, {\"credit_id\": \"54959f8cc3a36831b8001df2\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401808, \"job\": \"Animation Director\", \"name\": \"Jeremy Hollobon\"}, {\"credit_id\": \"54959fa0c3a36831b8001dfb\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401809, \"job\": \"Animation Director\", \"name\": \"Orlando Meunier\"}, {\"credit_id\": \"54959fb6c3a3686af3003f54\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401810, \"job\": \"Animation Director\", \"name\": \"Taisuke Tanimura\"}, {\"credit_id\": \"54959fd2c3a36831b8001e02\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1401812, \"job\": \"Set Costumer\", \"name\": \"Lilia Mishel Acevedo\"}, {\"credit_id\": \"54959ff9c3a3686ae300440c\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1401814, \"job\": \"Set Costumer\", \"name\": \"Alejandro M. Hernandez\"}, {\"credit_id\": \"5495a0ddc3a3686ae10046fe\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1401815, \"job\": \"Digital Intermediate\", \"name\": \"Marvin Hall\"}, {\"credit_id\": \"5495a1f7c3a3686ae3004443\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401816, \"job\": \"Publicist\", \"name\": \"Judy Alley\"}, {\"credit_id\": \"5592b29fc3a36869d100002f\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1418381, \"job\": \"CG Supervisor\", \"name\": \"Mike Perry\"}, {\"credit_id\": \"5592b23a9251415df8001081\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1426854, \"job\": \"CG Supervisor\", \"name\": \"Andrew Morley\"}, {\"credit_id\": \"55491e1192514104c40002d8\", \"department\": \"Art\", \"gender\": 0, \"id\": 1438901, \"job\": \"Conceptual Design\", \"name\": \"Seth Engstrom\"}, {\"credit_id\": \"5525d5809251417276002b06\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1447362, \"job\": \"Visual Effects Art Director\", \"name\": \"Eric Oliver\"}, {\"credit_id\": \"554427ca925141586500312a\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1447503, \"job\": \"Modeling\", \"name\": \"Matsune Suzuki\"}, {\"credit_id\": \"551906889251415aab001c88\", \"department\": \"Art\", \"gender\": 0, \"id\": 1447524, \"job\": \"Art Department Manager\", \"name\": \"Paul Tobin\"}, {\"credit_id\": \"5592af8492514152cc0010de\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1452643, \"job\": \"Hairstylist\", \"name\": \"Roxane Griffin\"}, {\"credit_id\": \"553d3c109251415852001318\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1453938, \"job\": \"Lighting Artist\", \"name\": \"Arun Ram-Mohan\"}, {\"credit_id\": \"5592af4692514152d5001355\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1457305, \"job\": \"Makeup Artist\", \"name\": \"Georgia Lockhart-Adams\"}, {\"credit_id\": \"5592b2eac3a36877470012a5\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1466035, \"job\": \"CG Supervisor\", \"name\": \"Thrain Shadbolt\"}, {\"credit_id\": \"5592b032c3a36877450015f1\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483220, \"job\": \"CG Supervisor\", \"name\": \"Brad Alexander\"}, {\"credit_id\": \"5592b05592514152d80012f6\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483221, \"job\": \"CG Supervisor\", \"name\": \"Shadi Almassizadeh\"}, {\"credit_id\": \"5592b090c3a36877570010b5\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483222, \"job\": \"CG Supervisor\", \"name\": \"Simon Clutterbuck\"}, {\"credit_id\": \"5592b0dbc3a368774b00112c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483223, \"job\": \"CG Supervisor\", \"name\": \"Graeme Demmocks\"}, {\"credit_id\": \"5592b0fe92514152db0010c1\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483224, \"job\": \"CG Supervisor\", \"name\": \"Adrian Fernandes\"}, {\"credit_id\": \"5592b11f9251415df8001059\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483225, \"job\": \"CG Supervisor\", \"name\": \"Mitch Gates\"}, {\"credit_id\": \"5592b15dc3a3687745001645\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483226, \"job\": \"CG Supervisor\", \"name\": \"Jerry Kung\"}, {\"credit_id\": \"5592b18e925141645a0004ae\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483227, \"job\": \"CG Supervisor\", \"name\": \"Andy Lomas\"}, {\"credit_id\": \"5592b1bfc3a368775d0010e7\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483228, \"job\": \"CG Supervisor\", \"name\": \"Sebastian Marino\"}, {\"credit_id\": \"5592b2049251415df8001078\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483229, \"job\": \"CG Supervisor\", \"name\": \"Matthias Menz\"}, {\"credit_id\": \"5592b27b92514152d800136a\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483230, \"job\": \"CG Supervisor\", \"name\": \"Sergei Nevshupov\"}, {\"credit_id\": \"5592b2c3c3a36869e800003c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483231, \"job\": \"CG Supervisor\", \"name\": \"Philippe Rebours\"}, {\"credit_id\": \"5592b317c3a36877470012af\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483232, \"job\": \"CG Supervisor\", \"name\": \"Michael Takarangi\"}, {\"credit_id\": \"5592b345c3a36877470012bb\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483233, \"job\": \"CG Supervisor\", \"name\": \"David Weitzberg\"}, {\"credit_id\": \"5592b37cc3a368775100113b\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483234, \"job\": \"CG Supervisor\", \"name\": \"Ben White\"}, {\"credit_id\": \"573c8e2f9251413f5d000094\", \"department\": \"Crew\", \"gender\": 1, \"id\": 1621932, \"job\": \"Stunts\", \"name\": \"Min Windle\"}]'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df['crew'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "805ac26b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# from the 'crew' feature, we only need Director's name which we can use in our recommendation model\n",
    "# so we need to create some function to search for \"job\": \"Director\" value in our dictionary\n",
    "# since a movie has only 1 director, once we have found the director's name in our dictionary\n",
    "# we need to terminate our loop using the break statement\n",
    "\n",
    "# for simplication:\n",
    "# {\"credit_id\": \"52fe48009251416c750aca23\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1721, \"job\": \"Editor\", \"name\": \"Stephen E. Rivkin\"}\n",
    "# considering above dictionary, this person named 'Stephen E. Rivkin' is the Editor for the movie titled 'Avatar'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "ecb42981",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fetch_director(obj):\n",
    "    list = []\n",
    "    for i in ast.literal_eval(obj):\n",
    "        if i['job'] == 'Director':\n",
    "            list.append(i['name'])\n",
    "            break\n",
    "    return list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "38ab65f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0           [James Cameron]\n",
       "1          [Gore Verbinski]\n",
       "2              [Sam Mendes]\n",
       "3       [Christopher Nolan]\n",
       "4          [Andrew Stanton]\n",
       "               ...         \n",
       "4804     [Robert Rodriguez]\n",
       "4805         [Edward Burns]\n",
       "4806          [Scott Smith]\n",
       "4807          [Daniel Hsia]\n",
       "4808     [Brian Herzlinger]\n",
       "Name: crew, Length: 4806, dtype: object"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df['crew'] = feat_df['crew'].apply(fetch_director)\n",
    "feat_df['crew']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "5d1a7789",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, Science Fiction]</td>\n",
       "      <td>[culture clash, future, space war, space colon...</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>[Sam Worthington, Zoe Saldana, Sigourney Weave...</td>\n",
       "      <td>[James Cameron]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drug abuse, exotic island, east india ...</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>[Johnny Depp, Orlando Bloom, Keira Knightley, ...</td>\n",
       "      <td>[Gore Verbinski]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, based on novel, secret agent, sequel, mi...</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>[Daniel Craig, Christoph Waltz, Léa Seydoux, R...</td>\n",
       "      <td>[Sam Mendes]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dc comics, crime fighter, terrorist, secret i...</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>[Christian Bale, Michael Caine, Gary Oldman, A...</td>\n",
       "      <td>[Christopher Nolan]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, Science Fiction]</td>\n",
       "      <td>[based on novel, mars, medallion, space travel...</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>[Taylor Kitsch, Lynn Collins, Samantha Morton,...</td>\n",
       "      <td>[Andrew Stanton]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                          genres  \\\n",
       "0  [Action, Adventure, Fantasy, Science Fiction]   \n",
       "1                   [Adventure, Fantasy, Action]   \n",
       "2                     [Action, Adventure, Crime]   \n",
       "3               [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, Science Fiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [culture clash, future, space war, space colon...   \n",
       "1  [ocean, drug abuse, exotic island, east india ...   \n",
       "2  [spy, based on novel, secret agent, sequel, mi...   \n",
       "3  [dc comics, crime fighter, terrorist, secret i...   \n",
       "4  [based on novel, mars, medallion, space travel...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...   \n",
       "1  Captain Barbossa, long believed to be dead, ha...   \n",
       "2  A cryptic message from Bond’s past sends him o...   \n",
       "3  Following the death of District Attorney Harve...   \n",
       "4  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                                cast                 crew  \n",
       "0  [Sam Worthington, Zoe Saldana, Sigourney Weave...      [James Cameron]  \n",
       "1  [Johnny Depp, Orlando Bloom, Keira Knightley, ...     [Gore Verbinski]  \n",
       "2  [Daniel Craig, Christoph Waltz, Léa Seydoux, R...         [Sam Mendes]  \n",
       "3  [Christian Bale, Michael Caine, Gary Oldman, A...  [Christopher Nolan]  \n",
       "4  [Taylor Kitsch, Lynn Collins, Samantha Morton,...     [Andrew Stanton]  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b2f19f8",
   "metadata": {},
   "source": [
    "#### V. TRANSFORMING THE \"OVERVIEW\" FEATURE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "e8a81c2e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In the 22nd century, a paraplegic Marine is dispatched to the moon Pandora on a unique mission, but becomes torn between following orders and protecting an alien civilization.'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df['overview'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6b83dd72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       [In, the, 22nd, century,, a, paraplegic, Marin...\n",
       "1       [Captain, Barbossa,, long, believed, to, be, d...\n",
       "2       [A, cryptic, message, from, Bond’s, past, send...\n",
       "3       [Following, the, death, of, District, Attorney...\n",
       "4       [John, Carter, is, a, war-weary,, former, mili...\n",
       "                              ...                        \n",
       "4804    [El, Mariachi, just, wants, to, play, his, gui...\n",
       "4805    [A, newlywed, couple's, honeymoon, is, upended...\n",
       "4806    [\"Signed,, Sealed,, Delivered\", introduces, a,...\n",
       "4807    [When, ambitious, New, York, attorney, Sam, is...\n",
       "4808    [Ever, since, the, second, grade, when, he, fi...\n",
       "Name: overview, Length: 4806, dtype: object"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df['overview'] = feat_df['overview'].apply(lambda x:x.split())\n",
    "feat_df['overview']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "5c8db03f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, Science Fiction]</td>\n",
       "      <td>[culture clash, future, space war, space colon...</td>\n",
       "      <td>[In, the, 22nd, century,, a, paraplegic, Marin...</td>\n",
       "      <td>[Sam Worthington, Zoe Saldana, Sigourney Weave...</td>\n",
       "      <td>[James Cameron]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drug abuse, exotic island, east india ...</td>\n",
       "      <td>[Captain, Barbossa,, long, believed, to, be, d...</td>\n",
       "      <td>[Johnny Depp, Orlando Bloom, Keira Knightley, ...</td>\n",
       "      <td>[Gore Verbinski]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, based on novel, secret agent, sequel, mi...</td>\n",
       "      <td>[A, cryptic, message, from, Bond’s, past, send...</td>\n",
       "      <td>[Daniel Craig, Christoph Waltz, Léa Seydoux, R...</td>\n",
       "      <td>[Sam Mendes]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dc comics, crime fighter, terrorist, secret i...</td>\n",
       "      <td>[Following, the, death, of, District, Attorney...</td>\n",
       "      <td>[Christian Bale, Michael Caine, Gary Oldman, A...</td>\n",
       "      <td>[Christopher Nolan]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, Science Fiction]</td>\n",
       "      <td>[based on novel, mars, medallion, space travel...</td>\n",
       "      <td>[John, Carter, is, a, war-weary,, former, mili...</td>\n",
       "      <td>[Taylor Kitsch, Lynn Collins, Samantha Morton,...</td>\n",
       "      <td>[Andrew Stanton]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                          genres  \\\n",
       "0  [Action, Adventure, Fantasy, Science Fiction]   \n",
       "1                   [Adventure, Fantasy, Action]   \n",
       "2                     [Action, Adventure, Crime]   \n",
       "3               [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, Science Fiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [culture clash, future, space war, space colon...   \n",
       "1  [ocean, drug abuse, exotic island, east india ...   \n",
       "2  [spy, based on novel, secret agent, sequel, mi...   \n",
       "3  [dc comics, crime fighter, terrorist, secret i...   \n",
       "4  [based on novel, mars, medallion, space travel...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  [In, the, 22nd, century,, a, paraplegic, Marin...   \n",
       "1  [Captain, Barbossa,, long, believed, to, be, d...   \n",
       "2  [A, cryptic, message, from, Bond’s, past, send...   \n",
       "3  [Following, the, death, of, District, Attorney...   \n",
       "4  [John, Carter, is, a, war-weary,, former, mili...   \n",
       "\n",
       "                                                cast                 crew  \n",
       "0  [Sam Worthington, Zoe Saldana, Sigourney Weave...      [James Cameron]  \n",
       "1  [Johnny Depp, Orlando Bloom, Keira Knightley, ...     [Gore Verbinski]  \n",
       "2  [Daniel Craig, Christoph Waltz, Léa Seydoux, R...         [Sam Mendes]  \n",
       "3  [Christian Bale, Michael Caine, Gary Oldman, A...  [Christopher Nolan]  \n",
       "4  [Taylor Kitsch, Lynn Collins, Samantha Morton,...     [Andrew Stanton]  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "2d667dff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now all the 5 features: (genres\tkeywords\toverview\tcast\tcrew) are listed as a list\n",
    "# however there is a problem specific to spaces between these words which would cause an error in our recommendation model\n",
    "\n",
    "# for simplication:\n",
    "    # 'Avatar' movie has an actor named 'Sam Worthington'\n",
    "    # Now though it is a single name, our model would still consider it as two separate entities owing to the spaces\n",
    "    # so it will classify it into two different tags as 'Sam' & 'Worthington'\n",
    "    \n",
    "    # 'Spectre' movie has a director named 'Sam Mendes'\n",
    "    # so our model would consider it as two separate tags: 'Sam' & 'Mendes'\n",
    "    \n",
    "# now the issue will arise when the model has to show recommendation specific to these actor/director's name\n",
    "# say we need our model to recommend movies specific 'Sam' tag\n",
    "# this may lead so some sort of confusion leading to incorrect recommendations\n",
    "# our model might share the recommendations pulling in results for 'Sam' tag corresponding to 'Sam Mendes'\n",
    "# to avoid this confusion, we need to omit the spaces so that our model could consider a name as a single entity\n",
    "\n",
    "# 'Sam Worthington' \t 'SamWorthington'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "974c333a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# so we need to omit spaces b/w strings in the list for 4 features: (genres\t\tkeywords\tcast\tcrew)\n",
    "\n",
    "feat_df['genres'] = feat_df['genres'].apply(lambda x:[i.replace(\" \", \"\") for i in x])\n",
    "feat_df['keywords'] = feat_df['keywords'].apply(lambda x:[i.replace(\" \", \"\") for i in x])\n",
    "feat_df['cast'] = feat_df['cast'].apply(lambda x:[i.replace(\" \", \"\") for i in x])\n",
    "feat_df['crew'] = feat_df['crew'].apply(lambda x:[i.replace(\" \", \"\") for i in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "c284ef02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, ScienceFiction]</td>\n",
       "      <td>[cultureclash, future, spacewar, spacecolony, ...</td>\n",
       "      <td>[In, the, 22nd, century,, a, paraplegic, Marin...</td>\n",
       "      <td>[SamWorthington, ZoeSaldana, SigourneyWeaver, ...</td>\n",
       "      <td>[JamesCameron]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drugabuse, exoticisland, eastindiatrad...</td>\n",
       "      <td>[Captain, Barbossa,, long, believed, to, be, d...</td>\n",
       "      <td>[JohnnyDepp, OrlandoBloom, KeiraKnightley, Ste...</td>\n",
       "      <td>[GoreVerbinski]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, basedonnovel, secretagent, sequel, mi6, ...</td>\n",
       "      <td>[A, cryptic, message, from, Bond’s, past, send...</td>\n",
       "      <td>[DanielCraig, ChristophWaltz, LéaSeydoux, Ralp...</td>\n",
       "      <td>[SamMendes]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dccomics, crimefighter, terrorist, secretiden...</td>\n",
       "      <td>[Following, the, death, of, District, Attorney...</td>\n",
       "      <td>[ChristianBale, MichaelCaine, GaryOldman, Anne...</td>\n",
       "      <td>[ChristopherNolan]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, ScienceFiction]</td>\n",
       "      <td>[basedonnovel, mars, medallion, spacetravel, p...</td>\n",
       "      <td>[John, Carter, is, a, war-weary,, former, mili...</td>\n",
       "      <td>[TaylorKitsch, LynnCollins, SamanthaMorton, Wi...</td>\n",
       "      <td>[AndrewStanton]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                         genres  \\\n",
       "0  [Action, Adventure, Fantasy, ScienceFiction]   \n",
       "1                  [Adventure, Fantasy, Action]   \n",
       "2                    [Action, Adventure, Crime]   \n",
       "3              [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, ScienceFiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [cultureclash, future, spacewar, spacecolony, ...   \n",
       "1  [ocean, drugabuse, exoticisland, eastindiatrad...   \n",
       "2  [spy, basedonnovel, secretagent, sequel, mi6, ...   \n",
       "3  [dccomics, crimefighter, terrorist, secretiden...   \n",
       "4  [basedonnovel, mars, medallion, spacetravel, p...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  [In, the, 22nd, century,, a, paraplegic, Marin...   \n",
       "1  [Captain, Barbossa,, long, believed, to, be, d...   \n",
       "2  [A, cryptic, message, from, Bond’s, past, send...   \n",
       "3  [Following, the, death, of, District, Attorney...   \n",
       "4  [John, Carter, is, a, war-weary,, former, mili...   \n",
       "\n",
       "                                                cast                crew  \n",
       "0  [SamWorthington, ZoeSaldana, SigourneyWeaver, ...      [JamesCameron]  \n",
       "1  [JohnnyDepp, OrlandoBloom, KeiraKnightley, Ste...     [GoreVerbinski]  \n",
       "2  [DanielCraig, ChristophWaltz, LéaSeydoux, Ralp...         [SamMendes]  \n",
       "3  [ChristianBale, MichaelCaine, GaryOldman, Anne...  [ChristopherNolan]  \n",
       "4  [TaylorKitsch, LynnCollins, SamanthaMorton, Wi...     [AndrewStanton]  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "001a8101",
   "metadata": {},
   "outputs": [],
   "source": [
    "# concatenating all the 5 features into one feature titled 'tags'\n",
    "feat_df['tags'] = feat_df['overview'] + feat_df['genres'] + feat_df['keywords'] + feat_df['cast'] + feat_df['crew']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "40a2cae0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, ScienceFiction]</td>\n",
       "      <td>[cultureclash, future, spacewar, spacecolony, ...</td>\n",
       "      <td>[In, the, 22nd, century,, a, paraplegic, Marin...</td>\n",
       "      <td>[SamWorthington, ZoeSaldana, SigourneyWeaver, ...</td>\n",
       "      <td>[JamesCameron]</td>\n",
       "      <td>[In, the, 22nd, century,, a, paraplegic, Marin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drugabuse, exoticisland, eastindiatrad...</td>\n",
       "      <td>[Captain, Barbossa,, long, believed, to, be, d...</td>\n",
       "      <td>[JohnnyDepp, OrlandoBloom, KeiraKnightley, Ste...</td>\n",
       "      <td>[GoreVerbinski]</td>\n",
       "      <td>[Captain, Barbossa,, long, believed, to, be, d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, basedonnovel, secretagent, sequel, mi6, ...</td>\n",
       "      <td>[A, cryptic, message, from, Bond’s, past, send...</td>\n",
       "      <td>[DanielCraig, ChristophWaltz, LéaSeydoux, Ralp...</td>\n",
       "      <td>[SamMendes]</td>\n",
       "      <td>[A, cryptic, message, from, Bond’s, past, send...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dccomics, crimefighter, terrorist, secretiden...</td>\n",
       "      <td>[Following, the, death, of, District, Attorney...</td>\n",
       "      <td>[ChristianBale, MichaelCaine, GaryOldman, Anne...</td>\n",
       "      <td>[ChristopherNolan]</td>\n",
       "      <td>[Following, the, death, of, District, Attorney...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, ScienceFiction]</td>\n",
       "      <td>[basedonnovel, mars, medallion, spacetravel, p...</td>\n",
       "      <td>[John, Carter, is, a, war-weary,, former, mili...</td>\n",
       "      <td>[TaylorKitsch, LynnCollins, SamanthaMorton, Wi...</td>\n",
       "      <td>[AndrewStanton]</td>\n",
       "      <td>[John, Carter, is, a, war-weary,, former, mili...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                         genres  \\\n",
       "0  [Action, Adventure, Fantasy, ScienceFiction]   \n",
       "1                  [Adventure, Fantasy, Action]   \n",
       "2                    [Action, Adventure, Crime]   \n",
       "3              [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, ScienceFiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [cultureclash, future, spacewar, spacecolony, ...   \n",
       "1  [ocean, drugabuse, exoticisland, eastindiatrad...   \n",
       "2  [spy, basedonnovel, secretagent, sequel, mi6, ...   \n",
       "3  [dccomics, crimefighter, terrorist, secretiden...   \n",
       "4  [basedonnovel, mars, medallion, spacetravel, p...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  [In, the, 22nd, century,, a, paraplegic, Marin...   \n",
       "1  [Captain, Barbossa,, long, believed, to, be, d...   \n",
       "2  [A, cryptic, message, from, Bond’s, past, send...   \n",
       "3  [Following, the, death, of, District, Attorney...   \n",
       "4  [John, Carter, is, a, war-weary,, former, mili...   \n",
       "\n",
       "                                                cast                crew  \\\n",
       "0  [SamWorthington, ZoeSaldana, SigourneyWeaver, ...      [JamesCameron]   \n",
       "1  [JohnnyDepp, OrlandoBloom, KeiraKnightley, Ste...     [GoreVerbinski]   \n",
       "2  [DanielCraig, ChristophWaltz, LéaSeydoux, Ralp...         [SamMendes]   \n",
       "3  [ChristianBale, MichaelCaine, GaryOldman, Anne...  [ChristopherNolan]   \n",
       "4  [TaylorKitsch, LynnCollins, SamanthaMorton, Wi...     [AndrewStanton]   \n",
       "\n",
       "                                                tags  \n",
       "0  [In, the, 22nd, century,, a, paraplegic, Marin...  \n",
       "1  [Captain, Barbossa,, long, believed, to, be, d...  \n",
       "2  [A, cryptic, message, from, Bond’s, past, send...  \n",
       "3  [Following, the, death, of, District, Attorney...  \n",
       "4  [John, Carter, is, a, war-weary,, former, mili...  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "f6a33633",
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessed_df = feat_df[['movie_id', 'title', 'tags']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "7060ffe5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[In, the, 22nd, century,, a, paraplegic, Marin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Captain, Barbossa,, long, believed, to, be, d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[A, cryptic, message, from, Bond’s, past, send...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Following, the, death, of, District, Attorney...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[John, Carter, is, a, war-weary,, former, mili...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                                tags  \n",
       "0  [In, the, 22nd, century,, a, paraplegic, Marin...  \n",
       "1  [Captain, Barbossa,, long, believed, to, be, d...  \n",
       "2  [A, cryptic, message, from, Bond’s, past, send...  \n",
       "3  [Following, the, death, of, District, Attorney...  \n",
       "4  [John, Carter, is, a, war-weary,, former, mili...  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessed_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "32fc5897",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4804</th>\n",
       "      <td>9367</td>\n",
       "      <td>El Mariachi</td>\n",
       "      <td>El Mariachi just wants to play his guitar and ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4805</th>\n",
       "      <td>72766</td>\n",
       "      <td>Newlyweds</td>\n",
       "      <td>A newlywed couple's honeymoon is upended by th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4806</th>\n",
       "      <td>231617</td>\n",
       "      <td>Signed, Sealed, Delivered</td>\n",
       "      <td>\"Signed, Sealed, Delivered\" introduces a dedic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4807</th>\n",
       "      <td>126186</td>\n",
       "      <td>Shanghai Calling</td>\n",
       "      <td>When ambitious New York attorney Sam is sent t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4808</th>\n",
       "      <td>25975</td>\n",
       "      <td>My Date with Drew</td>\n",
       "      <td>Ever since the second grade when he first saw ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4806 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      movie_id                                     title  \\\n",
       "0        19995                                    Avatar   \n",
       "1          285  Pirates of the Caribbean: At World's End   \n",
       "2       206647                                   Spectre   \n",
       "3        49026                     The Dark Knight Rises   \n",
       "4        49529                               John Carter   \n",
       "...        ...                                       ...   \n",
       "4804      9367                               El Mariachi   \n",
       "4805     72766                                 Newlyweds   \n",
       "4806    231617                 Signed, Sealed, Delivered   \n",
       "4807    126186                          Shanghai Calling   \n",
       "4808     25975                         My Date with Drew   \n",
       "\n",
       "                                                   tags  \n",
       "0     In the 22nd century, a paraplegic Marine is di...  \n",
       "1     Captain Barbossa, long believed to be dead, ha...  \n",
       "2     A cryptic message from Bond’s past sends him o...  \n",
       "3     Following the death of District Attorney Harve...  \n",
       "4     John Carter is a war-weary, former military ca...  \n",
       "...                                                 ...  \n",
       "4804  El Mariachi just wants to play his guitar and ...  \n",
       "4805  A newlywed couple's honeymoon is upended by th...  \n",
       "4806  \"Signed, Sealed, Delivered\" introduces a dedic...  \n",
       "4807  When ambitious New York attorney Sam is sent t...  \n",
       "4808  Ever since the second grade when he first saw ...  \n",
       "\n",
       "[4806 rows x 3 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now we will consolidate the tags in the list to formulate a string\n",
    "\n",
    "preprocessed_df['tags'] = preprocessed_df['tags'].apply(lambda x:\" \".join(x))\n",
    "preprocessed_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "9bc0946b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>in the 22nd century, a paraplegic marine is di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>captain barbossa, long believed to be dead, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>a cryptic message from bond’s past sends him o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>following the death of district attorney harve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>john carter is a war-weary, former military ca...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4804</th>\n",
       "      <td>9367</td>\n",
       "      <td>El Mariachi</td>\n",
       "      <td>el mariachi just wants to play his guitar and ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4805</th>\n",
       "      <td>72766</td>\n",
       "      <td>Newlyweds</td>\n",
       "      <td>a newlywed couple's honeymoon is upended by th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4806</th>\n",
       "      <td>231617</td>\n",
       "      <td>Signed, Sealed, Delivered</td>\n",
       "      <td>\"signed, sealed, delivered\" introduces a dedic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4807</th>\n",
       "      <td>126186</td>\n",
       "      <td>Shanghai Calling</td>\n",
       "      <td>when ambitious new york attorney sam is sent t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4808</th>\n",
       "      <td>25975</td>\n",
       "      <td>My Date with Drew</td>\n",
       "      <td>ever since the second grade when he first saw ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4806 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      movie_id                                     title  \\\n",
       "0        19995                                    Avatar   \n",
       "1          285  Pirates of the Caribbean: At World's End   \n",
       "2       206647                                   Spectre   \n",
       "3        49026                     The Dark Knight Rises   \n",
       "4        49529                               John Carter   \n",
       "...        ...                                       ...   \n",
       "4804      9367                               El Mariachi   \n",
       "4805     72766                                 Newlyweds   \n",
       "4806    231617                 Signed, Sealed, Delivered   \n",
       "4807    126186                          Shanghai Calling   \n",
       "4808     25975                         My Date with Drew   \n",
       "\n",
       "                                                   tags  \n",
       "0     in the 22nd century, a paraplegic marine is di...  \n",
       "1     captain barbossa, long believed to be dead, ha...  \n",
       "2     a cryptic message from bond’s past sends him o...  \n",
       "3     following the death of district attorney harve...  \n",
       "4     john carter is a war-weary, former military ca...  \n",
       "...                                                 ...  \n",
       "4804  el mariachi just wants to play his guitar and ...  \n",
       "4805  a newlywed couple's honeymoon is upended by th...  \n",
       "4806  \"signed, sealed, delivered\" introduces a dedic...  \n",
       "4807  when ambitious new york attorney sam is sent t...  \n",
       "4808  ever since the second grade when he first saw ...  \n",
       "\n",
       "[4806 rows x 3 columns]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# also we will have all these words converted to lowercase for simplification\n",
    "\n",
    "preprocessed_df['tags'] = preprocessed_df['tags'].apply(lambda x:x.lower())\n",
    "preprocessed_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "05b22844",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 4806 entries, 0 to 4808\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   movie_id  4806 non-null   int64 \n",
      " 1   title     4806 non-null   object\n",
      " 2   tags      4806 non-null   object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 279.2+ KB\n"
     ]
    }
   ],
   "source": [
    "preprocessed_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "d776cedc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'in the 22nd century, a paraplegic marine is dispatched to the moon pandora on a unique mission, but becomes torn between following orders and protecting an alien civilization. action adventure fantasy sciencefiction cultureclash future spacewar spacecolony society spacetravel futuristic romance space alien tribe alienplanet cgi marine soldier battle loveaffair antiwar powerrelations mindandsoul 3d samworthington zoesaldana sigourneyweaver stephenlang jamescameron'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessed_df['tags'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "3f670805",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"captain barbossa, long believed to be dead, has come back to life and is headed to the edge of the earth with will turner and elizabeth swann. but nothing is quite as it seems. adventure fantasy action ocean drugabuse exoticisland eastindiatradingcompany loveofone'slife traitor shipwreck strongwoman ship alliance calypso afterlife fighter pirate swashbuckler aftercreditsstinger johnnydepp orlandobloom keiraknightley stellanskarsgård goreverbinski\""
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessed_df['tags'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9489969",
   "metadata": {},
   "source": [
    "## COUNT VECTORIZATION\n",
    "\n",
    "The data under 'tags' feature is in textual format and to check for similarity between two tag records we need to evaluate the words that are common to both these records. Now we can't do this manually since there are over 4000 movies. For a real time applications, there might be even more records/data. So here we need to use the 'COUNT VECTORIZATION' feature wherein we would be transforming each of these tags into vectors. Since we are transforming *text into vectors* this process is termed as 'TEXT VECTORIZATION'. There are a number of techniques as shared below to apply text vectorization:\n",
    "- BAG OF WORDS (*we would be using it here in our problem*)\n",
    "- TF-IDF\n",
    "- word2vec\n",
    "\n",
    "### BAG OF WORDS\n",
    "'Bag of words' is the simplest of all techniques wherein at first we concatenate all the tag records together as a bag and then from here we will be extracting those set of words which occur most frequently throughout this complete bag. Our dataset houses nearly 4800 movies, so we can consider extracting around 5000 words in common that appear most frequently. \n",
    "\n",
    "For simplification let's assume:\n",
    "- **tag1** = 'in the 22nd century, a paraplegic marine is dispatched to the moon pandora on a unique mission, but becomes torn between following orders and protecting an alien civilization. action adventure fantasy sciencefiction cultureclash future spacewar spacecolony society spacetravel futuristic romance space alien tribe alienplanet cgi marine soldier battle loveaffair antiwar powerrelations mindandsoul 3d samworthington zoesaldana sigourneyweaver stephenlang jamescameron'\n",
    "- **tag2** = 'captain barbossa, long believed to be dead, has come back to life and is headed to the edge of the earth with will turner and elizabeth swann. but nothing is quite as it seems. adventure fantasy action ocean drugabuse exoticisland eastindiatradingcompany loveofone'slife traitor shipwreck strongwoman ship alliance calypso afterlife fighter pirate swashbuckler aftercreditsstinger johnnydepp orlandobloom keiraknightley stellanskarsgård goreverbinski'\n",
    "\n",
    "Similar to above there are around 4806 tag records in the form of textual data. So we will group them together as:\n",
    "[*tag1 + tag2 + tag3 + ........ + tag4806*] and let's say we will group under a single entity named 'bag'.\n",
    "- *bag* = *tag1 + tag2 + tag3 + ........ + tag4806*\n",
    "- now from this bag we would be extracting 5000 words that are most common/appear most frequently\n",
    "- we will refer them as *w1, w2, w3, .......... , w5000*\n",
    "- now we will check how many times each of these words are occuring under a single movie\n",
    "- this will derive the results in a form of a string\n",
    "\n",
    "for example, movies = [*w1, w2, w3, .......... , w5000*]\n",
    "- m1 = [*5, 3, 0, .......... , 2*]\n",
    "- m2 = [*6, 1, 2, .......... , 0*]\n",
    "- m3 = [*0, 0, 1, .......... , 0*]\n",
    "- m4 = [*4, 2, 0, .......... , 0*]\n",
    "- ..........\n",
    "- ..........\n",
    "- m4806 = [*1, 0, 0, .......... , 0*]\n",
    "\n",
    "> This indicates that first common word (w1) is occuring 5 times under the tag data for movie (m1),  second common word (w2) is occuring 3 times under the tag data for movie (m1), third common word (m3) is occuring 0 times under the tag data for movie (m1) and henceforth.\n",
    "\n",
    "Each of these movies are now represented as a vector mapped under a (x, y) dimensional plane. (x) refers to the number of movies and (y) refers to the number of common words. So in our problem we are dealing with (4806, 5000) dimensional plane, wherein we have 4806 vectors mapped onto this plane.\n",
    "\n",
    "#### SO EVERY MOVIE IS NOW A VECTOR MAPPED UNDER A 5000-dimensional SPACE\n",
    "\n",
    "**NOTE**: 'Stopwords' refer to those words which are used in english language with the sole purpose of formation of sentences however they do not have any contribution to the meaning of these sentences. For example: a, is, are, if, to, from, and, etc. So we would be excluding these words from our bag of commom words as they won't serve any meaning to our purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "2d6a1a19",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "3cb300f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer(stop_words='english',\n",
    "                     max_features=5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "0311539c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the result obtained after applying fit_transform() operation over the cv object\n",
    "# is basically in the form of a sciPy sparse matrix and we need to convert these values in the form of a list\n",
    "# so we would be applying the toarray() operation to convert these values into a numpy array\n",
    "\n",
    "vectors = cv.fit_transform(preprocessed_df['tags']).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "bee36d31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (4806, 5000)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       ...,\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Shape:', vectors.shape)\n",
    "vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "c2daee6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vector matrix for the first movie: [0 0 0 ... 0 0 0]\n",
      "Vector matrix for the second movie: [0 0 0 ... 0 0 0]\n",
      "Vector matrix for the third movie: [0 0 0 ... 0 0 0]\n",
      "Vector matrix for the fourth movie: [0 0 0 ... 0 0 0]\n",
      "\n",
      "Vector matrix for the second last movie: [0 0 0 ... 0 0 0]\n",
      "Vector matrix for the last movie: [0 0 0 ... 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "print('Vector matrix for the first movie:', vectors[0])\n",
    "print('Vector matrix for the second movie:', vectors[1])\n",
    "print('Vector matrix for the third movie:', vectors[2])\n",
    "print('Vector matrix for the fourth movie:', vectors[3])\n",
    "print(\"\")\n",
    "print('Vector matrix for the second last movie:', vectors[4084])\n",
    "print('Vector matrix for the last movie:', vectors[4085])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "88827c39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['000',\n",
       " '007',\n",
       " '10',\n",
       " '100',\n",
       " '11',\n",
       " '12',\n",
       " '13',\n",
       " '14',\n",
       " '15',\n",
       " '16',\n",
       " '17',\n",
       " '18',\n",
       " '18th',\n",
       " '19',\n",
       " '1930s',\n",
       " '1940s',\n",
       " '1950s',\n",
       " '1960s',\n",
       " '1970s',\n",
       " '1971',\n",
       " '1976',\n",
       " '1980',\n",
       " '1980s',\n",
       " '1985',\n",
       " '1990s',\n",
       " '19th',\n",
       " '19thcentury',\n",
       " '20',\n",
       " '200',\n",
       " '2003',\n",
       " '2009',\n",
       " '20th',\n",
       " '24',\n",
       " '25',\n",
       " '30',\n",
       " '300',\n",
       " '3d',\n",
       " '40',\n",
       " '50',\n",
       " '500',\n",
       " '60',\n",
       " '60s',\n",
       " '70',\n",
       " '70s',\n",
       " 'aaron',\n",
       " 'aaroneckhart',\n",
       " 'aarontaylor',\n",
       " 'abandoned',\n",
       " 'abducted',\n",
       " 'abigailbreslin',\n",
       " 'abilities',\n",
       " 'ability',\n",
       " 'able',\n",
       " 'aboard',\n",
       " 'abuse',\n",
       " 'abusive',\n",
       " 'academy',\n",
       " 'accept',\n",
       " 'accepted',\n",
       " 'accepts',\n",
       " 'access',\n",
       " 'accident',\n",
       " 'accidental',\n",
       " 'accidentally',\n",
       " 'accompanied',\n",
       " 'accomplish',\n",
       " 'account',\n",
       " 'accountant',\n",
       " 'accused',\n",
       " 'ace',\n",
       " 'achieve',\n",
       " 'act',\n",
       " 'acting',\n",
       " 'action',\n",
       " 'actionhero',\n",
       " 'actions',\n",
       " 'activist',\n",
       " 'activities',\n",
       " 'activity',\n",
       " 'actor',\n",
       " 'actors',\n",
       " 'actress',\n",
       " 'acts',\n",
       " 'actual',\n",
       " 'actually',\n",
       " 'adam',\n",
       " 'adambrody',\n",
       " 'adams',\n",
       " 'adamsandler',\n",
       " 'adamshankman',\n",
       " 'adaptation',\n",
       " 'adapted',\n",
       " 'addict',\n",
       " 'addicted',\n",
       " 'addiction',\n",
       " 'adolescence',\n",
       " 'adopt',\n",
       " 'adopted',\n",
       " 'adoption',\n",
       " 'adopts',\n",
       " 'adrienbrody',\n",
       " 'adult',\n",
       " 'adultery',\n",
       " 'adulthood',\n",
       " 'adults',\n",
       " 'advantage',\n",
       " 'adventure',\n",
       " 'adventures',\n",
       " 'advertising',\n",
       " 'advice',\n",
       " 'affair',\n",
       " 'affairs',\n",
       " 'affection',\n",
       " 'affections',\n",
       " 'afghanistan',\n",
       " 'africa',\n",
       " 'african',\n",
       " 'africanamerican',\n",
       " 'aftercreditsstinger',\n",
       " 'afterlife',\n",
       " 'aftermath',\n",
       " 'age',\n",
       " 'aged',\n",
       " 'agedifference',\n",
       " 'agency',\n",
       " 'agenda',\n",
       " 'agent',\n",
       " 'agents',\n",
       " 'aggressive',\n",
       " 'aging',\n",
       " 'ago',\n",
       " 'agree',\n",
       " 'agrees',\n",
       " 'ahead',\n",
       " 'aid',\n",
       " 'aidanquinn',\n",
       " 'aided',\n",
       " 'aids',\n",
       " 'ailing',\n",
       " 'air',\n",
       " 'airplane',\n",
       " 'airplanecrash',\n",
       " 'airport',\n",
       " 'aka',\n",
       " 'al',\n",
       " 'alabama',\n",
       " 'alan',\n",
       " 'alanalda',\n",
       " 'alanarkin',\n",
       " 'alaska',\n",
       " 'albert',\n",
       " 'albertbrooks',\n",
       " 'alcohol',\n",
       " 'alcoholic',\n",
       " 'alcoholism',\n",
       " 'alecbaldwin',\n",
       " 'alex',\n",
       " 'alexkendrick',\n",
       " 'alfredhitchcock',\n",
       " 'alfredmolina',\n",
       " 'alfrewoodard',\n",
       " 'ali',\n",
       " 'alice',\n",
       " 'alien',\n",
       " 'alieninvasion',\n",
       " 'alienlife',\n",
       " 'aliens',\n",
       " 'alike',\n",
       " 'alive',\n",
       " 'allen',\n",
       " 'alliance',\n",
       " 'allied',\n",
       " 'allies',\n",
       " 'allow',\n",
       " 'allows',\n",
       " 'ally',\n",
       " 'alongside',\n",
       " 'alpacino',\n",
       " 'alter',\n",
       " 'alternate',\n",
       " 'alternative',\n",
       " 'alzheimer',\n",
       " 'amanda',\n",
       " 'amandapeet',\n",
       " 'amandaseyfried',\n",
       " 'amateur',\n",
       " 'amazing',\n",
       " 'ambassador',\n",
       " 'ambition',\n",
       " 'ambitious',\n",
       " 'ambulance',\n",
       " 'ambush',\n",
       " 'america',\n",
       " 'american',\n",
       " 'americanabroad',\n",
       " 'americanfootball',\n",
       " 'americans',\n",
       " 'amid',\n",
       " 'amidst',\n",
       " 'amnesia',\n",
       " 'amp',\n",
       " 'amsterdam',\n",
       " 'amusementpark',\n",
       " 'amy',\n",
       " 'amyadams',\n",
       " 'amysmart',\n",
       " 'analyst',\n",
       " 'anarchiccomedy',\n",
       " 'ancient',\n",
       " 'ancientrome',\n",
       " 'ancientworld',\n",
       " 'anderson',\n",
       " 'andiemacdowell',\n",
       " 'andrew',\n",
       " 'android',\n",
       " 'andy',\n",
       " 'andygarcía',\n",
       " 'andyserkis',\n",
       " 'angel',\n",
       " 'angelabassett',\n",
       " 'angeles',\n",
       " 'angelinajolie',\n",
       " 'angels',\n",
       " 'anger',\n",
       " 'anglee',\n",
       " 'angry',\n",
       " 'animal',\n",
       " 'animalattack',\n",
       " 'animalhorror',\n",
       " 'animals',\n",
       " 'animated',\n",
       " 'animation',\n",
       " 'anjelicahuston',\n",
       " 'anna',\n",
       " 'annafaris',\n",
       " 'annakendrick',\n",
       " 'annasophiarobb',\n",
       " 'anne',\n",
       " 'annehathaway',\n",
       " 'annemoss',\n",
       " 'annettebening',\n",
       " 'annie',\n",
       " 'anniversary',\n",
       " 'annual',\n",
       " 'answer',\n",
       " 'answers',\n",
       " 'ant',\n",
       " 'anthology',\n",
       " 'anthony',\n",
       " 'anthonyanderson',\n",
       " 'anthonyhopkins',\n",
       " 'anthonymackie',\n",
       " 'anthropomorphism',\n",
       " 'anti',\n",
       " 'antics',\n",
       " 'antihero',\n",
       " 'antoinefuqua',\n",
       " 'antoniobanderas',\n",
       " 'antonyelchin',\n",
       " 'apart',\n",
       " 'apartheid',\n",
       " 'apartment',\n",
       " 'ape',\n",
       " 'apes',\n",
       " 'apocalypse',\n",
       " 'apocalyptic',\n",
       " 'apparent',\n",
       " 'apparently',\n",
       " 'appear',\n",
       " 'appears',\n",
       " 'apple',\n",
       " 'appointed',\n",
       " 'apprentice',\n",
       " 'approach',\n",
       " 'approaches',\n",
       " 'approaching',\n",
       " 'april',\n",
       " 'arab',\n",
       " 'arch',\n",
       " 'archaeologist',\n",
       " 'architect',\n",
       " 'arctic',\n",
       " 'area',\n",
       " 'aren',\n",
       " 'arena',\n",
       " 'argument',\n",
       " 'arise',\n",
       " 'aristocrat',\n",
       " 'armed',\n",
       " 'arms',\n",
       " 'army',\n",
       " 'arnold',\n",
       " 'arnoldschwarzenegger',\n",
       " 'arrangedmarriage',\n",
       " 'arrest',\n",
       " 'arrested',\n",
       " 'arrival',\n",
       " 'arrive',\n",
       " 'arrives',\n",
       " 'arriving',\n",
       " 'arrogant',\n",
       " 'art',\n",
       " 'arthur',\n",
       " 'artificialintelligence',\n",
       " 'artist',\n",
       " 'artistic',\n",
       " 'artists',\n",
       " 'arts',\n",
       " 'ashley',\n",
       " 'ashleyjudd',\n",
       " 'ashtonkutcher',\n",
       " 'asia',\n",
       " 'aside',\n",
       " 'ask',\n",
       " 'asked',\n",
       " 'asking',\n",
       " 'asks',\n",
       " 'aspirations',\n",
       " 'aspiring',\n",
       " 'assassin',\n",
       " 'assassinate',\n",
       " 'assassination',\n",
       " 'assassins',\n",
       " 'assault',\n",
       " 'assigned',\n",
       " 'assignment',\n",
       " 'assistant',\n",
       " 'assumes',\n",
       " 'asteroid',\n",
       " 'astronaut',\n",
       " 'astronauts',\n",
       " 'asylum',\n",
       " 'athlete',\n",
       " 'atomicbomb',\n",
       " 'attack',\n",
       " 'attacked',\n",
       " 'attacks',\n",
       " 'attempt',\n",
       " 'attempting',\n",
       " 'attempts',\n",
       " 'attending',\n",
       " 'attends',\n",
       " 'attention',\n",
       " 'attitude',\n",
       " 'attorney',\n",
       " 'attracted',\n",
       " 'attraction',\n",
       " 'attractive',\n",
       " 'audience',\n",
       " 'audiences',\n",
       " 'audition',\n",
       " 'august',\n",
       " 'aunt',\n",
       " 'austin',\n",
       " 'australia',\n",
       " 'australian',\n",
       " 'author',\n",
       " 'authorities',\n",
       " 'authority',\n",
       " 'autism',\n",
       " 'auto',\n",
       " 'avenge',\n",
       " 'average',\n",
       " 'avoid',\n",
       " 'awaits',\n",
       " 'awakens',\n",
       " 'award',\n",
       " 'away',\n",
       " 'awry',\n",
       " 'ax',\n",
       " 'babe',\n",
       " 'baby',\n",
       " 'bachelor',\n",
       " 'backdrop',\n",
       " 'background',\n",
       " 'backgrounds',\n",
       " 'bad',\n",
       " 'bag',\n",
       " 'bahamas',\n",
       " 'bail',\n",
       " 'balance',\n",
       " 'ball',\n",
       " 'ballet',\n",
       " 'baltimore',\n",
       " 'band',\n",
       " 'bandits',\n",
       " 'bangkok',\n",
       " 'banished',\n",
       " 'bank',\n",
       " 'banker',\n",
       " 'bankrobber',\n",
       " 'bankrobbery',\n",
       " 'bar',\n",
       " 'barely',\n",
       " 'bargained',\n",
       " 'barn',\n",
       " 'barney',\n",
       " 'barry',\n",
       " 'barrylevinson',\n",
       " 'bars',\n",
       " 'base',\n",
       " 'baseball',\n",
       " 'based',\n",
       " 'basedoncomicbook',\n",
       " 'basedongraphicnovel',\n",
       " 'basedonnovel',\n",
       " 'basedonplay',\n",
       " 'basedonstagemusical',\n",
       " 'basedontrueevents',\n",
       " 'basedontruestory',\n",
       " 'basedontvseries',\n",
       " 'basedonvideogame',\n",
       " 'basedonyoungadultnovel',\n",
       " 'basement',\n",
       " 'basketball',\n",
       " 'batman',\n",
       " 'battle',\n",
       " 'battlefield',\n",
       " 'battles',\n",
       " 'battling',\n",
       " 'bay',\n",
       " 'beach',\n",
       " 'bear',\n",
       " 'bears',\n",
       " 'beast',\n",
       " 'beasts',\n",
       " 'beat',\n",
       " 'beating',\n",
       " 'beautiful',\n",
       " 'beautifulwoman',\n",
       " 'beauty',\n",
       " 'becky',\n",
       " 'becominganadult',\n",
       " 'bed',\n",
       " 'bedroom',\n",
       " 'beer',\n",
       " 'befriends',\n",
       " 'began',\n",
       " 'begin',\n",
       " 'beginning',\n",
       " 'begins',\n",
       " 'behavior',\n",
       " 'beings',\n",
       " 'beliefs',\n",
       " 'believe',\n",
       " 'believed',\n",
       " 'believes',\n",
       " 'believing',\n",
       " 'beloved',\n",
       " 'ben',\n",
       " 'benaffleck',\n",
       " 'beneath',\n",
       " 'benfoster',\n",
       " 'beniciodeltoro',\n",
       " 'benjamin',\n",
       " 'benjaminbratt',\n",
       " 'benkingsley',\n",
       " 'bennett',\n",
       " 'benstiller',\n",
       " 'bent',\n",
       " 'berlin',\n",
       " 'berniemac',\n",
       " 'best',\n",
       " 'bestfriend',\n",
       " 'bet',\n",
       " 'beth',\n",
       " 'betrayal',\n",
       " 'betrayed',\n",
       " 'bettemidler',\n",
       " 'better',\n",
       " 'betty',\n",
       " 'beverly',\n",
       " 'bible',\n",
       " 'big',\n",
       " 'bigger',\n",
       " 'biggest',\n",
       " 'biker',\n",
       " 'bikini',\n",
       " 'billhader',\n",
       " 'billionaire',\n",
       " 'billmurray',\n",
       " 'billnighy',\n",
       " 'billpaxton',\n",
       " 'billpullman',\n",
       " 'billy',\n",
       " 'billybobthornton',\n",
       " 'billycrudup',\n",
       " 'billycrystal',\n",
       " 'biography',\n",
       " 'bird',\n",
       " 'birth',\n",
       " 'birthday',\n",
       " 'bisexual',\n",
       " 'bishop',\n",
       " 'bit',\n",
       " 'bite',\n",
       " 'bitter',\n",
       " 'bizarre',\n",
       " 'black',\n",
       " 'blackmagic',\n",
       " 'blackmail',\n",
       " 'blackpeople',\n",
       " 'blade',\n",
       " 'blame',\n",
       " 'blind',\n",
       " 'bliss',\n",
       " 'block',\n",
       " 'blonde',\n",
       " 'blood',\n",
       " 'bloodsplatter',\n",
       " 'bloodthirsty',\n",
       " 'bloody',\n",
       " 'blow',\n",
       " 'blue',\n",
       " 'board',\n",
       " 'boarding',\n",
       " 'boardingschool',\n",
       " 'boat',\n",
       " 'bob',\n",
       " 'bobby',\n",
       " 'bobbyfarrelly',\n",
       " 'bobhoskins',\n",
       " 'bodies',\n",
       " 'body',\n",
       " 'bodyguard',\n",
       " 'bold',\n",
       " 'bollywood',\n",
       " 'bomb',\n",
       " 'bombing',\n",
       " 'bond',\n",
       " 'bonds',\n",
       " 'bone',\n",
       " 'bonniehunt',\n",
       " 'book',\n",
       " 'books',\n",
       " 'border',\n",
       " 'bored',\n",
       " 'boredom',\n",
       " 'boring',\n",
       " 'born',\n",
       " 'boss',\n",
       " 'boston',\n",
       " 'botched',\n",
       " 'bound',\n",
       " 'boundaries',\n",
       " 'bounty',\n",
       " 'bountyhunter',\n",
       " 'box',\n",
       " 'boxer',\n",
       " 'boxing',\n",
       " 'boy',\n",
       " 'boyfriend',\n",
       " 'boys',\n",
       " 'bradleycooper',\n",
       " 'bradpitt',\n",
       " 'brain',\n",
       " 'brand',\n",
       " 'brave',\n",
       " 'bravery',\n",
       " 'brazil',\n",
       " 'brazilian',\n",
       " 'break',\n",
       " 'breakdown',\n",
       " 'breaking',\n",
       " 'breaks',\n",
       " 'breckinmeyer',\n",
       " 'brendanfraser',\n",
       " 'brendangleeson',\n",
       " 'brent',\n",
       " 'brettratner',\n",
       " 'brian',\n",
       " 'briancox',\n",
       " 'briandepalma',\n",
       " 'bride',\n",
       " 'bridge',\n",
       " 'brief',\n",
       " 'brien',\n",
       " 'bright',\n",
       " 'brilliant',\n",
       " 'bring',\n",
       " 'bringing',\n",
       " 'brings',\n",
       " 'brink',\n",
       " 'britain',\n",
       " 'british',\n",
       " 'britishsecretservice',\n",
       " 'brittanymurphy',\n",
       " 'broadway',\n",
       " 'broke',\n",
       " 'broken',\n",
       " 'broker',\n",
       " 'brooklyn',\n",
       " 'brooks',\n",
       " 'brothel',\n",
       " 'brother',\n",
       " 'brotherbrotherrelationship',\n",
       " 'brothers',\n",
       " 'brothersisterrelationship',\n",
       " 'brought',\n",
       " 'brown',\n",
       " 'bruce',\n",
       " 'brucedern',\n",
       " 'brucegreenwood',\n",
       " 'brucewillis',\n",
       " 'brutal',\n",
       " 'brutality',\n",
       " 'brutally',\n",
       " 'bryansinger',\n",
       " 'buck',\n",
       " 'buddies',\n",
       " 'buddy',\n",
       " 'buddycomedy',\n",
       " 'budget',\n",
       " 'build',\n",
       " 'building',\n",
       " 'built',\n",
       " 'bully',\n",
       " 'bullying',\n",
       " 'bumbling',\n",
       " 'bunny',\n",
       " 'burglar',\n",
       " 'buried',\n",
       " 'bus',\n",
       " 'bush',\n",
       " 'business',\n",
       " 'businessman',\n",
       " 'bust',\n",
       " 'busy',\n",
       " 'butler',\n",
       " 'buy',\n",
       " 'cabin',\n",
       " 'caesar',\n",
       " 'cage',\n",
       " 'cairo',\n",
       " 'cal',\n",
       " 'california',\n",
       " 'called',\n",
       " 'calls',\n",
       " 'calvin',\n",
       " 'camcorder',\n",
       " 'came',\n",
       " 'camera',\n",
       " 'cameraman',\n",
       " 'cameras',\n",
       " 'camerondiaz',\n",
       " 'camgigandet',\n",
       " 'camp',\n",
       " 'campaign',\n",
       " 'campbell',\n",
       " 'camping',\n",
       " 'campus',\n",
       " 'canada',\n",
       " 'canadian',\n",
       " 'cancer',\n",
       " 'candidate',\n",
       " 'candy',\n",
       " 'cannibal',\n",
       " 'capable',\n",
       " 'capital',\n",
       " 'capitalism',\n",
       " 'capt',\n",
       " 'captain',\n",
       " 'captive',\n",
       " 'capture',\n",
       " 'captured',\n",
       " 'captures',\n",
       " 'car',\n",
       " 'caraccident',\n",
       " 'carchase',\n",
       " 'carcrash',\n",
       " 'card',\n",
       " 'care',\n",
       " 'career',\n",
       " 'carefree',\n",
       " 'caretaker',\n",
       " 'careymulligan',\n",
       " 'caribbean',\n",
       " 'carjourney',\n",
       " 'carl',\n",
       " 'carlagugino',\n",
       " 'carmen',\n",
       " 'carol',\n",
       " 'carolina',\n",
       " 'carpenter',\n",
       " 'carrace',\n",
       " 'carrie',\n",
       " 'carry',\n",
       " 'carrying',\n",
       " 'cars',\n",
       " 'cartel',\n",
       " 'carter',\n",
       " 'cartoon',\n",
       " 'caryelwes',\n",
       " 'case',\n",
       " 'caseyaffleck',\n",
       " 'cash',\n",
       " 'casino',\n",
       " 'cast',\n",
       " 'castle',\n",
       " 'cat',\n",
       " 'cataclysm',\n",
       " 'catastrophe',\n",
       " 'catch',\n",
       " 'catches',\n",
       " 'cateblanchett',\n",
       " 'catherinekeener',\n",
       " 'catherineo',\n",
       " 'catherinezeta',\n",
       " 'catholic',\n",
       " 'catholicism',\n",
       " 'cattle',\n",
       " 'caught',\n",
       " 'cause',\n",
       " 'caused',\n",
       " 'causes',\n",
       " 'causing',\n",
       " 'cavalry',\n",
       " 'cave',\n",
       " 'cedrictheentertainer',\n",
       " 'celebrate',\n",
       " 'celebrated',\n",
       " 'celebration',\n",
       " 'celebrity',\n",
       " 'cellphone',\n",
       " 'cemetery',\n",
       " 'center',\n",
       " 'centered',\n",
       " 'centers',\n",
       " 'central',\n",
       " 'centuries',\n",
       " 'century',\n",
       " 'ceremony',\n",
       " 'certain',\n",
       " 'chad',\n",
       " 'chain',\n",
       " 'chainsaw',\n",
       " 'challenge',\n",
       " 'challenged',\n",
       " 'challenges',\n",
       " 'champion',\n",
       " 'championship',\n",
       " 'chance',\n",
       " 'change',\n",
       " 'changed',\n",
       " 'changes',\n",
       " 'changing',\n",
       " 'channingtatum',\n",
       " 'chaos',\n",
       " 'chaotic',\n",
       " 'chapter',\n",
       " 'character',\n",
       " 'characters',\n",
       " 'charge',\n",
       " 'charged',\n",
       " 'charismatic',\n",
       " 'charles',\n",
       " 'charless',\n",
       " 'charlie',\n",
       " 'charliesheen',\n",
       " 'charlizetheron',\n",
       " 'charlotte',\n",
       " 'charm',\n",
       " 'charming',\n",
       " 'chase',\n",
       " 'chased',\n",
       " 'chauffeur',\n",
       " 'chazzpalminteri',\n",
       " 'cheating',\n",
       " 'cheerleader',\n",
       " 'chef',\n",
       " 'chemical',\n",
       " 'cher',\n",
       " 'chevychase',\n",
       " 'chicago',\n",
       " 'chicken',\n",
       " 'chief',\n",
       " 'child',\n",
       " 'childabuse',\n",
       " 'childhero',\n",
       " 'childhood',\n",
       " 'children',\n",
       " 'chilling',\n",
       " 'china',\n",
       " 'chinese',\n",
       " 'chip',\n",
       " 'chiwetelejiofor',\n",
       " 'chloe',\n",
       " 'chloëgracemoretz',\n",
       " 'chloësevigny',\n",
       " 'chocolate',\n",
       " 'choice',\n",
       " 'choices',\n",
       " 'choose',\n",
       " 'chosen',\n",
       " 'chowyun',\n",
       " 'chris',\n",
       " 'chriscolumbus',\n",
       " 'chriscooper',\n",
       " 'chrisevans',\n",
       " 'chrishemsworth',\n",
       " 'chrisklein',\n",
       " 'chrispine',\n",
       " 'chrisrock',\n",
       " 'christ',\n",
       " 'christian',\n",
       " 'christianbale',\n",
       " 'christianity',\n",
       " 'christianslater',\n",
       " 'christinaapplegate',\n",
       " 'christinaricci',\n",
       " 'christine',\n",
       " 'christmas',\n",
       " 'christmasparty',\n",
       " 'christmastree',\n",
       " 'christopher',\n",
       " 'christopherguest',\n",
       " 'christopherlambert',\n",
       " 'christopherlloyd',\n",
       " 'christophernolan',\n",
       " 'christopherplummer',\n",
       " 'christopherwalken',\n",
       " 'christophwaltz',\n",
       " 'chronicle',\n",
       " 'chronicles',\n",
       " 'chuck',\n",
       " 'church',\n",
       " 'cia',\n",
       " 'ciaránhinds',\n",
       " 'cigarettesmoking',\n",
       " 'cillianmurphy',\n",
       " 'cinema',\n",
       " 'circle',\n",
       " 'circuit',\n",
       " 'circumstances',\n",
       " 'circus',\n",
       " 'cities',\n",
       " 'citizens',\n",
       " 'city',\n",
       " 'civil',\n",
       " 'civilization',\n",
       " 'civilwar',\n",
       " 'claim',\n",
       " 'claims',\n",
       " 'claire',\n",
       " 'clairedanes',\n",
       " 'claireforlani',\n",
       " 'clan',\n",
       " 'clark',\n",
       " 'clash',\n",
       " 'class',\n",
       " 'classes',\n",
       " 'classic',\n",
       " 'classmate',\n",
       " 'classmates',\n",
       " 'classroom',\n",
       " 'claudevandamme',\n",
       " 'clay',\n",
       " 'clean',\n",
       " 'clear',\n",
       " 'clerk',\n",
       " 'client',\n",
       " 'clients',\n",
       " 'climate',\n",
       " 'climbing',\n",
       " 'clinteastwood',\n",
       " 'cliveowen',\n",
       " 'clock',\n",
       " 'clone',\n",
       " 'cloning',\n",
       " 'close',\n",
       " 'closed',\n",
       " 'closer',\n",
       " 'club',\n",
       " 'clubs',\n",
       " 'clue',\n",
       " 'clues',\n",
       " 'clutches',\n",
       " 'coach',\n",
       " 'coast',\n",
       " 'cocaine',\n",
       " 'code',\n",
       " 'cody',\n",
       " 'coffin',\n",
       " 'cohen',\n",
       " 'col',\n",
       " 'cold',\n",
       " 'coldwar',\n",
       " 'cole',\n",
       " 'colehauser',\n",
       " 'colin',\n",
       " 'colinfarrell',\n",
       " 'colinfirth',\n",
       " 'collapse',\n",
       " 'collapses',\n",
       " 'colleague',\n",
       " 'colleagues',\n",
       " 'collect',\n",
       " 'collection',\n",
       " 'collector',\n",
       " 'college',\n",
       " 'collide',\n",
       " 'collision',\n",
       " 'colonel',\n",
       " 'colony',\n",
       " 'color',\n",
       " 'colorado',\n",
       " 'colorful',\n",
       " 'coma',\n",
       " 'combat',\n",
       " 'combined',\n",
       " 'come',\n",
       " 'comeback',\n",
       " 'comedian',\n",
       " 'comedic',\n",
       " 'comedy',\n",
       " 'comes',\n",
       " 'comet',\n",
       " 'comfort',\n",
       " 'comic',\n",
       " 'comics',\n",
       " 'coming',\n",
       " 'comingofage',\n",
       " 'comingout',\n",
       " 'command',\n",
       " 'commander',\n",
       " 'commercial',\n",
       " 'commit',\n",
       " 'commitment',\n",
       " 'committed',\n",
       " 'common',\n",
       " 'communication',\n",
       " 'community',\n",
       " 'companion',\n",
       " 'company',\n",
       " 'compete',\n",
       " 'competing',\n",
       " 'competition',\n",
       " 'complete',\n",
       " 'completely',\n",
       " 'complex',\n",
       " 'complicated',\n",
       " 'complications',\n",
       " 'composer',\n",
       " 'computer',\n",
       " 'computervirus',\n",
       " 'conan',\n",
       " 'concert',\n",
       " 'conclusion',\n",
       " 'condition',\n",
       " 'confession',\n",
       " 'confidence',\n",
       " 'confident',\n",
       " 'conflict',\n",
       " 'confront',\n",
       " 'confronted',\n",
       " 'confused',\n",
       " 'congress',\n",
       " 'conman',\n",
       " 'connected',\n",
       " 'connection',\n",
       " 'connell',\n",
       " 'connienielsen',\n",
       " 'connor',\n",
       " 'conquer',\n",
       " 'conscience',\n",
       " 'consequences',\n",
       " 'conservative',\n",
       " 'considered',\n",
       " 'conspiracy',\n",
       " 'conspire',\n",
       " 'constant',\n",
       " 'constantly',\n",
       " 'construction',\n",
       " 'consumed',\n",
       " 'contact',\n",
       " 'contain',\n",
       " 'contemporary',\n",
       " 'contend',\n",
       " 'contest',\n",
       " 'continue',\n",
       " 'continues',\n",
       " 'continuing',\n",
       " 'contract',\n",
       " 'control',\n",
       " 'controlled',\n",
       " 'controlling',\n",
       " 'controversial',\n",
       " 'convention',\n",
       " 'converge',\n",
       " 'convict',\n",
       " 'convicted',\n",
       " 'convince',\n",
       " 'convinced',\n",
       " 'convinces',\n",
       " 'cook',\n",
       " 'cooking',\n",
       " 'cool',\n",
       " 'cooper',\n",
       " 'cop',\n",
       " 'cope',\n",
       " ...]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing all the 5000 most frequent words that we had extracted from the corpus\n",
    "cv.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "3ead3e14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# there is a problem with the common words that have been extracted above\n",
    "# ['commit', 'commitment', 'committed']\n",
    "# more examples of such words in our list:\n",
    "    # - ['abilities', 'ability']\n",
    "    # - ['accident', 'accidental', 'accidentally']\n",
    "    # - ['activist', 'activities', 'activity']\n",
    "# basically these 3 words are similar however they are being treated as different by our count vectorizer object\n",
    "# so we will be applying the 'stemming' functionality to help avoid this issue\n",
    "\n",
    "# How (stemming) works:\n",
    "# Suppose we have three different words ['loved', 'loving', 'love']\n",
    "# So what stemming does is -> it will transform these words to the root term\n",
    "# final output will be: ['love', 'love', 'love']\n",
    "# we will be applying this functionality just before implementing the count vectorization technique"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "711dcfce",
   "metadata": {},
   "source": [
    "#### APPLYING STEMMING FUNCTIONALITY & RE-RUNNING COUNT VECTORIZATION TECHNIQUE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "212a1b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "0cce4cf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "ps = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "728cf274",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining a helper function to implement the stemming functionality to our tag records\n",
    "\n",
    "def stem(text): # --- text = preprocessed_df['tags'][0]\n",
    "    y = []\n",
    "    for i in text.split(): # --- splitting the input text based on spaces\n",
    "        y.append(ps.stem(i)) # --- applying the stemming functionality on these list of words\n",
    "    return \" \".join(y) # --- finally joining all the words as a combined string and returning it back"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "1fc03e74",
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessed_df['tags'] = preprocessed_df['tags'].apply(stem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "72a67771",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       in the 22nd century, a parapleg marin is dispa...\n",
       "1       captain barbossa, long believ to be dead, ha c...\n",
       "2       a cryptic messag from bond’ past send him on a...\n",
       "3       follow the death of district attorney harvey d...\n",
       "4       john carter is a war-weary, former militari ca...\n",
       "                              ...                        \n",
       "4804    el mariachi just want to play hi guitar and ca...\n",
       "4805    a newlyw couple' honeymoon is upend by the arr...\n",
       "4806    \"signed, sealed, delivered\" introduc a dedic q...\n",
       "4807    when ambiti new york attorney sam is sent to s...\n",
       "4808    ever sinc the second grade when he first saw h...\n",
       "Name: tags, Length: 4806, dtype: object"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessed_df['tags']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "58f98844",
   "metadata": {},
   "outputs": [],
   "source": [
    "vectors = cv.fit_transform(preprocessed_df['tags']).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "72f34809",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vector matrix for the first movie: [0 0 0 ... 0 0 0]\n",
      "Vector matrix for the second movie: [0 0 0 ... 0 0 0]\n",
      "Vector matrix for the third movie: [0 0 0 ... 0 0 0]\n",
      "Vector matrix for the fourth movie: [0 0 0 ... 0 0 0]\n",
      "\n",
      "Vector matrix for the second last movie: [0 0 0 ... 0 0 0]\n",
      "Vector matrix for the last movie: [0 0 0 ... 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "print('Vector matrix for the first movie:', vectors[0])\n",
    "print('Vector matrix for the second movie:', vectors[1])\n",
    "print('Vector matrix for the third movie:', vectors[2])\n",
    "print('Vector matrix for the fourth movie:', vectors[3])\n",
    "print(\"\")\n",
    "print('Vector matrix for the second last movie:', vectors[4084])\n",
    "print('Vector matrix for the last movie:', vectors[4085])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "c39072b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['000',\n",
       " '007',\n",
       " '10',\n",
       " '100',\n",
       " '11',\n",
       " '12',\n",
       " '13',\n",
       " '14',\n",
       " '15',\n",
       " '16',\n",
       " '17',\n",
       " '18',\n",
       " '18th',\n",
       " '19',\n",
       " '1910',\n",
       " '1920',\n",
       " '1930',\n",
       " '1940',\n",
       " '1950',\n",
       " '1950s',\n",
       " '1960',\n",
       " '1960s',\n",
       " '1970',\n",
       " '1970s',\n",
       " '1980',\n",
       " '1985',\n",
       " '1990',\n",
       " '19th',\n",
       " '19thcenturi',\n",
       " '20',\n",
       " '200',\n",
       " '2009',\n",
       " '20th',\n",
       " '24',\n",
       " '25',\n",
       " '30',\n",
       " '300',\n",
       " '3d',\n",
       " '40',\n",
       " '50',\n",
       " '500',\n",
       " '60',\n",
       " '70',\n",
       " 'aaron',\n",
       " 'aaroneckhart',\n",
       " 'aarontaylor',\n",
       " 'abandon',\n",
       " 'abduct',\n",
       " 'abigailbreslin',\n",
       " 'abil',\n",
       " 'abl',\n",
       " 'aboard',\n",
       " 'abov',\n",
       " 'abus',\n",
       " 'academ',\n",
       " 'academi',\n",
       " 'accept',\n",
       " 'access',\n",
       " 'accid',\n",
       " 'accident',\n",
       " 'acclaim',\n",
       " 'accompani',\n",
       " 'accomplish',\n",
       " 'account',\n",
       " 'accus',\n",
       " 'ace',\n",
       " 'achiev',\n",
       " 'acquaint',\n",
       " 'act',\n",
       " 'action',\n",
       " 'actionhero',\n",
       " 'activ',\n",
       " 'activist',\n",
       " 'activities',\n",
       " 'actor',\n",
       " 'actress',\n",
       " 'actual',\n",
       " 'ad',\n",
       " 'adam',\n",
       " 'adambrodi',\n",
       " 'adamsandl',\n",
       " 'adamshankman',\n",
       " 'adapt',\n",
       " 'add',\n",
       " 'addict',\n",
       " 'adjust',\n",
       " 'admir',\n",
       " 'admit',\n",
       " 'adolesc',\n",
       " 'adopt',\n",
       " 'ador',\n",
       " 'adrienbrodi',\n",
       " 'adult',\n",
       " 'adultanim',\n",
       " 'adulteri',\n",
       " 'adulthood',\n",
       " 'advanc',\n",
       " 'adventur',\n",
       " 'adventure',\n",
       " 'adventures',\n",
       " 'advertis',\n",
       " 'advic',\n",
       " 'advis',\n",
       " 'affair',\n",
       " 'affect',\n",
       " 'afghanistan',\n",
       " 'africa',\n",
       " 'african',\n",
       " 'africanamerican',\n",
       " 'aftercreditssting',\n",
       " 'afterlif',\n",
       " 'aftermath',\n",
       " 'ag',\n",
       " 'age',\n",
       " 'agediffer',\n",
       " 'agenc',\n",
       " 'agency',\n",
       " 'agenda',\n",
       " 'agent',\n",
       " 'agents',\n",
       " 'aggress',\n",
       " 'ago',\n",
       " 'agre',\n",
       " 'ahead',\n",
       " 'aid',\n",
       " 'aidanquinn',\n",
       " 'ail',\n",
       " 'aim',\n",
       " 'air',\n",
       " 'airplan',\n",
       " 'airplanecrash',\n",
       " 'airport',\n",
       " 'aka',\n",
       " 'al',\n",
       " 'alabama',\n",
       " 'alan',\n",
       " 'alanalda',\n",
       " 'alanarkin',\n",
       " 'alaska',\n",
       " 'albert',\n",
       " 'albertbrook',\n",
       " 'alcohol',\n",
       " 'alecbaldwin',\n",
       " 'alex',\n",
       " 'alexkendrick',\n",
       " 'alfredhitchcock',\n",
       " 'alfredmolina',\n",
       " 'alfrewoodard',\n",
       " 'ali',\n",
       " 'alic',\n",
       " 'alice',\n",
       " 'alicebraga',\n",
       " 'alien',\n",
       " 'alieninvas',\n",
       " 'alienlife',\n",
       " 'alienplanet',\n",
       " 'aliens',\n",
       " 'alik',\n",
       " 'aliv',\n",
       " 'alive',\n",
       " 'allen',\n",
       " 'alli',\n",
       " 'allianc',\n",
       " 'allow',\n",
       " 'alon',\n",
       " 'alongsid',\n",
       " 'alpacino',\n",
       " 'alpha',\n",
       " 'alreadi',\n",
       " 'alter',\n",
       " 'altern',\n",
       " 'alway',\n",
       " 'alzheimer',\n",
       " 'amanda',\n",
       " 'amandapeet',\n",
       " 'amandaseyfri',\n",
       " 'amateur',\n",
       " 'amaz',\n",
       " 'ambassador',\n",
       " 'ambit',\n",
       " 'ambiti',\n",
       " 'ambul',\n",
       " 'ambush',\n",
       " 'america',\n",
       " 'american',\n",
       " 'americanabroad',\n",
       " 'americanfootbal',\n",
       " 'amid',\n",
       " 'amidst',\n",
       " 'amnesia',\n",
       " 'amp',\n",
       " 'amsterdam',\n",
       " 'amus',\n",
       " 'amusementpark',\n",
       " 'amy',\n",
       " 'amyadam',\n",
       " 'amysmart',\n",
       " 'analyst',\n",
       " 'anarchiccomedi',\n",
       " 'ancient',\n",
       " 'ancientrom',\n",
       " 'ancientworld',\n",
       " 'anderson',\n",
       " 'andi',\n",
       " 'andiemacdowel',\n",
       " 'andrew',\n",
       " 'android',\n",
       " 'andy',\n",
       " 'andygarcía',\n",
       " 'andyserki',\n",
       " 'angel',\n",
       " 'angela',\n",
       " 'angelabassett',\n",
       " 'angeles',\n",
       " 'angelinajoli',\n",
       " 'anger',\n",
       " 'angle',\n",
       " 'angri',\n",
       " 'ani',\n",
       " 'anim',\n",
       " 'animalattack',\n",
       " 'animalhorror',\n",
       " 'animals',\n",
       " 'anjelicahuston',\n",
       " 'ann',\n",
       " 'anna',\n",
       " 'annafari',\n",
       " 'annakendrick',\n",
       " 'annasophiarobb',\n",
       " 'anne',\n",
       " 'annehathaway',\n",
       " 'annemoss',\n",
       " 'annetteben',\n",
       " 'anni',\n",
       " 'annie',\n",
       " 'announc',\n",
       " 'annual',\n",
       " 'anonym',\n",
       " 'anoth',\n",
       " 'answer',\n",
       " 'ant',\n",
       " 'antholog',\n",
       " 'anthoni',\n",
       " 'anthonyanderson',\n",
       " 'anthonyhopkin',\n",
       " 'anthonymacki',\n",
       " 'anthropomorph',\n",
       " 'anti',\n",
       " 'antihero',\n",
       " 'antoinefuqua',\n",
       " 'antoniobandera',\n",
       " 'antonyelchin',\n",
       " 'anyon',\n",
       " 'anyth',\n",
       " 'apart',\n",
       " 'apartheid',\n",
       " 'apartment',\n",
       " 'ape',\n",
       " 'apocalyps',\n",
       " 'apocalypt',\n",
       " 'appar',\n",
       " 'appear',\n",
       " 'appl',\n",
       " 'appoint',\n",
       " 'appreci',\n",
       " 'apprentic',\n",
       " 'approach',\n",
       " 'april',\n",
       " 'aquarium',\n",
       " 'arab',\n",
       " 'arch',\n",
       " 'archaeologist',\n",
       " 'archeolog',\n",
       " 'archer',\n",
       " 'architect',\n",
       " 'arctic',\n",
       " 'area',\n",
       " 'aren',\n",
       " 'arena',\n",
       " 'argument',\n",
       " 'aris',\n",
       " 'aristocrat',\n",
       " 'arm',\n",
       " 'armi',\n",
       " 'armiehamm',\n",
       " 'armor',\n",
       " 'armsdeal',\n",
       " 'army',\n",
       " 'arnold',\n",
       " 'arnoldschwarzenegg',\n",
       " 'arrang',\n",
       " 'arrangedmarriag',\n",
       " 'arrest',\n",
       " 'arriv',\n",
       " 'arrog',\n",
       " 'art',\n",
       " 'arthur',\n",
       " 'artifact',\n",
       " 'artifici',\n",
       " 'artificialintellig',\n",
       " 'artist',\n",
       " 'ash',\n",
       " 'ashley',\n",
       " 'ashleyjudd',\n",
       " 'ashtonkutch',\n",
       " 'asia',\n",
       " 'asian',\n",
       " 'asid',\n",
       " 'ask',\n",
       " 'aspect',\n",
       " 'aspir',\n",
       " 'assassin',\n",
       " 'assault',\n",
       " 'assembl',\n",
       " 'assign',\n",
       " 'assist',\n",
       " 'assistant',\n",
       " 'associ',\n",
       " 'assum',\n",
       " 'asteroid',\n",
       " 'astronaut',\n",
       " 'asylum',\n",
       " 'atheist',\n",
       " 'athlet',\n",
       " 'atom',\n",
       " 'atomicbomb',\n",
       " 'attack',\n",
       " 'attacks',\n",
       " 'attempt',\n",
       " 'attempttoescap',\n",
       " 'attend',\n",
       " 'attent',\n",
       " 'attic',\n",
       " 'attitud',\n",
       " 'attorney',\n",
       " 'attract',\n",
       " 'auction',\n",
       " 'audienc',\n",
       " 'audit',\n",
       " 'august',\n",
       " 'aunt',\n",
       " 'austin',\n",
       " 'australia',\n",
       " 'australian',\n",
       " 'author',\n",
       " 'autism',\n",
       " 'auto',\n",
       " 'aveng',\n",
       " 'averag',\n",
       " 'avoid',\n",
       " 'await',\n",
       " 'awaken',\n",
       " 'award',\n",
       " 'away',\n",
       " 'awkward',\n",
       " 'awri',\n",
       " 'awry',\n",
       " 'ax',\n",
       " 'babe',\n",
       " 'babi',\n",
       " 'baby',\n",
       " 'bachelor',\n",
       " 'backdrop',\n",
       " 'background',\n",
       " 'backpack',\n",
       " 'bad',\n",
       " 'bag',\n",
       " 'bahama',\n",
       " 'bail',\n",
       " 'balanc',\n",
       " 'ball',\n",
       " 'ballet',\n",
       " 'balloon',\n",
       " 'band',\n",
       " 'bandit',\n",
       " 'bangkok',\n",
       " 'banish',\n",
       " 'bank',\n",
       " 'banker',\n",
       " 'bankrobb',\n",
       " 'bankrobberi',\n",
       " 'bar',\n",
       " 'bare',\n",
       " 'bargain',\n",
       " 'barn',\n",
       " 'barney',\n",
       " 'baron',\n",
       " 'barri',\n",
       " 'barrylevinson',\n",
       " 'barrypepp',\n",
       " 'bas',\n",
       " 'base',\n",
       " 'basebal',\n",
       " 'basedoncomicbook',\n",
       " 'basedongraphicnovel',\n",
       " 'basedonnovel',\n",
       " 'basedonplay',\n",
       " 'basedonstagemus',\n",
       " 'basedontrueev',\n",
       " 'basedontruestori',\n",
       " 'basedontvseri',\n",
       " 'basedonvideogam',\n",
       " 'basedonyoungadultnovel',\n",
       " 'basement',\n",
       " 'basketbal',\n",
       " 'batman',\n",
       " 'battl',\n",
       " 'battle',\n",
       " 'battlefield',\n",
       " 'bay',\n",
       " 'beach',\n",
       " 'bear',\n",
       " 'beast',\n",
       " 'beat',\n",
       " 'beauti',\n",
       " 'beautiful',\n",
       " 'beautifulwoman',\n",
       " 'beauty',\n",
       " 'becam',\n",
       " 'becaus',\n",
       " 'becom',\n",
       " 'becominganadult',\n",
       " 'bed',\n",
       " 'bedroom',\n",
       " 'beer',\n",
       " 'befor',\n",
       " 'befriend',\n",
       " 'began',\n",
       " 'begin',\n",
       " 'begins',\n",
       " 'behavior',\n",
       " 'belief',\n",
       " 'believ',\n",
       " 'bell',\n",
       " 'bella',\n",
       " 'belong',\n",
       " 'belov',\n",
       " 'ben',\n",
       " 'benaffleck',\n",
       " 'bend',\n",
       " 'beneath',\n",
       " 'benefit',\n",
       " 'benfost',\n",
       " 'beniciodeltoro',\n",
       " 'benjamin',\n",
       " 'benjaminbratt',\n",
       " 'benkingsley',\n",
       " 'bennett',\n",
       " 'benstil',\n",
       " 'bent',\n",
       " 'berlin',\n",
       " 'berniemac',\n",
       " 'best',\n",
       " 'bestfriend',\n",
       " 'bestfriendsinlov',\n",
       " 'bet',\n",
       " 'beth',\n",
       " 'betray',\n",
       " 'bettemidl',\n",
       " 'better',\n",
       " 'betti',\n",
       " 'beverli',\n",
       " 'bibl',\n",
       " 'bid',\n",
       " 'big',\n",
       " 'bigger',\n",
       " 'biggest',\n",
       " 'bike',\n",
       " 'biker',\n",
       " 'bikini',\n",
       " 'billhad',\n",
       " 'billi',\n",
       " 'billionair',\n",
       " 'billmurray',\n",
       " 'billnighi',\n",
       " 'billpaxton',\n",
       " 'billpullman',\n",
       " 'billybobthornton',\n",
       " 'billyconnolli',\n",
       " 'billycrudup',\n",
       " 'billycryst',\n",
       " 'biographi',\n",
       " 'biolog',\n",
       " 'bird',\n",
       " 'birth',\n",
       " 'birthday',\n",
       " 'bisexu',\n",
       " 'bishop',\n",
       " 'bit',\n",
       " 'bite',\n",
       " 'bitter',\n",
       " 'bizarr',\n",
       " 'black',\n",
       " 'blackmag',\n",
       " 'blackmail',\n",
       " 'blackpeopl',\n",
       " 'blade',\n",
       " 'blame',\n",
       " 'blend',\n",
       " 'blind',\n",
       " 'bliss',\n",
       " 'block',\n",
       " 'blond',\n",
       " 'blood',\n",
       " 'bloodi',\n",
       " 'bloodsplatt',\n",
       " 'bloodthirsti',\n",
       " 'blow',\n",
       " 'blue',\n",
       " 'board',\n",
       " 'boardingschool',\n",
       " 'boat',\n",
       " 'bob',\n",
       " 'bobbi',\n",
       " 'bobbyfarrelli',\n",
       " 'bobhoskin',\n",
       " 'bodi',\n",
       " 'body',\n",
       " 'bodyguard',\n",
       " 'bold',\n",
       " 'bollywood',\n",
       " 'bomb',\n",
       " 'bond',\n",
       " 'bone',\n",
       " 'bonniehunt',\n",
       " 'book',\n",
       " 'border',\n",
       " 'bore',\n",
       " 'boredom',\n",
       " 'born',\n",
       " 'boss',\n",
       " 'boston',\n",
       " 'botch',\n",
       " 'bound',\n",
       " 'boundari',\n",
       " 'bounti',\n",
       " 'bountyhunt',\n",
       " 'box',\n",
       " 'boxer',\n",
       " 'boy',\n",
       " 'boyfriend',\n",
       " 'boys',\n",
       " 'bradleycoop',\n",
       " 'bradpitt',\n",
       " 'brain',\n",
       " 'brand',\n",
       " 'brave',\n",
       " 'braveri',\n",
       " 'brazil',\n",
       " 'brazilian',\n",
       " 'break',\n",
       " 'breakdown',\n",
       " 'breakup',\n",
       " 'breast',\n",
       " 'breath',\n",
       " 'breckinmey',\n",
       " 'breed',\n",
       " 'brendanfras',\n",
       " 'brendangleeson',\n",
       " 'brent',\n",
       " 'brettratn',\n",
       " 'brian',\n",
       " 'briancox',\n",
       " 'briandepalma',\n",
       " 'bride',\n",
       " 'bridesmaid',\n",
       " 'bridg',\n",
       " 'brief',\n",
       " 'brien',\n",
       " 'bright',\n",
       " 'brilliant',\n",
       " 'bring',\n",
       " 'brink',\n",
       " 'britain',\n",
       " 'british',\n",
       " 'britishsecretservic',\n",
       " 'brittanymurphi',\n",
       " 'broadcast',\n",
       " 'broadway',\n",
       " 'broke',\n",
       " 'broken',\n",
       " 'broker',\n",
       " 'brook',\n",
       " 'brooklyn',\n",
       " 'broom',\n",
       " 'brothel',\n",
       " 'brother',\n",
       " 'brotherbrotherrelationship',\n",
       " 'brothers',\n",
       " 'brothersisterrelationship',\n",
       " 'brought',\n",
       " 'brown',\n",
       " 'bruce',\n",
       " 'brucecampbel',\n",
       " 'brucedern',\n",
       " 'brucegreenwood',\n",
       " 'brucewilli',\n",
       " 'brutal',\n",
       " 'bryansing',\n",
       " 'brycedallashoward',\n",
       " 'bu',\n",
       " 'buck',\n",
       " 'bud',\n",
       " 'buddi',\n",
       " 'buddy',\n",
       " 'buddycomedi',\n",
       " 'buddycop',\n",
       " 'budget',\n",
       " 'build',\n",
       " 'building',\n",
       " 'built',\n",
       " 'bullet',\n",
       " 'bulli',\n",
       " 'bumbl',\n",
       " 'bunch',\n",
       " 'bunker',\n",
       " 'bunni',\n",
       " 'burglar',\n",
       " 'buri',\n",
       " 'burn',\n",
       " 'burtreynold',\n",
       " 'bush',\n",
       " 'busi',\n",
       " 'business',\n",
       " 'businessman',\n",
       " 'bust',\n",
       " 'butcher',\n",
       " 'butler',\n",
       " 'butt',\n",
       " 'button',\n",
       " 'buy',\n",
       " 'buzz',\n",
       " 'cabin',\n",
       " 'caesar',\n",
       " 'cage',\n",
       " 'cairo',\n",
       " 'cal',\n",
       " 'california',\n",
       " 'calvin',\n",
       " 'camcord',\n",
       " 'came',\n",
       " 'camera',\n",
       " 'cameraman',\n",
       " 'camerondiaz',\n",
       " 'camgigandet',\n",
       " 'camp',\n",
       " 'campaign',\n",
       " 'campbell',\n",
       " 'campu',\n",
       " 'canada',\n",
       " 'canadian',\n",
       " 'cancer',\n",
       " 'candid',\n",
       " 'canin',\n",
       " 'cannib',\n",
       " 'canuxploit',\n",
       " 'capabl',\n",
       " 'caper',\n",
       " 'capit',\n",
       " 'capt',\n",
       " 'captain',\n",
       " 'captiv',\n",
       " 'captur',\n",
       " 'capture',\n",
       " 'car',\n",
       " 'caraccid',\n",
       " 'carchas',\n",
       " 'carcrash',\n",
       " 'card',\n",
       " 'care',\n",
       " 'career',\n",
       " 'carefre',\n",
       " 'caretak',\n",
       " 'careymulligan',\n",
       " 'caribbean',\n",
       " 'carjourney',\n",
       " 'carl',\n",
       " 'carlagugino',\n",
       " 'carmen',\n",
       " 'carol',\n",
       " 'carolina',\n",
       " 'carrac',\n",
       " 'carri',\n",
       " 'carrie',\n",
       " 'cartel',\n",
       " 'carter',\n",
       " 'cartoon',\n",
       " 'caryelw',\n",
       " 'case',\n",
       " 'caseyaffleck',\n",
       " 'cash',\n",
       " 'casino',\n",
       " 'cast',\n",
       " 'castl',\n",
       " 'cat',\n",
       " 'cataclysm',\n",
       " 'catastroph',\n",
       " 'catch',\n",
       " 'cateblanchett',\n",
       " 'catherinedeneuv',\n",
       " 'catherinekeen',\n",
       " 'catherineo',\n",
       " 'catherinezeta',\n",
       " 'cathol',\n",
       " 'catholic',\n",
       " 'cattl',\n",
       " 'caught',\n",
       " 'caus',\n",
       " 'cavalri',\n",
       " 'cave',\n",
       " 'cavemen',\n",
       " 'cedrictheentertain',\n",
       " 'celebr',\n",
       " 'celebration',\n",
       " 'cell',\n",
       " 'cellphon',\n",
       " 'cemeteri',\n",
       " 'center',\n",
       " 'centr',\n",
       " 'central',\n",
       " 'centuri',\n",
       " 'centuries',\n",
       " 'century',\n",
       " 'ceo',\n",
       " 'certain',\n",
       " 'chad',\n",
       " 'chain',\n",
       " 'chainsaw',\n",
       " 'challeng',\n",
       " 'chamber',\n",
       " 'champion',\n",
       " 'championship',\n",
       " 'chanc',\n",
       " 'chance',\n",
       " 'chang',\n",
       " 'change',\n",
       " 'changed',\n",
       " 'changes',\n",
       " 'channingtatum',\n",
       " 'chao',\n",
       " 'chaos',\n",
       " 'chaotic',\n",
       " 'chapter',\n",
       " 'charact',\n",
       " 'character',\n",
       " 'characters',\n",
       " 'charg',\n",
       " 'charismat',\n",
       " 'charl',\n",
       " 'charless',\n",
       " 'charli',\n",
       " 'charlie',\n",
       " 'charliesheen',\n",
       " 'charlizetheron',\n",
       " 'charm',\n",
       " 'chart',\n",
       " 'chase',\n",
       " 'chauffeur',\n",
       " 'chazzpalminteri',\n",
       " 'cheat',\n",
       " 'check',\n",
       " 'cheerlead',\n",
       " 'chef',\n",
       " 'chemic',\n",
       " 'cher',\n",
       " 'chevychas',\n",
       " 'chicago',\n",
       " 'chicken',\n",
       " 'chief',\n",
       " 'child',\n",
       " 'childabus',\n",
       " 'childhero',\n",
       " 'childhood',\n",
       " 'childprodigi',\n",
       " 'children',\n",
       " 'chill',\n",
       " 'chimp',\n",
       " 'china',\n",
       " 'chines',\n",
       " 'chip',\n",
       " 'chipmunk',\n",
       " 'chiwetelejiofor',\n",
       " 'chloe',\n",
       " 'chloëgracemoretz',\n",
       " 'chloësevigni',\n",
       " 'chocol',\n",
       " 'choic',\n",
       " 'choice',\n",
       " 'choos',\n",
       " 'chosen',\n",
       " 'chowyun',\n",
       " 'chri',\n",
       " 'chriscolumbu',\n",
       " 'chriscoop',\n",
       " 'chrisevan',\n",
       " 'chrishemsworth',\n",
       " 'chrisklein',\n",
       " 'chrispin',\n",
       " 'chrispratt',\n",
       " 'chrisrock',\n",
       " 'christ',\n",
       " 'christian',\n",
       " 'christianbal',\n",
       " 'christianslat',\n",
       " 'christin',\n",
       " 'christinaappleg',\n",
       " 'christinaricci',\n",
       " 'christma',\n",
       " 'christmas',\n",
       " 'christmasparti',\n",
       " 'christmastre',\n",
       " 'christoph',\n",
       " 'christopherguest',\n",
       " 'christopherlambert',\n",
       " 'christopherlloyd',\n",
       " 'christophernolan',\n",
       " 'christopherplumm',\n",
       " 'christopherwalken',\n",
       " 'christophwaltz',\n",
       " 'chrisweitz',\n",
       " 'chronicl',\n",
       " 'chuck',\n",
       " 'church',\n",
       " 'cia',\n",
       " 'ciaránhind',\n",
       " 'cigarettesmok',\n",
       " 'cillianmurphi',\n",
       " 'cinema',\n",
       " 'circl',\n",
       " 'circu',\n",
       " 'circuit',\n",
       " 'circumst',\n",
       " 'citi',\n",
       " 'citizen',\n",
       " 'city',\n",
       " 'civil',\n",
       " 'civilian',\n",
       " 'civilwar',\n",
       " 'claim',\n",
       " 'clair',\n",
       " 'clairedan',\n",
       " 'claireforlani',\n",
       " 'clan',\n",
       " 'clark',\n",
       " 'clash',\n",
       " 'class',\n",
       " 'classdiffer',\n",
       " 'classic',\n",
       " 'classmat',\n",
       " 'classroom',\n",
       " 'claudevandamm',\n",
       " 'clay',\n",
       " 'cleaduval',\n",
       " 'clean',\n",
       " 'clear',\n",
       " 'clerk',\n",
       " 'clever',\n",
       " 'client',\n",
       " 'clients',\n",
       " 'cliff',\n",
       " 'climat',\n",
       " 'climb',\n",
       " 'clinteastwood',\n",
       " 'cliveowen',\n",
       " 'clock',\n",
       " 'clone',\n",
       " 'close',\n",
       " 'closer',\n",
       " 'cloud',\n",
       " 'clown',\n",
       " 'club',\n",
       " 'clue',\n",
       " 'clueless',\n",
       " 'clutch',\n",
       " 'coach',\n",
       " 'coast',\n",
       " 'cocain',\n",
       " 'code',\n",
       " 'coffin',\n",
       " 'cohen',\n",
       " 'col',\n",
       " 'cold',\n",
       " 'coldwar',\n",
       " 'cole',\n",
       " 'colehaus',\n",
       " 'colin',\n",
       " 'colinfarrel',\n",
       " 'colinfirth',\n",
       " 'collaps',\n",
       " 'colleagu',\n",
       " 'collect',\n",
       " 'collector',\n",
       " 'colleg',\n",
       " 'college',\n",
       " 'collid',\n",
       " 'collis',\n",
       " 'colombia',\n",
       " 'colonel',\n",
       " 'coloni',\n",
       " 'color',\n",
       " 'colorado',\n",
       " 'coma',\n",
       " 'combat',\n",
       " 'combin',\n",
       " 'come',\n",
       " 'comeback',\n",
       " 'comed',\n",
       " 'comedi',\n",
       " 'comedian',\n",
       " 'comedy',\n",
       " 'comet',\n",
       " 'comfort',\n",
       " 'comic',\n",
       " 'coming',\n",
       " 'comingofag',\n",
       " 'comingout',\n",
       " 'command',\n",
       " 'commando',\n",
       " 'commerci',\n",
       " 'commiss',\n",
       " 'commit',\n",
       " 'common',\n",
       " 'commun',\n",
       " 'communist',\n",
       " 'community',\n",
       " 'compani',\n",
       " 'companion',\n",
       " 'company',\n",
       " 'compet',\n",
       " 'competit',\n",
       " 'competition',\n",
       " 'complet',\n",
       " 'complex',\n",
       " 'complic',\n",
       " 'compos',\n",
       " 'compuls',\n",
       " 'comput',\n",
       " 'computerviru',\n",
       " 'conan',\n",
       " 'concern',\n",
       " 'concert',\n",
       " 'concoct',\n",
       " 'condit',\n",
       " 'condition',\n",
       " 'conduct',\n",
       " 'confeder',\n",
       " 'confess',\n",
       " 'confid',\n",
       " 'confin',\n",
       " 'conflict',\n",
       " 'confront',\n",
       " 'confus',\n",
       " 'congress',\n",
       " 'conman',\n",
       " 'connect',\n",
       " 'connecticut',\n",
       " 'connel',\n",
       " 'connienielsen',\n",
       " 'connor',\n",
       " 'conquer',\n",
       " 'consequ',\n",
       " 'consequences',\n",
       " 'conserv',\n",
       " 'consid',\n",
       " 'conspir',\n",
       " 'conspiraci',\n",
       " 'conspiracy',\n",
       " 'constant',\n",
       " 'constantli',\n",
       " 'construct',\n",
       " 'consum',\n",
       " 'contact',\n",
       " 'contain',\n",
       " 'contemporari',\n",
       " 'contend',\n",
       " 'content',\n",
       " 'contest',\n",
       " 'continu',\n",
       " 'contract',\n",
       " 'contractor',\n",
       " 'control',\n",
       " 'controversi',\n",
       " 'convent',\n",
       " 'converg',\n",
       " 'convers',\n",
       " 'convict',\n",
       " 'convinc',\n",
       " 'cook',\n",
       " 'cool',\n",
       " 'cooper',\n",
       " 'cop',\n",
       " 'cope',\n",
       " 'cops',\n",
       " 'core',\n",
       " 'corner',\n",
       " 'corp',\n",
       " 'corpor',\n",
       " 'corporation',\n",
       " 'corps',\n",
       " 'corrupt',\n",
       " ...]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "be01ee10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (4806, 5000)\n"
     ]
    }
   ],
   "source": [
    "print('Shape:', vectors.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "bcc8393d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the problem of repetition of common words in our tags feature has been resolved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "230a5b0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now that we have all our 4806 movies mapped as vectors on a 5000-dimensional plane\n",
    "# we will measure the distance between each of these vectors to check for similarity\n",
    "# we won't be measuring the 'Euclidean Distance' as it is not a reliable measure while dealing with a high dimensional plane\n",
    "# so we would be using the 'Cosine Similarity' technique to measure distance and check for similarity\n",
    "# [Distance] is always inversely proportional to [Similarity]\n",
    "\n",
    "# for simplication (Cosine Similarity):\n",
    "    # we will compare the similarity between two movies using 'theta' value\n",
    "    # 'theta' is the angle between two movies\n",
    "    # higher 'theta' value = cosine distance is more --> movies are less similar to each other\n",
    "    # lower 'theta' value = cosine distance is less --> movies are much similar to each other"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "179f8f3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "ff883a2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity = cosine_similarity(vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "15b225ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4806, 4806)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "7713ca96",
   "metadata": {},
   "outputs": [],
   "source": [
    "# so basically every movie will be have its distance comapred against the rest of the movies\n",
    "    # movie_1 compared against [movie_1, movie_2, movie_3, ..... , movie_4086]\n",
    "    # movie_2 compared against [movie_1, movie_2, movie_3, ..... , movie_4086]\n",
    "    # movie_3 compared against [movie_1, movie_2, movie_3, ..... , movie_4086]\n",
    "    # ............\n",
    "    # ............\n",
    "    # movie_4086 compared against [movie_1, movie_2, movie_3, ..... , movie_4086]\n",
    "\n",
    "# hence we are receiving a (4086x4086) matrix as the resultant matrix having the distance values expressed between (0,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "efc754c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 0.08226127, 0.0860309 , ..., 0.04543109, 0.        ,\n",
       "        0.        ],\n",
       "       [0.08226127, 1.        , 0.05976143, ..., 0.02366905, 0.        ,\n",
       "        0.02577696],\n",
       "       [0.0860309 , 0.05976143, 1.        , ..., 0.02475369, 0.        ,\n",
       "        0.        ],\n",
       "       ...,\n",
       "       [0.04543109, 0.02366905, 0.02475369, ..., 1.        , 0.04174829,\n",
       "        0.04270814],\n",
       "       [0.        , 0.        , 0.        , ..., 0.04174829, 1.        ,\n",
       "        0.09093258],\n",
       "       [0.        , 0.02577696, 0.        , ..., 0.04270814, 0.09093258,\n",
       "        1.        ]])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "007cfeac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 0.08226127, 0.0860309 , ..., 0.04543109, 0.        ,\n",
       "       0.        ])"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6e8e67e",
   "metadata": {},
   "source": [
    "## SAMPLE UNDERSTANDING OF HOW THE RECOMMENDATION MODEL WOULD WORK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "918c202e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now we need to define a helper function that would request a movie from the user\n",
    "# and would fetch for top 6 similar movies and share them back to the user as recommendation\n",
    "\n",
    "# Steps to be followed under the helper function:\n",
    "    # based on user's request a movie is provided as an input to the function\n",
    "    # now we need to fetch the index value for this movie and store it under a variable\n",
    "    # based on the index value we then need to check for the similarity scores for this movie against other movies\n",
    "    # once the similarity array for that particular movie has been identified -- it then needs to sort the distances in decreasing order\n",
    "    \n",
    "    ## -- meanwhile while sorting the distances we also need to hold the index position for our requested movie so that the sorted values don't mismatch\n",
    "    ## for simplication:\n",
    "        ## for 'Avatar' --> cosine distance = 1 against itself\n",
    "        ## for 'Avatar' --> cosine distance = 0.08226127 against movie_2\n",
    "        ## for 'Avatar' --> cosine distance = 0.0860309 against movie_3\n",
    "        \n",
    "        ## so while sorting results will follow up as: [1, 0.0860309, 0.08226127]\n",
    "        ## however we have our movies in the sequence: [movie_1, movie_2, movie_3]\n",
    "        ## but the correct sequence should be: [movie_1, movie_3, movie_2] (based on distance)\n",
    "\n",
    "# so we need to ensure that the index position is not lost while sorting the distance values\n",
    "# so for this we would be using the enumerate() functionality and transform it into a list (as illustrated below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "b39f2133",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 1.0000000000000002),\n",
       " (1, 0.08226127456606226),\n",
       " (2, 0.08603090020146065),\n",
       " (3, 0.07300534327409847),\n",
       " (4, 0.1873171623163388),\n",
       " (5, 0.10743376064838502),\n",
       " (6, 0.04024218182927669),\n",
       " (7, 0.14509525002200235),\n",
       " (8, 0.05923488777590923),\n",
       " (9, 0.0967301666813349),\n",
       " (10, 0.101338918387628),\n",
       " (11, 0.09365858115816941),\n",
       " (12, 0.08885233166386385),\n",
       " (13, 0.044151078568834795),\n",
       " (14, 0.12824729401064427),\n",
       " (15, 0.06282808624375433),\n",
       " (16, 0.07894736842105264),\n",
       " (17, 0.13977653617040256),\n",
       " (18, 0.09558988911273408),\n",
       " (19, 0.0837707816583391),\n",
       " (20, 0.057807331301608),\n",
       " (21, 0.106676149412533),\n",
       " (22, 0.0662266178532522),\n",
       " (23, 0.08603090020146065),\n",
       " (24, 0.05407380704358751),\n",
       " (25, 0.05101627678885769),\n",
       " (26, 0.15389675281277312),\n",
       " (27, 0.18848425873126295),\n",
       " (28, 0.10968169942141635),\n",
       " (29, 0.065033247714309),\n",
       " (30, 0.06622661785325219),\n",
       " (31, 0.15609763526361567),\n",
       " (32, 0.08447772061910234),\n",
       " (33, 0.09544271444636668),\n",
       " (34, 0.0),\n",
       " (35, 0.09733285267845754),\n",
       " (36, 0.1692777916923361),\n",
       " (37, 0.07894736842105264),\n",
       " (38, 0.08111071056538127),\n",
       " (39, 0.08226127456606226),\n",
       " (40, 0.076004188790721),\n",
       " (41, 0.16390251702679645),\n",
       " (42, 0.0),\n",
       " (43, 0.09086217008485092),\n",
       " (44, 0.0331133089266261),\n",
       " (45, 0.08240856434303293),\n",
       " (46, 0.13710212427677043),\n",
       " (47, 0.19529164171612676),\n",
       " (48, 0.0830812984794528),\n",
       " (49, 0.057353933467640436),\n",
       " (50, 0.11128297681493143),\n",
       " (51, 0.0792740035289418),\n",
       " (52, 0.14509525002200235),\n",
       " (53, 0.044151078568834795),\n",
       " (54, 0.03065696697424829),\n",
       " (55, 0.0695678476053965),\n",
       " (56, 0.13530201829834768),\n",
       " (57, 0.024182541670333724),\n",
       " (58, 0.06684847767323797),\n",
       " (59, 0.07209840939145001),\n",
       " (60, 0.018367958959266125),\n",
       " (61, 0.23179316248638276),\n",
       " (62, 0.07868894753646337),\n",
       " (63, 0.08175191193132876),\n",
       " (64, 0.029617443887954616),\n",
       " (65, 0.019389168358237032),\n",
       " (66, 0.02249606353329238),\n",
       " (67, 0.16222142113076252),\n",
       " (68, 0.1204950466257556),\n",
       " (69, 0.02640278405633409),\n",
       " (70, 0.076004188790721),\n",
       " (71, 0.07254762501100118),\n",
       " (72, 0.13518451760896877),\n",
       " (73, 0.03382550457458692),\n",
       " (74, 0.20054543301971392),\n",
       " (75, 0.10390486669322622),\n",
       " (76, 0.07175473098524099),\n",
       " (77, 0.014338483366910109),\n",
       " (78, 0.050669459193814),\n",
       " (79, 0.0623109738595896),\n",
       " (80, 0.050464733268936676),\n",
       " (81, 0.1095080149111477),\n",
       " (82, 0.07421560439929402),\n",
       " (83, 0.17996850826633903),\n",
       " (84, 0.12361284651454937),\n",
       " (85, 0.11470786693528087),\n",
       " (86, 0.07024393586862707),\n",
       " (87, 0.11128297681493143),\n",
       " (88, 0.08885233166386385),\n",
       " (89, 0.05757929360284594),\n",
       " (90, 0.06243905410544627),\n",
       " (91, 0.16452254913212452),\n",
       " (92, 0.0623109738595896),\n",
       " (93, 0.07175473098524099),\n",
       " (94, 0.18074256993863339),\n",
       " (95, 0.14048787173725413),\n",
       " (96, 0.076004188790721),\n",
       " (97, 0.06952346619889824),\n",
       " (98, 0.07509392614826384),\n",
       " (99, 0.027036903521793755),\n",
       " (100, 0.06765100914917384),\n",
       " (101, 0.076004188790721),\n",
       " (102, 0.12262786789699313),\n",
       " (103, 0.07868894753646337),\n",
       " (104, 0.05564148840746572),\n",
       " (105, 0.10147651372376076),\n",
       " (106, 0.061806423257274694),\n",
       " (107, 0.061806423257274694),\n",
       " (108, 0.14509525002200235),\n",
       " (109, 0.0662266178532522),\n",
       " (110, 0.04456565178215865),\n",
       " (111, 0.14843120879858804),\n",
       " (112, 0.07792865001991967),\n",
       " (113, 0.042973504259354006),\n",
       " (114, 0.04543108504242546),\n",
       " (115, 0.044151078568834795),\n",
       " (116, 0.04783648732349399),\n",
       " (117, 0.06243905410544627),\n",
       " (118, 0.033463724070512735),\n",
       " (119, 0.042973504259354006),\n",
       " (120, 0.0),\n",
       " (121, 0.10968169942141635),\n",
       " (122, 0.11470786693528087),\n",
       " (123, 0.11141412945539662),\n",
       " (124, 0.06748819059987714),\n",
       " (125, 0.07817962860855646),\n",
       " (126, 0.13157894736842107),\n",
       " (127, 0.14509525002200235),\n",
       " (128, 0.0),\n",
       " (129, 0.09269795493186431),\n",
       " (130, 0.027036903521793755),\n",
       " (131, 0.10390486669322622),\n",
       " (132, 0.04499212706658476),\n",
       " (133, 0.028676966733820218),\n",
       " (134, 0.12725695259515557),\n",
       " (135, 0.0),\n",
       " (136, 0.024455799402225926),\n",
       " (137, 0.02999475137772317),\n",
       " (138, 0.12824729401064427),\n",
       " (139, 0.11980845957463077),\n",
       " (140, 0.04456565178215865),\n",
       " (141, 0.08603090020146065),\n",
       " (142, 0.08471737420873576),\n",
       " (143, 0.08749572785196143),\n",
       " (144, 0.05006261743217589),\n",
       " (145, 0.019252140716412975),\n",
       " (146, 0.016222142113076255),\n",
       " (147, 0.05564148840746572),\n",
       " (148, 0.09933992677987831),\n",
       " (149, 0.1540171257313038),\n",
       " (150, 0.16531163063339513),\n",
       " (151, 0.19867985355975665),\n",
       " (152, 0.08226127456606226),\n",
       " (153, 0.0781166566864507),\n",
       " (154, 0.06131393394849658),\n",
       " (155, 0.02043797798283219),\n",
       " (156, 0.060833032924035954),\n",
       " (157, 0.09933992677987831),\n",
       " (158, 0.13245323570650439),\n",
       " (159, 0.06243905410544627),\n",
       " (160, 0.11650496642725611),\n",
       " (161, 0.03181423814878889),\n",
       " (162, 0.03181423814878889),\n",
       " (163, 0.051298917604257706),\n",
       " (164, 0.07894736842105264),\n",
       " (165, 0.04499212706658476),\n",
       " (166, 0.12369267399882337),\n",
       " (167, 0.08740748201220976),\n",
       " (168, 0.17521916101261562),\n",
       " (169, 0.08830215713766959),\n",
       " (170, 0.10471347707292386),\n",
       " (171, 0.02271554252121273),\n",
       " (172, 0.21239769762143662),\n",
       " (173, 0.02294157338705618),\n",
       " (174, 0.076004188790721),\n",
       " (175, 0.07098727864204515),\n",
       " (176, 0.02913582733740325),\n",
       " (177, 0.0),\n",
       " (178, 0.024182541670333724),\n",
       " (179, 0.07254762501100116),\n",
       " (180, 0.021873931962990357),\n",
       " (181, 0.04456565178215865),\n",
       " (182, 0.11707322644771176),\n",
       " (183, 0.0695678476053965),\n",
       " (184, 0.17521916101261562),\n",
       " (185, 0.050669459193814),\n",
       " (186, 0.05901671065234752),\n",
       " (187, 0.152008377581442),\n",
       " (188, 0.022282825891079324),\n",
       " (189, 0.11164843913471803),\n",
       " (190, 0.0837707816583391),\n",
       " (191, 0.04836508334066745),\n",
       " (192, 0.047324852428030105),\n",
       " (193, 0.07878168645790681),\n",
       " (194, 0.0),\n",
       " (195, 0.0582716546748065),\n",
       " (196, 0.051298917604257706),\n",
       " (197, 0.03673591791853225),\n",
       " (198, 0.057353933467640436),\n",
       " (199, 0.08749572785196143),\n",
       " (200, 0.1037571695799112),\n",
       " (201, 0.0),\n",
       " (202, 0.031219527052723135),\n",
       " (203, 0.08226127456606226),\n",
       " (204, 0.05006261743217589),\n",
       " (205, 0.07443229275647868),\n",
       " (206, 0.11092107999698035),\n",
       " (207, 0.07694837640638656),\n",
       " (208, 0.12262786789699316),\n",
       " (209, 0.0662266178532522),\n",
       " (210, 0.07421560439929402),\n",
       " (211, 0.024455799402225926),\n",
       " (212, 0.06622661785325219),\n",
       " (213, 0.058587492514838024),\n",
       " (214, 0.0),\n",
       " (215, 0.06684847767323797),\n",
       " (216, 0.06488856845230502),\n",
       " (217, 0.08594700851870801),\n",
       " (218, 0.039344473768231684),\n",
       " (219, 0.09037128496931669),\n",
       " (220, 0.1748149640244195),\n",
       " (221, 0.04783648732349399),\n",
       " (222, 0.0792740035289418),\n",
       " (223, 0.10390486669322622),\n",
       " (224, 0.060363272743915036),\n",
       " (225, 0.046348977465932154),\n",
       " (226, 0.029617443887954616),\n",
       " (227, 0.08111071056538127),\n",
       " (228, 0.11037769642208699),\n",
       " (229, 0.11020775375559674),\n",
       " (230, 0.09764582085806336),\n",
       " (231, 0.0),\n",
       " (232, 0.11128297681493143),\n",
       " (233, 0.10619884881071831),\n",
       " (234, 0.08594700851870801),\n",
       " (235, 0.076004188790721),\n",
       " (236, 0.11470786693528087),\n",
       " (237, 0.06488856845230502),\n",
       " (238, 0.11314714296070713),\n",
       " (239, 0.09183979479633063),\n",
       " (240, 0.046829290579084706),\n",
       " (241, 0.08111071056538127),\n",
       " (242, 0.07894736842105264),\n",
       " (243, 0.09269795493186431),\n",
       " (244, 0.09365858115816941),\n",
       " (245, 0.08000711205939975),\n",
       " (246, 0.07543142864047142),\n",
       " (247, 0.0),\n",
       " (248, 0.04456565178215865),\n",
       " (249, 0.13834289277321493),\n",
       " (250, 0.0),\n",
       " (251, 0.040555355282690636),\n",
       " (252, 0.020121090914638345),\n",
       " (253, 0.065033247714309),\n",
       " (254, 0.06952346619889824),\n",
       " (255, 0.0),\n",
       " (256, 0.10882143751650175),\n",
       " (257, 0.052631578947368425),\n",
       " (258, 0.017595383576944672),\n",
       " (259, 0.06131393394849658),\n",
       " (260, 0.20073876713674155),\n",
       " (261, 0.024182541670333724),\n",
       " (262, 0.10650358071057624),\n",
       " (263, 0.09933992677987831),\n",
       " (264, 0.0),\n",
       " (265, 0.02742042485535409),\n",
       " (266, 0.0978231976089037),\n",
       " (267, 0.0662266178532522),\n",
       " (268, 0.052631578947368425),\n",
       " (269, 0.02913582733740325),\n",
       " (270, 0.08749572785196143),\n",
       " (271, 0.06952346619889824),\n",
       " (272, 0.028239124736245257),\n",
       " (273, 0.0207703246198632),\n",
       " (274, 0.08977310580745099),\n",
       " (275, 0.1433848336691011),\n",
       " (276, 0.06131393394849658),\n",
       " (277, 0.052786150730834015),\n",
       " (278, 0.11587244366483038),\n",
       " (279, 0.09269795493186431),\n",
       " (280, 0.0),\n",
       " (281, 0.06446025638903101),\n",
       " (282, 0.044151078568834795),\n",
       " (283, 0.02742042485535409),\n",
       " (284, 0.018251335818524617),\n",
       " (285, 0.018986581228931064),\n",
       " (286, 0.0781166566864507),\n",
       " (287, 0.025649458802128853),\n",
       " (288, 0.02742042485535409),\n",
       " (289, 0.04588314677411236),\n",
       " (290, 0.06243905410544627),\n",
       " (291, 0.04947706959952935),\n",
       " (292, 0.08740748201220976),\n",
       " (293, 0.02913582733740325),\n",
       " (294, 0.1411956236812263),\n",
       " (295, 0.10526315789473685),\n",
       " (296, 0.038778336716474064),\n",
       " (297, 0.044151078568834795),\n",
       " (298, 0.0),\n",
       " (299, 0.05901671065234752),\n",
       " (300, 0.17316974359835272),\n",
       " (301, 0.09304036594559835),\n",
       " (302, 0.07024393586862707),\n",
       " (303, 0.02782074420373286),\n",
       " (304, 0.06557412294705281),\n",
       " (305, 0.18848425873126295),\n",
       " (306, 0.08111071056538127),\n",
       " (307, 0.10827043935570556),\n",
       " (308, 0.0662266178532522),\n",
       " (309, 0.06131393394849658),\n",
       " (310, 0.030521015231310435),\n",
       " (311, 0.14843120879858804),\n",
       " (312, 0.10259783520851541),\n",
       " (313, 0.05484084971070818),\n",
       " (314, 0.0),\n",
       " (315, 0.08346223261119858),\n",
       " (316, 0.0),\n",
       " (317, 0.0),\n",
       " (318, 0.0),\n",
       " (319, 0.09733285267845754),\n",
       " (320, 0.0301237616564389),\n",
       " (321, 0.0),\n",
       " (322, 0.20519567041703082),\n",
       " (323, 0.02913582733740325),\n",
       " (324, 0.025649458802128853),\n",
       " (325, 0.09933992677987831),\n",
       " (326, 0.05564148840746572),\n",
       " (327, 0.027036903521793755),\n",
       " (328, 0.02666903735313325),\n",
       " (329, 0.07792865001991967),\n",
       " (330, 0.0989541391990587),\n",
       " (331, 0.09086217008485092),\n",
       " (332, 0.10814761408717502),\n",
       " (333, 0.024182541670333724),\n",
       " (334, 0.0967301666813349),\n",
       " (335, 0.04543108504242546),\n",
       " (336, 0.06917144638660747),\n",
       " (337, 0.04499212706658476),\n",
       " (338, 0.0),\n",
       " (339, 0.047324852428030105),\n",
       " (340, 0.04891159880445185),\n",
       " (341, 0.08226127456606226),\n",
       " (342, 0.1602550282040466),\n",
       " (343, 0.019252140716412975),\n",
       " (344, 0.029617443887954616),\n",
       " (345, 0.023174488732966077),\n",
       " (346, 0.0),\n",
       " (347, 0.0),\n",
       " (348, 0.0602475233128778),\n",
       " (349, 0.076004188790721),\n",
       " (350, 0.04836508334066745),\n",
       " (351, 0.0),\n",
       " (352, 0.02742042485535409),\n",
       " (353, 0.0415406492397264),\n",
       " (354, 0.021486752129677003),\n",
       " (355, 0.03650267163704923),\n",
       " (356, 0.08885233166386385),\n",
       " (357, 0.0),\n",
       " (358, 0.05923488777590923),\n",
       " (359, 0.031219527052723135),\n",
       " (360, 0.0207703246198632),\n",
       " (361, 0.07509392614826384),\n",
       " (362, 0.09544271444636668),\n",
       " (363, 0.09909250441117726),\n",
       " (364, 0.023414645289542353),\n",
       " (365, 0.07509392614826384),\n",
       " (366, 0.05647824947249051),\n",
       " (367, 0.0),\n",
       " (368, 0.12977713690461004),\n",
       " (369, 0.126673647984535),\n",
       " (370, 0.050669459193814),\n",
       " (371, 0.09567297464698798),\n",
       " (372, 0.047324852428030105),\n",
       " (373, 0.1691275228729346),\n",
       " (374, 0.03797316245786213),\n",
       " (375, 0.060833032924035954),\n",
       " (376, 0.046348977465932154),\n",
       " (377, 0.0),\n",
       " (378, 0.0),\n",
       " (379, 0.050464733268936676),\n",
       " (380, 0.05195243334661311),\n",
       " (381, 0.08111071056538127),\n",
       " (382, 0.0),\n",
       " (383, 0.043355498476206004),\n",
       " (384, 0.028676966733820218),\n",
       " (385, 0.0),\n",
       " (386, 0.021677749238103002),\n",
       " (387, 0.024182541670333724),\n",
       " (388, 0.0),\n",
       " (389, 0.08740748201220976),\n",
       " (390, 0.025976216673306556),\n",
       " (391, 0.056573571480353566),\n",
       " (392, 0.029617443887954616),\n",
       " (393, 0.0),\n",
       " (394, 0.024455799402225926),\n",
       " (395, 0.05195243334661311),\n",
       " (396, 0.043747863925980714),\n",
       " (397, 0.051298917604257706),\n",
       " (398, 0.0),\n",
       " (399, 0.023662426214015053),\n",
       " (400, 0.08885233166386385),\n",
       " (401, 0.03065696697424829),\n",
       " (402, 0.08111071056538127),\n",
       " (403, 0.0830812984794528),\n",
       " (404, 0.06243905410544627),\n",
       " (405, 0.05647824947249051),\n",
       " (406, 0.02060214108575823),\n",
       " (407, 0.021486752129677003),\n",
       " (408, 0.0331133089266261),\n",
       " (409, 0.0),\n",
       " (410, 0.0967301666813349),\n",
       " (411, 0.051298917604257706),\n",
       " (412, 0.09365858115816941),\n",
       " (413, 0.1433848336691011),\n",
       " (414, 0.029617443887954616),\n",
       " (415, 0.053418342734682205),\n",
       " (416, 0.050669459193814),\n",
       " (417, 0.03181423814878889),\n",
       " (418, 0.03458572319330373),\n",
       " (419, 0.06814662756363819),\n",
       " (420, 0.11141412945539662),\n",
       " (421, 0.023174488732966077),\n",
       " (422, 0.11295649894498103),\n",
       " (423, 0.04188539082916955),\n",
       " (424, 0.036973693332326786),\n",
       " (425, 0.06390214842634574),\n",
       " (426, 0.053418342734682205),\n",
       " (427, 0.02666903735313325),\n",
       " (428, 0.05546053999849018),\n",
       " (429, 0.0),\n",
       " (430, 0.050669459193814),\n",
       " (431, 0.024182541670333724),\n",
       " (432, 0.11470786693528089),\n",
       " (433, 0.03065696697424829),\n",
       " (434, 0.0),\n",
       " (435, 0.03539961627023944),\n",
       " (436, 0.0),\n",
       " (437, 0.025649458802128853),\n",
       " (438, 0.12091270835166862),\n",
       " (439, 0.025976216673306556),\n",
       " (440, 0.024738534799764674),\n",
       " (441, 0.05647824947249051),\n",
       " (442, 0.04588314677411236),\n",
       " (443, 0.06282808624375433),\n",
       " (444, 0.0),\n",
       " (445, 0.0989541391990587),\n",
       " (446, 0.02294157338705618),\n",
       " (447, 0.06446025638903101),\n",
       " (448, 0.021300716142115247),\n",
       " (449, 0.10968169942141635),\n",
       " (450, 0.020942695414584774),\n",
       " (451, 0.025031308716087945),\n",
       " (452, 0.03065696697424829),\n",
       " (453, 0.024455799402225926),\n",
       " (454, 0.11587244366483038),\n",
       " (455, 0.060833032924035954),\n",
       " (456, 0.042973504259354006),\n",
       " (457, 0.08603090020146065),\n",
       " (458, 0.043355498476206004),\n",
       " (459, 0.03382550457458692),\n",
       " (460, 0.023918243661746996),\n",
       " (461, 0.1411956236812263),\n",
       " (462, 0.032283269414765066),\n",
       " (463, 0.03244428422615251),\n",
       " (464, 0.0),\n",
       " (465, 0.0),\n",
       " (466, 0.14509525002200233),\n",
       " (467, 0.10559715077387793),\n",
       " (468, 0.02742042485535409),\n",
       " (469, 0.05990422978731538),\n",
       " (470, 0.04499212706658476),\n",
       " (471, 0.06390214842634574),\n",
       " (472, 0.10814761408717502),\n",
       " (473, 0.09176629354822471),\n",
       " (474, 0.0),\n",
       " (475, 0.043747863925980714),\n",
       " (476, 0.12977713690461004),\n",
       " (477, 0.0),\n",
       " (478, 0.05407380704358751),\n",
       " (479, 0.08849904067559859),\n",
       " (480, 0.0967301666813349),\n",
       " (481, 0.0),\n",
       " (482, 0.021873931962990357),\n",
       " (483, 0.12262786789699316),\n",
       " (484, 0.028676966733820218),\n",
       " (485, 0.05195243334661311),\n",
       " (486, 0.08031157077540445),\n",
       " (487, 0.12725695259515557),\n",
       " (488, 0.06692744814102547),\n",
       " (489, 0.0),\n",
       " (490, 0.14159846508095775),\n",
       " (491, 0.09909250441117726),\n",
       " (492, 0.076004188790721),\n",
       " (493, 0.030521015231310435),\n",
       " (494, 0.0),\n",
       " (495, 0.1873171623163388),\n",
       " (496, 0.0),\n",
       " (497, 0.0),\n",
       " (498, 0.06765100914917384),\n",
       " (499, 0.0),\n",
       " (500, 0.025976216673306556),\n",
       " (501, 0.08175191193132876),\n",
       " (502, 0.13245323570650439),\n",
       " (503, 0.07421560439929402),\n",
       " (504, 0.0),\n",
       " (505, 0.13530201829834768),\n",
       " (506, 0.03458572319330373),\n",
       " (507, 0.255608593705383),\n",
       " (508, 0.10259783520851541),\n",
       " (509, 0.021873931962990357),\n",
       " (510, 0.0989541391990587),\n",
       " (511, 0.09037128496931669),\n",
       " (512, 0.02742042485535409),\n",
       " (513, 0.15585730003983933),\n",
       " (514, 0.04891159880445185),\n",
       " (515, 0.024738534799764674),\n",
       " (516, 0.02782074420373286),\n",
       " (517, 0.0),\n",
       " (518, 0.043355498476206004),\n",
       " (519, 0.0),\n",
       " (520, 0.0),\n",
       " (521, 0.024738534799764674),\n",
       " (522, 0.024182541670333724),\n",
       " (523, 0.04783648732349399),\n",
       " (524, 0.0),\n",
       " (525, 0.046829290579084706),\n",
       " (526, 0.02782074420373286),\n",
       " (527, 0.03244428422615251),\n",
       " (528, 0.022282825891079324),\n",
       " (529, 0.16742770563222897),\n",
       " (530, 0.05407380704358751),\n",
       " (531, 0.04588314677411236),\n",
       " (532, 0.021300716142115247),\n",
       " (533, 0.0602475233128778),\n",
       " (534, 0.0837707816583391),\n",
       " (535, 0.0978231976089037),\n",
       " (536, 0.1124803176664619),\n",
       " (537, 0.06765100914917384),\n",
       " (538, 0.0),\n",
       " (539, 0.2467838236981868),\n",
       " (540, 0.07421560439929402),\n",
       " (541, 0.103851623099316),\n",
       " (542, 0.03721614637823934),\n",
       " (543, 0.07098727864204515),\n",
       " (544, 0.05484084971070818),\n",
       " (545, 0.024738534799764674),\n",
       " (546, 0.06362847629757779),\n",
       " (547, 0.08226127456606226),\n",
       " (548, 0.0),\n",
       " (549, 0.11959121830873498),\n",
       " (550, 0.024455799402225926),\n",
       " (551, 0.03771571432023571),\n",
       " (552, 0.0),\n",
       " (553, 0.028239124736245257),\n",
       " (554, 0.0),\n",
       " (555, 0.04499212706658475),\n",
       " (556, 0.027036903521793755),\n",
       " (557, 0.12515654358043973),\n",
       " (558, 0.0),\n",
       " (559, 0.0301237616564389),\n",
       " (560, 0.03650267163704923),\n",
       " (561, 0.03181423814878889),\n",
       " (562, 0.029617443887954616),\n",
       " (563, 0.025334729596907),\n",
       " (564, 0.08740748201220976),\n",
       " (565, 0.043355498476206004),\n",
       " (566, 0.023662426214015053),\n",
       " (567, 0.025031308716087945),\n",
       " (568, 0.1411956236812263),\n",
       " (569, 0.04783648732349399),\n",
       " (570, 0.0662266178532522),\n",
       " (571, 0.04223886030955117),\n",
       " (572, 0.10650358071057624),\n",
       " (573, 0.0207703246198632),\n",
       " (574, 0.023174488732966077),\n",
       " (575, 0.04499212706658476),\n",
       " (576, 0.05195243334661311),\n",
       " (577, 0.1976738731537168),\n",
       " (578, 0.037463432463267755),\n",
       " (579, 0.1433848336691011),\n",
       " (580, 0.024455799402225926),\n",
       " (581, 0.14808721943977307),\n",
       " (582, 0.24511108480187255),\n",
       " (583, 0.03771571432023571),\n",
       " (584, 0.07868894753646337),\n",
       " (585, 0.024182541670333724),\n",
       " (586, 0.04456565178215865),\n",
       " (587, 0.09365858115816941),\n",
       " (588, 0.025334729596907),\n",
       " (589, 0.1163350101494222),\n",
       " (590, 0.0301237616564389),\n",
       " (591, 0.06282808624375433),\n",
       " (592, 0.021873931962990357),\n",
       " (593, 0.0),\n",
       " (594, 0.04891159880445185),\n",
       " (595, 0.16390251702679648),\n",
       " (596, 0.09269795493186431),\n",
       " (597, 0.0582716546748065),\n",
       " (598, 0.03539961627023944),\n",
       " (599, 0.02742042485535409),\n",
       " (600, 0.06882472016116853),\n",
       " (601, 0.03328713326493031),\n",
       " (602, 0.09086217008485092),\n",
       " (603, 0.05407380704358751),\n",
       " (604, 0.0662266178532522),\n",
       " (605, 0.025334729596907),\n",
       " (606, 0.0),\n",
       " (607, 0.05695974368679319),\n",
       " (608, 0.0),\n",
       " (609, 0.047324852428030105),\n",
       " (610, 0.058038100008800934),\n",
       " (611, 0.02294157338705618),\n",
       " (612, 0.06335829046432676),\n",
       " (613, 0.03458572319330373),\n",
       " (614, 0.04836508334066745),\n",
       " (615, 0.06131393394849658),\n",
       " (616, 0.03458572319330373),\n",
       " (617, 0.05006261743217589),\n",
       " (618, 0.08671099695241201),\n",
       " (619, 0.0),\n",
       " (620, 0.0),\n",
       " (621, 0.06488856845230502),\n",
       " (622, 0.021119430154775586),\n",
       " (623, 0.0),\n",
       " (624, 0.10147651372376076),\n",
       " (625, 0.031219527052723135),\n",
       " (626, 0.08226127456606226),\n",
       " (627, 0.11164843913471803),\n",
       " (628, 0.0),\n",
       " (629, 0.04836508334066745),\n",
       " (630, 0.024455799402225926),\n",
       " (631, 0.05006261743217589),\n",
       " (632, 0.07792865001991967),\n",
       " (633, 0.022075539284417398),\n",
       " (634, 0.07336739820667779),\n",
       " (635, 0.06622661785325219),\n",
       " (636, 0.023414645289542353),\n",
       " (637, 0.023918243661746996),\n",
       " (638, 0.0331133089266261),\n",
       " (639, 0.0),\n",
       " (640, 0.05990422978731538),\n",
       " (641, 0.0),\n",
       " (642, 0.0),\n",
       " (643, 0.0989541391990587),\n",
       " (644, 0.04891159880445185),\n",
       " (645, 0.043747863925980714),\n",
       " (646, 0.0533380747062665),\n",
       " (647, 0.04783648732349399),\n",
       " (648, 0.029617443887954616),\n",
       " (649, 0.0),\n",
       " (650, 0.05195243334661311),\n",
       " (651, 0.0),\n",
       " (652, 0.042973504259354006),\n",
       " (653, 0.10882143751650175),\n",
       " (654, 0.10301070542879115),\n",
       " (655, 0.023918243661746996),\n",
       " (656, 0.0),\n",
       " (657, 0.03458572319330373),\n",
       " (658, 0.04836508334066745),\n",
       " (659, 0.025334729596907),\n",
       " (660, 0.09365858115816941),\n",
       " (661, 0.16481712868606585),\n",
       " (662, 0.0),\n",
       " (663, 0.0),\n",
       " (664, 0.0602475233128778),\n",
       " (665, 0.06748819059987714),\n",
       " (666, 0.05006261743217589),\n",
       " (667, 0.0),\n",
       " (668, 0.1324532357065044),\n",
       " (669, 0.03382550457458692),\n",
       " (670, 0.017195436249022724),\n",
       " (671, 0.06814662756363819),\n",
       " (672, 0.028239124736245257),\n",
       " (673, 0.019968076595771794),\n",
       " (674, 0.07024393586862707),\n",
       " (675, 0.04456565178215865),\n",
       " (676, 0.028676966733820218),\n",
       " (677, 0.025031308716087945),\n",
       " (678, 0.116543309349613),\n",
       " (679, 0.0),\n",
       " (680, 0.07509392614826384),\n",
       " (681, 0.027036903521793755),\n",
       " (682, 0.019968076595771794),\n",
       " (683, 0.04783648732349399),\n",
       " (684, 0.0),\n",
       " (685, 0.03181423814878889),\n",
       " (686, 0.031219527052723135),\n",
       " (687, 0.10526315789473685),\n",
       " (688, 0.042973504259354006),\n",
       " (689, 0.0),\n",
       " (690, 0.024455799402225926),\n",
       " (691, 0.057353933467640436),\n",
       " (692, 0.10838874619051501),\n",
       " (693, 0.03382550457458692),\n",
       " (694, 0.021677749238103002),\n",
       " (695, 0.06488856845230502),\n",
       " (696, 0.021300716142115247),\n",
       " (697, 0.0),\n",
       " (698, 0.043355498476206004),\n",
       " (699, 0.03721614637823934),\n",
       " (700, 0.028239124736245257),\n",
       " (701, 0.057353933467640436),\n",
       " (702, 0.022282825891079324),\n",
       " (703, 0.021486752129677003),\n",
       " (704, 0.06748819059987714),\n",
       " (705, 0.024182541670333724),\n",
       " (706, 0.021486752129677003),\n",
       " (707, 0.0),\n",
       " (708, 0.024738534799764674),\n",
       " (709, 0.02043797798283219),\n",
       " (710, 0.05775642214923893),\n",
       " (711, 0.05195243334661311),\n",
       " (712, 0.0),\n",
       " (713, 0.024455799402225926),\n",
       " (714, 0.0),\n",
       " (715, 0.09733285267845754),\n",
       " (716, 0.02742042485535409),\n",
       " (717, 0.0),\n",
       " (718, 0.044151078568834795),\n",
       " (719, 0.0),\n",
       " (720, 0.025976216673306556),\n",
       " (721, 0.03181423814878889),\n",
       " (722, 0.0),\n",
       " (723, 0.024738534799764674),\n",
       " (724, 0.023414645289542353),\n",
       " (725, 0.0),\n",
       " (726, 0.0),\n",
       " (727, 0.030794751789719334),\n",
       " (728, 0.07098727864204515),\n",
       " (729, 0.04783648732349399),\n",
       " (730, 0.030254451040802585),\n",
       " (731, 0.05195243334661311),\n",
       " (732, 0.06814662756363819),\n",
       " (733, 0.061313933948496574),\n",
       " (734, 0.06748819059987714),\n",
       " (735, 0.06622661785325219),\n",
       " (736, 0.02913582733740325),\n",
       " (737, 0.07421560439929402),\n",
       " (738, 0.0),\n",
       " (739, 0.03823595564509363),\n",
       " (740, 0.10743376064838502),\n",
       " (741, 0.05407380704358751),\n",
       " (742, 0.0301237616564389),\n",
       " (743, 0.023174488732966077),\n",
       " (744, 0.06362847629757779),\n",
       " (745, 0.08603090020146065),\n",
       " (746, 0.06390214842634574),\n",
       " (747, 0.03905832834322535),\n",
       " (748, 0.03458572319330373),\n",
       " (749, 0.0),\n",
       " (750, 0.04223886030955117),\n",
       " (751, 0.031219527052723135),\n",
       " (752, 0.08671099695241201),\n",
       " (753, 0.02742042485535409),\n",
       " (754, 0.12515654358043973),\n",
       " (755, 0.058587492514838024),\n",
       " (756, 0.028239124736245257),\n",
       " (757, 0.023918243661746996),\n",
       " (758, 0.02249606353329238),\n",
       " (759, 0.0),\n",
       " (760, 0.0),\n",
       " (761, 0.028239124736245257),\n",
       " (762, 0.05923488777590923),\n",
       " (763, 0.0),\n",
       " (764, 0.02742042485535409),\n",
       " (765, 0.024455799402225926),\n",
       " (766, 0.03181423814878889),\n",
       " (767, 0.046348977465932154),\n",
       " (768, 0.060363272743915036),\n",
       " (769, 0.03065696697424829),\n",
       " (770, 0.042973504259354006),\n",
       " (771, 0.0),\n",
       " (772, 0.0),\n",
       " (773, 0.12725695259515557),\n",
       " (774, 0.03823595564509363),\n",
       " (775, 0.08603090020146065),\n",
       " (776, 0.07647191129018727),\n",
       " (777, 0.13369695534647594),\n",
       " (778, 0.2294157338705618),\n",
       " (779, 0.0),\n",
       " (780, 0.0),\n",
       " (781, 0.11846977555181847),\n",
       " (782, 0.06814662756363819),\n",
       " (783, 0.03181423814878889),\n",
       " (784, 0.07254762501100118),\n",
       " (785, 0.09243423333081695),\n",
       " (786, 0.10814761408717502),\n",
       " (787, 0.08111071056538127),\n",
       " (788, 0.046348977465932154),\n",
       " (789, 0.0),\n",
       " (790, 0.07254762501100116),\n",
       " (791, 0.09544271444636668),\n",
       " (792, 0.051298917604257706),\n",
       " (793, 0.050669459193814),\n",
       " (794, 0.08885233166386385),\n",
       " (795, 0.021873931962990357),\n",
       " (796, 0.0),\n",
       " (797, 0.03539961627023944),\n",
       " (798, 0.0),\n",
       " (799, 0.09183979479633063),\n",
       " (800, 0.07443229275647868),\n",
       " (801, 0.0),\n",
       " (802, 0.0),\n",
       " (803, 0.047324852428030105),\n",
       " (804, 0.051298917604257706),\n",
       " (805, 0.05484084971070818),\n",
       " (806, 0.04456565178215865),\n",
       " (807, 0.05407380704358751),\n",
       " (808, 0.11164843913471803),\n",
       " (809, 0.019389168358237032),\n",
       " (810, 0.028239124736245257),\n",
       " (811, 0.020942695414584774),\n",
       " (812, 0.08111071056538127),\n",
       " (813, 0.09176629354822471),\n",
       " (814, 0.10526315789473685),\n",
       " (815, 0.02043797798283219),\n",
       " (816, 0.04783648732349399),\n",
       " (817, 0.027036903521793755),\n",
       " (818, 0.024738534799764674),\n",
       " (819, 0.0),\n",
       " (820, 0.01583118967153259),\n",
       " (821, 0.015907119074394446),\n",
       " (822, 0.020942695414584774),\n",
       " (823, 0.04783648732349399),\n",
       " (824, 0.0),\n",
       " (825, 0.0),\n",
       " (826, 0.029617443887954616),\n",
       " (827, 0.08671099695241201),\n",
       " (828, 0.025334729596907),\n",
       " (829, 0.0),\n",
       " (830, 0.02666903735313325),\n",
       " (831, 0.047324852428030105),\n",
       " (832, 0.025976216673306556),\n",
       " (833, 0.07254762501100118),\n",
       " (834, 0.046348977465932154),\n",
       " (835, 0.025334729596907),\n",
       " (836, 0.061806423257274694),\n",
       " (837, 0.05564148840746572),\n",
       " (838, 0.1518926498314485),\n",
       " (839, 0.0),\n",
       " (840, 0.05564148840746572),\n",
       " (841, 0.023414645289542353),\n",
       " (842, 0.0),\n",
       " (843, 0.13518451760896877),\n",
       " (844, 0.051298917604257706),\n",
       " (845, 0.0),\n",
       " (846, 0.019389168358237032),\n",
       " (847, 0.04757860680698066),\n",
       " (848, 0.037463432463267755),\n",
       " (849, 0.0),\n",
       " (850, 0.07647191129018727),\n",
       " (851, 0.025031308716087945),\n",
       " (852, 0.021300716142115247),\n",
       " (853, 0.02913582733740325),\n",
       " (854, 0.08000711205939975),\n",
       " (855, 0.04947706959952935),\n",
       " (856, 0.023918243661746996),\n",
       " (857, 0.0),\n",
       " (858, 0.03627381250550058),\n",
       " (859, 0.13710212427677043),\n",
       " (860, 0.05158630874706817),\n",
       " (861, 0.02060214108575823),\n",
       " (862, 0.052631578947368425),\n",
       " (863, 0.05923488777590923),\n",
       " (864, 0.03823595564509363),\n",
       " (865, 0.0),\n",
       " (866, 0.08111071056538127),\n",
       " (867, 0.0),\n",
       " (868, 0.02060214108575823),\n",
       " (869, 0.04947706959952935),\n",
       " (870, 0.11959121830873498),\n",
       " (871, 0.05195243334661311),\n",
       " (872, 0.0),\n",
       " (873, 0.04188539082916955),\n",
       " (874, 0.025649458802128853),\n",
       " (875, 0.05006261743217589),\n",
       " (876, 0.0),\n",
       " (877, 0.0),\n",
       " (878, 0.09933992677987831),\n",
       " (879, 0.0),\n",
       " (880, 0.03181423814878889),\n",
       " (881, 0.0),\n",
       " (882, 0.027036903521793755),\n",
       " (883, 0.025031308716087945),\n",
       " (884, 0.0),\n",
       " (885, 0.07336739820667779),\n",
       " (886, 0.02294157338705618),\n",
       " (887, 0.03478392380269825),\n",
       " (888, 0.04783648732349399),\n",
       " (889, 0.11299144295167773),\n",
       " (890, 0.0),\n",
       " (891, 0.04188539082916955),\n",
       " (892, 0.0),\n",
       " (893, 0.016731862035256367),\n",
       " (894, 0.08226127456606226),\n",
       " (895, 0.0),\n",
       " (896, 0.0),\n",
       " (897, 0.039344473768231684),\n",
       " (898, 0.0978231976089037),\n",
       " (899, 0.08000711205939975),\n",
       " (900, 0.11831213107007527),\n",
       " (901, 0.08111071056538127),\n",
       " (902, 0.02294157338705618),\n",
       " (903, 0.031219527052723135),\n",
       " (904, 0.02060214108575823),\n",
       " (905, 0.0),\n",
       " (906, 0.0),\n",
       " (907, 0.023662426214015053),\n",
       " (908, 0.07254762501100118),\n",
       " (909, 0.0),\n",
       " (910, 0.019968076595771794),\n",
       " (911, 0.023414645289542353),\n",
       " (912, 0.03244428422615251),\n",
       " (913, 0.0),\n",
       " (914, 0.03721614637823934),\n",
       " (915, 0.026315789473684213),\n",
       " (916, 0.03382550457458692),\n",
       " (917, 0.028239124736245257),\n",
       " (918, 0.0),\n",
       " (919, 0.021486752129677003),\n",
       " (920, 0.09365858115816941),\n",
       " (921, 0.0),\n",
       " (922, 0.07792865001991967),\n",
       " (923, 0.0),\n",
       " (924, 0.0),\n",
       " (925, 0.020121090914638345),\n",
       " (926, 0.0),\n",
       " (927, 0.0),\n",
       " (928, 0.0),\n",
       " (929, 0.05195243334661311),\n",
       " (930, 0.04836508334066745),\n",
       " (931, 0.16943474841747155),\n",
       " (932, 0.07098727864204515),\n",
       " (933, 0.07024393586862707),\n",
       " (934, 0.02913582733740325),\n",
       " (935, 0.1324532357065044),\n",
       " (936, 0.0),\n",
       " (937, 0.13518451760896877),\n",
       " (938, 0.026315789473684213),\n",
       " (939, 0.1068366854693644),\n",
       " (940, 0.0),\n",
       " (941, 0.0301237616564389),\n",
       " (942, 0.1892994097121204),\n",
       " (943, 0.0),\n",
       " (944, 0.0),\n",
       " (945, 0.06131393394849658),\n",
       " (946, 0.024455799402225926),\n",
       " (947, 0.0),\n",
       " (948, 0.0),\n",
       " (949, 0.0),\n",
       " (950, 0.03721614637823934),\n",
       " (951, 0.06488856845230502),\n",
       " (952, 0.025649458802128853),\n",
       " (953, 0.023174488732966077),\n",
       " (954, 0.0),\n",
       " (955, 0.025976216673306556),\n",
       " (956, 0.05006261743217589),\n",
       " (957, 0.06243905410544627),\n",
       " (958, 0.024455799402225926),\n",
       " (959, 0.024738534799764674),\n",
       " (960, 0.08603090020146065),\n",
       " (961, 0.050669459193814),\n",
       " (962, 0.023918243661746996),\n",
       " (963, 0.0623109738595896),\n",
       " (964, 0.025976216673306556),\n",
       " (965, 0.0),\n",
       " (966, 0.0),\n",
       " (967, 0.03627381250550058),\n",
       " (968, 0.0),\n",
       " (969, 0.05775642214923893),\n",
       " (970, 0.029617443887954616),\n",
       " (971, 0.03244428422615251),\n",
       " (972, 0.2073221072156823),\n",
       " (973, 0.19767387315371682),\n",
       " (974, 0.02378930340349033),\n",
       " (975, 0.023353906807751994),\n",
       " (976, 0.02782074420373286),\n",
       " (977, 0.0),\n",
       " (978, 0.0),\n",
       " (979, 0.07254762501100118),\n",
       " (980, 0.09197090092274487),\n",
       " (981, 0.04456565178215865),\n",
       " (982, 0.020942695414584774),\n",
       " (983, 0.0),\n",
       " (984, 0.057353933467640436),\n",
       " (985, 0.03212462831016178),\n",
       " (986, 0.0),\n",
       " (987, 0.05484084971070818),\n",
       " (988, 0.0989541391990587),\n",
       " (989, 0.0582716546748065),\n",
       " (990, 0.0),\n",
       " (991, 0.03244428422615251),\n",
       " (992, 0.0415406492397264),\n",
       " (993, 0.06131393394849658),\n",
       " (994, 0.0),\n",
       " (995, 0.07254762501100116),\n",
       " (996, 0.05484084971070818),\n",
       " (997, 0.09086217008485092),\n",
       " (998, 0.052631578947368425),\n",
       " (999, 0.0),\n",
       " ...]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(enumerate(similarity[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba44b1b7",
   "metadata": {},
   "source": [
    "- (0, 1.0000000000000002) ---> distance of movie_1 compared against movie_1 = 1.0000000000000002\n",
    "- (1, 0.08226127456606226) ---> distance of movie_1 compared against movie_2 = 0.08226127456606226\n",
    "- (2, 0.08603090020146065) ---> distance of movie_1 compared against movie_3 = 0.08603090020146065"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "46368166",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 1.0000000000000002),\n",
       " (1216, 0.28676966733820225),\n",
       " (2409, 0.26310068027921696),\n",
       " (507, 0.255608593705383),\n",
       " (3730, 0.25391668753850405),\n",
       " (539, 0.2467838236981868),\n",
       " (582, 0.24511108480187255),\n",
       " (1204, 0.23918243661746996),\n",
       " (1194, 0.2367785320221084),\n",
       " (61, 0.23179316248638276),\n",
       " (778, 0.2294157338705618),\n",
       " (1920, 0.2252817784447915),\n",
       " (4048, 0.22329687826943603),\n",
       " (2786, 0.22269966704152225),\n",
       " (172, 0.21239769762143662),\n",
       " (972, 0.2073221072156823),\n",
       " (2971, 0.20602141085758227),\n",
       " (322, 0.20519567041703082),\n",
       " (2333, 0.20443988269091456),\n",
       " (3608, 0.20437977982832192),\n",
       " (4192, 0.2029530274475215),\n",
       " (1444, 0.20277677641345318),\n",
       " (1089, 0.2020475485519274),\n",
       " (260, 0.20073876713674155),\n",
       " (74, 0.20054543301971392),\n",
       " (151, 0.19867985355975665),\n",
       " (3675, 0.1979082783981174),\n",
       " (973, 0.19767387315371682),\n",
       " (577, 0.1976738731537168),\n",
       " (47, 0.19529164171612676),\n",
       " (3327, 0.19117977822546817),\n",
       " (1201, 0.19088542889273336),\n",
       " (942, 0.1892994097121204),\n",
       " (27, 0.18848425873126295),\n",
       " (305, 0.18848425873126295),\n",
       " (4405, 0.18731716231633883),\n",
       " (4, 0.1873171623163388),\n",
       " (495, 0.1873171623163388),\n",
       " (1329, 0.18394180184548975),\n",
       " (1071, 0.18183351671314588),\n",
       " (2999, 0.1813690625275029),\n",
       " (94, 0.18074256993863339),\n",
       " (3999, 0.18074256993863339),\n",
       " (83, 0.17996850826633903),\n",
       " (1321, 0.17699808135119718),\n",
       " (168, 0.17521916101261562),\n",
       " (184, 0.17521916101261562),\n",
       " (2204, 0.17521916101261562),\n",
       " (4336, 0.17521916101261562),\n",
       " (220, 0.1748149640244195),\n",
       " (1275, 0.1748149640244195),\n",
       " (300, 0.17316974359835272),\n",
       " (3538, 0.1711905958155815),\n",
       " (3162, 0.17040572913692198),\n",
       " (931, 0.16943474841747155),\n",
       " (36, 0.1692777916923361),\n",
       " (373, 0.1691275228729346),\n",
       " (529, 0.16742770563222897),\n",
       " (1831, 0.16742770563222897),\n",
       " (1982, 0.16692446522239718),\n",
       " (150, 0.16531163063339513),\n",
       " (661, 0.16481712868606585),\n",
       " (91, 0.16452254913212452),\n",
       " (595, 0.16390251702679648),\n",
       " (41, 0.16390251702679645),\n",
       " (1537, 0.16222142113076254),\n",
       " (67, 0.16222142113076252),\n",
       " (342, 0.1602550282040466),\n",
       " (1178, 0.1600142241187995),\n",
       " (2515, 0.15907119074394446),\n",
       " (4010, 0.15737789507292674),\n",
       " (31, 0.15609763526361567),\n",
       " (2731, 0.15609763526361567),\n",
       " (513, 0.15585730003983933),\n",
       " (149, 0.1540171257313038),\n",
       " (26, 0.15389675281277312),\n",
       " (187, 0.152008377581442),\n",
       " (1344, 0.152008377581442),\n",
       " (838, 0.1518926498314485),\n",
       " (111, 0.14843120879858804),\n",
       " (311, 0.14843120879858804),\n",
       " (1383, 0.14843120879858804),\n",
       " (581, 0.14808721943977307),\n",
       " (2075, 0.14808721943977307),\n",
       " (2760, 0.1478360110834291),\n",
       " (3354, 0.14567913668701626),\n",
       " (7, 0.14509525002200235),\n",
       " (52, 0.14509525002200235),\n",
       " (108, 0.14509525002200235),\n",
       " (127, 0.14509525002200235),\n",
       " (1297, 0.14509525002200235),\n",
       " (466, 0.14509525002200233),\n",
       " (1396, 0.14421498760030763),\n",
       " (1056, 0.14350946197048198),\n",
       " (1355, 0.14350946197048198),\n",
       " (1938, 0.14350946197048198),\n",
       " (275, 0.1433848336691011),\n",
       " (413, 0.1433848336691011),\n",
       " (579, 0.1433848336691011),\n",
       " (1479, 0.14306584587982532),\n",
       " (490, 0.14159846508095775),\n",
       " (294, 0.1411956236812263),\n",
       " (461, 0.1411956236812263),\n",
       " (568, 0.1411956236812263),\n",
       " (95, 0.14048787173725413),\n",
       " (1273, 0.14048787173725413),\n",
       " (17, 0.13977653617040256),\n",
       " (2496, 0.1391037210186643),\n",
       " (3409, 0.1391037210186643),\n",
       " (249, 0.13834289277321493),\n",
       " (1327, 0.13834289277321493),\n",
       " (1488, 0.13834289277321493),\n",
       " (3781, 0.13834289277321493),\n",
       " (1965, 0.1377056581888109),\n",
       " (46, 0.13710212427677043),\n",
       " (859, 0.13710212427677043),\n",
       " (2169, 0.13710212427677043),\n",
       " (4538, 0.13629325512727639),\n",
       " (56, 0.13530201829834768),\n",
       " (505, 0.13530201829834768),\n",
       " (2584, 0.13530201829834768),\n",
       " (3982, 0.13530201829834768),\n",
       " (72, 0.13518451760896877),\n",
       " (843, 0.13518451760896877),\n",
       " (937, 0.13518451760896877),\n",
       " (4348, 0.13497638119975428),\n",
       " (1322, 0.13476498501489081),\n",
       " (777, 0.13369695534647594),\n",
       " (1560, 0.13334518676566626),\n",
       " (1808, 0.13334518676566626),\n",
       " (668, 0.1324532357065044),\n",
       " (935, 0.1324532357065044),\n",
       " (158, 0.13245323570650439),\n",
       " (502, 0.13245323570650439),\n",
       " (126, 0.13157894736842107),\n",
       " (1015, 0.13157894736842107),\n",
       " (4079, 0.13157894736842107),\n",
       " (2133, 0.13124359177794215),\n",
       " (3622, 0.12988108336653278),\n",
       " (368, 0.12977713690461004),\n",
       " (476, 0.12977713690461004),\n",
       " (1373, 0.12977713690461004),\n",
       " (2378, 0.12977713690461004),\n",
       " (2873, 0.12892051277806202),\n",
       " (14, 0.12824729401064427),\n",
       " (138, 0.12824729401064427),\n",
       " (1086, 0.12824729401064427),\n",
       " (2181, 0.12824729401064427),\n",
       " (2686, 0.12824729401064427),\n",
       " (4230, 0.12824729401064427),\n",
       " (134, 0.12725695259515557),\n",
       " (487, 0.12725695259515557),\n",
       " (773, 0.12725695259515557),\n",
       " (1774, 0.12725695259515557),\n",
       " (1833, 0.12725695259515557),\n",
       " (2970, 0.12725695259515557),\n",
       " (369, 0.126673647984535),\n",
       " (557, 0.12515654358043973),\n",
       " (754, 0.12515654358043973),\n",
       " (2450, 0.12515654358043973),\n",
       " (3628, 0.12515654358043973),\n",
       " (1158, 0.12487810821089254),\n",
       " (1741, 0.12462194771917919),\n",
       " (166, 0.12369267399882337),\n",
       " (84, 0.12361284651454937),\n",
       " (208, 0.12262786789699316),\n",
       " (483, 0.12262786789699316),\n",
       " (1096, 0.12262786789699316),\n",
       " (2652, 0.12262786789699316),\n",
       " (102, 0.12262786789699313),\n",
       " (1878, 0.12262786789699313),\n",
       " (1299, 0.12227899701112963),\n",
       " (2331, 0.12166606584807191),\n",
       " (2480, 0.12166606584807191),\n",
       " (2674, 0.12166606584807191),\n",
       " (3083, 0.12166606584807191),\n",
       " (438, 0.12091270835166862),\n",
       " (3235, 0.12091270835166862),\n",
       " (68, 0.1204950466257556),\n",
       " (1658, 0.1204950466257556),\n",
       " (2060, 0.1204950466257556),\n",
       " (139, 0.11980845957463077),\n",
       " (1002, 0.11980845957463077),\n",
       " (549, 0.11959121830873498),\n",
       " (870, 0.11959121830873498),\n",
       " (781, 0.11846977555181847),\n",
       " (1004, 0.11846977555181847),\n",
       " (1047, 0.11846977555181847),\n",
       " (1274, 0.11846977555181847),\n",
       " (2741, 0.11846977555181847),\n",
       " (3376, 0.11846977555181847),\n",
       " (4673, 0.11846977555181847),\n",
       " (900, 0.11831213107007527),\n",
       " (1673, 0.11803342130469505),\n",
       " (1715, 0.11803342130469505),\n",
       " (2789, 0.11803342130469505),\n",
       " (4000, 0.11803342130469505),\n",
       " (182, 0.11707322644771176),\n",
       " (2284, 0.11707322644771176),\n",
       " (678, 0.116543309349613),\n",
       " (160, 0.11650496642725611),\n",
       " (589, 0.1163350101494222),\n",
       " (278, 0.11587244366483038),\n",
       " (454, 0.11587244366483038),\n",
       " (1617, 0.11587244366483038),\n",
       " (1497, 0.11551284429847784),\n",
       " (3212, 0.11542256460957984),\n",
       " (1220, 0.1147078669352809),\n",
       " (2547, 0.1147078669352809),\n",
       " (432, 0.11470786693528089),\n",
       " (85, 0.11470786693528087),\n",
       " (122, 0.11470786693528087),\n",
       " (236, 0.11470786693528087),\n",
       " (1301, 0.11470786693528087),\n",
       " (1418, 0.11357771260606365),\n",
       " (238, 0.11314714296070713),\n",
       " (3147, 0.11314714296070713),\n",
       " (889, 0.11299144295167773),\n",
       " (422, 0.11295649894498103),\n",
       " (536, 0.1124803176664619),\n",
       " (189, 0.11164843913471803),\n",
       " (627, 0.11164843913471803),\n",
       " (808, 0.11164843913471803),\n",
       " (1348, 0.11164843913471803),\n",
       " (3721, 0.11164843913471803),\n",
       " (123, 0.11141412945539662),\n",
       " (420, 0.11141412945539662),\n",
       " (2372, 0.11141412945539662),\n",
       " (1512, 0.11141412945539661),\n",
       " (50, 0.11128297681493143),\n",
       " (87, 0.11128297681493143),\n",
       " (232, 0.11128297681493143),\n",
       " (1616, 0.11128297681493143),\n",
       " (4468, 0.11128297681493143),\n",
       " (206, 0.11092107999698035),\n",
       " (228, 0.11037769642208699),\n",
       " (229, 0.11020775375559674),\n",
       " (28, 0.10968169942141635),\n",
       " (121, 0.10968169942141635),\n",
       " (449, 0.10968169942141635),\n",
       " (1031, 0.10968169942141635),\n",
       " (2817, 0.10968169942141635),\n",
       " (3008, 0.10968169942141635),\n",
       " (81, 0.1095080149111477),\n",
       " (1574, 0.1095080149111477),\n",
       " (3630, 0.10936965981495178),\n",
       " (256, 0.10882143751650175),\n",
       " (653, 0.10882143751650175),\n",
       " (1524, 0.10882143751650175),\n",
       " (692, 0.10838874619051501),\n",
       " (3093, 0.10838874619051501),\n",
       " (3309, 0.10838874619051501),\n",
       " (307, 0.10827043935570556),\n",
       " (332, 0.10814761408717502),\n",
       " (472, 0.10814761408717502),\n",
       " (786, 0.10814761408717502),\n",
       " (1237, 0.10814761408717502),\n",
       " (1945, 0.10814761408717502),\n",
       " (3140, 0.10814761408717502),\n",
       " (3428, 0.10814761408717502),\n",
       " (3498, 0.10814761408717502),\n",
       " (4777, 0.10814761408717502),\n",
       " (5, 0.10743376064838502),\n",
       " (740, 0.10743376064838502),\n",
       " (939, 0.1068366854693644),\n",
       " (21, 0.106676149412533),\n",
       " (262, 0.10650358071057624),\n",
       " (572, 0.10650358071057624),\n",
       " (233, 0.10619884881071831),\n",
       " (1575, 0.10619884881071831),\n",
       " (2161, 0.10619884881071831),\n",
       " (2521, 0.10619884881071831),\n",
       " (2634, 0.10619884881071831),\n",
       " (467, 0.10559715077387793),\n",
       " (1496, 0.10559715077387793),\n",
       " (295, 0.10526315789473685),\n",
       " (687, 0.10526315789473685),\n",
       " (814, 0.10526315789473685),\n",
       " (2072, 0.10526315789473685),\n",
       " (3340, 0.10526315789473685),\n",
       " (170, 0.10471347707292386),\n",
       " (2498, 0.10471347707292386),\n",
       " (75, 0.10390486669322622),\n",
       " (131, 0.10390486669322622),\n",
       " (223, 0.10390486669322622),\n",
       " (3152, 0.10390486669322622),\n",
       " (541, 0.103851623099316),\n",
       " (2439, 0.10385162309931599),\n",
       " (200, 0.1037571695799112),\n",
       " (1166, 0.1037571695799112),\n",
       " (1501, 0.1037571695799112),\n",
       " (1745, 0.1037571695799112),\n",
       " (3621, 0.1037571695799112),\n",
       " (3839, 0.1037571695799112),\n",
       " (4135, 0.1037571695799112),\n",
       " (654, 0.10301070542879115),\n",
       " (1455, 0.10301070542879115),\n",
       " (312, 0.10259783520851541),\n",
       " (508, 0.10259783520851541),\n",
       " (1082, 0.10259783520851541),\n",
       " (2148, 0.10259783520851541),\n",
       " (3634, 0.10259783520851541),\n",
       " (3755, 0.10259783520851541),\n",
       " (3874, 0.10259783520851541),\n",
       " (4097, 0.10259783520851541),\n",
       " (2106, 0.10218988991416095),\n",
       " (105, 0.10147651372376076),\n",
       " (624, 0.10147651372376076),\n",
       " (2323, 0.10147651372376076),\n",
       " (2456, 0.10147651372376076),\n",
       " (2762, 0.10147651372376076),\n",
       " (3055, 0.10147651372376076),\n",
       " (3663, 0.10147651372376076),\n",
       " (4198, 0.10147651372376076),\n",
       " (3441, 0.10138838820672659),\n",
       " (10, 0.101338918387628),\n",
       " (1037, 0.101338918387628),\n",
       " (1195, 0.101338918387628),\n",
       " (1294, 0.101338918387628),\n",
       " (1771, 0.101338918387628),\n",
       " (2247, 0.10133891838762799),\n",
       " (2099, 0.10060545457319173),\n",
       " (1614, 0.10012523486435178),\n",
       " (1405, 0.09984038297885897),\n",
       " (1439, 0.09984038297885897),\n",
       " (148, 0.09933992677987831),\n",
       " (157, 0.09933992677987831),\n",
       " (263, 0.09933992677987831),\n",
       " (325, 0.09933992677987831),\n",
       " (878, 0.09933992677987831),\n",
       " (1077, 0.09933992677987831),\n",
       " (1156, 0.09933992677987831),\n",
       " (1280, 0.09933992677987831),\n",
       " (1727, 0.09933992677987831),\n",
       " (1884, 0.09933992677987831),\n",
       " (4018, 0.09933992677987831),\n",
       " (4490, 0.09933992677987831),\n",
       " (363, 0.09909250441117726),\n",
       " (491, 0.09909250441117726),\n",
       " (330, 0.0989541391990587),\n",
       " (445, 0.0989541391990587),\n",
       " (510, 0.0989541391990587),\n",
       " (643, 0.0989541391990587),\n",
       " (988, 0.0989541391990587),\n",
       " (1823, 0.0989541391990587),\n",
       " (3620, 0.0989541391990587),\n",
       " (3783, 0.0989541391990587),\n",
       " (3935, 0.0989541391990587),\n",
       " (1775, 0.09836118442057921),\n",
       " (266, 0.0978231976089037),\n",
       " (535, 0.0978231976089037),\n",
       " (898, 0.0978231976089037),\n",
       " (1109, 0.0978231976089037),\n",
       " (1401, 0.0978231976089037),\n",
       " (2253, 0.0978231976089037),\n",
       " (230, 0.09764582085806336),\n",
       " (35, 0.09733285267845754),\n",
       " (319, 0.09733285267845754),\n",
       " (715, 0.09733285267845754),\n",
       " (1306, 0.09733285267845754),\n",
       " (1942, 0.09733285267845754),\n",
       " (2053, 0.09733285267845754),\n",
       " (2787, 0.09733285267845754),\n",
       " (3954, 0.09733285267845754),\n",
       " (4203, 0.09733285267845754),\n",
       " (4472, 0.09733285267845754),\n",
       " (9, 0.0967301666813349),\n",
       " (334, 0.0967301666813349),\n",
       " (410, 0.0967301666813349),\n",
       " (480, 0.0967301666813349),\n",
       " (2556, 0.0967301666813349),\n",
       " (371, 0.09567297464698798),\n",
       " (2241, 0.09567297464698798),\n",
       " (18, 0.09558988911273408),\n",
       " (4545, 0.09558988911273408),\n",
       " (33, 0.09544271444636668),\n",
       " (362, 0.09544271444636668),\n",
       " (791, 0.09544271444636668),\n",
       " (2562, 0.09544271444636668),\n",
       " (4256, 0.09544271444636668),\n",
       " (4403, 0.09544271444636668),\n",
       " (11, 0.09365858115816941),\n",
       " (244, 0.09365858115816941),\n",
       " (412, 0.09365858115816941),\n",
       " (587, 0.09365858115816941),\n",
       " (660, 0.09365858115816941),\n",
       " (920, 0.09365858115816941),\n",
       " (1276, 0.09365858115816941),\n",
       " (1357, 0.09365858115816941),\n",
       " (1474, 0.09365858115816941),\n",
       " (1573, 0.09365858115816941),\n",
       " (1686, 0.09365858115816941),\n",
       " (1692, 0.09365858115816941),\n",
       " (1721, 0.09365858115816941),\n",
       " (2488, 0.09365858115816941),\n",
       " (2592, 0.09365858115816941),\n",
       " (2645, 0.09365858115816941),\n",
       " (2650, 0.09365858115816941),\n",
       " (3407, 0.09365858115816941),\n",
       " (3537, 0.09365858115816941),\n",
       " (3720, 0.09365858115816941),\n",
       " (3736, 0.09365858115816941),\n",
       " (4331, 0.09365858115816941),\n",
       " (301, 0.09304036594559835),\n",
       " (129, 0.09269795493186431),\n",
       " (243, 0.09269795493186431),\n",
       " (279, 0.09269795493186431),\n",
       " (596, 0.09269795493186431),\n",
       " (1446, 0.09269795493186431),\n",
       " (1701, 0.09269795493186431),\n",
       " (2820, 0.09269795493186431),\n",
       " (785, 0.09243423333081695),\n",
       " (980, 0.09197090092274487),\n",
       " (2093, 0.09197090092274487),\n",
       " (2396, 0.09197090092274487),\n",
       " (2651, 0.09197090092274487),\n",
       " (239, 0.09183979479633063),\n",
       " (799, 0.09183979479633063),\n",
       " (2570, 0.09183979479633061),\n",
       " (473, 0.09176629354822471),\n",
       " (813, 0.09176629354822471),\n",
       " (43, 0.09086217008485092),\n",
       " (331, 0.09086217008485092),\n",
       " (602, 0.09086217008485092),\n",
       " (997, 0.09086217008485092),\n",
       " (1233, 0.09086217008485092),\n",
       " (1846, 0.09086217008485092),\n",
       " (3881, 0.09086217008485092),\n",
       " (219, 0.09037128496931669),\n",
       " (511, 0.09037128496931669),\n",
       " (1027, 0.09037128496931669),\n",
       " (1756, 0.09037128496931669),\n",
       " (2251, 0.09037128496931669),\n",
       " (2868, 0.09037128496931669),\n",
       " (2957, 0.09037128496931669),\n",
       " (3448, 0.09037128496931669),\n",
       " (2796, 0.08998425413316952),\n",
       " (2856, 0.08998425413316952),\n",
       " (3389, 0.08998425413316952),\n",
       " (3443, 0.08998425413316952),\n",
       " (3462, 0.08998425413316952),\n",
       " (4343, 0.08998425413316952),\n",
       " (4353, 0.08998425413316952),\n",
       " (274, 0.08977310580745099),\n",
       " (1783, 0.0891313035643173),\n",
       " (1906, 0.0891313035643173),\n",
       " (2363, 0.0891313035643173),\n",
       " (3553, 0.0891313035643173),\n",
       " (2115, 0.08903057122447033),\n",
       " (12, 0.08885233166386385),\n",
       " (88, 0.08885233166386385),\n",
       " (356, 0.08885233166386385),\n",
       " (400, 0.08885233166386385),\n",
       " (794, 0.08885233166386385),\n",
       " (1480, 0.08885233166386385),\n",
       " (2660, 0.08885233166386385),\n",
       " (3322, 0.08885233166386385),\n",
       " (479, 0.08849904067559859),\n",
       " (169, 0.08830215713766959),\n",
       " (2713, 0.08830215713766959),\n",
       " (143, 0.08749572785196143),\n",
       " (199, 0.08749572785196143),\n",
       " (270, 0.08749572785196143),\n",
       " (2109, 0.08749572785196143),\n",
       " (2740, 0.08749572785196143),\n",
       " (167, 0.08740748201220976),\n",
       " (292, 0.08740748201220976),\n",
       " (389, 0.08740748201220976),\n",
       " (564, 0.08740748201220976),\n",
       " (1249, 0.08740748201220976),\n",
       " (1259, 0.08740748201220976),\n",
       " (2448, 0.08695980950674562),\n",
       " (618, 0.08671099695241201),\n",
       " (752, 0.08671099695241201),\n",
       " (827, 0.08671099695241201),\n",
       " (1832, 0.08671099695241201),\n",
       " (2880, 0.08671099695241201),\n",
       " (3556, 0.08671099695241201),\n",
       " (4502, 0.08671099695241201),\n",
       " (4516, 0.08671099695241201),\n",
       " (4587, 0.08671099695241201),\n",
       " (2, 0.08603090020146065),\n",
       " (23, 0.08603090020146065),\n",
       " (141, 0.08603090020146065),\n",
       " (457, 0.08603090020146065),\n",
       " (745, 0.08603090020146065),\n",
       " (775, 0.08603090020146065),\n",
       " (960, 0.08603090020146065),\n",
       " (3135, 0.08603090020146065),\n",
       " (3449, 0.08603090020146065),\n",
       " (4754, 0.08603090020146065),\n",
       " (217, 0.08594700851870801),\n",
       " (234, 0.08594700851870801),\n",
       " (1019, 0.08594700851870801),\n",
       " (2354, 0.08594700851870801),\n",
       " (1639, 0.08520286456846099),\n",
       " (142, 0.08471737420873576),\n",
       " (1378, 0.08471737420873576),\n",
       " (2583, 0.08471737420873576),\n",
       " (4560, 0.08471737420873576),\n",
       " (32, 0.08447772061910234),\n",
       " (1305, 0.08447772061910234),\n",
       " (19, 0.0837707816583391),\n",
       " (190, 0.0837707816583391),\n",
       " (534, 0.0837707816583391),\n",
       " (1530, 0.0837707816583391),\n",
       " (2968, 0.0837707816583391),\n",
       " (3218, 0.0837707816583391),\n",
       " (3905, 0.0837707816583391),\n",
       " (3978, 0.0837707816583391),\n",
       " (4568, 0.0837707816583391),\n",
       " (4737, 0.0837707816583391),\n",
       " (315, 0.08346223261119858),\n",
       " (1142, 0.08346223261119858),\n",
       " (1589, 0.08346223261119858),\n",
       " (1749, 0.08346223261119858),\n",
       " (2167, 0.08346223261119858),\n",
       " (2304, 0.08346223261119858),\n",
       " (48, 0.0830812984794528),\n",
       " (403, 0.0830812984794528),\n",
       " (45, 0.08240856434303293),\n",
       " (1097, 0.08226127456606228),\n",
       " (1, 0.08226127456606226),\n",
       " (39, 0.08226127456606226),\n",
       " (152, 0.08226127456606226),\n",
       " (203, 0.08226127456606226),\n",
       " (341, 0.08226127456606226),\n",
       " (547, 0.08226127456606226),\n",
       " (626, 0.08226127456606226),\n",
       " (894, 0.08226127456606226),\n",
       " (1269, 0.08226127456606226),\n",
       " (1688, 0.08226127456606226),\n",
       " (2128, 0.08226127456606226),\n",
       " (2644, 0.08226127456606226),\n",
       " (2661, 0.08226127456606226),\n",
       " (2865, 0.08226127456606226),\n",
       " (3301, 0.08226127456606226),\n",
       " (3470, 0.08226127456606226),\n",
       " (3926, 0.08226127456606226),\n",
       " (63, 0.08175191193132876),\n",
       " (501, 0.08175191193132876),\n",
       " (2203, 0.08175191193132876),\n",
       " (38, 0.08111071056538127),\n",
       " (227, 0.08111071056538127),\n",
       " (241, 0.08111071056538127),\n",
       " (306, 0.08111071056538127),\n",
       " (381, 0.08111071056538127),\n",
       " (402, 0.08111071056538127),\n",
       " (787, 0.08111071056538127),\n",
       " (812, 0.08111071056538127),\n",
       " (866, 0.08111071056538127),\n",
       " (901, 0.08111071056538127),\n",
       " (1207, 0.08111071056538127),\n",
       " (1519, 0.08111071056538127),\n",
       " (1698, 0.08111071056538127),\n",
       " (1747, 0.08111071056538127),\n",
       " (2386, 0.08111071056538127),\n",
       " (2671, 0.08111071056538127),\n",
       " (2877, 0.08111071056538127),\n",
       " (3180, 0.08111071056538127),\n",
       " (3527, 0.08111071056538127),\n",
       " (3764, 0.08111071056538127),\n",
       " (4492, 0.08111071056538127),\n",
       " (4626, 0.08111071056538127),\n",
       " (4641, 0.08111071056538127),\n",
       " (4654, 0.08111071056538127),\n",
       " (4657, 0.08111071056538127),\n",
       " (2792, 0.08048436365855338),\n",
       " (4506, 0.08048436365855338),\n",
       " (486, 0.08031157077540445),\n",
       " (245, 0.08000711205939975),\n",
       " (854, 0.08000711205939975),\n",
       " (899, 0.08000711205939975),\n",
       " (2683, 0.08000711205939975),\n",
       " (3233, 0.08000711205939975),\n",
       " (3816, 0.08000711205939975),\n",
       " (3851, 0.08000711205939975),\n",
       " (1534, 0.07987230638308718),\n",
       " (1828, 0.07987230638308718),\n",
       " (51, 0.0792740035289418),\n",
       " (222, 0.0792740035289418),\n",
       " (16, 0.07894736842105264),\n",
       " (37, 0.07894736842105264),\n",
       " (164, 0.07894736842105264),\n",
       " (242, 0.07894736842105264),\n",
       " (1080, 0.07894736842105264),\n",
       " (1325, 0.07894736842105264),\n",
       " (3028, 0.07894736842105264),\n",
       " (4316, 0.07894736842105264),\n",
       " (193, 0.07878168645790681),\n",
       " (62, 0.07868894753646337),\n",
       " (103, 0.07868894753646337),\n",
       " (584, 0.07868894753646337),\n",
       " (1621, 0.07868894753646337),\n",
       " (1664, 0.07868894753646337),\n",
       " (1898, 0.07868894753646337),\n",
       " (2165, 0.07868894753646337),\n",
       " (2339, 0.07868894753646337),\n",
       " (4279, 0.07868894753646337),\n",
       " (4473, 0.07868894753646337),\n",
       " (125, 0.07817962860855646),\n",
       " (153, 0.0781166566864507),\n",
       " (286, 0.0781166566864507),\n",
       " (1926, 0.0781166566864507),\n",
       " (112, 0.07792865001991967),\n",
       " (329, 0.07792865001991967),\n",
       " (632, 0.07792865001991967),\n",
       " (922, 0.07792865001991967),\n",
       " (1161, 0.07792865001991967),\n",
       " (2295, 0.07792865001991967),\n",
       " (2938, 0.07792865001991967),\n",
       " (3164, 0.07792865001991967),\n",
       " (4341, 0.07792865001991967),\n",
       " (1282, 0.07755667343294813),\n",
       " (1659, 0.0770085628656519),\n",
       " (1924, 0.0770085628656519),\n",
       " (207, 0.07694837640638656),\n",
       " (1113, 0.07694837640638656),\n",
       " (2219, 0.07694837640638656),\n",
       " (2916, 0.07694837640638656),\n",
       " (3091, 0.07694837640638656),\n",
       " (3102, 0.07694837640638656),\n",
       " (776, 0.07647191129018727),\n",
       " (850, 0.07647191129018727),\n",
       " (1045, 0.07647191129018727),\n",
       " (1139, 0.07647191129018727),\n",
       " (1254, 0.07647191129018727),\n",
       " (2274, 0.07647191129018727),\n",
       " (3560, 0.07647191129018727),\n",
       " (3728, 0.07647191129018727),\n",
       " (3832, 0.07647191129018727),\n",
       " (4011, 0.07647191129018727),\n",
       " (40, 0.076004188790721),\n",
       " (70, 0.076004188790721),\n",
       " (96, 0.076004188790721),\n",
       " (101, 0.076004188790721),\n",
       " (174, 0.076004188790721),\n",
       " (235, 0.076004188790721),\n",
       " (349, 0.076004188790721),\n",
       " (492, 0.076004188790721),\n",
       " (1289, 0.076004188790721),\n",
       " (1290, 0.076004188790721),\n",
       " (2018, 0.076004188790721),\n",
       " (2449, 0.076004188790721),\n",
       " (4045, 0.076004188790721),\n",
       " (4268, 0.076004188790721),\n",
       " (4558, 0.076004188790721),\n",
       " (4624, 0.076004188790721),\n",
       " (246, 0.07543142864047142),\n",
       " (3331, 0.07530940414109724),\n",
       " (98, 0.07509392614826384),\n",
       " (361, 0.07509392614826384),\n",
       " (365, 0.07509392614826384),\n",
       " (680, 0.07509392614826384),\n",
       " (1009, 0.07509392614826384),\n",
       " (1293, 0.07509392614826384),\n",
       " (2021, 0.07509392614826384),\n",
       " (2163, 0.07509392614826384),\n",
       " (4205, 0.07509392614826384),\n",
       " (2005, 0.07492686492653551),\n",
       " (205, 0.07443229275647868),\n",
       " (800, 0.07443229275647868),\n",
       " (1005, 0.07443229275647868),\n",
       " (1323, 0.07443229275647868),\n",
       " (1521, 0.07443229275647868),\n",
       " (1770, 0.07443229275647868),\n",
       " (1990, 0.07443229275647868),\n",
       " (2256, 0.07443229275647868),\n",
       " (2326, 0.07443229275647868),\n",
       " (2445, 0.07443229275647868),\n",
       " (2712, 0.07443229275647868),\n",
       " (2801, 0.07443229275647868),\n",
       " (3117, 0.07443229275647868),\n",
       " (3126, 0.07443229275647868),\n",
       " (4165, 0.07443229275647868),\n",
       " (4412, 0.07443229275647868),\n",
       " (4542, 0.07443229275647868),\n",
       " (82, 0.07421560439929402),\n",
       " (210, 0.07421560439929402),\n",
       " (503, 0.07421560439929402),\n",
       " (540, 0.07421560439929402),\n",
       " (737, 0.07421560439929402),\n",
       " (1070, 0.07421560439929402),\n",
       " (1523, 0.07421560439929402),\n",
       " (3285, 0.07421560439929402),\n",
       " (3701, 0.07421560439929402),\n",
       " (2190, 0.07394738666465357),\n",
       " (3431, 0.07394738666465357),\n",
       " (1487, 0.0734718358370645),\n",
       " (634, 0.07336739820667779),\n",
       " (885, 0.07336739820667779),\n",
       " (1623, 0.07336739820667779),\n",
       " (2395, 0.07336739820667779),\n",
       " (2988, 0.07336739820667779),\n",
       " (3022, 0.07336739820667779),\n",
       " (3805, 0.07336739820667779),\n",
       " (4640, 0.07336739820667779),\n",
       " (2056, 0.07336739820667777),\n",
       " (3, 0.07300534327409847),\n",
       " (71, 0.07254762501100118),\n",
       " (784, 0.07254762501100118),\n",
       " (833, 0.07254762501100118),\n",
       " (908, 0.07254762501100118),\n",
       " (979, 0.07254762501100118),\n",
       " (1367, 0.07254762501100118),\n",
       " (1996, 0.07254762501100118),\n",
       " (2701, 0.07254762501100118),\n",
       " (3255, 0.07254762501100118),\n",
       " (3355, 0.07254762501100118),\n",
       " (179, 0.07254762501100116),\n",
       " (790, 0.07254762501100116),\n",
       " (995, 0.07254762501100116),\n",
       " (1489, 0.07254762501100116),\n",
       " (1655, 0.07254762501100116),\n",
       " (2212, 0.07254762501100116),\n",
       " (2297, 0.07254762501100116),\n",
       " (2317, 0.07254762501100116),\n",
       " (2772, 0.07254762501100116),\n",
       " (3069, 0.07254762501100116),\n",
       " (3072, 0.07254762501100116),\n",
       " (3103, 0.07254762501100116),\n",
       " (3155, 0.07254762501100116),\n",
       " (3430, 0.07254762501100116),\n",
       " (3716, 0.07254762501100116),\n",
       " (3976, 0.07254762501100116),\n",
       " (4431, 0.07254762501100116),\n",
       " (4443, 0.07254762501100116),\n",
       " (59, 0.07209840939145001),\n",
       " (76, 0.07175473098524099),\n",
       " (93, 0.07175473098524099),\n",
       " (1765, 0.07175473098524099),\n",
       " (1883, 0.07175473098524099),\n",
       " (2852, 0.07175473098524099),\n",
       " (3206, 0.07175473098524099),\n",
       " (3423, 0.07175473098524099),\n",
       " (3464, 0.07175473098524099),\n",
       " (4083, 0.07175473098524099),\n",
       " (1959, 0.07165743639000186),\n",
       " (175, 0.07098727864204515),\n",
       " (543, 0.07098727864204515),\n",
       " (728, 0.07098727864204515),\n",
       " (932, 0.07098727864204515),\n",
       " (1145, 0.07098727864204515),\n",
       " (1340, 0.07098727864204515),\n",
       " (2831, 0.07098727864204515),\n",
       " (3305, 0.07098727864204515),\n",
       " (3485, 0.07098727864204515),\n",
       " (3611, 0.07098727864204515),\n",
       " (4483, 0.07098727864204515),\n",
       " (1240, 0.07079923254047887),\n",
       " (1410, 0.07079923254047887),\n",
       " (2381, 0.07079923254047887),\n",
       " (2479, 0.07079923254047887),\n",
       " (2551, 0.07079923254047887),\n",
       " (2763, 0.07079923254047887),\n",
       " (3223, 0.07079923254047887),\n",
       " (4638, 0.07079923254047887),\n",
       " (86, 0.07024393586862707),\n",
       " (302, 0.07024393586862707),\n",
       " (674, 0.07024393586862707),\n",
       " (933, 0.07024393586862707),\n",
       " (1168, 0.07024393586862707),\n",
       " (1632, 0.07024393586862707),\n",
       " (2103, 0.07024393586862707),\n",
       " (2903, 0.07024393586862707),\n",
       " (3907, 0.07024393586862707),\n",
       " (55, 0.0695678476053965),\n",
       " (183, 0.0695678476053965),\n",
       " (97, 0.06952346619889824),\n",
       " (254, 0.06952346619889824),\n",
       " (271, 0.06952346619889824),\n",
       " (1457, 0.06952346619889824),\n",
       " (2527, 0.06952346619889824),\n",
       " (3300, 0.06952346619889824),\n",
       " (4180, 0.06952346619889824),\n",
       " (336, 0.06917144638660747),\n",
       " (1737, 0.06917144638660747),\n",
       " (2114, 0.06917144638660747),\n",
       " (2854, 0.06917144638660747),\n",
       " (3483, 0.06917144638660747),\n",
       " (3678, 0.06917144638660747),\n",
       " (3891, 0.06917144638660747),\n",
       " (600, 0.06882472016116853),\n",
       " (1278, 0.06882472016116853),\n",
       " (1538, 0.06882472016116853),\n",
       " (1847, 0.06882472016116853),\n",
       " (2209, 0.06882472016116853),\n",
       " (3310, 0.06882472016116853),\n",
       " (419, 0.06814662756363819),\n",
       " (671, 0.06814662756363819),\n",
       " (732, 0.06814662756363819),\n",
       " (782, 0.06814662756363819),\n",
       " (1146, 0.06814662756363819),\n",
       " (2002, 0.06814662756363819),\n",
       " (2243, 0.06814662756363819),\n",
       " (3910, 0.06814662756363819),\n",
       " (4123, 0.06814662756363819),\n",
       " (4392, 0.06814662756363819),\n",
       " (100, 0.06765100914917384),\n",
       " (498, 0.06765100914917384),\n",
       " (537, 0.06765100914917384),\n",
       " (1292, 0.06765100914917384),\n",
       " (1466, 0.06765100914917384),\n",
       " (1634, 0.06765100914917384),\n",
       " (1860, 0.06765100914917384),\n",
       " (2435, 0.06765100914917384),\n",
       " (2533, 0.06765100914917384),\n",
       " (2622, 0.06765100914917384),\n",
       " (2664, 0.06765100914917384),\n",
       " (2890, 0.06765100914917384),\n",
       " (4019, 0.06765100914917384),\n",
       " (4216, 0.06765100914917384),\n",
       " (4290, 0.06765100914917384),\n",
       " (124, 0.06748819059987714),\n",
       " (665, 0.06748819059987714),\n",
       " (704, 0.06748819059987714),\n",
       " (734, 0.06748819059987714),\n",
       " (2035, 0.06748819059987714),\n",
       " (2305, 0.06748819059987714),\n",
       " (3020, 0.06748819059987714),\n",
       " (488, 0.06692744814102547),\n",
       " (58, 0.06684847767323797),\n",
       " (215, 0.06684847767323797),\n",
       " (1445, 0.06684847767323797),\n",
       " (2596, 0.06684847767323797),\n",
       " (3225, 0.06684847767323797),\n",
       " (3588, 0.06684847767323797),\n",
       " (22, 0.0662266178532522),\n",
       " (109, 0.0662266178532522),\n",
       " (209, 0.0662266178532522),\n",
       " (267, 0.0662266178532522),\n",
       " (308, 0.0662266178532522),\n",
       " (570, 0.0662266178532522),\n",
       " (604, 0.0662266178532522),\n",
       " (1085, 0.0662266178532522),\n",
       " (1381, 0.0662266178532522),\n",
       " (2218, 0.0662266178532522),\n",
       " (2743, 0.0662266178532522),\n",
       " (2976, 0.0662266178532522),\n",
       " (3241, 0.0662266178532522),\n",
       " (3398, 0.0662266178532522),\n",
       " (3412, 0.0662266178532522),\n",
       " (3994, 0.0662266178532522),\n",
       " (4061, 0.0662266178532522),\n",
       " (4351, 0.0662266178532522),\n",
       " (4623, 0.0662266178532522),\n",
       " (30, 0.06622661785325219),\n",
       " (212, 0.06622661785325219),\n",
       " (635, 0.06622661785325219),\n",
       " (735, 0.06622661785325219),\n",
       " (1121, 0.06622661785325219),\n",
       " (1432, 0.06622661785325219),\n",
       " (2302, 0.06622661785325219),\n",
       " (2320, 0.06622661785325219),\n",
       " (2404, 0.06622661785325219),\n",
       " (1018, 0.06562179588897107),\n",
       " (1087, 0.06562179588897107),\n",
       " (2135, 0.06562179588897107),\n",
       " (2657, 0.06562179588897107),\n",
       " (3879, 0.06562179588897107),\n",
       " (304, 0.06557412294705281),\n",
       " (2832, 0.06521546507260247),\n",
       " (29, 0.065033247714309),\n",
       " (253, 0.065033247714309),\n",
       " (1175, 0.065033247714309),\n",
       " (2003, 0.065033247714309),\n",
       " (2052, 0.065033247714309),\n",
       " (3390, 0.065033247714309),\n",
       " (3635, 0.065033247714309),\n",
       " (3822, 0.065033247714309),\n",
       " (216, 0.06488856845230502),\n",
       " (237, 0.06488856845230502),\n",
       " (621, 0.06488856845230502),\n",
       " (695, 0.06488856845230502),\n",
       " (951, 0.06488856845230502),\n",
       " (1068, 0.06488856845230502),\n",
       " (1171, 0.06488856845230502),\n",
       " (1262, 0.06488856845230502),\n",
       " (1304, 0.06488856845230502),\n",
       " (1438, 0.06488856845230502),\n",
       " (1486, 0.06488856845230502),\n",
       " (2250, 0.06488856845230502),\n",
       " (2259, 0.06488856845230502),\n",
       " (2268, 0.06488856845230502),\n",
       " (2309, 0.06488856845230502),\n",
       " (2579, 0.06488856845230502),\n",
       " (2715, 0.06488856845230502),\n",
       " (2882, 0.06488856845230502),\n",
       " (3181, 0.06488856845230502),\n",
       " (3253, 0.06488856845230502),\n",
       " (3572, 0.06488856845230502),\n",
       " (3901, 0.06488856845230502),\n",
       " (3929, 0.06488856845230502),\n",
       " (4312, 0.06488856845230502),\n",
       " (281, 0.06446025638903101),\n",
       " (447, 0.06446025638903101),\n",
       " (1011, 0.06446025638903101),\n",
       " (1531, 0.06446025638903101),\n",
       " (3174, 0.06446025638903101),\n",
       " (2519, 0.06424925662032356),\n",
       " (425, 0.06390214842634574),\n",
       " (471, 0.06390214842634574),\n",
       " (746, 0.06390214842634574),\n",
       " (1848, 0.06390214842634574),\n",
       " (1979, 0.06390214842634574),\n",
       " (2264, 0.06390214842634574),\n",
       " (546, 0.06362847629757779),\n",
       " (744, 0.06362847629757779),\n",
       " (1365, 0.06362847629757779),\n",
       " (1854, 0.06362847629757779),\n",
       " (1950, 0.06362847629757779),\n",
       " (2113, 0.06362847629757779),\n",
       " (2270, 0.06362847629757779),\n",
       " (3374, 0.06362847629757779),\n",
       " (3931, 0.06362847629757779),\n",
       " (4006, 0.06362847629757779),\n",
       " (4355, 0.06362847629757779),\n",
       " (4357, 0.06362847629757779),\n",
       " (612, 0.06335829046432676),\n",
       " (1462, 0.06335829046432676),\n",
       " (15, 0.06282808624375433),\n",
       " (443, 0.06282808624375433),\n",
       " (591, 0.06282808624375433),\n",
       " (1093, 0.06282808624375433),\n",
       " (2319, 0.06282808624375433),\n",
       " (2888, 0.06282808624375433),\n",
       " (3344, 0.06282808624375433),\n",
       " (3607, 0.06282808624375433),\n",
       " (90, 0.06243905410544627),\n",
       " (117, 0.06243905410544627),\n",
       " (159, 0.06243905410544627),\n",
       " (290, 0.06243905410544627),\n",
       " (404, 0.06243905410544627),\n",
       " (957, 0.06243905410544627),\n",
       " (1707, 0.06243905410544627),\n",
       " (2228, 0.06243905410544627),\n",
       " (2620, 0.06243905410544627),\n",
       " (3118, 0.06243905410544627),\n",
       " (3133, 0.06243905410544627),\n",
       " (4664, 0.06243905410544627),\n",
       " (4704, 0.06243905410544627),\n",
       " (79, 0.0623109738595896),\n",
       " (92, 0.0623109738595896),\n",
       " (963, 0.0623109738595896),\n",
       " (1202, 0.0623109738595896),\n",
       " (2520, 0.0623109738595896),\n",
       " (3298, 0.0623109738595896),\n",
       " (3740, 0.0623109738595896),\n",
       " (1030, 0.06231097385958959),\n",
       " (106, 0.061806423257274694),\n",
       " (107, 0.061806423257274694),\n",
       " (836, 0.061806423257274694),\n",
       " (1191, 0.061806423257274694),\n",
       " (3078, 0.061806423257274694),\n",
       " (154, 0.06131393394849658),\n",
       " (259, 0.06131393394849658),\n",
       " (276, 0.06131393394849658),\n",
       " (309, 0.06131393394849658),\n",
       " (615, 0.06131393394849658),\n",
       " (945, 0.06131393394849658),\n",
       " (993, 0.06131393394849658),\n",
       " (1076, 0.06131393394849658),\n",
       " (1092, 0.06131393394849658),\n",
       " (1147, 0.06131393394849658),\n",
       " (1347, 0.06131393394849658),\n",
       " (1605, 0.06131393394849658),\n",
       " (1606, 0.06131393394849658),\n",
       " (1607, 0.06131393394849658),\n",
       " (1973, 0.06131393394849658),\n",
       " (2120, 0.06131393394849658),\n",
       " (2127, 0.06131393394849658),\n",
       " (2538, 0.06131393394849658),\n",
       " (2600, 0.06131393394849658),\n",
       " (2948, 0.06131393394849658),\n",
       " (3770, 0.06131393394849658),\n",
       " (3834, 0.06131393394849658),\n",
       " (4196, 0.06131393394849658),\n",
       " (4320, 0.06131393394849658),\n",
       " (4566, 0.06131393394849658),\n",
       " (4652, 0.06131393394849658),\n",
       " (733, 0.061313933948496574),\n",
       " (156, 0.060833032924035954),\n",
       " (375, 0.060833032924035954),\n",
       " (455, 0.060833032924035954),\n",
       " (1356, 0.060833032924035954),\n",
       " (3777, 0.060833032924035954),\n",
       " (4789, 0.060833032924035954),\n",
       " (224, 0.060363272743915036),\n",
       " (768, 0.060363272743915036),\n",
       " (1103, 0.060363272743915036),\n",
       " (4077, 0.060363272743915036),\n",
       " (348, 0.0602475233128778),\n",
       " (533, 0.0602475233128778),\n",
       " (664, 0.0602475233128778),\n",
       " (1408, 0.0602475233128778),\n",
       " (1434, 0.0602475233128778),\n",
       " (1584, 0.0602475233128778),\n",
       " (1631, 0.0602475233128778),\n",
       " (1944, 0.0602475233128778),\n",
       " (2159, 0.0602475233128778),\n",
       " ...]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# --- now we will sort this list of tuples\n",
    "sorted(list(enumerate(similarity[0])),\n",
    "      reverse = True,\n",
    "      key = lambda x:x[1])\n",
    "\n",
    "# ---- 'key' parameter is used to indicate that we want to sort the values based on second element in each of these tuples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "e68da564",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1216, 0.28676966733820225),\n",
       " (2409, 0.26310068027921696),\n",
       " (507, 0.255608593705383),\n",
       " (3730, 0.25391668753850405),\n",
       " (539, 0.2467838236981868),\n",
       " (582, 0.24511108480187255)]"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to suggest top 6 recommendations against movie titled 'Avatar'\n",
    "sorted(list(enumerate(similarity[0])), reverse = True, key = lambda x:x[1])[1:7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "f8553592",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now let us define the helper function to help implement the process illustrated above\n",
    "\n",
    "def recommend(movie):\n",
    "    movie_index = preprocessed_df[preprocessed_df['title'] == movie].index[0]\n",
    "    distance = similarity[movie_index]\n",
    "    movies_list = sorted(list(enumerate(distance)), reverse = True, key = lambda x:x[1])[1:6]\n",
    "    for i in movies_list:\n",
    "        print(preprocessed_df.iloc[i[0]].title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "af61126d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommendations similar to Batman Begins:\n",
      "The Dark Knight\n",
      "Batman\n",
      "Batman\n",
      "The Dark Knight Rises\n",
      "10th & Wolf\n",
      "\n",
      "Recommendations similar to John Carter:\n",
      "Riddick\n",
      "Krrish\n",
      "The Other Side of Heaven\n",
      "The Legend of Hercules\n",
      "Get Carter\n"
     ]
    }
   ],
   "source": [
    "# testing if our helper function: recommend() works successfully or not\n",
    "\n",
    "print('Recommendations similar to Batman Begins:')\n",
    "recommend('Batman Begins')\n",
    "print(\"\")\n",
    "print('Recommendations similar to John Carter:')\n",
    "recommend('John Carter')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "8cd967c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "70ea193c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(preprocessed_df,open('movie_list.pkl','wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "f7afe53d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(similarity,open('similarity_check.pkl','wb'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.5 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "11938c6bc6919ae2720b4d5011047913343b08a43b18698fd82dedb0d4417594"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}