MovieGenreFromPoster.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### <h1 style=\"text-align:center;\"> Movie Genres Classification from their Poster Image using CNNs</h1>\n",
    "\n",
    "<h3 style=\"text-align:center;\">- Final Project -</h3>\n",
    "<h5 style=\"text-align:center;\">author: Davide Iacobelli, 254435 </h5>\n",
    "\n",
    "# Introduction \n",
    "### Problem Description\n",
    "This project has the aim to achieve ***movie genre classi\u001bcation based only on movie poster images.*** <br>\n",
    "For movie viewers, movie posters are one of the first impressions used to get an idea about the movie content and its genre. Humans can get an idea based on things like color, objects, expressions on the faces of actors etc to quickly determine the genre (horror, comedy, animation etc). <br>\n",
    "If humans are more or less able to predict genre of a movie only giving a look at its poster, then we can assume that the poster possesses some characteristics which could be utilized in machine learning algorithms to predict its genre. \n",
    "\n",
    "### Proposed Approach\n",
    "In order to do that a _Deep Neural Network_ (**Convolutional Neural Network**) is constructed to classify a given movie poster image into genres. Since a movie may belong to multiple genres, this is a _multi-label image classication problem_. \n",
    "\n",
    "First of all, we use an online available **IMDB dataset** (source: https://www.kaggle.com/neha1703/movie-genre-from-its-poster) collected from the most famous movie website (https://www.imdb.com/). <br>\n",
    "Using the IMDB link of each movie (available in this dataset) we use a **Web Scraping approach** in order to retrieve its poster image from the IMDB movie page and save it locally. \n",
    "Once this is done, we can finally construct our Convolutional Neural Network in order to classify movie genre basing on poster characteristics.\n",
    "\n",
    "**Note:** since even a human can easily make mistakes in this task, our initial goal is to recognize correctly ***at least half of the movies***.\n",
    "\n",
    "# Step 1: _Webscraping_\n",
    " \n",
    "First of all we get the original IMDB dataset and using informations contained in it, namely IMDB link and IMDB id of each movie, we use a Webscraping Approach in order to retrieve movie poster images. \n",
    "In this task we use ***BeautifulSoup***, a Python Framework for Webscraping. <br>\n",
    "Since movie pages on IMDB website has all the same structure (see figure below), through Webscraping we can easily get the poster link of each film simply going on its IMDB page and taking the content of the _src_ HTML tag corresponding to the poster. \n",
    "Once we have all poster links, we add them to our dataset.\n",
    "<br><br>\n",
    "\n",
    "![alt text](./imdb_screen.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import glob\n",
    "import scipy.misc\n",
    "import imageio\n",
    "import skimage\n",
    "from tqdm import tqdm\n",
    "import requests  \n",
    "import re\n",
    "from bs4 import BeautifulSoup  \n",
    "from urllib.request import urlretrieve\n",
    "import ast \n",
    "import matplotlib.pyplot as plt \n",
    "\n",
    "\n",
    "savelocation = 'imdb_posters/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie = pd.read_csv(\"movies_metadata.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(movie)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie['imdb_link'] = [\"https://www.imdb.com/title/\"+str(x) for x in movie['imdb_id']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imdbURLS = movie['imdb_link'].tolist()\n",
    "imdbIDS = movie['imdb_id'].tolist()\n",
    "records = [] \n",
    "counter = 0\n",
    "\n",
    "for x in tqdm(imdbURLS): \n",
    "    # web scraping\n",
    "    imdbID = imdbIDS[counter]\n",
    "    r = requests.get(x)\n",
    "    soup = BeautifulSoup(r.text, 'html.parser')  \n",
    "    results = soup.find_all('div', attrs={'class':'poster'})  \n",
    "    if results:\n",
    "        first_result = results[0]  \n",
    "        postername = first_result.find('img')['alt'] \n",
    "        imgurl = first_result.find('img')['src'] \n",
    "        records.append((x, postername, imgurl))\n",
    "    else:\n",
    "        movie = movie[movie.imdb_id != imdbID]    \n",
    "counter += 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poster_df = pd.DataFrame(records, columns=['imdb_link', 'postername', 'poster_link'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal = pd.merge(movie, poster_df, on='imdb_link')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal.to_csv('movie_metadataWithPoster.csv', sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: _Posters Download_\n",
    "\n",
    "Once the _Webscraping step_ is completed, we have also poster image links in our dataset. <br>\n",
    "So now we are able to ***download poster images*** using those links. <br>\n",
    "Before doing that, we apply a simple step of data cleaning to the dataset, consisting in dropping all entries without a defined genre. \n",
    "\n",
    "So we start downloading all posters from the corrispondent link and we save each of them _using as name in the filesystem the IMDB id of the related movie_. In this way we maintain the relationship between movies and their poster images. \n",
    "\n",
    "**Note:** some images could be corrupted during the download or they may not be found at all. For those reasons we  check for corrupted images after download and also we drop from dataset rows corresponding to movie whose poster was not found. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal = pd.read_csv(\"movie_metadataWithPoster.csv\", sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "genres = []\n",
    "for entry in df_movietotal[\"genres\"]:\n",
    "    list_genres = \"\"\n",
    "    for genre in ast.literal_eval(entry):\n",
    "        list_genres = list_genres + genre[\"name\"] + \"|\"\n",
    "    genres.append(list_genres)\n",
    "df_movietotal[\"genres\"] = genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal['genres'].replace('', np.nan, inplace=True)\n",
    "df_movietotal.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_poster = df_movietotal[['imdb_id','poster_link']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "not_found = []\n",
    "for index, row in tqdm(df_poster.iterrows()):\n",
    "    url = row['poster_link']\n",
    "    if \"https://m.media-amazon.com/\" in str(url):\n",
    "        id = row['imdb_id']\n",
    "        jpgname = savelocation+id+'.jpg'\n",
    "        urlretrieve(url, jpgname)\n",
    "    else:\n",
    "        not_found.append(index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import listdir\n",
    "from PIL import Image\n",
    "   \n",
    "for filename in listdir(savelocation):\n",
    "    if filename.endswith('.jpg'):\n",
    "        try:\n",
    "            img = Image.open(savelocation+filename) # open the image file\n",
    "            img.verify() # verify that it is, in fact an image\n",
    "        except (IOError, SyntaxError) as e:\n",
    "            print('Bad file:', filename) # print out the names of corrupt files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal.drop(df_movietotal.index[not_found], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns_to_drop = []\n",
    "for i in df_movietotal.columns:\n",
    "    if \"Unnamed\" in i:\n",
    "        columns_to_drop.append(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal.drop(columns_to_drop, axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal.to_csv('movie_metadataWithPoster.csv', sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: _Dataset Manipulation_\n",
    "\n",
    "Before constructing our Machine Learning model, we manipulate the dataset obtained from previous steps in order to _increase our model performances_. <br>\n",
    "First of all we remove movies that has as genre _\"TV Movie\"_ or _\"Foreign\"_ since it is reasonable to think that there are no characteristics that identify them in their posters. \n",
    "\n",
    "Once this is done, we also get a copy of the whole dataset, that is composed by **19103 entries**, which is too big for the hardware we have. So we get a _random sample fraction of 0.4 of it_.\n",
    "\n",
    "Since the original dataset (and so also in the one obtained after sampling) is **heavily imbalanced with respect genres** (there are many occurrences of genres like \"Comedy\" and \"Drama\" and a very low number for other ones), we can add entries not picked in the random sample in order to reduce this imbalance. In this way we are facing the imbalance problem through an ***oversampling approach***, which consists in adding instances of less frequent classes in order to balance the dataset. <br>\n",
    "This step is necessary since _imbalance in the training set can negatively affect the model performances_. <br>\n",
    "Thus, for each genre which has a low number of occurences in the random sample, we get a number of instances from the remaining dataset (original - random sample) which allows to ***make the genre balanced with respect the most frequent one***. <br>\n",
    "Once this procedure is done for each imbalanced genre, we have that our random sample contains now balanced number of movies for each genre (remember that each movie can belong to many genres) and a total size of 8453 movies. <br>\n",
    "So finally we obtain a **balanced sample of movies** that we can use in our model (and which has size suitable to available hardware)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal = pd.read_csv(\"movie_metadataWithPoster.csv\", sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal = df_movietotal[~df_movietotal[\"genres\"].str.contains(\"TV Movie\")]\n",
    "df_movietotal = df_movietotal[~df_movietotal[\"genres\"].str.contains(\"Foreign\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal_copy = df_movietotal.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19103"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_movietotal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal = df_movietotal.sample(frac=.1)\n",
    "df_movietotal_copy = pd.concat([df_movietotal, df_movietotal_copy]).drop_duplicates(keep=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_dict = {\"word2idx\": {}, \"idx2word\": []}\n",
    "idx = 0\n",
    "genre_per_movie = df_movietotal[\"genres\"].apply(lambda x: str(x).split(\"|\")[:-1])\n",
    "for l in [g for d in genre_per_movie for g in d]:\n",
    "    if l in label_dict[\"idx2word\"]:\n",
    "        pass\n",
    "    else:\n",
    "        label_dict[\"idx2word\"].append(l)\n",
    "        label_dict[\"word2idx\"][l] = idx\n",
    "        idx += 1\n",
    "n_classes = len(label_dict[\"idx2word\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def genre_count(df, label_dict):\n",
    "    max_genre = 0\n",
    "    for label in label_dict[\"idx2word\"]:\n",
    "        occurrences = len((df[df['genres'].str.contains(label)]))\n",
    "        print(label, occurrences)\n",
    "        if occurrences > max_genre:\n",
    "            max_genre = occurrences\n",
    "    return max_genre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Crime 242\n",
      "Drama 896\n",
      "Thriller 462\n",
      "Western 56\n",
      "Animation 70\n",
      "Comedy 584\n",
      "Family 129\n",
      "Adventure 220\n",
      "Science Fiction 181\n",
      "Fantasy 122\n",
      "Horror 286\n",
      "Mystery 155\n",
      "Action 381\n",
      "War 75\n",
      "Documentary 106\n",
      "History 70\n",
      "Romance 339\n",
      "Music 88\n"
     ]
    }
   ],
   "source": [
    "max_genre = genre_count(df_movietotal, label_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random over-sampling:\n",
      "Crime 1107\n",
      "Drama 896\n",
      "Thriller 1876\n",
      "Western 1022\n",
      "Animation 1119\n",
      "Comedy 584\n",
      "Family 1137\n",
      "Adventure 1468\n",
      "Science Fiction 1153\n",
      "Fantasy 989\n",
      "Horror 1222\n",
      "Mystery 903\n",
      "Action 2152\n",
      "War 995\n",
      "Documentary 1361\n",
      "History 842\n",
      "Romance 957\n",
      "Music 839\n",
      "2152\n"
     ]
    }
   ],
   "source": [
    "# IMBALANCE: OVERSAMPLING SOLUTION\n",
    "df_movietotal_copy = df_movietotal_copy[~df_movietotal_copy[\"genres\"].str.contains(\"Comedy\")]\n",
    "df_movietotal_copy = df_movietotal_copy[~df_movietotal_copy[\"genres\"].str.contains(\"Drama\")]\n",
    "    \n",
    "for label in label_dict[\"idx2word\"]:\n",
    "    if label not in [\"Drama\", \"Comedy\"]:\n",
    "        len_genre = len(df_movietotal[df_movietotal['genres'].str.contains(label)])\n",
    "        df_genre = df_movietotal_copy[df_movietotal_copy['genres'].str.contains(label)]\n",
    "        #df_genre['genres'] = [label+\"|\" for i in range (0, len(df_genre))]    \n",
    "        if (max_genre - len_genre) > 0:\n",
    "            if len_genre > 3000:\n",
    "                param = 0\n",
    "            elif len_genre > 2000:\n",
    "                param = 0.3\n",
    "            elif len_genre > 1000:\n",
    "                param = 0.5\n",
    "            else:\n",
    "                param = 0.9\n",
    "            df_class_over = df_genre.sample(int((max_genre-len_genre)*param)+1, replace=True)\n",
    "            df_movietotal = pd.concat([df_movietotal, df_class_over], axis=0)\n",
    "\n",
    "print('Random over-sampling:')\n",
    "print(genre_count(df_movietotal, label_dict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8453"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_movietotal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Crime 1107\n",
      "Drama 896\n",
      "Thriller 1876\n",
      "Western 1022\n",
      "Animation 1119\n",
      "Comedy 584\n",
      "Family 1137\n",
      "Adventure 1468\n",
      "Science Fiction 1153\n",
      "Fantasy 989\n",
      "Horror 1222\n",
      "Mystery 903\n",
      "Action 2152\n",
      "War 995\n",
      "Documentary 1361\n",
      "History 842\n",
      "Romance 957\n",
      "Music 839\n",
      "2152\n"
     ]
    }
   ],
   "source": [
    "print(genre_count(df_movietotal, label_dict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: _Poster Images Preprocessing and Final Dataset Construction_\n",
    "\n",
    "Before starting with the Convolutional Neural Network, we need to preprocess the images in order to construct a final dataset that can be used to train our CNN. <br> \n",
    "In the following cells we define functions used for preprocessing. This functions allow to reshape poster images so that all of them has the same size, that will match the input size of our CNN. Once this is done, we read all poster images (using the Python library _imageio_), getting as output a numpy array, which comes with a dict of meta data at its ‘meta’ attribute.\n",
    "\n",
    "Furthermore in this step we perform the ***one-hot-encoding*** our target variable (_\"genres\"_). \n",
    "\n",
    "Finally we can obtain our Final Dataset which has as **_X variable_ poster images numpy arrays** (obtained processing each image) and as **_Y variable_ the target variable \"genres\" one-hot-encoded**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_glob = glob.glob(savelocation + \"*.jpg\")\n",
    "img_dict = {}\n",
    "\n",
    "\n",
    "def get_id(filename):\n",
    "    index_s = filename.rfind(\"/\") + 1\n",
    "    index_f = filename.rfind(\".jpg\")\n",
    "    return filename[index_s:index_f]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "for fn in image_glob:\n",
    "    try:\n",
    "        img_dict[get_id(fn)] = imageio.imread(fn)\n",
    "    except:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_img(id):\n",
    "    title = df_movietotal[df_movietotal[\"imdb_id\"] == id][\"original_title\"].values[0]\n",
    "    genre = df_movietotal[df_movietotal[\"imdb_id\"] == id][\"genres\"].values[0]\n",
    "    plt.imshow(img_dict[id])\n",
    "    plt.title(\"{} \\n {}\".format(title, genre))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(img, size=(150, 101, 3)):\n",
    "    img = skimage.transform.resize(img, size)\n",
    "    img = img.astype(np.float32)\n",
    "    img = (img / 127.5) - 1.\n",
    "    return img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_data(data, img_dict, label_dict, size=(150, 101, 3)):\n",
    "    print(\"Generation dataset...\")\n",
    "    dataset = []\n",
    "    y = []\n",
    "    ids = []\n",
    "    n_samples = len(img_dict)\n",
    "    print(\"got {} posters\".format(n_samples))\n",
    "    for k in img_dict:\n",
    "        if k in data[\"imdb_id\"].values:\n",
    "            G = data[data[\"imdb_id\"] == k][\"genres\"].values\n",
    "            for g in G: \n",
    "                g = g.split(\"|\")[:-1]\n",
    "                img = preprocess(img_dict[k], size)\n",
    "                if img.shape != (150, 101, 3):\n",
    "                    continue\n",
    "                l = np.sum([np.eye(n_classes, dtype=\"uint8\")[label_dict[\"word2idx\"][s]] \n",
    "                                                            for s in g], axis=0)\n",
    "                y.append(l)\n",
    "                dataset.append(img)\n",
    "                ids.append(k)\n",
    "    print(\"DONE\")\n",
    "    print(len(dataset))\n",
    "    return dataset, y, ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movietotal = df_movietotal[['genres', 'id', 'original_title', 'imdb_id', 'title']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation dataset...\n",
      "got 19587 posters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/skimage/transform/_warps.py:105: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.\n",
      "  warn(\"The default mode, 'constant', will be changed to 'reflect' in \"\n",
      "/usr/local/lib/python3.6/dist-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.\n",
      "  warn(\"Anti-aliasing will be enabled by default in skimage 0.15 to \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DONE\n",
      "8453\n"
     ]
    }
   ],
   "source": [
    "SIZE = (150, 101, 3)\n",
    "dataset, y, ids =  prepare_data(df_movietotal, img_dict, label_dict, size=SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 5: _Convolutional Neural Network (using Keras Framework)_\n",
    "\n",
    "### Model Construction \n",
    "Finally we are ready to build our model, namely a Convolutional Neural Network. For this purpose, we use **Keras**, a Python framework which allows to build Machine Learning models. <br>\n",
    "\n",
    "The Keras model type that we will be using is Sequential, which is the easiest way to build a model, since it allows to to build a model layer by layer. <br>\n",
    "Our first 4 layers are __Conv2D layers__. These are **convolution layers** that will deal with our **input images**, which are seen as 2-dimensional matrices. <br>\n",
    "The first of these has **32** nodes, the second and the last have **64** nodes, while the third has **128** nodes. <br>\n",
    "The first layer also takes an input shape, which is is the shape of each input image.\n",
    "\n",
    "Since we are modelling a Convolutional Neural Network, we need also **filter matrices**. These matrices are fundamental in the CNNs, since _this model multiplies a matrix of pixels with a filter matrix (or ‘kernel’) and sums up the multiplication values_. Then the convolution slides over to the next pixel and repeats the same process until all the image pixels have been covered. \n",
    "\n",
    "In our case the size of the filter matrix is 3, which means we will have a 3x3 filter matrix for each Conv2D layer. \n",
    "\n",
    "In between the Conv2D layers and the Dense layer, there is a **Flatten layer**, used as a connection between the convolution and dense layers.\n",
    "\n",
    "The last two layers are **Dense** layers, one with **128** nodes and one (the last one, namely the output layer) with **18** nodes, since our output classes are 18. \n",
    "\n",
    "The activation for the last layer is **sigmoid** since we are dealing with a _multi-label classication problem_.\n",
    "\n",
    "### Training the Model and Predict Genres\n",
    "Once the model is constructed, we are ready to train it. We perform a **training** using the first **4000** (due to hardware limitations). <br>\n",
    "We can see from the Keras log that the **training work quite fine**, since a **loss of 0.39** is achieved. <br> \n",
    "So finally we make the prediction on **100 test** instances and evaluate the results. <br>\n",
    "In order to **evaluate the prediction**, we consider the first 3 genres predicted and check if the each of the real movie genres is reported in this 3. For each contained in the first 3 we get a 1, while we get a 0 for each that is not contained in the first 3 predicted genres. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D, BatchNormalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3), activation='relu',\n",
    "                 input_shape=(SIZE[0], SIZE[1], 3)))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(18, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=keras.optimizers.Adagrad(),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 3600 samples, validate on 400 samples\n",
      "Epoch 1/5\n",
      "3600/3600 [==============================] - 376s 104ms/step - loss: 0.5457 - acc: 0.8561 - val_loss: 0.3905 - val_acc: 0.8642\n",
      "Epoch 2/5\n",
      "3600/3600 [==============================] - 368s 102ms/step - loss: 0.4040 - acc: 0.8653 - val_loss: 0.3888 - val_acc: 0.8642\n",
      "Epoch 3/5\n",
      "3600/3600 [==============================] - 368s 102ms/step - loss: 0.4015 - acc: 0.8652 - val_loss: 0.3877 - val_acc: 0.8642\n",
      "Epoch 4/5\n",
      "3600/3600 [==============================] - 383s 106ms/step - loss: 0.4002 - acc: 0.8653 - val_loss: 0.3879 - val_acc: 0.8642\n",
      "Epoch 5/5\n",
      "3600/3600 [==============================] - 381s 106ms/step - loss: 0.3990 - acc: 0.8654 - val_loss: 0.3878 - val_acc: 0.8642\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fca51898ba8>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 4000\n",
    "model.fit(np.array(dataset[: n]), np.array(y[: n]), batch_size=16, epochs=5,\n",
    "          verbose=1, validation_split=0.1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_test = 100\n",
    "X_test = dataset[n:n + n_test]\n",
    "y_test = y[n:n + n_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(np.array(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy_score(y_test, pred):\n",
    "    value = 0\n",
    "    for i in range(0, len(pred)):\n",
    "        first3_index = np.argsort(pred[i])[-3:]\n",
    "        correct = np.where(y_test[i] == 1)[0]\n",
    "        for j in first3_index:\n",
    "            if j in correct:\n",
    "                value += 1\n",
    "    print(value/len(pred))\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.64\n"
     ]
    }
   ],
   "source": [
    "accuracy_score(y_test, pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Consideration about Results\n",
    "\n",
    "***In the end, an accuracy of 0.65 is reached, which is satisfying considering our first assumption, which was to obtain at least 0.5. ***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}