hands-on/070_imdb_data_exploration_lazy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IMDB data set: opinions and recurrent neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Required imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from keras.datasets import imdb\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the data set, it will be downloaded and cached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = imdb.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data shape and types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shape and type of the input and output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train.shape, x_train.dtype, y_train.shape, y_train.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test.shape, x_test.dtype, y_test.shape, y_test.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both training and test sets have 25,000 examples each.  The input is a list of integers, the output either 0 or 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(x_train[0]), len(x_train[0]), type(x_train[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each training input consists of a list of integers.  Each integer uniquely represents a word, the list represents a text as an ordered sequence of words. The corresponding output is an integer, either 0 or 1, representing the opinion expressed in the review text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the distribution of the review lengths in a histogram, one for the training, the other for the test input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure, axes = plt.subplots(nrows=1, ncols=2, figsize=(15, 5))\n",
    "for i, reviews in enumerate((x_train, x_test)):\n",
    "    review_lengths = map(len, reviews)\n",
    "    axes[i].hist(list(review_lengths), bins=50);\n",
    "figure.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distribution of the words, or features, can also be visualized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following computation is rather time consuming, so its results are pickled, so that they can be reused for demonstartion purposes without redoing the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle_path = Path('feature_count.pkl')\n",
    "if not pickle_path.exists():\n",
    "    feature_counter = Counter()\n",
    "    for review in x_train:\n",
    "        for feature in review:\n",
    "            feature_counter[feature] += 1\n",
    "    with open('feature_count.pkl', 'wb') as pickle_file:\n",
    "        pickle.dump(feature_counter, pickle_file)\n",
    "else:\n",
    "    with open('feature_count.pkl', 'rb') as pickle_file:\n",
    "        feature_counter = pickle.load(pickle_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_counter.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the most common word starts at index 4, which may be unexpected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_counter[0], feature_counter[1], feature_counter[2], feature_counter[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Index 0 serves as padding, 1 as start of a review (note that it occurs as many times as there are reviews in the training set).  For more details, see the section on texts below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(feature_counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.semilogy(list(feature_counter.keys()), list(feature_counter.values()),\n",
    "             '.', alpha=0.3, markersize=1)\n",
    "plt.xlabel('feature')\n",
    "plt.ylabel('frequency');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The features, i.e., the words, follow a Zipf-like distribution, which doesn't come as a surprise.  Since this computation is time consuming, we assume similar results for the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the minimum index is 1, the maximum 88586."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min(feature_counter.keys()), max(feature_counter.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distirbution of the opinions, 0 or 1, can be visualized in a bar plot, again one for the training, the other for the test output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure, axes = plt.subplots(1, 2)\n",
    "for i, opinions in enumerate((y_train, y_test)):\n",
    "    counter = [0, 0]\n",
    "    for opinion in opinions:\n",
    "        counter[opinion] += 1\n",
    "    axes[i].bar(['0', '1'], counter, 0.5);\n",
    "figure.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Positive and negetive opinions are uniformly distributed in the training set and the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The texts represented as a list of integers, each integer representing a specific word.  The word index, i.e, a dictionary that has the words as keys, and the integers as values is also available in the IMDB dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index = imdb.get_word_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index['the']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, in order to translate the lists of integers into the original reviews, the data has to be loaded appropriately.  The `load_data` method has some optional arguments that should be specified.  Index 0 is usually reserved for padding, i.e., to ensure that short sequences can be extended to the required length.  Index 1 indicates the start of a review (`start_char`), while index 2 is used to represent words that have not been indexed, either because they were not part of the data set, they were too infrequently used, or, if the top words are left out, too common to be considered informative (`oov_char`).  Hence, the actual word index starts at 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The word index has to be shifted by `index_from`, and the strings representing padding, start and unknown added.  The following function will do this, compute the reverse dictionary, and return both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_indices(word_index=None, index_from=3, padding_idx=0, start_idx=1, unknown_idx=2):\n",
    "    if word_index is None:\n",
    "        word_index = imdb.get_word_index()\n",
    "    word_to_idx = {k: v + index_from for k, v in word_index.items()}\n",
    "    word_to_idx['<pad>'] = padding_idx\n",
    "    word_to_idx['<start>'] = start_idx\n",
    "    word_to_idx['<unknown>'] = unknown_idx\n",
    "    return word_to_idx, {v: k for k, v in word_to_idx.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_to_idx, idx_to_word = compute_indices(word_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first review in the training set can now be \"translated\" back to English.  Note that there is no punctuation, and, since only the 1,000 most common words were loaded, quite a number of `<unknown>` crop up in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(' '.join(idx_to_word[idx] for idx in x_train[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sentiment expressed in this review is positive, so the output should be 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the first review expressing a negative sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_idx = list(y_train).index(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(' '.join(idx_to_word[idx] for idx in x_train[neg_idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, the reviewer was not taken by the movie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stop words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you display the top-25 words, it is quite clear that most will not be very informative, except \"but\" at index 20 and \"not\" at index 23."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(4, 4 + 26):\n",
    "    print(i, idx_to_word[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would most likely be fine to exclude all most frequent words upto \"but\", and limit the number of words to the 5,000 most frequent ones in order to reduce the dataset size, and hence the computations when training the network."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}