hands-on/070_imdb_data_exploration_complete.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IMDB data set: opinions and recurrent neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Required imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from keras.datasets import imdb\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the data set, it will be downloaded and cached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = imdb.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data shape and types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shape and type of the input and output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((25000,), dtype('O'), (25000,), dtype('int64'))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape, x_train.dtype, y_train.shape, y_train.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((25000,), dtype('O'), (25000,), dtype('int64'))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape, x_test.dtype, y_test.shape, y_test.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both training and test sets have 25,000 examples each.  The input is a list of integers, the output either 0 or 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(list, 218, int)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(x_train[0]), len(x_train[0]), type(x_train[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0, 1}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each training input consists of a list of integers.  Each integer uniquely represents a word, the list represents a text as an ordered sequence of words. The corresponding output is an integer, either 0 or 1, representing the opinion expressed in the review text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the distribution of the review lengths in a histogram, one for the training, the other for the test input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABDAAAAFgCAYAAABNIolGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X3MXuV9J/jvrzikM+kLJjEI2XRNt1an9I8Q1gJGkapMmJq3Uc1IYUV31HgiJK80dJRKs5o61Wrp5EVyVtqmjWaLRINnTJWWsGkjrIYJtUiiaqSBYBJKAjRrlzDBawY8NaHtoKYi/e0fz+XMY/K8wuP7OTz+fKRb55zfue77uc6VmztHX59zruruAAAAAEzZD613BwAAAACWI8AAAAAAJk+AAQAAAEyeAAMAAACYPAEGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMkTYAAAAACTt2m9O7CUd7zjHb19+/b17gYA8BqPPfbYf+3uLevdj9VybgEA07PS84pJBxjbt2/PkSNH1rsbAMBrVNV/Xu8+vB7OLQBgelZ6XuEWEgAAAGDyBBgAAADA5AkwAAAAgMkTYAAAAACTJ8AAAAAAJk+AAQAAAEyeAAMAAACYPAEGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMkTYAAAAACTJ8AAAAAAJm/Tcg2q6qeTfGZe6SeT/B9J7hn17UmeTfI/d/dLVVVJfivJjUleSfLPu/ur47P2JPnfx+d8tLsPrs1hrI/t+z6/bJtn9980g54AABuBcwsAWNyyV2B09ze7+4ruviLJ/5S5UOJzSfYleai7dyR5aGwnyQ1JdozX3iR3JklVXZjkjiRXJ7kqyR1VtXltDwcAAADYiFZ7C8m1Sf68u/9zkt1JTl9BcTDJzWN9d5J7es7DSS6oqkuSXJfkcHef6u6XkhxOcv0bPgIAAABgw1ttgHFrkt8f6xd39/NJMpYXjfrWJM/Ne8/xUVusfoaq2ltVR6rqyMmTJ1fZPQAAAGAjWnGAUVXnJ/mFJP/Pck0XqPUS9TML3Xd1987u3rlly5aVdg8AAADYwFZzBcYNSb7a3S+M7RfGrSEZyxdH/XiSS+e9b1uSE0vUAQAAAJa0mgDjF/Pfbx9JkkNJ9oz1PUnun1d/f825JsnL4xaTB5PsqqrN4+Gdu0YNAAAAYEkrCjCq6u8n+fkkfzivvD/Jz1fV0bFv/6g/kOSZJMeS/E6Sf5Ek3X0qyUeSPDpeHx41AOAcU1UXVNVnq+rPqurpqvqHVXVhVR2uqqNjuXm0rar6ZFUdq6onqurKeZ+zZ7Q/OqZrBwA2qE0radTdryR5+2tqf5G5WUle27aT3L7I5xxIcmD13QQANpjfSvKF7n7feM7W30/ya5mbon1/Ve3L3BTtv5ozp2i/OnNTtF89b4r2nZl7rtZjVXVozHYGAGwwq52FBADgDamqH0vyc0nuTpLu/tvu/k5M0Q4ALEGAAQDM2k8mOZnk31XV16rqU1X1tpiiHQBYggADAJi1TUmuTHJnd78ryX/L3O0iizFFOwAgwAAAZu54kuPd/cjY/mzmAg1TtAMAixJgAAAz1d3/JclzVfXTo3RtkqdiinYAYAkrmoUEAGCN/csknx4zkDyT5AOZ+4eV+6rqtiTfTnLLaPtAkhszN0X7K6NtuvtUVZ2eoj0xRTsAbGgCDABg5rr78cxNf/papmgHABbkFhIAAABg8gQYAAAAwOQJMAAAAIDJE2AAAAAAkyfAAAAAACZPgAEAAABMngADAAAAmDwBBgAAADB5AgwAAABg8gQYAAAAwOQJMAAAAIDJE2AAAAAAkyfAAAAAACZPgAEAAABMngADAAAAmDwBBgAAADB5AgwAAABg8gQYAAAAwOQJMAAAAIDJE2AAAAAAkyfAAAAAACZPgAEAAABMngADAAAAmDwBBgAAADB5AgwAAABg8gQYAAAAwOQJMAAAAIDJW1GAUVUXVNVnq+rPqurpqvqHVXVhVR2uqqNjuXm0rar6ZFUdq6onqurKeZ+zZ7Q/WlV7ztZBAQAAABvLSq/A+K0kX+juf5DknUmeTrIvyUPdvSPJQ2M7SW5IsmO89ia5M0mq6sIkdyS5OslVSe44HXoAAAAALGXZAKOqfizJzyW5O0m6+2+7+ztJdic5OJodTHLzWN+d5J6e83CSC6rqkiTXJTnc3ae6+6Ukh5Ncv6ZHAwAAAGxIK7kC4yeTnEzy76rqa1X1qap6W5KLu/v5JBnLi0b7rUmem/f+46O2WP0MVbW3qo5U1ZGTJ0+u+oAAAACAjWclAcamJFcmubO735Xkv+W/3y6ykFqg1kvUzyx039XdO7t755YtW1bQPQAAAGCjW0mAcTzJ8e5+ZGx/NnOBxgvj1pCM5Yvz2l867/3bkpxYog4AAACwpGUDjO7+L0meq6qfHqVrkzyV5FCS0zOJ7Ely/1g/lOT9YzaSa5K8PG4xeTDJrqraPB7euWvUAAAAAJa0aYXt/mWST1fV+UmeSfKBzIUf91XVbUm+neSW0faBJDcmOZbkldE23X2qqj6S5NHR7sPdfWpNjgIAAADY0FYUYHT340l2LrDr2gXadpLbF/mcA0kOrKaDAAAAACt5BgYAAADAuhJgAAAAAJMnwAAAAAAmT4ABAAAATJ4AAwCYuap6tqq+XlWPV9WRUbuwqg5X1dGx3DzqVVWfrKpjVfVEVV0573P2jPZHq2rPYn8PAHjzE2AAAOvlH3X3Fd19eqazfUke6u4dSR4a20lyQ5Id47U3yZ3JXOCR5I4kVye5Kskdp0MPAGDjEWAAAFOxO8nBsX4wyc3z6vf0nIeTXFBVlyS5Lsnh7j7V3S8lOZzk+ll3GgCYDQEGALAeOskfV9VjVbV31C7u7ueTZCwvGvWtSZ6b997jo7ZY/QxVtbeqjlTVkZMnT67xYQAAs7JpvTsAAJyT3t3dJ6rqoiSHq+rPlmhbC9R6ifqZhe67ktyVJDt37vyB/QDAm4MrMACAmevuE2P5YpLPZe4ZFi+MW0Myli+O5seTXDrv7duSnFiiDgBsQAIMAGCmquptVfWjp9eT7EryjSSHkpyeSWRPkvvH+qEk7x+zkVyT5OVxi8mDSXZV1ebx8M5dowYAbEBuIQEAZu3iJJ+rqmTuXOT3uvsLVfVokvuq6rYk305yy2j/QJIbkxxL8kqSDyRJd5+qqo8keXS0+3B3n5rdYQAAsyTAAABmqrufSfLOBep/keTaBeqd5PZFPutAkgNr3UcAYHrcQgIAAABMniswAADeRLbv+/yybZ7df9MMegIAs+UKDAAAAGDyBBgAAADA5AkwAAAAgMkTYAAAAACTJ8AAAAAAJk+AAQAAAEyeAAMAAACYPAEGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMkTYAAAAACTJ8AAAAAAJk+AAQAAAEyeAAMAAACYPAEGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMlbUYBRVc9W1der6vGqOjJqF1bV4ao6OpabR72q6pNVdayqnqiqK+d9zp7R/mhV7Tk7hwQAAABsNKu5AuMfdfcV3b1zbO9L8lB370jy0NhOkhuS7BivvUnuTOYCjyR3JLk6yVVJ7jgdegAAAAAs5Y3cQrI7ycGxfjDJzfPq9/Sch5NcUFWXJLkuyeHuPtXdLyU5nOT6N/D3AQAAgHPESgOMTvLHVfVYVe0dtYu7+/kkGcuLRn1rkufmvff4qC1WP0NV7a2qI1V15OTJkys/EgAAAGDD2rTCdu/u7hNVdVGSw1X1Z0u0rQVqvUT9zEL3XUnuSpKdO3f+wH4AAADg3LOiKzC6+8RYvpjkc5l7hsUL49aQjOWLo/nxJJfOe/u2JCeWqAMAAAAsadkAo6reVlU/eno9ya4k30hyKMnpmUT2JLl/rB9K8v4xG8k1SV4et5g8mGRXVW0eD+/cNWoAAAAAS1rJLSQXJ/lcVZ1u/3vd/YWqejTJfVV1W5JvJ7lltH8gyY1JjiV5JckHkqS7T1XVR5I8Otp9uLtPrdmRAAAAABvWsgFGdz+T5J0L1P8iybUL1DvJ7Yt81oEkB1bfTQAAAOBc9kamUQUAAACYCQEGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMkTYAAAAACTJ8AAAAAAJk+AAQAAAEyeAAMAAACYPAEGAAAAMHkCDABg5qrqvKr6WlX90di+rKoeqaqjVfWZqjp/1N86to+N/dvnfcaHRv2bVXXd+hwJADArAgwAYD18MMnT87Y/nuQT3b0jyUtJbhv125K81N0/leQTo12q6vIktyb52STXJ/ntqjpvRn0HANaBAAMAmKmq2pbkpiSfGtuV5L1JPjuaHExy81jfPbYz9l872u9Ocm93f7e7v5XkWJKrZnMEAMB6EGAAALP2m0n+dZK/G9tvT/Kd7n51bB9PsnWsb03yXJKM/S+P9t+vL/CeM1TV3qo6UlVHTp48uZbHAQDMkAADAJiZqvonSV7s7sfmlxdo2svsW+o9Zxa77+rund29c8uWLavqLwAwHZvWuwMAwDnl3Ul+oapuTPLDSX4sc1dkXFBVm8ZVFtuSnBjtjye5NMnxqtqU5MeTnJpXP23+ewCADcgVGADAzHT3h7p7W3dvz9xDOL/Y3f8syZeSvG8025Pk/rF+aGxn7P9id/eo3zpmKbksyY4kX5nRYQAA68AVGADAFPxqknur6qNJvpbk7lG/O8nvVtWxzF15cWuSdPeTVXVfkqeSvJrk9u7+3uy7DQDMigADAFgX3f3lJF8e689kgVlEuvtvktyyyPs/luRjZ6+HAMCUuIUEAAAAmDwBBgAAADB5biE5y7bv+/yybZ7df9MMegIAAABvXq7AAAAAACZPgAEAAABMngADAAAAmDwBBgAAADB5AgwAAABg8gQYAAAAwOQJMAAAAIDJE2AAAAAAkyfAAAAAACZPgAEAAABMngADAAAAmLwVBxhVdV5Vfa2q/mhsX1ZVj1TV0ar6TFWdP+pvHdvHxv7t8z7jQ6P+zaq6bq0PBgAAANiYVnMFxgeTPD1v++NJPtHdO5K8lOS2Ub8tyUvd/VNJPjHapaouT3Jrkp9Ncn2S366q895Y9wEAAIBzwYoCjKraluSmJJ8a25XkvUk+O5ocTHLzWN89tjP2Xzva705yb3d/t7u/leRYkqvW4iAAAACAjW2lV2D8ZpJ/neTvxvbbk3ynu18d28eTbB3rW5M8lyRj/8uj/ffrC7wHAAAAYFHLBhhV9U+SvNjdj80vL9C0l9m31Hvm/729VXWkqo6cPHlyue4BAAAA54CVXIHx7iS/UFXPJrk3c7eO/GaSC6pq02izLcmJsX48yaVJMvb/eJJT8+sLvOf7uvuu7t7Z3Tu3bNmy6gMCAAAANp5lA4zu/lB3b+vu7Zl7COcXu/ufJflSkveNZnuS3D/WD43tjP1f7O4e9VvHLCWXJdmR5CtrdiQAAADAhrVp+SaL+tUk91bVR5N8Lcndo353kt+tqmOZu/Li1iTp7ier6r4kTyV5Ncnt3f29N/D3AQAAgHPEqgKM7v5yki+P9WeywCwi3f03SW5Z5P0fS/Kx1XYSAAAAOLetdBYSAAAAgHUjwAAAAAAmT4ABAAAATJ4AAwAAAJg8AQYAAAAweQIMAAAAYPIEGAAAAMDkCTAAAACAyRNgAAAAAJMnwAAAAAAmT4ABAAAATJ4AAwAAAJg8AQYAAAAweQIMAAAAYPIEGAAAAMDkCTAAAACAyRNgAAAzVVU/XFVfqao/raonq+rfjPplVfVIVR2tqs9U1fmj/taxfWzs3z7vsz406t+squvW54gAgFkQYAAAs/bdJO/t7ncmuSLJ9VV1TZKPJ/lEd+9I8lKS20b725K81N0/leQTo12q6vIktyb52STXJ/ntqjpvpkcCAMzMpvXuAABwbunuTvLXY/Mt49VJ3pvkfxn1g0l+PcmdSXaP9ST5bJJ/W1U16vd293eTfKuqjiW5Ksl/OvtHMW3b931+yf3P7r9pRj0BgLXjCgwAYOaq6ryqejzJi0kOJ/nzJN/p7ldHk+NJto71rUmeS5Kx/+Ukb59fX+A9AMAGI8AAAGauu7/X3Vck2Za5qyZ+ZqFmY1mL7Fusfoaq2ltVR6rqyMmTJ19vlwGAdSbAAADWTXd/J8mXk1yT5IKqOn1767YkJ8b68SSXJsnY/+NJTs2vL/Ce+X/jru7e2d07t2zZcjYOAwCYAQEGADBTVbWlqi4Y638vyT9O8nSSLyV532i2J8n9Y/3Q2M7Y/8XxHI1DSW4ds5RclmRHkq/M5igAgFnzEE8AYNYuSXJwzBjyQ0nu6+4/qqqnktxbVR9N8rUkd4/2dyf53fGQzlOZm3kk3f1kVd2X5Kkkrya5vbu/N+NjAQBmRIABAMxUdz+R5F0L1J/J3PMwXlv/myS3LPJZH0vysbXuIwAwPW4hAQAAACZPgAEAAABMngADAAAAmDwBBgAAADB5AgwAAABg8gQYAAAAwOQJMAAAAIDJE2AAAAAAkyfAAAAAACZv2QCjqn64qr5SVX9aVU9W1b8Z9cuq6pGqOlpVn6mq80f9rWP72Ni/fd5nfWjUv1lV152tgwIAAAA2lpVcgfHdJO/t7ncmuSLJ9VV1TZKPJ/lEd+9I8lKS20b725K81N0/leQTo12q6vIktyb52STXJ/ntqjpvLQ8GAAAA2JiWDTB6zl+PzbeMVyd5b5LPjvrBJDeP9d1jO2P/tVVVo35vd3+3u7+V5FiSq9bkKAAAAIANbUXPwKiq86rq8SQvJjmc5M+TfKe7Xx1NjifZOta3JnkuScb+l5O8fX59gffM/1t7q+pIVR05efLk6o8IAAAA2HBWFGB09/e6+4ok2zJ31cTPLNRsLGuRfYvVX/u37urund29c8uWLSvpHgAAALDBrWoWku7+TpIvJ7kmyQVVtWns2pbkxFg/nuTSJBn7fzzJqfn1Bd4DAAAAsKiVzEKypaouGOt/L8k/TvJ0ki8led9otifJ/WP90NjO2P/F7u5Rv3XMUnJZkh1JvrJWBwIAAABsXJuWb5JLkhwcM4b8UJL7uvuPquqpJPdW1UeTfC3J3aP93Ul+t6qOZe7Ki1uTpLufrKr7kjyV5NUkt3f399b2cAAAAICNaNkAo7ufSPKuBerPZIFZRLr7b5LcsshnfSzJx1bfTQAAAOBctqpnYAAAAACsBwEGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMkTYAAAAACTJ8AAAAAAJk+AAQAAAEyeAAMAAACYvE3r3QGS7fs+v2ybZ/ffNIOeAAAAwDS5AgMAAACYPAEGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMkzCwkAwAysZNYxAGBxAoxFOMkAAACA6XALCQAAADB5AgwAAABg8gQYAAAAwOQJMAAAAIDJE2AAAAAAkyfAAABmqqouraovVdXTVfVkVX1w1C+sqsNVdXQsN496VdUnq+pYVT1RVVfO+6w9o/3RqtqzXscEAJx9AgwAYNZeTfKvuvtnklyT5PaqujzJviQPdfeOJA+N7SS5IcmO8dqb5M5kLvBIckeSq5NcleSO06EHALDxCDAAgJnq7ue7+6tj/a+SPJ1ka5LdSQ6OZgeT3DzWdye5p+c8nOSCqrokyXVJDnf3qe5+KcnhJNfP8FAAgBkSYAAA66aqtid5V5JHklzc3c8ncyFHkotGs61Jnpv3tuOjtlj9tX9jb1UdqaojJ0+eXOtDAABmRIABAKyLqvqRJH+Q5Fe6+y+XarpArZeon1novqu7d3b3zi1btry+zgIA606AAQDMXFW9JXPhxae7+w9H+YVxa0jG8sVRP57k0nlv35bkxBJ1AGADEmAAADNVVZXk7iRPd/dvzNt1KMnpmUT2JLl/Xv39YzaSa5K8PG4xeTDJrqraPB7euWvUAIANaNN6dwAAOOe8O8kvJfl6VT0+ar+WZH+S+6rqtiTfTnLL2PdAkhuTHEvySpIPJEl3n6qqjyR5dLT7cHefms0hAACzJsAAAGaqu/9jFn5+RZJcu0D7TnL7Ip91IMmBtesdADBVbiEBAAAAJk+AAQAAAEyeW0gAAM4x2/d9ftk2z+6/aQY9AYCVW/YKjKq6tKq+VFVPV9WTVfXBUb+wqg5X1dGx3DzqVVWfrKpjVfVEVV0577P2jPZHq2rPYn8TAAAAYL6V3ELyapJ/1d0/k+SaJLdX1eVJ9iV5qLt3JHlobCfJDUl2jNfeJHcmc4FHkjuSXJ3kqiR3nA49AAAAAJaybIDR3c9391fH+l8leTrJ1iS7kxwczQ4muXms705yT895OMkFVXVJkuuSHO7uU939UpLDSa5f06MBAAAANqRVPcSzqrYneVeSR5Jc3N3PJ3MhR5KLRrOtSZ6b97bjo7ZY/bV/Y29VHamqIydPnlxN9wAAAIANasUBRlX9SJI/SPIr3f2XSzVdoNZL1M8sdN/V3Tu7e+eWLVtW2j0AAABgA1tRgFFVb8lcePHp7v7DUX5h3BqSsXxx1I8nuXTe27clObFEHQAAAGBJK5mFpJLcneTp7v6NebsOJTk9k8ieJPfPq79/zEZyTZKXxy0mDybZVVWbx8M7d40aAAAAwJI2raDNu5P8UpKvV9Xjo/ZrSfYnua+qbkvy7SS3jH0PJLkxybEkryT5QJJ096mq+kiSR0e7D3f3qTU5CgAAAGBDWzbA6O7/mIWfX5Ek1y7QvpPcvshnHUhyYDUdBAAAAFjVLCQAAAAA60GAAQAAAEyeAAMAAACYPAEGAAAAMHkCDAAAAGDyBBgAAADA5C07jSrTsH3f55dt8+z+m2bQEwAAAJg9V2AAAAAAkyfAAAAAACZPgAEAAABMnmdgAADwAzx/C4CpcQUGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMkTYAAAAACTJ8AAAAAAJk+AAQAAAEyeAAMAAACYPAEGAAAAMHkCDAAAAGDyBBgAAADA5AkwAAAAgMnbtN4dYO1s3/f5Zds8u/+mGfQEAAAA1pYrMAAAAIDJE2AAAAAAkyfAAAAAACZPgAEAAABMngADAJipqjpQVS9W1Tfm1S6sqsNVdXQsN496VdUnq+pYVT1RVVfOe8+e0f5oVe1Zj2MBAGbHLCQAwKz9+yT/Nsk982r7kjzU3furat/Y/tUkNyTZMV5XJ7kzydVVdWGSO5LsTNJJHquqQ9390syOAjOgATBTrsAAAGaqu/8kyanXlHcnOTjWDya5eV79np7zcJILquqSJNclOdzdp0ZocTjJ9We/9wDAehFgAABTcHF3P58kY3nRqG9N8ty8dsdHbbH6D6iqvVV1pKqOnDx5cs07DgDMhgADAJiyWqDWS9R/sNh9V3fv7O6dW7ZsWdPOAQCzI8AAAKbghXFrSMbyxVE/nuTSee22JTmxRB0A2KCWDTA8KRwAmIFDSU6fH+xJcv+8+vvHOcY1SV4et5g8mGRXVW0e5yG7Rg0A2KBWcgXGv88PPhTr9JPCdyR5aGwnZz4pfG/mnhSeeU8KvzrJVUnuOB16AADnlqr6/ST/KclPV9Xxqrotyf4kP19VR5P8/NhOkgeSPJPkWJLfSfIvkqS7TyX5SJJHx+vDowYAbFDLTqPa3X9SVdtfU96d5D1j/WCSL2duqrPvPyk8ycNVdfpJ4e/JeFJ4klTV6SeF//4bPgIA4E2lu39xkV3XLtC2k9y+yOccSHJgDbsGAEzYsgHGIs54UnhVremTwjN39UZ+4id+4nV2b3krmbccAAAAmIbXG2AsZk2eFJ7kriTZuXPngm14/VYS3Dy7/6YZ9AQAAABW7vXOQuJJ4QAAAMDMvN4Aw5PCAQAAgJlZ9haS8aTw9yR5R1Udz9xsIvuT3DeeGv7tJLeM5g8kuTFzTwp/JckHkrknhVfV6SeFJ54UDgBwTnD7KgBrZSWzkHhSOAAAALCuXu8tJAAAAAAzs9azkAAAnHNMzw4AZ58rMAAAAIDJE2AAAAAAkyfAAAAAACZPgAEAAABMnod48gOWexCZudoBAACYNQEGAADraiWzuPgHFADcQgIAAABMngADAAAAmDwBBgAAADB5AgwAAABg8jzEk1XzoC0AAABmzRUYAAAAwOQJMAAAAIDJE2AAAAAAk+cZGAAATJ5ncAHgCgwAAABg8lyBwVnhX0kAgFlz/gGwsbkCAwAAAJg8AQYAAAAweQIMAAAAYPI8A4N14z5VAGDWnH8AvHm5AgMAAACYPAEGAAAAMHluIWHSXOYJAABAIsAAAIAzLPcPKP7xBGB9uIUEAAAAmDxXYPCm5zYTAGCWnHsArA9XYAAAAACT5woMzgn+pQQAmCXnHgBrT4ABgxMNAGCWnHsArI4AA1ZhVk8ld0IDACTTOycwQwuwngQYsIZmeZIxtRMaAGB9rOScAGAjmHmAUVXXJ/mtJOcl+VR37591H2A9OckAWDvOKwDg3DHTWUiq6rwk/3eSG5JcnuQXq+ryWfYBANgYnFcAwLll1tOoXpXkWHc/091/m+TeJLtn3AcAYGNwXgEA55BZ30KyNclz87aPJ7l6foOq2ptk79j866r65hr34R1J/usafyZnMsZn34rGuD4+g55sbL7LZ58xPvvO1hj/D2fhM1dr2fOKxLnFOcD4r68zxt+5x8z5/q8v4792VnReMesAoxao9Rkb3XclueusdaDqSHfvPFufjzGeBWM8G8b57DPGZ98GH+NlzysS5xYbnfFfX8Z/fRn/9WX8Z2/Wt5AcT3LpvO1tSU7MuA8AwMbgvAIAziGzDjAeTbKjqi6rqvOT3Jrk0Iz7AABsDM4rAOAcMtNbSLr71ar65SQPZm66swPd/eQs+5CzeAkp32eMzz5jPBvG+ewzxmffhh3jiZxXJBt4jN8kjP/6Mv7ry/ivL+M/Y9X9A7eKAgAAAEzKrG8hAQAAAFg1AQYAAAAweedMgFFV11fVN6vqWFXtW+/+vJlV1bNV9fWqeryqjozahVV1uKqOjuXmUa+q+uQY9yeq6sr17f10VdWBqnqxqr4xr7bqca2qPaP90arasx7HMlWLjPGvV9X/N77Pj1fVjfP2fWiM8Ter6rp5db8ni6iqS6vqS1X1dFU9WVUfHHXf5TW0xDj7Ps/2gPQGAAAD9ElEQVSQsZsN5x2z55xkfTlfWV/OZSauuzf8K3MP9vrzJD+Z5Pwkf5rk8vXu15v1leTZJO94Te3/TLJvrO9L8vGxfmOS/5CkklyT5JH17v9UX0l+LsmVSb7xesc1yYVJnhnLzWN983of21Rei4zxryf53xZoe/n4rXhrksvGb8h5fk+WHeNLklw51n80yf87xtJ3eTbj7Ps8u/8NjN3sxtp5x+zH3DnJ9Mbf7/vsxt+5zIRf58oVGFclOdbdz3T33ya5N8nude7TRrM7ycGxfjDJzfPq9/Sch5NcUFWXrEcHp667/yTJqdeUVzuu1yU53N2nuvulJIeTXH/2e//msMgYL2Z3knu7+7vd/a0kxzL3W+L3ZAnd/Xx3f3Ws/1WSp5Nsje/ymlpinBfj+7z2jN36ct5xFjknWV/OV9aXc5lpO1cCjK1Jnpu3fTxLn+ixtE7yx1X1WFXtHbWLu/v5ZO4/+iQXjbqxf2NWO67G+/X55XHJ34HTlwPGGL9hVbU9ybuSPBLf5bPmNeOc+D7PirGbHecd0+B3fP35fZ8x5zLTc64EGLVAzfyxr9+7u/vKJDckub2qfm6Jtsb+7FhsXI336t2Z5H9MckWS55P8X6NujN+AqvqRJH+Q5Fe6+y+XarpAzTiv0ALj7Ps8O8Zudpx3TJvfl9nw+z5jzmWm6VwJMI4nuXTe9rYkJ9apL2963X1iLF9M8rnMXaL2wulLNMfyxdHc2L8xqx1X471K3f1Cd3+vu/8uye9k7vucGOPXrarekrn/w/90d//hKPsur7GFxtn3eaaM3Yw475gMv+PryO/7bDmXma5zJcB4NMmOqrqsqs5PcmuSQ+vcpzelqnpbVf3o6fUku5J8I3PjefrJunuS3D/WDyV5/3g67zVJXj596RUrstpxfTDJrqraPC4t3DVqLOI190b/08x9n5O5Mb61qt5aVZcl2ZHkK/F7sqSqqiR3J3m6u39j3i7f5TW02Dj7Ps+UsZsB5x2T4nd8Hfl9nx3nMtO2ab07MAvd/WpV/XLmvjDnJTnQ3U+uc7ferC5O8rm5/66zKcnvdfcXqurRJPdV1W1Jvp3kltH+gcw9mfdYkleSfGD2XX5zqKrfT/KeJO+oquNJ7kiyP6sY1+4+VVUfydz/aSXJh7t7pQ+B2vAWGeP3VNUVmbuk79kk/2uSdPeTVXVfkqeSvJrk9u7+3vgcvyeLe3eSX0ry9ap6fNR+Lb7La22xcf5F3+fZcG4xM8471oFzkvXlfGXdOZeZsOp2Gw4AAAAwbefKLSQAAADAm5gAAwAAAJg8AQYAAAAweQIMAAAAYPIEGAAAAMDkCTAAAACAyRNgAAAAAJP3/wNF0MPz6k0g/gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "figure, axes = plt.subplots(nrows=1, ncols=2, figsize=(15, 5))\n",
    "for i, reviews in enumerate((x_train, x_test)):\n",
    "    review_lengths = map(len, reviews)\n",
    "    axes[i].hist(list(review_lengths), bins=50);\n",
    "figure.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distribution of the words, or features, can also be visualized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following computation is rather time consuming, so its results are pickled, so that they can be reused for demonstartion purposes without redoing the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle_path = Path('feature_count.pkl')\n",
    "if not pickle_path.exists():\n",
    "    feature_counter = Counter()\n",
    "    for review in x_train:\n",
    "        for feature in review:\n",
    "            feature_counter[feature] += 1\n",
    "    with open('feature_count.pkl', 'wb') as pickle_file:\n",
    "        pickle.dump(feature_counter, pickle_file)\n",
    "else:\n",
    "    with open('feature_count.pkl', 'rb') as pickle_file:\n",
    "        feature_counter = pickle.load(pickle_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(4, 336148),\n",
       " (5, 164097),\n",
       " (6, 163040),\n",
       " (7, 145847),\n",
       " (8, 135708),\n",
       " (9, 107313),\n",
       " (10, 101871),\n",
       " (11, 93934),\n",
       " (12, 79058),\n",
       " (13, 77142)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_counter.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the most common word starts at index 4, which may be unexpected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 25000, 1, 0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_counter[0], feature_counter[1], feature_counter[2], feature_counter[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Index 0 serves as padding, 1 as start of a review (note that it occurs as many times as there are reviews in the training set).  For more details, see the section on texts below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "88585"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(feature_counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHAxJREFUeJzt3XmUXOV95vHvQ2tpJBoJIYFBQkhEoIkSH7aOWJI42Ma2gAg4NgngHAccgoIzYMcZJpGHzATbSQ5k5mQyGGyiiQkmxwFjwsFisQUHG/CCgRarWGQ6QkZtZC2jhZZEa/3NH3Vbqm5VV1d31+1bt+7zOadP1X2r7q23L0U/epf7XkUEZmZm/R2SdQXMzKwxOSDMzKwiB4SZmVXkgDAzs4ocEGZmVpEDwszMKnJAmJlZRQ4IMzOryAFhZmYVjcm6AiMxderUmDVrVtbVMDPLleXLl2+MiGmDvS+XASFpIbBwzpw5dHR0ZF0dM7NckfTzWt6Xyy6miHgwIhZNmjQp66qYmTWtXAaEmZmlL5cBIWmhpCVbt27NuipmZk0rlwHhLiYzs/TlMiDMzCx9DggzM6solwHhMQgzs/TlMiDqMQbR3bO7jjUyM2s+uQyIkeru2c2zb21ySJiZVVHIgGhrHcv82VNoax2bdVXMzBpWLgOiHmMQDgczs+pyGRC+DsLMLH25DAgzM0ufA8LMzCpyQJiZWUUOCDMzqyiXAeErqc3M0pfLgPAsJjOz9OUyIOph7Zb3sq6CmVlDK2RArN3yHnc/+3OHhJlZFYUMiGMmH8qFJ0/nmMmHZl0VM7OGVciA6O7ZzRu/fNeL9ZmZVVHIgChR1hUwM2toDRMQks6R9ENJt0s6J83Pamsdy+/MneYF+8zMqkg1ICTdIWm9pBX9yhdIWimpU9LipDiAbUAr0JVmvcCruZqZDSbtFsSdwILyAkktwG3AecA84HJJ84AfRsR5wF8CX0y5Xh5/MDMbRKoBERFPAZv6Fc8HOiNiVUTsAu4BLoqIfcnrm4Hxadaru2c3T67c4JAwM6tiTAafOR1YU7bdBZwh6ePAx4DJwK0D7SxpEbAIYObMmcOuRM+uvcPe18ysCLIIiErThyIi7gfuH2zniFgCLAFob2+P4VaidVzLcHc1MyuELGYxdQHHlW3PAN4ZygFGulifZzGZmQ0ui4B4DjhR0mxJ44DLgKVDOUA9Futbt7Vn2PuamRVB2tNc7waeBuZK6pJ0VUTsAa4FlgGvA/dGxKtDPO6IWhCd67r5P4+vpHNd97D2NzMrAkUMuxs/c+3t7dHR0TGsfV96ezMnzzyizjUyM2t8kpZHRPtg72uYK6mHYqQtiO6e3WzcvsvTXM3MqshlQIx0DKKtdSzzZ0/xILWZWRW5DIh63HLU4WBmVl0uA8K3HDUzS18uA6IePP5gZlZdLgOiHoPUXovJzKy6XAZEPbqYVm/0NRBmZtXkMiBGatX6bXz/jQ2sWr8t66qYmTWsXAbESLuYTp55BP/1Y3N9oZyZWRW5DIiRdjF19+xmx+59HoMwM6silwExUm2tY5l3zOG+FsLMrIpCBkR3z26W/3yzWxBmZlUUMiAAfrn1vayrYGbW0HIZECMdpF63tYcn3ljve0KYmVWRy4Coy1Ibyu8y52ZmoyGXATFSR09q5axfmcrRk1qzroqZWcMqZEBs69nD2i072dazJ+uqmJk1rEIGBMCuvXuzroKZWUMrbEDs2OXWg5lZNbkMiJHOYlr/bg+rN+5g/buexWRmNpBcBsRIZzGdcNRhzJ81hROOOqzONTMzax65DIiRWrV+G0+v2ujVXM3MqihkQEwcP4axY1qYOH5M1lUxM2tYhQyI7Tv30LNrD9t3eqDazGwghQyIow5vZe7RbRx1uC+UMzMbSCEDAmDXPi+1YWZWTUMFhKSJkpZL+t00P2f1xm28sGYzqzd6kNrMbCCpBoSkOyStl7SiX/kCSSsldUpaXPbSXwL3plmnXrFvHzt2+WpqM7OBpN2CuBNYUF4gqQW4DTgPmAdcLmmepHOB14B1KdeJaW2tHD+1jRfX+KZBZmYDSXWeZ0Q8JWlWv+L5QGdErAKQdA9wEXAYMJFSaLwn6ZGI2JdGveYc3cZnfudXuOPHb7Fua49vPWpmVkEWFwJMB9aUbXcBZ0TEtQCSrgQ2DhQOkhYBiwBmzpw57Eocf+RE3vYYhJnZgLIYpFaFsv1TiiLizoh4aKCdI2IJ8EXg+XHjxg27Et95oYs1W3fxnRe6hn0MM7NmlkVAdAHHlW3PAN4ZygHqcUe5c+e9j8mt4tx57xv2MczMmlkWAfEccKKk2ZLGAZcBS4dygJGu5gqwcdtO9uwLNm7bOexjmJk1s7Snud4NPA3MldQl6aqI2ANcCywDXgfujYhXh3LcerQgJoxroXXsWCaMaxn2MczMmlnas5guH6D8EeCR4R5X0kJg4Zw5c4Z7CKa1tdI2roVpbV5uw8yskoa6krpW9WhBvPD2ZlZv7uGFtzfXsWZmZs0jlwFRD1t27CKSRzMzO1guA6Ieg9TzZx/JEa2HMH/2kXWsmZlZ88hlQNSji2nNph1s79nHmk076lgzM7PmkcuAqJtKl+yZmRmQ04CoRxfTcVMmcFjrIRw3ZUIda2Zm1jxyGRD16GJ6uWsLm97bx8tdW+pYMzOz5pHLgKiH9e/29Hk0M7O+chkQ9ehi6r0fte9LbWZWWS4Doh5dTK1jW/o8mplZX7kMiHroXNfd59HMzPoqbEBs3r6rz6OZmfVV2IAwM7PqchkQ9RikPmLiuD6PZmbWVy4Doh6D1GZmVt2gASFpymhUZLSNG3NIn0czM+urlr+Oz0j6tqTzJXn1IjOzgqglIE4ClgCfAjol/Z2kk9KtVvqeeGN9n0czM+tr0ICIkseS24f+MXAF8KykJyWdlXoNU3Ls5NY+j2Zm1lctYxBHSvqcpA7geuA6YCrwX4B/S7l+A9VpxLOYTnrf4QDMmnpYvaplZtZUaulieho4HLg4Ii6IiPsjYk9EdAC3p1u9yuoxi+knnRsBuPNHq+nu2V2vqpmZNY1aAmJuRHw5Irr6vxARN6dQp1Hx79f+Nm1jYCdw/T3PZ10dM7OGU0tAPCppcu+GpCMkLUuxTqOmt3vpnS1e8tvMrL9aAmJaROy/q05EbAaOSq9Ko2fKhLF9Hs3M7IBaAmKvpJm9G5KOByK9Ko2eX767s8+jmZkdMKaG99wA/EjSk8n2B4BF6VVp9OzdF30ezczsgEEDIiK+J+k04ExAwOcjYmO9KyLpV4HPUZpC+3hEfK3en9Hfrr17+zyamdkBtS5ENB7YBGwF5kn6QC07SbpD0npJK/qVL5C0UlKnpMUAEfF6RFwD/D7QXvuvMHxjDjmkz6OZmR0waAtC0s3ApcCrwL6kOICnajj+ncCtwF1lx2sBbgM+AnQBz0laGhGvSboQWJzsk7pDkqWlDvESU2ZmB6llDOJiStdCDHkkNyKekjSrX/F8oDMiVgFIuge4CHgtIpYCSyU9zChcpb1tV+lucms2vZf2R5mZ5U4tfSurgHrOA50OrCnb7gKmSzpH0i2S/gl4ZKCdJS2S1CGpY8OGDSOqyHu7SmMPO4EvPvDKiI5lZtZsamlB7ABelPQ4pb+lAETEZ4f5mZX6cyIingCeGGzniFgiaS2wcNy4cacPsw4AvPSlC5i1+GEA7vnp2/z1xe8fyeHMzJpKLQGxNPmply7guLLtGcA7QzlARDwIPNje3n71SCtzbNsY3unewxFttZwKM7PiqGWa6zckHQrMjIiVdfjM54ATJc0GfgFcBnxyKAeQtBBYOGfOnBFXZlP3nj6PZmZWUsty3wuBF4HvJdunSKqpRSHpbkqrwc6V1CXpqojYA1wLLANeB+6NiFeHUmnfk9rMLH219KvcSGnm0RMAEfFi8q//QSU3GapU/ghVBqIHU88WxJgWYG/yaGZm+9Uyi2lPRPS/M0+ma1PUswWxPbmIepsvpjYz66OWgFgh6ZNAi6QTJX0F+EnK9Ro1P1n8of3PZyczmszMrLaAuA74NUpTXO8G3gX+LM1KDaYetxztdczkQxmfPPfdqc3MDhg0ICJiR0TcEBG/ERHtyfNM77BT70HqXcmjbxtkZnZALWsx/YAKYw4R8aEKbx8V9RykhgO/nBf9NjM7oJZZTNeXPW8FPgFketFAPS+UMzOzymrpYlpe9vPjiPhz4IxRqNuoebpsoHqWB6rNzIDaupimlG0eApwOvC+1GmXgmMmHZl0FM7OGU0sX03JK3fOi1LX0FnBVmpUaTL3HIMzM7GC1rMVU01XToyntMYi1W95zq8LMCk8R1efuSPp4tdcj4v661mgI2tvbo6Ojo27HKx9/WH3TBXU7rplZI5G0PCIGvbVzLV1MVwFnA99Ptj9IaV2mrZS6njILCDMzS08tARHAvIhYCyDpGOC2iPh0qjWrwmMQZmbpq2WpjVm94ZBYB5yUUn1qktZy36cc27b/uae7mlnR1RIQT0haJulKSVcADwM/SLlemXjgsx/IugpmZg2jlgvlrgVuB04GTgGWRMR1aVesEfzJnc9kXQUzs8zU0oIAeB54OCI+DyyT1DbYDnlVPntp2RsbM6yJmVm2arnl6NXAfcA/JUXTgQfSrJSZmWWvlhbEfwZ+k9J9IIiIN4Gj0qxU1trK5nZ5sNrMiqqWgNgZEb23TEDSGDJeGbueNwyq5JW/8UVyZma1BMSTkv4bcKikjwDfBh5Mt1rVpTXNdSD/ya0IMyugWgJiMbABeAX4E+AR4K/SrFQjKB+s9p3mzKyIqgaEpBbgroj4vxHxexFxSfK8cDdf81iEmRVN1YCIiL3ANEnjRqk+DcUL9plZkdWyFtNq4MeSlgLbewsj4h/SqlSjmrX4YYeGmRXGgC0ISf+aPL0UeCh5b1vZTyE4EMysqKq1IE6XdDzwNvCV0aiMpIuBCyhdZ3FbRDw6Gp87FG5FmFlRVBuDuB34HqWVWzvKfpYnjzWRdIek9ZJW9CtfIGmlpE5JiwEi4oGIuBq4klLLpSH0DwRPezWzIhgwICLiloj4VeBfIuKEsp/ZEXHCED7jTmBBeUEyO+o24DxgHnC5pHllb/mr5PWG4WmvZlY0tazm+pmRfEBEPAVs6lc8H+iMiFXJVdr3ABep5GbguxHxfKXjSVokqUNSx4YNG0ZStSGbO3XC/uee9mpmza7W1VzrbTqwpmy7Kym7DjgXuETSNZV2jIglEdEeEe3Tpk1Lv6Zlll3/wT7bDgkza2ZZBYQqlEXSrXV6RFwTEbcPuHPKazFV0388YrZDwsyaVFYB0QUcV7Y9A3in1p1Hey2m/spDIvCNhcysOWUVEM8BJ0qanVylfRmwtNads2xB9PKNhcys2aUeEJLuBp4G5krqknRVROwBrgWWAa8D90bEq7UeM+sWRK8zZh74fI9HmFmzUR7X3ZO0EFg4Z86cq998881M69I/GHwRnZk1OknLI6J9sPdl1cU0Io3SgoCDA8EtCTNrFrkMiEYYgyjnkDCzZpTLgGikFkQvh4SZNZtcBkSjqhQSV3396YxqY2Y2MrkMiEbrYirXPyQef3OTWxNmlku5DIhG7GIqV2kmk0PCzPImlwGRB6tvusDjEmaWaw6IlDkkzCyvchkQjTwGUUmlkFi75b2MamNmVptcXkndq729PTo6ar65XeYqtR585bWZjbamvpI6r1bfdAGnzzi8T5m7nMysUbkFkZH+wdAC/IdbE2Y2Cpq6BZG3MYhK+nct7cWtCTNrLG5BZGygUPDYhJmlpalbEM2k0vUS4NaEmWXPAdEgVt90AWP7lc1a/LCDwswy44BoIG9WaU1c+tUfZVAjMysyB0QDqhQSz7y91S0KMxtVuRykbqRbjqbtt/7uUbre3V3xNQ9km9lw1DpIncuA6NUMs5hqNVDLYdbk8Tyx+NxRro2Z5ZlnMTWZ3tlOR03s+59s9Zad7nYys1S4BZFTvn7CzIbLXUwFMVBQTB4PL37RYWFmB3MXU0GsvukC/qB9xkHlW3b6OgozGxm3IJpId89u3n/jo1Xf4y4oM8tdF5OkE4AbgEkRcUkt+zggBlZLy8FhYVZMDdHFJOkOSeslrehXvkDSSkmdkhYDRMSqiLgqzfoUyUBrPJVz95OZVTMm5ePfCdwK3NVbIKkFuA34CNAFPCdpaUS8lnJdCqk8JCoFQm+ZWxNm1l+qARERT0ma1a94PtAZEasAJN0DXAQ4IFLWGwLVgqL/e82suLKYxTQdWFO23QVMl3SkpNuBUyV9YaCdJS2S1CGpY8OGDWnXtSnV8sff3U9mlnYXUyWqUBYR8f+AawbbOSKWAEugNEhd57oVxmBdT/3Lv3zhPD519uzU62VmjSOLgOgCjivbngG8M5QDlC3WV896FVYtYfHfl5Z6AB0SZsWR+jTXZAzioYj49WR7DPAz4MPAL4DngE9GxKtDPbanuaaj1u6luVMnsOz6D6ZcGzOrt4a4DkLS3cA5wFRgHfDXEfF1SecD/wi0AHdExN8O8biFWe67EVQLDIeEWf40RECkzS2I0TPUQWvPgjJrXE0dEG5BZM9XapvlV1MHRC+3ILI1kqmwDg+z7DR1QLgF0biGEhoOCbNsNMRaTGmJiAcjYtGkSZOyror1U+sffYeDWePLZQuil7uY8smtDLNsNXULQtJCSUu2bt2adVVsiIY6buElP8yyk8uAcBdTfg21ReAWhFl23MVkDcuzpMzS4S4my7WRdi25a8ps5HIZEO5ian4jbQG4BWE2clms5mpWE/+RN8tWLlsQZmaWPrcgrOl09+zm/Tc+Wvfj/v6px/L3l55a9+OaNapctiA8SG0DSSscAO594R3+4lsvpHJss0aUy4DwILUNpK11LK/c+NFUju0WhBWNu5is6bS1jvUAt1kd5LIFYWZm6XNAmJlZRe5iMqtRI1+d7S41S0MuWxCexWSjrZHDARq/fpZPuQwIz2Ky0dbo/0Jv9PpZPrmLyaxG/iNsRZPLFoSZmaXPAWFmZhU5IMzMrCIHhJmZVdQwg9SSJgJfBXYBT0TENzOukplZoaXagpB0h6T1klb0K18gaaWkTkmLk+KPA/dFxNXAhWnWy8zMBpd2F9OdwILyAkktwG3AecA84HJJ84AZwJrkbXtTrpeZmQ0i1S6miHhK0qx+xfOBzohYBSDpHuAioItSSLzIKIyNdPfspq11bNofY5Y6X0VdTFMniI7/cX6qn5HFIPV0DrQUoBQM04H7gU9I+hrw4EA7S1okqUNSx4YNG4ZVge6e3Tz71ia6e3YPa3+zRuFwKK6NO4L2Lz2S6mdkERCqUBYRsT0iPh0Rn6k2QB0RSyKiPSLap02bNqwKtLWOZf7sKW5BWO756u7iGo0WRBazmLqA48q2ZwDvDOUAkhYCC+fMmTPsSjgcrFk4JCwtWbQgngNOlDRb0jjgMmBpBvUwM7Mq0p7mejfwNDBXUpekqyJiD3AtsAx4Hbg3Il4dynG9mquZWfrSnsV0+QDljwDDHl2pRxeTmZlVl8ulNtyCMDNLXy4DwszM0pfLgPAtR83M0pfLgHAXk5lZ+hQRWddh2CRtAH4+zN2nAhvrWJ1m4HNyMJ+Tvnw+DpbHc3J8RAx6pXGuA2IkJHVERHvW9WgkPicH8znpy+fjYM18TnLZxWRmZulzQJiZWUVFDoglWVegAfmcHMznpC+fj4M17Tkp7BiEmZlVV+QWhJmZVVHIgBjgnthNQdJxkn4g6XVJr0r6XFI+RdJjkt5MHo9IyiXpluRcvCzptLJjXZG8/01JV5SVny7plWSfWyRVusdHw5HUIukFSQ8l27MlPZP8ft9KVhdG0vhkuzN5fVbZMb6QlK+U9LGy8lx9pyRNlnSfpDeS78pZRf+OSPp88v/MCkl3S2ot8ncEgIgo1A/QAvwHcAIwDngJmJd1ver4+x0DnJY8bwN+Rune338PLE7KFwM3J8/PB75L6UZOZwLPJOVTgFXJ4xHJ8yOS154Fzkr2+S5wXta/d43n5s+BfwMeSrbvBS5Lnt8OfCZ5/qfA7cnzy4BvJc/nJd+X8cDs5HvUksfvFPAN4I+T5+OAyUX+jlC6q+VbwKFl340ri/wdiYhCtiD23xM7InYBvffEbgoRsTYink+ed1NaUn06pd/xG8nbvgFcnDy/CLgrSn4KTJZ0DPAx4LGI2BQRm4HHgAXJa4dHxNNR+j/irrJjNSxJM4ALgH9OtgV8CLgveUv/c9J7ru4DPpy8/yLgnojYGRFvAZ2Uvk+5+k5JOhz4APB1gIjYFRFbKPh3hNLq1odKGgNMANZS0O9IryIGxED3xG46SbP3VOAZ4OiIWAulEAGOSt420PmoVt5VobzR/SPwF8C+ZPtIYEuU7k8CfX+P/b978vrW5P1DPVeN6gRgA/AvSZfbP0uaSIG/IxHxC+B/AW9TCoatwHKK+x0BihkQFe+JPeq1SJmkw4B/B/4sIt6t9tYKZTGM8oYl6XeB9RGxvLy4wltjkNea5ZyMAU4DvhYRpwLbKXUpDaTZzwfJeMtFlLqFjgUmAudVeGtRviNAMQNixPfEbnSSxlIKh29GxP1J8bqk6U/yuD4pH+h8VCufUaG8kf0mcKGk1ZSa9h+i1KKYnHQnQN/fY//vnrw+CdjE0M9Vo+oCuiLimWT7PkqBUeTvyLnAWxGxISJ2A/cDZ1Pc7whQzIBo6ntiJ/2gXwdej4h/KHtpKdA7y+QK4Dtl5X+YzFQ5E9iadC8sAz4q6YjkX1cfBZYlr3VLOjP5rD8sO1ZDiogvRMSMiJhF6b/39yPiD4AfAJckb+t/TnrP1SXJ+yMpvyyZwTIbOJHSYGyuvlMR8UtgjaS5SdGHgdco8HeEUtfSmZImJHXuPSeF/I7sl/UoeRY/lGZl/IzSrIIbsq5PnX+336LUdH0ZeDH5OZ9S/+jjwJvJ45Tk/QJuS87FK0B72bH+iNIgWyfw6bLydmBFss+tJBdc5uEHOIcDs5hOoPQ/byfwbWB8Ut6abHcmr59Qtv8Nye+9krKZOXn7TgGnAB3J9+QBSrOQCv0dAb4IvJHU+18pzUQq7HckInwltZmZVVbELiYzM6uBA8LMzCpyQJiZWUUOCDMzq8gBYWZmFTkgzCqQ9NlkldNvDnG/WZI+mVa9zEaTA8Kssj8Fzo/SBXVDMQsYckBIahnqPmZpc0CY9SPpdkoXSC2VdIOkOyQ9lyxsd1HynlmSfijp+eTn7GT3m4DflvRicn+BKyXdWnbshySdkzzfJulLkp4BzkruofCkpOWSlvUue2GWFQeEWT8RcQ2ldXI+SGnRtu9HxG8k2/8zWfl0PfCRiDgNuBS4Jdl9MfDDiDglIv73IB81EVgREWdQWnH3K8AlEXE6cAfwt3X+1cyGZMzgbzErtI9SWujv+mS7FZhJKUBulXQKsBc4aRjH3ktpUUWAucCvA4+VlgKihdKy02aZcUCYVSfgExGxsk+hdCOwDjiZUku8Z4D999C3pd5a9rwnIvaWfc6rEXFWPSptVg/uYjKrbhlwXbLCJ5JOTconAWsjYh/wKUr/4gfopnSr116rgVMkHSLpOEp3FqtkJTBN0lnJ54yV9Gt1/U3MhsgBYVbdl4GxwMuSViTbAF8FrpD0U0rdS9uT8peBPZJekvR54MeU7nX8CqU7lj1f6UOidBvKS4CbJb1EaRXesyu912y0eDVXMzOryC0IMzOryAFhZmYVOSDMzKwiB4SZmVXkgDAzs4ocEGZmVpEDwszMKnJAmJlZRf8f8IHDYAWlacMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.semilogy(list(feature_counter.keys()), list(feature_counter.values()),\n",
    "             '.', alpha=0.3, markersize=1)\n",
    "plt.xlabel('feature')\n",
    "plt.ylabel('frequency');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The features, i.e., the words, follow a Zipf-like distribution, which doesn't come as a surprise.  Since this computation is time consuming, we assume similar results for the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the minimum index is 1, the maximum 88586."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 88586)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min(feature_counter.keys()), max(feature_counter.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distirbution of the opinions, 0 or 1, can be visualized in a bar plot, again one for the training, the other for the test output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEXdJREFUeJzt3H+o3fV9x/Hna8nsr62N1tvSJbJkNHSzZaMuqFthjGZotKXxjwqRMUMXCAy7dT9g1e2PQFuhsjE3WesINTOOog2uw7DaumAtZVCt11qsmrpcdDN3unpLousmbZfuvT/uJ+sxOUlu7rnmfE7u8wGH8/2+v5/v937OSd68zvmeL99UFZIk9eYnxj0BSZKGMaAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXVo57gks1vnnn19r164d9zS0zDzyyCPfraqpcc9jqdhHGoeF9tHEBtTatWuZnp4e9zS0zCT5t3HPYSnZRxqHhfaRp/gkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV2a2Fsdncza678w7imM5F8/+b5xT2Ekk/z+T/p7v5Qm+d8RJv/f0vffb1CSpE4ZUJKkLhlQkqQuGVCSpC4ZUJKkLp0yoJLsSvJCkscHan+W5NtJHkvyD0lWDWy7IclMkqeSXD5Q39RqM0muH6ivS/JQkgNJPpfknKV8gVIv7CXp9CzkG9TtwKZjavuAd1XVLwL/AtwAkORCYAvwzrbPp5OsSLIC+BRwBXAhcE0bC3ATcHNVrQcOA9tGekVSv27HXpIW7JQBVVVfBQ4dU/unqjrSVh8E1rTlzcBdVfWDqnoGmAEubo+Zqnq6qn4I3AVsThLgvcDdbf/dwFUjviapS/aSdHqW4jeo3wa+2JZXAwcHts222onqbwZeHGjQo/WhkmxPMp1kem5ubgmmLnXljPSSfaRJMVJAJflT4Ajw2aOlIcNqEfWhqmpnVW2oqg1TU1OnO12pW2eyl+wjTYpF3+ooyVbg/cDGqjraCLPABQPD1gDPteVh9e8Cq5KsbJ/8BsdLy4K9JA23qG9QSTYBHwU+UFUvD2zaC2xJ8pok64D1wNeBh4H17Sqjc5j/8Xdva8YHgA+2/bcC9yzupUiTx16STmwhl5nfCXwNeEeS2STbgL8GfhrYl+SbSf4GoKqeAPYATwJfAq6rqh+1T3QfBu4D9gN72liYb84/TDLD/Hn025b0FUqdsJek03PKU3xVdc2Q8gn/41fVjcCNQ+r3AvcOqT/N/JVJ0lnNXpJOj3eSkCR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR16ZQBlWRXkheSPD5QOy/JviQH2vO5rZ4ktySZSfJYkosG9tnaxh9IsnWg/stJvtX2uSVJlvpFSj2wl6TTs5BvULcDm46pXQ/cX1XrgfvbOsAVwPr22A7cCvNNCOwALgEuBnYcbcQ2ZvvAfsf+LelscTv2krRgpwyoqvoqcOiY8mZgd1veDVw1UL+j5j0IrEryNuByYF9VHaqqw8A+YFPb9saq+lpVFXDHwLGks4q9JJ2exf4G9daqeh6gPb+l1VcDBwfGzbbayeqzQ+pDJdmeZDrJ9Nzc3CKnLnXljPeSfaRJsdQXSQw7512LqA9VVTurakNVbZiamlrkFKWJ8Kr1kn2kSbHYgPpOO6VAe36h1WeBCwbGrQGeO0V9zZC6tFzYS9IJLDag9gJHrx7aCtwzUL+2XYF0KfBSO21xH3BZknPbD7qXAfe1bd9Lcmm74ujagWNJy4G9JJ3AylMNSHIn8OvA+Ulmmb+C6JPAniTbgGeBq9vwe4ErgRngZeBDAFV1KMnHgYfbuI9V1dEfi3+H+aubXgd8sT2ks469JJ2eUwZUVV1zgk0bh4wt4LoTHGcXsGtIfRp416nmIU06e0k6Pd5JQpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktSlkQIqyR8keSLJ40nuTPLaJOuSPJTkQJLPJTmnjX1NW59p29cOHOeGVn8qyeWjvSRp8thL0vEWHVBJVgO/B2yoqncBK4AtwE3AzVW1HjgMbGu7bAMOV9XbgZvbOJJc2PZ7J7AJ+HSSFYudlzRp7CVpuFFP8a0EXpdkJfB64HngvcDdbftu4Kq2vLmt07ZvTJJWv6uqflBVzwAzwMUjzkuaNPaSdIxFB1RV/Tvw58CzzDfTS8AjwItVdaQNmwVWt+XVwMG275E2/s2D9SH7vEKS7Ummk0zPzc0tdupSV850L9lHmhSjnOI7l/lPbOuAnwHeAFwxZGgd3eUE205UP75YtbOqNlTVhqmpqdOftNShM91L9pEmxSin+H4DeKaq5qrqf4DPA78KrGqnKQDWAM+15VngAoC2/U3AocH6kH2k5cBekoYYJaCeBS5N8vp2/nsj8CTwAPDBNmYrcE9b3tvWadu/XFXV6lvalUnrgPXA10eYlzRp7CVpiJWnHjJcVT2U5G7gG8AR4FFgJ/AF4K4kn2i129outwF/l2SG+U97W9pxnkiyh/mGPAJcV1U/Wuy8pEljL0nDLTqgAKpqB7DjmPLTDLlyqKq+D1x9guPcCNw4ylykSWYvScfzThKSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC6NFFBJViW5O8m3k+xP8itJzkuyL8mB9nxuG5sktySZSfJYkosGjrO1jT+QZOuoL0qaNPaSdLxRv0H9FfClqvp54JeA/cD1wP1VtR64v60DXAGsb4/twK0ASc4DdgCXABcDO442orSM2EvSMRYdUEneCPwacBtAVf2wql4ENgO727DdwFVteTNwR817EFiV5G3A5cC+qjpUVYeBfcCmxc5LmjT2kjTcKN+gfg6YA/42yaNJPpPkDcBbq+p5gPb8ljZ+NXBwYP/ZVjtR/ThJtieZTjI9Nzc3wtSlrpzRXrKPNClGCaiVwEXArVX1buC/+fEpiGEypFYnqR9frNpZVRuqasPU1NTpzlfq1RntJftIk2KUgJoFZqvqobZ+N/NN9p12uoH2/MLA+AsG9l8DPHeSurRc2EvSEIsOqKr6D+Bgkne00kbgSWAvcPTqoa3APW15L3BtuwLpUuCldtriPuCyJOe2H3QvazVpWbCXpOFWjrj/7wKfTXIO8DTwIeZDb0+SbcCzwNVt7L3AlcAM8HIbS1UdSvJx4OE27mNVdWjEeUmTxl6SjjFSQFXVN4ENQzZtHDK2gOtOcJxdwK5R5iJNMntJOp53kpAkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1aeSASrIiyaNJ/rGtr0vyUJIDST6X5JxWf01bn2nb1w4c44ZWfyrJ5aPOSZpE9pL0SkvxDeojwP6B9ZuAm6tqPXAY2Nbq24DDVfV24OY2jiQXAluAdwKbgE8nWbEE85Imjb0kDRgpoJKsAd4HfKatB3gvcHcbshu4qi1vbuu07Rvb+M3AXVX1g6p6BpgBLh5lXtKksZek4436DeovgT8G/retvxl4saqOtPVZYHVbXg0cBGjbX2rj/78+ZJ9XSLI9yXSS6bm5uRGnLnXljPWSfaRJseiASvJ+4IWqemSwPGRonWLbyfZ5ZbFqZ1VtqKoNU1NTpzVfqVdnupfsI02KlSPs+x7gA0muBF4LvJH5T4Grkqxsn+zWAM+18bPABcBskpXAm4BDA/WjBveRlgN7SRpi0d+gquqGqlpTVWuZ/2H2y1X1m8ADwAfbsK3APW15b1unbf9yVVWrb2lXJq0D1gNfX+y8pEljL0nDjfIN6kQ+CtyV5BPAo8BtrX4b8HdJZpj/tLcFoKqeSLIHeBI4AlxXVT96FeYlTRp7ScvakgRUVX0F+EpbfpohVw5V1feBq0+w/43AjUsxF2mS2UvSj3knCUlSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlxYdUEkuSPJAkv1JnkjykVY/L8m+JAfa87mtniS3JJlJ8liSiwaOtbWNP5Bk6+gvS5oc9pI03CjfoI4Af1RVvwBcClyX5ELgeuD+qloP3N/WAa4A1rfHduBWmG9CYAdwCXAxsONoI0rLhL0kDbHogKqq56vqG235e8B+YDWwGdjdhu0GrmrLm4E7at6DwKokbwMuB/ZV1aGqOgzsAzYtdl7SpLGXpOGW5DeoJGuBdwMPAW+tqudhvvGAt7Rhq4GDA7vNttqJ6sP+zvYk00mm5+bmlmLqUlfORC/ZR5oUIwdUkp8C/h74/ar6z5MNHVKrk9SPL1btrKoNVbVhamrq9CcrdexM9ZJ9pEkxUkAl+UnmG+qzVfX5Vv5OO91Ae36h1WeBCwZ2XwM8d5K6tGzYS9LxRrmKL8BtwP6q+ouBTXuBo1cPbQXuGahf265AuhR4qZ22uA+4LMm57Qfdy1pNWhbsJWm4lSPs+x7gt4BvJflmq/0J8ElgT5JtwLPA1W3bvcCVwAzwMvAhgKo6lOTjwMNt3Meq6tAI85Imjb0kDbHogKqqf2b4OW+AjUPGF3DdCY61C9i12LlIk8xekobzThKSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLhlQkqQuGVCSpC4ZUJKkLnUTUEk2JXkqyUyS68c9H2lS2Us6W3QRUElWAJ8CrgAuBK5JcuF4ZyVNHntJZ5MuAgq4GJipqqer6ofAXcDmMc9JmkT2ks4aK8c9gWY1cHBgfRa45NhBSbYD29vqfyV56gzMbZjzge++WgfPTa/Wkc8K437vf/bV+ttL5JS9ZB+pGef7v6A+6iWgMqRWxxWqdgI7X/3pnFyS6araMO55LEe+96d0yl6yjwST8f73copvFrhgYH0N8NyY5iJNMntJZ41eAuphYH2SdUnOAbYAe8c8J2kS2Us6a3Rxiq+qjiT5MHAfsALYVVVPjHlaJzP20yPLmO/9SUxYL/lvOV7dv/+pOu6nHkmSxq6XU3ySJL2CASVJ6pIBdRq8hcz4JNmV5IUkj497LhqdvTQ+k9RLBtQCeQuZsbsd2DTuSWh09tLY3c6E9JIBtXDeQmaMquqrwKFxz0NLwl4ao0nqJQNq4YbdQmb1mOYiTTJ7SQtiQC3cgm7HJOmU7CUtiAG1cN5CRloa9pIWxIBaOG8hIy0Ne0kLYkAtUFUdAY7eQmY/sKfjW8icdZLcCXwNeEeS2STbxj0nLY69NF6T1Eve6kiS1CW/QUmSumRASZK6ZEBJkrpkQEmSumRASZK6ZEBJkrpkQEmSuvR/T+gRKfOjT1YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "figure, axes = plt.subplots(1, 2)\n",
    "for i, opinions in enumerate((y_train, y_test)):\n",
    "    counter = [0, 0]\n",
    "    for opinion in opinions:\n",
    "        counter[opinion] += 1\n",
    "    axes[i].bar(['0', '1'], counter, 0.5);\n",
    "figure.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Positive and negetive opinions are uniformly distributed in the training set and the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The texts represented as a list of integers, each integer representing a specific word.  The word index, i.e, a dictionary that has the words as keys, and the integers as values is also available in the IMDB dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index = imdb.get_word_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_index['the']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, in order to translate the lists of integers into the original reviews, the data has to be loaded appropriately.  The `load_data` method has some optional arguments that should be specified.  Index 0 is usually reserved for padding, i.e., to ensure that short sequences can be extended to the required length.  Index 1 indicates the start of a review (`start_char`), while index 2 is used to represent words that have not been indexed, either because they were not part of the data set, they were too infrequently used, or, if the top words are left out, too common to be considered informative (`oov_char`).  Hence, the actual word index starts at 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The word index has to be shifted by `index_from`, and the strings representing padding, start and unknown added.  The following function will do this, compute the reverse dictionary, and return both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_indices(word_index=None, index_from=3, padding_idx=0, start_idx=1, unknown_idx=2):\n",
    "    if word_index is None:\n",
    "        word_index = imdb.get_word_index()\n",
    "    word_to_idx = {k: v + index_from for k, v in word_index.items()}\n",
    "    word_to_idx['<pad>'] = padding_idx\n",
    "    word_to_idx['<start>'] = start_idx\n",
    "    word_to_idx['<unknown>'] = unknown_idx\n",
    "    return word_to_idx, {v: k for k, v in word_to_idx.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_to_idx, idx_to_word = compute_indices(word_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first review in the training set can now be \"translated\" back to English.  Note that there is no punctuation, and, since only the 1,000 most common words were loaded, quite a number of `<unknown>` crop up in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<start> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert redford's is an amazing actor and now the same being director norman's father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for retail and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also congratulations to the two little boy's that played the part's of norman and paul they were just brilliant children are often left out of the praising list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\n"
     ]
    }
   ],
   "source": [
    "print(' '.join(idx_to_word[idx] for idx in x_train[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sentiment expressed in this review is positive, so the output should be 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the first review expressing a negative sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_idx = list(y_train).index(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<start> big hair big boobs bad music and a giant safety pin these are the words to best describe this terrible movie i love cheesy horror movies and i've seen hundreds but this had got to be on of the worst ever made the plot is paper thin and ridiculous the acting is an abomination the script is completely laughable the best is the end showdown with the cop and how he worked out who the killer is it's just so damn terribly written the clothes are sickening and funny in equal measures the hair is big lots of boobs bounce men wear those cut tee shirts that show off their stomachs sickening that men actually wore them and the music is just synthesiser trash that plays over and over again in almost every scene there is trashy music boobs and paramedics taking away bodies and the gym still doesn't close for bereavement all joking aside this is a truly bad film whose only charm is to look back on the disaster that was the 80's and have a good old laugh at how bad everything was back then\n"
     ]
    }
   ],
   "source": [
    "print(' '.join(idx_to_word[idx] for idx in x_train[neg_idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, the reviewer was not taken by the movie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stop words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you display the top-25 words, it is quite clear that most will not be very informative, except \"but\" at index 20 and \"not\" at index 23."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 the\n",
      "5 and\n",
      "6 a\n",
      "7 of\n",
      "8 to\n",
      "9 is\n",
      "10 br\n",
      "11 in\n",
      "12 it\n",
      "13 i\n",
      "14 this\n",
      "15 that\n",
      "16 was\n",
      "17 as\n",
      "18 for\n",
      "19 with\n",
      "20 movie\n",
      "21 but\n",
      "22 film\n",
      "23 on\n",
      "24 not\n",
      "25 you\n",
      "26 are\n",
      "27 his\n",
      "28 have\n",
      "29 he\n"
     ]
    }
   ],
   "source": [
    "for i in range(4, 4 + 26):\n",
    "    print(i, idx_to_word[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would most likely be fine to exclude all most frequent words upto \"but\", and limit the number of words to the 5,000 most frequent ones in order to reduce the dataset size, and hence the computations when training the network."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}