lab-07-6-IMDB-introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 07-6-IMDB-introduction\n",
    "* https://www.tensorflow.org/tutorials/keras/basic_text_classification\n",
    "### 기본 Library 선언 및 Tensorflow 버전 확인"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.12.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "tf.set_random_seed(777)  # for reproducibility\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 강의에 설명할 Data입니다\n",
    "* Tensorflow 2.0 버전에 맞게 Keras를 활용한 IMDB 분류 모델 생성\n",
    "\n",
    "## 학습에 쓰이는 Data\n",
    "* 50,000 movie reviews from the Internet Movie Database. (10000개의 빈도수가 높은 단어를 학습시 Vector에 사용)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training entries: 25000, labels: 25000\n",
      "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]\n"
     ]
    }
   ],
   "source": [
    "imdb = keras.datasets.imdb\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n",
    "print(\"Training entries: {}, labels: {}\".format(len(train_data), len(train_labels)))\n",
    "print(train_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*  IMDB Data를 Vector을 실재 값으로 변환하여 출력"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"<START> worst mistake of my life br br i picked this movie up at target for 5 because i figured hey it's sandler i can get some cheap laughs i was wrong completely wrong mid way through the film all three of my friends were asleep and i was still suffering worst plot worst script worst movie i have ever seen i wanted to hit my head up against a wall for an hour then i'd stop and you know why because it felt damn good upon bashing my head in i stuck that damn movie in the <UNK> and watched it burn and that felt better than anything else i've ever done it took american psycho army of darkness and kill bill just to get over that crap i hate you sandler for actually going through with this and ruining a whole day of my life\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A dictionary mapping words to an integer index\n",
    "word_index = imdb.get_word_index()\n",
    "\n",
    "# The first indices are reserved\n",
    "word_index = {k:(v+3) for k,v in word_index.items()}\n",
    "word_index[\"<PAD>\"] = 0\n",
    "word_index[\"<START>\"] = 1\n",
    "word_index[\"<UNK>\"] = 2  # unknown\n",
    "word_index[\"<UNUSED>\"] = 3\n",
    "\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "\n",
    "def decode_review(text):\n",
    "    return ' '.join([reverse_word_index.get(i, '?') for i in text])\n",
    "\n",
    "decode_review(train_data[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "print(train_labels[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Tensorflow Keras\n",
    "#### 위 Data를 기준으로 분류 모델을 만들도록 하겠습니다\n",
    "* 학습과 평가를 위해 동일길이인 256길이의 단어로 PAD값을 주어 맞춰줌 (뒤의 길이는 0값으로 맞춰줌)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "256 256\n",
      "[   1   14   22   16   43  530  973 1622 1385   65  458 4468   66 3941\n",
      "    4  173   36  256    5   25  100   43  838  112   50  670    2    9\n",
      "   35  480  284    5  150    4  172  112  167    2  336  385   39    4\n",
      "  172 4536 1111   17  546   38   13  447    4  192   50   16    6  147\n",
      " 2025   19   14   22    4 1920 4613  469    4   22   71   87   12   16\n",
      "   43  530   38   76   15   13 1247    4   22   17  515   17   12   16\n",
      "  626   18    2    5   62  386   12    8  316    8  106    5    4 2223\n",
      " 5244   16  480   66 3785   33    4  130   12   16   38  619    5   25\n",
      "  124   51   36  135   48   25 1415   33    6   22   12  215   28   77\n",
      "   52    5   14  407   16   82    2    8    4  107  117 5952   15  256\n",
      "    4    2    7 3766    5  723   36   71   43  530  476   26  400  317\n",
      "   46    7    4    2 1029   13  104   88    4  381   15  297   98   32\n",
      " 2071   56   26  141    6  194 7486   18    4  226   22   21  134  476\n",
      "   26  480    5  144   30 5535   18   51   36   28  224   92   25  104\n",
      "    4  226   65   16   38 1334   88   12   16  283    5   16 4472  113\n",
      "  103   32   15   16 5345   19  178   32    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0]\n"
     ]
    }
   ],
   "source": [
    "train_data = keras.preprocessing.sequence.pad_sequences(train_data,\n",
    "                                                        value=word_index[\"<PAD>\"],\n",
    "                                                        padding='post',\n",
    "                                                        maxlen=256)\n",
    "\n",
    "test_data = keras.preprocessing.sequence.pad_sequences(test_data,\n",
    "                                                       value=word_index[\"<PAD>\"],\n",
    "                                                       padding='post',\n",
    "                                                       maxlen=256)\n",
    "\n",
    "print(len(train_data[0]), len(test_data[0]))\n",
    "print(train_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Tensorflow keras API를 통해 모델에 대한 정의\n",
    "* 입력 Size와 학습시킬 Layer의 크기와 Activation Function 정의"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, None, 16)          160000    \n",
      "_________________________________________________________________\n",
      "global_average_pooling1d (Gl (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 16)                272       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 17        \n",
      "=================================================================\n",
      "Total params: 160,289\n",
      "Trainable params: 160,289\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# input shape is the vocabulary count used for the movie reviews (10,000 words)\n",
    "vocab_size = 10000\n",
    "\n",
    "model = keras.Sequential()\n",
    "model.add(keras.layers.Embedding(vocab_size, 16))\n",
    "model.add(keras.layers.GlobalAveragePooling1D())\n",
    "model.add(keras.layers.Dense(16, activation=tf.nn.relu))\n",
    "model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Adam Optimizer과 Cross Entropy Loss 선언"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 모델을 평가할 Test 데이타에 대한 정의(10000을 기준으로 학습과 평가 수행)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/40\n",
      "15000/15000 [==============================] - 1s 52us/step - loss: 0.6917 - acc: 0.5992 - val_loss: 0.6895 - val_acc: 0.6920\n",
      "Epoch 2/40\n",
      "15000/15000 [==============================] - 1s 33us/step - loss: 0.6854 - acc: 0.7370 - val_loss: 0.6809 - val_acc: 0.7296\n",
      "Epoch 3/40\n",
      "15000/15000 [==============================] - 1s 35us/step - loss: 0.6721 - acc: 0.7669 - val_loss: 0.6632 - val_acc: 0.7540\n",
      "Epoch 4/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.6467 - acc: 0.7661 - val_loss: 0.6347 - val_acc: 0.7770\n",
      "Epoch 5/40\n",
      "15000/15000 [==============================] - 1s 35us/step - loss: 0.6094 - acc: 0.7989 - val_loss: 0.5949 - val_acc: 0.7902\n",
      "Epoch 6/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.5633 - acc: 0.8173 - val_loss: 0.5510 - val_acc: 0.8031\n",
      "Epoch 7/40\n",
      "15000/15000 [==============================] - 1s 33us/step - loss: 0.5136 - acc: 0.8357 - val_loss: 0.5061 - val_acc: 0.8231\n",
      "Epoch 8/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.4653 - acc: 0.8519 - val_loss: 0.4647 - val_acc: 0.8367\n",
      "Epoch 9/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.4224 - acc: 0.8633 - val_loss: 0.4293 - val_acc: 0.8483\n",
      "Epoch 10/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.3848 - acc: 0.8773 - val_loss: 0.4004 - val_acc: 0.8535\n",
      "Epoch 11/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.3538 - acc: 0.8847 - val_loss: 0.3792 - val_acc: 0.8595\n",
      "Epoch 12/40\n",
      "15000/15000 [==============================] - 1s 35us/step - loss: 0.3283 - acc: 0.8907 - val_loss: 0.3584 - val_acc: 0.8668\n",
      "Epoch 13/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.3049 - acc: 0.8985 - val_loss: 0.3440 - val_acc: 0.8699\n",
      "Epoch 14/40\n",
      "15000/15000 [==============================] - 1s 33us/step - loss: 0.2856 - acc: 0.9035 - val_loss: 0.3313 - val_acc: 0.8733\n",
      "Epoch 15/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.2689 - acc: 0.9075 - val_loss: 0.3214 - val_acc: 0.8755\n",
      "Epoch 16/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.2545 - acc: 0.9118 - val_loss: 0.3133 - val_acc: 0.8776\n",
      "Epoch 17/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.2406 - acc: 0.9181 - val_loss: 0.3066 - val_acc: 0.8791\n",
      "Epoch 18/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.2284 - acc: 0.9225 - val_loss: 0.3011 - val_acc: 0.8821\n",
      "Epoch 19/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.2171 - acc: 0.9262 - val_loss: 0.2969 - val_acc: 0.8824\n",
      "Epoch 20/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.2072 - acc: 0.9295 - val_loss: 0.2931 - val_acc: 0.8831\n",
      "Epoch 21/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1974 - acc: 0.9338 - val_loss: 0.2902 - val_acc: 0.8834\n",
      "Epoch 22/40\n",
      "15000/15000 [==============================] - 1s 33us/step - loss: 0.1885 - acc: 0.9377 - val_loss: 0.2884 - val_acc: 0.8841\n",
      "Epoch 23/40\n",
      "15000/15000 [==============================] - 0s 32us/step - loss: 0.1804 - acc: 0.9410 - val_loss: 0.2871 - val_acc: 0.8841\n",
      "Epoch 24/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.1723 - acc: 0.9449 - val_loss: 0.2854 - val_acc: 0.8847\n",
      "Epoch 25/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1652 - acc: 0.9479 - val_loss: 0.2848 - val_acc: 0.8849\n",
      "Epoch 26/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1580 - acc: 0.9502 - val_loss: 0.2848 - val_acc: 0.8853\n",
      "Epoch 27/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.1521 - acc: 0.9529 - val_loss: 0.2856 - val_acc: 0.8842\n",
      "Epoch 28/40\n",
      "15000/15000 [==============================] - 1s 33us/step - loss: 0.1459 - acc: 0.9563 - val_loss: 0.2852 - val_acc: 0.8860\n",
      "Epoch 29/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1400 - acc: 0.9573 - val_loss: 0.2859 - val_acc: 0.8853\n",
      "Epoch 30/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.1351 - acc: 0.9600 - val_loss: 0.2874 - val_acc: 0.8866\n",
      "Epoch 31/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1291 - acc: 0.9624 - val_loss: 0.2888 - val_acc: 0.8861\n",
      "Epoch 32/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1244 - acc: 0.9645 - val_loss: 0.2907 - val_acc: 0.8853\n",
      "Epoch 33/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1191 - acc: 0.9670 - val_loss: 0.2926 - val_acc: 0.8851\n",
      "Epoch 34/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1147 - acc: 0.9685 - val_loss: 0.2955 - val_acc: 0.8846\n",
      "Epoch 35/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.1108 - acc: 0.9697 - val_loss: 0.2969 - val_acc: 0.8848\n",
      "Epoch 36/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.1060 - acc: 0.9718 - val_loss: 0.2998 - val_acc: 0.8846\n",
      "Epoch 37/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.1022 - acc: 0.9730 - val_loss: 0.3027 - val_acc: 0.8836\n",
      "Epoch 38/40\n",
      "15000/15000 [==============================] - 1s 33us/step - loss: 0.0988 - acc: 0.9737 - val_loss: 0.3060 - val_acc: 0.8823\n",
      "Epoch 39/40\n",
      "15000/15000 [==============================] - 1s 34us/step - loss: 0.0946 - acc: 0.9755 - val_loss: 0.3081 - val_acc: 0.8833\n",
      "Epoch 40/40\n",
      "15000/15000 [==============================] - 0s 33us/step - loss: 0.0910 - acc: 0.9775 - val_loss: 0.3116 - val_acc: 0.8825\n"
     ]
    }
   ],
   "source": [
    "x_val = train_data[:10000]\n",
    "partial_x_train = train_data[10000:]\n",
    "\n",
    "y_val = train_labels[:10000]\n",
    "partial_y_train = train_labels[10000:]\n",
    "\n",
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=40,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val),\n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 1s 22us/step\n",
      "[0.33182398523330686, 0.87232]\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(test_data, test_labels)\n",
    "print(results)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}