04_SQuAD_in_Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SQuAD in Keras (Question and Answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's solve a SQuAD problem using Keras and BERT\n",
    "\n",
    "Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.\n",
    "\n",
    "Bscailly when we input the question sentece, we give a corresponding answer\n",
    "Simply put, AI help solving Enligh comprehension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we build the model in Keras in 4 steps\n",
    "1) Understand SQuAD \n",
    "2) Create BERT Input \n",
    "3) Create SQUAD model \n",
    "4) Inference using testset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![squad]('./img/squad.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actaully SQuAD does not asnwer fully, it only gives a very first and last words in that sentence. So, if it knows the first and last words, it naturally makes sense to understand the whole sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Reference: SQuAD 2.0 [Github](https://rajpurkar.github.io/SQuAD-explorer/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from keras import backend as K\n",
    "from keras import Input, Model\n",
    "from keras import optimizers\n",
    "import keras as keras\n",
    "from keras.layers import Layer, Embedding, Dense, Input, LSTM, Bidirectional, Activation, Conv1D, GRU, TimeDistributed, Dropout\n",
    "from keras.models import Model, load_model\n",
    "from keras.preprocessing.text import Tokenizer, text_to_word_sequence\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import warnings\n",
    "import tensorflow as tf\n",
    "import os\n",
    "import re\n",
    "import pickle\n",
    "import codecs\n",
    "from tqdm import tqdm\n",
    "import shutil\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "warnings.filterwarnings(action='ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Install Keras-bert: Keras API of BERT model \n",
    "Install Keras_radam: Revised version of ADAM optimizer \n",
    "# !pip install keras-bert\n",
    "# !pip install keras-radam   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_bert import load_trained_model_from_checkpoint, load_vocabulary\n",
    "from keras_bert import Tokenizer\n",
    "from keras_bert import AdamWarmup, calc_train_steps\n",
    "\n",
    "from keras_radam import RAdam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Downaloding Model and Data to your directory, and building a few inital helper functions are the same as '02_Sentiment_Analysis_using_Bert' file. Please refer to the previousJupyter notebbok for a codeset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Download the SQuAD dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do Wget\n",
    "```\n",
    "!wget https://raw.githubusercontent.com/nate-parrott/squad/master/data/train-v1.1.json\n",
    "!wget https://raw.githubusercontent.com/nate-parrott/squad/master/data/dev-v1.1.json\n",
    "```\n",
    "Or direct download and save it in the data directory \n",
    "[Github Link](https://github.com/nate-parrott/squad/tree/master/data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.ipynb_checkpoints',\n",
       " 'bert',\n",
       " 'dev-v1.1.json',\n",
       " 'glove',\n",
       " 'News_Category_Dataset_v2.json',\n",
       " 'ratings_test.txt',\n",
       " 'ratings_train.txt',\n",
       " 'train-v1.1.json']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir('./data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A function to make SQuAD JSON file to Pandas Dataframe\n",
    "> Reference: https://www.kaggle.com/sanjay11100/squad-stanford-q-a-json-to-pandas-dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def squad_json_to_dataframe_train(input_file_path, record_path = ['data','paragraphs','qas','answers'],\n",
    "                           verbose = 1):\n",
    "    \"\"\"\n",
    "    input_file_path: path to the squad json file.\n",
    "    record_path: path to deepest level in json file default value is\n",
    "    ['data','paragraphs','qas','answers']\n",
    "    verbose: 0 to suppress it default is 1\n",
    "    \"\"\"\n",
    "    if verbose:\n",
    "        print(\"Reading the json file\")    \n",
    "    file = json.loads(open(input_file_path).read())\n",
    "    if verbose:\n",
    "        print(\"processing...\")\n",
    "    # parsing different level's in the json file\n",
    "    js = pd.io.json.json_normalize(file , record_path )\n",
    "    m = pd.io.json.json_normalize(file, record_path[:-1] )\n",
    "    r = pd.io.json.json_normalize(file,record_path[:-2])\n",
    "    \n",
    "    #combining it into single dataframe\n",
    "    idx = np.repeat(r['context'].values, r.qas.str.len())\n",
    "    ndx  = np.repeat(m['id'].values,m['answers'].str.len())\n",
    "    m['context'] = idx\n",
    "    js['q_idx'] = ndx\n",
    "    main = pd.concat([ m[['id','question','context']].set_index('id'),js.set_index('q_idx')],1,sort=False).reset_index()\n",
    "    main['c_id'] = main['context'].factorize()[0]\n",
    "    if verbose:\n",
    "        print(\"shape of the dataframe is {}\".format(main.shape))\n",
    "        print(\"Done\")\n",
    "    return main"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading the json file\n",
      "processing...\n",
      "shape of the dataframe is (87599, 6)\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "# Load train data\n",
    "train = squad_json_to_dataframe_train(\"./data/train-v1.1.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the training data is loaded suceessfully now\n",
    "The column 'Question' is the question part and 'context' is input to the model.\n",
    "\n",
    "the output (answer) is the first and last words in 'text' column. For example, if the value in 'text' column is 'Saint Bernadette Soubirous', the answer is 'Saint' and 'Soubrious' /\n",
    "\n",
    "And one characteristics of SQuAD problem is that, the answer of the 'text' is actually in 'context'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>question</th>\n",
       "      <th>context</th>\n",
       "      <th>answer_start</th>\n",
       "      <th>text</th>\n",
       "      <th>c_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5733be284776f41900661182</td>\n",
       "      <td>To whom did the Virgin Mary allegedly appear i...</td>\n",
       "      <td>Architecturally, the school has a Catholic cha...</td>\n",
       "      <td>515</td>\n",
       "      <td>Saint Bernadette Soubirous</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5733be284776f4190066117f</td>\n",
       "      <td>What is in front of the Notre Dame Main Building?</td>\n",
       "      <td>Architecturally, the school has a Catholic cha...</td>\n",
       "      <td>188</td>\n",
       "      <td>a copper statue of Christ</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5733be284776f41900661180</td>\n",
       "      <td>The Basilica of the Sacred heart at Notre Dame...</td>\n",
       "      <td>Architecturally, the school has a Catholic cha...</td>\n",
       "      <td>279</td>\n",
       "      <td>the Main Building</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5733be284776f41900661181</td>\n",
       "      <td>What is the Grotto at Notre Dame?</td>\n",
       "      <td>Architecturally, the school has a Catholic cha...</td>\n",
       "      <td>381</td>\n",
       "      <td>a Marian place of prayer and reflection</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5733be284776f4190066117e</td>\n",
       "      <td>What sits on top of the Main Building at Notre...</td>\n",
       "      <td>Architecturally, the school has a Catholic cha...</td>\n",
       "      <td>92</td>\n",
       "      <td>a golden statue of the Virgin Mary</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87594</th>\n",
       "      <td>5735d259012e2f140011a09d</td>\n",
       "      <td>In what US state did Kathmandu first establish...</td>\n",
       "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
       "      <td>229</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>18890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87595</th>\n",
       "      <td>5735d259012e2f140011a09e</td>\n",
       "      <td>What was Yangon previously known as?</td>\n",
       "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
       "      <td>414</td>\n",
       "      <td>Rangoon</td>\n",
       "      <td>18890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87596</th>\n",
       "      <td>5735d259012e2f140011a09f</td>\n",
       "      <td>With what Belorussian city does Kathmandu have...</td>\n",
       "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
       "      <td>476</td>\n",
       "      <td>Minsk</td>\n",
       "      <td>18890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87597</th>\n",
       "      <td>5735d259012e2f140011a0a0</td>\n",
       "      <td>In what year did Kathmandu create its initial ...</td>\n",
       "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
       "      <td>199</td>\n",
       "      <td>1975</td>\n",
       "      <td>18890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87598</th>\n",
       "      <td>5735d259012e2f140011a0a1</td>\n",
       "      <td>What is KMC an initialism of?</td>\n",
       "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
       "      <td>0</td>\n",
       "      <td>Kathmandu Metropolitan City</td>\n",
       "      <td>18890</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>87599 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          index  \\\n",
       "0      5733be284776f41900661182   \n",
       "1      5733be284776f4190066117f   \n",
       "2      5733be284776f41900661180   \n",
       "3      5733be284776f41900661181   \n",
       "4      5733be284776f4190066117e   \n",
       "...                         ...   \n",
       "87594  5735d259012e2f140011a09d   \n",
       "87595  5735d259012e2f140011a09e   \n",
       "87596  5735d259012e2f140011a09f   \n",
       "87597  5735d259012e2f140011a0a0   \n",
       "87598  5735d259012e2f140011a0a1   \n",
       "\n",
       "                                                question  \\\n",
       "0      To whom did the Virgin Mary allegedly appear i...   \n",
       "1      What is in front of the Notre Dame Main Building?   \n",
       "2      The Basilica of the Sacred heart at Notre Dame...   \n",
       "3                      What is the Grotto at Notre Dame?   \n",
       "4      What sits on top of the Main Building at Notre...   \n",
       "...                                                  ...   \n",
       "87594  In what US state did Kathmandu first establish...   \n",
       "87595               What was Yangon previously known as?   \n",
       "87596  With what Belorussian city does Kathmandu have...   \n",
       "87597  In what year did Kathmandu create its initial ...   \n",
       "87598                      What is KMC an initialism of?   \n",
       "\n",
       "                                                 context  answer_start  \\\n",
       "0      Architecturally, the school has a Catholic cha...           515   \n",
       "1      Architecturally, the school has a Catholic cha...           188   \n",
       "2      Architecturally, the school has a Catholic cha...           279   \n",
       "3      Architecturally, the school has a Catholic cha...           381   \n",
       "4      Architecturally, the school has a Catholic cha...            92   \n",
       "...                                                  ...           ...   \n",
       "87594  Kathmandu Metropolitan City (KMC), in order to...           229   \n",
       "87595  Kathmandu Metropolitan City (KMC), in order to...           414   \n",
       "87596  Kathmandu Metropolitan City (KMC), in order to...           476   \n",
       "87597  Kathmandu Metropolitan City (KMC), in order to...           199   \n",
       "87598  Kathmandu Metropolitan City (KMC), in order to...             0   \n",
       "\n",
       "                                          text   c_id  \n",
       "0                   Saint Bernadette Soubirous      0  \n",
       "1                    a copper statue of Christ      0  \n",
       "2                            the Main Building      0  \n",
       "3      a Marian place of prayer and reflection      0  \n",
       "4           a golden statue of the Virgin Mary      0  \n",
       "...                                        ...    ...  \n",
       "87594                                   Oregon  18890  \n",
       "87595                                  Rangoon  18890  \n",
       "87596                                    Minsk  18890  \n",
       "87597                                     1975  18890  \n",
       "87598              Kathmandu Metropolitan City  18890  \n",
       "\n",
       "[87599 rows x 6 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Max-length of a sentence. If the sentence is shorter than 384, remaining is filled with 0. \n",
    "# I chose 384 as random (due to memory)\n",
    "SEQ_LEN = 384\n",
    "BATCH_SIZE = 10\n",
    "EPOCHS=2\n",
    "LR=3e-5\n",
    "\n",
    "# folder that has the pretrained BERT model\n",
    "pretrained_path = os.path.abspath('./data/bert')\n",
    "\n",
    "config_path = os.path.join(pretrained_path, 'bert_config.json')\n",
    "checkpoint_path = os.path.join(pretrained_path, 'bert_model.ckpt')\n",
    "vocab_path = os.path.join(pretrained_path, 'vocab.txt')\n",
    "\n",
    "# Specify lables \n",
    "DATA_COLUMN = \"context\"\n",
    "QUESTION_COLUMN = \"question\"\n",
    "TEXT = \"text\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Same step as Sentiment Analysis Notebook\n",
    "Create a dictionary called 'token_dict' that adds numbering to words in vocab.txt \n",
    "So the flow of NLP is\n",
    "**Tokonize the sentence into words ==> Words converted to Index (numbers) ==> Fed into the BERT model**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Tokenize the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_dict = {}\n",
    "with codecs.open(vocab_path, 'r', 'utf8') as reader:\n",
    "    for line in reader:\n",
    "        token = line.strip()\n",
    "        if \"_\" in token:\n",
    "            token = token.replace(\"_\",\"\")\n",
    "            token = \"##\" + token\n",
    "        token_dict[token] = len(token_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(token_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras_bert.tokenizer.Tokenizer at 0x207d7e5f448>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check if tokenization is done well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'keras', 'is', 'real', '##l', 'fun', '.', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.tokenize(\"keras is reall fun.\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = train['question'][0]\n",
    "context = train['context'][0]\n",
    "text = train['text'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at sample question, context and answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Saint Bernadette Soubirous'"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# answer\n",
    "text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In raw format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to get 'question' and 'context' and creata a model to get 'text'. /\n",
    "Tokenized answer is ['[CLS]', 'saint', 'bern', '##ade', '##tte', 'sou', '##bir', '##ous', '[SEP]'], but we are trying to fine-tuning the model to predict the location of 'saint' and '##ous'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'to', 'whom', 'did', 'the', 'vir', '##gin', 'mar', '##y', 'allegedly', 'appear', 'in', '1858', 'in', 'lo', '##urde', '##s', 'franc', '##e', '?', '[SEP]', 'architectural', '##ly', ',', 'the', 'school', 'has', 'a', 'cat', '##hol', '##ic', 'character', '.', 'ato', '##p', 'the', 'main', 'building', \"'\", 's', 'gold', 'dome', 'is', 'a', 'golden', 'statue', 'of', 'the', 'vir', '##gin', 'mar', '##y', '.', 'immediately', 'in', 'front', 'of', 'the', 'main', 'building', 'and', 'facing', 'it', ',', 'is', 'a', 'copper', 'statue', 'of', 'ch', '##rist', 'with', 'arms', 'up', '##rais', '##ed', 'with', 'the', 'legend', '\"', 'ven', '##ite', 'ad', 'me', 'om', '##nes', '\"', '.', 'next', 'to', 'the', 'main', 'building', 'is', 'the', 'basilica', 'of', 'the', 'sacred', 'heart', '.', 'immediately', 'behind', 'the', 'basilica', 'is', 'the', 'gr', '##otto', ',', 'a', 'mari', '##an', 'place', 'of', 'prayer', 'and', 'reflect', '##ion', '.', 'it', 'is', 'a', 'replica', 'of', 'the', 'gr', '##otto', 'at', 'lo', '##urde', '##s', ',', 'franc', '##e', 'where', 'the', 'vir', '##gin', 'mar', '##y', 'rep', '##uted', '##ly', 'appeared', 'to', 'saint', 'bern', '##ade', '##tte', 'sou', '##bir', '##ous', 'in', '1858', '.', 'at', 'the', 'end', 'of', 'the', 'main', 'drive', '(', 'and', 'in', 'a', 'direct', 'line', 'that', 'connects', 'through', '3', 'statues', 'and', 'the', 'gold', 'dome', ')', ',', 'is', 'a', 'simple', ',', 'modern', 'stone', 'statue', 'of', 'mar', '##y', '.', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.tokenize(question, context))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'saint', 'bern', '##ade', '##tte', 'sou', '##bir', '##ous', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.tokenize(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This converts Words into Index (numbers)\n",
    "def convert_data(data_df):\n",
    "    global tokenizer\n",
    "    indices, segments, target_start, target_end = [], [], [], []\n",
    "    for i in tqdm(range(len(data_df))):\n",
    "        \n",
    "        ids, segment = tokenizer.encode(data_df[QUESTION_COLUMN][i], data_df[DATA_COLUMN][i], max_len=SEQ_LEN)\n",
    "        \n",
    "\n",
    "        text = tokenizer.encode(data_df[TEXT][i])[0]\n",
    "\n",
    "        text_slide_len = len(text[1:-1])\n",
    "        for i in range(1,len(ids)-text_slide_len-1):  \n",
    "            exist_flag = 0\n",
    "            if text[1:-1] == ids[i:i+text_slide_len]:\n",
    "                ans_start = i\n",
    "                ans_end = i + text_slide_len - 1\n",
    "                exist_flag = 1\n",
    "                break\n",
    "        \n",
    "        if exist_flag == 0:\n",
    "            ans_start = SEQ_LEN\n",
    "            ans_end = SEQ_LEN\n",
    "\n",
    "        indices.append(ids)\n",
    "        segments.append(segment)\n",
    "\n",
    "        target_start.append(ans_start)\n",
    "        target_end.append(ans_end)\n",
    "\n",
    "    indices_x = np.array(indices)\n",
    "    segments = np.array(segments)\n",
    "    target_start = np.array(target_start)\n",
    "    target_end = np.array(target_end)\n",
    "    \n",
    "    del_list = np.where(target_start!=SEQ_LEN)[0]\n",
    "\n",
    "    indices_x = indices_x[del_list]\n",
    "    segments = segments[del_list]\n",
    "    target_start = target_start[del_list]\n",
    "    target_end = target_end[del_list]\n",
    "\n",
    "    train_y_0 = keras.utils.to_categorical(target_start, num_classes=SEQ_LEN, dtype='int64')\n",
    "    train_y_1 = keras.utils.to_categorical(target_end, num_classes=SEQ_LEN, dtype='int64')\n",
    "    train_y_cat = [train_y_0, train_y_1]\n",
    "    \n",
    "    return [indices_x, segments], train_y_cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load dataframe and split it into train/test\n",
    "\n",
    "def load_data(df):\n",
    "    data_df = df\n",
    "    \n",
    "    data_df[DATA_COLUMN] = data_df[DATA_COLUMN].astype(str)\n",
    "    data_df[QUESTION_COLUMN] = data_df[QUESTION_COLUMN].astype(str)\n",
    "\n",
    "\n",
    "    data_x, data_y = convert_data(data_df)\n",
    "\n",
    "    return data_x, data_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████| 87599/87599 [03:21<00:00, 434.57it/s]\n"
     ]
    }
   ],
   "source": [
    "train_x, train_y = load_data(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[  101, 10114, 18104, ...,     0,     0,     0],\n",
       "        [  101, 12976, 10124, ...,     0,     0,     0],\n",
       "        [  101, 10105, 78253, ...,     0,     0,     0],\n",
       "        ...,\n",
       "        [  101, 10169, 12976, ...,     0,     0,     0],\n",
       "        [  101, 10106, 12976, ...,     0,     0,     0],\n",
       "        [  101, 12976, 10124, ...,     0,     0,     0]]),\n",
       " array([[0, 0, 0, ..., 0, 0, 0],\n",
       "        [0, 0, 0, ..., 0, 0, 0],\n",
       "        [0, 0, 0, ..., 0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0, ..., 0, 0, 0],\n",
       "        [0, 0, 0, ..., 0, 0, 0],\n",
       "        [0, 0, 0, ..., 0, 0, 0]])]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Pre-trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_6\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Input-Token (InputLayer)        (None, 384)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Input-Segment (InputLayer)      (None, 384)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Token (TokenEmbedding [(None, 384, 768), ( 91812096    Input-Token[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Segment (Embedding)   (None, 384, 768)     1536        Input-Segment[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Token-Segment (Add)   (None, 384, 768)     0           Embedding-Token[0][0]            \n",
      "                                                                 Embedding-Segment[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Position (PositionEmb (None, 384, 768)     294912      Embedding-Token-Segment[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Dropout (Dropout)     (None, 384, 768)     0           Embedding-Position[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Norm (LayerNormalizat (None, 384, 768)     1536        Embedding-Dropout[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     2362368     Embedding-Norm[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Embedding-Norm[0][0]             \n",
      "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-1-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-1-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-1-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-2-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-2-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-2-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-3-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-3-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-3-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-4-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-4-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-4-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-5-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-5-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-5-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-6-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-6-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-6-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-7-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-7-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-7-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-8-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-8-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-8-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-9-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-9-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-9-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-9-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Dropout  (None, 384, 768)     0           Encoder-10-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Add (Add (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
      "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-10-FeedForward-Add[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-10-FeedForward-Norm[0][0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-FeedForward-Norm[0][0]\n",
      "                                                                 Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Dropout  (None, 384, 768)     0           Encoder-11-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Add (Add (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
      "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-11-FeedForward-Add[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-11-FeedForward-Norm[0][0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-FeedForward-Norm[0][0]\n",
      "                                                                 Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Dropout  (None, 384, 768)     0           Encoder-12-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Add (Add (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
      "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-12-FeedForward-Add[0][0] \n",
      "==================================================================================================\n",
      "Total params: 177,164,544\n",
      "Trainable params: 177,164,544\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "layer_num = 12\n",
    "model = load_trained_model_from_checkpoint(\n",
    "    config_path,\n",
    "    checkpoint_path,\n",
    "    training=False,\n",
    "    trainable=True,\n",
    "    seq_len=SEQ_LEN)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a couple custom layer for transfer learning to be implemented\n",
    "We need to specify NonMaking function to unleash the tensor that was in BERT's default masking.\n",
    "This is a necessary step to train BERT model in SQuAD problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NonMasking(Layer):   \n",
    "    def __init__(self, **kwargs):   \n",
    "        self.supports_masking = True  \n",
    "        super(NonMasking, self).__init__(**kwargs)   \n",
    "  \n",
    "    def build(self, input_shape):   \n",
    "        input_shape = input_shape   \n",
    "  \n",
    "    def compute_mask(self, input, input_mask=None):   \n",
    "        return None   \n",
    "  \n",
    "    def call(self, x, mask=None):   \n",
    "        return x   \n",
    "  \n",
    "    def get_output_shape_for(self, input_shape):   \n",
    "        return input_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create two Keras Custom Layer \n",
    "'MyLayer_Start' predicts the first word of the answer \n",
    "'MyLaer_End' predicts the last word of the answer \n",
    "\n",
    "The fundamental of two layers are basically the same.\n",
    "Get the last layer of the BERT as input, we make (batch_size, 384, 768) tensor to (batch_size, 384, 2)\n",
    "Then we divide tensor into two, to be able to have two different output: (batch_size, 384) and (batch_size, 384)\n",
    "\n",
    "It's 350, because it predicts the location of 384 words "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyLayer_Start(Layer):\n",
    "\n",
    "    def __init__(self,seq_len, **kwargs):\n",
    "        \n",
    "        self.seq_len = seq_len\n",
    "        self.supports_masking = True\n",
    "        super(MyLayer_Start, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        \n",
    "        self.W = self.add_weight(name='kernel', \n",
    "                                 shape=(768,2),\n",
    "                                 initializer='uniform',\n",
    "                                 trainable=True)\n",
    "        super(MyLayer_Start, self).build(input_shape)\n",
    "\n",
    "    def call(self, x):\n",
    "        \n",
    "        x = K.reshape(x, shape=(-1,384,768))\n",
    "        x = K.dot(x, self.W)\n",
    "        \n",
    "        x = K.permute_dimensions(x, (2,0,1))\n",
    "\n",
    "        self.start_logits, self.end_logits = x[0], x[1]\n",
    "        \n",
    "        self.start_logits = K.softmax(self.start_logits, axis=-1)\n",
    "        \n",
    "        return self.start_logits\n",
    "\n",
    "    def compute_output_shape(self, input_shape):\n",
    "        return (input_shape[0], self.seq_len)\n",
    "\n",
    "\n",
    "class MyLayer_End(Layer):\n",
    "    def __init__(self,seq_len, **kwargs):\n",
    "\n",
    "            self.seq_len = seq_len\n",
    "            self.supports_masking = True\n",
    "            super(MyLayer_End, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "\n",
    "            self.W = self.add_weight(name='kernel', \n",
    "                                     shape=(768, 2),\n",
    "                                     initializer='uniform',\n",
    "                                     trainable=True)\n",
    "            super(MyLayer_End, self).build(input_shape)\n",
    "\n",
    "\n",
    "    def call(self, x):\n",
    "\n",
    "\n",
    "            x = K.reshape(x, shape=(-1,384,768))\n",
    "            x = K.dot(x, self.W)\n",
    "            x = K.permute_dimensions(x, (2,0,1))\n",
    "\n",
    "            self.start_logits, self.end_logits = x[0], x[1]\n",
    "\n",
    "            self.end_logits = K.softmax(self.end_logits, axis=-1)\n",
    "\n",
    "            return self.end_logits\n",
    "\n",
    "    def compute_output_shape(self, input_shape):\n",
    "            return (input_shape[0], self.seq_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Create a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a model that ouputs the BERT output\n",
    "it will predict 'start_answer' and 'end_answer'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import merge, dot, concatenate\n",
    "from keras import metrics\n",
    "\n",
    "def get_bert_finetuning_model(model):\n",
    "    \n",
    "    inputs = model.inputs[:2]\n",
    "    dense = model.output\n",
    "    x = NonMasking()(dense)\n",
    "    outputs_start = MyLayer_Start(384)(x)\n",
    "    outputs_end = MyLayer_End(384)(x)\n",
    "    \n",
    "    bert_model = keras.models.Model(inputs, [outputs_start, outputs_end])\n",
    "    \n",
    "    bert_model.compile(\n",
    "          optimizer=RAdam(learning_rate=LR, decay=0.001),\n",
    "          loss='categorical_crossentropy',\n",
    "          metrics=['accuracy'])\n",
    "\n",
    "    return bert_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5.Start Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_7\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Input-Token (InputLayer)        (None, 384)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Input-Segment (InputLayer)      (None, 384)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Token (TokenEmbedding [(None, 384, 768), ( 91812096    Input-Token[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Segment (Embedding)   (None, 384, 768)     1536        Input-Segment[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Token-Segment (Add)   (None, 384, 768)     0           Embedding-Token[0][0]            \n",
      "                                                                 Embedding-Segment[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Position (PositionEmb (None, 384, 768)     294912      Embedding-Token-Segment[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Dropout (Dropout)     (None, 384, 768)     0           Embedding-Position[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "Embedding-Norm (LayerNormalizat (None, 384, 768)     1536        Embedding-Dropout[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     2362368     Embedding-Norm[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Embedding-Norm[0][0]             \n",
      "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-1-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-1-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-1-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-1-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-1-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-2-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-2-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-2-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-2-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-2-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-3-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-3-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-3-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-3-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-3-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-4-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-4-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-4-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-4-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-4-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-5-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-5-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-5-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-5-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-5-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-6-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-6-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-6-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-6-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-6-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-7-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-7-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-7-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-7-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-7-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-8-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-8-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-8-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-8-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-8-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention[\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-9-MultiHeadSelfAttention-\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-9-FeedForward[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention-\n",
      "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-9-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-9-FeedForward-Add[0][0]  \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-9-FeedForward-Norm[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-9-FeedForward-Norm[0][0] \n",
      "                                                                 Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-10-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Dropout  (None, 384, 768)     0           Encoder-10-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Add (Add (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
      "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-10-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-10-FeedForward-Add[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-10-FeedForward-Norm[0][0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-FeedForward-Norm[0][0]\n",
      "                                                                 Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-11-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Dropout  (None, 384, 768)     0           Encoder-11-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Add (Add (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
      "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-11-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-11-FeedForward-Add[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-11-FeedForward-Norm[0][0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-FeedForward-Norm[0][0]\n",
      "                                                                 Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-12-MultiHeadSelfAttention\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Dropout  (None, 384, 768)     0           Encoder-12-FeedForward[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Add (Add (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
      "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
      "__________________________________________________________________________________________________\n",
      "Encoder-12-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-12-FeedForward-Add[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "non_masking_3 (NonMasking)      (None, 384, 768)     0           Encoder-12-FeedForward-Norm[0][0]\n",
      "__________________________________________________________________________________________________\n",
      "my_layer__start_3 (MyLayer_Star (None, 384)          1536        non_masking_3[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "my_layer__end_3 (MyLayer_End)   (None, 384)          1536        non_masking_3[0][0]              \n",
      "==================================================================================================\n",
      "Total params: 177,167,616\n",
      "Trainable params: 177,167,616\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "bert_model = get_bert_finetuning_model(model)\n",
    "bert_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 78555 samples, validate on 8729 samples\n",
      "Epoch 1/1\n",
      "   50/78555 [..............................] - ETA: 66:56:04 - loss: 11.8252 - my_layer__start_3_loss: 6.0306 - my_layer__end_3_loss: 5.7946 - my_layer__start_3_accuracy: 0.0000e+00 - my_layer__end_3_accuracy: 0.0000e+00"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "KeyboardInterrupt\n",
      "\n"
     ]
    }
   ],
   "source": [
    "history = bert_model.fit(train_x, \n",
    "                         train_y, \n",
    "                         batch_size=10, \n",
    "                         validation_split=0.1, # we can do validation_data=(test_x, test_y) instead\n",
    "                         shuffle=False, \n",
    "                         verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Save the best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = os.path.abspath('./data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_model.save_weights(path+\"/squad_wordpiece.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Retrain the model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(now we empty the 'validation_split' string, so that it trains on a whole dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_model.compile(optimizer=RAdam(learning_rate=0.00003, decay=0.0001), loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "bert_model.fit(train_x, train_y, batch_size=10, shuffle=False, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2nd retraining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_model.save_weights(path+\"/squad_wordpiece_3.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_model = get_bert_finetuning_model(model)\n",
    "bert_model.load_weights(path+\"/squad_wordpiece_3.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test data set에 대한 bert_input을 만들어 줍니다.\n",
    "Train data set과는 다르게 label을 생성하지 않습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_pred_data(question, doc):\n",
    "    global tokenizer\n",
    "    indices, segments = [], []\n",
    "    ids, segment = tokenizer.encode(question, doc, max_len=SEQ_LEN)\n",
    "    indices.append(ids)\n",
    "    segments.append(segment)\n",
    "    indices_x = np.array(indices)\n",
    "    segments = np.array(segments)\n",
    "    return [indices_x, segments]\n",
    "\n",
    "def load_pred_data(question, doc):\n",
    "    data_x = convert_pred_data(question, doc)\n",
    "    return data_x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a helper function that returns an answer from the question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_letter(question, doc):\n",
    "  \n",
    "    test_input = load_pred_data(question, doc)\n",
    "    test_start, test_end = bert_model.predict(test_input)\n",
    "\n",
    "    indexes = tokenizer.encode(question, doc, max_len=SEQ_LEN)[0]\n",
    "    start = np.argmax(test_start, axis=1).item()\n",
    "    end = np.argmax(test_end, axis=1).item()\n",
    "    start_tok = indexes[start]\n",
    "    end_tok = indexes[end]\n",
    "    print(\"Question : \", question)\n",
    "\n",
    "    print(\"-\"*50)\n",
    "    print(\"Context : \", end = \" \")\n",
    "\n",
    "    def split_text(text, n):\n",
    "        for line in text.splitlines():\n",
    "            while len(line) > n:\n",
    "                x, line = line[:n], line[n:]\n",
    "                yield x\n",
    "            yield line\n",
    "\n",
    "\n",
    "\n",
    "    for line in split_text(doc, 150):\n",
    "        print(line)\n",
    "\n",
    "    print(\"-\"*50)\n",
    "    print(\"ANSWER : \", end = \" \")\n",
    "    print(\"\\n\")\n",
    "    sentences = []\n",
    "\n",
    "    for i in range(start, end+1):\n",
    "        token_based_word = reverse_token_dict[indexes[i]]\n",
    "        sentences.append(token_based_word)\n",
    "        print(token_based_word, end= \" \")\n",
    "\n",
    "    print(\"\\n\")\n",
    "    print(\"Untokenized Answer : \", end = \"\")\n",
    "    for w in sentences:\n",
    "        if w.startswith(\"##\"):\n",
    "            w = w.replace(\"##\", \"\")\n",
    "        else:\n",
    "            w = \" \" + w\n",
    "\n",
    "    print(w, end=\"\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Inference on Test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "def squad_json_to_dataframe_dev(input_file_path, record_path = ['data','paragraphs','qas','answers'],\n",
    "                           verbose = 1):\n",
    "    \"\"\"\n",
    "    input_file_path: path to the squad json file.\n",
    "    record_path: path to deepest level in json file default value is\n",
    "    ['data','paragraphs','qas','answers']\n",
    "    verbose: 0 to suppress it default is 1\n",
    "    \"\"\"\n",
    "    if verbose:\n",
    "        print(\"Reading the json file\")    \n",
    "    file = json.loads(open(input_file_path).read())\n",
    "    if verbose:\n",
    "        print(\"processing...\")\n",
    "    # parsing different level's in the json file\n",
    "    js = pd.io.json.json_normalize(file , record_path )\n",
    "    m = pd.io.json.json_normalize(file, record_path[:-1] )\n",
    "    r = pd.io.json.json_normalize(file,record_path[:-2])\n",
    "    \n",
    "    #combining it into single dataframe\n",
    "    idx = np.repeat(r['context'].values, r.qas.str.len())\n",
    "    m['context'] = idx\n",
    "    main = m[['id','question','context','answers']].set_index('id').reset_index()\n",
    "    main['c_id'] = main['context'].factorize()[0]\n",
    "    if verbose:\n",
    "        print(\"shape of the dataframe is {}\".format(main.shape))\n",
    "        print(\"Done\")\n",
    "    return main"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading the json file\n",
      "processing...\n",
      "shape of the dataframe is (10570, 5)\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "input_file_path ='./data/dev-v1.1.json'\n",
    "record_path = ['data','paragraphs','qas','answers']\n",
    "verbose = 0\n",
    "dev = squad_json_to_dataframe_dev(input_file_path=input_file_path,record_path=record_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>question</th>\n",
       "      <th>context</th>\n",
       "      <th>answers</th>\n",
       "      <th>c_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56be4db0acb8001400a502ec</td>\n",
       "      <td>Which NFL team represented the AFC at Super Bo...</td>\n",
       "      <td>Super Bowl 50 was an American football game to...</td>\n",
       "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>56be4db0acb8001400a502ed</td>\n",
       "      <td>Which NFL team represented the NFC at Super Bo...</td>\n",
       "      <td>Super Bowl 50 was an American football game to...</td>\n",
       "      <td>[{'answer_start': 249, 'text': 'Carolina Panth...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>56be4db0acb8001400a502ee</td>\n",
       "      <td>Where did Super Bowl 50 take place?</td>\n",
       "      <td>Super Bowl 50 was an American football game to...</td>\n",
       "      <td>[{'answer_start': 403, 'text': 'Santa Clara, C...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56be4db0acb8001400a502ef</td>\n",
       "      <td>Which NFL team won Super Bowl 50?</td>\n",
       "      <td>Super Bowl 50 was an American football game to...</td>\n",
       "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>56be4db0acb8001400a502f0</td>\n",
       "      <td>What color was used to emphasize the 50th anni...</td>\n",
       "      <td>Super Bowl 50 was an American football game to...</td>\n",
       "      <td>[{'answer_start': 488, 'text': 'gold'}, {'answ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10565</th>\n",
       "      <td>5737aafd1c456719005744fb</td>\n",
       "      <td>What is the metric term less used than the New...</td>\n",
       "      <td>The pound-force has a metric counterpart, less...</td>\n",
       "      <td>[{'answer_start': 82, 'text': 'kilogram-force'...</td>\n",
       "      <td>2066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10566</th>\n",
       "      <td>5737aafd1c456719005744fc</td>\n",
       "      <td>What is the kilogram-force sometimes reffered ...</td>\n",
       "      <td>The pound-force has a metric counterpart, less...</td>\n",
       "      <td>[{'answer_start': 114, 'text': 'kilopond'}, {'...</td>\n",
       "      <td>2066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10567</th>\n",
       "      <td>5737aafd1c456719005744fd</td>\n",
       "      <td>What is a very seldom used unit of mass in the...</td>\n",
       "      <td>The pound-force has a metric counterpart, less...</td>\n",
       "      <td>[{'answer_start': 274, 'text': 'slug'}, {'answ...</td>\n",
       "      <td>2066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10568</th>\n",
       "      <td>5737aafd1c456719005744fe</td>\n",
       "      <td>What seldom used term of a unit of force equal...</td>\n",
       "      <td>The pound-force has a metric counterpart, less...</td>\n",
       "      <td>[{'answer_start': 712, 'text': 'kip'}, {'answe...</td>\n",
       "      <td>2066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10569</th>\n",
       "      <td>5737aafd1c456719005744ff</td>\n",
       "      <td>What is the seldom used force unit equal to on...</td>\n",
       "      <td>The pound-force has a metric counterpart, less...</td>\n",
       "      <td>[{'answer_start': 665, 'text': 'sthène'}, {'an...</td>\n",
       "      <td>2066</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10570 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                             id  \\\n",
       "0      56be4db0acb8001400a502ec   \n",
       "1      56be4db0acb8001400a502ed   \n",
       "2      56be4db0acb8001400a502ee   \n",
       "3      56be4db0acb8001400a502ef   \n",
       "4      56be4db0acb8001400a502f0   \n",
       "...                         ...   \n",
       "10565  5737aafd1c456719005744fb   \n",
       "10566  5737aafd1c456719005744fc   \n",
       "10567  5737aafd1c456719005744fd   \n",
       "10568  5737aafd1c456719005744fe   \n",
       "10569  5737aafd1c456719005744ff   \n",
       "\n",
       "                                                question  \\\n",
       "0      Which NFL team represented the AFC at Super Bo...   \n",
       "1      Which NFL team represented the NFC at Super Bo...   \n",
       "2                    Where did Super Bowl 50 take place?   \n",
       "3                      Which NFL team won Super Bowl 50?   \n",
       "4      What color was used to emphasize the 50th anni...   \n",
       "...                                                  ...   \n",
       "10565  What is the metric term less used than the New...   \n",
       "10566  What is the kilogram-force sometimes reffered ...   \n",
       "10567  What is a very seldom used unit of mass in the...   \n",
       "10568  What seldom used term of a unit of force equal...   \n",
       "10569  What is the seldom used force unit equal to on...   \n",
       "\n",
       "                                                 context  \\\n",
       "0      Super Bowl 50 was an American football game to...   \n",
       "1      Super Bowl 50 was an American football game to...   \n",
       "2      Super Bowl 50 was an American football game to...   \n",
       "3      Super Bowl 50 was an American football game to...   \n",
       "4      Super Bowl 50 was an American football game to...   \n",
       "...                                                  ...   \n",
       "10565  The pound-force has a metric counterpart, less...   \n",
       "10566  The pound-force has a metric counterpart, less...   \n",
       "10567  The pound-force has a metric counterpart, less...   \n",
       "10568  The pound-force has a metric counterpart, less...   \n",
       "10569  The pound-force has a metric counterpart, less...   \n",
       "\n",
       "                                                 answers  c_id  \n",
       "0      [{'answer_start': 177, 'text': 'Denver Broncos...     0  \n",
       "1      [{'answer_start': 249, 'text': 'Carolina Panth...     0  \n",
       "2      [{'answer_start': 403, 'text': 'Santa Clara, C...     0  \n",
       "3      [{'answer_start': 177, 'text': 'Denver Broncos...     0  \n",
       "4      [{'answer_start': 488, 'text': 'gold'}, {'answ...     0  \n",
       "...                                                  ...   ...  \n",
       "10565  [{'answer_start': 82, 'text': 'kilogram-force'...  2066  \n",
       "10566  [{'answer_start': 114, 'text': 'kilopond'}, {'...  2066  \n",
       "10567  [{'answer_start': 274, 'text': 'slug'}, {'answ...  2066  \n",
       "10568  [{'answer_start': 712, 'text': 'kip'}, {'answe...  2066  \n",
       "10569  [{'answer_start': 665, 'text': 'sthène'}, {'an...  2066  \n",
       "\n",
       "[10570 rows x 5 columns]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question :  Who was the main performer at this year's halftime show?\n",
      "--------------------------------------------------\n",
      "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
      "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
      "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
      "--------------------------------------------------\n",
      "ANSWER :  \n",
      "\n",
      "\n",
      "\n",
      "Untokenized Answer : "
     ]
    },
    {
     "ename": "UnboundLocalError",
     "evalue": "local variable 'w' referenced before assignment",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mUnboundLocalError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-79-deb504855dff>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      4\u001b[0m     \u001b[0mquestion\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdev\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'question'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m     \u001b[0manswers\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdev\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'answers'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m     \u001b[0mpredict_letter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mquestion\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdoc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      7\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"real answer : \"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0manswers\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-74-56bc109b8b87>\u001b[0m in \u001b[0;36mpredict_letter\u001b[1;34m(question, doc)\u001b[0m\n\u001b[0;32m     44\u001b[0m             \u001b[0mw\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m\" \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mw\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     45\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 46\u001b[1;33m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     47\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mUnboundLocalError\u001b[0m: local variable 'w' referenced before assignment"
     ]
    }
   ],
   "source": [
    "import random\n",
    "for i in random.sample(range(100),100):\n",
    "    doc = dev['context'][i]\n",
    "    question = dev['question'][i]\n",
    "    answers = dev['answers'][i]\n",
    "    predict_letter(question, doc)\n",
    "    print(\"\")\n",
    "    print(\"real answer : \", answers)\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}