06_Named_Entity_Recognition_(NER)_in_Tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Names Entity Recognition (NER) in Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, we solve the another application of Natural Language Processing - Names Entity Recognition (NER) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![ner]('./img/ner.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Named entity recognition (NER) ‒ also called entity identification or entity extraction ‒ is an information extraction technique that automatically identifies named entities in a text and classifies them into predefined categories. Entities can be names of people, organizations, locations, times, quantities, monetary values, percentages, and more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Named entity recognition (NER)is probably the first step towards information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![ner]('./img/ner1.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will tackle this Names Entity Recognition (NER) problem in four steps:  \n",
    "1) Load data and preprocessing \n",
    "2) Create BERT input \n",
    "3) Create Model using BERT \n",
    "4) Check loss \n",
    "5) Inference with testset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0626 14:28:07.756202 20232 file_utils.py:39] PyTorch version 1.5.0 available.\n",
      "I0626 14:28:07.759197 20232 file_utils.py:55] TensorFlow version 2.2.0 available.\n",
      "wandb: WARNING W&B installed but not logged in.  Run `wandb login` or set the WANDB_API_KEY env variable.\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from transformers import *\n",
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "import os\n",
    "import re\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Load data and preprocessing "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data is from Naver NLP challenge [github](https://github.com/naver/nlp-challenge/blob/master/missions/ner/data/train/train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or from Wget\n",
    "# !wget https://github.com/naver/nlp-challenge/raw/master/missions/ner/data/train/train_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = os.path.abspath('./data/ner')\n",
    "train = pd.read_table(os.path.join(path,\"train_data.txt\"),names=['src', 'tar'], sep=\"\\t\")\n",
    "train = train.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>src</th>\n",
       "      <th>tar</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>비토리오</td>\n",
       "      <td>PER_B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>양일</td>\n",
       "      <td>DAT_B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>만에</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>영사관</td>\n",
       "      <td>ORG_B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>감호</td>\n",
       "      <td>CVL_B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769060</th>\n",
       "      <td>2</td>\n",
       "      <td>어째</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769061</th>\n",
       "      <td>3</td>\n",
       "      <td>뭔가</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769062</th>\n",
       "      <td>4</td>\n",
       "      <td>수상쩍은</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769063</th>\n",
       "      <td>5</td>\n",
       "      <td>좌담</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769064</th>\n",
       "      <td>6</td>\n",
       "      <td>．</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>769065 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        index   src    tar\n",
       "0           1  비토리오  PER_B\n",
       "1           2    양일  DAT_B\n",
       "2           3    만에      -\n",
       "3           4   영사관  ORG_B\n",
       "4           5    감호  CVL_B\n",
       "...       ...   ...    ...\n",
       "769060      2    어째      -\n",
       "769061      3    뭔가      -\n",
       "769062      4  수상쩍은      -\n",
       "769063      5    좌담      -\n",
       "769064      6     ．      -\n",
       "\n",
       "[769065 rows x 3 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>src</th>\n",
       "      <th>tar</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>6</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>15</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>10</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>24</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>19</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769013</th>\n",
       "      <td>11</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769036</th>\n",
       "      <td>23</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769053</th>\n",
       "      <td>17</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769058</th>\n",
       "      <td>5</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769064</th>\n",
       "      <td>6</td>\n",
       "      <td>.</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>57254 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        index src tar\n",
       "15          6   .   -\n",
       "30         15   .   -\n",
       "40         10   .   -\n",
       "77         24   .   -\n",
       "96         19   .   -\n",
       "...       ...  ..  ..\n",
       "769013     11   .   -\n",
       "769036     23   .   -\n",
       "769053     17   .   -\n",
       "769058      5   .   -\n",
       "769064      6   .   -\n",
       "\n",
       "[57254 rows x 3 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['src'] = train['src'].str.replace(\"．\", \".\", regex=False)\n",
    "train.loc[train['src']=='.']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lower the case, and delete all other except Korean / English words\n",
    "train['src'] = train['src'].astype(str)\n",
    "train['tar'] = train['tar'].astype(str)\n",
    "\n",
    "train['src'] = train['src'].str.replace(r'[^ㄱ-ㅣ가-힣0-9a-zA-Z.]+', \"\", regex=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change Pandas datframe into list formar\n",
    "data = [list(x) for x in train[['index', 'src', 'tar']].to_numpy()]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1, '비토리오', 'PER_B'], [2, '양일', 'DAT_B'], [3, '만에', '-'], [4, '영사관', 'ORG_B'], [5, '감호', 'CVL_B'], [6, '용퇴', '-'], [7, '항룡', '-'], [8, '압력설', '-'], [9, '의심만', '-'], [10, '가율', '-'], [1, '이', '-'], [2, '음경동맥의', '-'], [3, '직경이', '-'], [4, '8', 'NUM_B'], [5, '19mm입니다', 'NUM_B'], [6, '.', '-'], [1, '9세이브로', 'NUM_B'], [2, '구완', '-'], [3, '30위인', 'NUM_B'], [4, 'LG', 'ORG_B']]\n"
     ]
    }
   ],
   "source": [
    "# It's a one sentence before the index goes back to 1\n",
    "print(data[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract Labels and save in dictionary format\n",
    "label = train['tar'].unique().tolist()\n",
    "label_dict = {word:i for i, word in enumerate(label)}\n",
    "label_dict.update({\"[PAD]\":len(label_dict)})\n",
    "index_to_ner = {i:j for j, i in label_dict.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'PER_B': 0, 'DAT_B': 1, '-': 2, 'ORG_B': 3, 'CVL_B': 4, 'NUM_B': 5, 'LOC_B': 6, 'EVT_B': 7, 'TRM_B': 8, 'TRM_I': 9, 'EVT_I': 10, 'PER_I': 11, 'CVL_I': 12, 'NUM_I': 13, 'TIM_B': 14, 'TIM_I': 15, 'ORG_I': 16, 'DAT_I': 17, 'ANM_B': 18, 'MAT_B': 19, 'MAT_I': 20, 'AFW_B': 21, 'FLD_B': 22, 'LOC_I': 23, 'AFW_I': 24, 'PLT_B': 25, 'FLD_I': 26, 'ANM_I': 27, 'PLT_I': 28, '[PAD]': 29}\n"
     ]
    }
   ],
   "source": [
    "print(label_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 'PER_B', 1: 'DAT_B', 2: '-', 3: 'ORG_B', 4: 'CVL_B', 5: 'NUM_B', 6: 'LOC_B', 7: 'EVT_B', 8: 'TRM_B', 9: 'TRM_I', 10: 'EVT_I', 11: 'PER_I', 12: 'CVL_I', 13: 'NUM_I', 14: 'TIM_B', 15: 'TIM_I', 16: 'ORG_I', 17: 'DAT_I', 18: 'ANM_B', 19: 'MAT_B', 20: 'MAT_I', 21: 'AFW_B', 22: 'FLD_B', 23: 'LOC_I', 24: 'AFW_I', 25: 'PLT_B', 26: 'FLD_I', 27: 'ANM_I', 28: 'PLT_I', 29: '[PAD]'}\n"
     ]
    }
   ],
   "source": [
    "print(index_to_ner)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In each tups (tups[0], tups[1],..), words and index number would be stored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['비토리오', 'PER_B']]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tups = []\n",
    "temp_tup = []\n",
    "temp_tup.append(data[0][1:])\n",
    "sentences = []\n",
    "targets = []\n",
    "for i, j, k in data:\n",
    "    \n",
    "    if i != 1:\n",
    "        temp_tup.append([j,label_dict[k]])\n",
    "    if i == 1:\n",
    "        if len(temp_tup) != 0:\n",
    "            tups.append(temp_tup)\n",
    "            temp_tup = []\n",
    "            temp_tup.append([j,label_dict[k]])\n",
    "\n",
    "tups.pop(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['비토리오', 0], ['양일', 1], ['만에', 2], ['영사관', 3], ['감호', 4], ['용퇴', 2], ['항룡', 2], ['압력설', 2], ['의심만', 2], ['가율', 2]] [['이', 2], ['음경동맥의', 2], ['직경이', 2], ['8', 5], ['19mm입니다', 5], ['.', 2]]\n"
     ]
    }
   ],
   "source": [
    "print(tups[0], tups[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## If we look at each tups, it is stored as (word, index),(word, index),(word, index),,, but we would like to change this to (word,word,word,word,,,) and (index,index,index,index,,,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = []\n",
    "targets = []\n",
    "for tup in tups:\n",
    "    sentence = []\n",
    "    target = []\n",
    "    sentence.append(\"[CLS]\")\n",
    "    target.append(label_dict['-'])\n",
    "    for i, j in tup:\n",
    "        sentence.append(i)\n",
    "        target.append(j)\n",
    "    sentence.append(\"[SEP]\")\n",
    "    target.append(label_dict['-'])\n",
    "    sentences.append(sentence)\n",
    "    targets.append(target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[CLS]',\n",
       " '비토리오',\n",
       " '양일',\n",
       " '만에',\n",
       " '영사관',\n",
       " '감호',\n",
       " '용퇴',\n",
       " '항룡',\n",
       " '압력설',\n",
       " '의심만',\n",
       " '가율',\n",
       " '[SEP]']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 0, 1, 2, 3, 4, 2, 2, 2, 2, 2, 2]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targets[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Create BERT input "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From KoBert [Github](https://github.com/monologg/KoBERT-NER/blob/master/tokenization_kobert.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import unicodedata\n",
    "from shutil import copyfile\n",
    "\n",
    "from transformers import PreTrainedTokenizer\n",
    "\n",
    "\n",
    "logger = logging.getLogger(__name__)\n",
    "\n",
    "VOCAB_FILES_NAMES = {\"vocab_file\": \"tokenizer_78b3253a26.model\",\n",
    "                     \"vocab_txt\": \"vocab.txt\"}\n",
    "\n",
    "PRETRAINED_VOCAB_FILES_MAP = {\n",
    "    \"vocab_file\": {\n",
    "        \"monologg/kobert\": \"https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert/tokenizer_78b3253a26.model\",\n",
    "        \"monologg/kobert-lm\": \"https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert-lm/tokenizer_78b3253a26.model\",\n",
    "        \"monologg/distilkobert\": \"https://s3.amazonaws.com/models.huggingface.co/bert/monologg/distilkobert/tokenizer_78b3253a26.model\"\n",
    "    },\n",
    "    \"vocab_txt\": {\n",
    "        \"monologg/kobert\": \"https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert/vocab.txt\",\n",
    "        \"monologg/kobert-lm\": \"https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert-lm/vocab.txt\",\n",
    "        \"monologg/distilkobert\": \"https://s3.amazonaws.com/models.huggingface.co/bert/monologg/distilkobert/vocab.txt\"\n",
    "    }\n",
    "}\n",
    "\n",
    "PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {\n",
    "    \"monologg/kobert\": 512,\n",
    "    \"monologg/kobert-lm\": 512,\n",
    "    \"monologg/distilkobert\": 512\n",
    "}\n",
    "\n",
    "PRETRAINED_INIT_CONFIGURATION = {\n",
    "    \"monologg/kobert\": {\"do_lower_case\": False},\n",
    "    \"monologg/kobert-lm\": {\"do_lower_case\": False},\n",
    "    \"monologg/distilkobert\": {\"do_lower_case\": False}\n",
    "}\n",
    "\n",
    "SPIECE_UNDERLINE = u'▁'\n",
    "\n",
    "\n",
    "class KoBertTokenizer(PreTrainedTokenizer):\n",
    "    \"\"\"\n",
    "        SentencePiece based tokenizer. Peculiarities:\n",
    "            - requires `SentencePiece <https://github.com/google/sentencepiece>`_\n",
    "    \"\"\"\n",
    "    vocab_files_names = VOCAB_FILES_NAMES\n",
    "    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP\n",
    "    pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION\n",
    "    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES\n",
    "\n",
    "    def __init__(\n",
    "            self,\n",
    "            vocab_file,\n",
    "            vocab_txt,\n",
    "            do_lower_case=False,\n",
    "            remove_space=True,\n",
    "            keep_accents=False,\n",
    "            unk_token=\"[UNK]\",\n",
    "            sep_token=\"[SEP]\",\n",
    "            pad_token=\"[PAD]\",\n",
    "            cls_token=\"[CLS]\",\n",
    "            mask_token=\"[MASK]\",\n",
    "            **kwargs):\n",
    "        super().__init__(\n",
    "            unk_token=unk_token,\n",
    "            sep_token=sep_token,\n",
    "            pad_token=pad_token,\n",
    "            cls_token=cls_token,\n",
    "            mask_token=mask_token,\n",
    "            **kwargs\n",
    "        )\n",
    "\n",
    "        # Build vocab\n",
    "        self.token2idx = dict()\n",
    "        self.idx2token = []\n",
    "        with open(vocab_txt, 'r', encoding='utf-8') as f:\n",
    "            for idx, token in enumerate(f):\n",
    "                token = token.strip()\n",
    "                self.token2idx[token] = idx\n",
    "                self.idx2token.append(token)\n",
    "\n",
    "        try:\n",
    "            import sentencepiece as spm\n",
    "        except ImportError:\n",
    "            logger.warning(\"You need to install SentencePiece to use KoBertTokenizer: https://github.com/google/sentencepiece\"\n",
    "                           \"pip install sentencepiece\")\n",
    "\n",
    "        self.do_lower_case = do_lower_case\n",
    "        self.remove_space = remove_space\n",
    "        self.keep_accents = keep_accents\n",
    "        self.vocab_file = vocab_file\n",
    "        self.vocab_txt = vocab_txt\n",
    "\n",
    "        self.sp_model = spm.SentencePieceProcessor()\n",
    "        self.sp_model.Load(vocab_file)\n",
    "\n",
    "    @property\n",
    "    def vocab_size(self):\n",
    "        return len(self.idx2token)\n",
    "\n",
    "    def get_vocab(self):\n",
    "        return dict(self.token2idx, **self.added_tokens_encoder)\n",
    "\n",
    "    def __getstate__(self):\n",
    "        state = self.__dict__.copy()\n",
    "        state[\"sp_model\"] = None\n",
    "        return state\n",
    "\n",
    "    def __setstate__(self, d):\n",
    "        self.__dict__ = d\n",
    "        try:\n",
    "            import sentencepiece as spm\n",
    "        except ImportError:\n",
    "            logger.warning(\"You need to install SentencePiece to use KoBertTokenizer: https://github.com/google/sentencepiece\"\n",
    "                           \"pip install sentencepiece\")\n",
    "        self.sp_model = spm.SentencePieceProcessor()\n",
    "        self.sp_model.Load(self.vocab_file)\n",
    "\n",
    "    def preprocess_text(self, inputs):\n",
    "        if self.remove_space:\n",
    "            outputs = \" \".join(inputs.strip().split())\n",
    "        else:\n",
    "            outputs = inputs\n",
    "        outputs = outputs.replace(\"``\", '\"').replace(\"''\", '\"')\n",
    "\n",
    "        if not self.keep_accents:\n",
    "            outputs = unicodedata.normalize('NFKD', outputs)\n",
    "            outputs = \"\".join([c for c in outputs if not unicodedata.combining(c)])\n",
    "        if self.do_lower_case:\n",
    "            outputs = outputs.lower()\n",
    "\n",
    "        return outputs\n",
    "\n",
    "    def _tokenize(self, text, return_unicode=True, sample=False):\n",
    "        \"\"\" Tokenize a string. \"\"\"\n",
    "        text = self.preprocess_text(text)\n",
    "\n",
    "        if not sample:\n",
    "            pieces = self.sp_model.EncodeAsPieces(text)\n",
    "        else:\n",
    "            pieces = self.sp_model.SampleEncodeAsPieces(text, 64, 0.1)\n",
    "        new_pieces = []\n",
    "        for piece in pieces:\n",
    "            if len(piece) > 1 and piece[-1] == str(\",\") and piece[-2].isdigit():\n",
    "                cur_pieces = self.sp_model.EncodeAsPieces(piece[:-1].replace(SPIECE_UNDERLINE, \"\"))\n",
    "                if piece[0] != SPIECE_UNDERLINE and cur_pieces[0][0] == SPIECE_UNDERLINE:\n",
    "                    if len(cur_pieces[0]) == 1:\n",
    "                        cur_pieces = cur_pieces[1:]\n",
    "                    else:\n",
    "                        cur_pieces[0] = cur_pieces[0][1:]\n",
    "                cur_pieces.append(piece[-1])\n",
    "                new_pieces.extend(cur_pieces)\n",
    "            else:\n",
    "                new_pieces.append(piece)\n",
    "\n",
    "        return new_pieces\n",
    "\n",
    "    def _convert_token_to_id(self, token):\n",
    "        \"\"\" Converts a token (str/unicode) in an id using the vocab. \"\"\"\n",
    "        return self.token2idx.get(token, self.token2idx[self.unk_token])\n",
    "\n",
    "    def _convert_id_to_token(self, index, return_unicode=True):\n",
    "        \"\"\"Converts an index (integer) in a token (string/unicode) using the vocab.\"\"\"\n",
    "        return self.idx2token[index]\n",
    "\n",
    "    def convert_tokens_to_string(self, tokens):\n",
    "        \"\"\"Converts a sequence of tokens (strings for sub-words) in a single string.\"\"\"\n",
    "        out_string = \"\".join(tokens).replace(SPIECE_UNDERLINE, \" \").strip()\n",
    "        return out_string\n",
    "\n",
    "    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):\n",
    "        \"\"\"\n",
    "        Build model inputs from a sequence or a pair of sequence for sequence classification tasks\n",
    "        by concatenating and adding special tokens.\n",
    "        A KoBERT sequence has the following format:\n",
    "            single sequence: [CLS] X [SEP]\n",
    "            pair of sequences: [CLS] A [SEP] B [SEP]\n",
    "        \"\"\"\n",
    "        if token_ids_1 is None:\n",
    "            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]\n",
    "        cls = [self.cls_token_id]\n",
    "        sep = [self.sep_token_id]\n",
    "        return cls + token_ids_0 + sep + token_ids_1 + sep\n",
    "\n",
    "    def get_special_tokens_mask(self, token_ids_0, token_ids_1=None, already_has_special_tokens=False):\n",
    "        \"\"\"\n",
    "        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding\n",
    "        special tokens using the tokenizer ``prepare_for_model`` or ``encode_plus`` methods.\n",
    "        Args:\n",
    "            token_ids_0: list of ids (must not contain special tokens)\n",
    "            token_ids_1: Optional list of ids (must not contain special tokens), necessary when fetching sequence ids\n",
    "                for sequence pairs\n",
    "            already_has_special_tokens: (default False) Set to True if the token list is already formated with\n",
    "                special tokens for the model\n",
    "        Returns:\n",
    "            A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.\n",
    "        \"\"\"\n",
    "\n",
    "        if already_has_special_tokens:\n",
    "            if token_ids_1 is not None:\n",
    "                raise ValueError(\n",
    "                    \"You should not supply a second sequence if the provided sequence of \"\n",
    "                    \"ids is already formated with special tokens for the model.\"\n",
    "                )\n",
    "            return list(map(lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0, token_ids_0))\n",
    "\n",
    "        if token_ids_1 is not None:\n",
    "            return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]\n",
    "        return [1] + ([0] * len(token_ids_0)) + [1]\n",
    "\n",
    "    def create_token_type_ids_from_sequences(self, token_ids_0, token_ids_1=None):\n",
    "        \"\"\"\n",
    "        Creates a mask from the two sequences passed to be used in a sequence-pair classification task.\n",
    "        A KoBERT sequence pair mask has the following format:\n",
    "        0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1\n",
    "        | first sequence    | second sequence\n",
    "        if token_ids_1 is None, only returns the first portion of the mask (0's).\n",
    "        \"\"\"\n",
    "        sep = [self.sep_token_id]\n",
    "        cls = [self.cls_token_id]\n",
    "        if token_ids_1 is None:\n",
    "            return len(cls + token_ids_0 + sep) * [0]\n",
    "        return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]\n",
    "\n",
    "    def save_vocabulary(self, save_directory):\n",
    "        \"\"\" Save the sentencepiece vocabulary (copy original file) and special tokens file\n",
    "            to a directory.\n",
    "        \"\"\"\n",
    "        if not os.path.isdir(save_directory):\n",
    "            logger.error(\"Vocabulary path ({}) should be a directory\".format(save_directory))\n",
    "            return\n",
    "\n",
    "        # 1. Save sentencepiece model\n",
    "        out_vocab_model = os.path.join(save_directory, VOCAB_FILES_NAMES[\"vocab_file\"])\n",
    "\n",
    "        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_model):\n",
    "            copyfile(self.vocab_file, out_vocab_model)\n",
    "\n",
    "        # 2. Save vocab.txt\n",
    "        index = 0\n",
    "        out_vocab_txt = os.path.join(save_directory, VOCAB_FILES_NAMES[\"vocab_txt\"])\n",
    "        with open(out_vocab_txt, \"w\", encoding=\"utf-8\") as writer:\n",
    "            for token, token_index in sorted(self.token2idx.items(), key=lambda kv: kv[1]):\n",
    "                if index != token_index:\n",
    "                    logger.warning(\n",
    "                        \"Saving vocabulary to {}: vocabulary indices are not consecutive.\"\n",
    "                        \" Please check that the vocabulary is not corrupted!\".format(out_vocab_txt)\n",
    "                    )\n",
    "                    index = token_index\n",
    "                writer.write(token + \"\\n\")\n",
    "                index += 1\n",
    "\n",
    "        return out_vocab_model, out_vocab_txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0626 14:45:36.301329 20232 filelock.py:274] Lock 1573363001672 acquired on C:\\Users\\bokhy/.cache\\torch\\transformers\\1f871cf1d80a4c09742b5f2094cc7ab9163dfbcfb930d99fdc9149aebf6a13bb.7555cd8e0e341a4d19f33dcb20eaf8614c72f56827d1b28481046caabba18eb3.lock\n",
      "I0626 14:45:36.305331 20232 file_utils.py:436] https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert/tokenizer_78b3253a26.model not found in cache or force_download set to True, downloading to C:\\Users\\bokhy\\.cache\\torch\\transformers\\tmpfkj04ifw\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5573a562a993431abbc93cd1ccbdde9d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=371391.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0626 14:45:37.082502 20232 file_utils.py:440] storing https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert/tokenizer_78b3253a26.model in cache at C:\\Users\\bokhy/.cache\\torch\\transformers\\1f871cf1d80a4c09742b5f2094cc7ab9163dfbcfb930d99fdc9149aebf6a13bb.7555cd8e0e341a4d19f33dcb20eaf8614c72f56827d1b28481046caabba18eb3\n",
      "I0626 14:45:37.088491 20232 file_utils.py:443] creating metadata file for C:\\Users\\bokhy/.cache\\torch\\transformers\\1f871cf1d80a4c09742b5f2094cc7ab9163dfbcfb930d99fdc9149aebf6a13bb.7555cd8e0e341a4d19f33dcb20eaf8614c72f56827d1b28481046caabba18eb3\n",
      "I0626 14:45:37.093480 20232 filelock.py:318] Lock 1573363001672 released on C:\\Users\\bokhy/.cache\\torch\\transformers\\1f871cf1d80a4c09742b5f2094cc7ab9163dfbcfb930d99fdc9149aebf6a13bb.7555cd8e0e341a4d19f33dcb20eaf8614c72f56827d1b28481046caabba18eb3.lock\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0626 14:45:37.453058 20232 filelock.py:274] Lock 1573335400584 acquired on C:\\Users\\bokhy/.cache\\torch\\transformers\\c247d0225676d99ae7f3e6d39dba68b77756181415828f922c6393e4aecb53ce.5d8505da4be274704344000e8d9a798ff03b513f5053f98f7e97d719a9a9891b.lock\n",
      "I0626 14:45:37.456047 20232 file_utils.py:436] https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert/vocab.txt not found in cache or force_download set to True, downloading to C:\\Users\\bokhy\\.cache\\torch\\transformers\\tmpeni7fmal\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "139230968a874285a2faa627c3a46a1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=77779.0, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0626 14:45:38.039345 20232 file_utils.py:440] storing https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert/vocab.txt in cache at C:\\Users\\bokhy/.cache\\torch\\transformers\\c247d0225676d99ae7f3e6d39dba68b77756181415828f922c6393e4aecb53ce.5d8505da4be274704344000e8d9a798ff03b513f5053f98f7e97d719a9a9891b\n",
      "I0626 14:45:38.044367 20232 file_utils.py:443] creating metadata file for C:\\Users\\bokhy/.cache\\torch\\transformers\\c247d0225676d99ae7f3e6d39dba68b77756181415828f922c6393e4aecb53ce.5d8505da4be274704344000e8d9a798ff03b513f5053f98f7e97d719a9a9891b\n",
      "I0626 14:45:38.048319 20232 filelock.py:318] Lock 1573335400584 released on C:\\Users\\bokhy/.cache\\torch\\transformers\\c247d0225676d99ae7f3e6d39dba68b77756181415828f922c6393e4aecb53ce.5d8505da4be274704344000e8d9a798ff03b513f5053f98f7e97d719a9a9891b.lock\n",
      "I0626 14:45:38.049320 20232 tokenization_utils.py:1015] loading file https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert/tokenizer_78b3253a26.model from cache at C:\\Users\\bokhy/.cache\\torch\\transformers\\1f871cf1d80a4c09742b5f2094cc7ab9163dfbcfb930d99fdc9149aebf6a13bb.7555cd8e0e341a4d19f33dcb20eaf8614c72f56827d1b28481046caabba18eb3\n",
      "I0626 14:45:38.051313 20232 tokenization_utils.py:1015] loading file https://s3.amazonaws.com/models.huggingface.co/bert/monologg/kobert/vocab.txt from cache at C:\\Users\\bokhy/.cache\\torch\\transformers\\c247d0225676d99ae7f3e6d39dba68b77756181415828f922c6393e4aecb53ce.5d8505da4be274704344000e8d9a798ff03b513f5053f98f7e97d719a9a9891b\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = KoBertTokenizer.from_pretrained('monologg/kobert')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['▁이', '▁책', '▁정말', '▁재미있', '다', '!']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.tokenize(\"이 책 정말 재미있다!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important steps from here\n",
    "We will tokenize the sentences, and have target to the fit into the tokenized sentences\n",
    "\n",
    "For example, above sentence \"이 책 정말 재미있다!\" actually shows ('▁이', '▁책', '▁정말', '▁재미있', '다', '!') when tokenized\n",
    "So we need to convert it into ( ▁이, entity1) , (▁책, entity2), (▁정말, entity2),,,, (다, entity3) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_and_preserve_labels(sentence, text_labels):\n",
    "    tokenized_sentence = []\n",
    "    labels = []\n",
    "\n",
    "    for word, label in zip(sentence, text_labels):\n",
    "        tokenized_word = tokenizer.tokenize(word)\n",
    "        n_subwords = len(tokenized_word)\n",
    "\n",
    "        tokenized_sentence.extend(tokenized_word)\n",
    "        labels.extend([label] * n_subwords)\n",
    "\n",
    "    return tokenized_sentence, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_texts_and_labels = [tokenize_and_preserve_labels(sent, labs)\n",
    "                              for sent, labs in zip(sentences, targets)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['[CLS]', '▁비', '토', '리', '오', '▁양', '일', '▁만에', '▁영', '사', '관', '▁감', '호', '▁용', '퇴', '▁항', '룡', '▁압력', '설', '▁의심', '만', '▁', '가', '율', '[SEP]'], [2, 0, 0, 0, 0, 1, 1, 2, 3, 3, 3, 4, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]), (['[CLS]', '▁이', '▁음', '경', '동', '맥', '의', '▁직', '경', '이', '▁8', '▁19', 'mm', '입니다', '▁', '.', '[SEP]'], [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 5, 5, 5, 5, 2, 2, 2])]\n"
     ]
    }
   ],
   "source": [
    "# It's saved as (sentence, entity),(sentence, entity),(sentence, entity),...\n",
    "print(tokenized_texts_and_labels[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### So, we would divide into (sentence, sentence, sentence,...) and (entity,entity,entity,...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_texts = [token_label_pair[0] for token_label_pair in tokenized_texts_and_labels]\n",
    "labels = [token_label_pair[1] for token_label_pair in tokenized_texts_and_labels]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[CLS]',\n",
       " '▁이',\n",
       " '▁음',\n",
       " '경',\n",
       " '동',\n",
       " '맥',\n",
       " '의',\n",
       " '▁직',\n",
       " '경',\n",
       " '이',\n",
       " '▁8',\n",
       " '▁19',\n",
       " 'mm',\n",
       " '입니다',\n",
       " '▁',\n",
       " '.',\n",
       " '[SEP]']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_texts[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 5, 5, 5, 5, 2, 2, 2]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's cut the sentence to save memory, if the sentence length if greater than 88, it's cut. If it's shorter than 88, paddings are added to make it all 88 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "88.0\n"
     ]
    }
   ],
   "source": [
    "print(np.quantile(np.array([len(x) for x in tokenized_texts]), 0.975))\n",
    "max_len = 88\n",
    "bs = 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's create a train input \n",
    "input_ids : numeric tokenized sentence\n",
    "attention_masks : If not padding it's 1, if padding it's \n",
    "\n",
    "[input_ids, attention_masks] are used as input to BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sentence as input\n",
    "input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts],\n",
    "                          maxlen=max_len, dtype = \"int\", value=tokenizer.convert_tokens_to_ids(\"[PAD]\"), truncating=\"post\", padding=\"post\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2, 3647, 3606, 5424, 5872, 6172, 7095, 4349, 5424, 7096,  624,\n",
       "        548,  424, 7139,  517,   54,    3,    1,    1,    1,    1,    1,\n",
       "          1,    1,    1,    1,    1,    1,    1,    1,    1,    1,    1,\n",
       "          1,    1,    1,    1,    1,    1,    1,    1,    1,    1,    1,\n",
       "          1,    1,    1,    1,    1,    1,    1,    1,    1,    1,    1,\n",
       "          1,    1,    1,    1,    1,    1,    1,    1,    1,    1,    1,\n",
       "          1,    1,    1,    1,    1,    1,    1,    1,    1,    1,    1,\n",
       "          1,    1,    1,    1,    1,    1,    1,    1,    1,    1,    1])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_ids[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer as input\n",
    "tags = pad_sequences([lab for lab in labels], maxlen=max_len, value=label_dict[\"[PAD]\"], padding='post',\\\n",
    "                     dtype='int', truncating='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  5,  5,  5,  5,  2,  2,  2,\n",
       "       29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29,\n",
       "       29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29,\n",
       "       29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29,\n",
       "       29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29,\n",
       "       29, 29, 29])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# attention_masks\n",
    "attention_masks = np.array([[int(i != tokenizer.convert_tokens_to_ids(\"[PAD]\")) for i in ii] for ii in input_ids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 1, ..., 0, 0, 0],\n",
       "       [1, 1, 1, ..., 0, 0, 0],\n",
       "       [1, 1, 1, ..., 0, 0, 0],\n",
       "       ...,\n",
       "       [1, 1, 1, ..., 0, 0, 0],\n",
       "       [1, 1, 1, ..., 0, 0, 0],\n",
       "       [1, 1, 1, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attention_masks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set 10% of train dataset to Validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "tr_inputs, val_inputs, tr_tags, val_tags = train_test_split(input_ids, \n",
    "                                                            tags,\n",
    "                                                            random_state=623, \n",
    "                                                            test_size=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "tr_masks, val_masks, _, _ = train_test_split(attention_masks, \n",
    "                                             input_ids,\n",
    "                                             random_state=623, \n",
    "                                             test_size=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Create Model using BERT "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "SEQ_LEN = max_len\n",
    "\n",
    "def create_model():\n",
    "    model = TFBertModel.from_pretrained(\"monologg/kobert\", from_pt=True, num_labels=len(label_dict), output_attentions = False,\n",
    "        output_hidden_states = False)\n",
    "\n",
    "    token_inputs = tf.keras.layers.Input((SEQ_LEN,), dtype=tf.int32, name='input_word_ids')\n",
    "    mask_inputs = tf.keras.layers.Input((SEQ_LEN,), dtype=tf.int32, name='input_masks') \n",
    "\n",
    "    bert_outputs = model([token_inputs, mask_inputs])\n",
    "    bert_outputs = bert_outputs[0] # shape : (Batch_size, max_len, 30)\n",
    "    nr = tf.keras.layers.Dense(30, activation='softmax')(bert_outputs) # shape : (Batch_size, max_len, 30)\n",
    "\n",
    "    nr_model = tf.keras.Model([token_inputs, mask_inputs], nr)\n",
    "\n",
    "    nr_model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.0001), loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),\n",
    "          metrics=['sparse_categorical_accuracy'])\n",
    "    nr_model.summary()\n",
    "    \n",
    "    return nr_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start Traning\n",
    "nr_model = create_model()\n",
    "nr_model.fit([tr_inputs, tr_masks], tr_tags, validation_data=([val_inputs, val_masks], val_tags), epochs=1, shuffle=False, batch_size=bs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![progree]('./img/progress.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Check loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check F1 Score\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score, classification_report\n",
    "y_predicted = nr_model.predict([val_inputs, val_masks])\n",
    "\n",
    "f_label = [i for i, j in label_dict.items()]\n",
    "val_tags_l = [index_to_ner[x] for x in np.ravel(val_tags).astype(int).tolist()]\n",
    "y_predicted_l = [index_to_ner[x] for x in np.ravel(np.argmax(y_predicted, axis=2)).astype(int).tolist()]\n",
    "f_label.remove(\"[PAD]\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(val_tags_l, y_predicted_l, labels=f_label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 Inference with testset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ner_inference(test_sentence):\n",
    "  \n",
    "    tokenized_sentence = np.array([tokenizer.encode(test_sentence, max_length=max_len, pad_to_max_length=True)])\n",
    "    tokenized_mask = np.array([[int(x!=1) for x in tokenized_sentence[0].tolist()]])\n",
    "    ans = nr_model.predict([tokenized_sentence, tokenized_mask])\n",
    "    ans = np.argmax(ans, axis=2)\n",
    "\n",
    "    tokens = tokenizer.convert_ids_to_tokens(tokenized_sentence[0])\n",
    "    new_tokens, new_labels = [], []\n",
    "    for token, label_idx in zip(tokens, ans[0]):\n",
    "        \n",
    "        if (token.startswith(\"▁\")):\n",
    "            new_labels.append(index_to_ner[label_idx])\n",
    "            new_tokens.append(token[1:])\n",
    "        elif (token=='[CLS]'):\n",
    "            pass\n",
    "        elif (token=='[SEP]'):\n",
    "            pass\n",
    "        elif (token=='[PAD]'):\n",
    "            pass\n",
    "        elif (token != '[CLS]' or token != '[SEP]'):\n",
    "            new_tokens[-1] = new_tokens[-1] + token\n",
    "\n",
    "    for token, label in zip(new_tokens, new_labels):\n",
    "          print(\"{}\\t{}\".format(label, token))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_inference(\"9세이브로 구완 30위인 LG 박찬형은 평균자책점이 16.45로 준수한 편이지만 22이닝 동안 피홈런이 31개나 된다.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reference: \n",
    "- https://github.com/monologg/KoBERT-NER\n",
    "- https://monkeylearn.com/blog/named-entity-recognition/\n",
    "- https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}