E15/E15_Transformer_Chatbot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "official-commitment",
   "metadata": {},
   "source": [
    "# Exploration 15\n",
    "## 트랜스포머 모델로 챗봇 만들기 🎯\n",
    "\n",
    "### Table of contents\n",
    "1. [데이터 수집]()\n",
    "2. [데이터 전처리]()\n",
    "3. [SubwordTextEncoder]()\n",
    "4. [Transfomer components]()\n",
    "\t\n",
    "    - Positional Encoding\n",
    "\t- Scaled_dot_product_attention\n",
    "\t- MultiHeadAttention\n",
    "\t- Masks\n",
    "\t- Encoder\n",
    "\t- Decoder\n",
    "\t- Transformer\n",
    "    \n",
    "\n",
    "5. [모델링]()\n",
    "6. [개선을 위한 전처리 수정]()\n",
    "7. [하이퍼파리미터 튜닝]()\n",
    "8. [회고]()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rapid-treaty",
   "metadata": {},
   "source": [
    "## 1. 데이터 수집"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 449,
   "id": "terminal-henry",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "import os\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 450,
   "id": "split-antenna",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = os.getenv(\"HOME\") + '/aiffel/transformer_chatbot/data/ChatbotData .csv'\n",
    "chatbotdata = pd.read_csv(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 451,
   "id": "apparent-assessment",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12시 땡!</td>\n",
       "      <td>하루가 또 가네요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1지망 학교 떨어졌어</td>\n",
       "      <td>위로해 드립니다.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3박4일 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3박4일 정도 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PPL 심하네</td>\n",
       "      <td>눈살이 찌푸려지죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Q            A  label\n",
       "0           12시 땡!   하루가 또 가네요.      0\n",
       "1      1지망 학교 떨어졌어    위로해 드립니다.      0\n",
       "2     3박4일 놀러가고 싶다  여행은 언제나 좋죠.      0\n",
       "3  3박4일 정도 놀러가고 싶다  여행은 언제나 좋죠.      0\n",
       "4          PPL 심하네   눈살이 찌푸려지죠.      0"
      ]
     },
     "execution_count": 451,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chatbotdata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 452,
   "id": "tribal-poland",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5911</th>\n",
       "      <td>너무 가슴이 아프네</td>\n",
       "      <td>무슨 마음인지 알겠어서 더 마음이 아프네요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5912</th>\n",
       "      <td>너무 괴롭네.</td>\n",
       "      <td>무슨 마음인지 알겠어서 더 마음이 아프네요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5913</th>\n",
       "      <td>너무 끔직한 5월의 연휴도</td>\n",
       "      <td>다 지나갈 거예요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5914</th>\n",
       "      <td>너무 너무 억울하다.</td>\n",
       "      <td>억울함이 풀리길 바랄게요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5915</th>\n",
       "      <td>너무 늦은 깨달음</td>\n",
       "      <td>너무 늦지 않았을 거예요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Q                         A  label\n",
       "5911      너무 가슴이 아프네  무슨 마음인지 알겠어서 더 마음이 아프네요.      1\n",
       "5912         너무 괴롭네.  무슨 마음인지 알겠어서 더 마음이 아프네요.      1\n",
       "5913  너무 끔직한 5월의 연휴도                다 지나갈 거예요.      1\n",
       "5914     너무 너무 억울하다.            억울함이 풀리길 바랄게요.      1\n",
       "5915       너무 늦은 깨달음            너무 늦지 않았을 거예요.      1"
      ]
     },
     "execution_count": 452,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mid = len(chatbotdata) // 2\n",
    "chatbotdata.iloc[mid:mid+5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 453,
   "id": "flush-anime",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11818</th>\n",
       "      <td>훔쳐보는 것도 눈치 보임.</td>\n",
       "      <td>티가 나니까 눈치가 보이는 거죠!</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11819</th>\n",
       "      <td>훔쳐보는 것도 눈치 보임.</td>\n",
       "      <td>훔쳐보는 거 티나나봐요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11820</th>\n",
       "      <td>흑기사 해주는 짝남.</td>\n",
       "      <td>설렜겠어요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11821</th>\n",
       "      <td>힘든 연애 좋은 연애라는게 무슨 차이일까?</td>\n",
       "      <td>잘 헤어질 수 있는 사이 여부인 거 같아요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11822</th>\n",
       "      <td>힘들어서 결혼할까봐</td>\n",
       "      <td>도피성 결혼은 하지 않길 바라요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Q                         A  label\n",
       "11818           훔쳐보는 것도 눈치 보임.        티가 나니까 눈치가 보이는 거죠!      2\n",
       "11819           훔쳐보는 것도 눈치 보임.             훔쳐보는 거 티나나봐요.      2\n",
       "11820              흑기사 해주는 짝남.                    설렜겠어요.      2\n",
       "11821  힘든 연애 좋은 연애라는게 무슨 차이일까?  잘 헤어질 수 있는 사이 여부인 거 같아요.      2\n",
       "11822               힘들어서 결혼할까봐        도피성 결혼은 하지 않길 바라요.      2"
      ]
     },
     "execution_count": 453,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chatbotdata.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "royal-complex",
   "metadata": {},
   "source": [
    "## 2. 데이터 전처리\n",
    "\n",
    "구두점을 띄워주고 중복되는 공백을 하나의 공백으로 바꿔주는 최소한의 전처리를 진행해보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 454,
   "id": "narrative-arbitration",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_sentence(sentence):\n",
    "    sentence = re.sub(r\"([?.!,])\", r\" \\1 \", sentence)\n",
    "    sentence = sentence.strip()\n",
    "    \n",
    "    return sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 455,
   "id": "suspected-steps",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_conversations():\n",
    "    \n",
    "    questions = chatbotdata['Q'].apply(preprocess_sentence)\n",
    "    answers = chatbotdata['A'].apply(preprocess_sentence)\n",
    "    \n",
    "    return questions.to_numpy(), answers.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 456,
   "id": "veterinary-academy",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 질문 샘플 수: 11823\n",
      "전체 답변 샘플 수: 11823\n"
     ]
    }
   ],
   "source": [
    "questions, answers = load_conversations()\n",
    "print(f\"전체 질문 샘플 수: {len(questions)}\")\n",
    "print(f\"전체 답변 샘플 수: {len(answers)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 457,
   "id": "finished-appraisal",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전처리 후의 125번째 질문 샘플: 게임 같이 하자고 할까 ?\n",
      "전처리 후의 125번째 답변 샘플: 안 될 것도 없죠 .\n",
      "==================================================\n",
      "전처리 후의 11554번째 질문 샘플: 짝녀한테 의미 없는 선톡 해도 될까요 ?\n",
      "전처리 후의 11554번째 답변 샘플: 선톡을 해야 대화도 이어나가니까요 .\n",
      "==================================================\n",
      "전처리 후의 6152번째 질문 샘플: 또 다시 낮술\n",
      "전처리 후의 6152번째 답변 샘플: 낮술 조금만 하세요 .\n",
      "==================================================\n",
      "전처리 후의 1169번째 질문 샘플: 담배 끊는 거 어려워\n",
      "전처리 후의 1169번째 답변 샘플: 자신을 이겨야해요 .\n",
      "==================================================\n",
      "전처리 후의 7263번째 질문 샘플: 외로워서\n",
      "전처리 후의 7263번째 답변 샘플: 많이 힘든가봅니다 .\n",
      "==================================================\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng()\n",
    "\n",
    "\n",
    "for _ in range(5):\n",
    "    idx = rng.integers(len(answers))\n",
    "    print(f'전처리 후의 {idx}번째 질문 샘플: {questions[idx]}')\n",
    "    print(f'전처리 후의 {idx}번째 답변 샘플: {answers[idx]}')\n",
    "    print('=' * 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unexpected-pottery",
   "metadata": {},
   "source": [
    "## 3. SubwordTextEncoder 사용하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 458,
   "id": "large-colorado",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "tokenizer = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(questions + answers, \n",
    "                                                                      target_vocab_size=2**13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 459,
   "id": "buried-centre",
   "metadata": {},
   "outputs": [],
   "source": [
    "START_TOKEN, END_TOKEN = [tokenizer.vocab_size], [tokenizer.vocab_size+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 460,
   "id": "surprised-preparation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "START_TOKEN의 번호 : [8369]\n",
      "END_TOKEN의 번호 : [8370]\n"
     ]
    }
   ],
   "source": [
    "print('START_TOKEN의 번호 :' ,[tokenizer.vocab_size])\n",
    "print('END_TOKEN의 번호 :' ,[tokenizer.vocab_size + 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 461,
   "id": "critical-scott",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8371\n"
     ]
    }
   ],
   "source": [
    "# 시작 토큰과 종료 토큰을 고려하여 +2를 하여 단어장의 크기를 산정한다.\n",
    "VOCAB_SIZE = tokenizer.vocab_size + 2\n",
    "print(VOCAB_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 462,
   "id": "opponent-advertising",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전처리 후의 10509번째 질문 샘플: [6758, 1105, 525, 116, 3395, 174, 2331, 2]\n",
      "전처리 후의 10509번째 답변 샘플: [7364, 98, 7464, 2473, 1082, 93, 1]\n",
      "==================================================\n",
      "전처리 후의 7348번째 질문 샘플: [79, 229, 179, 2087, 4987, 1361]\n",
      "전처리 후의 7348번째 답변 샘플: [143, 294, 13, 2510, 1]\n",
      "==================================================\n",
      "전처리 후의 4004번째 질문 샘플: [6409, 2745, 2]\n",
      "전처리 후의 4004번째 답변 샘플: [311, 96, 783, 5, 30, 3, 1]\n",
      "==================================================\n",
      "전처리 후의 7308번째 질문 샘플: [2536, 6619, 534, 214, 1]\n",
      "전처리 후의 7308번째 답변 샘플: [485, 4810, 575, 4074, 27, 230, 137, 17, 21, 1]\n",
      "==================================================\n",
      "전처리 후의 4557번째 질문 샘플: [4837, 8145, 20, 159]\n",
      "전처리 후의 4557번째 답변 샘플: [768, 1938, 4836, 8145, 5, 30, 3, 1]\n",
      "==================================================\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng()\n",
    "\n",
    "\n",
    "for _ in range(5):\n",
    "    idx = rng.integers(len(answers))\n",
    "    print(f'전처리 후의 {idx}번째 질문 샘플: {tokenizer.encode(questions[idx])}')\n",
    "    print(f'전처리 후의 {idx}번째 답변 샘플: {tokenizer.encode(answers[idx])}')\n",
    "    print('=' * 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 463,
   "id": "changing-logan",
   "metadata": {},
   "outputs": [],
   "source": [
    "q_len = [len(s.split()) for s in questions]\n",
    "a_len = [len(s.split()) for s in answers]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 464,
   "id": "forbidden-pontiac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Answers')"
      ]
     },
     "execution_count": 464,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAegklEQVR4nO3df5xddX3n8debZEwIombCSANkEitIQ8Yf4IiWpGoWVKRU6NZ1zQILbUyaPuqoTShBpo+K3U1KKGI13TpNHASFjsuCv+oCykIAB1CZIMLEUaFIfkAgA0H5JSEhn/3jnElPhpnMnbl37jlz7/v5eNzH3PPj3vO5k/nmfb7nnPs9igjMzMyK5qC8CzAzMxuKA8rMzArJAWVmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUFUiaY6kkDS5Qu/3PyU9KenxSrzfeJB0kaQv512H1a56bFf1pC4CStIjkk6plW1KagZWAMdFxO+MxzZGS9J7JW3LzouI1RHxsbxqsvHldmXjrS4CqgY1A09FxI68CzGrITXTrirVo8xbXQeUpIMkXSjp3yU9JelaSY3psoFDB+dK2pJ2+9szrz1Y0lWSnpbUJ+mCgR6EpK+R/LH/m6TnJF2Q2exZQ73fELW9VtJXJfVL2izpb9J6TwFuBo5I3/vKYV7/15K2S3pM0p+ln+XodNltkj6WWfc8Sd2Z6d+TdLOknZJ+IekjmWWnSfqZpGclPSrpfEmHADdmanpO0hGSLpZ0dea1H5K0SdKv0xrmZpY9kr7X/ZJ+I+l/S5qaLjtM0nfT1+2U9ANJdf23W2S12q4kTU//DvvT+r4r6ajM8tsk/Q9Jd6bt4/uSDkuXTZV0dfr7+LWkeyQdLmmhpAcy73GzpHsy0z+QdGb6/AhJ16fb/5WkT2TWu1jSdek2ngHOk3SipB5Jz0h6QtLlpf4bFkZE1PwDeAQ4ZYj5nwR+CBwFTAH+BehKl80BAlgPHAy8FdgFzE2XXwLcDkxPX38/sG24bY70fkPU9lXg28Ch6Wt/CSxOl703u60hXnsq8ATQAhwC/Gu67aPT5bcBH8usfx7QnT4/BNgK/CkwGTgeeJLksAfAduAP0ufTgROGqwm4GLg6ff4m4HngfUADcAHwEPCqzO/rx8ARQCPQByxLl/090JG+rgH4A0B5/13V+6MO29UM4E+Aaenr/w/wrczy24B/T//WD06nL0mX/Tnwb+lrJwFvB16TrvcicFj6t/0E8Gj6/gcDv023exCwEfhb4FXA7wIPAx/ItLXdwJnpugcDdwPnpMtfDbwr77+Z0T7qfS90GdAeEdsiYhfJP/KHtX/3+LMR8duI+CnwU5IGAPARYHVEPB0R24AvlrjN4d5vH0mTgI8Cn46IZyPiEeBzwDklbuMjwFciojcink8/V6lOBx6JiK9ExJ6I+AlwPfBf0uW7geMkvSb97PeW+L7/Ffi/EXFzROwGLiNpRCdl1vliRDwWETtJGvPbMtucCcyOiN0R8YNIW50VUk22q4h4KiKuj4gXIuJZYBXwnkGrfSUifhkRvwWuZf+/4RkkO4kvR8TGiHgmXe8e4N0kofVT4E5gPvAu4MGIeAp4B9AUEX8XES9FxMMkofzRzLbvjohvRcTe9H13A0dLOiwinouIH5byOYuk3gNqNvDNtMv9a5K99peBwzPrZK/meYFkTwSSPf2tmWXZ5wcy3PtlDexNbc7M2wwcWeI2Bte2ebgVhzAbeOfA7yT9vZwFDJw0/hPgNGCzpNsl/f4oatpXR0TsTWvMfqbhfjf/QNLb+r6khyVdOIrPY9VXk+1K0jRJ/5IeGnwGuAN4XRp8I9XxNeB7wNeVHHa/VFJDuux2kt7bu9Pnt5EE33vSaUh+p0cMapcXsf/vdPDvajFJb+7n6SHF00v5nEVS7wG1FfhgRLwu85gaEY+W8NrtJIcgBswatLycPfwnSfZ+ZmfmNZN0/UuxfVA9zYOWP09yqGFA9oqlrcDtg34nr46IvwCIiHsi4gzg9cC3SPYSYeTP+xiZzyNJaY0jfqZ0b3dFRPwu8CFguaSTR3qd5aZW29UK4FjgnRHxGpJAAdBIL0x7/p+NiONIjhqcDvz3dPHggLqdVwbUVuBXg36nh0bEadnNDNrmgxGxiKStrgGuU3K+eMKop4BqSE9UDjwmk5zXWCVpNoCkJklnlPh+1wKfTk+cHgl8fNDyJ0iOE49aRLycvv8qSYem9S0Hrj7wK/er7TxJx0maBnxm0PL7gP+c7hEeTbKnNeC7wJsknSOpIX28Q9JcSa+SdJak16aH6Z4B9mY+7wxJrz1ATX8o6eR0z3EFybmCu0b6MJJOl3R0Gmq/Idkb3zvCy6w66qldHUpyTujXSi76GNyuhpVeDPHmtLf1DElQDvwN30USfCcCP46ITaRHMkh6aZCcn31W0kolF5JMktQi6R0H2ObZkprSoxW/TmdPqHZTTwF1A8kf18DjYuALwHdIDh09S3Ji950lvt/fAduAXwH/D7iO5D/cAX8P/E3aHT9/DPW2kfR0Hga6SS50uKKUF0bEjcA/AreSHBq7ddAqnwdeImnsVwHXZF77LPB+kmPbj5EcslhDcrIbkuP1j6SHOJaRHP4jIn4OdAEPp5/5iEE1/QI4G1hLsif7R8AfRcRLJXykY0h+x8+RnPj954jYACDpRkkXlfAeNj7qpl2RtKmDSf5+fwjcNIrt/g7JZ3mG5JDn7SSH/UjPE98LbMq0h7uBzZFe8p6G6+kk57R+ldbwZWC4HUJILpbaJOk5kn+Tj6bnplBypeIfjKL+XMjnmitD0l+Q/AEMPmlaCJICOCYiHsq7FrNSFb1d2fiqpx5URUmaKWm+ku9QHEtyyOqbeddlNpG5XVlWTXzbOCevIvl+xxtIju9+HfjnPAsyqwFuV7aPD/GZmVkh+RCfWZVImiVpg5KhojZJ+mQ6/2Ilw0bdlz5OG+m9zOpBVXtQhx12WMyZM6dq2zMbTxs3bnwyIppKXV/STGBmRNwr6VCSoWvOJBk94bmIuKzU93JbsloyXFuq6jmoOXPm0NPTU81Nmo0bSaMZoYOI2E7yRVQi4llJfZQ+Osh+3JaslgzXlnyIzywHkuaQDMT7o3TWx5WM5H6FpOnDvGapktGpe/r7+6tVqlluHFBmVSbp1SQD8H4qIp4BvgS8keRLmNtJBjB9hYhYFxGtEdHa1FTykUWzCcsBZVZF6TBP1wPXRMQ3ACLiiXSE670kI1SfmGeNZkXhgDKrknQswU6gLyIuz8yfmVntj4HeatdmVkT+oq5Z9cwnGcvwAUn3pfMuAhZJehvJaNSPkNzczqzuOaDMqiQiuhn61gw3VLsWs4lgxEN86VVFOyT1DprfJunn6RcOLx2/Eq1UXV1dtLS0MGnSJFpaWujq6sq7JLMJyW2pGErpQV0J/BPw1YEZkhYCZwBvjYhdkl4/PuVZqbq6umhvb6ezs5MFCxbQ3d3N4sXJbZ4WLVqUc3VmE4fbUoFExIgPYA7Qm5m+FjillNdmH29/+9vDxse8efPi1ltv3W/erbfeGvPmzcupotoH9MQo20ClHm5L48dtqfqGa0slDXWUfqnwuxHRkk7fB3yb5IZYLwLnR8Q9w7x2KbAUoLm5+e2bN4/qy/dWokmTJvHiiy/S0NCwb97u3buZOnUqL7/8co6V1S5JGyOiNY9tt7a2hkeSGB9uS9U3XFsa62Xmk4FG4F3AXwPXppfQvkL4y4VVMXfuXLq7u/eb193dzdy5c3OqyGxiclsqjrEG1DbgG2nv7Mck97k/rHJl2Wi1t7ezePFiNmzYwO7du9mwYQOLFy+mvb0979LMJhS3peIY62Xm3wIWAhskvYnkJmNPVqooG72Bk7dtbW309fUxd+5cVq1a5ZO6ZqPktlQcI56DktQFvJekh/QE8Bnga8AVJGOHvURyDurWkTbm4+ZWS3wOyqwyhmtLI/agImK43Yazy67KzMxsGB6Lz8zMCskBZWZmheSAMjOzQnJAmZlZITmgzMyskBxQZmZWSA4oMzMrJAeUmZkVkgPKzMwKyQFlZjaI76hbDGMdLNbMrCb5jrrF4R6UmVnGqlWr6OzsZOHChTQ0NLBw4UI6OztZtWpV3qXVHQeUmVlGX18fCxYs2G/eggUL6Ovry6mi+uWAMjPL8B11i8MBZWaW4TvqFocvkjAzy/AddYvDAWVmNsiiRYscSAXgQ3xmZlZIDigzMyukEQNK0hWSdkjqHWLZCkkh6bDxKc9Go62tjalTpyKJqVOn0tbWlndJZmZjVkoP6krg1MEzJc0C3g9sqXBNNgZtbW10dHSwevVqnn/+eVavXk1HR4dDyswmrBEDKiLuAHYOsejzwAVAVLooG73169ezZs0ali9fzrRp01i+fDlr1qxh/fr1eZdmZjYmYzoHJekM4NGI+GkJ6y6V1COpp7+/fyybsxLs2rWLZcuW7Tdv2bJl7Nq1K6eKzMzKM+qAkjQNuAj421LWj4h1EdEaEa1NTU2j3ZyVaMqUKXR0dOw3r6OjgylTpuRUkZlZecbyPag3Am8AfioJ4CjgXkknRsTjlSzOSrdkyRJWrlwJJD2njo4OVq5c+YpelZnZRDHqgIqIB4DXD0xLegRojYgnK1iXjdLatWsBuOiii1ixYgVTpkxh2bJl++abmU00pVxm3gXcDRwraZukxeNflo3F2rVrefHFF4kIXnzxRYeTmU1oI/agIuKA431ExJyKVWNmZpbySBJmVSJplqQNkn4maZOkT6bzGyXdLOnB9Of0vGs1KwIHlFn17AFWRMRxwLuAv5R0HHAhcEtEHAPckk6b1T0HlFmVRMT2iLg3ff4s0AccCZwBXJWudhVwZi4FmhWMA8osB5LmAMcDPwIOj4jt6aLHgcOHeY2/9G51xQFlVmWSXg1cD3wqIp7JLouIYJjhw/yld6s3DqgaMmPGDCTte8yYMSPvkmwQSQ0k4XRNRHwjnf2EpJnp8pnAjrzqMysSB1SNmDFjBjt37mTevHls3ryZefPmsXPnTodUgSgZeqUT6IuIyzOLvgOcmz4/F/h2tWszKyLf8r1GDIRTb29y267e3l5aWlrYtGlTzpVZxnzgHOABSfel8y4CLgGuTb8Evxn4SD7lmRWLA6qG3HDDDa+Ynj17dk7V2GAR0Q1omMUnV7MWs4nAh/hqyGmnnXbAaTMrTVdXFy0tLUyaNImWlha6urryLqkuOaBqRGNjI5s2baKlpYUtW7bsO7zX2NiYd2lmE0pXVxft7e37xrZcu3Yt7e3tDqkcOKBqxFNPPbUvpGbPnr0vnJ566qm8SzObUFatWkVnZycLFy6koaGBhQsX0tnZyapVq/Iure74HFQNcRiZla+vr48FCxbsN2/BggX09fXlVFH9cg/KzCxj7ty5dHd37zevu7ubuXPn5lRR/XJAmZlltLe3s3jxYjZs2MDu3bvZsGEDixcvpr29Pe/S6o4P8ZmZZSxalNwCr62tjb6+PubOncuqVav2zbfqcUCZmQ2yaNEiB1IB+BCfmZkVkgPKzGyQtrY2pk6diiSmTp1KW1tb3iXVpREDStIVknZI6s3M+wdJP5d0v6RvSnrduFZpZlYlbW1tdHR0sHr1ap5//nlWr15NR0eHQyoHpfSgrgROHTTvZqAlIt4C/BL4dIXrsjHI3mpj4GFmo7N+/XrWrFnD8uXLmTZtGsuXL2fNmjWsX78+79LqzogBFRF3ADsHzft+ROxJJ38IHDUOtdkoDISRJG666ab9ps2sdLt27WLZsmX7zVu2bBm7du3KqaL6VYlzUH8G3FiB97EySWLv3r184AMfYO/evQ4nszGYMmUKHR0d+83r6OhgypQpOVVUv8oKKEntwB7gmgOss1RSj6Se/v7+cjZnI7jxxhsPOG1mI1uyZAkrV67k8ssv54UXXuDyyy9n5cqVLFmyJO/S6o4iYuSVpDnAdyOiJTPvPODPgZMj4oVSNtba2ho9PT1jq9QOaOCc0969e/fNO+igg4gISvk3ttGTtDEiWvPYttvS+Gpra2P9+vXs2rWLKVOmsGTJEtauXZt3WTVruLY0ph6UpFOBC4APlRpONv4igoMOOojvfe97+8LJzEZv4FYbEbHvlhtWfaVcZt4F3A0cK2lbelvqfwIOBW6WdJ+kjgO+iY27gTCKCE499dT9ps3MJqIRhzqKiKHG++gch1qsTA4jM6slHknCzGwQ3/K9GDxYrJlZxsAt3zs7O1mwYAHd3d0sXrwYwAPIVpl7UGZmGb7le3E4oMzMMnzL9+JwQJmZZfiW78XhgDIzy/At34vDF0nUkKHG3vOl52aj41u+F4d7UDUiG05r1qwZcr6ZlWbRokX09vby8ssv09vb63DKiQOqxkQEF1xwgXtOZjbhOaBqSLbnNNS0mZWmubl5vxt/Njc3511SXXJA1ZCVK1cecNrMRtbc3MzWrVs56aSTeOyxxzjppJPYunWrQyoHDqgaI4lLL73U557MxmggnO68805mzpzJnXfeuS+krLocUDUie84p23PyuSiz0bvuuusOOG3V4YCqIQM3J8w+zGz0PvzhDx9w2qrDAWVmljFr1izuuusu5s+fz/bt25k/fz533XUXs2bNyru0uuMv6pqZZWzZsoXm5mbuuusujjjiCCAJrS1btuRcWf1xQJmZDeIwKgYf4jMzs0JyQJlViaQrJO2Q1JuZd7GkRyXdlz5Oy7NGSzQ0NOz3Rd2Ghoa8S6pLDiiz6rkSOHWI+Z+PiLeljxuqXJMN0tDQwJ49e5g+fTr3338/06dPZ8+ePQ6pHIwYUMPs9TVKulnSg+nP6eNbppUiu8c38LDiiIg7gJ1512EHNhBOO3fu5M1vfjM7d+7cF1JWXaX0oK7klXt9FwK3RMQxwC3ptOUoG0bHH3/8kPOtsD4u6f50Z3DYnT1JSyX1SOrp7++vZn115/bbbz/gtFXHiAE1zF7fGcBV6fOrgDMrW5aNVURw7733+ku6E8eXgDcCbwO2A58bbsWIWBcRrRHR2tTUVKXy6tN73vOeA05bdYz1HNThEbE9ff44cPhwK3qvr3qyPaehpq14IuKJiHg5IvYC64ET866p3k2ePJmnn36axsZGHnjgARobG3n66aeZPNnfyqm2si+SiGRXfdjdde/1Vc9PfvKTA05b8UiamZn8Y6B3uHWtOnbv3r0vpN7ylrfsC6fdu3fnXVrdGWtAPTHQsNKfOypXkpVDEieccILPPRWQpC7gbuBYSdskLQYulfSApPuBhcBf5VqkAUlIZce0dDjlY6x91u8A5wKXpD+/XbGKbEwiYl8oZXtOPhdVHBEx1H3DO6teiNkEUcpl5kPt9V0CvE/Sg8Ap6bTlzKOZm1WGv7JRDCP2oIbZ6wM4ucK1mJnlLhtGV199NWefffa++d7pqy6PJGFmNoSI4KyzznIo5cgBZWY2yNVXX33AaasOB5SZ2SADh/WGm7bqcECZmQ1BEtdcc40vkMiRA8rMLCN7zinbc/K5qOrz2B01ZKg9PTcqs9FzuykG96BqxHCHIXx4wswmKvegakx2z8/hZDY2PhpRDO5BmZllZMPpsssuG3K+VYcDysxsCBHBihUr3HPKkQOqxnjsMLPyZXtOQ01bdTigasRwe3ne+zMbvfPPP/+A01YdDqga4tHMzSpHEp/73Od8NCJHDigzs4zsjl225+QdvurzZeZmZoM4jIrBPSgzMyskB5SZmRWSD/GZmQ3ikSSKwT0oM7OMbDh98YtfHHK+VUdZASXpryRtktQrqUvS1EoVZmaWp4igra3NPaccjTmgJB0JfAJojYgWYBLw0UoVZmaWl2zPaahpq45yD/FNBg6WNBmYBjxWfklmZvn6xCc+ccBpq44xB1REPApcBmwBtgO/iYjvD15P0lJJPZJ6+vv7x16p7Sc75t5oH2Y2MkmsXbvWbSZH5Rzimw6cAbwBOAI4RNLZg9eLiHUR0RoRrU1NTWOv1PYz1LBG2eGNRlpuZkPLtpFsz8ltp/rKOcR3CvCriOiPiN3AN4CTKlOWmVl+vGNXDOUE1BbgXZKmKekDnwz0VaYsMzOrd+Wcg/oRcB1wL/BA+l7rKlSXmVlufO62GMq6ii8iPhMRvxcRLRFxTkTsqlRhZmZ5yIbR0UcfPeR8qw4PdWRmNoTseSeHUz481JGZ2SDZntNQ01YdDigzs0EeeuihA05bdTigzMyGIIljjjnGh/dy5IAyM8vInnvK9pz8Xajq80USZmaDOIyKwT0oMzMrJAeUWZVIukLSDkm9mXmNkm6W9GD6c3qeNZoViQPKrHquBE4dNO9C4JaIOAa4JZ02MxxQZlUTEXcAOwfNPgO4Kn1+FXBmNWsyKzJfJGGWr8MjYnv6/HHg8OFWlLQUWArQ3NxchdLqw1gvI/eFFOPPPSizgojkf7xh/9fzvdXGx1jvrWbjzwFllq8nJM0ESH/uyLkes8JwQJnl6zvAuenzc4Fv51iLWaE4oMyqRFIXcDdwrKRtkhYDlwDvk/QgyV2qL8mzRrMi8UUSZlUSEYuGWXRyVQsxmyDcgzIzs0JyQJmZWSE5oMzMrJDKCihJr5N0naSfS+qT9PuVKszMzOpbuRdJfAG4KSI+LOlVwLQK1GRmZjb2gJL0WuDdwHkAEfES8FJlyjIzs3pXziG+NwD9wFck/UTSlyUdMnglSUsl9Ujq6e/vL2NzZmZWT8oJqMnACcCXIuJ44HmGuFWAxw8zM7OxKCegtgHbIuJH6fR1JIFlZmZWtjEHVEQ8DmyVdGw662TgZxWpyszM6l65V/G1AdekV/A9DPxp+SWZmZmVGVARcR/QWplSzMzM/oNHkjAzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCskBZWZmheSAMjOzQnJAFVhjYyOSRv0ARv2axsbGnD+tmdn+yh2Lz8bR008/TURUZVsDwWZmVhTuQZmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCskBZWZmheSAMjOzQnJAmVnN86gsE1PZI0lImgT0AI9GxOnll2RmVlkelWViqkQP6pNAXwXex8zMbJ+yAkrSUcAfAl+uTDlmZmaJcg/x/SNwAXDocCtIWgosBWhubi5zc/UlPvMauPi11duWmVmBjDmgJJ0O7IiIjZLeO9x6EbEOWAfQ2tpanYPANUKffaaqx83j4qpsysysJOX0oOYDH5J0GjAVeI2kqyPi7MqUZlY/JD0CPAu8DOyJiNZ8KzLL35jPQUXEpyPiqIiYA3wUuNXhZFaWhRHxNoeTWcLfgzIzs0KqSEBFxG3+DpRZWQL4vqSN6YVFryBpqaQeST39/f1VLs+s+tyDMiuGBRFxAvBB4C8lvXvwChGxLiJaI6K1qamp+hWaVZkDyqwAIuLR9OcO4JvAiflWZJY/B5RZziQdIunQgefA+4HefKsyy1/ZY/GZWdkOB76ZjuE2GfjXiLgp35LM8ueAMstZRDwMvDXvOsyKxof4zMyskBxQZmZWSA4oMzMrJJ+DKrhq3fxs+vTpVdmOmVmpHFAFNtaRzCVVbRR0s4nAt66ZmBxQZlbzfOuaicnnoMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCskBZWZmheSAMjOzQvL3oMysLnhUlolnzAElaRbwVZJ72QSwLiK+UKnCzMwqxaOyTEzl9KD2ACsi4t70bqAbJd0cET+rUG1mZlbHxnwOKiK2R8S96fNngT7gyEoVZmZm9a0iF0lImgMcD/xoiGVLJfVI6unv76/E5szMrA6UHVCSXg1cD3wqIp4ZvDwi1kVEa0S0NjU1lbs5MzOrE2UFlKQGknC6JiK+UZmSzMzMyggoJddsdgJ9EXF55UoyMzMrrwc1HzgH+E+S7ksfp1WoLjMzq3Njvsw8IrqB6nzzzczM6o6HOjIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCskBZWZmheQ76k5QI90d9EDLfQM2s/8w1rbkdjT+HFATlBuHWWW4LRWXD/GZmVkhOaDMzKyQHFBmZlZIDigzMyskB5RZAUg6VdIvJD0k6cK86zErAgeUWc4kTQL+F/BB4DhgkaTj8q3KLH8OKLP8nQg8FBEPR8RLwNeBM3KuySx3Diiz/B0JbM1Mb0vn7UfSUkk9knr6+/urVpxZXhxQZhNERKyLiNaIaG1qasq7HLNxV9WRJDZu3PikpM3V3GadOgx4Mu8i6sDsCr3Po8CszPRR6bxhuS1VjdtSdQzZluRhPmqPpJ6IaM27DiuNpMnAL4GTSYLpHuC/RcSmXAszt6WceSw+s5xFxB5JHwe+B0wCrnA4mTmgzAohIm4Absi7DrMi8UUStWld3gWY1Qi3pRz5HJSZmRWSe1BmZlZIDigzMyskB1QNkXSFpB2SevOuxWwic1sqBgdUbbkSODXvIsxqwJW4LeXOAVVDIuIOYGfedZhNdG5LxeCAMjOzQnJAmZlZITmgzMyskBxQZmZWSA6oGiKpC7gbOFbSNkmL867JbCJyWyoGD3VkZmaF5B6UmZkVkgPKzMwKyQFlZmaF5IAyM7NCckCZmVkhOaDMzKyQHFBmZlZI/x+tX3VaWh0MmAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeCElEQVR4nO3de7xVZb3v8c9XvF8QFSTk0kKj2mSJHvJ+3KapJBZmZloZut2HPEdTT+72xnZlZRZZ6tFdWpRssdySqSkpqWReslIBLyig25WigiB4R00T+Z0/xrNstFhzjQGsOedYzO/79ZqvOcYzbr+Fwm89l/E8igjMzMy6s0GzAzAzs+pzsjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMKkDSZyTd3Ow4zGpxsrCWJ+k4SQ9Kek3SUkkXSdq6js9rkxSSNuwoi4jLI+Lgej3TbF05WVhLk3Q68F3gS8DWwJ5AG3CzpI2aGJpZpThZWMuS1Bf4BvCFiLgxIt6MiIXAUcCOwKclXSrpW7lr9pe0KLe/g6SrJS2X9LikU3LHdpc0W9LLkp6RdF46dEf6flHSK5L2SrWbO3PX7i1plqSX0vfeuWO3STpL0h8krZB0s6T+6dimkn4u6TlJL6ZrB/b8n561GicLa2V7A5sC1+QLI+IVYAbQbbOQpA2AXwMPAIOBA4HTJB2STrkAuCAi+gI7AVem8v3Sd7+I2DIi/tTpvtsCNwAXAtsB5wE3SNoud9qngeOB7YGNgX9J5ePJakhD07UnAn/p9k/BrAQnC2tl/YFnI2JlF8eWAAMKrv8gMCAivhkRf42Ix4CfAEen428C75LUPyJeiYi7SsY1Fng0In4WESsj4grgYeCjuXP+MyL+OyL+QpaERuWeuR3wroh4KyLmRMTLJZ9rVpOThbWyZ4H++Y7mnEHpeHfeCeyQmntelPQi8GWgo9nnBODdwMOpOeiwknHtADzRqewJstpLh6W57deALdP2z4CbgGmSnpZ0jvterCc4WVgr+xPwBnBEvlDSlsBHgNuAV4HNc4ffkdt+Cng8IvrlPltFxKEAEfFoRBxD1lT0XeAqSVsAResCPE2WiPKGAYuLfqDU7/KNiBhJ1sx2GPC5ouvMijhZWMuKiJfIOrj/Q9IYSRtJaiNr1nkWuBy4HzhU0raS3gGclrvFPcAKSf8maTNJfSTtLOmDAJI+K2lARKwCXkzXrAKWp+8da4Q2A3i3pE9L2lDSp4CRwPVFP5OkD0l6v6Q+wMtkzVKryv6ZmNXiZGEtLSLOIWs6+j6wAnicrCbx4Yh4laxZ5wFgIXAz8IvctW+R/eY+Kl33LPBTsg5mgDHAPEmvkHV2Hx0Rf4mI14CzgT+k5qs9O8X0XLrv6cBzwL8Ch0VEUbMYZDWfq8gSxQLg9vQzmK0TeaU8s7+RdDzwTWCfiHiy2fGYVYWThVknko4F3oyIac2OxawqnCzMzKyQ+yzMzKxQV+PLe73+/ftHW1tbs8MwM+tV5syZ82xEdPky6nqZLNra2pg9e3azwzAz61UkdX4Z9G1uhjIzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0Lr5Rvc66u2iTfUPLZw0tgGRmJmraZuNQtJm0q6R9IDkuZJ+kYqHy7pbkntkn4haeNUvknab0/H23L3OiOVPyLpkHrFbGZmXatnM9QbwAERsQvZSmJj0opg3wXOj4h3AS+QLWpP+n4hlZ+fzkPSSOBo4H1kK49dlJaMNDOzBqlbsojMK2l3o/QJ4ACyZR8BpgKHp+1xaZ90/EBJSuXTIuKNiHgcaAd2r1fcZma2urp2cKcF7O8HlgEzgT8DL0bEynTKImBw2h4MPAWQjr8EbJcv7+Ka/LMmSJotafby5cvr8NOYmbWuuiaLiHgrIkYBQ8hqA++t47MmR8ToiBg9YECX07GbmdlaasjQ2Yh4EbgV2AvoJ6ljFNYQYHHaXgwMBUjHtwaey5d3cY2ZmTVAPUdDDZDUL21vBhwELCBLGkem08YD16Xt6WmfdPx3kS0QPh04Oo2WGg6MAO6pV9xmZra6er5nMQiYmkYubQBcGRHXS5oPTJP0LeA+4JJ0/iXAzyS1A8+TjYAiIuZJuhKYD6wEToqIt+oYt5mZdVK3ZBERc4Fduyh/jC5GM0XE68Ana9zrbODsno7RzMzK8XQfZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWqG7JQtJQSbdKmi9pnqRTU/nXJS2WdH/6HJq75gxJ7ZIekXRIrnxMKmuXNLFeMZuZWdc2rOO9VwKnR8S9krYC5kiamY6dHxHfz58saSRwNPA+YAfgt5LenQ7/EDgIWATMkjQ9IubXMXYzM8upW7KIiCXAkrS9QtICYHA3l4wDpkXEG8DjktqB3dOx9oh4DEDStHSuk0UPapt4Q81jCyeNbWAkZlZFDemzkNQG7ArcnYpOljRX0hRJ26SywcBTucsWpbJa5Z2fMUHSbEmzly9f3tM/gplZS6t7spC0JXA1cFpEvAxcDOwEjCKreZzbE8+JiMkRMToiRg8YMKAnbmlmZkk9+yyQtBFZorg8Iq4BiIhncsd/AlyfdhcDQ3OXD0lldFNuZmYNUM/RUAIuARZExHm58kG50z4OPJS2pwNHS9pE0nBgBHAPMAsYIWm4pI3JOsGn1ytuMzNbXT1rFvsAxwIPSro/lX0ZOEbSKCCAhcDnASJinqQryTquVwInRcRbAJJOBm4C+gBTImJeHeM2M7NO6jka6k5AXRya0c01ZwNnd1E+o7vrzMysvvwGt5mZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrFBhspB0qqS+ylwi6V5JBzciODMzq4YyNYt/SrPFHgxsQzaFx6S6RmVmZpVSJll0TNlxKPCzNC9TV9N4mJnZeqpMspgj6WayZHFTWiJ1VX3DMjOzKikzkeAJZAsVPRYRr0naDji+rlGZmVmllKlZBDASOCXtbwFsWreIzMyscsoki4uAvYBj0v4K4Id1i8jMzCqnTDPUHhGxm6T7ACLihbRinZmZtYgyNYs3JfUha45C0gDcwW1m1lLKJIsLgV8B20s6G7gT+HZdozIzs0opbIaKiMslzQEOJHu/4vCIWFD3yMzMrDJqJgtJ2+Z2lwFX5I9FxPP1DMzMzKqju5rFHLJ+iq7e1g5gx7pEZGZmlVMzWUTE8EYGYmZm1VVm6CySjgD2JatR/D4irq1nUGZmVi1lpii/CDgReBB4CDhRkl/KMzNrIWVqFgcA/xARHe9ZTAXmFV0kaShwGTCQrEYyOSIuSB3nvwDagIXAUelFPwEXkE1Y+BpwXETcm+41HvhKuvW3ImJq6Z+wwdom3lDz2MJJYxsYiZlZzynznkU7MCy3PzSVFVkJnB4RI4E9gZMkjQQmArdExAjglrQP8BFgRPpMAC6Gt0dlnQnsAewOnClpmxLPNzOzHlImWWwFLJB0m6TbgPlAX0nTJU2vdVFELOmoGUTECmABMBgYB3TUDKYCh6ftccBlkbkL6CdpEHAIMDMino+IF4CZwJg1/DnNzGwdlGmG+tq6PkRSG7ArcDcwMCKWpENLyZqpIEskT+UuW5TKapWbmVmDlHmD+3YASX3z55d9KU/SlsDVwGkR8XLWNfH2PUJSrGnQNZ4zgaz5imHDhhWcbWZma6LMaKgJkpYCc4HZZC/rzS5zc0kbkSWKyyPimlT8TGpeIn0vS+WLyfpDOgxJZbXK/05ETI6I0RExesCAAWXCMzOzksr0WXwJ2Dki2iJix4gYHhGFb2+n0U2XAAsi4rzcoenA+LQ9HrguV/45ZfYEXkrNVTcBB0vaJnVsH5zKzMysQcr0WfyZbCjrmtoHOBZ4UNL9qezLwCTgSkknAE8AR6VjM8iGzban5x0PWXOXpLOAWem8b3peKjOzxiqTLM4A/ijpbuCNjsKIOKX2JRARd9L1vFKQzWDb+fwATqpxrynAlBKxmplZHZRJFj8Gfkf2BrcXPTIza0FlksVGEfHFukdiZmaVVaaD+zdpRNQgSdt2fOoemZmZVUaZmsUx6fuMXJnXszAzayFlXsrzuhZmZi2u7HoWOwMjgU07yiLisnoFZWZm1VKYLCSdCexPlixmkM0OeyfZ9ONmZtYCynRwH0n2XsTSiDge2AXYuq5RmZlZpZRJFn+JiFXAyjSZ4DL+fq4mMzNbz5Xps5gtqR/wE7JJBF8B/lTPoMzMrFrKjIb6P2nzR5JuBPpGxNz6hmVmZlVSZoryfSRtkXb3BY6T9M76hmVmZlVSps/iYuA1SbsAp5PNQuuRUGZmLaRMsliZZoQdB/wgIn5Iti63mZm1iDId3CsknQF8FthP0gbARvUNy8zMqqRMzeJTZOtYnBARS8mWNf1eXaMyM7NKKTMaailwXm7/SdxnYWbWUsrULMzMrMU5WZiZWaGayULSLen7u40Lx8zMqqi7PotBkvYGPiZpGqD8wYi4t66RmZlZZXSXLL4GfJVs9NN5nY4FcEC9grLepW3iDTWPLZw0toGRmFm91EwWEXEVcJWkr0bEWQ2MyczMKqbM0NmzJH0M2C8V3RYR19c3LDMzq5IyEwl+BzgVmJ8+p0r6dr0DMzOz6igzdHYscFBETImIKcAY4LCiiyRNkbRM0kO5sq9LWizp/vQ5NHfsDEntkh6RdEiufEwqa5c0cc1+PDMz6wll37Pol9suu6TqpWSJpbPzI2JU+swAkDQSOBp4X7rmIkl9JPUBfki27vdI4Jh0rpmZNVCZiQS/A9wn6Vay4bP7AYW/4UfEHZLaSsYxDpgWEW8Aj0tqB3ZPx9oj4jGANIR3HFlzmJmZNUhhzSIirgD2BK4Brgb2iohfrMMzT5Y0NzVTbZPKBgNP5c5ZlMpqla9G0gRJsyXNXr58+TqEZ2ZmnZVqhoqIJRExPX2WrsPzLgZ2AkYBS4Bz1+FefyciJkfE6IgYPWDAgJ66rZmZUa4ZqsdExDMd25J+AnQMwV0MDM2dOiSV0U25mZk1SEMnEpQ0KLf7caBjpNR04GhJm0gaDowA7gFmASMkDZe0MVkn+PRGxmxmZgU1izQaaV5EvHdNbyzpCmB/oL+kRcCZwP6SRpFNF7IQ+DxARMyTdCVZx/VK4KSIeCvd52TgJqAPMCUi5q1pLGZmtm66TRYR8VZ6x2FYWvSotIg4poviS7o5/2zg7C7KZwAz1uTZZmbWs8r0WWwDzJN0D/BqR2FEfKxuUZmZWaWUSRZfrXsUZmZWaWUmErxd0juBERHxW0mbk/UfmJlZiygzkeD/Aq4CfpyKBgPX1jEmMzOrmDJDZ08C9gFeBoiIR4Ht6xmUmZlVS5lk8UZE/LVjR9KGZENfzcysRZRJFrdL+jKwmaSDgF8Cv65vWGZmViVlksVEYDnwINlLdDOAr9QzKDMzq5Yyo6FWSZoK3E3W/PRIRLgZysyshRQmC0ljgR8BfyZbz2K4pM9HxG/qHZyZmVVDmZfyzgU+FBHtAJJ2Am4AnCzMzFpEmT6LFR2JInkMWFGneMzMrIJq1iwkHZE2Z0uaAVxJ1mfxSbKpw83MrEV01wz10dz2M8A/pu3lwGZ1i8jMzCqnZrKIiOMbGYiZmVVXmdFQw4EvAG358z1FuZlZ6ygzGupaskWLfg2sqms0ZmZWSWWSxesRcWHdIzEzs8oqkywukHQmcDPwRkdhRNxbt6jMzKxSyiSL9wPHAgfwt2aoSPtmZtYCyiSLTwI75qcpNzOz1lLmDe6HgH51jsPMzCqsTM2iH/CwpFn8fZ+Fh86ambWIMsnizLpHYWZmlVZmPYvbGxGImZlVV2GfhaQVkl5On9clvSXp5RLXTZG0TNJDubJtJc2U9Gj63iaVS9KFktolzZW0W+6a8en8RyWNX9sf1MzM1l5hsoiIrSKib0T0JZtA8BPARSXufSkwplPZROCWiBgB3JL2AT4CjEifCcDFkCUXsmawPYDdgTM7EoyZmTVOmdFQb4vMtcAhJc69A3i+U/E4YGrangocniu/LN3/LqCfpEHpOTMj4vmIeAGYyeoJyMzM6qzMRIJH5HY3AEYDr6/l8wZGxJK0vRQYmLYHA0/lzluUymqVdxXnBLJaCcOGDVvL8MzMrCtlRkPl17VYCSwkqwmsk4gISbGu98ndbzIwGWD06NE9dl8zMys3Gqon17V4RtKgiFiSmpmWpfLFwNDceUNS2WJg/07lt/VgPGZmVkJ3y6p+rZvrIiLOWovnTQfGA5PS93W58pMlTSPrzH4pJZSbgG/nOrUPBs5Yi+eamdk66K5m8WoXZVsAJwDbAd0mC0lXkNUK+ktaRDaqaRJwpaQTgCeAo9LpM4BDgXbgNeB4gIh4XtJZ/G3N729GROdOc1uPtU28oeaxhZPGNjASs9bW3bKq53ZsS9oKOJXsH/FpwLm1rstdf0yNQwd2cW4AJ9W4zxRgStHzzMysfrrts0jvOXwR+AzZUNfd0hBWMzNrId31WXwPOIJshNH7I+KVhkVlZmaV0t1LeacDOwBfAZ7OTfmxosx0H2Zmtv7ors9ijd7uNjOz9ZcTgpmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWqOYa3Gbrs7aJN3R7fOGksQ2KxKx3cM3CzMwKNSVZSFoo6UFJ90uancq2lTRT0qPpe5tULkkXSmqXNFfSbs2I2cyslTWzZvGhiBgVEaPT/kTglogYAdyS9gE+AoxInwnAxQ2P1MysxVWpGWocMDVtTwUOz5VfFpm7gH6SBjUhPjOzltWsDu4AbpYUwI8jYjIwMCKWpONLgYFpezDwVO7aRalsSa4MSRPIah4MGzZsnYLrrvPTHZ9m1oqalSz2jYjFkrYHZkp6OH8wIiIlktJSwpkMMHr06DW61szMuteUZqiIWJy+lwG/AnYHnuloXkrfy9Lpi4GhucuHpDIzM2uQhicLSVtI2qpjGzgYeAiYDoxPp40Hrkvb04HPpVFRewIv5ZqrzMysAZrRDDUQ+JWkjuf/V0TcKGkWcKWkE4AngKPS+TOAQ4F24DXg+MaHbGbW2hqeLCLiMWCXLsqfAw7sojyAkxoQmpmZ1VClobNmZlZRThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskFfKM1sLnmzSWo1rFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCnm6D7MK6W4aEfBUItY8rlmYmVkhJwszMyvkZGFmZoWcLMzMrJA7uM1ahDvPbV24ZmFmZoV6TbKQNEbSI5LaJU1sdjxmZq2kVzRDSeoD/BA4CFgEzJI0PSLmNzcys9bhpWRbW69IFsDuQHtEPAYgaRowDnCyMOvl6tmX4n6anqOIaHYMhSQdCYyJiH9O+8cCe0TEyblzJgAT0u57gEcaHmht/YFnmx1EgarHWPX4oPoxVj0+qH6MVY8P1i3Gd0bEgK4O9JaaRaGImAxMbnYcXZE0OyJGNzuO7lQ9xqrHB9WPserxQfVjrHp8UL8Ye0sH92JgaG5/SCozM7MG6C3JYhYwQtJwSRsDRwPTmxyTmVnL6BXNUBGxUtLJwE1AH2BKRMxrclhropLNY51UPcaqxwfVj7Hq8UH1Y6x6fFCnGHtFB7eZmTVXb2mGMjOzJnKyMDOzQk4WdSRpqKRbJc2XNE/Sqc2OqSuS+ki6T9L1zY6lK5L6SbpK0sOSFkjaq9kx5Un6v+m/70OSrpC0aQVimiJpmaSHcmXbSpop6dH0vU0FY/xe+u88V9KvJPWrUny5Y6dLCkn9mxFbLo4uY5T0hfTnOE/SOT3xLCeL+loJnB4RI4E9gZMkjWxyTF05FVjQ7CC6cQFwY0S8F9iFCsUqaTBwCjA6InYmG4BxdHOjAuBSYEynsonALRExArgl7TfTpawe40xg54j4APDfwBmNDirnUlaPD0lDgYOBJxsdUBcupVOMkj5ENsPFLhHxPuD7PfEgJ4s6ioglEXFv2l5B9o/c4OZG9fckDQHGAj9tdixdkbQ1sB9wCUBE/DUiXmxqUKvbENhM0obA5sDTTY6HiLgDeL5T8ThgatqeChzeyJg66yrGiLg5Ilam3bvI3qlqihp/hgDnA/8KNH10UI0Y/zcwKSLeSOcs64lnOVk0iKQ2YFfg7iaH0tn/I/sff1WT46hlOLAc+M/UVPZTSVs0O6gOEbGY7De3J4ElwEsRcXNzo6ppYEQsSdtLgYHNDKaEfwJ+0+wg8iSNAxZHxAPNjqUb7wb+p6S7Jd0u6YM9cVMniwaQtCVwNXBaRLzc7Hg6SDoMWBYRc5odSzc2BHYDLo6IXYFXaX7zydtSu/84sqS2A7CFpM82N6pikY2Zb/pvxrVI+neyZtzLmx1LB0mbA18GvtbsWApsCGxL1vT9JeBKSVrXmzpZ1JmkjcgSxeURcU2z4+lkH+BjkhYC04ADJP28uSGtZhGwKCI6amRXkSWPqvgw8HhELI+IN4FrgL2bHFMtz0gaBJC+e6R5oqdJOg44DPhMVOtFsJ3Ifil4IP2dGQLcK+kdTY1qdYuAayJzD1mrwTp3xDtZ1FHK5pcACyLivGbH01lEnBERQyKijaxT9ncRUanfiiNiKfCUpPekogOp1tT0TwJ7Sto8/fc+kAp1wHcyHRiftscD1zUxli5JGkPWLPqxiHit2fHkRcSDEbF9RLSlvzOLgN3S/6NVci3wIQBJ7wY2pgdmynWyqK99gGPJfmO/P30ObXZQvdAXgMslzQVGAd9ubjh/k2o8VwH3Ag+S/Z1q+pQQkq4A/gS8R9IiSScAk4CDJD1KViOaVMEYfwBsBcxMf19+VLH4KqVGjFOAHdNw2mnA+J6ooXm6DzMzK+SahZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwvr1SS9Uuf7n5be3F3n50naRNJv05DQT/VMhGsVx3GSftCs51vv5GRh1r3TyCYH7Am7AkTEqIj4RQ/d06whnCxsvSNpJ0k3Spoj6feS3pvKL5V0oaQ/SnpM0pGpfANJF6X5/2dKmiHpSEmnkM33dKukW3P3P1vSA5LukrTaZHxp3Yhr05oMd0n6gKTtgZ8DH0w1i506XXOKsnVP5kqalsp2l/SnNIHiHzveYk81g2tTrAslnSzpi+m8uyRtm867TdIF6XkPSdq9i1gHSLpa0qz02SeV/2PuRdL7JG3VM/91rNeKCH/86bUf4JUuym4BRqTtPcimMYFs7v9fkv2SNBJoT+VHAjNS+TuAF4Aj07GFQP/cvQP4aNo+B/hKF8//D+DMtH0AcH/a3h+4vsbP8TSwSdrul777Ahum7Q8DV6ft44B2sjedBwAvASemY+eTTVgJcBvwk7S9H/BQ7vofpO3/AvZN28PIpqYB+DWwT9resiMOf1r3s+Ea5BWzyksz/O4N/DI30eYmuVOujYhVwPxcrWBf4JepfGm+FtGFvwIdKwrOAQ7q4px9gU8ARMTvJG0nqW9B6HPJpjS5lmxuH4CtgamSRpAlqY1y598a2RopKyS9RPaPO2RTjnwgd94VKY47JPXV6ivPfRgYmfuz6pv+DP8AnCfpcrJJ6RYVxG/rOScLW99sALwYEaNqHH8jt7020za/GREdc+S8Rc/9HRpL9tv/R4F/l/R+4CyypPBxZeuh3JY7P/9zrMrtr+oUU+f5fDrvbwDsGRGvdyqfJOkG4FDgD5IOiYiH1+xHsvWJ+yxsvRLZeiGPS/okZDP/Stql4LI/AJ9IfRcDyZqLOqwga+5ZE78HPpOevz/wbHSzjomkDYChEXEr8G9kNYot0/fidNpxaxhDh0+lZ+xLtjDTS52O30w2UWNHLKPS906RzbL6XWAW8N61fL6tJ5wsrLfbPM222fH5Itk/1CdIegCYR7Y4UXeuJptuej5ZJ/S9ZP0AkM0ge2NB01RnXwf+h7JZcifxt2nBa+kD/FzSg8B9wIWRLR17DvAdSfex9jWY19P1PwK6mjX1FGB06lifD5yYyk9LneJzgTep2Ip11nieddaMrK8jIl6RtB1wD1nnbtXWKVgjkm4D/iUiZjc7Fuv93Gdhlrk+df5uDJzV2xOFWU9zzcLMzAq5z8LMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMys0P8HTATPVR3kfGYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEWCAYAAACe8xtsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcY0lEQVR4nO3de7RdZXnv8e8v4aICIQFCGpJAIsZq1BJoBFpygGK5Dw0qIBwrgaaNjkKBoafH4FGh0misAi1y0TDIIVgkRhCIEIVwSQBbIAlgyAUO2xCaxNy4BxE05Dl/zHfJZLP32nMme+619l6/zxhrrDmfeXv2HIs8zPed852KCMzMzMro1+gEzMys93HxMDOz0lw8zMysNBcPMzMrzcXDzMxKc/EwM7PSXDzMzKw0Fw+zTkiaL+lFSTs3OhezZuPiYdYBSSOB/wEE8InGZtMxSTs0OgdrXS4eZh07A3gIuA6YWAtKuk7SlZLukLRZ0sOS9k/LJOkySRslvSLpCUkfljRK0kuS+qX1rpG0MbfPH0o6P03vLulaSeskrZX0L5L6p2VnSvplOsbzwEWS3idpgaSXJT0n6cc9dYKstbl4mHXsDOCG9DlW0pDcstOAfwYGAW3A1BQ/BjgceD+wO3Aq8HxEPAO8AhyY1jsceFXSB9P8EcCCNH0dsAV4X1r/GODvcsc+BFgJDEnHvRi4K+UyHPje9v3ZZsW4eJi1I2k8sB8wOyIWA78G/mdulVsi4pGI2EJWXMam+B+A3YAPAIqIFRGxLi1bABwh6U/S/E1pfhQwAPhVKlAnAOdHxG8jYiNwGVmxqvlNRHwvIrZExO/SMfcD9omI1yPiwe48F2adcfEwe6eJwF0R8Vya/xG5pitgfW76NWBXgIi4F7gCuBLYKGm6pAFpvQXAkWRXHfcD88muOI4AHoiIrWRFYEdgXWrmegn4AbB37nir2+X6vwEBj0haJulvt/FvNivFHW5mOZLeTdbc1F9SrUjsDAyUdEBX20fE5cDlkvYGZgP/BHyNrHh8B1iTph8Evg+8zltNVquBN4C90lVNh4dod7z1wN+n3McDd0u6PyLaiv3FZtvGVx5mb3cS8CYwhqw5aizwQeABsn6QTkn6qKRDJO0I/JasMGwFiIingd8BfwMsiIhXgA3Ap0nFIzVx3QVcImmApH6S9pd0RJ1jniJpeJp9kay4bC3/Z5uV4+Jh9nYTgf8bEf8dEetrH7LmqM9S/2p9AHAN2T/izwLPk11t1Cwg60BfnZsX8GhunTOAnYDlaT83AUPrHPOjwMOSXgXmAOdFxMpCf6nZdpBfBmVmZmX5ysPMzEpz8TAzs9JcPMzMrDQXDzMzK61PPuex1157xciRIxudhplZr7J48eLnImJwkXX7ZPEYOXIkixYtanQaZma9iqRni67rZiszMyvNxcPMzEpz8TAzs9JcPMzMrDQXDzMzK83Fw8zMSnPxMDOz0lw8zMysNBcPMzMrrU8+Yd5XjZxyR6fLVk07sQczMbNW5ysPMzMrrbLiIeldkh6R9CtJyyT9c4qPkvSwpDZJP5a0U4rvnObb0vKRuX1dkOJPSTq2qpzNzKyYKq883gCOiogDgLHAcZIOBb4NXBYR7yN7R/OktP4k4MUUvyyth6QxwGnAh4DjgKsk9a8wbzMz60JlxSMyr6bZHdMngKOAm1J8JnBSmp6Q5knLPyZJKT4rIt6IiGeANuDgqvI2M7OuVdrnIam/pMeBjcA84NfASxGxJa2yBhiWpocBqwHS8peBPfPxDrbJH2uypEWSFm3atKmCv8bMzGoqLR4R8WZEjAWGk10tfKDCY02PiHERMW7w4ELvMjEzs23UI3dbRcRLwH3AXwADJdVuER4OrE3Ta4ERAGn57sDz+XgH25iZWQNUebfVYEkD0/S7gaOBFWRF5OS02kTgtjQ9J82Tlt8bEZHip6W7sUYBo4FHqsrbzMy6VuVDgkOBmenOqH7A7Ii4XdJyYJakfwEeA65N618L/FBSG/AC2R1WRMQySbOB5cAW4OyIeLPCvM3MrAuVFY+IWAIc2EF8JR3cLRURrwOndLKvqcDU7s7RzMy2jZ8wNzOz0lw8zMysNBcPMzMrzcXDzMxKc/EwM7PSXDzMzKw0Fw8zMyvNxcPMzEpz8TAzs9JcPMzMrDQXDzMzK83Fw8zMSnPxMDOz0lw8zMysNBcPMzMrzcXDzMxKc/EwM7PSXDzMzKw0Fw8zMyvNxcPMzEpz8TAzs9JcPMzMrDQXDzMzK83Fw8zMSnPxMDOz0iorHpJGSLpP0nJJyySdl+IXSVor6fH0OSG3zQWS2iQ9JenYXPy4FGuTNKWqnM3MrJgdKtz3FuBLEfGopN2AxZLmpWWXRcR38ytLGgOcBnwI2Ae4W9L70+IrgaOBNcBCSXMiYnmFuZuZWR2VFY+IWAesS9ObJa0AhtXZZAIwKyLeAJ6R1AYcnJa1RcRKAEmz0rouHmZmDdIjfR6SRgIHAg+n0DmSlkiaIWlQig0DVuc2W5NincXbH2OypEWSFm3atKm7/wQzM8upvHhI2hW4GTg/Il4Brgb2B8aSXZlc0h3HiYjpETEuIsYNHjy4O3ZpZmadqLLPA0k7khWOGyLipwARsSG3/Brg9jS7FhiR23x4ilEn3quMnHJH3eWrpp3YQ5mYmW2fKu+2EnAtsCIiLs3Fh+ZW+ySwNE3PAU6TtLOkUcBo4BFgITBa0ihJO5F1qs+pKm8zM+talVcehwGfA56Q9HiKfQU4XdJYIIBVwOcBImKZpNlkHeFbgLMj4k0ASecAdwL9gRkRsazCvM3MrAtV3m31IKAOFs2ts81UYGoH8bn1tjMzs57lJ8zNzKw0Fw8zMyut0rutrOf4Ti4z60m+8jAzs9JcPMzMrDQXDzMzK83Fw8zMSuuyeEg6T9IAZa6V9KikY3oiOTMza05Frjz+Ng1oeAwwiOyp8WmVZmVmZk2tSPGoPSV+AvDDNDRIR0+Om5lZiyhSPBZLuouseNyZ3gq4tdq0zMysmRV5SHAS2bs3VkbEa5L2BM6qNCszM2tqRa48AhgDnJvmdwHeVVlGZmbW9IoUj6uAvwBOT/ObgSsry8jMzJpekWarQyLiIEmPAUTEi+mlTGZm1qKKXHn8QVJ/suYrJA3GHeZmZi2tSPG4HLgF2FvSVOBB4JuVZmVmZk2ty2ariLhB0mLgY2TPd5wUESsqz8zMzJpWp8VD0h652Y3AjfllEfFClYmZmVnzqnflsZisn6Ojp8kDeG8lGZmZWdPrtHhExKieTMTMzHqPQq+hlfQpYDzZFccDEXFrlUmZmVlzKzIk+1XAF4AngKXAFyT5IUEzsxZW5MrjKOCDEVF7zmMmsKzSrMzMrKkVec6jDdg3Nz8ixeqSNELSfZKWS1om6bwU30PSPElPp+9BKS5Jl0tqk7RE0kG5fU1M6z8taWK5P9HMzLpbkeKxG7BC0nxJ84HlwABJcyTNqbPdFuBLETEGOBQ4W9IYYApwT0SMBu5J8wDHA6PTZzJwNfzxluELgUOAg4ELawXHzMwao0iz1de3ZccRsQ5Yl6Y3S1oBDAMmAEem1WYC84Evp/j1qXnsIUkDJQ1N686rPVciaR5wHLnnTszMrGcVecJ8AYCkAfn1yzwkKGkkcCDwMDAkFRaA9cCQND0MWJ3bbE2KdRY3M7MG6bJ4SJoMfAN4nWxARFHiIUFJuwI3A+dHxCvSW88cRkRIim3Iu7M8JwPsu+++XaxtZmbbo0ifxz8BH46IkRHx3ogYFRFFC8eOZIXjhoj4aQpvSM1RpO+NKb6WrDO+ZniKdRZ/m4iYHhHjImLc4MGDi6RnZmbbqEjx+DXwWtkdK7vEuBZYERGX5hbNAWp3TE0EbsvFz0h3XR0KvJyat+4EjpE0KHWUH5NiZmbWIEU6zC8A/lPSw8AbtWBEnNv5JgAcBnwOeELS4yn2FWAaMFvSJOBZ4NS0bC5wAtltwK+R3pMeES9IuhhYmNb7hgdlNDNrrCLF4wfAvWRPmBd+CVREPEjHgypCNrx7+/UDOLuTfc0AZhQ9tpmZVatI8dgxIr5YeSZmZtZrFOnz+LmkyZKGpqfD92j3rg8zM2sxRa48Tk/fF+Rifp+HmVkLK/KQoN/rYWZmb1P0fR4fBsYA76rFIuL6qpIyM7PmVuQJ8wvJxpcaQ3Y77fHAg4CLh5lZiyrSYX4y2a216yPiLOAAYPdKszIzs6ZWpHj8LiK2AlvS4IgbeftwIWZm1mKK9HkskjQQuAZYDLwK/FeVSZmZWXMrcrfVP6TJ70v6BTAgIpZUm5aZmTWzLputJB0maZc0Ox44U9J+1aZlZmbNrEifx9XAa5IOAL5ENsqu77QyM2thRYrHljRo4QTgioi4kuy95mZm1qKKdJhvlnQB8DfA4ZL6ATtWm5aZmTWzIlcenyF7j8ekiFhP9ia/71SalZmZNbUid1utBy7Nzf837vMwM2tpRa48zMzM3sbFw8zMSuu0eEi6J31/u+fSMTOz3qBen8dQSX8JfELSLNq9jzwiHq00MzMza1r1isfXga+R3V11abtlARxVVVJmZtbcOi0eEXETcJOkr0XExT2Yk5mZNbkit+peLOkTwOEpND8ibq82LTMza2ZFBkb8FnAesDx9zpP0zaoTMzOz5lVkeJITgbHphVBImgk8BnylysTMzKx5FX3OY2BuutAraCXNkLRR0tJc7CJJayU9nj4n5JZdIKlN0lOSjs3Fj0uxNklTCuZrZmYVKnLl8S3gMUn3kd2uezhQ5B/x64AreOdQJpdFxHfzAUljgNOADwH7AHdLen9afCVwNLAGWChpTkQsL3B8MzOrSJEO8xslzQc+mkJfTuNddbXd/ZJGFsxjAjArIt4AnpHUBhyclrVFxEqA9LzJBLK+FzMza5BCzVYRsS4i5qRPl4WjC+dIWpKatQal2DBgdW6dNSnWWfwdJE2WtEjSok2bNm1nimZmVk9Pj211NbA/MBZYB1zSXTuOiOkRMS4ixg0ePLi7dmtmZh0o0ufRbSJiQ21a0jVA7XmRtcCI3KrDU4w6cTMza5C6Vx6S+kt6srsOJmlobvaTQO1OrDnAaZJ2ljQKGA08AiwERksaJWknsk71Od2Vj5mZbZu6Vx4R8Wa6TXbf9BKowiTdCBwJ7CVpDXAhcKSksWRjY60CPp+Os0zSbLKO8C3A2RHxZtrPOcCdQH9gRkQsK5OHmZl1vyLNVoOAZZIeAX5bC0bEJ+ptFBGndxC+ts76U4GpHcTnAnML5GlmZj2kSPH4WuVZWMONnHJHp8tWTTuxBzMxs96gyHMeCyTtB4yOiLslvYesCcnMzFpUkYER/x64CfhBCg0Dbq0wJzMza3JFnvM4GzgMeAUgIp4G9q4yKTMza25FiscbEfH72oykHcjuljIzsxZVpHgskPQV4N2SjgZ+Avys2rTMzKyZFSkeU4BNwBNkz2XMBb5aZVJmZtbcitxttTW9AOphsuaqpyLCzVZmZi2sy+Ih6UTg+8Cvyd7nMUrS5yPi51UnZ2ZmzanIQ4KXAH8VEW0AkvYH7gBcPMzMWlSRPo/NtcKRrAQ2V5SPmZn1Ap1eeUj6VJpcJGkuMJusz+MUstFuzcysRdVrtvp4bnoDcESa3gS8u7KMzMys6XVaPCLirJ5MxMzMeo8id1uNAv4RGJlfv6sh2c3MrO8qcrfVrWTv4fgZsLXSbMzMrFcoUjxej4jLK8/EzMx6jSLF498lXQjcBbxRC0bEo5VlZWZmTa1I8fgI8DngKN5qtoo0b2ZmLahI8TgFeG9+WHYzM2ttRZ4wXwoMrDgPMzPrRYpceQwEnpS0kLf3efhWXTOzFlWkeFxYeRZmZtarFHmfx4KeSMTMzHqPLvs8JG2W9Er6vC7pTUmvFNhuhqSNkpbmYntImifp6fQ9KMUl6XJJbZKWSDoot83EtP7TkiZu6x9qZmbdp8viERG7RcSAiBhANiDip4GrCuz7OuC4drEpwD0RMRq4J80DHA+MTp/JwNWQFRuyZrNDgIOBC2sFx8zMGqfI3VZ/FJlbgWMLrHs/8EK78ARgZpqeCZyUi1+f9v8QMFDS0HSceRHxQkS8CMzjnQXJzMx6WJGBET+Vm+0HjANe38bjDYmIdWl6PTAkTQ8DVufWW5NincXNzKyBitxtlX+vxxZgFdmVwnaJiJAU27ufGkmTyZq82Hfffbtrt2Zm1oEid1t153s9NkgaGhHrUrPUxhRfC4zIrTc8xdYCR7aLz+8kz+nAdIBx48Z1W1EyM7N3qvca2q/X2S4i4uJtON4cYCIwLX3floufI2kWWef4y6nA3Al8M9dJfgxwwTYc18zMulG9K4/fdhDbBZgE7AnULR6SbiS7athL0hqyu6amAbMlTQKeBU5Nq88FTgDagNeAswAi4gVJF/PWO9O/ERHtO+HNzKyH1XsN7SW1aUm7AeeR/aM+C7iks+1y25/eyaKPdbBuAGd3sp8ZwIyujmdmZj2nbp9Hes7ii8BnyW6tPSjdMmtmZi2sXp/Hd4BPkXVCfyQiXu2xrMzMrKnVe0jwS8A+wFeB3+SGKNlcZHgSMzPru+r1eZR6+tzMzFqHC4SZmZXm4mFmZqUVGZ7ErK6RU+7odNmqaSf2YCZm1lN85WFmZqW5eJiZWWkuHmZmVpqLh5mZlebiYWZmpfluq5Lq3VkEvrvIzFqDrzzMzKw0Fw8zMyvNxcPMzEpz8TAzs9JcPMzMrDQXDzMzK83Fw8zMSnPxMDOz0lw8zMysNBcPMzMrzcXDzMxKc/EwM7PSGlI8JK2S9ISkxyUtSrE9JM2T9HT6HpTiknS5pDZJSyQd1IiczczsLY288viriBgbEePS/BTgnogYDdyT5gGOB0anz2Tg6h7P1MzM3qaZmq0mADPT9EzgpFz8+sg8BAyUNLQB+ZmZWdKo4hHAXZIWS5qcYkMiYl2aXg8MSdPDgNW5bdek2NtImixpkaRFmzZtqipvMzOjcS+DGh8RayXtDcyT9GR+YUSEpCizw4iYDkwHGDduXKltzcysnIZceUTE2vS9EbgFOBjYUGuOSt8b0+prgRG5zYenmJmZNUiPFw9Ju0jarTYNHAMsBeYAE9NqE4Hb0vQc4Ix019WhwMu55i0zM2uARjRbDQFukVQ7/o8i4heSFgKzJU0CngVOTevPBU4A2oDXgLN6PmUzM8vr8eIRESuBAzqIPw98rIN4AGf3QGpmZlZQM92qa2ZmvUSj7rYyA2DklDvqLl817cQeysTMyvCVh5mZlebiYWZmpbl4mJlZaS4eZmZWmouHmZmV5uJhZmaluXiYmVlpLh5mZlaai4eZmZXm4mFmZqV5eBJravWGL/HQJWaN4ysPMzMrzcXDzMxKc/EwM7PSXDzMzKw0Fw8zMyvNxcPMzEpz8TAzs9JcPMzMrDQ/JGh9lh8wNKuOi4dZB+oVHnDxMXOzlZmZlebiYWZmpfWa4iHpOElPSWqTNKXR+ZiZtbJe0echqT9wJXA0sAZYKGlORCyv4nhdtXeb1bM9v5+u+lLcF2PNolcUD+BgoC0iVgJImgVMACopHmZ90fYWHt+9ZnmKiEbn0CVJJwPHRcTfpfnPAYdExDm5dSYDk9PsnwJPAXsBz/Vwus3K5yLj85Dxecj4PGRq52G/iBhcZIPecuXRpYiYDkzPxyQtiohxDUqpqfhcZHweMj4PGZ+HzLach97SYb4WGJGbH55iZmbWAL2leCwERksaJWkn4DRgToNzMjNrWb2i2Soitkg6B7gT6A/MiIhlBTad3vUqLcPnIuPzkPF5yPg8ZEqfh17RYW5mZs2ltzRbmZlZE3HxMDOz0vps8fBwJhlJqyQ9IelxSYsanU9PkjRD0kZJS3OxPSTNk/R0+h7UyBx7Qifn4SJJa9Pv4nFJJzQyx54gaYSk+yQtl7RM0nkp3lK/iTrnodRvok/2eaThTP4fueFMgNOrGs6kmUlaBYyLiJZ7EErS4cCrwPUR8eEU+1fghYiYlv6nYlBEfLmReVatk/NwEfBqRHy3kbn1JElDgaER8aik3YDFwEnAmbTQb6LOeTiVEr+Jvnrl8cfhTCLi90BtOBNrIRFxP/BCu/AEYGaankn2H02f1sl5aDkRsS4iHk3Tm4EVwDBa7DdR5zyU0leLxzBgdW5+DdtwcvqIAO6StDgN4dLqhkTEujS9HhjSyGQa7BxJS1KzVp9uqmlP0kjgQOBhWvg30e48QInfRF8tHvaW8RFxEHA8cHZqwjAgsjbbvtduW8zVwP7AWGAdcElDs+lBknYFbgbOj4hX8sta6TfRwXko9Zvoq8XDw5kkEbE2fW8EbiFr0mtlG1Kbb63td2OD82mIiNgQEW9GxFbgGlrkdyFpR7J/MG+IiJ+mcMv9Jjo6D2V/E321eHg4E0DSLqlDDEm7AMcAS+tv1efNASam6YnAbQ3MpWFq/1gmn6QFfheSBFwLrIiIS3OLWuo30dl5KPub6JN3WwGk28z+jbeGM5na2Ix6nqT3kl1tQDYUzY9a6TxIuhE4kmy46Q3AhcCtwGxgX+BZ4NSI6NOdyZ2chyPJmicCWAV8Ptfu3ydJGg88ADwBbE3hr5C197fMb6LOeTidEr+JPls8zMysOn212crMzCrk4mFmZqW5eJiZWWkuHmZmVpqLh5mZlebiYb2apFcr3v/5kt7THceTtLOku9OIpZ/pngy3KY8zJV3RqONb3+DiYVbf+cB7ulqpoAMBImJsRPy4m/Zp1hAuHtbnSNpf0i/SYJAPSPpAil8n6XJJ/ylppaSTU7yfpKskPZne5zBX0smSzgX2Ae6TdF9u/1Ml/UrSQ5LeMYheej/ErWmAuYck/ZmkvYH/AD6arjz2b7fNuen9CkskzUqxgyX9l6THUs5/muJnpv3PU/a+lnMkfTGt95CkPdJ68yX9ezreUknvGG5C0mBJN0tamD6HpfgRufc6PFYbqcDsjyLCH3967Yfs/QPtY/cAo9P0IcC9afo64Cdk/9M0hmzYfoCTgbkp/ifAi8DJadkqYK/cvgP4eJr+V+CrHRz/e8CFafoo4PE0fSRweyd/x2+AndP0wPQ9ANghTf81cHOaPhNoA3YDBgMvA19Iyy4jG+gOYD5wTZo+HFia2/6KNP0jssEzIXvCekWa/hlwWJretZaHP/7UPjuUqDNmTS+NFPqXwE+yIXwA2Dm3yq2RDfy2PHfVMB74SYqvz19ldOD3wO1pejHZC8faGw98GiAi7pW0p6QBXaS+BLhB0q1kQ6gA7A7MlDSarGjtmFv/vsjexbBZ0stk/9hDNuTEn+XWuzHlcb+kAZIGtjvuXwNjcudqQDqHvwQulXQD8NOIWNNF/tZiXDysr+kHvBQRYztZ/kZuWp2sU88fIqI2ps+bdN9/QyeSXR18HPg/kj4CXExWJD6Z3rswP7d+/u/Ympvf2i6n9uMPtZ/vBxwaEa+3i0+TdAdwAvBLScdGxJPl/iTry9znYX1KZO8leEbSKZCNICrpgC42+yXw6dT3MYSsealmM1nzUBkPAJ9Nxz8SeC7avTciT1I/YERE3Ad8meyKY9f0XXuVwJklc6j5TDrGeODliHi53fK7gH/M5TI2fe8fEU9ExLfJRqn+wDYe3/ooFw/r7d4jaU3u80Wyf7gnSfoVsIyuX0F8M9nbJpeTdWo/StaPADAd+EUXTVntXQT8uaQlwDTeGu67M/2B/5D0BPAYcHlEvETWp/ItSY+x7Vc4r6ftvw9M6mD5ucC41FG/HPhCip+fOtmXAH8Afr6Nx7c+yqPqmpH1lUTEq5L2BB4h6yxe3+i8toek+cD/iohFjc7F+h73eZhlbk+dyTsBF/f2wmFWNV95mJlZae7zMDOz0lw8zMysNBcPMzMrzcXDzMxKc/EwM7PS/j8NjG8kOxmQqQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(nrows=1, ncols=2)\n",
    "axs = axes.ravel()\n",
    "\n",
    "axs[0].boxplot(q_len)\n",
    "axs[0].set_title(\"Length of questions.\")\n",
    "\n",
    "axs[1].boxplot(a_len)\n",
    "axs[1].set_title(\"Length of answers.\")\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "fig2, ax2 = plt.subplots()\n",
    "ax2.hist(q_len, bins=40)\n",
    "ax2.set_xlabel(\"Length of samples\")\n",
    "ax2.set_ylabel(\"Number of samples\")\n",
    "ax2.set_title(\"Questions\")\n",
    "\n",
    "fig3, ax3 = plt.subplots()\n",
    "ax3.hist(a_len, bins=40)\n",
    "ax3.set_xlabel(\"Length of samples\")\n",
    "ax3.set_ylabel(\"Number of samples\")\n",
    "ax3.set_title(\"Answers\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 465,
   "id": "democratic-anderson",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "95% of q_len is :7.0\n",
      "95% of a_len is :8.0\n"
     ]
    }
   ],
   "source": [
    "print(f\"95% of q_len is :{np.percentile(q_len, 95)}\")\n",
    "print(f\"95% of a_len is :{np.percentile(a_len, 95)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daily-promise",
   "metadata": {},
   "source": [
    "데이터셋의 크기가 크지 않으므로 최대길이를 예제와 같이 40으로 정해 모든 단어를 포함하는 방향으로 진행한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 466,
   "id": "apparent-cornwall",
   "metadata": {},
   "outputs": [],
   "source": [
    "def below_threshold_len(max_len, nested_list):\n",
    "    cnt = 0\n",
    "    for s in nested_list:\n",
    "        if(len(s.split()) <= max_len):\n",
    "            cnt = cnt+1\n",
    "            \n",
    "    print(f\"전체 샘플 중 길이가 {max_len} 이하인 샘플의 비율: {cnt/len(nested_list):.2%}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 467,
   "id": "controversial-pastor",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Questions\n",
      ">>>\n",
      "전체 샘플 중 길이가 40 이하인 샘플의 비율: 100.00%\n",
      "Answers\n",
      ">>>\n",
      "전체 샘플 중 길이가 40 이하인 샘플의 비율: 100.00%\n"
     ]
    }
   ],
   "source": [
    "print(\"Questions\\n>>>\")\n",
    "below_threshold_len(40, questions)\n",
    "print(\"Answers\\n>>>\")\n",
    "below_threshold_len(40, answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 468,
   "id": "spanish-dylan",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40\n"
     ]
    }
   ],
   "source": [
    "MAX_LENGTH = 40\n",
    "print(MAX_LENGTH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 469,
   "id": "unlikely-couple",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_and_filter(inputs, outputs):\n",
    "    tokenized_inputs, tokenized_outputs = [], []\n",
    "    \n",
    "    for (sentence1, sentence2) in zip(inputs, outputs):\n",
    "        sentence1 = START_TOKEN + tokenizer.encode(sentence1) + END_TOKEN\n",
    "        sentence2 = START_TOKEN + tokenizer.encode(sentence2) + END_TOKEN\n",
    "        \n",
    "        if len(sentence1) <= MAX_LENGTH and len(sentence2) <= MAX_LENGTH:\n",
    "            tokenized_inputs.append(sentence1)\n",
    "            tokenized_outputs.append(sentence2)\n",
    "    \n",
    "    tokenized_inputs = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_inputs, maxlen=MAX_LENGTH, padding='post')\n",
    "    \n",
    "    tokenized_outputs = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_outputs, maxlen=MAX_LENGTH, padding='post')\n",
    "    \n",
    "    return tokenized_inputs, tokenized_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 470,
   "id": "alternate-heavy",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어장의 크기 : 8371\n",
      "필터링 후의 질문 샘플 개수: 11823\n",
      "필터링 후의 답변 샘플 개수: 11823\n"
     ]
    }
   ],
   "source": [
    "questions, answers = tokenize_and_filter(questions, answers)\n",
    "print('단어장의 크기 :',(VOCAB_SIZE))\n",
    "print('필터링 후의 질문 샘플 개수: {}'.format(len(questions)))\n",
    "print('필터링 후의 답변 샘플 개수: {}'.format(len(answers)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 471,
   "id": "filled-surname",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "BUFFER_SIZE = 20000\n",
    "\n",
    "dataset = tf.data.Dataset.from_tensor_slices((\n",
    "    {\n",
    "        'inputs': questions,\n",
    "        'dec_inputs': answers[:, :-1]\n",
    "    },\n",
    "    {\n",
    "        'outputs': answers[:, 1:]\n",
    "    }\n",
    "))\n",
    "\n",
    "dataset = dataset.cache()\n",
    "dataset = dataset.shuffle(BUFFER_SIZE)\n",
    "dataset = dataset.batch(BATCH_SIZE)\n",
    "dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unnecessary-gross",
   "metadata": {},
   "source": [
    "[BUFFER_SIZE in shuffle](https://helloyjam.github.io/tensorflow/buffer-size-in-shuffle/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cosmetic-energy",
   "metadata": {},
   "source": [
    "## 4. 모델링"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "optimum-toolbox",
   "metadata": {},
   "source": [
    "### Postional Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 472,
   "id": "noted-province",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(tf.keras.layers.Layer):\n",
    "    \n",
    "    def __init__(self, position, d_model):\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        self.pos_encoding = self.positional_encoding(position, d_model)\n",
    "    \n",
    "    def get_angles(self, position, i, d_model):\n",
    "        angles = 1 / tf.pow(10000, (2 * (i // 2)) / tf.cast(d_model, tf.float32))\n",
    "        return position * angles\n",
    "    \n",
    "    def positional_encoding(self, position, d_model):\n",
    "        \n",
    "        angle_rads = self.get_angles(\n",
    "            position=tf.range(position, dtype=tf.float32)[:, tf.newaxis],\n",
    "            i=tf.range(d_model, dtype=tf.float32)[tf.newaxis, :],\n",
    "            d_model=d_model)\n",
    "        \n",
    "        sines = tf.math.sin(angle_rads[:, 0::2])\n",
    "        cosines = tf.math.cos(angle_rads[:, 1::2])\n",
    "        \n",
    "        pos_encoding = tf.stack([sines, cosines], axis=0)\n",
    "        pos_encoding = tf.transpose(pos_encoding, [1, 2, 0])\n",
    "        pos_encoding = tf.reshape(pos_encoding, [position, d_model])\n",
    "        \n",
    "        pos_encoding = pos_encoding[tf.newaxis, ...]\n",
    "        return tf.cast(pos_encoding, tf.float32)\n",
    "    \n",
    "    def call(self, inputs):\n",
    "        return inputs + self.pos_encoding[:, :tf.shape(inputs)[1], :]\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "marine-duplicate",
   "metadata": {},
   "source": [
    "### Scaled_dot_product_attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 473,
   "id": "surprised-accent",
   "metadata": {},
   "outputs": [],
   "source": [
    "def scaled_dot_product_attention(query, key, value, mask):\n",
    "    matmul_qk = tf.matmul(query, key, transpose_b = True)\n",
    "    \n",
    "    depth = tf.cast(tf.shape(key)[-1], tf.float32)\n",
    "    logits = matmul_qk / tf.math.sqrt(depth)\n",
    "    \n",
    "    if mask is not None:\n",
    "        logits += (mask * -1e9)\n",
    "    \n",
    "    attention_weights = tf.nn.softmax(logits, axis=-1)\n",
    "    \n",
    "    output = tf.matmul(attention_weights, value)\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "latter-parts",
   "metadata": {},
   "source": [
    "### MultiHeadAttention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 474,
   "id": "listed-subscription",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(tf.keras.layers.Layer):\n",
    "    \n",
    "    def __init__(self, d_model, num_heads, name=\"multi_head_attention\"):\n",
    "        super(MultiHeadAttention, self).__init__(name=name)\n",
    "        self.num_heads = num_heads\n",
    "        self.d_model = d_model\n",
    "        \n",
    "        assert d_model % self.num_heads == 0\n",
    "        \n",
    "        self.depth = d_model // self.num_heads\n",
    "        \n",
    "        self.query_dense = tf.keras.layers.Dense(units=d_model)\n",
    "        self.key_dense = tf.keras.layers.Dense(units=d_model)\n",
    "        self.value_dense = tf.keras.layers.Dense(units=d_model)\n",
    "        \n",
    "        self.dense = tf.keras.layers.Dense(units=d_model)\n",
    "        \n",
    "    def split_heads(self, inputs, batch_size):\n",
    "        inputs = tf.reshape(\n",
    "            inputs, shape=(batch_size, -1, self.num_heads, self.depth))\n",
    "        \n",
    "        return tf.transpose(inputs, perm=[0, 2, 1, 3])\n",
    "    \n",
    "    def call(self, inputs):\n",
    "        query, key, value, mask = inputs['query'], inputs['key'], inputs['value'], inputs['mask']\n",
    "        \n",
    "        batch_size = tf.shape(query)[0]\n",
    "        \n",
    "        query = self.query_dense(query)\n",
    "        key = self.key_dense(key)\n",
    "        value = self.value_dense(value)\n",
    "        \n",
    "        query = self.split_heads(query, batch_size)\n",
    "        key = self.split_heads(key, batch_size)\n",
    "        value = self.split_heads(value, batch_size)\n",
    "        \n",
    "        scaled_attention = scaled_dot_product_attention(query, key, value, mask)\n",
    "        \n",
    "        scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3])\n",
    "        \n",
    "        concat_attention = tf.reshape(scaled_attention,\n",
    "                                      (batch_size, -1, self.d_model))\n",
    "        \n",
    "        outputs = self.dense(concat_attention)\n",
    "        \n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "understood-vault",
   "metadata": {},
   "source": [
    "### Masks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 475,
   "id": "funded-creator",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_padding_mask(x):\n",
    "    mask = tf.cast(tf.math.equal(x, 0), tf.float32)\n",
    "    return mask[:, tf.newaxis, tf.newaxis, :]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 476,
   "id": "boolean-insured",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_look_ahead_mask(x):\n",
    "    seq_len = tf.shape(x)[1]\n",
    "    look_ahead_mask = 1 - tf.linalg.band_part(tf.ones((seq_len, seq_len)), -1, 0)\n",
    "    padding_mask = create_padding_mask(x)\n",
    "    return tf.maximum(look_ahead_mask, padding_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "operating-decrease",
   "metadata": {},
   "source": [
    "### Encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 477,
   "id": "cutting-short",
   "metadata": {},
   "outputs": [],
   "source": [
    "def encoder_layer(units, d_model, num_heads, dropout, name=\"encoder_layer\"):\n",
    "    inputs = tf.keras.Input(shape=(None, d_model), name=\"inputs\")\n",
    "    \n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name=\"padding_mask\")\n",
    "    \n",
    "    attention = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention\")({\n",
    "            'query': inputs,\n",
    "            'key': inputs,\n",
    "            'value': inputs,\n",
    "            'mask': padding_mask\n",
    "        })\n",
    "    \n",
    "    attention = tf.keras.layers.Dropout(rate=dropout)(attention)\n",
    "    attention = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(inputs+attention)\n",
    "    \n",
    "    outputs = tf.keras.layers.Dense(units=units, activation='relu')(attention)\n",
    "    outputs = tf.keras.layers.Dense(units=d_model)(outputs)\n",
    "    \n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(outputs)\n",
    "    outputs = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention + outputs)\n",
    "    \n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, padding_mask], outputs=outputs, name=name)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 478,
   "id": "correct-tunnel",
   "metadata": {},
   "outputs": [],
   "source": [
    "def encoder(vocab_size,\n",
    "            num_layers,\n",
    "            units,\n",
    "            d_model,\n",
    "            num_heads,\n",
    "            dropout,\n",
    "            name=\"encoder\"):\n",
    "    inputs = tf.keras.Input(shape=(None,), name=\"inputs\")\n",
    "    \n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name=\"padding_mask\")\n",
    "    \n",
    "    embeddings = tf.keras.layers.Embedding(vocab_size, d_model)(inputs)\n",
    "    embeddings *= tf.math.sqrt(tf.cast(d_model, tf.float32))\n",
    "    \n",
    "    embedding = PositionalEncoding(vocab_size, d_model)(embeddings)\n",
    "    \n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(embeddings)\n",
    "    \n",
    "    for i in range(num_layers):\n",
    "        outputs = encoder_layer(\n",
    "            units=units,\n",
    "            d_model=d_model,\n",
    "            num_heads=num_heads,\n",
    "            dropout=dropout,\n",
    "            name=\"encoder_layer_{}\".format(i),\n",
    "        )([outputs, padding_mask])\n",
    "        \n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, padding_mask], outputs=outputs, name=name)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "latin-cuisine",
   "metadata": {},
   "source": [
    "### Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 479,
   "id": "material-grass",
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder_layer(units, d_model, num_heads, dropout, name=\"decoder_layer\"):\n",
    "    inputs = tf.keras.Input(shape=(None, d_model), name=\"inputs\")\n",
    "    enc_outputs = tf.keras.Input(shape=(None, d_model))\n",
    "    look_ahead_mask = tf.keras.Input(\n",
    "        shape=(1, None, None), name=\"look_ahead_mask\")\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name=\"padding_mask\")\n",
    "    \n",
    "    attention1 = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention_1\")(inputs={\n",
    "            'query': inputs,\n",
    "            'key': inputs,\n",
    "            'value': inputs,\n",
    "            'mask': look_ahead_mask\n",
    "    })\n",
    "    \n",
    "    attention1 = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention1 + inputs)\n",
    "    \n",
    "    attention2 = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention2\")(inputs={\n",
    "            'query': attention1,\n",
    "            'key': enc_outputs,\n",
    "            'value': enc_outputs,\n",
    "            'mask': padding_mask\n",
    "    })\n",
    "    \n",
    "    attention2 = tf.keras.layers.Dropout(rate=dropout)(attention2)\n",
    "    attention2 = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention2 + attention1)\n",
    "    \n",
    "    outputs = tf.keras.layers.Dense(units=units, activation='relu')(attention2)\n",
    "    outputs = tf.keras.layers.Dense(units=d_model)(outputs)\n",
    "    \n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(outputs)\n",
    "    outputs = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(outputs + attention2)\n",
    "    \n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, enc_outputs, look_ahead_mask, padding_mask],\n",
    "        outputs=outputs,\n",
    "        name=name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 480,
   "id": "declared-columbia",
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder(vocab_size,\n",
    "            num_layers,\n",
    "            units,\n",
    "            d_model,\n",
    "            num_heads,\n",
    "            dropout,\n",
    "            name='decoder'):\n",
    "    inputs = tf.keras.Input(shape=(None,), name='inputs')\n",
    "    enc_outputs = tf.keras.Input(shape=(None, d_model), name='encoder_outputs')\n",
    "    look_ahead_mask = tf.keras.Input(\n",
    "        shape=(1, None, None), name='look_ahead_mask')\n",
    "    \n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name='padding_mask')\n",
    "    \n",
    "    embeddings = tf.keras.layers.Embedding(vocab_size, d_model)(inputs)\n",
    "    embeddings *= tf.math.sqrt(tf.cast(d_model, tf.float32))\n",
    "    \n",
    "    embeddings = PositionalEncoding(vocab_size, d_model)(embeddings)\n",
    "    \n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(embeddings)\n",
    "    \n",
    "    for i in range(num_layers):\n",
    "        outputs = decoder_layer(\n",
    "            units=units,\n",
    "            d_model=d_model,\n",
    "            num_heads=num_heads,\n",
    "            dropout=dropout,\n",
    "            name='decoder_layer_{}'.format(i),\n",
    "        )(inputs=[outputs, enc_outputs, look_ahead_mask, padding_mask])\n",
    "        \n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, enc_outputs, look_ahead_mask, padding_mask],\n",
    "        outputs=outputs,\n",
    "        name=name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "continued-passing",
   "metadata": {},
   "source": [
    "### Transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 481,
   "id": "muslim-asthma",
   "metadata": {},
   "outputs": [],
   "source": [
    "def transformer(vocab_size,\n",
    "                num_layers,\n",
    "                units,\n",
    "                d_model,\n",
    "                num_heads,\n",
    "                dropout,\n",
    "                name=\"transformer\"):\n",
    "    inputs = tf.keras.Input(shape=(None,), name=\"inputs\")\n",
    "    dec_inputs = tf.keras.Input(shape=(None,), name=\"dec_inputs\")\n",
    "    \n",
    "    enc_padding_mask = tf.keras.layers.Lambda(\n",
    "        create_padding_mask, output_shape=(1, 1, None),\n",
    "        name='enc_padding_mask')(inputs)\n",
    "    \n",
    "    look_ahead_mask = tf.keras.layers.Lambda(\n",
    "        create_look_ahead_mask,\n",
    "        output_shape=(1, None, None),\n",
    "        name='look_ahead_mask')(dec_inputs)\n",
    "    \n",
    "    dec_padding_mask = tf.keras.layers.Lambda(\n",
    "        create_padding_mask, output_shape=(1, 1, None),\n",
    "        name='dec_padding_mask')(inputs)\n",
    "    \n",
    "    enc_outputs = encoder(\n",
    "        vocab_size=vocab_size,\n",
    "        num_layers=num_layers,\n",
    "        units=units,\n",
    "        d_model=d_model,\n",
    "        num_heads=num_heads,\n",
    "        dropout=dropout,\n",
    "    )(inputs=[inputs, enc_padding_mask])\n",
    "    \n",
    "    dec_outputs = decoder(\n",
    "        vocab_size=vocab_size,\n",
    "        num_layers=num_layers,\n",
    "        units=units,\n",
    "        d_model=d_model,\n",
    "        num_heads=num_heads,\n",
    "        dropout=dropout,\n",
    "    )(inputs=[dec_inputs, enc_outputs, look_ahead_mask, dec_padding_mask])\n",
    "    \n",
    "    outputs = tf.keras.layers.Dense(units=vocab_size, name=\"outputs\")(dec_outputs)\n",
    "    \n",
    "    return tf.keras.Model(inputs=[inputs, dec_inputs], outputs=outputs, name=name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "green-twelve",
   "metadata": {},
   "source": [
    "## 5. 모델링"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 482,
   "id": "compatible-tampa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"transformer\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "inputs (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dec_inputs (InputLayer)         [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "enc_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "encoder (Functional)            (None, None, 256)    3197184     inputs[0][0]                     \n",
      "                                                                 enc_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "look_ahead_mask (Lambda)        (None, 1, None, None 0           dec_inputs[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dec_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "decoder (Functional)            (None, None, 256)    3724544     dec_inputs[0][0]                 \n",
      "                                                                 encoder[0][0]                    \n",
      "                                                                 look_ahead_mask[0][0]            \n",
      "                                                                 dec_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "outputs (Dense)                 (None, None, 8371)   2151347     decoder[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 9,073,075\n",
      "Trainable params: 9,073,075\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "\n",
    "\n",
    "NUM_LAYERS = 2\n",
    "D_MODEL = 256\n",
    "NUM_HEADS = 8\n",
    "UNITS = 512\n",
    "DROPOUT = 0.1\n",
    "\n",
    "model = transformer(\n",
    "    vocab_size=VOCAB_SIZE,\n",
    "    num_layers=NUM_LAYERS,\n",
    "    units=UNITS,\n",
    "    d_model=D_MODEL,\n",
    "    num_heads=NUM_HEADS,\n",
    "    dropout=DROPOUT)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 483,
   "id": "corresponding-applicant",
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_function(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH - 1))\n",
    "    \n",
    "    loss = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "        from_logits=True, reduction='none')(y_true, y_pred)\n",
    "    \n",
    "    mask = tf.cast(tf.not_equal(y_true, 0), tf.float32)\n",
    "    loss = tf.multiply(loss, mask)\n",
    "    \n",
    "    return tf.reduce_mean(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 484,
   "id": "measured-ballet",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "    \n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super(CustomSchedule, self).__init__()\n",
    "        \n",
    "        self.d_model = d_model\n",
    "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "        \n",
    "        self.warmup_steps = warmup_steps\n",
    "        \n",
    "    def __call__(self, step):\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps**-1.5)\n",
    "        \n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 485,
   "id": "municipal-fleece",
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = CustomSchedule(D_MODEL)\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(\n",
    "    learning_rate, beta_1=0.9, beta_2=0.98, epsilon=1e-9)\n",
    "\n",
    "def accuracy(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH-1))\n",
    "    return tf.keras.metrics.sparse_categorical_accuracy(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 486,
   "id": "resistant-conjunction",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer, loss=loss_function, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 487,
   "id": "upper-spyware",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "185/185 [==============================] - 32s 141ms/step - loss: 1.5416 - accuracy: 0.0113\n",
      "Epoch 2/50\n",
      "185/185 [==============================] - 26s 138ms/step - loss: 1.2531 - accuracy: 0.0471\n",
      "Epoch 3/50\n",
      "185/185 [==============================] - 27s 147ms/step - loss: 1.0301 - accuracy: 0.0501\n",
      "Epoch 4/50\n",
      "185/185 [==============================] - 26s 142ms/step - loss: 0.9396 - accuracy: 0.0534\n",
      "Epoch 5/50\n",
      "185/185 [==============================] - 26s 143ms/step - loss: 0.8789 - accuracy: 0.0570\n",
      "Epoch 6/50\n",
      "185/185 [==============================] - 26s 142ms/step - loss: 0.8225 - accuracy: 0.0612\n",
      "Epoch 7/50\n",
      "185/185 [==============================] - 26s 142ms/step - loss: 0.7430 - accuracy: 0.0670\n",
      "Epoch 8/50\n",
      "185/185 [==============================] - 25s 137ms/step - loss: 0.6657 - accuracy: 0.0763\n",
      "Epoch 9/50\n",
      "185/185 [==============================] - 26s 139ms/step - loss: 0.5702 - accuracy: 0.0863\n",
      "Epoch 10/50\n",
      "185/185 [==============================] - 26s 140ms/step - loss: 0.4777 - accuracy: 0.0977\n",
      "Epoch 11/50\n",
      "185/185 [==============================] - 26s 141ms/step - loss: 0.3894 - accuracy: 0.1112\n",
      "Epoch 12/50\n",
      "185/185 [==============================] - 25s 137ms/step - loss: 0.2965 - accuracy: 0.1245\n",
      "Epoch 13/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.2186 - accuracy: 0.1377\n",
      "Epoch 14/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.1535 - accuracy: 0.1487\n",
      "Epoch 15/50\n",
      "185/185 [==============================] - 26s 143ms/step - loss: 0.1025 - accuracy: 0.1582\n",
      "Epoch 16/50\n",
      "185/185 [==============================] - 26s 142ms/step - loss: 0.0707 - accuracy: 0.1630\n",
      "Epoch 17/50\n",
      "185/185 [==============================] - 26s 141ms/step - loss: 0.0490 - accuracy: 0.1668\n",
      "Epoch 18/50\n",
      "185/185 [==============================] - 26s 138ms/step - loss: 0.0393 - accuracy: 0.1668\n",
      "Epoch 19/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0337 - accuracy: 0.1674\n",
      "Epoch 20/50\n",
      "185/185 [==============================] - 27s 144ms/step - loss: 0.0331 - accuracy: 0.1688\n",
      "Epoch 21/50\n",
      "185/185 [==============================] - 27s 144ms/step - loss: 0.0320 - accuracy: 0.1689\n",
      "Epoch 22/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0307 - accuracy: 0.1676\n",
      "Epoch 23/50\n",
      "185/185 [==============================] - 27s 144ms/step - loss: 0.0288 - accuracy: 0.1675\n",
      "Epoch 24/50\n",
      "185/185 [==============================] - 26s 142ms/step - loss: 0.0246 - accuracy: 0.1698\n",
      "Epoch 25/50\n",
      "185/185 [==============================] - 26s 141ms/step - loss: 0.0212 - accuracy: 0.1707\n",
      "Epoch 26/50\n",
      "185/185 [==============================] - 27s 147ms/step - loss: 0.0186 - accuracy: 0.1697\n",
      "Epoch 27/50\n",
      "185/185 [==============================] - 27s 144ms/step - loss: 0.0170 - accuracy: 0.1717\n",
      "Epoch 28/50\n",
      "185/185 [==============================] - 27s 148ms/step - loss: 0.0157 - accuracy: 0.1710\n",
      "Epoch 29/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0130 - accuracy: 0.1715\n",
      "Epoch 30/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0132 - accuracy: 0.1728\n",
      "Epoch 31/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0116 - accuracy: 0.1731\n",
      "Epoch 32/50\n",
      "185/185 [==============================] - 27s 148ms/step - loss: 0.0103 - accuracy: 0.1735\n",
      "Epoch 33/50\n",
      "185/185 [==============================] - 27s 147ms/step - loss: 0.0105 - accuracy: 0.1731\n",
      "Epoch 34/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0094 - accuracy: 0.1724\n",
      "Epoch 35/50\n",
      "185/185 [==============================] - 27s 148ms/step - loss: 0.0083 - accuracy: 0.1737\n",
      "Epoch 36/50\n",
      "185/185 [==============================] - 27s 148ms/step - loss: 0.0082 - accuracy: 0.1732\n",
      "Epoch 37/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0082 - accuracy: 0.1735\n",
      "Epoch 38/50\n",
      "185/185 [==============================] - 27s 148ms/step - loss: 0.0073 - accuracy: 0.1739\n",
      "Epoch 39/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0069 - accuracy: 0.1737\n",
      "Epoch 40/50\n",
      "185/185 [==============================] - 26s 142ms/step - loss: 0.0065 - accuracy: 0.1746\n",
      "Epoch 41/50\n",
      "185/185 [==============================] - 27s 146ms/step - loss: 0.0062 - accuracy: 0.1748\n",
      "Epoch 42/50\n",
      "185/185 [==============================] - 28s 151ms/step - loss: 0.0058 - accuracy: 0.1741\n",
      "Epoch 43/50\n",
      "185/185 [==============================] - 28s 149ms/step - loss: 0.0055 - accuracy: 0.1741\n",
      "Epoch 44/50\n",
      "185/185 [==============================] - 27s 148ms/step - loss: 0.0055 - accuracy: 0.1742\n",
      "Epoch 45/50\n",
      "185/185 [==============================] - 27s 145ms/step - loss: 0.0053 - accuracy: 0.1745\n",
      "Epoch 46/50\n",
      "185/185 [==============================] - 26s 139ms/step - loss: 0.0048 - accuracy: 0.1749\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fbf553d76d0>"
      ]
     },
     "execution_count": 487,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EPOCHS = 50\n",
    "callback = tf.keras.callbacks.EarlyStopping(monitor='loss')\n",
    "model.fit(dataset, epochs=EPOCHS, verbose=1, callbacks=[callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 504,
   "id": "periodic-lodging",
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder_inference(sentence):\n",
    "    sentence = preprocess_sentence(sentence)\n",
    "\n",
    "  # 입력된 문장을 정수 인코딩 후, 시작 토큰과 종료 토큰을 앞뒤로 추가.\n",
    "  # ex) Where have you been? → [[8331   86   30    5 1059    7 8332]]\n",
    "    sentence = tf.expand_dims(\n",
    "      START_TOKEN + tokenizer.encode(sentence) + END_TOKEN, axis=0)\n",
    "\n",
    "  # 디코더의 현재까지의 예측한 출력 시퀀스가 지속적으로 저장되는 변수.\n",
    "  # 처음에는 예측한 내용이 없음으로 시작 토큰만 별도 저장. ex) 8331\n",
    "    output_sequence = tf.expand_dims(START_TOKEN, 0)\n",
    "\n",
    "  # 디코더의 인퍼런스 단계\n",
    "    for i in range(MAX_LENGTH):\n",
    "        predictions = model(inputs=[sentence, output_sequence], training=False)\n",
    "        predictions = predictions[:, -1:, :]\n",
    "\n",
    "        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
    "\n",
    "    # 만약 현재 예측한 단어가 종료 토큰이라면 for문을 종료\n",
    "        if tf.equal(predicted_id, END_TOKEN[0]):\n",
    "            break\n",
    "\n",
    "    # 예측한 단어들은 지속적으로 output_sequence에 추가됩니다.\n",
    "\n",
    "        output_sequence = tf.concat([output_sequence, predicted_id], axis=-1)\n",
    "\n",
    "    return tf.squeeze(output_sequence, axis=0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 505,
   "id": "impressed-lodging",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence_generation(sentence):\n",
    "  # 입력 문장에 대해서 디코더를 동작 시켜 예측된 정수 시퀀스를 리턴받습니다.\n",
    "    prediction = decoder_inference(sentence)\n",
    "\n",
    "    # 정수 시퀀스를 다시 텍스트 시퀀스로 변환합니다.\n",
    "    predicted_sentence = tokenizer.decode(\n",
    "      [i for i in prediction if i < tokenizer.vocab_size])\n",
    "\n",
    "    print('입력 : {}'.format(sentence))\n",
    "    print('출력 : {}'.format(predicted_sentence))\n",
    "\n",
    "    return predicted_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 506,
   "id": "herbal-lounge",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 출근길 너무 힘들어\n",
      "출력 : 회사 근처로 이사를 가보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'회사 근처로 이사를 가보세요 .'"
      ]
     },
     "execution_count": 506,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('출근길 너무 힘들어')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 507,
   "id": "affiliated-transmission",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 안녕, 넌 이름이 뭐니?\n",
      "출력 : 저는 위로봇입니다 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'저는 위로봇입니다 .'"
      ]
     },
     "execution_count": 507,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('안녕, 넌 이름이 뭐니?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 508,
   "id": "loaded-thing",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때?\n",
      "출력 : 친구들과 같이 놀러가세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'친구들과 같이 놀러가세요 .'"
      ]
     },
     "execution_count": 508,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 510,
   "id": "liquid-zimbabwe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어떠니?\n",
      "출력 : 저랑 이야기 해요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'저랑 이야기 해요 .'"
      ]
     },
     "execution_count": 510,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어떠니?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 511,
   "id": "continent-navigation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어땡?\n",
      "출력 : 정말 기분 좋으시겠어요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'정말 기분 좋으시겠어요 .'"
      ]
     },
     "execution_count": 511,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어땡?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 513,
   "id": "affiliated-venice",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때??\n",
      "출력 : 저랑 같이 놀아요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'저랑 같이 놀아요 .'"
      ]
     },
     "execution_count": 513,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때??')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 515,
   "id": "freelance-disorder",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때ㅋㅋㅋㅋㅋ\n",
      "출력 : 이젠 기억을 놔주는 건 어떨까요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'이젠 기억을 놔주는 건 어떨까요 .'"
      ]
     },
     "execution_count": 515,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때ㅋㅋㅋㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 517,
   "id": "experimental-reproduction",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋ\n",
      "출력 : 새로운 사람으로 잊혀질 거예요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'새로운 사람으로 잊혀질 거예요 .'"
      ]
     },
     "execution_count": 517,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 516,
   "id": "entire-stack",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋㅋ\n",
      "출력 : 좋은 사람이랑 결혼할 수 있을 거예요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'좋은 사람이랑 결혼할 수 있을 거예요 .'"
      ]
     },
     "execution_count": 516,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 518,
   "id": "sized-bride",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋㅋㅋ\n",
      "출력 : 좋은 소식길 바라요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'좋은 소식길 바라요 .'"
      ]
     },
     "execution_count": 518,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 550,
   "id": "interesting-webster",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 날씨 좋아?\n",
      "출력 : 날씨 어플에 물어보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'날씨 어플에 물어보세요 .'"
      ]
     },
     "execution_count": 550,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('날씨 좋아?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 551,
   "id": "continuing-creek",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 날씨 좋아!\n",
      "출력 : 날씨 어플에 물어보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'날씨 어플에 물어보세요 .'"
      ]
     },
     "execution_count": 551,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('날씨 좋아!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reliable-protocol",
   "metadata": {},
   "source": [
    "정확도는 약 18%에서 Earlystopping을 했지만 문장자체는 비교적 자연스럽게 생성하는 것 같습니다. 하지만 같은 의미의 문장이여도 의미 자체를 다르게 받아들이고 부자연스러운 답변을 생성하는 경우도 많네요."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "external-transformation",
   "metadata": {},
   "source": [
    "## 6. 개선을 위한 전처리 수정"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coral-holiday",
   "metadata": {},
   "source": [
    "### 1. 'ㅋㅋㅋ', 'ㅠㅠㅠ'와 같은 단어들을 'ㅋㅋ', 'ㅠㅠ' 로 바꿔주기."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 640,
   "id": "banned-pastor",
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = chatbotdata['Q'].to_numpy()\n",
    "answers = chatbotdata['A'].to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 641,
   "id": "experimental-cemetery",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8758</th>\n",
       "      <td>혼자ㅠㅠㅠ</td>\n",
       "      <td>많이 힘들어하지 마요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Q             A  label\n",
       "8758  혼자ㅠㅠㅠ  많이 힘들어하지 마요.      1"
      ]
     },
     "execution_count": 641,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chatbotdata.loc[chatbotdata['Q'].str.contains('ㅠㅠㅠ')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 642,
   "id": "thirty-president",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "혼자ㅠㅠ\n",
      "혼자ㅠㅠ\n",
      "뭐야ㅋㅋ\n",
      "뭐야ㅋㅋ\n"
     ]
    }
   ],
   "source": [
    "sentence = '혼자ㅠㅠㅠㅠㅠㅠㅠㅠㅠㅠㅠㅠ'\n",
    "sentence = re.sub(\"(.)\\\\1{1,}\", r\"\\1\\1\", sentence)\n",
    "print(sentence)\n",
    "\n",
    "sentence = '혼자ㅠㅠ'\n",
    "sentence = re.sub(\"(.)\\\\1{1,}\", r\"\\1\\1\", sentence)\n",
    "print(sentence)\n",
    "\n",
    "sentence = '뭐야ㅋㅋㅋㅋㅋㅋㅋㅋㅋ'\n",
    "sentence = re.sub(\"(.)\\\\1{1,}\", r\"\\1\\1\", sentence)\n",
    "print(sentence)\n",
    "\n",
    "sentence = '뭐야ㅋㅋㅋㅋ'\n",
    "sentence = re.sub(\"(.)\\\\1{1,}\", r\"\\1\\1\", sentence)\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dried-barrier",
   "metadata": {},
   "source": [
    "추가적으로 느낌표와 물음표 등은 \"!!\", \"??\"가 각각 \"!\", \"?\"와 같은 의미를 가진다고 생각할 수 있습니다.\n",
    "(\"..\" 와 \".\" 는 조금 의미가 다르다고 생각해 전처리 과정에 포함시키지 않았습니다.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 643,
   "id": "sudden-clause",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "뭐야?!\n"
     ]
    }
   ],
   "source": [
    "sentence = '뭐야????!!!!!!!!!'\n",
    "sentence = re.sub(\"(!)\\\\1{1,}\", r\"\\1\", sentence)\n",
    "sentence = re.sub(\"(\\?)\\\\1{1,}\", r\"\\1\", sentence)\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ranking-speaking",
   "metadata": {},
   "source": [
    "### 2. 특수문자 제거"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 644,
   "id": "super-budget",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 저를 만들어 준 사람을 부모님 , 저랑 이야기해 주는 사람을 친구로 생각하고 있어요 \n"
     ]
    }
   ],
   "source": [
    "sentence = '\"저를 만들어 준 사람을 부모님, 저랑 이야기해 주는 사람을 친구로 생각하고 있어요\"'\n",
    "sentence = re.sub(r\"([?.!,])\", r\" \\1 \", sentence)\n",
    "sentence = re.sub(r\"[^가-힣0-9?.!,]+\", \" \", sentence)\n",
    "sentence = re.sub(r'[\" \"]+', \" \", sentence)\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collectible-breeding",
   "metadata": {},
   "source": [
    "위 두 전처리를 전처리 함수에 추가해보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 645,
   "id": "danish-advocacy",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"저를 만들어 준 사람을 부모님 ,  저랑 이야기해 주는 사람을 친구로 생각하고 있어요 . ㅋㅋㅋㅋㅋ\"\n",
      " 저를 만들어 준 사람을 부모님 , 저랑 이야기해 주는 사람을 친구로 생각하고 있어요 . ㅋㅋㅋㅋㅋ \n",
      " 저를 만들어 준 사람을 부모님 , 저랑 이야기해 주는 사람을 친구로 생각하고 있어요 . ㅋㅋㅋㅋㅋ \n",
      " 저를 만들어 준 사람을 부모님 , 저랑 이야기해 주는 사람을 친구로 생각하고 있어요 . ㅋㅋ \n",
      "저를 만들어 준 사람을 부모님 , 저랑 이야기해 주는 사람을 친구로 생각하고 있어요 . ㅋㅋ\n"
     ]
    }
   ],
   "source": [
    "sentence = '\"저를 만들어 준 사람을 부모님, 저랑 이야기해 주는 사람을 친구로 생각하고 있어요.ㅋㅋㅋㅋㅋ\"'\n",
    "sentence = re.sub(r\"([?.!,])\", r\" \\1 \", sentence)\n",
    "print(sentence)\n",
    "sentence = re.sub(r\"[^ㄱ-ㅎ가-힣0-9?.!,]+\", \" \", sentence)\n",
    "print(sentence)\n",
    "sentence = re.sub(r'[\" \"]+', \" \", sentence)\n",
    "print(sentence)\n",
    "sentence = re.sub(\"(.)\\\\1{1,}\", r\"\\1\\1\", sentence)\n",
    "print(sentence)\n",
    "sentence = sentence.strip()\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 646,
   "id": "associate-metro",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"이게 무슨일이야??!??!\"\n",
      "\"이게 무슨일이야?!?!\"\n",
      "\"이게 무슨일이야 ?  !  ?  ! \"\n",
      " 이게 무슨일이야 ? ! ? ! \n",
      " 이게 무슨일이야 ? ! ? ! \n",
      " 이게 무슨일이야 ? ! ? ! \n",
      "이게 무슨일이야 ? ! ? !\n"
     ]
    }
   ],
   "source": [
    "sentence = '\"이게 무슨일이야??!??!!!\"'\n",
    "sentence = re.sub(\"(!)\\\\1{1,}\", r\"\\1\", sentence)\n",
    "print(sentence)\n",
    "sentence = re.sub(\"(\\?)\\\\1{1,}\", r\"\\1\", sentence)\n",
    "print(sentence)\n",
    "sentence = re.sub(r\"([?.!,])\", r\" \\1 \", sentence)\n",
    "print(sentence)\n",
    "sentence = re.sub(r\"[^ㄱ-ㅎ가-힣0-9?.!,]+\", \" \", sentence)\n",
    "print(sentence)\n",
    "sentence = re.sub(r'[\" \"]+', \" \", sentence)\n",
    "print(sentence)\n",
    "sentence = re.sub(\"(.)\\\\1{1,}\", r\"\\1\\1\", sentence)\n",
    "print(sentence)\n",
    "sentence = sentence.strip()\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 647,
   "id": "charming-primary",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_sentence(sentence):\n",
    "    sentence = re.sub(\"(!)\\\\1{1,}\", r\"\\1\", sentence) ## 연속 느낌표 제거\n",
    "    sentence = re.sub(\"(\\?)\\\\1{1,}\", r\"\\1\", sentence) ## 연속 물음표 제거\n",
    "    sentence = re.sub(r\"([?.!,])\", r\" \\1 \", sentence) ## 구둣점 분리\n",
    "    sentence = re.sub(r\"[^가-힣0-9?.!,]+\", \" \", sentence)\n",
    "    sentence = re.sub(\"(.)\\\\1{1,}\", r\"\\1\\1\", sentence) ## 반복되는 ㅋㅋㅋ => ㅋㅋ\n",
    "    sentence = sentence.strip()\n",
    "    \n",
    "    return sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 648,
   "id": "completed-dealer",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_conversations():\n",
    "    \n",
    "    questions = chatbotdata['Q'].apply(preprocess_sentence)\n",
    "    answers = chatbotdata['A'].apply(preprocess_sentence)\n",
    "    \n",
    "    return questions.to_numpy(), answers.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 649,
   "id": "inclusive-potential",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 질문 샘플 수: 11823\n",
      "전체 답변 샘플 수: 11823\n"
     ]
    }
   ],
   "source": [
    "questions, answers = load_conversations()\n",
    "print(f\"전체 질문 샘플 수: {len(questions)}\")\n",
    "print(f\"전체 답변 샘플 수: {len(answers)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 650,
   "id": "focal-coordinator",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전처리 후의 1523번째 질문 샘플: 말 잘하는 법\n",
      "전처리 후의 1523번째 답변 샘플: 책을 많이 읽어보세요 .\n",
      "==================================================\n",
      "전처리 후의 11532번째 질문 샘플: 짝녀랑 연락이 끊겼는데 더 연락하고 싶어 .\n",
      "전처리 후의 11532번째 답변 샘플: 연락할 핑계를 찾아보세요 .\n",
      "==================================================\n",
      "전처리 후의 4328번째 질문 샘플: 진짜 별 사람이 다 있어\n",
      "전처리 후의 4328번째 답변 샘플: 정말 특이한 사람 많죠 .\n",
      "==================================================\n",
      "전처리 후의 2180번째 질문 샘플: 불안해\n",
      "전처리 후의 2180번째 답변 샘플: 두려워하지 않아도 돼요 .\n",
      "==================================================\n",
      "전처리 후의 9761번째 질문 샘플: 몰래 동거하고 있어\n",
      "전처리 후의 9761번째 답변 샘플: 허락을 맡고 당당하게 하는 게 좋을 것 같아요 .\n",
      "==================================================\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng()\n",
    "\n",
    "\n",
    "for _ in range(5):\n",
    "    idx = rng.integers(len(answers))\n",
    "    print(f'전처리 후의 {idx}번째 질문 샘플: {questions[idx]}')\n",
    "    print(f'전처리 후의 {idx}번째 답변 샘플: {answers[idx]}')\n",
    "    print('=' * 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sapphire-starter",
   "metadata": {},
   "source": [
    "## SubwordTextEncoder 사용하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 651,
   "id": "accompanied-brooks",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "tokenizer = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(questions + answers, \n",
    "                                                                      target_vocab_size=2**13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 652,
   "id": "southern-organic",
   "metadata": {},
   "outputs": [],
   "source": [
    "START_TOKEN, END_TOKEN = [tokenizer.vocab_size], [tokenizer.vocab_size+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 653,
   "id": "elect-fisher",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "START_TOKEN의 번호 : [8348]\n",
      "END_TOKEN의 번호 : [8349]\n"
     ]
    }
   ],
   "source": [
    "print('START_TOKEN의 번호 :' ,[tokenizer.vocab_size])\n",
    "print('END_TOKEN의 번호 :' ,[tokenizer.vocab_size + 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 654,
   "id": "boolean-willow",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8350\n"
     ]
    }
   ],
   "source": [
    "# 시작 토큰과 종료 토큰을 고려하여 +2를 하여 단어장의 크기를 산정합니다.\n",
    "VOCAB_SIZE = tokenizer.vocab_size + 2\n",
    "print(VOCAB_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 655,
   "id": "moral-passport",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전처리 후의 5869번째 질문 샘플: [50, 3664, 977, 687, 1]\n",
      "전처리 후의 5869번째 답변 샘플: [831, 22, 1420, 1]\n",
      "==================================================\n",
      "전처리 후의 3583번째 질문 샘플: [4065, 1233, 428, 6148, 4871, 786]\n",
      "전처리 후의 3583번째 답변 샘플: [1413, 13, 3954, 1]\n",
      "==================================================\n",
      "전처리 후의 6944번째 질문 샘플: [58, 205, 43, 5775]\n",
      "전처리 후의 6944번째 답변 샘플: [4488, 3439, 1]\n",
      "==================================================\n",
      "전처리 후의 3600번째 질문 샘플: [765, 2305, 312, 9, 59]\n",
      "전처리 후의 3600번째 답변 샘플: [466, 912, 131, 7436, 1]\n",
      "==================================================\n",
      "전처리 후의 4754번째 질문 샘플: [1774, 3033]\n",
      "전처리 후의 4754번째 답변 샘플: [678, 387, 1]\n",
      "==================================================\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng()\n",
    "\n",
    "\n",
    "for _ in range(5):\n",
    "    idx = rng.integers(len(answers))\n",
    "    print(f'전처리 후의 {idx}번째 질문 샘플: {tokenizer.encode(questions[idx])}')\n",
    "    print(f'전처리 후의 {idx}번째 답변 샘플: {tokenizer.encode(answers[idx])}')\n",
    "    print('=' * 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 656,
   "id": "early-scratch",
   "metadata": {},
   "outputs": [],
   "source": [
    "q_len = [len(s.split()) for s in questions]\n",
    "a_len = [len(s.split()) for s in answers]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 657,
   "id": "knowing-sudan",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Answers')"
      ]
     },
     "execution_count": 657,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeg0lEQVR4nO3df3xddZ3n8debNrYUUZsSmQJN64AypfEHGNGh9UcXVGQYYXZc1y6wMFPb6TzGqNMyFMk8RpzddiiDONrZMdMaBIWJy4K/xgWUhQIGUEkRITUqivQHFBooyi8pLf3sH+ekcxqS5ib35p6Te9/Px+M+cs+Pe8/npvn2fb7nnPs9igjMzMyK5qC8CzAzMxuKA8rMzArJAWVmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUFUiaY6kkDS5Qu/3PyU9IemxSrzfeJB0kaQv5V2H1a56bFf1pC4CStLDkk6plW1KagZWAMdFxO+NxzZGS9J7JG3LzouI1RHx0bxqsvHldmXjrS4CqgY1A09GxI68CzGrITXTrirVo8xbXQeUpIMkXSjpV5KelHStpMZ02cChg3MlbUm7/e2Z1x4s6SpJT0nqk3TBQA9C0ldJ/tj/XdKzki7IbPasod5viNpeLekrkvolbZb0t2m9pwA3A0ek733lMK//G0nbJT0q6c/Tz3JMuuw2SR/NrHuepO7M9B9IulnSTkk/l/ThzLLTJP1U0jOSHpF0vqRDgBszNT0r6QhJF0u6OvPaD0raJOk3aQ1zM8seTt/rfkm/lfS/JU1Nlx0m6Tvp63ZK+r6kuv7bLbJabVeSpqd/h/1pfd+RdFRm+W2S/oekO9P28T1Jh6XLpkq6Ov19/EbSPZIOl7RQ0gOZ97hZ0j2Z6e9LOjN9foSk69Pt/1rSxzPrXSzpunQbTwPnSTpRUo+kpyU9LunyUv8NCyMiav4BPAycMsT8TwA/AI4CpgD/CnSly+YAAawHDgbeDOwC5qbLLwFuB6anr78f2DbcNkd6vyFq+wrwLeDQ9LW/ABany96T3dYQrz0VeBxoAQ4B/i3d9jHp8tuAj2bWPw/oTp8fAmwF/gyYDBwPPEFy2ANgO/DO9Pl04IThagIuBq5On78BeA54L9AAXAD8EnhF5vf1I+AIoBHoA5aly/4B6Ehf1wC8E1Def1f1/qjDdjUD+FNgWvr6/wN8M7P8NuBX6d/6wen0JemyvwD+PX3tJOCtwKvS9V4ADkv/th8HHknf/2Dgd+l2DwI2An8HvAL4feAh4P2ZtrYbODNd92DgbuCcdPkrgXfk/Tcz2ke974UuA9ojYltE7CL5R/6Q9u8efyYifhcRPwF+QtIAAD4MrI6IpyJiG/CFErc53PvtI2kS8BHgUxHxTEQ8DHwWOKfEbXwY+HJE9EbEc+nnKtXpwMMR8eWI2BMRPwauB/5Lunw3cJykV6Wf/d4S3/e/Av83Im6OiN3AZSSN6KTMOl+IiEcjYidJY35LZpszgdkRsTsivh9pq7NCqsl2FRFPRsT1EfF8RDwDrALePWi1L0fELyLid8C17P83PINkJ/GliNgYEU+n690DvIsktH4C3AnMB94BPBgRTwJvA5oi4u8j4sWIeIgklD+S2fbdEfHNiNibvu9u4BhJh0XEsxHxg1I+Z5HUe0DNBr6Rdrl/Q7LX/hJweGad7NU8z5PsiUCyp781syz7/ECGe7+sgb2pzZl5m4EjS9zG4No2D7fiEGYDbx/4naS/l7OAgZPGfwqcBmyWdLukPxxFTfvqiIi9aY3ZzzTc7+YfSXpb35P0kKQLR/F5rPpqsl1JmibpX9NDg08DdwCvSYNvpDq+CnwX+JqSw+6XSmpIl91O0nt7V/r8NpLge3c6Dcnv9IhB7fIi9v+dDv5dLSbpzf0sPaR4eimfs0jqPaC2Ah+IiNdkHlMj4pESXrud5BDEgFmDlpezh/8Eyd7P7My8ZpKufym2D6qnedDy50gONQzIXrG0Fbh90O/klRHxlwARcU9EnAG8FvgmyV4ijPx5HyXzeSQprXHEz5Tu7a6IiN8HPggsl3TySK+z3NRqu1oBHAu8PSJeRRIoABrphWnP/zMRcRzJUYPTgf+eLh4cULfz8oDaCvx60O/00Ig4LbuZQdt8MCIWkbTVNcB1Ss4XTxj1FFAN6YnKgcdkkvMaqyTNBpDUJOmMEt/vWuBT6YnTI4GPDVr+OMlx4lGLiJfS918l6dC0vuXA1Qd+5X61nSfpOEnTgE8PWn4f8J/TPcJjSPa0BnwHeIOkcyQ1pI+3SZor6RWSzpL06vQw3dPA3sznnSHp1Qeo6Y8knZzuOa4gOVdw10gfRtLpko5JQ+23JHvje0d4mVVHPbWrQ0nOCf1GyUUfg9vVsNKLId6Y9raeJgnKgb/hu0iC70TgRxGxifRIBkkvDZLzs89IWqnkQpJJklokve0A2zxbUlN6tOI36ewJ1W7qKaBuIPnjGnhcDHwe+DbJoaNnSE7svr3E9/t7YBvwa+D/AdeR/Ic74B+Av0274+ePod42kp7OQ0A3yYUOV5Tywoi4Efgn4FaSQ2O3Dlrlc8CLJI39KuCazGufAd5Hcmz7UZJDFmtITnZDcrz+4fQQxzKSw39ExM+ALuCh9DMfMaimnwNnA2tJ9mT/GPjjiHixhI/0epLf8bMkJ37/JSI2AEi6UdJFJbyHjY+6aVckbepgkr/fHwA3jWK7v0fyWZ4mOeR5O8lhP9LzxPcCmzLt4W5gc6SXvKfhejrJOa1fpzV8CRhuhxCSi6U2SXqW5N/kI+m5KZRcqfjOUdSfC/lcc2VI+kuSP4DBJ00LQVIAr4+IX+Zdi1mpit6ubHzVUw+qoiTNlDRfyXcojiU5ZPWNvOsym8jcriyrJr5tnJNXkHy/43Ukx3e/BvxLngWZ1QC3K9vHh/jMzKyQfIjPrEokzZK0QclQUZskfSKdf7GSYaPuSx+njfReZvWgqj2oww47LObMmVO17ZmNp40bNz4REU2lri9pJjAzIu6VdCjJ0DVnkoye8GxEXFbqe7ktWS0Zri1V9RzUnDlz6OnpqeYmzcaNpNGM0EFEbCf5IioR8YykPkofHWQ/bktWS4ZrSz7EZ5YDSXNIBuL9YTrrY0pGcr9C0vRhXrNUyejUPf39/dUq1Sw3DiizKpP0SpIBeD8ZEU8DXwSOJvkS5naSAUxfJiLWRURrRLQ2NZV8ZNFswnJAmVVROszT9cA1EfF1gIh4PB3hei/JCNUn5lmjWVE4oMyqJB1LsBPoi4jLM/NnZlb7E6C32rWZFZG/qGtWPfNJxjJ8QNJ96byLgEWS3kIyGvXDJDe3M6t7DiizKomIboa+NcMN1a7FbCIY8RBfelXRDkm9g+a3SfpZ+oXDS8evRCtVV1cXLS0tTJo0iZaWFrq6uvIuyWxCclsqhlJ6UFcC/wx8ZWCGpIXAGcCbI2KXpNeOT3lWqq6uLtrb2+ns7GTBggV0d3ezeHFym6dFixblXJ3ZxOG2VCARMeIDmAP0ZqavBU4p5bXZx1vf+taw8TFv3ry49dZb95t36623xrx583KqqPYBPTHKNlCph9vS+HFbqr7h2lJJQx2lXyr8TkS0pNP3Ad8iuSHWC8D5EXHPMK9dCiwFaG5ufuvmzaP68r2VaNKkSbzwwgs0NDTsm7d7926mTp3KSy+9lGNltUvSxohozWPbra2t4ZEkxofbUvUN15bGepn5ZKAReAfwN8C16SW0LxP+cmFVzJ07l+7u7v3mdXd3M3fu3JwqMpuY3JaKY6wBtQ34eto7+xHJfe4Pq1xZNlrt7e0sXryYDRs2sHv3bjZs2MDixYtpb2/PuzSzCcVtqTjGepn5N4GFwAZJbyC5ydgTlSrKRm/g5G1bWxt9fX3MnTuXVatW+aSu2Si5LRXHiOegJHUB7yHpIT0OfBr4KnAFydhhL5Kcg7p1pI35uLnVEp+DMquM4drSiD2oiBhut+HssqsyMzMbhsfiMzOzQnJAmZlZITmgzMyskBxQZmZWSA4oMzMrJAeUmZkVkgPKzMwKyQFlZmaF5IAyM7NCckCZmQ3iO+oWw1gHizUzq0m+o25xuAdlZpaxatUqOjs7WbhwIQ0NDSxcuJDOzk5WrVqVd2l1xwFlZpbR19fHggUL9pu3YMEC+vr6cqqofjmgzMwyfEfd4nBAmZll+I66xeGLJMzMMnxH3eJwQJmZDbJo0SIHUgH4EJ+ZmRWSA8rMzAppxICSdIWkHZJ6h1i2QlJIOmx8yrPRaGtrY+rUqUhi6tSptLW15V2SmdmYldKDuhI4dfBMSbOA9wFbKlyTjUFbWxsdHR2sXr2a5557jtWrV9PR0eGQMrMJa8SAiog7gJ1DLPoccAEQlS7KRm/9+vWsWbOG5cuXM23aNJYvX86aNWtYv3593qWZmY3JmM5BSToDeCQiflLCuksl9Ujq6e/vH8vmrAS7du1i2bJl+81btmwZu3btyqkiM7PyjDqgJE0DLgL+rpT1I2JdRLRGRGtTU9NoN2clmjJlCh0dHfvN6+joYMqUKTlVZGZWnrF8D+po4HXATyQBHAXcK+nEiHisksVZ6ZYsWcLKlSuBpOfU0dHBypUrX9arMjObKEYdUBHxAPDagWlJDwOtEfFEBeuyUVq7di0AF110EStWrGDKlCksW7Zs33wzs4mmlMvMu4C7gWMlbZO0ePzLsrFYu3YtL7zwAhHBCy+84HAyswltxB5URBxwvI+ImFOxaszMzFIeScKsSiTNkrRB0k8lbZL0iXR+o6SbJT2Y/pyed61mReCAMquePcCKiDgOeAfwV5KOAy4EbomI1wO3pNNmdc8BZVYlEbE9Iu5Nnz8D9AFHAmcAV6WrXQWcmUuBZgXjgDLLgaQ5wPHAD4HDI2J7uugx4PBhXuMvvVtdcUCZVZmkVwLXA5+MiKezyyIiGGb4MH/p3eqNA6qGzJgxA0n7HjNmzMi7JBtEUgNJOF0TEV9PZz8uaWa6fCawI6/6zIrEAVUjZsyYwc6dO5k3bx6bN29m3rx57Ny50yFVIEqGXukE+iLi8syibwPnps/PBb5V7drMisi3fK8RA+HU25vctqu3t5eWlhY2bdqUc2WWMR84B3hA0n3pvIuAS4Br0y/BbwY+nE95ZsXigKohN9xww8umZ8+enVM1NlhEdAMaZvHJ1azFbCLwIb4actpppx1w2sxK09XVRUtLC5MmTaKlpYWurq68S6pLDqga0djYyKZNm2hpaWHLli37Du81NjbmXZrZhNLV1UV7e/u+sS3Xrl1Le3u7QyoHDqga8eSTT+4LqdmzZ+8LpyeffDLv0swmlFWrVtHZ2cnChQtpaGhg4cKFdHZ2smrVqrxLqzs+B1VDHEZm5evr62PBggX7zVuwYAF9fX05VVS/3IMyM8uYO3cu3d3d+83r7u5m7ty5OVVUvxxQZmYZ7e3tLF68mA0bNrB79242bNjA4sWLaW9vz7u0uuNDfGZmGYsWJbfAa2tro6+vj7lz57Jq1ap98616HFBmZoMsWrTIgVQAPsRnZmaF5IAyMxukra2NqVOnIompU6fS1taWd0l1acSAknSFpB2SejPz/lHSzyTdL+kbkl4zrlWamVVJW1sbHR0drF69mueee47Vq1fT0dHhkMpBKT2oK4FTB827GWiJiDcBvwA+VeG6bAyyt9oYeJjZ6Kxfv541a9awfPlypk2bxvLly1mzZg3r16/Pu7S6M2JARcQdwM5B874XEXvSyR8AR41DbTYKA2EkiZtuumm/aTMr3a5du1i2bNl+85YtW8auXbtyqqh+VeIc1J8DN1bgfaxMkti7dy/vf//72bt3r8PJbAymTJlCR0fHfvM6OjqYMmVKThXVr7ICSlI7sAe45gDrLJXUI6mnv7+/nM3ZCG688cYDTpvZyJYsWcLKlSu5/PLLef7557n88stZuXIlS5Ysybu0uqOIGHklaQ7wnYhoycw7D/gL4OSIeL6UjbW2tkZPT8/YKrUDGjjntHfv3n3zDjroICKCUv6NbfQkbYyI1jy27bY0vtra2li/fj27du1iypQpLFmyhLVr1+ZdVs0ari2NqQcl6VTgAuCDpYaTjb+I4KCDDuK73/3uvnAys9EbuNVGROy75YZVXymXmXcBdwPHStqW3pb6n4FDgZsl3Sep44BvYuNuIIwiglNPPXW/aTOziWjEoY4iYqjxPjrHoRYrk8PIzGqJR5IwMxvEt3wvBg8Wa2aWMXDL987OThYsWEB3dzeLFy8G8ACyVeYelJlZhm/5XhwOKDOzDN/yvTgcUGZmGb7le3E4oMzMMnzL9+LwRRI1ZKix93zpudno+JbvxeEeVI3IhtOaNWuGnG9mpVm0aBG9vb289NJL9Pb2Opxy4oCqMRHBBRdc4J6TmU14Dqgaku05DTVtZqVpbm7e78afzc3NeZdUlxxQNWTlypUHnDazkTU3N7N161ZOOukkHn30UU466SS2bt3qkMqBA6rGSOLSSy/1uSezMRoIpzvvvJOZM2dy55137gspqy4HVI3InnPK9px8Lsps9K677roDTlt1OKBqyMDNCbMPMxu9D33oQwectupwQJmZZcyaNYu77rqL+fPns337dubPn89dd93FrFmz8i6t7viLumZmGVu2bKG5uZm77rqLI444AkhCa8uWLTlXVn8cUGZmgziMisGH+MzMrJAcUGZVIukKSTsk9WbmXSzpEUn3pY/T8qzREg0NDft9UbehoSHvkuqSA8qseq4ETh1i/uci4i3p44Yq12SDNDQ0sGfPHqZPn87999/P9OnT2bNnj0MqByMG1DB7fY2Sbpb0YPpz+viWaaXI7vENPKw4IuIOYGfeddiBDYTTzp07eeMb38jOnTv3hZRVVyk9qCt5+V7fhcAtEfF64JZ02nKUDaPjjz9+yPlWWB+TdH+6Mzjszp6kpZJ6JPX09/dXs766c/vttx9w2qpjxIAaZq/vDOCq9PlVwJmVLcvGKiK49957/SXdieOLwNHAW4DtwGeHWzEi1kVEa0S0NjU1Vam8+vTud7/7gNNWHWM9B3V4RGxPnz8GHD7cit7rq55sz2moaSueiHg8Il6KiL3AeuDEvGuqd5MnT+app56isbGRBx54gMbGRp566ikmT/a3cqqt7IskItlVH3Z33Xt91fPjH//4gNNWPJJmZib/BOgdbl2rjt27d+8LqTe96U37wmn37t15l1Z3xhpQjw80rPTnjsqVZOWQxAknnOBzTwUkqQu4GzhW0jZJi4FLJT0g6X5gIfDXuRZpQBJS2TEtHU75GGuf9dvAucAl6c9vVawiG5OI2BdK2Z6Tz0UVR0QMdd/wzqoXYjZBlHKZ+VB7fZcA75X0IHBKOm0582jmZpXhr2wUw4g9qGH2+gBOrnAtZma5y4bR1Vdfzdlnn71vvnf6qssjSZiZDSEiOOussxxKOXJAmZkNcvXVVx9w2qrDAWVmNsjAYb3hpq06HFBmZkOQxDXXXOMLJHLkgDIzy8iec8r2nHwuqvo8dkcNGWpPz43KbPTcborBPagaMdxhCB+eMLOJyj2oGpPd83M4mY2Nj0YUg3tQZmYZ2XC67LLLhpxv1eGAMjMbQkSwYsUK95xy5ICqMR47zKx82Z7TUNNWHQ6oGjHcXp73/sxG7/zzzz/gtFWHA6qGeDRzs8qRxGc/+1kfjciRA8rMLCO7Y5ftOXmHr/p8mbmZ2SAOo2JwD8rMzArJAWVmZoXkQ3xmZoN4JIlicA/KzCwjG05f+MIXhpxv1VFWQEn6a0mbJPVK6pI0tVKFmZnlKSJoa2tzzylHYw4oSUcCHwdaI6IFmAR8pFKFmZnlJdtzGmraqqPcQ3yTgYMlTQamAY+WX5KZWb4+/vGPH3DaqmPMARURjwCXAVuA7cBvI+J7g9eTtFRSj6Se/v7+sVdq+8mOuTfah5mNTBJr1651m8lROYf4pgNnAK8DjgAOkXT24PUiYl1EtEZEa1NT09grtf0MNaxRdnijkZab2dCybSTbc3Lbqb5yDvGdAvw6IvojYjfwdeCkypRlZpYf79gVQzkBtQV4h6RpSvrAJwN9lSnLzMzqXTnnoH4IXAfcCzyQvte6CtVlZpYbn7sthrKu4ouIT0fEH0RES0ScExG7KlWYmVkesmF09NFHDznfqsNDHZmZDSF73snhlA8PdWRmNki25zTUtFWHA8rMbJBf/epXB5y26nBAmZkNQRLHHHOMD+/lyAFlZpaRPfeU7Tn5u1DV54skzMwGcRgVg3tQZmZWSA4osyqRdIWkHZJ6M/MaJd0s6cH05/Q8azQrEgeUWfVcCZw6aN6FwC0R8XrglnTazHBAmVVNRNwB7Bw0+wzgqvT5VcCZ1azJrMh8kYRZvg6PiO3p88eAw4dbUdJSYClAc3NzFUqrD2O9jNwXUow/96DMCiKS//GG/V/P91YbH2O9t5qNPweUWb4elzQTIP25I+d6zArDAWWWr28D56bPzwW+lWMtZoXigDKrEkldwN3AsZK2SVoMXAK8V9KDJHepviTPGs2KxBdJmFVJRCwaZtHJVS3EbIJwD8rMzArJAWVmZoXkgDIzs0IqK6AkvUbSdZJ+JqlP0h9WqjAzM6tv5V4k8Xngpoj4kKRXANMqUJOZmdnYA0rSq4F3AecBRMSLwIuVKcvMzOpdOYf4Xgf0A1+W9GNJX5J0yOCVJC2V1COpp7+/v4zNmZlZPSknoCYDJwBfjIjjgecY4lYBHj/MzMzGopyA2gZsi4gfptPXkQSWmZlZ2cYcUBHxGLBV0rHprJOBn1akKjMzq3vlXsXXBlyTXsH3EPBn5ZdkZmZWZkBFxH1Aa2VKMTMz+w8eScLMzArJAWVmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCskBVWCNjY1IGvUDGPVrGhsbc/60Zmb7K3csPhtHTz31FBFRlW0NBJuZWVG4B2VmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCskBZWY1z6OyTExljyQhaRLQAzwSEaeXX5KZWWV5VJaJqRI9qE8AfRV4HzMzs33KCihJRwF/BHypMuWYmZklyj3E90/ABcChw60gaSmwFKC5ubnMzdWX+PSr4OJXV29bZmYFMuaAknQ6sCMiNkp6z3DrRcQ6YB1Aa2trdQ4C1wh95umqHjePi6uyKTOzkpTTg5oPfFDSacBU4FWSro6IsytTmln9kPQw8AzwErAnIlrzrcgsf2M+BxURn4qIoyJiDvAR4FaHk1lZFkbEWxxOZgl/D8rMzAqpIgEVEbf5O1BmZQnge5I2phcWvYykpZJ6JPX09/dXuTyz6nMPyqwYFkTECcAHgL+S9K7BK0TEuohojYjWpqam6ldoVmUOKLMCiIhH0p87gG8AJ+ZbkVn+HFBmOZN0iKRDB54D7wN6863KLH9lj8VnZmU7HPhGOobbZODfIuKmfEsyy58DyixnEfEQ8Oa86zArGh/iMzOzQnJAmZlZITmgzMyskHwOquCqdfOz6dOnV2U7ZmalckAV2FhHMpdUtVHQzSYC37pmYnJAmVnN861rJiafgzIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCsnfgzKzuuBRWSaeMQeUpFnAV0juZRPAuoj4fKUKMzOrFI/KMjGV04PaA6yIiHvTu4FulHRzRPy0QrWZmVkdG/M5qIjYHhH3ps+fAfqAIytVmJmZ1beKXCQhaQ5wPPDDIZYtldQjqae/v78SmzMzszpQdkBJeiVwPfDJiHh68PKIWBcRrRHR2tTUVO7mzMysTpQVUJIaSMLpmoj4emVKMjMzKyOglFyz2Qn0RcTllSvJzMysvB7UfOAc4D9Jui99nFahuszMrM6N+TLziOgGqvPNNzMzqzse6sjMzArJAWVmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZLvqDtBjXR30AMt9w3YzP7DWNuS29H4c0BNUG4cZpXhtlRcPsRnZmaF5IAyM7NCckCZmVkhOaDMzKyQHFBmBSDpVEk/l/RLSRfmXY9ZETigzHImaRLwv4APAMcBiyQdl29VZvlzQJnl70TglxHxUES8CHwNOCPnmsxy54Ayy9+RwNbM9LZ03n4kLZXUI6mnv7+/asWZ5cUBZTZBRMS6iGiNiNampqa8yzEbd1UdSWLjxo1PSNpczW3WqcOAJ/Iuog7MrtD7PALMykwflc4blttS1bgtVceQbUke5qP2SOqJiNa867DSSJoM/AI4mSSY7gH+W0RsyrUwc1vKmcfiM8tZROyR9DHgu8Ak4AqHk5kDyqwQIuIG4Ia86zArEl8kUZvW5V2AWY1wW8qRz0GZmVkhuQdlZmaF5IAyM7NCckDVEElXSNohqTfvWswmMrelYnBA1ZYrgVPzLsKsBlyJ21LuHFA1JCLuAHbmXYfZROe2VAwOKDMzKyQHlJmZFZIDyszMCskBZWZmheSAqiGSuoC7gWMlbZO0OO+azCYit6Vi8FBHZmZWSO5BmZlZITmgzMyskBxQZmZWSA4oMzMrJAeUmZkVkgPKzMwKyQFlZmaF9P8BeyN1VzX9IgsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeCklEQVR4nO3de7xXdZ3v8ddb8JIXRAUJubTRsIYuoofU1OOYppJUmFlpZegwhzpHU09OEzqVlWORpR6d0rJkpHIkUzNSUsnErirgBQV03CklBIJ5w0wT+Zw/1nfrarP3XgvY6/dbm9/7+Xj8Hr+1vuv22Sh89veyvl9FBGZmZj3ZotkBmJlZ/TlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmY1IOkjkm5pdhxm3XGysJYn6URJ90t6XtJKSZdI2rHC57VJCkn9O8oi4sqIOKKqZ5ptKicLa2mSzgC+Cnwa2BHYH2gDbpG0ZRNDM6sVJwtrWZIGAF8EPhkRN0XESxGxFPggsDvwYUlXSPr33DWHSFqW299N0rWSVkt6VNKpuWP7Spov6VlJj0u6IB36Zfp+WtJzkt6eaje/zl17gKR5kp5J3wfkjs2VdI6k30haI+kWSYPSsW0k/UDSnyU9na4d0vt/etZqnCyslR0AbANcly+MiOeA2UCPzUKStgB+CtwHDAMOA06XdGQ65SLgoogYAOwBXJ3KD07fAyNi+4j4Xaf77gzcCFwM7AJcANwoaZfcaR8GTgJ2BbYC/iWVTyKrIY1I134C+GuPfwpmJThZWCsbBDwREWu7OLYCGFxw/duAwRHxpYj4W0Q8AnwHOC4dfwl4vaRBEfFcRNxRMq4JwMMR8f2IWBsRVwEPAu/JnfOfEfHfEfFXsiQ0NvfMXYDXR8TLEbEgIp4t+VyzbjlZWCt7AhiU72jOGZqO9+R1wG6puedpSU8DZwEdzT6TgT2BB1Nz0LtLxrUb8IdOZX8gq710WJnbfh7YPm1/H7gZmCnpT5LOc9+L9QYnC2tlvwNeBI7JF0raHngXMBf4C7Bt7vBrc9uPAY9GxMDcZ4eIOAogIh6OiOPJmoq+ClwjaTugaF2AP5EloryRwPKiHyj1u3wxIsaQNbO9G/hY0XVmRZwsrGVFxDNkHdz/IWm8pC0ltZE16zwBXAncCxwlaWdJrwVOz93iLmCNpM9Ieo2kfpLeLOltAJI+KmlwRKwDnk7XrANWp+/duwltNrCnpA9L6i/pQ8AY4Iain0nSOyS9RVI/4FmyZql1Zf9MzLrjZGEtLSLOI2s6+jqwBniUrCbxzoj4C1mzzn3AUuAW4Ie5a18m+819bLruCeC7ZB3MAOOBRZKeI+vsPi4i/hoRzwPnAr9JzVf7d4rpz+m+ZwB/Bv4VeHdEFDWLQVbzuYYsUSwBbk8/g9kmkVfKM3uVpJOALwEHRsQfmx2PWV04WZh1IukE4KWImNnsWMzqwsnCzMwKuc/CzMwKdTW+vM8bNGhQtLW1NTsMM7M+ZcGCBU9ERJcvo26WyaKtrY358+c3Owwzsz5FUueXQV/hZigzMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyu0Wb7Bvblqm3pjt8eWTpvQwEjMrNW4ZmFmZoUqSxaStpF0l6T7JC2S9MVUPkrSnZLaJf1Q0lapfOu0356Ot+XudWYqf0jSkVXFbGZmXauyZvEicGhE7EW27OT4tHzkV4ELI+L1wFPA5HT+ZOCpVH5hOg9JY4DjgDeRLVN5SVpf2MzMGqSyZBGZ59LulukTwKFkawQDzACOTtsT0z7p+GGSlMpnRsSLEfEo0A7sW1XcZma2vkr7LCT1k3QvsAqYA/weeDoi1qZTlgHD0vYw4DGAdPwZYJd8eRfX5J81RdJ8SfNXr15dwU9jZta6Kk0WEfFyRIwFhpPVBt5Y4bMui4hxETFu8OAu1+4wM7ON1JDRUBHxNHAb8HZgoKSOIbvDgeVpezkwAiAd3xH4c768i2vMzKwBqhwNNVjSwLT9GuBwYAlZ0jg2nTYJ+EnanpX2Scd/ERGRyo9Lo6VGAaOBu6qK28zM1lflS3lDgRlp5NIWwNURcYOkxcBMSf8O3ANcns6/HPi+pHbgSbIRUETEIklXA4uBtcDJEfFyhXGbmVknlSWLiFgI7N1F+SN0MZopIl4APtDNvc4Fzu3tGM3MrBy/wW1mZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMytUWbKQNELSbZIWS1ok6bRU/gVJyyXdmz5H5a45U1K7pIckHZkrH5/K2iVNrSpmMzPrWv8K770WOCMi7pa0A7BA0px07MKI+Hr+ZEljgOOANwG7AT+XtGc6/E3gcGAZME/SrIhYXGHsZmaWU1myiIgVwIq0vUbSEmBYD5dMBGZGxIvAo5LagX3TsfaIeARA0sx0rpOFmVmDVFmzeIWkNmBv4E7gQOAUSR8D5pPVPp4iSyR35C5bxqvJ5bFO5ft18YwpwBSAkSNH9vJPsPlrm3pjt8eWTpvQwEjMrI4q7+CWtD1wLXB6RDwLXArsAYwlq3mc3xvPiYjLImJcRIwbPHhwb9zSzMySSmsWkrYkSxRXRsR1ABHxeO74d4Ab0u5yYETu8uGpjB7KzcysAaocDSXgcmBJRFyQKx+aO+19wANpexZwnKStJY0CRgN3AfOA0ZJGSdqKrBN8VlVxm5nZ+qqsWRwInADcL+neVHYWcLyksUAAS4GPA0TEIklXk3VcrwVOjoiXASSdAtwM9AOmR8SiCuM2M7NOqhwN9WtAXRya3cM15wLndlE+u6frzMysWn6D28zMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVqgwWUg6TdIAZS6XdLekIxoRnJmZ1UOZmsU/pdlijwB2IpvCY1qlUZmZWa2USRYdU3YcBXw/zcvU1TQeZma2mSqTLBZIuoUsWdyclkhdV21YZmZWJ2UmEpxMtlDRIxHxvKRdgJMqjcrMzGqlTM0igDHAqWl/O2CbyiIyM7PaKZMsLgHeDhyf9tcA36wsIjMzq50yzVD7RcQ+ku4BiIin0op1ZmbWIsrULF6S1I+sOQpJg3EHt5lZSymTLC4GfgzsKulc4NfAlyuNyszMaqWwGSoirpS0ADiM7P2KoyNiSeWRmZlZbXSbLCTtnNtdBVyVPxYRT1YZmJmZ1UdPNYsFZP0UXb2tHcDulURkZma1022yiIhRjQzEzMzqq8zQWSQdAxxEVqP4VURcX2VQZmZWL2WmKL8E+ARwP/AA8AlJfinPzKyFlKlZHAr8Q0R0vGcxA1hUaVR9WNvUG7s9tnTahAZGYmbWe8q8Z9EOjMztj0hlPZI0QtJtkhZLWiTptFS+s6Q5kh5O3zulckm6WFK7pIWS9snda1I6/2FJkzbsRzQzs01VJlnsACyRNFfSXGAxMEDSLEmzerhuLXBGRIwB9gdOljQGmArcGhGjgVvTPsC7gNHpMwW4FF4Zwns2sB+wL3B2R4IxM7PGKNMM9fmNuXFErABWpO01kpYAw4CJwCHptBnAXOAzqfx7qbnrDkkDJQ1N587peK9D0hxgPLn3PszMrFpl3uC+HUDSgPz5G/JSnqQ2YG/gTmBISiQAK4EhaXsY8FjusmWprLvyzs+YQlYjYeTIkZ0Pm5nZJigzGmqKpJXAQmA+2ct688s+QNL2wLXA6Wkt71ekWkRsUMTdiIjLImJcRIwbPHhwb9zSzMySMs1QnwbeHBFPbOjNJW1JliiujIjrUvHjkoZGxIrUzLQqlS8n6zzvMDyVLefVZquO8rkbGouZmW28Mh3cvwee39AbSxJwObAkIi7IHZoFdIxomgT8JFf+sTQqan/gmdRcdTNwhKSdUsf2EanMzMwapEzN4kzgt5LuBF7sKIyIU7u/BIADgROA+yXdm8rOAqYBV0uaDPwB+GA6Nhs4imxY7vOkdb4j4klJ5wDz0nlf8iSGZmaNVSZZfBv4Bdkb3KUXPYqIX9P1JISQTXfe+fwATu7mXtOB6WWfbWZmvatMstgyIj5VeSRmZlZbZfosfpZGRA1Nb1/v3GmtCzMz28yVqVkcn77PzJV5PQszsxZS5qU8r2thZtbiyq5n8WZgDLBNR1lEfK+qoMzMrF4Kk4Wks8leihtDNrz1XcCvAScLM7MWUaaD+1iyoa4rI+IkYC9gx0qjMjOzWimTLP4aEeuAtWkywVX8/bQcZma2mSvTZzFf0kDgO2STCD4H/K7KoMzMrF7KjIb6P2nzW5JuAgZExMJqwzIzszopM0X5gZK2S7sHASdKel21YZmZWZ2U6bO4FHhe0l7AGWSz0HoklJlZCymTLNamSf4mAt+IiG+SrcttZmYtokwH9xpJZwIfBQ6WtAWwZbVhmZlZnZSpWXyIbB2LyRGxkmyluq9VGpWZmdVKmdFQK4ELcvt/xH0WZmYtpUzNwszMWpyThZmZFeo2WUi6NX1/tXHhmJlZHfXUZzFU0gHAeyXNpNN62hFxd6WRmZlZbfSULD4PfI5s9NMFnY4FcGhVQZmZWb10mywi4hrgGkmfi4hzGhiT9TFtU2/s9tjSaRMaGImZVaXM0NlzJL0XODgVzY2IG6oNy8zM6qTMRIJfAU4DFqfPaZK+XHVgZmZWH2Wm+5gAjE0LICFpBnAPcFaVgZmZWX2Ufc9iYG671JKqkqZLWiXpgVzZFyQtl3Rv+hyVO3ampHZJD0k6Mlc+PpW1S5paMl4zM+tFZWoWXwHukXQb2fDZg4Ey/2hfAXyD9acGuTAivp4vkDQGOA54E7Ab8HNJe6bD3wQOB5YB8yTNiojFJZ5vZma9pEwH91WS5gJvS0WfSfNFFV33S0ltJeOYCMyMiBeBRyW1A/umY+0R8QhAet9jIlnfiZmZNUipZqiIWBERs9KnMFEUOEXSwtRMtVMqGwY8ljtnWSrrrnw9kqZImi9p/urVqzcxRDMzy2v03FCXAnsAY4EVwPm9deOIuCwixkXEuMGDB/fWbc3MjHJ9Fr0mIh7v2Jb0HaDjfY3lwIjcqcNTGT2Um5lZg/RYs5DUT9KDvfUwSUNzu+8DOkZKzQKOk7S1pFHAaOAuYB4wWtIoSVuRdYLP6q14zMysnB5rFhHxchq2OjItelSapKuAQ4BBkpYBZwOHSBpLNrfUUuDj6TmLJF1N1nG9Fjg5Il5O9zkFuBnoB0yPiEUbEoeZmW26Ms1QOwGLJN0F/KWjMCLe29NFEXF8F8WX93D+ucC5XZTPBmaXiNPMzCpSJll8rvIozMys1sq8Z3G7pNcBoyPi55K2JWsSMjOzFlFmIsH/BVwDfDsVDQOurzAmMzOrmTLvWZwMHAg8CxARDwO7VhmUmZnVS5lk8WJE/K1jR1J/stFMZmbWIsoki9slnQW8RtLhwI+An1YblpmZ1UmZZDEVWA3cT/ZexGzgs1UGZWZm9VJmNNS6tODRnWTNTw9FhJuhzMxaSGGykDQB+Bbwe7L1LEZJ+nhE/Kzq4MzMrB7KvJR3PvCOiGgHkLQHcCPgZGFm1iLK9Fms6UgUySPAmoriMTOzGuq2ZiHpmLQ5X9Js4GqyPosPkM0Ga2ZmLaKnZqj35LYfB/4xba8GXlNZRGZmVjvdJouIOKmRgZiZWX2VGQ01Cvgk0JY/v2iKcjMz23yUGQ11Pdk6FD8F1lUajZmZ1VKZZPFCRFxceSRmZlZbZZLFRZLOBm4BXuwojIi7K4vKzMxqpUyyeAtwAnAorzZDRdo3M7MWUCZZfADYPT9NuZmZtZYyb3A/AAysOA4zM6uxMjWLgcCDkubx930WHjprZtYiyiSLsyuPwszMaq3Meha3NyIQMzOrr8I+C0lrJD2bPi9IelnSsyWumy5plaQHcmU7S5oj6eH0vVMql6SLJbVLWihpn9w1k9L5D0uatLE/qJmZbbzCZBERO0TEgIgYQDaB4PuBS0rc+wpgfKeyqcCtETEauDXtA7wLGJ0+U4BLIUsuZM1g+wH7Amd3JBgzM2ucMqOhXhGZ64EjS5z7S+DJTsUTgRlpewZwdK78e+n+dwADJQ1Nz5kTEU9GxFPAHNZPQGZmVrEyEwkek9vdAhgHvLCRzxsSESvS9kpgSNoeBjyWO29ZKuuuvKs4p5DVShg5cuRGhmdmZl0pMxoqv67FWmApWU1gk0RESIpNvU/ufpcBlwGMGzeu1+5rZmblRkP15roWj0saGhErUjPTqlS+HBiRO294KlsOHNKpfG4vxmNmZiX0tKzq53u4LiLinI143ixgEjAtff8kV36KpJlkndnPpIRyM/DlXKf2EcCZG/FcMzPbBD3VLP7SRdl2wGRgF6DHZCHpKrJawSBJy8hGNU0DrpY0GfgD8MF0+mzgKKAdeB44CSAinpR0Dq+u+f2liOjcaW5mZhXraVnV8zu2Je0AnEb2j/hM4Pzurstdf3w3hw7r4twATu7mPtOB6UXPs81T29Qbuz22dNqEBkZi1tp67LNI7zl8CvgI2VDXfdIQVjMzayE99Vl8DTiGbITRWyLiuYZFZWZmtdLTS3lnALsBnwX+lJvyY02Z6T7MzGzz0VOfxQa93W1mZpsvJwQzMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFerf7ADMmqFt6o09Hl86bUKDIjHrG5pSs5C0VNL9ku6VND+V7SxpjqSH0/dOqVySLpbULmmhpH2aEbOZWStrZjPUOyJibESMS/tTgVsjYjRwa9oHeBcwOn2mAJc2PFIzsxZXpz6LicCMtD0DODpX/r3I3AEMlDS0CfGZmbWsZvVZBHCLpAC+HRGXAUMiYkU6vhIYkraHAY/lrl2WylbkypA0hazmwciRIzcpuJ7as92WbWatqFnJ4qCIWC5pV2COpAfzByMiUiIpLSWcywDGjRu3QdeamVnPmtIMFRHL0/cq4MfAvsDjHc1L6XtVOn05MCJ3+fBUZmZmDdLwZCFpO0k7dGwDRwAPALOASem0ScBP0vYs4GNpVNT+wDO55iozM2uAZjRDDQF+LKnj+f8VETdJmgdcLWky8Afgg+n82cBRQDvwPHBS40M2M2ttDU8WEfEIsFcX5X8GDuuiPICTGxCamZl1o05DZ83MrKacLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhL35kthE82aS1GtcszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkV8nQfZjXS0zQi4KlErHlcszAzs0JOFmZmVsjJwszMCjlZmJlZIXdwm7UId57bpnDNwszMCvWZZCFpvKSHJLVLmtrseMzMWkmfaIaS1A/4JnA4sAyYJ2lWRCxubmRmrcNLyba2PpEsgH2B9oh4BEDSTGAi4GRh1sdV2Zfifpreo4hodgyFJB0LjI+If077JwD7RcQpuXOmAFPS7huAhxoeaPcGAU80O4gCdY+x7vFB/WOse3xQ/xjrHh9sWoyvi4jBXR3oKzWLQhFxGXBZs+PoiqT5ETGu2XH0pO4x1j0+qH+MdY8P6h9j3eOD6mLsKx3cy4ERuf3hqczMzBqgrySLecBoSaMkbQUcB8xqckxmZi2jTzRDRcRaSacANwP9gOkRsajJYW2IWjaPdVL3GOseH9Q/xrrHB/WPse7xQUUx9okObjMza66+0gxlZmZN5GRhZmaFnCwqJGmEpNskLZa0SNJpzY6pK5L6SbpH0g3NjqUrkgZKukbSg5KWSHp7s2PKk/R/03/fByRdJWmbGsQ0XdIqSQ/kynaWNEfSw+l7pxrG+LX033mhpB9LGlin+HLHzpAUkgY1I7ZcHF3GKOmT6c9xkaTzeuNZThbVWgucERFjgP2BkyWNaXJMXTkNWNLsIHpwEXBTRLwR2IsaxSppGHAqMC4i3kw2AOO45kYFwBXA+E5lU4FbI2I0cGvab6YrWD/GOcCbI+KtwH8DZzY6qJwrWD8+JI0AjgD+2OiAunAFnWKU9A6yGS72iog3AV/vjQc5WVQoIlZExN1pew3ZP3LDmhvV35M0HJgAfLfZsXRF0o7AwcDlABHxt4h4uqlBra8/8BpJ/YFtgT81OR4i4pfAk52KJwIz0vYM4OhGxtRZVzFGxC0RsTbt3kH2TlVTdPNnCHAh8K9A00cHdRPj/wamRcSL6ZxVvfEsJ4sGkdQG7A3c2eRQOvt/ZP/jr2tyHN0ZBawG/jM1lX1X0nbNDqpDRCwn+83tj8AK4JmIuKW5UXVrSESsSNsrgSHNDKaEfwJ+1uwg8iRNBJZHxH3NjqUHewL/U9Kdkm6X9LbeuKmTRQNI2h64Fjg9Ip5tdjwdJL0bWBURC5odSw/6A/sAl0bE3sBfaH7zyStSu/9EsqS2G7CdpI82N6pikY2Zb/pvxt2R9G9kzbhXNjuWDpK2Bc4CPt/sWAr0B3Yma/r+NHC1JG3qTZ0sKiZpS7JEcWVEXNfseDo5EHivpKXATOBQST9obkjrWQYsi4iOGtk1ZMmjLt4JPBoRqyPiJeA64IAmx9SdxyUNBUjfvdI80dsknQi8G/hI1OtFsD3Ifim4L/2dGQ7cLem1TY1qfcuA6yJzF1mrwSZ3xDtZVChl88uBJRFxQbPj6SwizoyI4RHRRtYp+4uIqNVvxRGxEnhM0htS0WHUa2r6PwL7S9o2/fc+jBp1wHcyC5iUticBP2liLF2SNJ6sWfS9EfF8s+PJi4j7I2LXiGhLf2eWAfuk/0fr5HrgHQCS9gS2ohdmynWyqNaBwAlkv7Hfmz5HNTuoPuiTwJWSFgJjgS83N5xXpRrPNcDdwP1kf6eaPiWEpKuA3wFvkLRM0mRgGnC4pIfJakTTahjjN4AdgDnp78u3ahZfrXQT43Rg9zScdiYwqTdqaJ7uw8zMCrlmYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycL6NEnPVXz/09Obu5v8PElbS/p5GhL6od6JcKPiOFHSN5r1fOubnCzMenY62eSAvWFvgIgYGxE/7KV7mjWEk4VtdiTtIekmSQsk/UrSG1P5FZIulvRbSY9IOjaVbyHpkjT//xxJsyUdK+lUsvmebpN0W+7+50q6T9IdktabjC+tG3F9WpPhDklvlbQr8APgbalmsUena05Vtu7JQkkzU9m+kn6XJlD8bcdb7KlmcH2KdamkUyR9Kp13h6Sd03lzJV2UnveApH27iHWwpGslzUufA1P5P+ZeJL1H0g6981/H+qyI8MefPvsBnuui7FZgdNrej2waE8jm/v8R2S9JY4D2VH4sMDuVvxZ4Cjg2HVsKDMrdO4D3pO3zgM928fz/AM5O24cC96btQ4Abuvk5/gRsnbYHpu8BQP+0/U7g2rR9ItBO9qbzYOAZ4BPp2IVkE1YCzAW+k7YPBh7IXf+NtP1fwEFpeyTZ1DQAPwUOTNvbd8ThT+t++m9AXjGrvTTD7wHAj3ITbW6dO+X6iFgHLM7VCg4CfpTKV+ZrEV34G9CxouAC4PAuzjkIeD9ARPxC0i6SBhSEvpBsSpPryeb2AdgRmCFpNFmS2jJ3/m2RrZGyRtIzZP+4QzblyFtz512V4vilpAFaf+W5dwJjcn9WA9Kf4W+ACyRdSTYp3bKC+G0z52Rhm5stgKcjYmw3x1/MbW/MtM0vRUTHHDkv03t/hyaQ/fb/HuDfJL0FOIcsKbxP2Xooc3Pn53+Odbn9dZ1i6jyfT+f9LYD9I+KFTuXTJN0IHAX8RtKREfHghv1Itjlxn4VtViJbL+RRSR+AbOZfSXsVXPYb4P2p72IIWXNRhzVkzT0b4lfAR9LzDwGeiB7WMZG0BTAiIm4DPkNWo9g+fS9Pp524gTF0+FB6xkFkCzM90+n4LWQTNXbEMjZ97xHZLKtfBeYBb9zI59tmwsnC+rpt02ybHZ9Pkf1DPVnSfcAissWJenIt2XTTi8k6oe8m6weAbAbZmwqapjr7AvA/lM2SO41XpwXvTj/gB5LuB+4BLo5s6djzgK9IuoeNr8G8kK7/FtDVrKmnAuNSx/pi4BOp/PTUKb4QeImarVhnjedZZ83I+joi4jlJuwB3kXXu1m2dgg0iaS7wLxExv9mxWN/nPguzzA2p83cr4Jy+nijMeptrFmZmVsh9FmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaF/j/u1rDjB60EpAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEWCAYAAACe8xtsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcbklEQVR4nO3de7RdZXnv8e8v4aICIQFCGpJAIsZq1BI4EWjJAYrlPjSogFArgaaNjgMFhp4eA0eFlkZjFThFLhoGOQQLxAgCEaIQLglgyyUBDLnAYRtCk5gb9yCCJnnOH/NdMNnsvfacYc+91t7r9xljjfXOd96ePcciD+/7zvlORQRmZmZl9Gt0AGZm1vs4eZiZWWlOHmZmVpqTh5mZlebkYWZmpTl5mJlZaU4eZmZWmpOHWSckzZf0kqQdGx2LWbNx8jDrgKSRwH8HAvhMY6PpmKTtGh2DtS4nD7OOnQY8BFwLTKxVSrpW0hWS7pC0SdLDkvZN6yTpUkkbJL0q6UlJH5c0StLLkvql7a6WtCF3zB9LOjeVd5V0jaS1ktZI+hdJ/dO60yX9Kp3jBeBCSR+StEDSK5Kel/STnrpA1tqcPMw6dhpwffocLWlIbt0pwD8Bg4A2YGqqPwo4FPgwsCtwMvBCRDwLvArsn7Y7FHhN0kfT8mHAglS+FtgMfChtfxTwd7lzHwSsAIak814E3JViGQ784L392WbFOHmYtSNpPLAPMDsiFgG/Af46t8ktEfFIRGwmSy5jU/0fgV2AjwCKiOURsTatWwAcJulP0vJNaXkUMAD4dUpQxwHnRsTvImIDcClZsqr5bUT8ICI2R8Tv0zn3AfaKiDci4sHuvBZmnXHyMHu3icBdEfF8Wr6BXNcVsC5Xfh3YGSAi7gUuB64ANkiaLmlA2m4BcDhZq+N+YD5Zi+Mw4IGI2EqWBLYH1qZurpeBHwF75s63ql2s/wsQ8IikpZL+dhv/ZrNSPOBmliPp/WTdTf0l1ZLEjsBASft1tX9EXAZcJmlPYDbwj8A3yZLH94DVqfwg8EPgDd7usloFvAnskVo1HZ6i3fnWAX+fYh8P3C3p/ohoK/YXm20btzzM3ukEYAswhqw7aizwUeABsnGQTkn6pKSDJG0P/I4sMWwFiIhngN8DfwMsiIhXgfXA50nJI3Vx3QVcLGmApH6S9pV0WJ1zniRpeFp8iSy5bC3/Z5uV4+Rh9k4Tgf8bEf8VEetqH7LuqC9Sv7U+ALia7B/x54AXyFobNQvIBtBX5ZYFPJbb5jRgB2BZOs5NwNA65/wk8LCk14A5wDkRsaLQX2r2HsgvgzIzs7Lc8jAzs9KcPMzMrDQnDzMzK83Jw8zMSuuTz3nsscceMXLkyEaHYWbWqyxatOj5iBhcZNs+mTxGjhzJwoULGx2GmVmvIum5otu628rMzEpz8jAzs9KcPMzMrDQnDzMzK83Jw8zMSnPyMDOz0pw8zMysNCcPMzMrzcnDzMxK65NPmPdVI6fc0em6ldOO78FIzKzVueVhZmalVZY8JL1P0iOSfi1pqaR/SvWjJD0sqU3STyTtkOp3TMttaf3I3LHOS/VPSzq6qpjNzKyYKlsebwJHRMR+wFjgGEkHA98FLo2ID5G9o3lS2n4S8FKqvzRth6QxwCnAx4BjgCsl9a8wbjMz60JlySMyr6XF7dMngCOAm1L9TOCEVJ6QlknrPyVJqX5WRLwZEc8CbcCBVcVtZmZdq3TMQ1J/SU8AG4B5wG+AlyNic9pkNTAslYcBqwDS+leA3fP1HeyTP9dkSQslLdy4cWMFf42ZmdVUmjwiYktEjAWGk7UWPlLhuaZHxLiIGDd4cKF3mZiZ2TbqkbutIuJl4D7gz4GBkmq3CA8H1qTyGmAEQFq/K/BCvr6DfczMrAGqvNtqsKSBqfx+4EhgOVkSOTFtNhG4LZXnpGXS+nsjIlL9KelurFHAaOCRquI2M7OuVfmQ4FBgZrozqh8wOyJul7QMmCXpX4DHgWvS9tcAP5bUBrxIdocVEbFU0mxgGbAZODMitlQYt5mZdaGy5BERi4H9O6hfQQd3S0XEG8BJnRxrKjC1u2M0M7Nt4yfMzcysNCcPMzMrzcnDzMxKc/IwM7PSnDzMzKw0Jw8zMyvNycPMzEpz8jAzs9KcPMzMrDQnDzMzK83Jw8zMSnPyMDOz0pw8zMysNCcPMzMrzcnDzMxKc/IwM7PSnDzMzKw0Jw8zMyvNycPMzEpz8jAzs9KcPMzMrDQnDzMzK83Jw8zMSnPyMDOz0ipLHpJGSLpP0jJJSyWdk+ovlLRG0hPpc1xun/MktUl6WtLRufpjUl2bpClVxWxmZsVsV+GxNwNfi4jHJO0CLJI0L627NCK+n99Y0hjgFOBjwF7A3ZI+nFZfARwJrAYelTQnIpZVGLuZmdVRWfKIiLXA2lTeJGk5MKzOLhOAWRHxJvCspDbgwLSuLSJWAEialbZ18jAza5AeGfOQNBLYH3g4VZ0labGkGZIGpbphwKrcbqtTXWf1ZmbWIJUnD0k7AzcD50bEq8BVwL7AWLKWycXddJ7JkhZKWrhx48buOKSZmXWi0uQhaXuyxHF9RPwMICLWR8SWiNgKXM3bXVNrgBG53Yenus7q3yEipkfEuIgYN3jw4O7/Y8zM7C2VjXlIEnANsDwiLsnVD03jIQCfBZak8hzgBkmXkA2YjwYeAQSMljSKLGmcAvx1VXFXaeSUO+quXznt+B6KxMzsvanybqtDgC8BT0p6ItWdD5wqaSwQwErgywARsVTSbLKB8M3AmRGxBUDSWcCdQH9gRkQsrTBuMzPrQpV3Wz1I1mpob26dfaYCUzuon1tvPzMz61l+wtzMzEpz8jAzs9KcPMzMrLQqB8ytB/lOLjPrSW55mJlZaU4eZmZWmpOHmZmV5uRhZmaldZk8JJ0jaYAy10h6TNJRPRGcmZk1pyItj79Ns+EeBQwim3JkWqVRmZlZUyuSPGpTjBwH/DjNK9XRtCNmZtYiiiSPRZLuIksed6ZXym6tNiwzM2tmRR4SnET24qYVEfG6pN2BMyqNyszMmlqRlkcAY4Cz0/JOwPsqi8jMzJpekeRxJfDnwKlpeRNwRWURmZlZ0yvSbXVQRBwg6XGAiHhJ0g4Vx2VmZk2sSMvjj5L6k3VfIWkwHjA3M2tpRZLHZcAtwJ6SpgIPAt+uNCozM2tqXXZbRcT1khYBnyJ7vuOEiFheeWRmZta0Ok0eknbLLW4Absyvi4gXqwzMzMyaV72WxyKycY6OniYP4IOVRGRmZk2v0+QREaN6MhAzM+s9Cr2GVtLngPFkLY4HIuLWKoMyM7PmVmRK9iuBrwBPAkuAr0jyQ4JmZi2sSMvjCOCjEVF7zmMmsLTSqMzMrKkVec6jDdg7tzwi1dUlaYSk+yQtk7RU0jmpfjdJ8yQ9k74HpXpJukxSm6TFkg7IHWti2v4ZSRPL/YlmZtbdiiSPXYDlkuZLmg8sAwZImiNpTp39NgNfi4gxwMHAmZLGAFOAeyJiNHBPWgY4FhidPpOBq+CtW4YvAA4CDgQuqCUcMzNrjCLdVt/algNHxFpgbSpvkrQcGAZMAA5Pm80E5gNfT/XXpe6xhyQNlDQ0bTuv9lyJpHnAMeSeOzEzs55V5AnzBQCSBuS3L/OQoKSRwP7Aw8CQlFgA1gFDUnkYsCq32+pU11l9+3NMJmuxsPfee7dfbWZm3ajI3VaTJa0DFgMLyR4eXFj0BJJ2Bm4Gzk3vQn9LamVEqYg7ERHTI2JcRIwbPHhwdxzSzMw6UaTb6h+Bj0fE82UPLml7ssRxfUT8LFWvlzQ0ItambqkNqX4N2WB8zfBUt4a3u7lq9fPLxmJmZt2nyID5b4DXyx5YkoBrgOURcUlu1RygdsfUROC2XP1p6a6rg4FXUvfWncBRkgalgfKjUp2ZmTVIkZbHecB/SHoYeLNWGRFnd74LAIcAXwKelPREqjsfmAbMljQJeA44Oa2bCxxHdhvw66T3pEfEi5IuAh5N2/2zJ2U0M2usIsnjR8C9ZE+YF34JVEQ8SMeTKkI2vXv77QM4s5NjzQBmFD23mZlVq0jy2D4ivlp5JGZm1msUGfP4Rbrjamh6Ony3du/6MDOzFlOk5XFq+j4vV+f3eZiZtbAiDwn6vR5mZvYORd/n8XFgDPC+Wl1EXFdVUGZm1ty6TB6SLiB7SG8M2e20xwIPAk4eZmYtqsiA+Ylkt9aui4gzgP2AXSuNyszMmlqR5PH7iNgKbE6TI27gndOImJlZiyky5rFQ0kDgarJJEV8D/rPKoMzMrLkVudvqf6TiDyX9EhgQEYurDcvMzJpZkSnZD5G0U1ocD5wuaZ9qwzIzs2ZWZMzjKuB1SfsBXyObZdd3WpmZtbAiyWNzmrRwAnB5RFxB9l5zMzNrUUUGzDdJOg/4G+BQSf2A7asNy8zMmlmRlscXyN7jMSki1pG9ye97lUZlZmZNrcjdVuuAS3LL/4XHPMzMWlqRloeZmdk7OHmYmVlpnSYPSfek7+/2XDhmZtYb1BvzGCrpL4DPSJpFu/eRR8RjlUZmZmZNq17y+BbwTbK7qy5pty6AI6oKyszMmlunySMibgJukvTNiLioB2MyM7MmV+RW3YskfQY4NFXNj4jbqw3LzMyaWZGJEb8DnAMsS59zJH276sDMzKx5FZme5HhgbHohFJJmAo8D51cZmJmZNa+iz3kMzJULvYJW0gxJGyQtydVdKGmNpCfS57jcuvMktUl6WtLRufpjUl2bpCkF4zUzswoVaXl8B3hc0n1kt+seChT5R/xa4HLePZXJpRHx/XyFpDHAKcDHgL2AuyV9OK2+AjgSWA08KmlORCwrcH4zM6tIkQHzGyXNBz6Zqr6e5rvqar/7JY0sGMcEYFZEvAk8K6kNODCta4uIFQDpeZMJZGMvZmbWIIW6rSJibUTMSZ8uE0cXzpK0OHVrDUp1w4BVuW1Wp7rO6t9F0mRJCyUt3Lhx43sM0czM6unpua2uAvYFxgJrgYu768ARMT0ixkXEuMGDB3fXYc3MrANFxjy6TUSsr5UlXQ3UnhdZA4zIbTo81VGn3szMGqRuy0NSf0lPddfJJA3NLX4WqN2JNQc4RdKOkkYBo4FHgEeB0ZJGSdqBbFB9TnfFY2Zm26ZuyyMitqTbZPdOL4EqTNKNwOHAHpJWAxcAh0saSzY31krgy+k8SyXNJhsI3wycGRFb0nHOAu4E+gMzImJpmTjMzKz7Fem2GgQslfQI8LtaZUR8pt5OEXFqB9XX1Nl+KjC1g/q5wNwCcZqZWQ8pkjy+WXkU1nAjp9zR6bqV047vwUjMrDco8pzHAkn7AKMj4m5JHyDrQjIzsxZVZGLEvwduAn6UqoYBt1YYk5mZNbkiz3mcCRwCvAoQEc8Ae1YZlJmZNbciyePNiPhDbUHSdmR3S5mZWYsqkjwWSDofeL+kI4GfAj+vNiwzM2tmRZLHFGAj8CTZcxlzgW9UGZSZmTW3IndbbU0vgHqYrLvq6Yhwt5WZWQvrMnlIOh74IfAbsvd5jJL05Yj4RdXBmZlZcyrykODFwF9GRBuApH2BOwAnDzOzFlVkzGNTLXEkK4BNFcVjZma9QKctD0mfS8WFkuYCs8nGPE4im+3WzMxaVL1uq0/nyuuBw1J5I/D+yiIyM7Om12nyiIgzejIQMzPrPYrcbTUK+AdgZH77rqZkNzOzvqvI3Va3kr2H4+fA1kqjMTOzXqFI8ngjIi6rPBIzM+s1iiSPf5N0AXAX8GatMiIeqywqMzNrakWSxyeALwFH8Ha3VaRlMzNrQUWSx0nAB/PTspuZWWsr8oT5EmBgxXGYmVkvUqTlMRB4StKjvHPMw7fqmpm1qCLJ44LKozAzs16lyPs8FvREIGZm1nt0OeYhaZOkV9PnDUlbJL1aYL8ZkjZIWpKr203SPEnPpO9BqV6SLpPUJmmxpANy+0xM2z8jaeK2/qFmZtZ9ukweEbFLRAyIiAFkEyJ+HriywLGvBY5pVzcFuCciRgP3pGWAY4HR6TMZuAqyZEPWbXYQcCBwQS3hmJlZ4xS52+otkbkVOLrAtvcDL7arngDMTOWZwAm5+uvS8R8CBkoams4zLyJejIiXgHm8OyGZmVkPKzIx4udyi/2AccAb23i+IRGxNpXXAUNSeRiwKrfd6lTXWb2ZmTVQkbut8u/12AysJGspvCcREZLivR6nRtJksi4v9t577+46rJmZdaDI3Vbd+V6P9ZKGRsTa1C21IdWvAUbkthue6tYAh7ern99JnNOB6QDjxo3rtqRkZmbvVu81tN+qs19ExEXbcL45wERgWvq+LVd/lqRZZIPjr6QEcyfw7dwg+VHAedtwXjMz60b1Wh6/66BuJ2ASsDtQN3lIupGs1bCHpNVkd01NA2ZLmgQ8B5ycNp8LHAe0Aa8DZwBExIuSLuLtd6b/c0S0H4Q3M7MeVu81tBfXypJ2Ac4h+0d9FnBxZ/vl9j+1k1Wf6mDbAM7s5DgzgBldnc/MzHpO3TGP9JzFV4Evkt1ae0C6ZdbMzFpYvTGP7wGfIxuE/kREvNZjUZmZWVOr95Dg14C9gG8Av81NUbKpyPQkZmbWd9Ub8yj19LmZmbUOJwgzMyvNycPMzEorMj2JWV0jp9zR6bqV047vwUjMrKe45WFmZqU5eZiZWWlOHmZmVpqTh5mZlebkYWZmpfluq5Lq3VkEvrvIzFqDWx5mZlaak4eZmZXm5GFmZqU5eZiZWWlOHmZmVpqTh5mZlebkYWZmpTl5mJlZaU4eZmZWmpOHmZmV5uRhZmalOXmYmVlpDUkeklZKelLSE5IWprrdJM2T9Ez6HpTqJekySW2SFks6oBExm5nZ2xrZ8vjLiBgbEePS8hTgnogYDdyTlgGOBUanz2Tgqh6P1MzM3qGZuq0mADNTeSZwQq7+usg8BAyUNLQB8ZmZWdKo5BHAXZIWSZqc6oZExNpUXgcMSeVhwKrcvqtT3TtImixpoaSFGzdurCpuMzOjcS+DGh8RayTtCcyT9FR+ZUSEpChzwIiYDkwHGDduXKl9zcysnIa0PCJiTfreANwCHAisr3VHpe8NafM1wIjc7sNTnZmZNUiPJw9JO0napVYGjgKWAHOAiWmzicBtqTwHOC3ddXUw8Eque8vMzBqgEd1WQ4BbJNXOf0NE/FLSo8BsSZOA54CT0/ZzgeOANuB14IyeD9nMzPJ6PHlExApgvw7qXwA+1UF9AGf2QGhmZlZQM92qa2ZmvUSj7rYyA2DklDvqrl857fgeisTMynDLw8zMSnPyMDOz0pw8zMysNCcPMzMrzcnDzMxKc/IwM7PSnDzMzKw0Jw8zMyvNycPMzEpz8jAzs9I8PYk1tXrTl3jqErPGccvDzMxKc/IwM7PSnDzMzKw0Jw8zMyvNycPMzEpz8jAzs9KcPMzMrDQnDzMzK80PCVqf5QcMzarj5GHWgXqJB5x8zNxtZWZmpTl5mJlZab0meUg6RtLTktokTWl0PGZmraxXjHlI6g9cARwJrAYelTQnIpZVcb6u+rvN6nkvv5+uxlI8FmPNolckD+BAoC0iVgBImgVMACpJHmZ90XtNPL57zfIUEY2OoUuSTgSOiYi/S8tfAg6KiLNy20wGJqfFPwWeBvYAnu/hcJuVr0XG1yHj65DxdcjUrsM+ETG4yA69peXRpYiYDkzP10laGBHjGhRSU/G1yPg6ZHwdMr4OmW25Dr1lwHwNMCK3PDzVmZlZA/SW5PEoMFrSKEk7AKcAcxock5lZy+oV3VYRsVnSWcCdQH9gRkQsLbDr9K43aRm+Fhlfh4yvQ8bXIVP6OvSKAXMzM2suvaXbyszMmoiTh5mZldZnk4enM8lIWinpSUlPSFrY6Hh6kqQZkjZIWpKr203SPEnPpO9BjYyxJ3RyHS6UtCb9Lp6QdFwjY+wJkkZIuk/SMklLJZ2T6lvqN1HnOpT6TfTJMY80ncn/IzedCXBqVdOZNDNJK4FxEdFyD0JJOhR4DbguIj6e6v4VeDEipqX/qRgUEV9vZJxV6+Q6XAi8FhHfb2RsPUnSUGBoRDwmaRdgEXACcDot9Juocx1OpsRvoq+2PN6aziQi/gDUpjOxFhIR9wMvtqueAMxM5Zlk/9H0aZ1ch5YTEWsj4rFU3gQsB4bRYr+JOtehlL6aPIYBq3LLq9mGi9NHBHCXpEVpCpdWNyQi1qbyOmBII4NpsLMkLU7dWn26q6Y9SSOB/YGHaeHfRLvrACV+E301edjbxkfEAcCxwJmpC8OAyPps+16/bTFXAfsCY4G1wMUNjaYHSdoZuBk4NyJeza9rpd9EB9eh1G+iryYPT2eSRMSa9L0BuIWsS6+VrU99vrW+3w0NjqchImJ9RGyJiK3A1bTI70LS9mT/YF4fET9L1S33m+joOpT9TfTV5OHpTABJO6UBMSTtBBwFLKm/V583B5iYyhOB2xoYS8PU/rFMPksL/C4kCbgGWB4Rl+RWtdRvorPrUPY30SfvtgJIt5n9H96ezmRqYyPqeZI+SNbagGwqmhta6TpIuhE4nGy66fXABcCtwGxgb+A54OSI6NODyZ1ch8PJuicCWAl8Odfv3ydJGg88ADwJbE3V55P197fMb6LOdTiVEr+JPps8zMysOn2128rMzCrk5GFmZqU5eZiZWWlOHmZmVpqTh5mZlebkYb2apNcqPv65kj7QHeeTtKOku9OMpV/ongi3KY7TJV3eqPNb3+DkYVbfucAHutqooP0BImJsRPykm45p1hBOHtbnSNpX0i/TZJAPSPpIqr9W0mWS/kPSCkknpvp+kq6U9FR6n8NcSSdKOhvYC7hP0n2540+V9GtJD0l61yR66f0Qt6YJ5h6S9GeS9gT+Hfhkanns226fs9P7FRZLmpXqDpT0n5IeTzH/aao/PR1/nrL3tZwl6atpu4ck7Za2my/p39L5lkh613QTkgZLulnSo+lzSKo/LPdeh8drMxWYvSUi/PGn137I3j/Qvu4eYHQqHwTcm8rXAj8l+5+mMWTT9gOcCMxN9X8CvAScmNatBPbIHTuAT6fyvwLf6OD8PwAuSOUjgCdS+XDg9k7+jt8CO6bywPQ9ANgulf8KuDmVTwfagF2AwcArwFfSukvJJroDmA9cncqHAkty+1+eyjeQTZ4J2RPWy1P558AhqbxzLQ5//Kl9tiuRZ8yaXpop9C+An2ZT+ACwY26TWyOb+G1ZrtUwHvhpql+Xb2V04A/A7am8iOyFY+2NBz4PEBH3Stpd0oAuQl8MXC/pVrIpVAB2BWZKGk2WtLbPbX9fZO9i2CTpFbJ/7CGbcuLPctvdmOK4X9IASQPbnfevgDG5azUgXcNfAZdIuh74WUSs7iJ+azFOHtbX9ANejoixnax/M1dWJ9vU88eIqM3ps4Xu+2/oeLLWwaeB/y3pE8BFZEnis+m9C/Nz2+f/jq255a3tYmo//1D75X7AwRHxRrv6aZLuAI4DfiXp6Ih4qtyfZH2ZxzysT4nsvQTPSjoJshlEJe3XxW6/Aj6fxj6GkHUv1Wwi6x4q4wHgi+n8hwPPR7v3RuRJ6geMiIj7gK+TtTh2Tt+1VwmcXjKGmi+kc4wHXomIV9qtvwv4h1wsY9P3vhHxZER8l2yW6o9s4/mtj3LysN7uA5JW5z5fJfuHe5KkXwNL6foVxDeTvW1yGdmg9mNk4wgA04FfdtGV1d6FwH+TtBiYxtvTfXemP/Dvkp4EHgcui4iXycZUviPpcba9hfNG2v+HwKQO1p8NjEsD9cuAr6T6c9Mg+2Lgj8AvtvH81kd5Vl0zsrGSiHhN0u7AI2SDxesaHdd7IWk+8D8jYmGjY7G+x2MeZpnb02DyDsBFvT1xmFXNLQ8zMyvNYx5mZlaak4eZmZXm5GFmZqU5eZiZWWlOHmZmVtr/BwHod7r9Zcz7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(nrows=1, ncols=2)\n",
    "axs = axes.ravel()\n",
    "\n",
    "axs[0].boxplot(q_len)\n",
    "axs[0].set_title(\"Length of questions.\")\n",
    "\n",
    "axs[1].boxplot(a_len)\n",
    "axs[1].set_title(\"Length of answers.\")\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "fig2, ax2 = plt.subplots()\n",
    "ax2.hist(q_len, bins=40)\n",
    "ax2.set_xlabel(\"Length of samples\")\n",
    "ax2.set_ylabel(\"Number of samples\")\n",
    "ax2.set_title(\"Questions\")\n",
    "\n",
    "fig3, ax3 = plt.subplots()\n",
    "ax3.hist(a_len, bins=40)\n",
    "ax3.set_xlabel(\"Length of samples\")\n",
    "ax3.set_ylabel(\"Number of samples\")\n",
    "ax3.set_title(\"Answers\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 658,
   "id": "floating-castle",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "95% of q_len is :7.0\n",
      "95% of a_len is :8.0\n"
     ]
    }
   ],
   "source": [
    "print(f\"95% of q_len is :{np.percentile(q_len, 95)}\")\n",
    "print(f\"95% of a_len is :{np.percentile(a_len, 95)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lined-delta",
   "metadata": {},
   "source": [
    "데이터셋의 크기가 크지 않으므로 최대길이를 예제와 같이 25으로 정해 모든 단어를 포함하는 방향으로 진행해보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 659,
   "id": "offensive-bundle",
   "metadata": {},
   "outputs": [],
   "source": [
    "def below_threshold_len(max_len, nested_list):\n",
    "    cnt = 0\n",
    "    for s in nested_list:\n",
    "        if(len(s.split()) <= max_len):\n",
    "            cnt = cnt+1\n",
    "            \n",
    "    print(f\"전체 샘플 중 길이가 {max_len} 이하인 샘플의 비율: {cnt/len(nested_list):.2%}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 660,
   "id": "empty-physiology",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Questions\n",
      ">>>\n",
      "전체 샘플 중 길이가 25 이하인 샘플의 비율: 100.00%\n",
      "Answers\n",
      ">>>\n",
      "전체 샘플 중 길이가 25 이하인 샘플의 비율: 100.00%\n"
     ]
    }
   ],
   "source": [
    "print(\"Questions\\n>>>\")\n",
    "below_threshold_len(25, questions)\n",
    "print(\"Answers\\n>>>\")\n",
    "below_threshold_len(25, answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 661,
   "id": "abroad-coach",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25\n"
     ]
    }
   ],
   "source": [
    "MAX_LENGTH = 25\n",
    "print(MAX_LENGTH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 662,
   "id": "complicated-cycling",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_and_filter(inputs, outputs):\n",
    "    tokenized_inputs, tokenized_outputs = [], []\n",
    "    \n",
    "    for (sentence1, sentence2) in zip(inputs, outputs):\n",
    "        sentence1 = START_TOKEN + tokenizer.encode(sentence1) + END_TOKEN\n",
    "        sentence2 = START_TOKEN + tokenizer.encode(sentence2) + END_TOKEN\n",
    "        \n",
    "        if len(sentence1) <= MAX_LENGTH and len(sentence2) <= MAX_LENGTH:\n",
    "            tokenized_inputs.append(sentence1)\n",
    "            tokenized_outputs.append(sentence2)\n",
    "    \n",
    "    tokenized_inputs = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_inputs, maxlen=MAX_LENGTH, padding='post')\n",
    "    \n",
    "    tokenized_outputs = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_outputs, maxlen=MAX_LENGTH, padding='post')\n",
    "    \n",
    "    return tokenized_inputs, tokenized_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 663,
   "id": "solar-adobe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어장의 크기 : 8350\n",
      "필터링 후의 질문 샘플 개수: 11817\n",
      "필터링 후의 답변 샘플 개수: 11817\n"
     ]
    }
   ],
   "source": [
    "questions, answers = tokenize_and_filter(questions, answers)\n",
    "print('단어장의 크기 :',(VOCAB_SIZE))\n",
    "print('필터링 후의 질문 샘플 개수: {}'.format(len(questions)))\n",
    "print('필터링 후의 답변 샘플 개수: {}'.format(len(answers)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 664,
   "id": "offshore-identity",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "BUFFER_SIZE = 20000\n",
    "\n",
    "dataset = tf.data.Dataset.from_tensor_slices((\n",
    "    {\n",
    "        'inputs': questions,\n",
    "        'dec_inputs': answers[:, :-1]\n",
    "    },\n",
    "    {\n",
    "        'outputs': answers[:, 1:]\n",
    "    }\n",
    "))\n",
    "\n",
    "dataset = dataset.cache()\n",
    "dataset = dataset.shuffle(BUFFER_SIZE)\n",
    "dataset = dataset.batch(BATCH_SIZE)\n",
    "dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "applicable-organic",
   "metadata": {},
   "source": [
    "### 모델링"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 665,
   "id": "presidential-seven",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"transformer\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "inputs (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dec_inputs (InputLayer)         [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "enc_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "encoder (Functional)            (None, None, 256)    3191808     inputs[0][0]                     \n",
      "                                                                 enc_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "look_ahead_mask (Lambda)        (None, 1, None, None 0           dec_inputs[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dec_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "decoder (Functional)            (None, None, 256)    3719168     dec_inputs[0][0]                 \n",
      "                                                                 encoder[0][0]                    \n",
      "                                                                 look_ahead_mask[0][0]            \n",
      "                                                                 dec_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "outputs (Dense)                 (None, None, 8350)   2145950     decoder[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 9,056,926\n",
      "Trainable params: 9,056,926\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "\n",
    "\n",
    "NUM_LAYERS = 2\n",
    "D_MODEL = 256\n",
    "NUM_HEADS = 8\n",
    "UNITS = 512\n",
    "DROPOUT = 0.1\n",
    "\n",
    "model = transformer(\n",
    "    vocab_size=VOCAB_SIZE,\n",
    "    num_layers=NUM_LAYERS,\n",
    "    units=UNITS,\n",
    "    d_model=D_MODEL,\n",
    "    num_heads=NUM_HEADS,\n",
    "    dropout=DROPOUT)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 666,
   "id": "mechanical-solomon",
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_function(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH - 1))\n",
    "    \n",
    "    loss = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "        from_logits=True, reduction='none')(y_true, y_pred)\n",
    "    \n",
    "    mask = tf.cast(tf.not_equal(y_true, 0), tf.float32)\n",
    "    loss = tf.multiply(loss, mask)\n",
    "    \n",
    "    return tf.reduce_mean(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 667,
   "id": "younger-salvation",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "    \n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super(CustomSchedule, self).__init__()\n",
    "        \n",
    "        self.d_model = d_model\n",
    "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "        \n",
    "        self.warmup_steps = warmup_steps\n",
    "        \n",
    "    def __call__(self, step):\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps**-1.5)\n",
    "        \n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 668,
   "id": "delayed-camera",
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = CustomSchedule(D_MODEL)\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(\n",
    "    learning_rate, beta_1=0.9, beta_2=0.98, epsilon=1e-9)\n",
    "\n",
    "def accuracy(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH-1))\n",
    "    return tf.keras.metrics.sparse_categorical_accuracy(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 669,
   "id": "martial-mirror",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer, loss=loss_function, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 670,
   "id": "massive-canyon",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "185/185 [==============================] - 24s 100ms/step - loss: 2.4919 - accuracy: 0.0203\n",
      "Epoch 2/50\n",
      "185/185 [==============================] - 19s 102ms/step - loss: 2.0264 - accuracy: 0.0799\n",
      "Epoch 3/50\n",
      "185/185 [==============================] - 19s 103ms/step - loss: 1.6745 - accuracy: 0.0815\n",
      "Epoch 4/50\n",
      "185/185 [==============================] - 19s 104ms/step - loss: 1.5262 - accuracy: 0.0875\n",
      "Epoch 5/50\n",
      "185/185 [==============================] - 19s 103ms/step - loss: 1.4305 - accuracy: 0.0924\n",
      "Epoch 6/50\n",
      "185/185 [==============================] - 19s 104ms/step - loss: 1.3278 - accuracy: 0.0989\n",
      "Epoch 7/50\n",
      "185/185 [==============================] - 19s 103ms/step - loss: 1.2127 - accuracy: 0.1080\n",
      "Epoch 8/50\n",
      "185/185 [==============================] - 19s 103ms/step - loss: 1.0895 - accuracy: 0.1220\n",
      "Epoch 9/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.9416 - accuracy: 0.1394\n",
      "Epoch 10/50\n",
      "185/185 [==============================] - 19s 104ms/step - loss: 0.7798 - accuracy: 0.1585\n",
      "Epoch 11/50\n",
      "185/185 [==============================] - 19s 104ms/step - loss: 0.6340 - accuracy: 0.1808\n",
      "Epoch 12/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.4837 - accuracy: 0.2044\n",
      "Epoch 13/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.3579 - accuracy: 0.2237\n",
      "Epoch 14/50\n",
      "185/185 [==============================] - 19s 103ms/step - loss: 0.2466 - accuracy: 0.2413\n",
      "Epoch 15/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.1642 - accuracy: 0.2556\n",
      "Epoch 16/50\n",
      "185/185 [==============================] - 19s 102ms/step - loss: 0.1092 - accuracy: 0.2655\n",
      "Epoch 17/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0771 - accuracy: 0.2715\n",
      "Epoch 18/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0623 - accuracy: 0.2721\n",
      "Epoch 19/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0564 - accuracy: 0.2723\n",
      "Epoch 20/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0511 - accuracy: 0.2716\n",
      "Epoch 21/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0498 - accuracy: 0.2723\n",
      "Epoch 22/50\n",
      "185/185 [==============================] - 19s 104ms/step - loss: 0.0488 - accuracy: 0.2717\n",
      "Epoch 23/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0460 - accuracy: 0.2724\n",
      "Epoch 24/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0398 - accuracy: 0.2761\n",
      "Epoch 25/50\n",
      "185/185 [==============================] - 20s 105ms/step - loss: 0.0343 - accuracy: 0.2770\n",
      "Epoch 26/50\n",
      "185/185 [==============================] - 20s 105ms/step - loss: 0.0300 - accuracy: 0.2794\n",
      "Epoch 27/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0256 - accuracy: 0.2773\n",
      "Epoch 28/50\n",
      "185/185 [==============================] - 19s 104ms/step - loss: 0.0254 - accuracy: 0.2784\n",
      "Epoch 29/50\n",
      "185/185 [==============================] - 20s 106ms/step - loss: 0.0219 - accuracy: 0.2803\n",
      "Epoch 30/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0203 - accuracy: 0.2780\n",
      "Epoch 31/50\n",
      "185/185 [==============================] - 20s 106ms/step - loss: 0.0189 - accuracy: 0.2791\n",
      "Epoch 32/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0178 - accuracy: 0.2808\n",
      "Epoch 33/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0154 - accuracy: 0.2807\n",
      "Epoch 34/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0153 - accuracy: 0.2808\n",
      "Epoch 35/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0142 - accuracy: 0.2808\n",
      "Epoch 36/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0129 - accuracy: 0.2807\n",
      "Epoch 37/50\n",
      "185/185 [==============================] - 20s 106ms/step - loss: 0.0114 - accuracy: 0.2812\n",
      "Epoch 38/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0114 - accuracy: 0.2815\n",
      "Epoch 39/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0115 - accuracy: 0.2815\n",
      "Epoch 40/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0102 - accuracy: 0.2817\n",
      "Epoch 41/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0102 - accuracy: 0.2831\n",
      "Epoch 42/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0092 - accuracy: 0.2825\n",
      "Epoch 43/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0091 - accuracy: 0.2812\n",
      "Epoch 44/50\n",
      "185/185 [==============================] - 19s 104ms/step - loss: 0.0088 - accuracy: 0.2821\n",
      "Epoch 45/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0077 - accuracy: 0.2825\n",
      "Epoch 46/50\n",
      "185/185 [==============================] - 20s 105ms/step - loss: 0.0072 - accuracy: 0.2836\n",
      "Epoch 47/50\n",
      "185/185 [==============================] - 19s 105ms/step - loss: 0.0082 - accuracy: 0.2837\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fbf4ea0b810>"
      ]
     },
     "execution_count": 670,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EPOCHS = 50\n",
    "callback = tf.keras.callbacks.EarlyStopping(monitor='loss')\n",
    "model.fit(dataset, epochs=EPOCHS, verbose=1, callbacks=[callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 671,
   "id": "approximate-living",
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder_inference(sentence):\n",
    "    sentence = preprocess_sentence(sentence)\n",
    "\n",
    "  # 입력된 문장을 정수 인코딩 후, 시작 토큰과 종료 토큰을 앞뒤로 추가.\n",
    "  # ex) Where have you been? → [[8331   86   30    5 1059    7 8332]]\n",
    "    sentence = tf.expand_dims(\n",
    "      START_TOKEN + tokenizer.encode(sentence) + END_TOKEN, axis=0)\n",
    "\n",
    "  # 디코더의 현재까지의 예측한 출력 시퀀스가 지속적으로 저장되는 변수.\n",
    "  # 처음에는 예측한 내용이 없음으로 시작 토큰만 별도 저장. ex) 8331\n",
    "    output_sequence = tf.expand_dims(START_TOKEN, 0)\n",
    "\n",
    "  # 디코더의 인퍼런스 단계\n",
    "    for i in range(MAX_LENGTH):\n",
    "        predictions = model(inputs=[sentence, output_sequence], training=False)\n",
    "        predictions = predictions[:, -1:, :]\n",
    "\n",
    "        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
    "\n",
    "    # 만약 현재 예측한 단어가 종료 토큰이라면 for문을 종료\n",
    "        if tf.equal(predicted_id, END_TOKEN[0]):\n",
    "            break\n",
    "\n",
    "    # 예측한 단어들은 지속적으로 output_sequence에 추가됩니다.\n",
    "\n",
    "        output_sequence = tf.concat([output_sequence, predicted_id], axis=-1)\n",
    "\n",
    "    return tf.squeeze(output_sequence, axis=0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 672,
   "id": "proprietary-conspiracy",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence_generation(sentence):\n",
    "  # 입력 문장에 대해서 디코더를 동작 시켜 예측된 정수 시퀀스를 리턴받습니다.\n",
    "    prediction = decoder_inference(sentence)\n",
    "\n",
    "    # 정수 시퀀스를 다시 텍스트 시퀀스로 변환합니다.\n",
    "    predicted_sentence = tokenizer.decode(\n",
    "      [i for i in prediction if i < tokenizer.vocab_size])\n",
    "\n",
    "    print('입력 : {}'.format(sentence))\n",
    "    print('출력 : {}'.format(predicted_sentence))\n",
    "\n",
    "    return predicted_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 673,
   "id": "voluntary-architect",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 출근길 너무 힘들어\n",
      "출력 : 회사 근처로 이사를 가보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'회사 근처로 이사를 가보세요 .'"
      ]
     },
     "execution_count": 673,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('출근길 너무 힘들어')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 674,
   "id": "adjustable-geneva",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 안녕, 넌 이름이 뭐니?\n",
      "출력 : 안녕하세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'안녕하세요 .'"
      ]
     },
     "execution_count": 674,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('안녕, 넌 이름이 뭐니?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 675,
   "id": "thirty-enterprise",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때?\n",
      "출력 : 오늘은 예능이요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'오늘은 예능이요 .'"
      ]
     },
     "execution_count": 675,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 676,
   "id": "meaningful-example",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때??\n",
      "출력 : 오늘은 예능이요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'오늘은 예능이요 .'"
      ]
     },
     "execution_count": 676,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때??')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 677,
   "id": "wrong-riding",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어떠니?\n",
      "출력 : 당신만 아픈 것도 , 당신만 겪은 것도 아니에요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'당신만 아픈 것도 , 당신만 겪은 것도 아니에요 .'"
      ]
     },
     "execution_count": 677,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어떠니?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 678,
   "id": "gross-timing",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어땡?\n",
      "출력 : 오늘은 예능이요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'오늘은 예능이요 .'"
      ]
     },
     "execution_count": 678,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어땡?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 679,
   "id": "functioning-lambda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때ㅋㅋㅋㅋㅋ\n",
      "출력 : 오늘은 예능이요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'오늘은 예능이요 .'"
      ]
     },
     "execution_count": 679,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때ㅋㅋㅋㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 680,
   "id": "potential-sigma",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋ\n",
      "출력 : 지금은 힘들겠지만 조금만 더 견뎌봐요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'지금은 힘들겠지만 조금만 더 견뎌봐요 .'"
      ]
     },
     "execution_count": 680,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 681,
   "id": "twenty-portal",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋㅋ\n",
      "출력 : 지금은 힘들겠지만 조금만 더 견뎌봐요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'지금은 힘들겠지만 조금만 더 견뎌봐요 .'"
      ]
     },
     "execution_count": 681,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 682,
   "id": "expired-fever",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋㅋㅋ\n",
      "출력 : 지금은 힘들겠지만 조금만 더 견뎌봐요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'지금은 힘들겠지만 조금만 더 견뎌봐요 .'"
      ]
     },
     "execution_count": 682,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 691,
   "id": "advised-michigan",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 날씨 좋아?\n",
      "출력 : 날씨 어플에 물어보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'날씨 어플에 물어보세요 .'"
      ]
     },
     "execution_count": 691,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('날씨 좋아?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 692,
   "id": "welsh-chess",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 날씨 좋아!\n",
      "출력 : 하늘을 보고 웃어보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'하늘을 보고 웃어보세요 .'"
      ]
     },
     "execution_count": 692,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('날씨 좋아!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "configured-orbit",
   "metadata": {},
   "source": [
    "'오늘 날씨는 어때?' 와 '오늘 날씨는 어때ㅋㅋㅋㅋㅋ' 가 같은 답변을 만들어낸 것을 보면 전처리가 효과가 있었다고 생각할 수 있을 것도 같습니다. 'ㅋ'와 'ㅋㅋ', 'ㅋㅋㅋ' 모두 다 같은 답변을 이끌어내고 있기도 합니다.\n",
    "\n",
    "'날씨 좋아?'와 '날씨 좋아!'도 어느정도 구분을 하고 답변을 만들어내고 있는 것 같습니다. 날씨가 어떠냐는 질문에 '오늘은 예능이요.'라고 맞지 않는 대답을 하는 경우도 아직 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "average-petersburg",
   "metadata": {},
   "source": [
    "## 7. 하이퍼파라미터 수정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 693,
   "id": "selective-bronze",
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = chatbotdata['Q'].to_numpy()\n",
    "answers = chatbotdata['A'].to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 694,
   "id": "brazilian-inventory",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_sentence(sentence):\n",
    "    sentence = re.sub(\"(!)\\\\1{1,}\", r\"\\1\", sentence) ## 연속 느낌표 제거\n",
    "    sentence = re.sub(\"(\\?)\\\\1{1,}\", r\"\\1\", sentence) ## 연속 물음표 제거\n",
    "    sentence = re.sub(r\"([?.!,])\", r\" \\1 \", sentence) ## 구둣점 분리\n",
    "    sentence = re.sub(r\"[^가-힣0-9?.!,]+\", \" \", sentence)\n",
    "    sentence = re.sub(\"(.)\\\\1{1,}\", r\"\\1\\1\", sentence) ## 반복되는 ㅋㅋㅋ => ㅋㅋ\n",
    "    sentence = sentence.strip()\n",
    "    \n",
    "    return sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 695,
   "id": "expired-withdrawal",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_conversations():\n",
    "    \n",
    "    questions = chatbotdata['Q'].apply(preprocess_sentence)\n",
    "    answers = chatbotdata['A'].apply(preprocess_sentence)\n",
    "    \n",
    "    return questions.to_numpy(), answers.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 696,
   "id": "naked-taylor",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 질문 샘플 수: 11823\n",
      "전체 답변 샘플 수: 11823\n"
     ]
    }
   ],
   "source": [
    "questions, answers = load_conversations()\n",
    "print(f\"전체 질문 샘플 수: {len(questions)}\")\n",
    "print(f\"전체 답변 샘플 수: {len(answers)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 697,
   "id": "divine-scanning",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전처리 후의 4027번째 질문 샘플: 재테크 하는 방법 알려줘\n",
      "전처리 후의 4027번째 답변 샘플: 우선 관련 서적을 읽어보세요 .\n",
      "==================================================\n",
      "전처리 후의 6442번째 질문 샘플: 받아들이기가 많이 힘드네\n",
      "전처리 후의 6442번째 답변 샘플: 이해하기 힘드니까요 .\n",
      "==================================================\n",
      "전처리 후의 7320번째 질문 샘플: 으휴 집에 들어가기가 싫네\n",
      "전처리 후의 7320번째 답변 샘플: 집에 들어가 뜨거운 물로 샤워하고 나면 쌓였던 생각들이 사라질지도 몰라요 .\n",
      "==================================================\n",
      "전처리 후의 10377번째 질문 샘플: 썸이 좋아\n",
      "전처리 후의 10377번째 답변 샘플: 사귈락말락 할 때가 좋지요 .\n",
      "==================================================\n",
      "전처리 후의 2233번째 질문 샘플: 비행기표가 왜이렇게 비싸\n",
      "전처리 후의 2233번째 답변 샘플: 일찍 구매하면 좀 더 저렴하게 살 수 있어요 .\n",
      "==================================================\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng()\n",
    "\n",
    "\n",
    "for _ in range(5):\n",
    "    idx = rng.integers(len(answers))\n",
    "    print(f'전처리 후의 {idx}번째 질문 샘플: {questions[idx]}')\n",
    "    print(f'전처리 후의 {idx}번째 답변 샘플: {answers[idx]}')\n",
    "    print('=' * 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 698,
   "id": "suited-fashion",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "tokenizer = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(questions + answers, \n",
    "                                                                      target_vocab_size=2**13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 699,
   "id": "educational-quest",
   "metadata": {},
   "outputs": [],
   "source": [
    "START_TOKEN, END_TOKEN = [tokenizer.vocab_size], [tokenizer.vocab_size+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 700,
   "id": "willing-cleveland",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "START_TOKEN의 번호 : [8348]\n",
      "END_TOKEN의 번호 : [8349]\n"
     ]
    }
   ],
   "source": [
    "print('START_TOKEN의 번호 :' ,[tokenizer.vocab_size])\n",
    "print('END_TOKEN의 번호 :' ,[tokenizer.vocab_size + 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 701,
   "id": "trying-association",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8350\n"
     ]
    }
   ],
   "source": [
    "# 시작 토큰과 종료 토큰을 고려하여 +2를 하여 단어장의 크기를 산정합니다.\n",
    "VOCAB_SIZE = tokenizer.vocab_size + 2\n",
    "print(VOCAB_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 702,
   "id": "split-graphics",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전처리 후의 3349번째 질문 샘플: [464, 729, 14, 20, 5337]\n",
      "전처리 후의 3349번째 답변 샘플: [267, 854, 23, 5212, 94, 3846, 1219, 5327, 1]\n",
      "==================================================\n",
      "전처리 후의 323번째 질문 샘플: [1456, 8124, 55, 1742, 162]\n",
      "전처리 후의 323번째 답변 샘플: [22, 4628, 30, 3, 1]\n",
      "==================================================\n",
      "전처리 후의 4918번째 질문 샘플: [3853, 5771, 132]\n",
      "전처리 후의 4918번째 답변 샘플: [2433, 23, 138, 4646, 5, 30, 3, 35]\n",
      "==================================================\n",
      "전처리 후의 9172번째 질문 샘플: [592, 10, 394, 3643, 49, 1]\n",
      "전처리 후의 9172번째 답변 샘플: [1007, 3102, 140, 1]\n",
      "==================================================\n",
      "전처리 후의 11257번째 질문 샘플: [998, 7993, 67, 778, 58, 93, 2]\n",
      "전처리 후의 11257번째 답변 샘플: [8141, 802, 27, 3739, 66, 1296, 1203, 7047, 8124, 3739, 66, 26, 1]\n",
      "==================================================\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng()\n",
    "\n",
    "\n",
    "for _ in range(5):\n",
    "    idx = rng.integers(len(answers))\n",
    "    print(f'전처리 후의 {idx}번째 질문 샘플: {tokenizer.encode(questions[idx])}')\n",
    "    print(f'전처리 후의 {idx}번째 답변 샘플: {tokenizer.encode(answers[idx])}')\n",
    "    print('=' * 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 703,
   "id": "living-citizenship",
   "metadata": {},
   "outputs": [],
   "source": [
    "q_len = [len(s.split()) for s in questions]\n",
    "a_len = [len(s.split()) for s in answers]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 704,
   "id": "monthly-registration",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Answers')"
      ]
     },
     "execution_count": 704,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeg0lEQVR4nO3df3xddZ3n8debNrYUUZsSmQJN64AypfEHGNGh9UcXVGQYYXZc1y6wMFPb6TzGqNMyFMk8RpzddiiDONrZMdMaBIWJy4K/xgWUhQIGUEkRITUqivQHFBooyi8pLf3sH+ekcxqS5ib35p6Te9/Px+M+cs+Pe8/npvn2fb7nnPs9igjMzMyK5qC8CzAzMxuKA8rMzArJAWVmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUFUiaY6kkDS5Qu/3PyU9IemxSrzfeJB0kaQv5V2H1a56bFf1pC4CStLDkk6plW1KagZWAMdFxO+NxzZGS9J7JG3LzouI1RHx0bxqsvHldmXjrS4CqgY1A09GxI68CzGrITXTrirVo8xbXQeUpIMkXSjpV5KelHStpMZ02cChg3MlbUm7/e2Z1x4s6SpJT0nqk3TBQA9C0ldJ/tj/XdKzki7IbPasod5viNpeLekrkvolbZb0t2m9pwA3A0ek733lMK//G0nbJT0q6c/Tz3JMuuw2SR/NrHuepO7M9B9IulnSTkk/l/ThzLLTJP1U0jOSHpF0vqRDgBszNT0r6QhJF0u6OvPaD0raJOk3aQ1zM8seTt/rfkm/lfS/JU1Nlx0m6Tvp63ZK+r6kuv7bLbJabVeSpqd/h/1pfd+RdFRm+W2S/oekO9P28T1Jh6XLpkq6Ov19/EbSPZIOl7RQ0gOZ97hZ0j2Z6e9LOjN9foSk69Pt/1rSxzPrXSzpunQbTwPnSTpRUo+kpyU9LunyUv8NCyMiav4BPAycMsT8TwA/AI4CpgD/CnSly+YAAawHDgbeDOwC5qbLLwFuB6anr78f2DbcNkd6vyFq+wrwLeDQ9LW/ABany96T3dYQrz0VeBxoAQ4B/i3d9jHp8tuAj2bWPw/oTp8fAmwF/gyYDBwPPEFy2ANgO/DO9Pl04IThagIuBq5On78BeA54L9AAXAD8EnhF5vf1I+AIoBHoA5aly/4B6Ehf1wC8E1Def1f1/qjDdjUD+FNgWvr6/wN8M7P8NuBX6d/6wen0JemyvwD+PX3tJOCtwKvS9V4ADkv/th8HHknf/2Dgd+l2DwI2An8HvAL4feAh4P2ZtrYbODNd92DgbuCcdPkrgXfk/Tcz2ke974UuA9ojYltE7CL5R/6Q9u8efyYifhcRPwF+QtIAAD4MrI6IpyJiG/CFErc53PvtI2kS8BHgUxHxTEQ8DHwWOKfEbXwY+HJE9EbEc+nnKtXpwMMR8eWI2BMRPwauB/5Lunw3cJykV6Wf/d4S3/e/Av83Im6OiN3AZSSN6KTMOl+IiEcjYidJY35LZpszgdkRsTsivh9pq7NCqsl2FRFPRsT1EfF8RDwDrALePWi1L0fELyLid8C17P83PINkJ/GliNgYEU+n690DvIsktH4C3AnMB94BPBgRTwJvA5oi4u8j4sWIeIgklD+S2fbdEfHNiNibvu9u4BhJh0XEsxHxg1I+Z5HUe0DNBr6Rdrl/Q7LX/hJweGad7NU8z5PsiUCyp781syz7/ECGe7+sgb2pzZl5m4EjS9zG4No2D7fiEGYDbx/4naS/l7OAgZPGfwqcBmyWdLukPxxFTfvqiIi9aY3ZzzTc7+YfSXpb35P0kKQLR/F5rPpqsl1JmibpX9NDg08DdwCvSYNvpDq+CnwX+JqSw+6XSmpIl91O0nt7V/r8NpLge3c6Dcnv9IhB7fIi9v+dDv5dLSbpzf0sPaR4eimfs0jqPaC2Ah+IiNdkHlMj4pESXrud5BDEgFmDlpezh/8Eyd7P7My8ZpKufym2D6qnedDy50gONQzIXrG0Fbh90O/klRHxlwARcU9EnAG8FvgmyV4ijPx5HyXzeSQprXHEz5Tu7a6IiN8HPggsl3TySK+z3NRqu1oBHAu8PSJeRRIoABrphWnP/zMRcRzJUYPTgf+eLh4cULfz8oDaCvx60O/00Ig4LbuZQdt8MCIWkbTVNcB1Ss4XTxj1FFAN6YnKgcdkkvMaqyTNBpDUJOmMEt/vWuBT6YnTI4GPDVr+OMlx4lGLiJfS918l6dC0vuXA1Qd+5X61nSfpOEnTgE8PWn4f8J/TPcJjSPa0BnwHeIOkcyQ1pI+3SZor6RWSzpL06vQw3dPA3sznnSHp1Qeo6Y8knZzuOa4gOVdw10gfRtLpko5JQ+23JHvje0d4mVVHPbWrQ0nOCf1GyUUfg9vVsNKLId6Y9raeJgnKgb/hu0iC70TgRxGxifRIBkkvDZLzs89IWqnkQpJJklokve0A2zxbUlN6tOI36ewJ1W7qKaBuIPnjGnhcDHwe+DbJoaNnSE7svr3E9/t7YBvwa+D/AdeR/Ic74B+Av0274+ePod42kp7OQ0A3yYUOV5Tywoi4Efgn4FaSQ2O3Dlrlc8CLJI39KuCazGufAd5Hcmz7UZJDFmtITnZDcrz+4fQQxzKSw39ExM+ALuCh9DMfMaimnwNnA2tJ9mT/GPjjiHixhI/0epLf8bMkJ37/JSI2AEi6UdJFJbyHjY+6aVckbepgkr/fHwA3jWK7v0fyWZ4mOeR5O8lhP9LzxPcCmzLt4W5gc6SXvKfhejrJOa1fpzV8CRhuhxCSi6U2SXqW5N/kI+m5KZRcqfjOUdSfC/lcc2VI+kuSP4DBJ00LQVIAr4+IX+Zdi1mpit6ubHzVUw+qoiTNlDRfyXcojiU5ZPWNvOsym8jcriyrJr5tnJNXkHy/43Ukx3e/BvxLngWZ1QC3K9vHh/jMzKyQfIjPrEokzZK0QclQUZskfSKdf7GSYaPuSx+njfReZvWgqj2oww47LObMmVO17ZmNp40bNz4REU2lri9pJjAzIu6VdCjJ0DVnkoye8GxEXFbqe7ktWS0Zri1V9RzUnDlz6OnpqeYmzcaNpNGM0EFEbCf5IioR8YykPkofHWQ/bktWS4ZrSz7EZ5YDSXNIBuL9YTrrY0pGcr9C0vRhXrNUyejUPf39/dUq1Sw3DiizKpP0SpIBeD8ZEU8DXwSOJvkS5naSAUxfJiLWRURrRLQ2NZV8ZNFswnJAmVVROszT9cA1EfF1gIh4PB3hei/JCNUn5lmjWVE4oMyqJB1LsBPoi4jLM/NnZlb7E6C32rWZFZG/qGtWPfNJxjJ8QNJ96byLgEWS3kIyGvXDJDe3M6t7DiizKomIboa+NcMN1a7FbCIY8RBfelXRDkm9g+a3SfpZ+oXDS8evRCtVV1cXLS0tTJo0iZaWFrq6uvIuyWxCclsqhlJ6UFcC/wx8ZWCGpIXAGcCbI2KXpNeOT3lWqq6uLtrb2+ns7GTBggV0d3ezeHFym6dFixblXJ3ZxOG2VCARMeIDmAP0ZqavBU4p5bXZx1vf+taw8TFv3ry49dZb95t36623xrx583KqqPYBPTHKNlCph9vS+HFbqr7h2lJJQx2lXyr8TkS0pNP3Ad8iuSHWC8D5EXHPMK9dCiwFaG5ufuvmzaP68r2VaNKkSbzwwgs0NDTsm7d7926mTp3KSy+9lGNltUvSxohozWPbra2t4ZEkxofbUvUN15bGepn5ZKAReAfwN8C16SW0LxP+cmFVzJ07l+7u7v3mdXd3M3fu3JwqMpuY3JaKY6wBtQ34eto7+xHJfe4Pq1xZNlrt7e0sXryYDRs2sHv3bjZs2MDixYtpb2/PuzSzCcVtqTjGepn5N4GFwAZJbyC5ydgTlSrKRm/g5G1bWxt9fX3MnTuXVatW+aSu2Si5LRXHiOegJHUB7yHpIT0OfBr4KnAFydhhL5Kcg7p1pI35uLnVEp+DMquM4drSiD2oiBhut+HssqsyMzMbhsfiMzOzQnJAmZlZITmgzMyskBxQZmZWSA4oMzMrJAeUmZkVkgPKzMwKyQFlZmaF5IAyM7NCckCZmQ3iO+oWw1gHizUzq0m+o25xuAdlZpaxatUqOjs7WbhwIQ0NDSxcuJDOzk5WrVqVd2l1xwFlZpbR19fHggUL9pu3YMEC+vr6cqqofjmgzMwyfEfd4nBAmZll+I66xeGLJMzMMnxH3eJwQJmZDbJo0SIHUgH4EJ+ZmRWSA8rMzAppxICSdIWkHZJ6h1i2QlJIOmx8yrPRaGtrY+rUqUhi6tSptLW15V2SmdmYldKDuhI4dfBMSbOA9wFbKlyTjUFbWxsdHR2sXr2a5557jtWrV9PR0eGQMrMJa8SAiog7gJ1DLPoccAEQlS7KRm/9+vWsWbOG5cuXM23aNJYvX86aNWtYv3593qWZmY3JmM5BSToDeCQiflLCuksl9Ujq6e/vH8vmrAS7du1i2bJl+81btmwZu3btyqkiM7PyjDqgJE0DLgL+rpT1I2JdRLRGRGtTU9NoN2clmjJlCh0dHfvN6+joYMqUKTlVZGZWnrF8D+po4HXATyQBHAXcK+nEiHisksVZ6ZYsWcLKlSuBpOfU0dHBypUrX9arMjObKEYdUBHxAPDagWlJDwOtEfFEBeuyUVq7di0AF110EStWrGDKlCksW7Zs33wzs4mmlMvMu4C7gWMlbZO0ePzLsrFYu3YtL7zwAhHBCy+84HAyswltxB5URBxwvI+ImFOxaszMzFIeScKsSiTNkrRB0k8lbZL0iXR+o6SbJT2Y/pyed61mReCAMquePcCKiDgOeAfwV5KOAy4EbomI1wO3pNNmdc8BZVYlEbE9Iu5Nnz8D9AFHAmcAV6WrXQWcmUuBZgXjgDLLgaQ5wPHAD4HDI2J7uugx4PBhXuMvvVtdcUCZVZmkVwLXA5+MiKezyyIiGGb4MH/p3eqNA6qGzJgxA0n7HjNmzMi7JBtEUgNJOF0TEV9PZz8uaWa6fCawI6/6zIrEAVUjZsyYwc6dO5k3bx6bN29m3rx57Ny50yFVIEqGXukE+iLi8syibwPnps/PBb5V7drMisi3fK8RA+HU25vctqu3t5eWlhY2bdqUc2WWMR84B3hA0n3pvIuAS4Br0y/BbwY+nE95ZsXigKohN9xww8umZ8+enVM1NlhEdAMaZvHJ1azFbCLwIb4actpppx1w2sxK09XVRUtLC5MmTaKlpYWurq68S6pLDqga0djYyKZNm2hpaWHLli37Du81NjbmXZrZhNLV1UV7e/u+sS3Xrl1Le3u7QyoHDqga8eSTT+4LqdmzZ+8LpyeffDLv0swmlFWrVtHZ2cnChQtpaGhg4cKFdHZ2smrVqrxLqzs+B1VDHEZm5evr62PBggX7zVuwYAF9fX05VVS/3IMyM8uYO3cu3d3d+83r7u5m7ty5OVVUvxxQZmYZ7e3tLF68mA0bNrB79242bNjA4sWLaW9vz7u0uuNDfGZmGYsWJbfAa2tro6+vj7lz57Jq1ap98616HFBmZoMsWrTIgVQAPsRnZmaF5IAyMxukra2NqVOnIompU6fS1taWd0l1acSAknSFpB2SejPz/lHSzyTdL+kbkl4zrlWamVVJW1sbHR0drF69mueee47Vq1fT0dHhkMpBKT2oK4FTB827GWiJiDcBvwA+VeG6bAyyt9oYeJjZ6Kxfv541a9awfPlypk2bxvLly1mzZg3r16/Pu7S6M2JARcQdwM5B874XEXvSyR8AR41DbTYKA2EkiZtuumm/aTMr3a5du1i2bNl+85YtW8auXbtyqqh+VeIc1J8DN1bgfaxMkti7dy/vf//72bt3r8PJbAymTJlCR0fHfvM6OjqYMmVKThXVr7ICSlI7sAe45gDrLJXUI6mnv7+/nM3ZCG688cYDTpvZyJYsWcLKlSu5/PLLef7557n88stZuXIlS5Ysybu0uqOIGHklaQ7wnYhoycw7D/gL4OSIeL6UjbW2tkZPT8/YKrUDGjjntHfv3n3zDjroICKCUv6NbfQkbYyI1jy27bY0vtra2li/fj27du1iypQpLFmyhLVr1+ZdVs0ari2NqQcl6VTgAuCDpYaTjb+I4KCDDuK73/3uvnAys9EbuNVGROy75YZVXymXmXcBdwPHStqW3pb6n4FDgZsl3Sep44BvYuNuIIwiglNPPXW/aTOziWjEoY4iYqjxPjrHoRYrk8PIzGqJR5IwMxvEt3wvBg8Wa2aWMXDL987OThYsWEB3dzeLFy8G8ACyVeYelJlZhm/5XhwOKDOzDN/yvTgcUGZmGb7le3E4oMzMMnzL9+LwRRI1ZKix93zpudno+JbvxeEeVI3IhtOaNWuGnG9mpVm0aBG9vb289NJL9Pb2Opxy4oCqMRHBBRdc4J6TmU14Dqgaku05DTVtZqVpbm7e78afzc3NeZdUlxxQNWTlypUHnDazkTU3N7N161ZOOukkHn30UU466SS2bt3qkMqBA6rGSOLSSy/1uSezMRoIpzvvvJOZM2dy55137gspqy4HVI3InnPK9px8Lsps9K677roDTlt1OKBqyMDNCbMPMxu9D33oQwectupwQJmZZcyaNYu77rqL+fPns337dubPn89dd93FrFmz8i6t7viLumZmGVu2bKG5uZm77rqLI444AkhCa8uWLTlXVn8cUGZmgziMisGH+MzMrJAcUGZVIukKSTsk9WbmXSzpEUn3pY/T8qzREg0NDft9UbehoSHvkuqSA8qseq4ETh1i/uci4i3p44Yq12SDNDQ0sGfPHqZPn87999/P9OnT2bNnj0MqByMG1DB7fY2Sbpb0YPpz+viWaaXI7vENPKw4IuIOYGfeddiBDYTTzp07eeMb38jOnTv3hZRVVyk9qCt5+V7fhcAtEfF64JZ02nKUDaPjjz9+yPlWWB+TdH+6Mzjszp6kpZJ6JPX09/dXs766c/vttx9w2qpjxIAaZq/vDOCq9PlVwJmVLcvGKiK49957/SXdieOLwNHAW4DtwGeHWzEi1kVEa0S0NjU1Vam8+vTud7/7gNNWHWM9B3V4RGxPnz8GHD7cit7rq55sz2moaSueiHg8Il6KiL3AeuDEvGuqd5MnT+app56isbGRBx54gMbGRp566ikmT/a3cqqt7IskItlVH3Z33Xt91fPjH//4gNNWPJJmZib/BOgdbl2rjt27d+8LqTe96U37wmn37t15l1Z3xhpQjw80rPTnjsqVZOWQxAknnOBzTwUkqQu4GzhW0jZJi4FLJT0g6X5gIfDXuRZpQBJS2TEtHU75GGuf9dvAucAl6c9vVawiG5OI2BdK2Z6Tz0UVR0QMdd/wzqoXYjZBlHKZ+VB7fZcA75X0IHBKOm0582jmZpXhr2wUw4g9qGH2+gBOrnAtZma5y4bR1Vdfzdlnn71vvnf6qssjSZiZDSEiOOussxxKOXJAmZkNcvXVVx9w2qrDAWVmNsjAYb3hpq06HFBmZkOQxDXXXOMLJHLkgDIzy8iec8r2nHwuqvo8dkcNGWpPz43KbPTcborBPagaMdxhCB+eMLOJyj2oGpPd83M4mY2Nj0YUg3tQZmYZ2XC67LLLhpxv1eGAMjMbQkSwYsUK95xy5ICqMR47zKx82Z7TUNNWHQ6oGjHcXp73/sxG7/zzzz/gtFWHA6qGeDRzs8qRxGc/+1kfjciRA8rMLCO7Y5ftOXmHr/p8mbmZ2SAOo2JwD8rMzArJAWVmZoXkQ3xmZoN4JIlicA/KzCwjG05f+MIXhpxv1VFWQEn6a0mbJPVK6pI0tVKFmZnlKSJoa2tzzylHYw4oSUcCHwdaI6IFmAR8pFKFmZnlJdtzGmraqqPcQ3yTgYMlTQamAY+WX5KZWb4+/vGPH3DaqmPMARURjwCXAVuA7cBvI+J7g9eTtFRSj6Se/v7+sVdq+8mOuTfah5mNTBJr1651m8lROYf4pgNnAK8DjgAOkXT24PUiYl1EtEZEa1NT09grtf0MNaxRdnijkZab2dCybSTbc3Lbqb5yDvGdAvw6IvojYjfwdeCkypRlZpYf79gVQzkBtQV4h6RpSvrAJwN9lSnLzMzqXTnnoH4IXAfcCzyQvte6CtVlZpYbn7sthrKu4ouIT0fEH0RES0ScExG7KlWYmVkesmF09NFHDznfqsNDHZmZDSF73snhlA8PdWRmNki25zTUtFWHA8rMbJBf/epXB5y26nBAmZkNQRLHHHOMD+/lyAFlZpaRPfeU7Tn5u1DV54skzMwGcRgVg3tQZmZWSA4osyqRdIWkHZJ6M/MaJd0s6cH05/Q8azQrEgeUWfVcCZw6aN6FwC0R8XrglnTazHBAmVVNRNwB7Bw0+wzgqvT5VcCZ1azJrMh8kYRZvg6PiO3p88eAw4dbUdJSYClAc3NzFUqrD2O9jNwXUow/96DMCiKS//GG/V/P91YbH2O9t5qNPweUWb4elzQTIP25I+d6zArDAWWWr28D56bPzwW+lWMtZoXigDKrEkldwN3AsZK2SVoMXAK8V9KDJHepviTPGs2KxBdJmFVJRCwaZtHJVS3EbIJwD8rMzArJAWVmZoXkgDIzs0IqK6AkvUbSdZJ+JqlP0h9WqjAzM6tv5V4k8Xngpoj4kKRXANMqUJOZmdnYA0rSq4F3AecBRMSLwIuVKcvMzOpdOYf4Xgf0A1+W9GNJX5J0yOCVJC2V1COpp7+/v4zNmZlZPSknoCYDJwBfjIjjgecY4lYBHj/MzMzGopyA2gZsi4gfptPXkQSWmZlZ2cYcUBHxGLBV0rHprJOBn1akKjMzq3vlXsXXBlyTXsH3EPBn5ZdkZmZWZkBFxH1Aa2VKMTMz+w8eScLMzArJAWVmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCskBVWCNjY1IGvUDGPVrGhsbc/60Zmb7K3csPhtHTz31FBFRlW0NBJuZWVG4B2VmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCskBZWY1z6OyTExljyQhaRLQAzwSEaeXX5KZWWV5VJaJqRI9qE8AfRV4HzMzs33KCihJRwF/BHypMuWYmZklyj3E90/ABcChw60gaSmwFKC5ubnMzdWX+PSr4OJXV29bZmYFMuaAknQ6sCMiNkp6z3DrRcQ6YB1Aa2trdQ4C1wh95umqHjePi6uyKTOzkpTTg5oPfFDSacBU4FWSro6IsytTmln9kPQw8AzwErAnIlrzrcgsf2M+BxURn4qIoyJiDvAR4FaHk1lZFkbEWxxOZgl/D8rMzAqpIgEVEbf5O1BmZQnge5I2phcWvYykpZJ6JPX09/dXuTyz6nMPyqwYFkTECcAHgL+S9K7BK0TEuohojYjWpqam6ldoVmUOKLMCiIhH0p87gG8AJ+ZbkVn+HFBmOZN0iKRDB54D7wN6863KLH9lj8VnZmU7HPhGOobbZODfIuKmfEsyy58DyixnEfEQ8Oa86zArGh/iMzOzQnJAmZlZITmgzMyskHwOquCqdfOz6dOnV2U7ZmalckAV2FhHMpdUtVHQzSYC37pmYnJAmVnN861rJiafgzIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZIDyszMCsnfgzKzuuBRWSaeMQeUpFnAV0juZRPAuoj4fKUKMzOrFI/KMjGV04PaA6yIiHvTu4FulHRzRPy0QrWZmVkdG/M5qIjYHhH3ps+fAfqAIytVmJmZ1beKXCQhaQ5wPPDDIZYtldQjqae/v78SmzMzszpQdkBJeiVwPfDJiHh68PKIWBcRrRHR2tTUVO7mzMysTpQVUJIaSMLpmoj4emVKMjMzKyOglFyz2Qn0RcTllSvJzMysvB7UfOAc4D9Jui99nFahuszMrM6N+TLziOgGqvPNNzMzqzse6sjMzArJAWVmZoXkgDIzs0JyQJmZWSE5oMzMrJAcUGZmVkgOKDMzKyQHlJmZFZLvqDtBjXR30AMt9w3YzP7DWNuS29H4c0BNUG4cZpXhtlRcPsRnZmaF5IAyM7NCckCZmVkhOaDMzKyQHFBmBSDpVEk/l/RLSRfmXY9ZETigzHImaRLwv4APAMcBiyQdl29VZvlzQJnl70TglxHxUES8CHwNOCPnmsxy54Ayy9+RwNbM9LZ03n4kLZXUI6mnv7+/asWZ5cUBZTZBRMS6iGiNiNampqa8yzEbd1UdSWLjxo1PSNpczW3WqcOAJ/Iuog7MrtD7PALMykwflc4blttS1bgtVceQbUke5qP2SOqJiNa867DSSJoM/AI4mSSY7gH+W0RsyrUwc1vKmcfiM8tZROyR9DHgu8Ak4AqHk5kDyqwQIuIG4Ia86zArEl8kUZvW5V2AWY1wW8qRz0GZmVkhuQdlZmaF5IAyM7NCckDVEElXSNohqTfvWswmMrelYnBA1ZYrgVPzLsKsBlyJ21LuHFA1JCLuAHbmXYfZROe2VAwOKDMzKyQHlJmZFZIDyszMCskBZWZmheSAqiGSuoC7gWMlbZO0OO+azCYit6Vi8FBHZmZWSO5BmZlZITmgzMyskBxQZmZWSA4oMzMrJAeUmZkVkgPKzMwKyQFlZmaF9P8BeyN1VzX9IgsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeCklEQVR4nO3de7xXdZ3v8ddb8JIXRAUJubTRsIYuoofU1OOYppJUmFlpZegwhzpHU09OEzqVlWORpR6d0rJkpHIkUzNSUsnErirgBQV03CklBIJ5w0wT+Zw/1nfrarP3XgvY6/dbm9/7+Xj8Hr+1vuv22Sh89veyvl9FBGZmZj3ZotkBmJlZ/TlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmY1IOkjkm5pdhxm3XGysJYn6URJ90t6XtJKSZdI2rHC57VJCkn9O8oi4sqIOKKqZ5ptKicLa2mSzgC+Cnwa2BHYH2gDbpG0ZRNDM6sVJwtrWZIGAF8EPhkRN0XESxGxFPggsDvwYUlXSPr33DWHSFqW299N0rWSVkt6VNKpuWP7Spov6VlJj0u6IB36Zfp+WtJzkt6eaje/zl17gKR5kp5J3wfkjs2VdI6k30haI+kWSYPSsW0k/UDSnyU9na4d0vt/etZqnCyslR0AbANcly+MiOeA2UCPzUKStgB+CtwHDAMOA06XdGQ65SLgoogYAOwBXJ3KD07fAyNi+4j4Xaf77gzcCFwM7AJcANwoaZfcaR8GTgJ2BbYC/iWVTyKrIY1I134C+GuPfwpmJThZWCsbBDwREWu7OLYCGFxw/duAwRHxpYj4W0Q8AnwHOC4dfwl4vaRBEfFcRNxRMq4JwMMR8f2IWBsRVwEPAu/JnfOfEfHfEfFXsiQ0NvfMXYDXR8TLEbEgIp4t+VyzbjlZWCt7AhiU72jOGZqO9+R1wG6puedpSU8DZwEdzT6TgT2BB1Nz0LtLxrUb8IdOZX8gq710WJnbfh7YPm1/H7gZmCnpT5LOc9+L9QYnC2tlvwNeBI7JF0raHngXMBf4C7Bt7vBrc9uPAY9GxMDcZ4eIOAogIh6OiOPJmoq+ClwjaTugaF2AP5EloryRwPKiHyj1u3wxIsaQNbO9G/hY0XVmRZwsrGVFxDNkHdz/IWm8pC0ltZE16zwBXAncCxwlaWdJrwVOz93iLmCNpM9Ieo2kfpLeLOltAJI+KmlwRKwDnk7XrANWp+/duwltNrCnpA9L6i/pQ8AY4Iain0nSOyS9RVI/4FmyZql1Zf9MzLrjZGEtLSLOI2s6+jqwBniUrCbxzoj4C1mzzn3AUuAW4Ie5a18m+819bLruCeC7ZB3MAOOBRZKeI+vsPi4i/hoRzwPnAr9JzVf7d4rpz+m+ZwB/Bv4VeHdEFDWLQVbzuYYsUSwBbk8/g9kmkVfKM3uVpJOALwEHRsQfmx2PWV04WZh1IukE4KWImNnsWMzqwsnCzMwKuc/CzMwKdTW+vM8bNGhQtLW1NTsMM7M+ZcGCBU9ERJcvo26WyaKtrY358+c3Owwzsz5FUueXQV/hZigzMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyu0Wb7Bvblqm3pjt8eWTpvQwEjMrNW4ZmFmZoUqSxaStpF0l6T7JC2S9MVUPkrSnZLaJf1Q0lapfOu0356Ot+XudWYqf0jSkVXFbGZmXauyZvEicGhE7EW27OT4tHzkV4ELI+L1wFPA5HT+ZOCpVH5hOg9JY4DjgDeRLVN5SVpf2MzMGqSyZBGZ59LulukTwKFkawQDzACOTtsT0z7p+GGSlMpnRsSLEfEo0A7sW1XcZma2vkr7LCT1k3QvsAqYA/weeDoi1qZTlgHD0vYw4DGAdPwZYJd8eRfX5J81RdJ8SfNXr15dwU9jZta6Kk0WEfFyRIwFhpPVBt5Y4bMui4hxETFu8OAu1+4wM7ON1JDRUBHxNHAb8HZgoKSOIbvDgeVpezkwAiAd3xH4c768i2vMzKwBqhwNNVjSwLT9GuBwYAlZ0jg2nTYJ+EnanpX2Scd/ERGRyo9Lo6VGAaOBu6qK28zM1lflS3lDgRlp5NIWwNURcYOkxcBMSf8O3ANcns6/HPi+pHbgSbIRUETEIklXA4uBtcDJEfFyhXGbmVknlSWLiFgI7N1F+SN0MZopIl4APtDNvc4Fzu3tGM3MrBy/wW1mZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMytUWbKQNELSbZIWS1ok6bRU/gVJyyXdmz5H5a45U1K7pIckHZkrH5/K2iVNrSpmMzPrWv8K770WOCMi7pa0A7BA0px07MKI+Hr+ZEljgOOANwG7AT+XtGc6/E3gcGAZME/SrIhYXGHsZmaWU1myiIgVwIq0vUbSEmBYD5dMBGZGxIvAo5LagX3TsfaIeARA0sx0rpOFmVmDVFmzeIWkNmBv4E7gQOAUSR8D5pPVPp4iSyR35C5bxqvJ5bFO5ft18YwpwBSAkSNH9vJPsPlrm3pjt8eWTpvQwEjMrI4q7+CWtD1wLXB6RDwLXArsAYwlq3mc3xvPiYjLImJcRIwbPHhwb9zSzMySSmsWkrYkSxRXRsR1ABHxeO74d4Ab0u5yYETu8uGpjB7KzcysAaocDSXgcmBJRFyQKx+aO+19wANpexZwnKStJY0CRgN3AfOA0ZJGSdqKrBN8VlVxm5nZ+qqsWRwInADcL+neVHYWcLyksUAAS4GPA0TEIklXk3VcrwVOjoiXASSdAtwM9AOmR8SiCuM2M7NOqhwN9WtAXRya3cM15wLndlE+u6frzMysWn6D28zMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVqgwWUg6TdIAZS6XdLekIxoRnJmZ1UOZmsU/pdlijwB2IpvCY1qlUZmZWa2USRYdU3YcBXw/zcvU1TQeZma2mSqTLBZIuoUsWdyclkhdV21YZmZWJ2UmEpxMtlDRIxHxvKRdgJMqjcrMzGqlTM0igDHAqWl/O2CbyiIyM7PaKZMsLgHeDhyf9tcA36wsIjMzq50yzVD7RcQ+ku4BiIin0op1ZmbWIsrULF6S1I+sOQpJg3EHt5lZSymTLC4GfgzsKulc4NfAlyuNyszMaqWwGSoirpS0ADiM7P2KoyNiSeWRmZlZbXSbLCTtnNtdBVyVPxYRT1YZmJmZ1UdPNYsFZP0UXb2tHcDulURkZma1022yiIhRjQzEzMzqq8zQWSQdAxxEVqP4VURcX2VQZmZWL2WmKL8E+ARwP/AA8AlJfinPzKyFlKlZHAr8Q0R0vGcxA1hUaVR9WNvUG7s9tnTahAZGYmbWe8q8Z9EOjMztj0hlPZI0QtJtkhZLWiTptFS+s6Q5kh5O3zulckm6WFK7pIWS9snda1I6/2FJkzbsRzQzs01VJlnsACyRNFfSXGAxMEDSLEmzerhuLXBGRIwB9gdOljQGmArcGhGjgVvTPsC7gNHpMwW4FF4Zwns2sB+wL3B2R4IxM7PGKNMM9fmNuXFErABWpO01kpYAw4CJwCHptBnAXOAzqfx7qbnrDkkDJQ1N587peK9D0hxgPLn3PszMrFpl3uC+HUDSgPz5G/JSnqQ2YG/gTmBISiQAK4EhaXsY8FjusmWprLvyzs+YQlYjYeTIkZ0Pm5nZJigzGmqKpJXAQmA+2ct688s+QNL2wLXA6Wkt71ekWkRsUMTdiIjLImJcRIwbPHhwb9zSzMySMs1QnwbeHBFPbOjNJW1JliiujIjrUvHjkoZGxIrUzLQqlS8n6zzvMDyVLefVZquO8rkbGouZmW28Mh3cvwee39AbSxJwObAkIi7IHZoFdIxomgT8JFf+sTQqan/gmdRcdTNwhKSdUsf2EanMzMwapEzN4kzgt5LuBF7sKIyIU7u/BIADgROA+yXdm8rOAqYBV0uaDPwB+GA6Nhs4imxY7vOkdb4j4klJ5wDz0nlf8iSGZmaNVSZZfBv4Bdkb3KUXPYqIX9P1JISQTXfe+fwATu7mXtOB6WWfbWZmvatMstgyIj5VeSRmZlZbZfosfpZGRA1Nb1/v3GmtCzMz28yVqVkcn77PzJV5PQszsxZS5qU8r2thZtbiyq5n8WZgDLBNR1lEfK+qoMzMrF4Kk4Wks8leihtDNrz1XcCvAScLM7MWUaaD+1iyoa4rI+IkYC9gx0qjMjOzWimTLP4aEeuAtWkywVX8/bQcZma2mSvTZzFf0kDgO2STCD4H/K7KoMzMrF7KjIb6P2nzW5JuAgZExMJqwzIzszopM0X5gZK2S7sHASdKel21YZmZWZ2U6bO4FHhe0l7AGWSz0HoklJlZCymTLNamSf4mAt+IiG+SrcttZmYtokwH9xpJZwIfBQ6WtAWwZbVhmZlZnZSpWXyIbB2LyRGxkmyluq9VGpWZmdVKmdFQK4ELcvt/xH0WZmYtpUzNwszMWpyThZmZFeo2WUi6NX1/tXHhmJlZHfXUZzFU0gHAeyXNpNN62hFxd6WRmZlZbfSULD4PfI5s9NMFnY4FcGhVQZmZWb10mywi4hrgGkmfi4hzGhiT9TFtU2/s9tjSaRMaGImZVaXM0NlzJL0XODgVzY2IG6oNy8zM6qTMRIJfAU4DFqfPaZK+XHVgZmZWH2Wm+5gAjE0LICFpBnAPcFaVgZmZWX2Ufc9iYG671JKqkqZLWiXpgVzZFyQtl3Rv+hyVO3ampHZJD0k6Mlc+PpW1S5paMl4zM+tFZWoWXwHukXQb2fDZg4Ey/2hfAXyD9acGuTAivp4vkDQGOA54E7Ab8HNJe6bD3wQOB5YB8yTNiojFJZ5vZma9pEwH91WS5gJvS0WfSfNFFV33S0ltJeOYCMyMiBeBRyW1A/umY+0R8QhAet9jIlnfiZmZNUipZqiIWBERs9KnMFEUOEXSwtRMtVMqGwY8ljtnWSrrrnw9kqZImi9p/urVqzcxRDMzy2v03FCXAnsAY4EVwPm9deOIuCwixkXEuMGDB/fWbc3MjHJ9Fr0mIh7v2Jb0HaDjfY3lwIjcqcNTGT2Um5lZg/RYs5DUT9KDvfUwSUNzu+8DOkZKzQKOk7S1pFHAaOAuYB4wWtIoSVuRdYLP6q14zMysnB5rFhHxchq2OjItelSapKuAQ4BBkpYBZwOHSBpLNrfUUuDj6TmLJF1N1nG9Fjg5Il5O9zkFuBnoB0yPiEUbEoeZmW26Ms1QOwGLJN0F/KWjMCLe29NFEXF8F8WX93D+ucC5XZTPBmaXiNPMzCpSJll8rvIozMys1sq8Z3G7pNcBoyPi55K2JWsSMjOzFlFmIsH/BVwDfDsVDQOurzAmMzOrmTLvWZwMHAg8CxARDwO7VhmUmZnVS5lk8WJE/K1jR1J/stFMZmbWIsoki9slnQW8RtLhwI+An1YblpmZ1UmZZDEVWA3cT/ZexGzgs1UGZWZm9VJmNNS6tODRnWTNTw9FhJuhzMxaSGGykDQB+Bbwe7L1LEZJ+nhE/Kzq4MzMrB7KvJR3PvCOiGgHkLQHcCPgZGFm1iLK9Fms6UgUySPAmoriMTOzGuq2ZiHpmLQ5X9Js4GqyPosPkM0Ga2ZmLaKnZqj35LYfB/4xba8GXlNZRGZmVjvdJouIOKmRgZiZWX2VGQ01Cvgk0JY/v2iKcjMz23yUGQ11Pdk6FD8F1lUajZmZ1VKZZPFCRFxceSRmZlZbZZLFRZLOBm4BXuwojIi7K4vKzMxqpUyyeAtwAnAorzZDRdo3M7MWUCZZfADYPT9NuZmZtZYyb3A/AAysOA4zM6uxMjWLgcCDkubx930WHjprZtYiyiSLsyuPwszMaq3Meha3NyIQMzOrr8I+C0lrJD2bPi9IelnSsyWumy5plaQHcmU7S5oj6eH0vVMql6SLJbVLWihpn9w1k9L5D0uatLE/qJmZbbzCZBERO0TEgIgYQDaB4PuBS0rc+wpgfKeyqcCtETEauDXtA7wLGJ0+U4BLIUsuZM1g+wH7Amd3JBgzM2ucMqOhXhGZ64EjS5z7S+DJTsUTgRlpewZwdK78e+n+dwADJQ1Nz5kTEU9GxFPAHNZPQGZmVrEyEwkek9vdAhgHvLCRzxsSESvS9kpgSNoeBjyWO29ZKuuuvKs4p5DVShg5cuRGhmdmZl0pMxoqv67FWmApWU1gk0RESIpNvU/ufpcBlwGMGzeu1+5rZmblRkP15roWj0saGhErUjPTqlS+HBiRO294KlsOHNKpfG4vxmNmZiX0tKzq53u4LiLinI143ixgEjAtff8kV36KpJlkndnPpIRyM/DlXKf2EcCZG/FcMzPbBD3VLP7SRdl2wGRgF6DHZCHpKrJawSBJy8hGNU0DrpY0GfgD8MF0+mzgKKAdeB44CSAinpR0Dq+u+f2liOjcaW5mZhXraVnV8zu2Je0AnEb2j/hM4Pzurstdf3w3hw7r4twATu7mPtOB6UXPs81T29Qbuz22dNqEBkZi1tp67LNI7zl8CvgI2VDXfdIQVjMzayE99Vl8DTiGbITRWyLiuYZFZWZmtdLTS3lnALsBnwX+lJvyY02Z6T7MzGzz0VOfxQa93W1mZpsvJwQzMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFerf7ADMmqFt6o09Hl86bUKDIjHrG5pSs5C0VNL9ku6VND+V7SxpjqSH0/dOqVySLpbULmmhpH2aEbOZWStrZjPUOyJibESMS/tTgVsjYjRwa9oHeBcwOn2mAJc2PFIzsxZXpz6LicCMtD0DODpX/r3I3AEMlDS0CfGZmbWsZvVZBHCLpAC+HRGXAUMiYkU6vhIYkraHAY/lrl2WylbkypA0hazmwciRIzcpuJ7as92WbWatqFnJ4qCIWC5pV2COpAfzByMiUiIpLSWcywDGjRu3QdeamVnPmtIMFRHL0/cq4MfAvsDjHc1L6XtVOn05MCJ3+fBUZmZmDdLwZCFpO0k7dGwDRwAPALOASem0ScBP0vYs4GNpVNT+wDO55iozM2uAZjRDDQF+LKnj+f8VETdJmgdcLWky8Afgg+n82cBRQDvwPHBS40M2M2ttDU8WEfEIsFcX5X8GDuuiPICTGxCamZl1o05DZ83MrKacLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhL35kthE82aS1GtcszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkV8nQfZjXS0zQi4KlErHlcszAzs0JOFmZmVsjJwszMCjlZmJlZIXdwm7UId57bpnDNwszMCvWZZCFpvKSHJLVLmtrseMzMWkmfaIaS1A/4JnA4sAyYJ2lWRCxubmRmrcNLyba2PpEsgH2B9oh4BEDSTGAi4GRh1sdV2Zfifpreo4hodgyFJB0LjI+If077JwD7RcQpuXOmAFPS7huAhxoeaPcGAU80O4gCdY+x7vFB/WOse3xQ/xjrHh9sWoyvi4jBXR3oKzWLQhFxGXBZs+PoiqT5ETGu2XH0pO4x1j0+qH+MdY8P6h9j3eOD6mLsKx3cy4ERuf3hqczMzBqgrySLecBoSaMkbQUcB8xqckxmZi2jTzRDRcRaSacANwP9gOkRsajJYW2IWjaPdVL3GOseH9Q/xrrHB/WPse7xQUUx9okObjMza66+0gxlZmZN5GRhZmaFnCwqJGmEpNskLZa0SNJpzY6pK5L6SbpH0g3NjqUrkgZKukbSg5KWSHp7s2PKk/R/03/fByRdJWmbGsQ0XdIqSQ/kynaWNEfSw+l7pxrG+LX033mhpB9LGlin+HLHzpAUkgY1I7ZcHF3GKOmT6c9xkaTzeuNZThbVWgucERFjgP2BkyWNaXJMXTkNWNLsIHpwEXBTRLwR2IsaxSppGHAqMC4i3kw2AOO45kYFwBXA+E5lU4FbI2I0cGvab6YrWD/GOcCbI+KtwH8DZzY6qJwrWD8+JI0AjgD+2OiAunAFnWKU9A6yGS72iog3AV/vjQc5WVQoIlZExN1pew3ZP3LDmhvV35M0HJgAfLfZsXRF0o7AwcDlABHxt4h4uqlBra8/8BpJ/YFtgT81OR4i4pfAk52KJwIz0vYM4OhGxtRZVzFGxC0RsTbt3kH2TlVTdPNnCHAh8K9A00cHdRPj/wamRcSL6ZxVvfEsJ4sGkdQG7A3c2eRQOvt/ZP/jr2tyHN0ZBawG/jM1lX1X0nbNDqpDRCwn+83tj8AK4JmIuKW5UXVrSESsSNsrgSHNDKaEfwJ+1uwg8iRNBJZHxH3NjqUHewL/U9Kdkm6X9LbeuKmTRQNI2h64Fjg9Ip5tdjwdJL0bWBURC5odSw/6A/sAl0bE3sBfaH7zyStSu/9EsqS2G7CdpI82N6pikY2Zb/pvxt2R9G9kzbhXNjuWDpK2Bc4CPt/sWAr0B3Yma/r+NHC1JG3qTZ0sKiZpS7JEcWVEXNfseDo5EHivpKXATOBQST9obkjrWQYsi4iOGtk1ZMmjLt4JPBoRqyPiJeA64IAmx9SdxyUNBUjfvdI80dsknQi8G/hI1OtFsD3Ifim4L/2dGQ7cLem1TY1qfcuA6yJzF1mrwSZ3xDtZVChl88uBJRFxQbPj6SwizoyI4RHRRtYp+4uIqNVvxRGxEnhM0htS0WHUa2r6PwL7S9o2/fc+jBp1wHcyC5iUticBP2liLF2SNJ6sWfS9EfF8s+PJi4j7I2LXiGhLf2eWAfuk/0fr5HrgHQCS9gS2ohdmynWyqNaBwAlkv7Hfmz5HNTuoPuiTwJWSFgJjgS83N5xXpRrPNcDdwP1kf6eaPiWEpKuA3wFvkLRM0mRgGnC4pIfJakTTahjjN4AdgDnp78u3ahZfrXQT43Rg9zScdiYwqTdqaJ7uw8zMCrlmYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycL6NEnPVXz/09Obu5v8PElbS/p5GhL6od6JcKPiOFHSN5r1fOubnCzMenY62eSAvWFvgIgYGxE/7KV7mjWEk4VtdiTtIekmSQsk/UrSG1P5FZIulvRbSY9IOjaVbyHpkjT//xxJsyUdK+lUsvmebpN0W+7+50q6T9IdktabjC+tG3F9WpPhDklvlbQr8APgbalmsUena05Vtu7JQkkzU9m+kn6XJlD8bcdb7KlmcH2KdamkUyR9Kp13h6Sd03lzJV2UnveApH27iHWwpGslzUufA1P5P+ZeJL1H0g6981/H+qyI8MefPvsBnuui7FZgdNrej2waE8jm/v8R2S9JY4D2VH4sMDuVvxZ4Cjg2HVsKDMrdO4D3pO3zgM928fz/AM5O24cC96btQ4Abuvk5/gRsnbYHpu8BQP+0/U7g2rR9ItBO9qbzYOAZ4BPp2IVkE1YCzAW+k7YPBh7IXf+NtP1fwEFpeyTZ1DQAPwUOTNvbd8ThT+t++m9AXjGrvTTD7wHAj3ITbW6dO+X6iFgHLM7VCg4CfpTKV+ZrEV34G9CxouAC4PAuzjkIeD9ARPxC0i6SBhSEvpBsSpPryeb2AdgRmCFpNFmS2jJ3/m2RrZGyRtIzZP+4QzblyFtz512V4vilpAFaf+W5dwJjcn9WA9Kf4W+ACyRdSTYp3bKC+G0z52Rhm5stgKcjYmw3x1/MbW/MtM0vRUTHHDkv03t/hyaQ/fb/HuDfJL0FOIcsKbxP2Xooc3Pn53+Odbn9dZ1i6jyfT+f9LYD9I+KFTuXTJN0IHAX8RtKREfHghv1Itjlxn4VtViJbL+RRSR+AbOZfSXsVXPYb4P2p72IIWXNRhzVkzT0b4lfAR9LzDwGeiB7WMZG0BTAiIm4DPkNWo9g+fS9Pp524gTF0+FB6xkFkCzM90+n4LWQTNXbEMjZ97xHZLKtfBeYBb9zI59tmwsnC+rpt02ybHZ9Pkf1DPVnSfcAissWJenIt2XTTi8k6oe8m6weAbAbZmwqapjr7AvA/lM2SO41XpwXvTj/gB5LuB+4BLo5s6djzgK9IuoeNr8G8kK7/FtDVrKmnAuNSx/pi4BOp/PTUKb4QeImarVhnjedZZ83I+joi4jlJuwB3kXXu1m2dgg0iaS7wLxExv9mxWN/nPguzzA2p83cr4Jy+nijMeptrFmZmVsh9FmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaF/j/u1rDjB60EpAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEWCAYAAACe8xtsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcbklEQVR4nO3de7RdZXnv8e8v4aICIQFCGpJAIsZq1BI4EWjJAYrlPjSogFArgaaNjgMFhp4eA0eFlkZjFThFLhoGOQQLxAgCEaIQLglgyyUBDLnAYRtCk5gb9yCCJnnOH/NdMNnsvfacYc+91t7r9xljjfXOd96ePcciD+/7zvlORQRmZmZl9Gt0AGZm1vs4eZiZWWlOHmZmVpqTh5mZlebkYWZmpTl5mJlZaU4eZmZWmpOHWSckzZf0kqQdGx2LWbNx8jDrgKSRwH8HAvhMY6PpmKTtGh2DtS4nD7OOnQY8BFwLTKxVSrpW0hWS7pC0SdLDkvZN6yTpUkkbJL0q6UlJH5c0StLLkvql7a6WtCF3zB9LOjeVd5V0jaS1ktZI+hdJ/dO60yX9Kp3jBeBCSR+StEDSK5Kel/STnrpA1tqcPMw6dhpwffocLWlIbt0pwD8Bg4A2YGqqPwo4FPgwsCtwMvBCRDwLvArsn7Y7FHhN0kfT8mHAglS+FtgMfChtfxTwd7lzHwSsAIak814E3JViGQ784L392WbFOHmYtSNpPLAPMDsiFgG/Af46t8ktEfFIRGwmSy5jU/0fgV2AjwCKiOURsTatWwAcJulP0vJNaXkUMAD4dUpQxwHnRsTvImIDcClZsqr5bUT8ICI2R8Tv0zn3AfaKiDci4sHuvBZmnXHyMHu3icBdEfF8Wr6BXNcVsC5Xfh3YGSAi7gUuB64ANkiaLmlA2m4BcDhZq+N+YD5Zi+Mw4IGI2EqWBLYH1qZurpeBHwF75s63ql2s/wsQ8IikpZL+dhv/ZrNSPOBmliPp/WTdTf0l1ZLEjsBASft1tX9EXAZcJmlPYDbwj8A3yZLH94DVqfwg8EPgDd7usloFvAnskVo1HZ6i3fnWAX+fYh8P3C3p/ohoK/YXm20btzzM3ukEYAswhqw7aizwUeABsnGQTkn6pKSDJG0P/I4sMWwFiIhngN8DfwMsiIhXgfXA50nJI3Vx3QVcLGmApH6S9pV0WJ1zniRpeFp8iSy5bC3/Z5uV4+Rh9k4Tgf8bEf8VEetqH7LuqC9Sv7U+ALia7B/x54AXyFobNQvIBtBX5ZYFPJbb5jRgB2BZOs5NwNA65/wk8LCk14A5wDkRsaLQX2r2HsgvgzIzs7Lc8jAzs9KcPMzMrDQnDzMzK83Jw8zMSuuTz3nsscceMXLkyEaHYWbWqyxatOj5iBhcZNs+mTxGjhzJwoULGx2GmVmvIum5otu628rMzEpz8jAzs9KcPMzMrDQnDzMzK83Jw8zMSnPyMDOz0pw8zMysNCcPMzMrzcnDzMxK65NPmPdVI6fc0em6ldOO78FIzKzVueVhZmalVZY8JL1P0iOSfi1pqaR/SvWjJD0sqU3STyTtkOp3TMttaf3I3LHOS/VPSzq6qpjNzKyYKlsebwJHRMR+wFjgGEkHA98FLo2ID5G9o3lS2n4S8FKqvzRth6QxwCnAx4BjgCsl9a8wbjMz60JlySMyr6XF7dMngCOAm1L9TOCEVJ6QlknrPyVJqX5WRLwZEc8CbcCBVcVtZmZdq3TMQ1J/SU8AG4B5wG+AlyNic9pkNTAslYcBqwDS+leA3fP1HeyTP9dkSQslLdy4cWMFf42ZmdVUmjwiYktEjAWGk7UWPlLhuaZHxLiIGDd4cKF3mZiZ2TbqkbutIuJl4D7gz4GBkmq3CA8H1qTyGmAEQFq/K/BCvr6DfczMrAGqvNtqsKSBqfx+4EhgOVkSOTFtNhG4LZXnpGXS+nsjIlL9KelurFHAaOCRquI2M7OuVfmQ4FBgZrozqh8wOyJul7QMmCXpX4DHgWvS9tcAP5bUBrxIdocVEbFU0mxgGbAZODMitlQYt5mZdaGy5BERi4H9O6hfQQd3S0XEG8BJnRxrKjC1u2M0M7Nt4yfMzcysNCcPMzMrzcnDzMxKc/IwM7PSnDzMzKw0Jw8zMyvNycPMzEpz8jAzs9KcPMzMrDQnDzMzK83Jw8zMSnPyMDOz0pw8zMysNCcPMzMrzcnDzMxKc/IwM7PSnDzMzKw0Jw8zMyvNycPMzEpz8jAzs9KcPMzMrDQnDzMzK83Jw8zMSnPyMDOz0ipLHpJGSLpP0jJJSyWdk+ovlLRG0hPpc1xun/MktUl6WtLRufpjUl2bpClVxWxmZsVsV+GxNwNfi4jHJO0CLJI0L627NCK+n99Y0hjgFOBjwF7A3ZI+nFZfARwJrAYelTQnIpZVGLuZmdVRWfKIiLXA2lTeJGk5MKzOLhOAWRHxJvCspDbgwLSuLSJWAEialbZ18jAza5AeGfOQNBLYH3g4VZ0labGkGZIGpbphwKrcbqtTXWf1ZmbWIJUnD0k7AzcD50bEq8BVwL7AWLKWycXddJ7JkhZKWrhx48buOKSZmXWi0uQhaXuyxHF9RPwMICLWR8SWiNgKXM3bXVNrgBG53Yenus7q3yEipkfEuIgYN3jw4O7/Y8zM7C2VjXlIEnANsDwiLsnVD03jIQCfBZak8hzgBkmXkA2YjwYeAQSMljSKLGmcAvx1VXFXaeSUO+quXznt+B6KxMzsvanybqtDgC8BT0p6ItWdD5wqaSwQwErgywARsVTSbLKB8M3AmRGxBUDSWcCdQH9gRkQsrTBuMzPrQpV3Wz1I1mpob26dfaYCUzuon1tvPzMz61l+wtzMzEpz8jAzs9KcPMzMrLQqB8ytB/lOLjPrSW55mJlZaU4eZmZWmpOHmZmV5uRhZmaldZk8JJ0jaYAy10h6TNJRPRGcmZk1pyItj79Ns+EeBQwim3JkWqVRmZlZUyuSPGpTjBwH/DjNK9XRtCNmZtYiiiSPRZLuIksed6ZXym6tNiwzM2tmRR4SnET24qYVEfG6pN2BMyqNyszMmlqRlkcAY4Cz0/JOwPsqi8jMzJpekeRxJfDnwKlpeRNwRWURmZlZ0yvSbXVQRBwg6XGAiHhJ0g4Vx2VmZk2sSMvjj5L6k3VfIWkwHjA3M2tpRZLHZcAtwJ6SpgIPAt+uNCozM2tqXXZbRcT1khYBnyJ7vuOEiFheeWRmZta0Ok0eknbLLW4Absyvi4gXqwzMzMyaV72WxyKycY6OniYP4IOVRGRmZk2v0+QREaN6MhAzM+s9Cr2GVtLngPFkLY4HIuLWKoMyM7PmVmRK9iuBrwBPAkuAr0jyQ4JmZi2sSMvjCOCjEVF7zmMmsLTSqMzMrKkVec6jDdg7tzwi1dUlaYSk+yQtk7RU0jmpfjdJ8yQ9k74HpXpJukxSm6TFkg7IHWti2v4ZSRPL/YlmZtbdiiSPXYDlkuZLmg8sAwZImiNpTp39NgNfi4gxwMHAmZLGAFOAeyJiNHBPWgY4FhidPpOBq+CtW4YvAA4CDgQuqCUcMzNrjCLdVt/algNHxFpgbSpvkrQcGAZMAA5Pm80E5gNfT/XXpe6xhyQNlDQ0bTuv9lyJpHnAMeSeOzEzs55V5AnzBQCSBuS3L/OQoKSRwP7Aw8CQlFgA1gFDUnkYsCq32+pU11l9+3NMJmuxsPfee7dfbWZm3ajI3VaTJa0DFgMLyR4eXFj0BJJ2Bm4Gzk3vQn9LamVEqYg7ERHTI2JcRIwbPHhwdxzSzMw6UaTb6h+Bj0fE82UPLml7ssRxfUT8LFWvlzQ0ItambqkNqX4N2WB8zfBUt4a3u7lq9fPLxmJmZt2nyID5b4DXyx5YkoBrgOURcUlu1RygdsfUROC2XP1p6a6rg4FXUvfWncBRkgalgfKjUp2ZmTVIkZbHecB/SHoYeLNWGRFnd74LAIcAXwKelPREqjsfmAbMljQJeA44Oa2bCxxHdhvw66T3pEfEi5IuAh5N2/2zJ2U0M2usIsnjR8C9ZE+YF34JVEQ8SMeTKkI2vXv77QM4s5NjzQBmFD23mZlVq0jy2D4ivlp5JGZm1msUGfP4Rbrjamh6Ony3du/6MDOzFlOk5XFq+j4vV+f3eZiZtbAiDwn6vR5mZvYORd/n8XFgDPC+Wl1EXFdVUGZm1ty6TB6SLiB7SG8M2e20xwIPAk4eZmYtqsiA+Ylkt9aui4gzgP2AXSuNyszMmlqR5PH7iNgKbE6TI27gndOImJlZiyky5rFQ0kDgarJJEV8D/rPKoMzMrLkVudvqf6TiDyX9EhgQEYurDcvMzJpZkSnZD5G0U1ocD5wuaZ9qwzIzs2ZWZMzjKuB1SfsBXyObZdd3WpmZtbAiyWNzmrRwAnB5RFxB9l5zMzNrUUUGzDdJOg/4G+BQSf2A7asNy8zMmlmRlscXyN7jMSki1pG9ye97lUZlZmZNrcjdVuuAS3LL/4XHPMzMWlqRloeZmdk7OHmYmVlpnSYPSfek7+/2XDhmZtYb1BvzGCrpL4DPSJpFu/eRR8RjlUZmZmZNq17y+BbwTbK7qy5pty6AI6oKyszMmlunySMibgJukvTNiLioB2MyM7MmV+RW3YskfQY4NFXNj4jbqw3LzMyaWZGJEb8DnAMsS59zJH276sDMzKx5FZme5HhgbHohFJJmAo8D51cZmJmZNa+iz3kMzJULvYJW0gxJGyQtydVdKGmNpCfS57jcuvMktUl6WtLRufpjUl2bpCkF4zUzswoVaXl8B3hc0n1kt+seChT5R/xa4HLePZXJpRHx/XyFpDHAKcDHgL2AuyV9OK2+AjgSWA08KmlORCwrcH4zM6tIkQHzGyXNBz6Zqr6e5rvqar/7JY0sGMcEYFZEvAk8K6kNODCta4uIFQDpeZMJZGMvZmbWIIW6rSJibUTMSZ8uE0cXzpK0OHVrDUp1w4BVuW1Wp7rO6t9F0mRJCyUt3Lhx43sM0czM6unpua2uAvYFxgJrgYu768ARMT0ixkXEuMGDB3fXYc3MrANFxjy6TUSsr5UlXQ3UnhdZA4zIbTo81VGn3szMGqRuy0NSf0lPddfJJA3NLX4WqN2JNQc4RdKOkkYBo4FHgEeB0ZJGSdqBbFB9TnfFY2Zm26ZuyyMitqTbZPdOL4EqTNKNwOHAHpJWAxcAh0saSzY31krgy+k8SyXNJhsI3wycGRFb0nHOAu4E+gMzImJpmTjMzKz7Fem2GgQslfQI8LtaZUR8pt5OEXFqB9XX1Nl+KjC1g/q5wNwCcZqZWQ8pkjy+WXkU1nAjp9zR6bqV047vwUjMrDco8pzHAkn7AKMj4m5JHyDrQjIzsxZVZGLEvwduAn6UqoYBt1YYk5mZNbkiz3mcCRwCvAoQEc8Ae1YZlJmZNbciyePNiPhDbUHSdmR3S5mZWYsqkjwWSDofeL+kI4GfAj+vNiwzM2tmRZLHFGAj8CTZcxlzgW9UGZSZmTW3IndbbU0vgHqYrLvq6Yhwt5WZWQvrMnlIOh74IfAbsvd5jJL05Yj4RdXBmZlZcyrykODFwF9GRBuApH2BOwAnDzOzFlVkzGNTLXEkK4BNFcVjZma9QKctD0mfS8WFkuYCs8nGPE4im+3WzMxaVL1uq0/nyuuBw1J5I/D+yiIyM7Om12nyiIgzejIQMzPrPYrcbTUK+AdgZH77rqZkNzOzvqvI3Va3kr2H4+fA1kqjMTOzXqFI8ngjIi6rPBIzM+s1iiSPf5N0AXAX8GatMiIeqywqMzNrakWSxyeALwFH8Ha3VaRlMzNrQUWSx0nAB/PTspuZWWsr8oT5EmBgxXGYmVkvUqTlMRB4StKjvHPMw7fqmpm1qCLJ44LKozAzs16lyPs8FvREIGZm1nt0OeYhaZOkV9PnDUlbJL1aYL8ZkjZIWpKr203SPEnPpO9BqV6SLpPUJmmxpANy+0xM2z8jaeK2/qFmZtZ9ukweEbFLRAyIiAFkEyJ+HriywLGvBY5pVzcFuCciRgP3pGWAY4HR6TMZuAqyZEPWbXYQcCBwQS3hmJlZ4xS52+otkbkVOLrAtvcDL7arngDMTOWZwAm5+uvS8R8CBkoams4zLyJejIiXgHm8OyGZmVkPKzIx4udyi/2AccAb23i+IRGxNpXXAUNSeRiwKrfd6lTXWb2ZmTVQkbut8u/12AysJGspvCcREZLivR6nRtJksi4v9t577+46rJmZdaDI3Vbd+V6P9ZKGRsTa1C21IdWvAUbkthue6tYAh7ern99JnNOB6QDjxo3rtqRkZmbvVu81tN+qs19ExEXbcL45wERgWvq+LVd/lqRZZIPjr6QEcyfw7dwg+VHAedtwXjMz60b1Wh6/66BuJ2ASsDtQN3lIupGs1bCHpNVkd01NA2ZLmgQ8B5ycNp8LHAe0Aa8DZwBExIuSLuLtd6b/c0S0H4Q3M7MeVu81tBfXypJ2Ac4h+0d9FnBxZ/vl9j+1k1Wf6mDbAM7s5DgzgBldnc/MzHpO3TGP9JzFV4Evkt1ae0C6ZdbMzFpYvTGP7wGfIxuE/kREvNZjUZmZWVOr95Dg14C9gG8Av81NUbKpyPQkZmbWd9Ub8yj19LmZmbUOJwgzMyvNycPMzEorMj2JWV0jp9zR6bqV047vwUjMrKe45WFmZqU5eZiZWWlOHmZmVpqTh5mZlebkYWZmpfluq5Lq3VkEvrvIzFqDWx5mZlaak4eZmZXm5GFmZqU5eZiZWWlOHmZmVpqTh5mZlebkYWZmpTl5mJlZaU4eZmZWmpOHmZmV5uRhZmalOXmYmVlpDUkeklZKelLSE5IWprrdJM2T9Ez6HpTqJekySW2SFks6oBExm5nZ2xrZ8vjLiBgbEePS8hTgnogYDdyTlgGOBUanz2Tgqh6P1MzM3qGZuq0mADNTeSZwQq7+usg8BAyUNLQB8ZmZWdKo5BHAXZIWSZqc6oZExNpUXgcMSeVhwKrcvqtT3TtImixpoaSFGzdurCpuMzOjcS+DGh8RayTtCcyT9FR+ZUSEpChzwIiYDkwHGDduXKl9zcysnIa0PCJiTfreANwCHAisr3VHpe8NafM1wIjc7sNTnZmZNUiPJw9JO0napVYGjgKWAHOAiWmzicBtqTwHOC3ddXUw8Eque8vMzBqgEd1WQ4BbJNXOf0NE/FLSo8BsSZOA54CT0/ZzgeOANuB14IyeD9nMzPJ6PHlExApgvw7qXwA+1UF9AGf2QGhmZlZQM92qa2ZmvUSj7rYyA2DklDvqrl857fgeisTMynDLw8zMSnPyMDOz0pw8zMysNCcPMzMrzcnDzMxKc/IwM7PSnDzMzKw0Jw8zMyvNycPMzEpz8jAzs9I8PYk1tXrTl3jqErPGccvDzMxKc/IwM7PSnDzMzKw0Jw8zMyvNycPMzEpz8jAzs9KcPMzMrDQnDzMzK80PCVqf5QcMzarj5GHWgXqJB5x8zNxtZWZmpTl5mJlZab0meUg6RtLTktokTWl0PGZmraxXjHlI6g9cARwJrAYelTQnIpZVcb6u+rvN6nkvv5+uxlI8FmPNolckD+BAoC0iVgBImgVMACpJHmZ90XtNPL57zfIUEY2OoUuSTgSOiYi/S8tfAg6KiLNy20wGJqfFPwWeBvYAnu/hcJuVr0XG1yHj65DxdcjUrsM+ETG4yA69peXRpYiYDkzP10laGBHjGhRSU/G1yPg6ZHwdMr4OmW25Dr1lwHwNMCK3PDzVmZlZA/SW5PEoMFrSKEk7AKcAcxock5lZy+oV3VYRsVnSWcCdQH9gRkQsLbDr9K43aRm+Fhlfh4yvQ8bXIVP6OvSKAXMzM2suvaXbyszMmoiTh5mZldZnk4enM8lIWinpSUlPSFrY6Hh6kqQZkjZIWpKr203SPEnPpO9BjYyxJ3RyHS6UtCb9Lp6QdFwjY+wJkkZIuk/SMklLJZ2T6lvqN1HnOpT6TfTJMY80ncn/IzedCXBqVdOZNDNJK4FxEdFyD0JJOhR4DbguIj6e6v4VeDEipqX/qRgUEV9vZJxV6+Q6XAi8FhHfb2RsPUnSUGBoRDwmaRdgEXACcDot9Juocx1OpsRvoq+2PN6aziQi/gDUpjOxFhIR9wMvtqueAMxM5Zlk/9H0aZ1ch5YTEWsj4rFU3gQsB4bRYr+JOtehlL6aPIYBq3LLq9mGi9NHBHCXpEVpCpdWNyQi1qbyOmBII4NpsLMkLU7dWn26q6Y9SSOB/YGHaeHfRLvrACV+E301edjbxkfEAcCxwJmpC8OAyPps+16/bTFXAfsCY4G1wMUNjaYHSdoZuBk4NyJeza9rpd9EB9eh1G+iryYPT2eSRMSa9L0BuIWsS6+VrU99vrW+3w0NjqchImJ9RGyJiK3A1bTI70LS9mT/YF4fET9L1S33m+joOpT9TfTV5OHpTABJO6UBMSTtBBwFLKm/V583B5iYyhOB2xoYS8PU/rFMPksL/C4kCbgGWB4Rl+RWtdRvorPrUPY30SfvtgJIt5n9H96ezmRqYyPqeZI+SNbagGwqmhta6TpIuhE4nGy66fXABcCtwGxgb+A54OSI6NODyZ1ch8PJuicCWAl8Odfv3ydJGg88ADwJbE3V55P197fMb6LOdTiVEr+JPps8zMysOn2128rMzCrk5GFmZqU5eZiZWWlOHmZmVpqTh5mZlebkYb2apNcqPv65kj7QHeeTtKOku9OMpV/ongi3KY7TJV3eqPNb3+DkYVbfucAHutqooP0BImJsRPykm45p1hBOHtbnSNpX0i/TZJAPSPpIqr9W0mWS/kPSCkknpvp+kq6U9FR6n8NcSSdKOhvYC7hP0n2540+V9GtJD0l61yR66f0Qt6YJ5h6S9GeS9gT+Hfhkanns226fs9P7FRZLmpXqDpT0n5IeTzH/aao/PR1/nrL3tZwl6atpu4ck7Za2my/p39L5lkh613QTkgZLulnSo+lzSKo/LPdeh8drMxWYvSUi/PGn137I3j/Qvu4eYHQqHwTcm8rXAj8l+5+mMWTT9gOcCMxN9X8CvAScmNatBPbIHTuAT6fyvwLf6OD8PwAuSOUjgCdS+XDg9k7+jt8CO6bywPQ9ANgulf8KuDmVTwfagF2AwcArwFfSukvJJroDmA9cncqHAkty+1+eyjeQTZ4J2RPWy1P558AhqbxzLQ5//Kl9tiuRZ8yaXpop9C+An2ZT+ACwY26TWyOb+G1ZrtUwHvhpql+Xb2V04A/A7am8iOyFY+2NBz4PEBH3Stpd0oAuQl8MXC/pVrIpVAB2BWZKGk2WtLbPbX9fZO9i2CTpFbJ/7CGbcuLPctvdmOK4X9IASQPbnfevgDG5azUgXcNfAZdIuh74WUSs7iJ+azFOHtbX9ANejoixnax/M1dWJ9vU88eIqM3ps4Xu+2/oeLLWwaeB/y3pE8BFZEnis+m9C/Nz2+f/jq255a3tYmo//1D75X7AwRHxRrv6aZLuAI4DfiXp6Ih4qtyfZH2ZxzysT4nsvQTPSjoJshlEJe3XxW6/Aj6fxj6GkHUv1Wwi6x4q4wHgi+n8hwPPR7v3RuRJ6geMiIj7gK+TtTh2Tt+1VwmcXjKGmi+kc4wHXomIV9qtvwv4h1wsY9P3vhHxZER8l2yW6o9s4/mtj3LysN7uA5JW5z5fJfuHe5KkXwNL6foVxDeTvW1yGdmg9mNk4wgA04FfdtGV1d6FwH+TtBiYxtvTfXemP/Dvkp4EHgcui4iXycZUviPpcba9hfNG2v+HwKQO1p8NjEsD9cuAr6T6c9Mg+2Lgj8AvtvH81kd5Vl0zsrGSiHhN0u7AI2SDxesaHdd7IWk+8D8jYmGjY7G+x2MeZpnb02DyDsBFvT1xmFXNLQ8zMyvNYx5mZlaak4eZmZXm5GFmZqU5eZiZWWlOHmZmVtr/BwHod7r9Zcz7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(nrows=1, ncols=2)\n",
    "axs = axes.ravel()\n",
    "\n",
    "axs[0].boxplot(q_len)\n",
    "axs[0].set_title(\"Length of questions.\")\n",
    "\n",
    "axs[1].boxplot(a_len)\n",
    "axs[1].set_title(\"Length of answers.\")\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "fig2, ax2 = plt.subplots()\n",
    "ax2.hist(q_len, bins=40)\n",
    "ax2.set_xlabel(\"Length of samples\")\n",
    "ax2.set_ylabel(\"Number of samples\")\n",
    "ax2.set_title(\"Questions\")\n",
    "\n",
    "fig3, ax3 = plt.subplots()\n",
    "ax3.hist(a_len, bins=40)\n",
    "ax3.set_xlabel(\"Length of samples\")\n",
    "ax3.set_ylabel(\"Number of samples\")\n",
    "ax3.set_title(\"Answers\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 705,
   "id": "hundred-recipient",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "95% of q_len is :7.0\n",
      "95% of a_len is :8.0\n"
     ]
    }
   ],
   "source": [
    "print(f\"95% of q_len is :{np.percentile(q_len, 95)}\")\n",
    "print(f\"95% of a_len is :{np.percentile(a_len, 95)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stable-joyce",
   "metadata": {},
   "source": [
    "데이터셋의 크기가 크지 않으므로 최대길이를 예제와 같이 25으로 정해 모든 단어를 포함하는 방향으로 진행해보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 706,
   "id": "false-cancer",
   "metadata": {},
   "outputs": [],
   "source": [
    "def below_threshold_len(max_len, nested_list):\n",
    "    cnt = 0\n",
    "    for s in nested_list:\n",
    "        if(len(s.split()) <= max_len):\n",
    "            cnt = cnt+1\n",
    "            \n",
    "    print(f\"전체 샘플 중 길이가 {max_len} 이하인 샘플의 비율: {cnt/len(nested_list):.2%}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 707,
   "id": "literary-frederick",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Questions\n",
      ">>>\n",
      "전체 샘플 중 길이가 25 이하인 샘플의 비율: 100.00%\n",
      "Answers\n",
      ">>>\n",
      "전체 샘플 중 길이가 25 이하인 샘플의 비율: 100.00%\n"
     ]
    }
   ],
   "source": [
    "print(\"Questions\\n>>>\")\n",
    "below_threshold_len(25, questions)\n",
    "print(\"Answers\\n>>>\")\n",
    "below_threshold_len(25, answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 708,
   "id": "designed-cambodia",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25\n"
     ]
    }
   ],
   "source": [
    "MAX_LENGTH = 25\n",
    "print(MAX_LENGTH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 709,
   "id": "bronze-tackle",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_and_filter(inputs, outputs):\n",
    "    tokenized_inputs, tokenized_outputs = [], []\n",
    "    \n",
    "    for (sentence1, sentence2) in zip(inputs, outputs):\n",
    "        sentence1 = START_TOKEN + tokenizer.encode(sentence1) + END_TOKEN\n",
    "        sentence2 = START_TOKEN + tokenizer.encode(sentence2) + END_TOKEN\n",
    "        \n",
    "        if len(sentence1) <= MAX_LENGTH and len(sentence2) <= MAX_LENGTH:\n",
    "            tokenized_inputs.append(sentence1)\n",
    "            tokenized_outputs.append(sentence2)\n",
    "    \n",
    "    tokenized_inputs = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_inputs, maxlen=MAX_LENGTH, padding='post')\n",
    "    \n",
    "    tokenized_outputs = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_outputs, maxlen=MAX_LENGTH, padding='post')\n",
    "    \n",
    "    return tokenized_inputs, tokenized_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 710,
   "id": "special-process",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어장의 크기 : 8350\n",
      "필터링 후의 질문 샘플 개수: 11817\n",
      "필터링 후의 답변 샘플 개수: 11817\n"
     ]
    }
   ],
   "source": [
    "questions, answers = tokenize_and_filter(questions, answers)\n",
    "print('단어장의 크기 :',(VOCAB_SIZE))\n",
    "print('필터링 후의 질문 샘플 개수: {}'.format(len(questions)))\n",
    "print('필터링 후의 답변 샘플 개수: {}'.format(len(answers)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 711,
   "id": "afraid-control",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "BUFFER_SIZE = 20000\n",
    "\n",
    "dataset = tf.data.Dataset.from_tensor_slices((\n",
    "    {\n",
    "        'inputs': questions,\n",
    "        'dec_inputs': answers[:, :-1]\n",
    "    },\n",
    "    {\n",
    "        'outputs': answers[:, 1:]\n",
    "    }\n",
    "))\n",
    "\n",
    "dataset = dataset.cache()\n",
    "dataset = dataset.shuffle(BUFFER_SIZE)\n",
    "dataset = dataset.batch(BATCH_SIZE)\n",
    "dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 752,
   "id": "expired-lawrence",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"transformer\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "inputs (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dec_inputs (InputLayer)         [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "enc_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "encoder (Functional)            (None, None, 512)    8480768     inputs[0][0]                     \n",
      "                                                                 enc_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "look_ahead_mask (Lambda)        (None, 1, None, None 0           dec_inputs[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dec_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "decoder (Functional)            (None, None, 512)    10584064    dec_inputs[0][0]                 \n",
      "                                                                 encoder[0][0]                    \n",
      "                                                                 look_ahead_mask[0][0]            \n",
      "                                                                 dec_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "outputs (Dense)                 (None, None, 8350)   4283550     decoder[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 23,348,382\n",
      "Trainable params: 23,348,382\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "\n",
    "\n",
    "NUM_LAYERS = 2\n",
    "D_MODEL = 512\n",
    "NUM_HEADS = 8\n",
    "UNITS = 1024\n",
    "DROPOUT = 0.1\n",
    "\n",
    "model = transformer(\n",
    "    vocab_size=VOCAB_SIZE,\n",
    "    num_layers=NUM_LAYERS,\n",
    "    units=UNITS,\n",
    "    d_model=D_MODEL,\n",
    "    num_heads=NUM_HEADS,\n",
    "    dropout=DROPOUT)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 753,
   "id": "precious-dependence",
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_function(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH - 1))\n",
    "    \n",
    "    loss = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "        from_logits=True, reduction='none')(y_true, y_pred)\n",
    "    \n",
    "    mask = tf.cast(tf.not_equal(y_true, 0), tf.float32)\n",
    "    loss = tf.multiply(loss, mask)\n",
    "    \n",
    "    return tf.reduce_mean(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 754,
   "id": "billion-leave",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "    \n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super(CustomSchedule, self).__init__()\n",
    "        \n",
    "        self.d_model = d_model\n",
    "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "        \n",
    "        self.warmup_steps = warmup_steps\n",
    "        \n",
    "    def __call__(self, step):\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps**-1.5)\n",
    "        \n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 755,
   "id": "incredible-flour",
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = CustomSchedule(D_MODEL)\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(\n",
    "    learning_rate, beta_1=0.9, beta_2=0.98, epsilon=1e-9)\n",
    "\n",
    "def accuracy(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH-1))\n",
    "    return tf.keras.metrics.sparse_categorical_accuracy(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 756,
   "id": "sized-cisco",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer, loss=loss_function, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 757,
   "id": "dominant-accountability",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "185/185 [==============================] - 36s 160ms/step - loss: 2.3819 - accuracy: 0.0329\n",
      "Epoch 2/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 1.7825 - accuracy: 0.0803\n",
      "Epoch 3/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 1.5807 - accuracy: 0.0845\n",
      "Epoch 4/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 1.4597 - accuracy: 0.0915\n",
      "Epoch 5/50\n",
      "185/185 [==============================] - 29s 158ms/step - loss: 1.3298 - accuracy: 0.0987\n",
      "Epoch 6/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 1.2034 - accuracy: 0.1090\n",
      "Epoch 7/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 1.0534 - accuracy: 0.1257\n",
      "Epoch 8/50\n",
      "185/185 [==============================] - 29s 158ms/step - loss: 0.8746 - accuracy: 0.1474\n",
      "Epoch 9/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 0.7124 - accuracy: 0.1723\n",
      "Epoch 10/50\n",
      "185/185 [==============================] - 29s 155ms/step - loss: 0.5377 - accuracy: 0.1981\n",
      "Epoch 11/50\n",
      "185/185 [==============================] - 29s 158ms/step - loss: 0.3750 - accuracy: 0.2274\n",
      "Epoch 12/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 0.2491 - accuracy: 0.2464\n",
      "Epoch 13/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.1522 - accuracy: 0.2648\n",
      "Epoch 14/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.0925 - accuracy: 0.2720\n",
      "Epoch 15/50\n",
      "185/185 [==============================] - 29s 158ms/step - loss: 0.0620 - accuracy: 0.2727\n",
      "Epoch 16/50\n",
      "185/185 [==============================] - 30s 161ms/step - loss: 0.0493 - accuracy: 0.2746\n",
      "Epoch 17/50\n",
      "185/185 [==============================] - 29s 157ms/step - loss: 0.0469 - accuracy: 0.2746\n",
      "Epoch 18/50\n",
      "185/185 [==============================] - 30s 162ms/step - loss: 0.0430 - accuracy: 0.2752\n",
      "Epoch 19/50\n",
      "185/185 [==============================] - 29s 157ms/step - loss: 0.0437 - accuracy: 0.2730\n",
      "Epoch 20/50\n",
      "185/185 [==============================] - 29s 156ms/step - loss: 0.0420 - accuracy: 0.2771\n",
      "Epoch 21/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 0.0433 - accuracy: 0.2742\n",
      "Epoch 22/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 0.0429 - accuracy: 0.2725\n",
      "Epoch 23/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.0412 - accuracy: 0.2763\n",
      "Epoch 24/50\n",
      "185/185 [==============================] - 30s 165ms/step - loss: 0.0395 - accuracy: 0.2750\n",
      "Epoch 25/50\n",
      "185/185 [==============================] - 30s 161ms/step - loss: 0.0331 - accuracy: 0.2772\n",
      "Epoch 26/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 0.0296 - accuracy: 0.2772\n",
      "Epoch 27/50\n",
      "185/185 [==============================] - 29s 158ms/step - loss: 0.0278 - accuracy: 0.2780\n",
      "Epoch 28/50\n",
      "185/185 [==============================] - 31s 166ms/step - loss: 0.0252 - accuracy: 0.2795\n",
      "Epoch 29/50\n",
      "185/185 [==============================] - 30s 161ms/step - loss: 0.0234 - accuracy: 0.2789\n",
      "Epoch 30/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.0214 - accuracy: 0.2784\n",
      "Epoch 31/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 0.0212 - accuracy: 0.2786\n",
      "Epoch 32/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.0186 - accuracy: 0.2804\n",
      "Epoch 33/50\n",
      "185/185 [==============================] - 30s 161ms/step - loss: 0.0177 - accuracy: 0.2804\n",
      "Epoch 34/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 0.0159 - accuracy: 0.2799\n",
      "Epoch 35/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.0172 - accuracy: 0.2792\n",
      "Epoch 36/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.0146 - accuracy: 0.2807\n",
      "Epoch 37/50\n",
      "185/185 [==============================] - 29s 158ms/step - loss: 0.0148 - accuracy: 0.2801\n",
      "Epoch 38/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.0135 - accuracy: 0.2805\n",
      "Epoch 39/50\n",
      "185/185 [==============================] - 30s 164ms/step - loss: 0.0123 - accuracy: 0.2813\n",
      "Epoch 40/50\n",
      "185/185 [==============================] - 30s 161ms/step - loss: 0.0108 - accuracy: 0.2816\n",
      "Epoch 41/50\n",
      "185/185 [==============================] - 30s 162ms/step - loss: 0.0119 - accuracy: 0.2824\n",
      "Epoch 42/50\n",
      "185/185 [==============================] - 30s 163ms/step - loss: 0.0117 - accuracy: 0.2813\n",
      "Epoch 43/50\n",
      "185/185 [==============================] - 30s 161ms/step - loss: 0.0111 - accuracy: 0.2800\n",
      "Epoch 44/50\n",
      "185/185 [==============================] - 30s 160ms/step - loss: 0.0101 - accuracy: 0.2827\n",
      "Epoch 45/50\n",
      "185/185 [==============================] - 30s 164ms/step - loss: 0.0107 - accuracy: 0.2833\n",
      "Epoch 46/50\n",
      "185/185 [==============================] - 30s 162ms/step - loss: 0.0090 - accuracy: 0.2835\n",
      "Epoch 47/50\n",
      "185/185 [==============================] - 29s 159ms/step - loss: 0.0097 - accuracy: 0.2819\n",
      "Epoch 48/50\n",
      "185/185 [==============================] - 30s 164ms/step - loss: 0.0078 - accuracy: 0.2814\n",
      "Epoch 49/50\n",
      "185/185 [==============================] - 30s 161ms/step - loss: 0.0088 - accuracy: 0.2814\n",
      "Epoch 50/50\n",
      "185/185 [==============================] - 30s 162ms/step - loss: 0.0091 - accuracy: 0.2830\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fbf374ffa90>"
      ]
     },
     "execution_count": 757,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EPOCHS = 50\n",
    "callback = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5)\n",
    "model.fit(dataset, epochs=EPOCHS, verbose=1, callbacks=[callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 758,
   "id": "unknown-reason",
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder_inference(sentence):\n",
    "    sentence = preprocess_sentence(sentence)\n",
    "\n",
    "  # 입력된 문장을 정수 인코딩 후, 시작 토큰과 종료 토큰을 앞뒤로 추가.\n",
    "  # ex) Where have you been? → [[8331   86   30    5 1059    7 8332]]\n",
    "    sentence = tf.expand_dims(\n",
    "      START_TOKEN + tokenizer.encode(sentence) + END_TOKEN, axis=0)\n",
    "\n",
    "  # 디코더의 현재까지의 예측한 출력 시퀀스가 지속적으로 저장되는 변수.\n",
    "  # 처음에는 예측한 내용이 없음으로 시작 토큰만 별도 저장. ex) 8331\n",
    "    output_sequence = tf.expand_dims(START_TOKEN, 0)\n",
    "\n",
    "  # 디코더의 인퍼런스 단계\n",
    "    for i in range(MAX_LENGTH):\n",
    "        predictions = model(inputs=[sentence, output_sequence], training=False)\n",
    "        predictions = predictions[:, -1:, :]\n",
    "\n",
    "        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
    "\n",
    "    # 만약 현재 예측한 단어가 종료 토큰이라면 for문을 종료\n",
    "        if tf.equal(predicted_id, END_TOKEN[0]):\n",
    "            break\n",
    "\n",
    "    # 예측한 단어들은 지속적으로 output_sequence에 추가됩니다.\n",
    "\n",
    "        output_sequence = tf.concat([output_sequence, predicted_id], axis=-1)\n",
    "\n",
    "    return tf.squeeze(output_sequence, axis=0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 759,
   "id": "backed-making",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence_generation(sentence):\n",
    "  # 입력 문장에 대해서 디코더를 동작 시켜 예측된 정수 시퀀스를 리턴받습니다.\n",
    "    prediction = decoder_inference(sentence)\n",
    "\n",
    "    # 정수 시퀀스를 다시 텍스트 시퀀스로 변환합니다.\n",
    "    predicted_sentence = tokenizer.decode(\n",
    "      [i for i in prediction if i < tokenizer.vocab_size])\n",
    " \n",
    "    print('입력 : {}'.format(sentence))\n",
    "    print('출력 : {}'.format(predicted_sentence))\n",
    "\n",
    "    return predicted_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 760,
   "id": "homeless-ribbon",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 출근길 너무 힘들어\n",
      "출력 : 회사 근처로 이사를 가보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'회사 근처로 이사를 가보세요 .'"
      ]
     },
     "execution_count": 760,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('출근길 너무 힘들어')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 761,
   "id": "broad-hospital",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 안녕, 넌 이름이 뭐니?\n",
      "출력 : 저는 위로봇입니다 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'저는 위로봇입니다 .'"
      ]
     },
     "execution_count": 761,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('안녕, 넌 이름이 뭐니?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 762,
   "id": "radical-contract",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때?\n",
      "출력 : 사랑에 나이는 중요하지 않다고 생각해요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'사랑에 나이는 중요하지 않다고 생각해요 .'"
      ]
     },
     "execution_count": 762,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 763,
   "id": "spread-sleeve",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때??\n",
      "출력 : 사랑에 나이는 중요하지 않다고 생각해요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'사랑에 나이는 중요하지 않다고 생각해요 .'"
      ]
     },
     "execution_count": 763,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때??')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 764,
   "id": "perceived-pontiac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어떠니?\n",
      "출력 : 사랑을 말과 행동으로 표현해보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'사랑을 말과 행동으로 표현해보세요 .'"
      ]
     },
     "execution_count": 764,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어떠니?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 765,
   "id": "commercial-recruitment",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어땡?\n",
      "출력 : 친구들과 같이 놀러가세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'친구들과 같이 놀러가세요 .'"
      ]
     },
     "execution_count": 765,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어땡?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 766,
   "id": "variable-pioneer",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘 날씨는 어때ㅋㅋㅋㅋㅋ\n",
      "출력 : 정말 나쁜 생각 하지 마세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'정말 나쁜 생각 하지 마세요 .'"
      ]
     },
     "execution_count": 766,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘 날씨는 어때ㅋㅋㅋㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 767,
   "id": "boring-algorithm",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋ\n",
      "출력 : 항상 사랑했던 걸 수도 있어요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'항상 사랑했던 걸 수도 있어요 .'"
      ]
     },
     "execution_count": 767,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 768,
   "id": "sixth-iraqi",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋㅋ\n",
      "출력 : 항상 사랑했던 걸 수도 있어요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'항상 사랑했던 걸 수도 있어요 .'"
      ]
     },
     "execution_count": 768,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 769,
   "id": "independent-recipient",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : ㅋㅋㅋ\n",
      "출력 : 항상 사랑했던 걸 수도 있어요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'항상 사랑했던 걸 수도 있어요 .'"
      ]
     },
     "execution_count": 769,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('ㅋㅋㅋ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 770,
   "id": "alternate-bridal",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 날씨 좋아?\n",
      "출력 : 날씨 어플에 물어보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'날씨 어플에 물어보세요 .'"
      ]
     },
     "execution_count": 770,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('날씨 좋아?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 771,
   "id": "surgical-texture",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 날씨 좋아!\n",
      "출력 : 하늘을 보고 웃어보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'하늘을 보고 웃어보세요 .'"
      ]
     },
     "execution_count": 771,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('날씨 좋아!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "illegal-child",
   "metadata": {},
   "source": [
    "## 8. 회고\n",
    "\n",
    "1. 전처리가 효과가 있었는 지 문장을 생성할 때 문제점이 개선된 것 같아서 좋았다.\n",
    "2. 하지만 아직 부자연스러운 문장들이 생성될 때가 많고 정확도도 높은 수준이 아니라 더 학습을 오래 시켜야겠다.\n",
    "3. 추가적으로 KoBERT를 이용한 챗봇만들기를 따라서 구현하면서 PyTorch도 익혀야겠습니다.\n",
    "[KoBERT-chatbot](https://github.com/nawnoes/WellnessConversation-LanguageModel)\n",
    "4. Tokenizer로 이전에 사용한 Mecab()등을 사용해 개선되는 지 확인해봐야겠다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}