EXPLORATION/Node_08/[E-08] news_summary_bot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8.뉴스 요약봇 만들기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8-13. 프로젝트: 뉴스기사 요약해보기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 목차\n",
    "\n",
    "### Step 1. 데이터 수집하기\n",
    "\n",
    "### Step 2. 데이터 전처리하기 (추상적 요약)\n",
    "\n",
    "### Step 3. 어텐션 메커니즘 사용하기 (추상적 요약)\n",
    "\n",
    "### Step 4. 실제 결과와 요약문 비교하기 (추상적 요약)\n",
    "\n",
    "### Step 5. Summa을 이용해서 추출적 요약해보기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\User\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')  # NLTK 패키지에서 불용어 사전 다운로드\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import re\n",
    "import matplotlib.pyplot as plt\n",
    "from nltk.corpus import stopwords\n",
    "from bs4 import BeautifulSoup \n",
    "from tensorflow.keras.preprocessing.text import Tokenizer \n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "import urllib.request\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning, module='bs4')\n",
    "\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. 데이터 수집하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "데이터는 뉴스 기사 데이터([news_summary_more.csv](https://github.com/sunnysai12345/News_Summary))를 사용\n",
    "\n",
    "=> 기사의 본문에 해당되는 text와 headlines 두 가지 열로 구성됨\n",
    "\n",
    "- 추상적 요약: text 본문, headlines 이미 요약된 데이터로 삼아서 모델 학습\n",
    "- 추출적 요약: 오직 text열만 사용하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "\n",
    "# 데이터 다운\n",
    "DATA_PATH = 'news_summarization/data/news_summary_more.csv'\n",
    "if not os.path.exists(DATA_PATH):\n",
    "    urllib.request.urlretrieve(\"https://raw.githubusercontent.com/sunnysai12345/News_Summary/master/news_summary_more.csv\", filename=DATA_PATH)\n",
    "\n",
    "data = pd.read_csv(DATA_PATH, encoding='iso-8859-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>23018</th>\n",
       "      <td>Computer virus halts iPhone chipmaker TSMC's f...</td>\n",
       "      <td>A computer virus on Friday night halted the pr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77079</th>\n",
       "      <td>Over 6,000 female defectors 'returned' to N Korea</td>\n",
       "      <td>In a report to the UN Committee on the Elimina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13226</th>\n",
       "      <td>UP policeman who shouted 'Thain Thain' nominat...</td>\n",
       "      <td>The UP police has hailed Manoj Kumar for his \"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98228</th>\n",
       "      <td>Trump blasts Snoop Dogg over clown shooting vi...</td>\n",
       "      <td>After rapper Snoop Dogg made a music video fea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22065</th>\n",
       "      <td>SRK to narrate first 3 episodes of 'Kasautii.....</td>\n",
       "      <td>Shah Rukh Khan will introduce the lead charact...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34238</th>\n",
       "      <td>Veg diet healthy, non-veg unhealthy: Health Mi...</td>\n",
       "      <td>The Health Ministry had recently posted an ima...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85882</th>\n",
       "      <td>216-kg ice cream sculpture displayed at Crimea...</td>\n",
       "      <td>Crimea's Sevastopol city hosted the first ever...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4880</th>\n",
       "      <td>Huawei gear to be removed from UK emergency se...</td>\n",
       "      <td>Britain's BT Group will remove Huawei equipmen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25246</th>\n",
       "      <td>4-year-old allegedly raped by Congress leader'...</td>\n",
       "      <td>A four-year-old girl was allegedly raped by a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1250</th>\n",
       "      <td>Noida, Gurugram have most number of unsold hom...</td>\n",
       "      <td>NCR's Gurugram and Noida have the most number ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               headlines  \\\n",
       "23018  Computer virus halts iPhone chipmaker TSMC's f...   \n",
       "77079  Over 6,000 female defectors 'returned' to N Korea   \n",
       "13226  UP policeman who shouted 'Thain Thain' nominat...   \n",
       "98228  Trump blasts Snoop Dogg over clown shooting vi...   \n",
       "22065  SRK to narrate first 3 episodes of 'Kasautii.....   \n",
       "34238  Veg diet healthy, non-veg unhealthy: Health Mi...   \n",
       "85882  216-kg ice cream sculpture displayed at Crimea...   \n",
       "4880   Huawei gear to be removed from UK emergency se...   \n",
       "25246  4-year-old allegedly raped by Congress leader'...   \n",
       "1250   Noida, Gurugram have most number of unsold hom...   \n",
       "\n",
       "                                                    text  \n",
       "23018  A computer virus on Friday night halted the pr...  \n",
       "77079  In a report to the UN Committee on the Elimina...  \n",
       "13226  The UP police has hailed Manoj Kumar for his \"...  \n",
       "98228  After rapper Snoop Dogg made a music video fea...  \n",
       "22065  Shah Rukh Khan will introduce the lead charact...  \n",
       "34238  The Health Ministry had recently posted an ima...  \n",
       "85882  Crimea's Sevastopol city hosted the first ever...  \n",
       "4880   Britain's BT Group will remove Huawei equipmen...  \n",
       "25246  A four-year-old girl was allegedly raped by a ...  \n",
       "1250   NCR's Gurugram and Noida have the most number ...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 랜덤 10개 샘플 출력\n",
    "data.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플수 : 98401\n"
     ]
    }
   ],
   "source": [
    "print('전체 샘플수 :', (len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. 데이터 전처리하기 (추상적 요약)\n",
    "실습에서 사용된 전처리를 참고하여 각자 필요하다고 생각하는 전처리를 추가 사용하여 텍스트를 정규화 또는 정제해 보세요. 만약, 불용어 제거를 선택한다면 상대적으로 길이가 짧은 요약 데이터에 대해서도 불용어를 제거하는 것이 좋을지 고민해 보세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 데이터 중복 샘플 제거\n",
    "- 유일값 개수 세기: pandas.unique()\n",
    "- 중복 제거: pandas.drop_duplicates()\n",
    "\n",
    "text 중복은 제거해야 함\n",
    "\n",
    "text가 달라도 headlines는 같을 수 있어서 중복 제거X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text 열에서 중복을 배제한 유일한 샘플의 수 : 98360\n",
      "headlines 열에서 중복을 배제한 유일한 샘플의 수 : 98280\n"
     ]
    }
   ],
   "source": [
    "print('text 열에서 중복을 배제한 유일한 샘플의 수 :', data['text'].nunique())\n",
    "print('headlines 열에서 중복을 배제한 유일한 샘플의 수 :', data['headlines'].nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플수 : 98360\n"
     ]
    }
   ],
   "source": [
    "# inplace=True 를 설정하면 DataFrame 타입 값을 return 하지 않고 data 내부를 직접적으로 바꿉니다\n",
    "data.drop_duplicates(subset = ['text'], inplace=True)\n",
    "print('전체 샘플수 :', (len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Null 값 제거\n",
    "- Null 개수 확인: data.isnull().sum()\n",
    "- Null 제거: data.dropna()\n",
    "\n",
    "다행히 현재 데이터에는 Null 값이 없음"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "headlines    0\n",
      "text         0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(data.isnull().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플수 : 98360\n"
     ]
    }
   ],
   "source": [
    "data.dropna(axis=0, inplace=True)\n",
    "print('전체 샘플수 :', (len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 텍스트 정규화와 불용어 제거\n",
    "- 텍스트 정규화(text normalization)\n",
    "    - 같은 의미 다른 표현의 단어를 같은 표현으로 통일시킴\n",
    "    - => 기계 연산량 감소\n",
    "    - [정규화 사전](https://stackoverflow.com/questions/19790188/expanding-english-language-contractions-in-python) 구성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "정규화 사전의 수:  120\n"
     ]
    }
   ],
   "source": [
    "# 정규화 사전 구성\n",
    "contractions = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\",\n",
    "                           \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\",\n",
    "                           \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",\n",
    "                           \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\",\n",
    "                           \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\",\n",
    "                           \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\",\n",
    "                           \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\",\n",
    "                           \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\",\n",
    "                           \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\",\n",
    "                           \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\",\n",
    "                           \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\",\n",
    "                           \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\",\n",
    "                           \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\",\n",
    "                           \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\",\n",
    "                           \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\",\n",
    "                           \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",\n",
    "                           \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\",\n",
    "                           \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\",\n",
    "                           \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\",\n",
    "                           \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\",\n",
    "                           \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\n",
    "                           \"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\",\n",
    "                           \"you're\": \"you are\", \"you've\": \"you have\"}\n",
    "\n",
    "print(\"정규화 사전의 수: \", len(contractions))  # 120"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 불용어(stopwords)\n",
    "    - 텍스트에 자주 등장하지만 자연어 처리에 실질적 도움이 되지 않는 단어\n",
    "    - NLTK 제공 불용어 리스트 사용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "불용어 개수 : 179\n",
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
     ]
    }
   ],
   "source": [
    "# NLTK 제공 불용어 리스트\n",
    "print('불용어 개수 :', len(stopwords.words('english') ))  # 179\n",
    "print(stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 데이터 전처리\n",
    "    - 모든 영어를 소문자로 변경\n",
    "    - html 태그 제거\n",
    "    - 정규 표현식 사용하여 각종 특수문자 제거\n",
    "    - 불용어 제거\n",
    "        - text 전처리에서만 사용\n",
    "        - 자연스러운 요약을 위해 headlines는 불용어를 남김"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "# 데이터 전처리 함수\n",
    "def preprocess_sentence(sentence, remove_stopwords=True):\n",
    "    sentence = sentence.lower() # 텍스트 소문자화\n",
    "    sentence = BeautifulSoup(sentence, \"lxml\").text # <br />, <a href = ...> 등의 html 태그 제거\n",
    "    sentence = re.sub(r'\\([^)]*\\)', '', sentence) # 괄호로 닫힌 문자열 (...) 제거 Ex) my husband (and myself!) for => my husband for\n",
    "    sentence = re.sub('\"','', sentence) # 쌍따옴표 \" 제거\n",
    "    sentence = ' '.join([contractions[t] if t in contractions else t for t in sentence.split(\" \")]) # 약어 정규화\n",
    "    sentence = re.sub(r\"'s\\b\",\"\", sentence) # 소유격 제거. Ex) roland's -> roland\n",
    "    sentence = re.sub(\"[^a-zA-Z]\", \" \", sentence) # 영어 외 문자(숫자, 특수문자 등) 공백으로 변환\n",
    "    sentence = re.sub('[m]{2,}', 'mm', sentence) # m이 3개 이상이면 2개로 변경. Ex) ummmmmmm yeah -> umm yeah\n",
    "    \n",
    "    # 불용어 제거 (text)\n",
    "    if remove_stopwords:\n",
    "        tokens = ' '.join(word for word in sentence.split() if not word in stopwords.words('english') if len(word) > 1)\n",
    "    # 불용어 미제거 (headlines)\n",
    "    else:\n",
    "        tokens = ' '.join(word for word in sentence.split() if len(word) > 1)\n",
    "    return tokens\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text 전처리 후 결과:  ['saurav kant alumnus upgrad iiit pg program machine learning artificial intelligence sr systems engineer infosys almost years work experience program upgrad degree career support helped transition data scientist tech mahindra salary hike upgrad online power learning powered lakh careers', 'kunal shah credit card bill payment platform cred gave users chance win free food swiggy one year pranav kaushik delhi techie bagged reward spending cred coins users get one cred coin per rupee bill paid used avail rewards brands like ixigo bookmyshow ubereats cult fit', 'new zealand defeated india wickets fourth odi hamilton thursday win first match five match odi series india lost international match rohit sharma captaincy consecutive victories dating back march match witnessed india getting seventh lowest total odi cricket history', 'aegon life iterm insurance plan customers enjoy tax benefits premiums paid save taxes plan provides life cover age years also customers options insure critical illnesses disability accidental death benefit rider life cover age years', 'speaking sexual harassment allegations rajkumar hirani sonam kapoor said known hirani many years true metoo movement get derailed metoo movement always believe woman case need reserve judgment added hirani accused assistant worked sanju']\n"
     ]
    }
   ],
   "source": [
    "# 전체 text 데이터에 대한 전처리 : 10분 이상 시간이 걸릴 수 있습니다. \n",
    "clean_text = []\n",
    "for s in data['text']:\n",
    "    clean_text.append(preprocess_sentence(s))\n",
    "\n",
    "# 전처리 후 출력\n",
    "print(\"text 전처리 후 결과: \", clean_text[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "headlines 전처리 후 결과:  ['upgrad learner switches to career in ml al with salary hike', 'delhi techie wins free food from swiggy for one year on cred', 'new zealand end rohit sharma led india match winning streak', 'aegon life iterm insurance plan helps customers save tax', 'have known hirani for yrs what if metoo claims are not true sonam']\n"
     ]
    }
   ],
   "source": [
    "# 전체 headlines 데이터에 대한 전처리 : 5분 이상 시간이 걸릴 수 있습니다. \n",
    "clean_headlines = []\n",
    "for s in data['headlines']:\n",
    "    clean_headlines.append(preprocess_sentence(s, False))\n",
    "\n",
    "print(\"headlines 전처리 후 결과: \", clean_headlines[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 텍스트 정제 후 샘플 확인\n",
    "    - empty 샘플 모두 Null 값으로 대체\n",
    "    - Null 샘플 모두 제거"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "data['text'] = clean_text\n",
    "data['headlines'] = clean_headlines\n",
    "\n",
    "# 빈 값을 Null 값으로 변환\n",
    "data.replace('', np.nan, inplace=True)\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "headlines    0\n",
       "text         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플수 : 98360\n"
     ]
    }
   ],
   "source": [
    "data.dropna(axis=0, inplace=True)\n",
    "print('전체 샘플수 :', (len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 샘플 최대 길이 정하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "텍스트의 최소 길이 : 1\n",
      "텍스트의 최대 길이 : 60\n",
      "텍스트의 평균 길이 : 35.09968483123221\n",
      "요약의 최소 길이 : 1\n",
      "요약의 최대 길이 : 16\n",
      "요약의 평균 길이 : 9.299532330215534\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAb/UlEQVR4nO3dfZBV9Z3n8fenW9KIISJL63ZU0pkNaqfZaGJv4gzuJkQQJskKWyUTqUmWaEemddPJrGbTak/WWFMQ2Y0zyZAUvTgwUBO31XVMZFLZ8NiYwrgmjVEjtEY34wORQCtgHFwINt/94x5I03bD7Yd7z7n3fl5Vp+49v3MfviA/P/d3Hn5HEYGZmVnWVKVdgJmZ2WAcUGZmlkkOKDMzyyQHlJmZZZIDyszMMskBZWZmmeSAMrOSIekFSbMK/B31kkLSacn6VkmfT57/qaQNhfx++z0HVIkYq45ZjA5uVq4i4p6IuDLtOiqFA8rMzDLJAVUCJP09MBX4R0n/LOkrki6T9BNJByQ9KeljyWv/SNKrks5P1i9OXnPRYJ+T1p/JbBQukfSUpNcl3SdpPICkT0l6Ivn3/hNJHzj2Bkm3SPq/kt6QtFPSf+i3rVrSN5J+8yvgk0N9saTPSdrWbz0ktUh6TtJ+Sd+RpH7br5PUk2xbL+k9Sbsk/bWkvcmf4ylJ08f476n0RYSXEliAF4BZyfNzgdeAT5D7kTE7Wa9Nti8BtgCnA08BXxjsc7x4KbUl+ff7U+DdwGSgB2gBPgTsBT4CVAOLktfWJO9bkLynCvg0cBCoS7a1AM8A5yef2QUEcFqyfSvw+eT554Bt/eoJ4AfAJHI//nqBucm2+cDzQANwGvAXwE+SbXOA7cn7lLymLu2/36wtHkGVps8AP4yIH0bE0YjYCHSTCyyArwFnkuvIrwDfSaVKs8L4m4h4JSL2Af8IXAJcD/yPiHgsIvoiYi1wGLgMICL+V/KeoxFxH/Ac8OHk8/4E+GZEvJx85teHWc+dEXEgIl4iF26XJO1/Bnw9Inoi4i1gKbnR33uAI8BE4CJAyWt2j+Qvo5w5oErTe4AFya6MA5IOAJcDdQARcQRYA0wH7orkJ5tZmfhNv+dvAu8k1yduHtAnzic3akLSf+y3++8Aub4xJfmMdwMv9/vMF8egHpKavtXvO/eRGy2dGxFbgG+T+/G4R9JKSe8a5veWPQdU6egfMi8Dfx8Rk/otZ0TEnQCSzgVuB/4OuEtSzRCfY1YuXgaWDOgTEyKiMxmx3A18AfgXETEJeJpcWADsJhdmx0wdw5r+bEBNp0fETwAi4m8i4lKgEbgA+C9j9L1lwwFVOvYAf5A8/y7w7yXNSQ7wjpf0MUnnJQdo1wCrgGZyne8vh/gcs3JxN9Ai6SPJCQhnSPqkpInAGeR+mPUCSLqW3AjqmPuBLyb95yzgljGqqQO4VVJj8r1nSlqQPP83Sa3jyB0POwT0jdH3lg0HVOn4OvAXya6CTwPzgNvIdbqXyf36qgK+CJwDfDXZtXctcK2kfzvwcyR9ubh/BLPCiIhucsehvg3sJ3dywueSbTuBu4BHyf1A+9fAI/3efjewHngSeBx4cIxq+h6wDLhX0m/Jjdr+ONn8ruR795Pbpfga8I2x+N5yIh+eMDOzLPIIyszMMskBZWZmmeSAMjOzTHJAmZlZJp1WzC+bMmVK1NfXF/MrzQpm+/btr0ZEbRrf7b5k5WSovlTUgKqvr6e7u7uYX2lWMJKGO+PAmHFfsnIyVF/yLj4zM8skB5SZmWWSA8rMzDLJAWVmZpnkgDIzs0xyQJmZWSblFVCSJkl6QNIzknok/aGkyZI2SnoueTyr0MXayXV2djJ9+nSqq6uZPn06nZ2daZdk/UhaLWmvpKcHtLdKelbSDkn/La367PfmzJlDVVUVkqiqqmLOnDlpl1SR8h1BfQv4UURcBFwM9JC7Z8rmiJgGbGbs7qFiI9DZ2Ul7ezvLly/n0KFDLF++nPb2dodUtqwB5vZvkDST3K1TPhARjfiWC6mbM2cOGzZsoKWlhQMHDtDS0sKGDRscUmmIiJMu5O5b8k8kt+bo1/4sUJc8rwOePdVnXXrppWGF0djYGFu2bDmhbcuWLdHY2JhSReUP6I5T/JsfuAD1wNP91u8HZg33c9yXCkdS3HDDDSe03XDDDSEppYrK31B96ZT3g5J0CbAS2Elu9LQd+BLw68jdOvnY6/ZHxNt280laDCwGmDp16qUvvpjaxfdlrbq6mkOHDjFu3LjjbUeOHGH8+PH09flGnYUgaXtENA3zPfXADyJierL+BPAQuZHVIeDLEfGzId7rvlQEkjhw4ABnnnnm8bbXX3+dSZMmcar/X9rIDNWX8tnFdxrwIWBFRHyQ3O2J896dFxErI6IpIppqa1OZtqwiNDQ0sG3bthPatm3bRkNDQ0oVWZ5OA84CLiN3V+T7JWmwF7ovFYckbr311hPabr31Vob4z2IFlE9A7QJ2RcRjyfoD5AJrj6Q6gORxb2FKtHy0t7fT3NxMV1cXR44coauri+bmZtrb29MuzU5uF/Bgsqfjp8BRYErKNVW02bNns2LFCm688UZef/11brzxRlasWMHs2bPTLq3inHKy2Ij4jaSXJV0YEc8CV5Db3bcTWATcmTw+VNBK7aQWLlwIQGtrKz09PTQ0NLBkyZLj7ZZZ3wc+DmyVdAHwDuDVVCuqcOvXr2fOnDl0dHSwYsUKJHHllVeyfv36tEurOPnOZt4K3CPpHcCvgGvJjb7ul9QMvAQsKEyJlq+FCxc6kDJMUifwMWCKpF3A7cBqYHVy6vnvgEXhAx2pcxhlQ14BFRFPAIMdDL5iTKsxK2MRMdSvh88UtRCzEuGZJMzMLJMcUGZmlkkOKDMzyyQHlJmZZZIDyszMMinf08zNzCrGYLNG+Oz/4vMIysysn/7hdO+99w7absXhgDIzG0RE8OlPf9ojpxQ5oMzMBug/chps3YrDAVVGfEdds7FxzTXXnHTdisMBVSZ8R12zsSWJ++67z8eeUuSAKhNLlixh1apVzJw5k3HjxjFz5kxWrVrFkiVL0i7NrKT0P+bUf+TkY1HF59PMy0RPTw+XX375CW2XX345PT09KVVkVrocRtngEVSZaGho4I477jjhGNQdd9zhO+qaWclyQJWJmTNnsmzZMq677jreeOMNrrvuOpYtW8bMmTPTLs3MbEQcUGWiq6uLtrY2Vq9ezcSJE1m9ejVtbW10dXWlXZqZ2Yj4GFSZ6Onpoa6ujp07dxIR7Ny5k7q6Oh+DMrOS5RFUmTj99NPZtGkTLS0tHDhwgJaWFjZt2sTpp5+edmlmZiPigCoTBw8eZOLEiSxYsIAJEyawYMECJk6cyMGDB9MuzcxsRBxQZeSuu+6itbWV8ePH09rayl133ZV2SdaPpNWS9kp6epBtX5YUkqakUZudSNLbFis+B1SZkERbWxs7duzg6NGj7Nixg7a2NnesbFkDzB3YKOl8YDbwUrELsrcbqs+4LxWfA6pMTJgwgf3791NfX8/zzz9PfX09+/fvZ8KECWmXZomI+DGwb5BNfw18BfDVoRkSEccXS4fP4isTBw8eZMqUKbz44ou8733vQxJTpkzh1VdfTbs0OwlJVwG/jognT/ULXdJiYDHA1KlTi1CdWbo8giojtbW1x3/tRQS1tbUpV2QnI2kC0A7813xeHxErI6IpIpr839YqgQOqjPT09HDVVVfR29vLVVdd5Wugsu9fAe8FnpT0AnAe8Likf5lqVQbgEyQywLv4zFISEb8Azj62noRUU0R4v2yKImLQUPKxqOJzQJWRiy66iHXr1h3ftXfRRRfxzDPPpFyVHSOpE/gYMEXSLuD2iFiVblU2GIdRNuQVUMkvuzeAPuCtiGiSNBm4D6gHXgD+JCL2F6ZMy8fAMHI4ZUtELDzF9voilWJWEoZzDGpmRFwSEU3J+i3A5oiYBmxO1i0DHnjggbRLMDMbtdGcJDEPWJs8XwvMH3U1NiauvvrqtEswMxu1fAMqgA2StifXYgCcExG7AZLHswd7o6TFkroldff29o6+YhvSpk2bTri4cNOmTWmXZGY2YvmeJDEjIl6RdDawUVLeBzciYiWwEqCpqclHHgto1qxZaZdgZjZm8hpBRcQryeNe4HvAh4E9kuoAkse9hSrShmfZsmVpl2BmNmqnDChJZ0iaeOw5cCXwNLAOWJS8bBHwUKGKtOFpa2tLuwQzs1HLZxffOcD3kgvXTgP+Z0T8SNLPgPslNZObhXlB4co0M7NKc8oRVET8KiIuTpbGiFiStL8WEVdExLTkcbBZmi0FX/3qV9Muwcxs1DwXX5mpqqriox/9KFVV/k9rlo/Bbk6Yz2KF56mOyszRo0d9Np/ZMJxsWiNJnvYoRf6ZbWZmmeSAMjOzTHJAmZlZJjmgzMwskxxQZmaWSQ6oMnTOOeekXYKZ2ag5oMrQnj170i7BzGzUfB1Umel/zYYvJjSzUuaAKjMOJTMrF97FVyaGutrdV8Fnh6TVkvZKerpf23+X9IykpyR9T9KkFEs0yxQHVInKd24wzyGWKWuAuQPaNgLTI+IDwC+BW4tdlFlWOaBKVP9buw9c8tluxRcRPwb2DWjbEBFvJav/Bziv6IWZZZQDyiw7rgP+d9pFmGWFA8osAyS1A28B95zkNYsldUvq7u3tLV5xZilxQJmlTNIi4FPAn8ZJ9sFGxMqIaIqIptra2uIVaJYSn2ZuliJJc4E24KMR8Wba9ZhliUdQZkUiqRN4FLhQ0i5JzcC3gYnARklPSOpItUizDPEIyqxIImLhIM2ril6IWYnwCMrMzDLJAWVmZpnkgDIzs0xyQJmZWSY5oMzMLJMcUGZmlkkOKDMzy6S8A0pStaSfS/pBsj5Z0kZJzyWPZxWuTDMzqzTDGUF9Cejpt34LsDkipgGbk3UzM7MxkVdASToP+CTwt/2a5wFrk+drgfljWpmZmVW0fEdQ3wS+Ahzt13ZOROwGSB7PHuyNvkWAmZmNxCkDStKngL0RsX0kX+BbBJiZ2UjkM1nsDOAqSZ8AxgPvkvRdYI+kuojYLakO2FvIQs3MrLKccgQVEbdGxHkRUQ9cA2yJiM8A64BFycsWAQ8VrEozM6s4o7kO6k5gtqTngNnJupmZ2ZgY1v2gImIrsDV5/hpwxdiXZGZm5pkkzMwsoxxQGTZ58mQkDXsBhv2eyZMnp/ynNTM7kW/5nmH79+8nIoryXceCzcwsKzyCMjOzTHJAmRWJpNWS9kp6ul+bJ102G4IDyqx41gBzB7R50mWzITigzIokIn4M7BvQ7EmXzYbggDJLV16TLoMnXh4NnxFbmnwWn1mJiIiVwEqApqam4pzeWSZ8Rmxp8gjKLF17ksmW8aTLZidyQJmly5Mumw3BAWVWJJI6gUeBCyXtktSMJ102G5KPQWVY3P4u+NqZxfsuK6iIWDjEJk+6bDYIB1SG6Y7fFvXAbnytKF9lZpYX7+IzM7NMckCZmVkmOaDMzCyTHFBmZpZJDigzM8skn8WXccWaNuWss3yXBzPLFgdUho30FHNJRTs93cysUBxQZlb2fNF7aXJAmVnZ80XvpcknSZiZWSY5oMzMLJMcUGZmlkkOKDMzy6RTBpSk8ZJ+KulJSTsk3ZG0T5a0UdJzyaMvpDEzszGTzwjqMPDxiLgYuASYK+ky4BZgc0RMAzYn62ZmZmPilAEVOf+crI5LlgDmAWuT9rXA/EIUaGZmlSmvY1CSqiU9AewFNkbEY8A5EbEbIHk8e4j3LpbULam7t7d3jMo2M7Nyl1dARURfRFwCnAd8WNL0fL8gIlZGRFNENNXW1o6wTDMzqzTDOosvIg4AW4G5wB5JdQDJ496xLs6sUkj6z8lJSE9L6pQ0Pu2azNKWz1l8tZImJc9PB2YBzwDrgEXJyxYBDxWoRrOyJulc4ItAU0RMB6qBa9Ktyix9+czFVweslVRNLtDuj4gfSHoUuF9SM/ASsKCAdZqVu9OA0yUdASYAr6Rcj1nqThlQEfEU8MFB2l8DrihEUWaVJCJ+Lekb5H7o/T9gQ0RsGPg6SYuBxQBTp04tbpFlwPdWKz2eScIsZclF7vOA9wLvBs6Q9JmBr/MJRyMXESNaRvLeffv2pfynLR8OKLP0zQL+KSJ6I+II8CDwRynXZJY6B5RZ+l4CLpM0Qbn9UFcAPSnXZJY6B5RZypIL3x8AHgd+Qa5frky1KLMM8B11zTIgIm4Hbk+7DrMs8QjKzMwyyQFlZmaZ5IAyM7NM8jGoEnWqiw5Ptv3Y9R1mZlnmgCpRg4XMYKHkMDKzUuVdfGViqBFTsaZ3MTMbax5BlZn+IyaHk5mVMgdUmXEomVm58C4+MzPLJAeUmZllkgPKzMwyyQFlZmaZ5IAyM7NMckCZmVkmOaDKyLx580649fS8efPSLsnMbMR8HVQZeeihh3wdlJmVDY+gytDFF1+cdglmZqPmgCpDTz75ZNolmJmNmgPKzMwyyQFVZqqrq9m6dSvV1dVpl2LDIGmSpAckPSOpR9Ifpl2TWdp8kkSZ6evr49VXX6Wvry/tUmx4vgX8KCKulvQOYELaBZmlzQFVhq6++uq0S7BhkPQu4N8BnwOIiN8Bv0uzJrMsOOUuPknnS+pKdjvskPSlpH2ypI2Snksezyp8uWZl6Q+AXuDvJP1c0t9KOmPgiyQtltQtqbu3t7f4VZoVWT7HoN4Cbo6IBuAy4D9Jej9wC7A5IqYBm5N1y4Dvf//7aZdgw3Ma8CFgRUR8EDjIIP0pIlZGRFNENNXW1ha7RrOiO2VARcTuiHg8ef4G0AOcC8wD1iYvWwvML1CNNkzz589PuwQbnl3Aroh4LFl/gFxgmVW0YZ3FJ6ke+CDwGHBOROyGXIgBZw/xHu+WKJJrr72WmpoaAGpqarj22mtTrsjyERG/AV6WdGHSdAWwM8WSzDIh74CS9E7gH4A/j4jf5vs+75YonjVr1rB06VIOHjzI0qVLWbNmTdolWf5agXskPQVcAixNtxyz9OUVUJLGkQuneyLiwaR5j6S6ZHsdsLcwJVo+JBERPPzww7z55ps8/PDDRITn5isREfFE8kPuAxExPyL2p12TWdryOYtPwCqgJyL+qt+mdcCi5Pki4KGxL8/yFRE0Njaybt06amtrWbduHY2NjURE2qWZmY1IPtdBzQA+C/xC0hNJ223AncD9kpqBl4AFBanQ8lJTU8OkSZOoqanh8OHDJ6ybmZWifM7i2xYRSnY9XJIsP4yI1yLiioiYljzuK0bBNrgLLriARx55hDlz5tDb28ucOXN45JFHuOCCC9IuzcxsRDyTRJn45S9/yYwZM1i/fj21tbXU1NQwY8YMuru70y7NzGxEHFBl4vDhw2zYsIEJE34/hdubb77JGWe8bUICM7OS4NnMy0RNTQ0dHR0ntHV0dPgYlJmVLI+gysT1119PW1sbAC0tLXR0dNDW1kZLS0vKlZmZjYwDqkwsX74cgNtuu42bb76ZmpoaWlpajrebmZUaB1QZWb58uQPJzMqGA8rMKtqpZlsZarsvgi88B5SZVTQHTXb5LD4zM8skB5SZmWWSA8rMzDLJAWVmZpnkgDIzs0xyQJmZWSY5oMzMLJMcUGZmlkkOKDMzyyQHlFlGSKqW9HNJP0i7lkon6W2LFZ8Dyiw7vgT0pF1EpTsWRlVVVWzatImqqqoT2q14PBefWQZIOg/4JLAEuCnlcipeVVUVfX19APT19VFdXc3Ro0dTrqryeARllg3fBL4CDPl/QUmLJXVL6u7t7S1aYZVow4YNJ1234nBAmaVM0qeAvRGx/WSvi4iVEdEUEU21tbVFqq4yXXnllSddt+JwQJmlbwZwlaQXgHuBj0v6brolVbajR49SXV3N5s2bvXsvRQ4os5RFxK0RcV5E1APXAFsi4jMpl1Wxjt0f6ujRo8yaNet4OPm+UcXnkyTMzAZwGGWDA8osQyJiK7A15TLMMsG7+MzMLJNOGVCSVkvaK+npfm2TJW2U9FzyeFZhyzQzs0qTzwhqDTB3QNstwOaImAZsTtbNzMzGzCkDKiJ+DOwb0DwPWJs8XwvMH9uyzMys0o30GNQ5EbEbIHk8e6gX+up3MzMbiYKfJOGr382s1LS2tjJ+/HgkMX78eFpbW9MuqSKNNKD2SKoDSB73jl1JZmbpaW1tpaOjg6VLl3Lw4EGWLl1KR0eHQyoFIw2odcCi5Pki4KGxKcfMLF133303y5Yt46abbmLChAncdNNNLFu2jLvvvjvt0ipOPqeZdwKPAhdK2iWpGbgTmC3pOWB2sm5mVvIOHz5MS0vLCW0tLS0cPnw4pYoqVz5n8S2MiLqIGJfMF7YqIl6LiCsiYlryOPAsPzOzklRTU0NHR8cJbR0dHdTU1KRUUeXyVEdmZv1cf/31tLW1AbmRU0dHB21tbW8bVVnhOaDMzPpZvnw5ALfddhs333wzNTU1tLS0HG+34nFAmZkNsHz5cgdSBniyWDMzyyQHlJmZZZIDyszMMskBZWZmmeSAMjOzTHJAmZlZJjmgzFIm6XxJXZJ6JO2Q9KW0azLLAl8HZZa+t4CbI+JxSROB7ZI2RsTOtAszS5NHUGYpi4jdEfF48vwNoAc4N92qzNLngDLLEEn1wAeBxwbZ5rtTW0VxQJllhKR3Av8A/HlE/Hbgdt+d2iqNA8osAySNIxdO90TEg2nXY5YFDiizlEkSsAroiYi/Srses6xwQJmlbwbwWeDjkp5Ilk+kXZRZ2nyauVnKImIboLTrMMsaj6DMzCyTHFBmZpZJDigzM8skB5SZmWWSA8rMzDLJAVVGOjs7mT59OtXV1UyfPp3Ozs60SzIrSe5L2eDTzMtEZ2cn7e3trFq1issvv5xt27bR3NwMwMKFC1Ouzqx0uC9lSEQUbbn00kvDCqOxsTG2bNlyQtuWLVuisbExpYrKH9AdRew/4b5UFO5LxTdUX1JuW3E0NTVFd3d30b6vklRXV3Po0CHGjRt3vO3IkSOMHz+evr6+FCsrX5K2R0RTGt/tvlQ47kvFN1RfGtUxKElzJT0r6XlJt4zms2x0Ghoa2LZt2wlt27Zto6GhIaWKzEqT+1J2jDigJFUD3wH+GHg/sFDS+8eqMBue9vZ2mpub6erq4siRI3R1ddHc3Ex7e3vapZmVFPel7BjNSRIfBp6PiF8BSLoXmAf4NtUpOHbwtrW1lZ6eHhoaGliyZIkP6poNk/tSdoz4GJSkq4G5EfH5ZP2zwEci4gsDXrcYWAwwderUS1988cXRVWyWET4GZTY2CnEMarDZl9+WduG7gJqZ2QiMJqB2Aef3Wz8PeGV05ZiZmeWMJqB+BkyT9F5J7wCuAdaNTVlmZlbpRnySRES8JekLwHqgGlgdETvGrDIzM6too5rqKCJ+CPxwjGoxMzM7zpPFmplZJhV1qiNJvYDPMy+8KcCraRdRAd4TEamcmuq+VDTuS8UxaF8qakBZcUjqTuv6HLNy4r6ULu/iMzOzTHJAmZlZJjmgytPKtAswKxPuSynyMSgzM8skj6DMzCyTHFBmZpZJDqgyImm1pL2Snk67FrNS5r6UDQ6o8rIGmJt2EWZlYA3uS6lzQJWRiPgxsC/tOsxKnftSNjigzMwskxxQZmaWSQ4oMzPLJAeUmZllkgOqjEjqBB4FLpS0S1Jz2jWZlSL3pWzwVEdmZpZJHkGZmVkmOaDMzCyTHFBmZpZJDigzM8skB5SZmWWSA8rMzDLJAWVmZpn0/wHMTfHmqAKdIwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAgPElEQVR4nO3de7hdVX3u8e9LUKQKyiVy0lzcIPECVAOJaaxo0aikYgv2cAnnUShSUykWrJeexFqhnpMjHKtYbI3GggTklgMiqaAYQUo9xuAGIgkgxwCpbJNDoiBELakJb/+YY8vKzto7a2futXZW8n6eZz5rrt+8rDEMyc8xxpxjyDYRERE7ao/RLkBERHS3JJKIiKgliSQiImpJIomIiFqSSCIiopYkkoiIqCWJJCIiakkiiWgzSWskvXlnuU/ESEsiiYiIWpJIItpI0hXAJOCfJf1C0l9JmiHpu5J+LukHko4p5/6epJ9Kmli+v7qc84pm9xmtOkUMpEyREtFektYAf2r7W5LGA/cC7wK+AcwErgFeYXuDpPnAa4HjgOXAQtv/MPA+na9FxODSIonorHcCN9u+2fYztpcCvcDbyvHzgRcCdwJrgX8clVJGDEMSSURnvQQ4qXRZ/VzSz4GjgXEAtn8NXAYcAXzK6TKILrDnaBcgYjfQmAweBa6w/Z5mJ5aur/OALwGfkvQa25ua3Cdip5EWSUT7PQYcUva/DPyhpGMljZH0PEnHSJogSVStkUuAM4F1wP8Y5D4RO40kkoj2+wTw0dKNdQpwPPARYANVC+XDVH8XzwEOAv6mdGmdAZwh6fUD7yPpQ52tQsTg8tRWRETUkhZJRETUkkQSERG1JJFEREQtSSQREVHLbvceyYEHHuienp7RLkZERFe56667fmp7bLNju10i6enpobe3d7SLERHRVST922DH0rUVERG1JJFEREQtSSQREVFLEklERNSSRBIREbUkkURERC1JJBERUUsSSURE1JJEEhERtbTtzXZJE4HLgf8CPAMstP33kvYHrgV6gDXAybafKNfMo1oZbgtwju1bSnwq1cpxewM3A+fatqS9ym9MBX4GnGJ7TbvqFNGteubeNOTxNRcc16GSxK6onS2SzcAHbb8SmAGcLekwYC5wq+3JwK3lO+XYbOBwYBbwOUljyr0WAHOAyWWbVeJnAk/YPhS4CLiwjfWJiIgm2pZIbK+zfXfZ3wg8AIynWmZ0UTltEXBC2T8euMb2JtuPAKuB6ZLGAfvaXlaWH718wDX997oOmFnWvY6IiA7pyBiJpB7gSGA5cJDtdVAlG+DF5bTxVOtX9+srsfFlf2B8q2tsbwaeBA5o8vtzJPVK6t2wYcMI1SoiIqADiUTSC4DrgffbfmqoU5vEPER8qGu2DtgLbU+zPW3s2KazIEdExA5qayKR9ByqJHKl7a+U8GOlu4ryub7E+4CJDZdPANaW+IQm8a2ukbQn8ELg8ZGvSUREDKZtiaSMVVwCPGD70w2HlgCnl/3TgRsb4rMl7SXpYKpB9TtL99dGSTPKPU8bcE3/vU4EbivjKBER0SHtXNjqdcC7gJWSVpTYR4ALgMWSzgR+DJwEYPs+SYuB+6me+Drb9pZy3Vk8+/jv18sGVaK6QtJqqpbI7DbWJyIimmhbIrH9HZqPYQDMHOSa+cD8JvFe4Igm8acpiSgiIkZH3myPiIhakkgiIqKWJJKIiKgliSQiImpJIomIiFqSSCIiopYkkoiIqCWJJCIiakkiiYiIWpJIIiKiliSSiIioJYkkIiJqSSKJiIhakkgiIqKWJJKIiKgliSQiImpp51K7l0paL2lVQ+xaSSvKtqZ/5URJPZL+veHY5xuumSpppaTVki4uy+1SluS9tsSXS+ppV10iImJw7WyRXAbMagzYPsX2FNtTgOuBrzQcfqj/mO33NsQXAHOo1nCf3HDPM4EnbB8KXARc2JZaRETEkNqWSGzfQbWO+jZKq+Jk4Oqh7iFpHLCv7WW2DVwOnFAOHw8sKvvXATP7WysREdE5ozVG8nrgMds/aogdLOkeSf8i6fUlNh7oazinr8T6jz0KYHsz8CRwQHuLHRERA+05Sr97Klu3RtYBk2z/TNJU4KuSDgeatTBcPoc6thVJc6i6x5g0adIOFzoiIrbV8RaJpD2BPwau7Y/Z3mT7Z2X/LuAh4GVULZAJDZdPANaW/T5gYsM9X8ggXWm2F9qeZnva2LFjR7ZCERG7udHo2noz8EPbv+mykjRW0piyfwjVoPrDttcBGyXNKOMfpwE3lsuWAKeX/ROB28o4SkREdFA7H/+9GlgGvFxSn6Qzy6HZbDvI/gbgXkk/oBo4f6/t/tbFWcA/AaupWipfL/FLgAMkrQY+AMxtV10iImJwbRsjsX3qIPE/aRK7nupx4Gbn9wJHNIk/DZxUr5QREVFX3myPiIhakkgiIqKWJJKIiKgliSQiImpJIomIiFqSSCIiopYkkoiIqGW05tqKiGHqmXvTkMfXXHBch0oSsbW0SCIiopYkkoiIqCWJJCIiakkiiYiIWpJIIiKiliSSiIioJYkkIiJqSSKJiIhakkgiIqKWdi61e6mk9ZJWNcTOl/QTSSvK9raGY/MkrZb0oKRjG+JTJa0sxy4ua7cjaS9J15b4ckk97apLREQMbruJRNJJkvYp+x+V9BVJR7Vw78uAWU3iF9meUraby30Po1rL/fByzeckjSnnLwDmAJPL1n/PM4EnbB8KXARc2EKZIiJihLXSIvkb2xslHQ0cCyyi+sd9SLbvAB5vsRzHA9fY3mT7EWA1MF3SOGBf28tsG7gcOKHhmkVl/zpgZn9rJSIiOqeVRLKlfB4HLLB9I/DcGr/5Pkn3lq6v/UpsPPBowzl9JTa+7A+Mb3WN7c3Ak8ABzX5Q0hxJvZJ6N2zYUKPoERExUCuJ5CeSvgCcDNwsaa8Wr2tmAfBSYAqwDvhUiTdrSXiI+FDXbBu0F9qeZnva2LFjh1XgiIgYWisJ4WTgFmCW7Z8D+wMf3pEfs/2Y7S22nwG+CEwvh/qAiQ2nTgDWlviEJvGtrpG0J/BCWu9Ki4iIEbLdRGL7V8B64OgS2gz8aEd+rIx59HsH0P9E1xJgdnkS62CqQfU7ba8DNkqaUcY/TgNubLjm9LJ/InBbGUeJiIgO2u7CVpLOA6YBLwe+BDwH+DLwuu1cdzVwDHCgpD7gPOAYSVOouqDWAH8GYPs+SYuB+6kS1dm2+8dmzqJ6Amxv4OtlA7gEuELSaqqWyOwW6hsRESOslRUS3wEcCdwNYHtt/+PAQ7F9apPwJUOcPx+Y3yTeCxzRJP40cNL2yhEREe3VyhjJf5QuIwNIen57ixQREd2klUSyuDy19SJJ7wG+RTVQHhERsf2uLdt/J+ktwFNU4yQfs7207SWLiIiu0MoYCSVxJHlERMQ2Bk0kkjbS/AU/Aba9b9tKFRERXWPQRGJ7u09mRUREtNS1VWb7PZqqhfId2/e0tVQRsVPpmXvToMfWXHBcB0sSO6NWppH/GNUsuwcABwKXSfpouwsWERHdoZUWyanAkeUFQCRdQPVy4v9sZ8EiIqI7tPIeyRrgeQ3f9wIeaktpIiKi67TSItkE3CdpKdUYyVuA70i6GMD2OW0sX0RE7ORaSSQ3lK3f7e0pSkREdKNW3mxftL1zIiJi99XKU1tvl3SPpMclPSVpo6SnOlG4iIjY+bXStfUZ4I+BlVk4KiIiBmrlqa1HgVVJIhER0UwrieSvgJslzZP0gf5texdJulTSekmrGmKflPRDSfdKukHSi0q8R9K/S1pRts83XDNV0kpJqyVdXJbcpSzLe22JL5fUM9zKR0REfa0kkvnAr6jeJdmnYduey4BZA2JLgSNsvwr4f8C8hmMP2Z5Stvc2xBcAc6jWcZ/ccM8zgSdsHwpcBFzYQpkiImKEtTJGsr/ttw73xrbvGNhKsP3Nhq/fA04c6h6SxgH72l5Wvl8OnEC1bvvxwPnl1OuAf5CkdMFFRHRWKy2Sb0kadiJpwbupEkK/g8vTYf8i6fUlNh7oazinr8T6jz0KYHsz8CTVfGDbkDRHUq+k3g0bNoxkHSIidnutJJKzgW+UMYwRefxX0l8Dm4ErS2gdMMn2kcAHgKsk7Uu19slA/S2OoY5tHbQX2p5me9rYsWPrFD0iIgZo5YXEEV2XRNLpwNuBmf3dULY3UU3Fgu27JD0EvIyqBTKh4fIJwNqy3wdMBPok7Qm8EHh8JMsaERHb1+p6JPtRDXT/ZvJG23cM98ckzQL+O/D7tn/VEB8LPG57i6RDym89bPvx0gKaASwHTgM+Wy5bApwOLKMaa7kt4yMREZ233UQi6U+Bc6laAyuAGVT/eL9pO9ddDRwDHCipDziP6imtvYCl5Sne75UntN4AfFzSZmAL8F7b/a2Ls6ieANubakylf1zlEuAKSaupWiKzW6lwRESMrFZaJOcCr6H6R/+Nkl4B/O32LrJ9apPwJYOcez1w/SDHeoEjmsSfBk7aXjkiIqK9Whlsf7phUau9bP8QeHl7ixUREd2ilRZJX3kD/atUXVJP8OyAd0RE7OZaeWrrHWX3fEnfpno66httLVVERHSNVqaRf6mkvfq/Aj3Ab7WzUBER0T1aGSO5Htgi6VCqwfKDgavaWqqIiOgarSSSZ8oUJO8APmP7L4Fx7S1WRER0i1YSya8lnUr18t/XSuw57StSRER0k1YSyRnAa4H5th+RdDDw5fYWKyIiukUrT23dD5zT8P0R4IJ2FioiIrpHKy2SiIiIQSWRRERELYMmEklXlM9zO1eciIjoNkO1SKZKegnwbkn7Sdq/cetUASMiYuc21GD756mmQjkEuIutVyR0iUdExG5u0BaJ7YttvxK41PYhtg9u2JJEIiICaO3x37MkvRp4fQndYfve9hYrIiK6RSuTNp4DXAm8uGxXSvqLdhcsIiK6QyuP//4p8Lu2P2b7Y1RL7b5nexdJulTSekmrGmL7S1oq6Uflc7+GY/MkrZb0oKRjG+JTJa0sxy5WWaNX0l6Sri3x5ZJ6hlHviIgYIa0kElGto95vC1sPvA/mMmDWgNhc4Fbbk4Fby3ckHUa15vrh5ZrPSRpTrlkAzAEml63/nmcCT9g+FLgIuLCFMkVExAhrJZF8CVgu6XxJ5wPfY5C11xvZvgN4fED4eGBR2V8EnNAQv8b2pjIFy2pguqRxwL62l9k2cPmAa/rvdR0ws7+1EhERndPKYPunJd0OHE3VEjnD9j07+HsH2V5X7rtO0otLfDxVgurXV2K/LvsD4/3XPFrutVnSk8ABwE8H/qikOVStGiZNmrSDRY/YufXMvWm0ixC7qVbWbMf23cDdbSxHs5aEh4gPdc22QXshsBBg2rRpTc+JiIgd0+m5th4r3VWUz/Ul3gdMbDhvArC2xCc0iW91jaQ9qdaSH9iVFhERbdbpRLKEaoEsyueNDfHZ5Umsg6kG1e8s3WAbJc0o4x+nDbim/14nAreVcZSIiOigIbu2ypNTt9h+83BvLOlq4BjgQEl9wHlU65gslnQm8GPgJADb90laDNwPbAbOtt3/pNhZVE+A7Q18vWxQDfhfIWk1VUtk9nDLGBER9Q2ZSGxvkfQrSS+0/eRwbmz71EEOzRzk/PnA/CbxXuCIJvGnKYkoIiJGTyuD7U8DKyUtBX7ZH7R9zuCXRETE7qKVRHJT2SJiF5VHh6OOVt4jWSRpb2CS7Qc7UKaIiOgirUza+IfACqq1SZA0RdKSNpcrIiK6RCuP/54PTAd+DmB7BXBw20oUERFdpZVEsrnJE1t5XyMiIoDWBttXSfpvwBhJk4FzgO+2t1gREdEtWmmR/AXV9O6bgKuBp4D3t7FMERHRRVp5autXwF9LurD66o3tL1ZERHSLVp7aeo2klcC9VC8m/kDS1PYXLSIiukErYySXAH9u+18BJB1NtdjVq9pZsIiI6A6tjJFs7E8iALa/A6R7KyIigCFaJJKOKrt3SvoC1UC7gVOA29tftIiI6AZDdW19asD38xr28x5JREQAQyQS22/sZEEiIqI7bXewXdKLqFYm7Gk8P9PIR0QEtDbYfjNVElkJ3NWw7RBJL5e0omF7StL7JZ0v6ScN8bc1XDNP0mpJD0o6tiE+VdLKcuzishxvRER0UCuP/z7P9gdG6gfLVPRT4DdL+f4EuAE4A7jI9t81ni/pMKpldA8Hfhv4lqSXlaV4FwBzgO9RJbxZPLsUb0REdEArLZIrJL1H0jhJ+/dvI/T7M4GHbP/bEOccD1xje5PtR4DVwHRJ44B9bS+zbeBy4IQRKldERLSolUTyH8AngWU8263VO0K/P5vqseJ+75N0r6RLJe1XYuOBRxvO6Sux8WV/YHwbkuZI6pXUu2HDhhEqekREQGuJ5APAobZ7bB9ctkPq/rCk5wJ/BPyfEloAvJSq22sdzz5+3Gzcw0PEtw3aC21Psz1t7NixdYodEREDtJJI7gN+1Ybf/gPgbtuPAdh+zPYW288AX6RaTAuqlsbEhusmAGtLfEKTeEREdFArg+1bgBWSvk01lTwwIo//nkpDt5akcbbXla/vAFaV/SXAVZI+TTXYPhm40/YWSRslzQCWUz2i/NmaZYqIiGFqJZF8tWwjRtJvAW8B/qwh/L8lTaHqnlrTf8z2fZIWA/cDm4GzyxNbAGcBlwF7Uz2tlSe2IiI6rJX1SBaN9I+WNU4OGBB71xDnzwfmN4n3AkeMdPkiIqJ1rbzZ/ghNBrFHYsA9IiK6XytdW9Ma9p8HnASM1HskERHR5bb71JbtnzVsP7H9GeBN7S9aRER0g1a6to5q+LoHVQtln7aVKCIiukorXVuN65Jspnqi6uS2lCYiIrpOK09tZV2SiIgYVCtdW3sB/5Vt1yP5ePuKFRER3aKVrq0bgSepJmvctJ1zIyJiN9NKIplge1bbSxIREV2plUkbvyvpd9pekoiI6EqttEiOBv6kvOG+iWr6dtt+VVtLFhERXaGVRPIHbS9FRER0rVYe/x1qGdyIGCE9c28a7SJE7JBWxkgiIiIGlUQSERG1JJFEREQtSSQREVHLqCQSSWskrZS0QlJvie0vaamkH5XP/RrOnydptaQHJR3bEJ9a7rNa0sWSNBr1iYjYnY1mi+SNtqfY7l84ay5wq+3JwK3lO5IOA2YDhwOzgM9JGlOuWQDMASaXLW/gR0R02M7UtXU80L8+/CLghIb4NbY32X4EWA1MlzQO2Nf2MtsGLm+4JiIiOqSVFxLbwcA3JRn4gu2FwEG21wHYXifpxeXc8cD3Gq7tK7Ffl/2B8W1ImkPVcmHSpEkjWY+I2I6h3o9Zc8FxHSxJtMtoJZLX2V5bksVSST8c4txm4x4eIr5tsEpUCwGmTZvW9JyIiNgxo9K1ZXtt+VwP3ABMBx4r3VWUz/Xl9D5gYsPlE4C1JT6hSTwiIjqo4y0SSc8H9rC9sey/Ffg4sAQ4HbigfN5YLlkCXCXp08BvUw2q32l7i6SNkmYAy4HTgM92tjYRW9veNCfpyold0Wh0bR0E3FCe1N0TuMr2NyR9H1gs6Uzgx8BJALbvk7QYuJ9qzfizbW8p9zoLuAzYG/h62SIiooM6nkhsPwy8ukn8Z8DMQa6ZD8xvEu8FjhjpMkZE6zLZZOxMj/9GREQXSiKJiIhakkgiIqKW0XqPJGK3lPGE2BWlRRIREbUkkURERC1JJBERUUsSSURE1JJEEhERtSSRRERELUkkERFRSxJJRETUkkQSERG1JJFEREQtSSQREVFLEklERNTS8UQiaaKkb0t6QNJ9ks4t8fMl/UTSirK9reGaeZJWS3pQ0rEN8amSVpZjF6ssuxgREZ0zGrP/bgY+aPtuSfsAd0laWo5dZPvvGk+WdBgwGzicas32b0l6WVludwEwB/gecDMwiyy3GxHRUR1vkdheZ/vusr8ReAAYP8QlxwPX2N5k+xFgNTBd0jhgX9vLbBu4HDihvaWPiIiBRnWMRFIPcCSwvITeJ+leSZdK2q/ExgOPNlzWV2Ljy/7AeLPfmSOpV1Lvhg0bRrIKERG7vVFLJJJeAFwPvN/2U1TdVC8FpgDrgE/1n9rkcg8R3zZoL7Q9zfa0sWPH1i16REQ0GJVEIuk5VEnkSttfAbD9mO0ttp8BvghML6f3ARMbLp8ArC3xCU3iERHRQaPx1JaAS4AHbH+6IT6u4bR3AKvK/hJgtqS9JB0MTAbutL0O2ChpRrnnacCNHalERET8xmg8tfU64F3ASkkrSuwjwKmSplB1T60B/gzA9n2SFgP3Uz3xdXZ5YgvgLOAyYG+qp7XyxFZERId1PJHY/g7NxzduHuKa+cD8JvFe4IiRK11ERAxX3myPiIhakkgiIqKWJJKIiKgliSQiImpJIomIiFqSSCIiopYkkoiIqCWJJCIiahmNN9sjIgDomXvTkMfXXHBch0oSdSSRRAzT9v7xi9jdJJFEDJBEsfNIi6U7ZIwkIiJqSSKJiIhakkgiIqKWJJKIiKgliSQiImpJIomIiFq6PpFImiXpQUmrJc0d7fJEROxuuvo9EkljgH8E3gL0Ad+XtMT2/aNbstiZ5T2RXcdQf5Z5x6RzujqRANOB1bYfBpB0DXA8kESym0uyiLzM2DndnkjGA482fO8DfnfgSZLmAHPK119IerCFex8I/LR2CXceu1J9dqW6wK5Vn66piy5s6bSuqU+L6tTnJYMd6PZEoiYxbxOwFwILh3Vjqdf2tB0t2M5mV6rPrlQX2LXqsyvVBVKfVnX7YHsfMLHh+wRg7SiVJSJit9TtieT7wGRJB0t6LjAbWDLKZYqI2K10ddeW7c2S3gfcAowBLrV93wjdflhdYV1gV6rPrlQX2LXqsyvVBVKflsjeZkghIiKiZd3etRUREaMsiSQiImpJImmim6ddkXSppPWSVjXE9pe0VNKPyud+o1nG4ZA0UdK3JT0g6T5J55Z419VJ0vMk3SnpB6Uuf1viXVeXRpLGSLpH0tfK966tj6Q1klZKWiGpt8S6sj6SXiTpOkk/LH9/XtuuuiSRDNAw7cofAIcBp0o6bHRLNSyXAbMGxOYCt9qeDNxavneLzcAHbb8SmAGcXf48urFOm4A32X41MAWYJWkG3VmXRucCDzR87/b6vNH2lIb3Lbq1Pn8PfMP2K4BXU/0ZtacutrM1bMBrgVsavs8D5o12uYZZhx5gVcP3B4FxZX8c8OBol7FG3W6kmlutq+sE/BZwN9VMDF1bF6p3t24F3gR8rcS6uT5rgAMHxLquPsC+wCOUB6raXZe0SLbVbNqV8aNUlpFykO11AOXzxaNcnh0iqQc4ElhOl9apdAOtANYDS213bV2KzwB/BTzTEOvm+hj4pqS7ytRK0J31OQTYAHypdDv+k6Tn06a6JJFsq6VpV6KzJL0AuB54v+2nRrs8O8r2FttTqP6f/HRJR4xykXaYpLcD623fNdplGUGvs30UVdf22ZLeMNoF2kF7AkcBC2wfCfySNnbJJZFsa1ecduUxSeMAyuf6US7PsEh6DlUSudL2V0q4q+tk++fA7VTjWd1al9cBfyRpDXAN8CZJX6Z764PtteVzPXAD1Qzj3VifPqCvtHgBrqNKLG2pSxLJtnbFaVeWAKeX/dOpxhm6giQBlwAP2P50w6Guq5OksZJeVPb3Bt4M/JAurAuA7Xm2J9juofp7cpvtd9Kl9ZH0fEn79O8DbwVW0YX1sf3/gUclvbyEZlItr9GWuuTN9iYkvY2q77d/2pX5o1ui1km6GjiGarrox4DzgK8Ci4FJwI+Bk2w/PkpFHBZJRwP/Cqzk2X74j1CNk3RVnSS9ClhE9d/VHsBi2x+XdABdVpeBJB0DfMj227u1PpIOoWqFQNU1dJXt+V1cnynAPwHPBR4GzqD8d8cI1yWJJCIiaknXVkRE1JJEEhERtSSRRERELUkkERFRSxJJRETUkkQSuzRJv2jDPaeUR8T7v58v6UM17ndSmZ312yNTwh0uxxpJB45mGaI7JZFEDN8U4G3bO2kYzgT+3PYbR/CeER2TRBK7DUkflvR9Sfc2rAXSU1oDXyxrhHyzvHWOpNeUc5dJ+qSkVWW2g48Dp5Q1K04ptz9M0u2SHpZ0ziC/f2pZ62KVpAtL7GPA0cDnJX1ywPnjJN1RfmeVpNeX+AJJvWpY06TE10j6X6W8vZKOknSLpIckvbecc0y55w2S7pf0eUnb/Dsg6Z2q1k5ZIekLZbLJMZIuK2VZKekva/6RxK5itKc7zpatnRvwi/L5VmAh1aScewBfA95ANeX+ZmBKOW8x8M6yvwr4vbJ/AWVqfuBPgH9o+I3zge8Ce1HNKPAz4DkDyvHbVG8Sj6V6a/o24IRy7HZgWpOyfxD467I/Btin7O/fELsdeFX5vgY4q+xfBNwL7FN+c32JHwM8TTU77BhgKXBiw/UHAq8E/rm/DsDngNOAqVQzFveX70Wj/eebbefY0iKJ3cVby3YP1TogrwAml2OP2F5R9u8CesqcWPvY/m6JX7Wd+99ke5Ptn1JNhHfQgOOvAW63vcH2ZuBKqkQ2lO8DZ0g6H/gd2xtL/GRJd5e6HE61AFu//nnhVgLLbW+0vQF4un+eL+BO2w/b3gJcTdUiajSTKml8v0x5P5Mq8TwMHCLps5JmAV07C3OMrD1HuwARHSLgE7a/sFWwWuNkU0NoC7A3zZcTGMrAewz8uzXc+2H7jjKN+XHAFaXr61+BDwGvsf2EpMuA5zUpxzMDyvRMQ5kGzos08LuARbbnDSyTpFcDxwJnAycD7x5uvWLXkxZJ7C5uAd5d1jVB0nhJgy7qY/sJYKOqpXChmt2230aqLqPhWA78vqQDVS3nfCrwL0NdIOklVF1SX6SaAfkoqpXvfgk8KekgqnUzhmt6md16D+AU4DsDjt8KnNj/v4+qdb5fUp7o2sP29cDflPJEpEUSuwfb35T0SmBZNTM9vwDeSdV6GMyZwBcl/ZJqLOLJEv82MLd0+3yixd9fJ2leuVbAzba3N4X3McCHJf26lPc0249Iuge4j6qr6f+28vsDLKMa8/kd4A6enfG2v6z3S/oo1UqBewC/pmqB/DvVinv9/wd0mxZL7J4y+2/EICS9wPYvyv5cqrWuzx3lYtXSON37KBcldiFpkUQM7rjSitgT+Deqp7UiYoC0SCIiopYMtkdERC1JJBERUUsSSURE1JJEEhERtSSRRERELf8JSOBrQXlU7NYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAfV0lEQVR4nO3dfbxVZZ338c83MCQDH9GbePBgkvmQoh6JmazbspTSO3VGDWcKKopyKK2xJqiZsnndFN492JBJ4uiAZipjmkxqSqiZI4GoJE95exKSE4xoIqKOJPi7/1jXudts9tlnHdbZe5/t+b5fr/Xaa//Wvtb6bR7O71zrWutaigjMzMx21+sanYCZmTU3FxIzMyvEhcTMzApxITEzs0JcSMzMrBAXEjMzK8SFxKwTktZJem+Nj9EiKST1T+/vlfSJtP63ku6q5fHNeoILiVkvFRHXRcQpjc7DrCsuJGZmVogLiVl1YyQ9KmmLpBsl7Qkg6XRJyyU9J+kBSUd3NJA0TdLvJG2VtFrSWSXb+kn6tqRnJD0BnNbZgSV9VNL9Je9D0qclPS5ps6QfSFLJ9o9LWpO23Snp4BSXpEslbUrf41FJR/Xwn5P1YS4kZtWdC4wHRgFHAx+VdBxwNfApYH/gCmCBpAGpze+AdwJ7A18HfiRpaNr2SeB04FigFTi7m/mcDpwAHJNyOxVA0pnAl4G/AoYAvwKuT21OAd4FvAXYB/gQ8MduHtesUy4kZtXNiogNEfEs8B/AGLJicEVELImIHRExD9gGjAOIiH9PbV6NiBuBx4GxaX/nAt+LiPVpn9/sZj4zI+K5iHgSuCflA1lR+2ZErImI7cA3yHpTBwOvAIOAtwJKn9m4O38YZpW4kJhV918l6y8BbwQOBi5Kp7Wek/QcMAJ4E4CkiSWnvZ4DjgIOSPt4E7C+ZJ+/74F8SDn9S8kxnwUEDIuIu4HLgB8AT0maI2lwN49r1ikXErPuWw/MiIh9SpY3RMT1qQdwJfAZYP+I2AdYSfZDHWAjWdHpMLIHc/pUWU4DI+IBgIiYFRHHA0eSneL6Yg8d18yFxGw3XAl8WtLb00D2XpJOkzQI2AsI4GkASR8j65F0mA9cIGm4pH2BaT2U0w+B6ZKOTMfdW9I5af2ElOsewIvAy8COHjqumQuJWXdFxDKycZLLgM1AG/DRtG018B1gMfAU8DbgP0uaXwncCfwGeBi4uYdyugW4BLhB0vNkvaD3p82D03E3k51K+yPw7Z44rhlkA2+NzsHMzJqYeyRmZlaIC4mZmRVSs0IiaU9JSyX9RtIqSV9P8f0kLUx35y5MA44dbaZLapP0mKRTS+LHS1qRts3quJtX0oB0t3GbpCWSWmr1fczMrLJa9ki2Ae+JiGPIbpoaL2kc2VUqiyJiNLAovUfSEcAEsssTxwOXS+qX9jUbmAKMTsv4FJ8MbI6IQ4FLyQYbzcysjvrXaseRjeK/kN7ukZYAzgBOSvF5wL3Al1L8hojYBqyV1AaMlbQOGBwRiwEkXQOcCdyR2lyc9nUTcJkkRZUrCA444IBoaWnpia9oZtZnPPTQQ89ExJBK22pWSCCboA54CDgU+EFELJF0UMf0DBGxUdKB6ePDgF+XNG9PsVfSenm8o836tK/tkraQzX30TFkeU8h6NIwcOZJly5b13Jc0M+sDJHU6C0NNB9vTPERjgOFkvYtqM46qQiyqxKu1Kc9jTkS0RkTrkCEVC6qZme2muly1FRHPkZ3CGk82189QgPS6KX2snZ2njhgObEjx4RXiO7VJT5jbm2yOITMzq5NaXrU1RNI+aX0g8F7gt8ACYFL62CTg1rS+AJiQrsQaRTaovjSdBtsqaVy6WmtiWZuOfZ0N3F1tfMTMzHpeLcdIhgLz0jjJ64D5EfEzSYuB+ZImA08C5wBExCpJ84HVwHZgakR0zAd0PjAXGEg2yH5Hil8FXJsG5p8lu+rLzMzqqM9NkdLa2hoebDcz6x5JD0VEa6VtvrPdzMwKcSExM7NCXEjMzKwQFxIzMyukpne2m1nPaZl2W9Xt62aeVqdMzHbmHomZmRXiQmJmZoW4kJiZWSEuJGZmVogLiZmZFeJCYmZmhbiQmJlZIS4kZmZWiAuJmZkV4kJiZmaFuJCYmVkhLiRmZlaIC4mZmRXiQmJmZoW4kJiZWSEuJGZmVogLiZmZFeJCYmZmhbiQmJlZIS4kZmZWiAuJmZkV4kJiZmaF1KyQSBoh6R5JayStknRhil8s6Q+SlqflAyVtpktqk/SYpFNL4sdLWpG2zZKkFB8g6cYUXyKppVbfx8zMKqtlj2Q7cFFEHA6MA6ZKOiJtuzQixqTldoC0bQJwJDAeuFxSv/T52cAUYHRaxqf4ZGBzRBwKXApcUsPvY2ZmFdSskETExoh4OK1vBdYAw6o0OQO4ISK2RcRaoA0YK2koMDgiFkdEANcAZ5a0mZfWbwJO7uitmJlZfdRljCSdcjoWWJJCn5H0qKSrJe2bYsOA9SXN2lNsWFovj+/UJiK2A1uA/Sscf4qkZZKWPf300z3zpczMDKhDIZH0RuAnwOci4nmy01RvBsYAG4HvdHy0QvOoEq/WZudAxJyIaI2I1iFDhnTvC5iZWVU1LSSS9iArItdFxM0AEfFUROyIiFeBK4Gx6ePtwIiS5sOBDSk+vEJ8pzaS+gN7A8/W5tuYmVkl/Wu14zRWcRWwJiK+WxIfGhEb09uzgJVpfQHwY0nfBd5ENqi+NCJ2SNoqaRzZqbGJwPdL2kwCFgNnA3encRQz64aWabdV3b5u5ml1ysSaUc0KCfAO4CPACknLU+zLwHmSxpCdgloHfAogIlZJmg+sJrvia2pE7EjtzgfmAgOBO9ICWaG6VlIbWU9kQg2/j5mZVVCzQhIR91N5DOP2Km1mADMqxJcBR1WIvwycUyBNMzMryHe2m5lZIS4kZmZWiAuJmZkV4kJiZmaFuJCYmVkhLiRmZlaIC4mZmRXiQmJmZoW4kJiZWSEuJGZmVogLiZmZFeJCYmZmhbiQmJlZIS4kZmZWiAuJmZkV4kJiZmaFuJCYmVkhLiRmZlaIC4mZmRXiQmJmZoV0WUgknSNpUFr/R0k3Szqu9qmZmVkzyNMj+aeI2CrpROBUYB4wu7ZpmZlZs8hTSHak19OA2RFxK/D62qVkZmbNJE8h+YOkK4BzgdslDcjZzszM+oA8BeFc4E5gfEQ8B+wHfLGWSZmZWfPospBExEvAJuDEFNoOPF7LpMzMrHnkuWrra8CXgOkptAfwo1omZWZmzSPPqa2zgA8CLwJExAZgUFeNJI2QdI+kNZJWSbowxfeTtFDS4+l135I20yW1SXpM0qkl8eMlrUjbZklSig+QdGOKL5HU0q1vb2ZmheUpJH+KiAACQNJeOfe9HbgoIg4HxgFTJR0BTAMWRcRoYFF6T9o2ATgSGA9cLqlf2tdsYAowOi3jU3wysDkiDgUuBS7JmZuZmfWQPIVkfrpqax9JnwR+AVzZVaOI2BgRD6f1rcAaYBhwBtm9KKTXM9P6GcANEbEtItYCbcBYSUOBwRGxOBW0a8radOzrJuDkjt6KmZnVR/+uPhAR35b0PuB54DDgqxGxsDsHSaecjgWWAAdFxMa0742SDkwfGwb8uqRZe4q9ktbL4x1t1qd9bZe0BdgfeKbs+FPIejSMHDmyO6mbmVkXuiwkAKlwdKt4dJD0RuAnwOci4vkqHYZKG6JKvFqbnQMRc4A5AK2trbtsNzOz3ddpIZG0lQo/lMl+eEdEDO5q55L2ICsi10XEzSn8lKShqTcylOzSYsh6GiNKmg8HNqT48Arx0jbtkvoDewPPdpWXmZn1nE7HSCJiUEQMrrAMyllEBFwFrImI75ZsWgBMSuuTgFtL4hPSlVijyAbVl6bTYFsljUv7nFjWpmNfZwN3p3EUMzOrk1ynttJsvyeS9VDuj4hHcjR7B/ARYIWk5Sn2ZWAm2QD+ZOBJ4ByAiFglaT6wmuyKr6kR0THP1/nAXGAgcEdaICtU10pqI+uJTMjzfczMrOd0WUgkfZXsh33Hqam5kv49Iv53tXYRcT+VxzAATu6kzQxgRoX4MuCoCvGXU25mZtYgeXok5wHHph/aSJoJPAxULSRmZtY35LmPZB2wZ8n7AcDvapKNmZk1nTw9km3AKkkLycZI3gfcL2kWQERcUMP8zMysl8tTSG5JS4d7a5OKmZk1ozx3ts/r6jNmZtZ35ZlG/nRJj0h6VtLzkrZKer4eyZmZWe+X59TW94C/Alb4Zj+z6lqm3VZ1+7qZp9UpE7P6yXPV1npgpYuImZlVkqdH8g/A7ZJ+SXYFFwBl056YmVkflaeQzABeILuX5PW1TcfMzJpNnkKyX0ScUvNMzMysKeUZI/mFJBcSMzOrKE8hmQr8XNJ/+/JfMzMrl+eGxEH1SMTMzJpT3ueR7Ev2oKn/P3ljRNxXq6TMzKx55HkeySeAC8kecbscGAcsBt5T08zMzKwp5BkjuRA4Afh9RLwbOBZ4uqZZmZlZ08hTSF4ueajVgIj4LXBYbdMyM7NmkWeMpF3SPsBPgYWSNgMbapmUmZk1jzxXbZ2VVi+WdA+wN/DzmmZlZmZNI8808m+WNKDjLdACvKGWSZmZWfPIM0byE2CHpEOBq4BRwI9rmpWZmTWNPIXk1YjYDpwFfC8iPg8MrW1aZmbWLPIUklcknQdMAn6WYnvULiUzM2smeQrJx4C/AGZExFpJo4Af1TYtMzNrFnmu2loNXFDyfi0ws5ZJmZlZ88jTIzEzM+tUzQqJpKslbZK0siR2saQ/SFqelg+UbJsuqU3SY5JOLYkfL2lF2jZLklJ8gKQbU3yJpJZafRczM+tcp4VE0rXp9cLd3PdcYHyF+KURMSYtt6djHAFMAI5MbS6X1C99fjYwhWz24dEl+5wMbI6IQ4FLgUt2M08zMyugWo/keEkHAx+XtK+k/UqXrnacppl/NmceZwA3RMS2NAbTBoyVNBQYHBGLIyKAa4AzS9rMS+s3ASd39FbMzKx+qg22/5BsKpRDgIfI7mrvECm+Oz4jaSKwDLgoIjYDw4Bfl3ymPcVeSevlcdLreoCI2C5pC7A/8Ez5ASVNIevVMHLkyN1M28zMKum0RxIRsyLicODqiDgkIkaVLLtbRGYDbwbGABuB76R4pZ5EVIlXa7NrMGJORLRGROuQIUO6lbCZmVWX5/Lf8yUdA7wzhe6LiEd352AR8VTHuqQr+fMNju3AiJKPDiebYbg9rZfHS9u0S+pPNplk3lNpZmbWQ/JM2ngBcB1wYFquk/TZ3TlYGvPocBbQcUXXAmBCuhJrFNmg+tKI2AhslTQujX9MBG4taTMprZ8N3J3GUczMrI7yPI/kE8DbI+JFAEmXkD1q9/vVGkm6HjgJOEBSO/A14CRJY8hOQa0DPgUQEaskzQdWA9uBqRGxI+3qfLIrwAYCd6QFsgkkr5XURtYTmZDju5iZWQ/LU0gE7Ch5v4PK4xM7iYjzKoSvqvL5GcCMCvFlwFEV4i8D53SVh5mZ1VaeQvJvwBJJt6T3Z1KlIJiZWd+SZ7D9u5LuBU4k64l8LCIeqXViZmbWHPL0SIiIh4GHa5yLmZk1IU/aaGZmhbiQmJlZIVULiaR+kn5Rr2TMzKz5VC0k6V6OlyTtXad8zMysyeQZbH8ZWCFpIfBiRzAiLui8iZmZ9RV5CsltaTEzM9tFnvtI5kkaCIyMiMfqkJOZmTWRPJM2/i9gOdmzSZA0RtKCGudlZmZNIs+prYuBscC9ABGxPM3Qa2ZGy7TqZ77XzTytTplYo+S5j2R7RGwpi3m6djMzA/L1SFZK+hugn6TRwAXAA7VNy8zMmkWeHslngSOBbcD1wPPA52qYk5mZNZE8V229BHwlPdAqImJr7dMyM7NmkeeqrRMkrQAeJbsx8TeSjq99amZm1gzyjJFcBfxdRPwKQNKJZA+7OrqWiZmZWXPIM0aytaOIAETE/YBPb5mZGVClRyLpuLS6VNIVZAPtAXyIdE+JmZlZtVNb3yl7/7WSdd9HYmZmQJVCEhHvrmciZmbWnLocbJe0DzARaCn9vKeRNzMzyHfV1u3Ar4EVwKu1TcfMzJpNnkKyZ0T8fc0zMTOzppTn8t9rJX1S0lBJ+3UsNc/MzMyaQp4eyZ+AbwFf4c9XawVwSK2SMjOz5pGnR/L3wKER0RIRo9LSZRGRdLWkTZJWlsT2k7RQ0uPpdd+SbdMltUl6TNKpJfHjJa1I22ZJUooPkHRjii+R1NKtb25mZj0iTyFZBby0G/ueC4wvi00DFkXEaGBReo+kI4AJZLMMjwcul9QvtZkNTAFGp6Vjn5OBzRFxKHApcMlu5GhmZgXlObW1A1gu6R6yqeSBri//jYj7KvQSzgBOSuvzyO6Q/1KK3xAR24C1ktqAsZLWAYMjYjGApGuAM4E7UpuL075uAi6TpIjwzZJmZnWUp5D8NC094aCI2AgQERslHZjiw8guMe7QnmKvpPXyeEeb9Wlf2yVtAfYHnik/qKQpZL0aRo4c2UNfxczMIN/zSObVIQ9VOnSVeLU2uwYj5gBzAFpbW91jMTPrQXnubF9LhR/QeQbcK3hK0tDUGxkKbErxdmBEyeeGAxtSfHiFeGmbdkn9gb2BZ3cjJzMzKyDPYHsrcEJa3gnMAn60m8dbAExK65OAW0viE9KVWKPIBtWXptNgWyWNS1drTSxr07Gvs4G7PT5iZlZ/eU5t/bEs9D1J9wNfrdZO0vVkA+sHSGonmz14JjBf0mTgSeCcdIxVkuYDq4HtwNSI2JF2dT7ZFWADyQbZ70jxq8hulmwj64lM6Oq7mJlZz8tzauu4krevI+uhDOqqXUSc18mmkzv5/AxgRoX4MuCoCvGXSYXIzMwaJ89VW6XPJdkOrAPOrUk2ZmbWdPKc2vJzSczMrFN5Tm0NAP6aXZ9H8s+1S8vMzJpFnlNbtwJbgIcoubPdzMwM8hWS4RFRPmeWmZkZkO8+kgckva3mmZiZWVPK0yM5EfhousN9G9nUJBERR9c0MzMzawp5Csn7a56FmZk1rTyX//6+HomYmVlzyjNGYmZm1ikXEjMzK8SFxMzMCnEhMTOzQlxIzMysEBcSMzMrxIXEzMwKcSExM7NCXEjMzKyQPFOkmPUpLdNuq7p93czT6pSJWXNwj8TMzApxITEzs0JcSMzMrBAXEjMzK8SFxMzMCnEhMTOzQlxIzMysEBcSMzMrpCGFRNI6SSskLZe0LMX2k7RQ0uPpdd+Sz0+X1CbpMUmnlsSPT/tpkzRLkhrxfczM+rJG9kjeHRFjIqI1vZ8GLIqI0cCi9B5JRwATgCOB8cDlkvqlNrOBKcDotIyvY/5mZkbvOrV1BjAvrc8DziyJ3xAR2yJiLdAGjJU0FBgcEYsjIoBrStqYmVmdNKqQBHCXpIckTUmxgyJiI0B6PTDFhwHrS9q2p9iwtF4eNzOzOmrUpI3viIgNkg4EFkr6bZXPVhr3iCrxXXeQFaspACNHjuxurmZmVkVDeiQRsSG9bgJuAcYCT6XTVaTXTenj7cCIkubDgQ0pPrxCvNLx5kREa0S0DhkypCe/iplZn1f3QiJpL0mDOtaBU4CVwAJgUvrYJODWtL4AmCBpgKRRZIPqS9Ppr62SxqWrtSaWtDEzszppxKmtg4Bb0pW6/YEfR8TPJT0IzJc0GXgSOAcgIlZJmg+sBrYDUyNiR9rX+cBcYCBwR1rMzKyO6l5IIuIJ4JgK8T8CJ3fSZgYwo0J8GXBUT+doZmb5+QmJZtZr+WmVzaE33UdiZmZNyIXEzMwKcSExM7NCXEjMzKwQFxIzMyvEhcTMzApxITEzs0JcSMzMrBAXEjMzK8SFxMzMCnEhMTOzQlxIzMysEBcSMzMrxIXEzMwKcSExM7NCXEjMzKwQFxIzMyvEhcTMzArxo3atKfkRrGa9h3skZmZWiAuJmZkV4kJiZmaFuJCYmVkhHmw3sz6p2gUbvlije9wjMTOzQlxIzMysEBcSMzMrpOkLiaTxkh6T1CZpWqPzMTPra5p6sF1SP+AHwPuAduBBSQsiYnVjMzPw3edmfUVTFxJgLNAWEU8ASLoBOANwITGzmvEvSTtTRDQ6h90m6WxgfER8Ir3/CPD2iPhM2eemAFPS28OAx+qaaHUHAM80Ookqent+0Ptz7O35Qe/PsbfnB6/9HA+OiCGVNjR7j0QVYrtUxoiYA8ypfTrdJ2lZRLQ2Oo/O9Pb8oPfn2Nvzg96fY2/PD/p2js0+2N4OjCh5PxzY0KBczMz6pGYvJA8CoyWNkvR6YAKwoME5mZn1KU19aisitkv6DHAn0A+4OiJWNTit7uqVp9xK9Pb8oPfn2Nvzg96fY2/PD/pwjk092G5mZo3X7Ke2zMyswVxIzMysEBeSBpA0QtI9ktZIWiXpwkbnVImkfpIekfSzRudSiaR9JN0k6bfpz/IvGp1TOUmfT3/HKyVdL2nPXpDT1ZI2SVpZEttP0kJJj6fXfXtZft9Kf8+PSrpF0j6Nyi/ls0uOJdu+ICkkHdCI3FIOFfOT9Nk0pdQqSf+np47nQtIY24GLIuJwYBwwVdIRDc6pkguBNY1Ooop/AX4eEW8FjqGX5SppGHAB0BoRR5FdEDKhsVkBMBcYXxabBiyKiNHAovS+Ueaya34LgaMi4mjg/wLT651UmbnsmiOSRpBN2fRkvRMqM5ey/CS9m2zmj6Mj4kjg2z11MBeSBoiIjRHxcFrfSvYDcFhjs9qZpOHAacC/NjqXSiQNBt4FXAUQEX+KiOcamlRl/YGBkvoDb6AX3OcUEfcBz5aFzwDmpfV5wJn1zKlUpfwi4q6I2J7e/prsnrGG6eTPEOBS4B+ocGN0PXWS3/nAzIjYlj6zqaeO50LSYJJagGOBJQ1Opdz3yP5DvNrgPDpzCPA08G/p9Nu/Stqr0UmViog/kP3W9ySwEdgSEXc1NqtOHRQRGyH7RQc4sMH5VPNx4I5GJ1FO0geBP0TEbxqdSyfeArxT0hJJv5R0Qk/t2IWkgSS9EfgJ8LmIeL7R+XSQdDqwKSIeanQuVfQHjgNmR8SxwIs09nTMLtI4wxnAKOBNwF6SPtzYrJqbpK+QnRq+rtG5lJL0BuArwFcbnUsV/YF9yU6nfxGYL6nSNFPd5kLSIJL2ICsi10XEzY3Op8w7gA9KWgfcALxH0o8am9Iu2oH2iOjoyd1EVlh6k/cCayPi6Yh4BbgZ+MsG59SZpyQNBUivPXbao6dImgScDvxt9L4b4N5M9gvDb9L/m+HAw5L+R0Oz2lk7cHNklpKdbeiRCwJcSBog/RZwFbAmIr7b6HzKRcT0iBgeES1kg8N3R0Sv+k06Iv4LWC/psBQ6md73+IAngXGS3pD+zk+ml10QUGIBMCmtTwJubWAuu5A0HvgS8MGIeKnR+ZSLiBURcWBEtKT/N+3AcenfaW/xU+A9AJLeAryeHpqt2IWkMd4BfITsN/3laflAo5NqQp8FrpP0KDAG+EZj09lZ6i3dBDwMrCD7/9bwaTQkXQ8sBg6T1C5pMjATeJ+kx8muOprZy/K7DBgELEz/X37YqPyq5NhrdJLf1cAh6ZLgG4BJPdWz8xQpZmZWiHskZmZWiAuJmZkV4kJiZmaFuJCYmVkhLiRmZlaIC4m9pkl6oQb7HFN6ubakiyV9ocD+zkmzF9/TMxnudh7rGjljrTUvFxKz7hsD9OR9P5OBv4uId/fgPs3qxoXE+gxJX5T0YHqmxddTrCX1Bq5Mz2i4S9LAtO2E9NnF6XkYKyW9Hvhn4EPpxrgPpd0fIeleSU9IuqCT458naUXazyUp9lXgROCHkr5V9vmhku5Lx1kp6Z0pPlvSspTv10s+v07SN1K+yyQdJ+lOSb+T9On0mZPSPm+RtFrSDyXt8nNA0oclLU3HvkLZs2n6SZqbclkh6fMF/0rstSIivHh5zS7AC+n1FLK7ykX2C9TPyKahbyGbBHBM+tx84MNpfSXwl2l9JrAyrX8UuKzkGBcDDwADyOYu+iOwR1kebyKbMmUI2eR5dwNnpm33kj2zpDz3i4CvpPV+wKC0vl9J7F6y50sArAPOT+uXAo+S3Q0+hGwSToCTgJfJZk/uR/acj7NL2h8AHA78R8d3AC4HJgLHAwtL8tun0X+/XnrH4h6J9RWnpOURsilL3gqMTtvWRsTytP4Q0KLsCXyDIuKBFP9xF/u/LSK2RcQzZBMeHlS2/QTg3sgmcOyYvfZdXezzQeBjki4G3hbZs2sAzpX0cPouRwKlD0VbkF5XAEsiYmtEPA28rD8/VXBpRDwRETuA68l6RKVOJisaD0pant4fAjxBNsXG99PcV71mxmprrP6NTsCsTgR8MyKu2CmYPQ9mW0loBzAwfb47yvdR/n+r29N1R8R9kt5F9oCxa9Opr18BXwBOiIjNkuYCpY/v7cjj1bKcXi3JqXxepPL3AuZFxC5PIZR0DHAqMBU4l+zZINbHuUdifcWdwMfTM2CQNExSpw9viojNwFZJ41Ko9BG5W8lOGXXHEuB/SjpAUj/gPOCX1RpIOpjslNSVZLNFHwcMJnv2yhZJBwHv72YeAGMljUpjIx8C7i/bvgg4u+PPR9nz3A9OV3S9LiJ+AvwTvW/afmsQ90isT4iIuyQdDizOZnTnBeDDZL2HzkwGrpT0ItlYxJYUvweYlk77fDPn8TdKmp7aCrg9Irqaqv0k4IuSXkn5ToyItZIeAVaRnWr6zzzHL7OYbMznbcB9wC1lua6W9I/AXanYvELWA/lvsidSdvwC2ujnplsv4dl/zToh6Y0R8UJanwYMjYgLG5xWIZJOAr4QEac3OBV7DXGPxKxzp6VeRH/g92RXa5lZGfdIzMysEA+2m5lZIS4kZmZWiAuJmZkV4kJiZmaFuJCYmVkh/w9GVXAyaJPoDQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 길이 분포 출력\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "text_len = [len(s.split()) for s in data['text']]\n",
    "headlines_len = [len(s.split()) for s in data['headlines']]\n",
    "\n",
    "print('텍스트의 최소 길이 : {}'.format(np.min(text_len)))  # 1\n",
    "print('텍스트의 최대 길이 : {}'.format(np.max(text_len)))  # 60\n",
    "print('텍스트의 평균 길이 : {}'.format(np.mean(text_len)))  # 35.09968483123221\n",
    "print('요약의 최소 길이 : {}'.format(np.min(headlines_len)))  # 1\n",
    "print('요약의 최대 길이 : {}'.format(np.max(headlines_len)))  # 16\n",
    "print('요약의 평균 길이 : {}'.format(np.mean(headlines_len)))  # 9.299532330215534\n",
    "\n",
    "plt.subplot(1,2,1)\n",
    "plt.boxplot(text_len)\n",
    "plt.title('text')\n",
    "plt.subplot(1,2,2)\n",
    "plt.boxplot(headlines_len)\n",
    "plt.title('headlines')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "plt.title('text')\n",
    "plt.hist(text_len, bins = 40)\n",
    "plt.xlabel('length of samples')\n",
    "plt.ylabel('number of samples')\n",
    "plt.show()\n",
    "\n",
    "plt.title('headlines')\n",
    "plt.hist(headlines_len, bins = 40)\n",
    "plt.xlabel('length of samples')\n",
    "plt.ylabel('number of samples')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-text: 히스토그램을 보면 대체적으로 40 내외의 길이를 가짐\n",
    "\n",
    "-headlines: 히스토그램을 보면 대체적으로 11 이하의 길이를 가짐\n",
    "\n",
    "|column|min|max|int(mean)|\n",
    "|:---:|:---:|:---:|:---:|\n",
    "|text|1|60|35|\n",
    "|headlines|1|16|9|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "text_max_len = 40  # text 최대 길이 설정\n",
    "headlines_max_len = 11  # headlines 최대 길이 설정\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 설정한 길이가 몇 %의 샘플까지 포함하는지 확인"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "def below_threshold_len(max_len, nested_list):\n",
    "  cnt = 0\n",
    "  for s in nested_list:\n",
    "    if(len(s.split()) <= max_len):\n",
    "        cnt = cnt + 1\n",
    "  print('전체 샘플 중 길이가 %s 이하인 샘플의 비율: %s'%(max_len, (cnt / len(nested_list))))\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플 중 길이가 40 이하인 샘플의 비율: 0.9238714924766165\n",
      "전체 샘플 중 길이가 11 이하인 샘플의 비율: 0.9449877999186661\n"
     ]
    }
   ],
   "source": [
    "below_threshold_len(text_max_len, data['text'])  # 0.9750305002033347\n",
    "below_threshold_len(headlines_max_len,  data['headlines'])  # 0.9880337535583571"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 정해진 길이보다 길면 제외"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플수 : 85843\n"
     ]
    }
   ],
   "source": [
    "data = data[data['text'].apply(lambda x: len(x.split()) <= text_max_len)]\n",
    "data = data[data['headlines'].apply(lambda x: len(x.split()) <= headlines_max_len)]\n",
    "print('전체 샘플수 :', (len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 시작 토큰(sostoken), 종료 토큰(eostoken) 추가\n",
    "    - 디코더는 시작 토큰 입력받아서 문장 생성함, 종료 토큰 예측한 순간 멈춤\n",
    "    - SOS(start of a sequence): 시작 토큰\n",
    "        - sos 토큰이 맨 앞에 있는 문장: decoder_input\n",
    "    - EOS(end of a sequence): 종료 토큰\n",
    "        - eos 토큰이 맨 뒤에 있는 문장: decoder_target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "      <th>decoder_input</th>\n",
       "      <th>decoder_target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>upgrad learner switches to career in ml al wit...</td>\n",
       "      <td>saurav kant alumnus upgrad iiit pg program mac...</td>\n",
       "      <td>sostoken upgrad learner switches to career in ...</td>\n",
       "      <td>upgrad learner switches to career in ml al wit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>new zealand end rohit sharma led india match w...</td>\n",
       "      <td>new zealand defeated india wickets fourth odi ...</td>\n",
       "      <td>sostoken new zealand end rohit sharma led indi...</td>\n",
       "      <td>new zealand end rohit sharma led india match w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aegon life iterm insurance plan helps customer...</td>\n",
       "      <td>aegon life iterm insurance plan customers enjo...</td>\n",
       "      <td>sostoken aegon life iterm insurance plan helps...</td>\n",
       "      <td>aegon life iterm insurance plan helps customer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>rahat fateh ali khan denies getting notice for...</td>\n",
       "      <td>pakistani singer rahat fateh ali khan denied r...</td>\n",
       "      <td>sostoken rahat fateh ali khan denies getting n...</td>\n",
       "      <td>rahat fateh ali khan denies getting notice for...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>govt directs alok verma to join work day befor...</td>\n",
       "      <td>weeks ex cbi director alok verma told departme...</td>\n",
       "      <td>sostoken govt directs alok verma to join work ...</td>\n",
       "      <td>govt directs alok verma to join work day befor...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           headlines  \\\n",
       "0  upgrad learner switches to career in ml al wit...   \n",
       "2  new zealand end rohit sharma led india match w...   \n",
       "3  aegon life iterm insurance plan helps customer...   \n",
       "5  rahat fateh ali khan denies getting notice for...   \n",
       "7  govt directs alok verma to join work day befor...   \n",
       "\n",
       "                                                text  \\\n",
       "0  saurav kant alumnus upgrad iiit pg program mac...   \n",
       "2  new zealand defeated india wickets fourth odi ...   \n",
       "3  aegon life iterm insurance plan customers enjo...   \n",
       "5  pakistani singer rahat fateh ali khan denied r...   \n",
       "7  weeks ex cbi director alok verma told departme...   \n",
       "\n",
       "                                       decoder_input  \\\n",
       "0  sostoken upgrad learner switches to career in ...   \n",
       "2  sostoken new zealand end rohit sharma led indi...   \n",
       "3  sostoken aegon life iterm insurance plan helps...   \n",
       "5  sostoken rahat fateh ali khan denies getting n...   \n",
       "7  sostoken govt directs alok verma to join work ...   \n",
       "\n",
       "                                      decoder_target  \n",
       "0  upgrad learner switches to career in ml al wit...  \n",
       "2  new zealand end rohit sharma led india match w...  \n",
       "3  aegon life iterm insurance plan helps customer...  \n",
       "5  rahat fateh ali khan denies getting notice for...  \n",
       "7  govt directs alok verma to join work day befor...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 헤드라인 데이터에는 시작 토큰과 종료 토큰을 추가한다.\n",
    "data['decoder_input'] = data['headlines'].apply(lambda x : 'sostoken '+ x)\n",
    "data['decoder_target'] = data['headlines'].apply(lambda x : x + ' eostoken')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 인코더의 입력, 디코더의 입력, 레이블 모두 numpy 로 저장"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "encoder_input = np.array(data['text']) # 인코더의 입력\n",
    "decoder_input = np.array(data['decoder_input']) # 디코더의 입력\n",
    "decoder_target = np.array(data['decoder_target']) # 디코더의 레이블\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 훈련 데이터, 테스트 데이터 분리\n",
    "    - 방법1: 분리 패키지 사용\n",
    "    - 방법2: 직접 코딩으로 분리\n",
    "\n",
    "방법2를 사용하겠다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "테스트 데이터의 수 : 17168\n"
     ]
    }
   ],
   "source": [
    "# encoder_input과 크기와 형태가 같은 순서가 섞인 정수 시퀀스 생성\n",
    "indices = np.arange(encoder_input.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "\n",
    "# 정수 시퀀스로 데이터의 샘플 순서를 정의 => 샘플이 섞이게 된다.\n",
    "encoder_input = encoder_input[indices]\n",
    "decoder_input = decoder_input[indices]\n",
    "decoder_target = decoder_target[indices]\n",
    "\n",
    "# 섞인 데이터를 8:2 비율로 train, test 분리할 예정\n",
    "n_of_val = int(len(encoder_input)*0.2)\n",
    "print('테스트 데이터의 수 :', n_of_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "훈련 데이터의 개수 : 68675\n",
      "훈련 레이블의 개수 : 68675\n",
      "테스트 데이터의 개수 : 17168\n",
      "테스트 레이블의 개수 : 17168\n"
     ]
    }
   ],
   "source": [
    "# train, test 분리\n",
    "encoder_input_train = encoder_input[:-n_of_val]\n",
    "decoder_input_train = decoder_input[:-n_of_val]\n",
    "decoder_target_train = decoder_target[:-n_of_val]\n",
    "\n",
    "encoder_input_test = encoder_input[-n_of_val:]\n",
    "decoder_input_test = decoder_input[-n_of_val:]\n",
    "decoder_target_test = decoder_target[-n_of_val:]\n",
    "\n",
    "print('훈련 데이터의 개수 :', len(encoder_input_train))\n",
    "print('훈련 레이블의 개수 :', len(decoder_input_train))\n",
    "print('테스트 데이터의 개수 :', len(encoder_input_test))\n",
    "print('테스트 레이블의 개수 :', len(decoder_input_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 단어 집합(vocabulary) 생성 및 정수 인코딩\n",
    "    - 단어 집합(vocabulary): 데이터의 각 단어를 고유한 정수로 맵핑하는 작업\n",
    "    - Keras의 Tokenizer()로 생성할 수 있음\n",
    "        - Tokenizer().fit_on_texts(데이터)  # 입력 데이터로부터 단어 집합 생성(동시에 각 단어에 고유한 정수가 부여됨)\n",
    "    - encoder_input_train  # 원문 데이터\n",
    "- 만들어진 단어 집합 중 빈도수가 낮은 단어 제외하기\n",
    "    - src_tokenizer.word_index  # 현재 생성된 단어 집합\n",
    "    - src_tokenizer.word_counts.items()  # 단어, 각 단어의 등장 빈도수\n",
    "- 설정한 등장 빈도 값(threshold) 보다 작은 단어를 단어 집합과 훈련 데이터에서 제외\n",
    "    - total_cnt - rare_cnt  # 등장 빈도가 적은 단어를 제외한 단어 집합 크기\n",
    "    - Tokenizer(num_words=src_vocab) # 단어 집합의 크기 설정\n",
    "- 정수인코딩 진행\n",
    "    - decoder_input_train, decoder_target_train 에는 더 이상 len(tokenizer.word_index) 을 넘는 숫자는 존재하지 않음!\n",
    "        - decoder_input_train: sostoken 추가된 상태\n",
    "        - decoder_target_train: eostoken 추가된 상태\n",
    "    - => 빈도수가 낮은 단어가 삭제됨\n",
    "    - => 빈도수가 낮은 단어만으로 구성된 샘플은 empty 샘플이 되었을 가능성이 있다. 주로 text 보다는 headlines 에서 나타날 것이다.\n",
    "    - => headlines 에서 길이가 0이 된 샘플 인덱스 찾아야 함!\n",
    "    - => sostoken or eostoken 추가된 상태라서 길이가 1인 상태인 샘플 인덱스를 찾으면 됨!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 'text' 에 대한 처리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "# 'text'\n",
    "src_tokenizer = Tokenizer() # 토크나이저 정의\n",
    "src_tokenizer.fit_on_texts(encoder_input_train) # 입력된 데이터로부터 단어 집합 생성\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어 집합(vocabulary)의 크기 : 64641\n",
      "등장 빈도가 6번 이하인 희귀 단어의 수: 44104\n",
      "단어 집합에서 희귀 단어를 제외시킬 경우의 단어 집합의 크기 20537\n",
      "단어 집합에서 희귀 단어의 비율: 68.2291424947015\n",
      "전체 등장 빈도에서 희귀 단어 등장 빈도 비율: 3.7834991612694067\n"
     ]
    }
   ],
   "source": [
    "# 만들어진 단어 집합 중 희귀 단어(한계값 미만) 등장 빈도 비율 관련 내용 출력\n",
    "\n",
    "threshold = 7\n",
    "total_cnt = len(src_tokenizer.word_index) # 단어의 수\n",
    "rare_cnt = 0 # 등장 빈도수가 threshold보다 작은 단어의 개수를 카운트\n",
    "total_freq = 0 # 훈련 데이터의 전체 단어 빈도수 총 합\n",
    "rare_freq = 0 # 등장 빈도수가 threshold보다 작은 단어의 등장 빈도수의 총 합\n",
    "\n",
    "# 단어와 빈도수의 쌍(pair)을 key와 value로 받는다.\n",
    "for key, value in src_tokenizer.word_counts.items():\n",
    "    total_freq = total_freq + value\n",
    "\n",
    "    # 단어의 등장 빈도수가 threshold보다 작으면\n",
    "    if(value < threshold):\n",
    "        rare_cnt = rare_cnt + 1\n",
    "        rare_freq = rare_freq + value\n",
    "\n",
    "print('단어 집합(vocabulary)의 크기 :', total_cnt)\n",
    "print('등장 빈도가 %s번 이하인 희귀 단어의 수: %s'%(threshold - 1, rare_cnt))\n",
    "print('단어 집합에서 희귀 단어를 제외시킬 경우의 단어 집합의 크기 %s'%(total_cnt - rare_cnt))\n",
    "print(\"단어 집합에서 희귀 단어의 비율:\", (rare_cnt / total_cnt)*100)\n",
    "print(\"전체 등장 빈도에서 희귀 단어 등장 빈도 비율:\", (rare_freq / total_freq)*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "src_vocab = 20000  # 등장 빈도가 적은 단어를 제외한 단어 집합 크기\n",
    "\n",
    "src_tokenizer = Tokenizer(num_words=src_vocab)  # 단어 집합 크기 설정해서 다시 생성\n",
    "src_tokenizer.fit_on_texts(decoder_input_train)\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2547, 5970, 2855, 102, 2611, 916, 3959, 30, 66, 186, 11373, 426, 3959, 416, 6266, 5970, 3959, 2014, 2504, 916, 916, 919, 32, 5970, 4438, 1372, 802], [2202, 4884, 2408, 1152, 2031, 1049, 49, 3077, 4938, 840, 4557, 10827, 1767, 301, 422, 3077, 2725, 2031, 588, 137, 2101, 4383, 2101, 301, 512, 523, 4032, 301, 717, 3136], [671, 298, 587, 389, 1365, 265, 265, 4744, 3415, 39, 61, 214, 35, 595, 808, 8290, 389, 1365, 583, 1104, 187, 783, 2504, 983, 8290, 1365, 583, 187, 858, 983, 2951, 2042, 3147, 4010, 3415]]\n"
     ]
    }
   ],
   "source": [
    "# 텍스트 시퀀스를 정수 시퀀스로 변환\n",
    "encoder_input_train = src_tokenizer.texts_to_sequences(encoder_input_train) \n",
    "encoder_input_test = src_tokenizer.texts_to_sequences(encoder_input_test)\n",
    "\n",
    "# 잘 진행되었는지 샘플 출력\n",
    "print(encoder_input_train[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 'headlines' 에 대한 처리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "# 'headlines'\n",
    "tar_tokenizer = Tokenizer() # 토크나이저 정의\n",
    "tar_tokenizer.fit_on_texts(decoder_input_train) # 입력된 데이터로부터 단어 집합 생성\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어 집합(vocabulary)의 크기 : 28467\n",
      "등장 빈도가 5번 이하인 희귀 단어의 수: 18775\n",
      "단어 집합에서 희귀 단어를 제외시킬 경우의 단어 집합의 크기 9692\n",
      "단어 집합에서 희귀 단어의 비율: 65.95356026276039\n",
      "전체 등장 빈도에서 희귀 단어 등장 빈도 비율: 5.185495106517886\n"
     ]
    }
   ],
   "source": [
    "# 만들어진 단어 집합 중 희귀 단어(한계값 미만) 등장 빈도 비율 관련 내용 출력\n",
    "\n",
    "threshold = 6\n",
    "total_cnt = len(tar_tokenizer.word_index) # 단어의 수\n",
    "rare_cnt = 0 # 등장 빈도수가 threshold보다 작은 단어의 개수를 카운트\n",
    "total_freq = 0 # 훈련 데이터의 전체 단어 빈도수 총 합\n",
    "rare_freq = 0 # 등장 빈도수가 threshold보다 작은 단어의 등장 빈도수의 총 합\n",
    "\n",
    "# 단어와 빈도수의 쌍(pair)을 key와 value로 받는다.\n",
    "for key, value in tar_tokenizer.word_counts.items():\n",
    "    total_freq = total_freq + value\n",
    "\n",
    "    # 단어의 등장 빈도수가 threshold보다 작으면\n",
    "    if(value < threshold):\n",
    "        rare_cnt = rare_cnt + 1\n",
    "        rare_freq = rare_freq + value\n",
    "\n",
    "print('단어 집합(vocabulary)의 크기 :', total_cnt)\n",
    "print('등장 빈도가 %s번 이하인 희귀 단어의 수: %s'%(threshold - 1, rare_cnt))\n",
    "print('단어 집합에서 희귀 단어를 제외시킬 경우의 단어 집합의 크기 %s'%(total_cnt - rare_cnt))\n",
    "print(\"단어 집합에서 희귀 단어의 비율:\", (rare_cnt / total_cnt)*100)\n",
    "print(\"전체 등장 빈도에서 희귀 단어 등장 빈도 비율:\", (rare_freq / total_freq)*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "tar_vocab = 10000  # 등장 빈도가 적은 단어를 제외한 단어 집합 크기\n",
    "\n",
    "tar_tokenizer = Tokenizer(num_words=tar_vocab)  # 단어 집합 크기 설정해서 다시 생성\n",
    "tar_tokenizer.fit_on_texts(decoder_input_train)\n",
    "tar_tokenizer.fit_on_texts(decoder_target_train)\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input\n",
      "input  [[1, 5971, 699, 917, 3960, 5, 152, 31], [1, 108, 3078, 886, 4558, 520, 26, 22, 423, 2203], [1, 390, 57, 42, 266, 3, 266, 4745, 3416, 4, 99], [1, 23, 1453, 119, 5, 2333, 473, 738], [1, 7091, 3, 827, 188, 13, 1319, 3618, 34]]\n",
      "target\n",
      "decoder  [[5971, 699, 917, 3960, 5, 152, 31, 2], [108, 3078, 886, 4558, 520, 26, 22, 423, 2203, 2], [390, 57, 42, 266, 3, 266, 4745, 3416, 4, 99, 2], [23, 1453, 119, 5, 2333, 473, 738, 2], [7091, 3, 827, 188, 13, 1319, 3618, 34, 2]]\n"
     ]
    }
   ],
   "source": [
    "# 텍스트 시퀀스를 정수 시퀀스로 변환\n",
    "decoder_input_train = tar_tokenizer.texts_to_sequences(decoder_input_train) \n",
    "decoder_target_train = tar_tokenizer.texts_to_sequences(decoder_target_train)\n",
    "decoder_input_test = tar_tokenizer.texts_to_sequences(decoder_input_test)\n",
    "decoder_target_test = tar_tokenizer.texts_to_sequences(decoder_target_test)\n",
    "\n",
    "# 잘 변환되었는지 확인\n",
    "print('input')\n",
    "print('input ',decoder_input_train[:5])\n",
    "print('target')\n",
    "print('decoder ',decoder_target_train[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "삭제할 훈련 데이터의 개수 : 0\n",
      "삭제할 테스트 데이터의 개수 : 0\n",
      "훈련 데이터의 개수 : 68675\n",
      "훈련 레이블의 개수 : 68675\n",
      "테스트 데이터의 개수 : 17168\n",
      "테스트 레이블의 개수 : 17168\n"
     ]
    }
   ],
   "source": [
    "# 길이 0인 샘플 인덱스 저장 => token 추가해놔서 실질적으로 길이 1인 샘플이다.\n",
    "drop_train = [index for index, sentence in enumerate(decoder_input_train) if len(sentence) == 1]\n",
    "drop_test = [index for index, sentence in enumerate(decoder_input_test) if len(sentence) == 1]\n",
    "\n",
    "print('삭제할 훈련 데이터의 개수 :', len(drop_train))\n",
    "print('삭제할 테스트 데이터의 개수 :', len(drop_test))\n",
    "\n",
    "# 길이 0인 샘플 인덱스(=token 추가로 인해 실질적으로 길이 1인 샘플 인덱스) 삭제\n",
    "encoder_input_train = [sentence for index, sentence in enumerate(encoder_input_train) if index not in drop_train]\n",
    "decoder_input_train = [sentence for index, sentence in enumerate(decoder_input_train) if index not in drop_train]\n",
    "decoder_target_train = [sentence for index, sentence in enumerate(decoder_target_train) if index not in drop_train]\n",
    "\n",
    "encoder_input_test = [sentence for index, sentence in enumerate(encoder_input_test) if index not in drop_test]\n",
    "decoder_input_test = [sentence for index, sentence in enumerate(decoder_input_test) if index not in drop_test]\n",
    "decoder_target_test = [sentence for index, sentence in enumerate(decoder_target_test) if index not in drop_test]\n",
    "\n",
    "# 샘플 삭제 결과 확인\n",
    "print('훈련 데이터의 개수 :', len(encoder_input_train))\n",
    "print('훈련 레이블의 개수 :', len(decoder_input_train))\n",
    "print('테스트 데이터의 개수 :', len(encoder_input_test))\n",
    "print('테스트 레이블의 개수 :', len(decoder_input_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 패딩하기\n",
    "    - 짧은 데이터는 뒤에 숫자 0을 넣어서 최대 길이에 맞춘다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "encoder_input_train = pad_sequences(encoder_input_train, maxlen=text_max_len, padding='post')\n",
    "encoder_input_test = pad_sequences(encoder_input_test, maxlen=text_max_len, padding='post')\n",
    "decoder_input_train = pad_sequences(decoder_input_train, maxlen=headlines_max_len, padding='post')\n",
    "decoder_target_train = pad_sequences(decoder_target_train, maxlen=headlines_max_len, padding='post')\n",
    "decoder_input_test = pad_sequences(decoder_input_test, maxlen=headlines_max_len, padding='post')\n",
    "decoder_target_test = pad_sequences(decoder_target_test, maxlen=headlines_max_len, padding='post')\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 기본적인 seq2seq 모델 설계\n",
    "    - 기본적인 seq2seq 모델: hidden state, cell state 를 디코더의 초기 state 로 사용\n",
    "    - 인코더\n",
    "        - 함수형 API를 이용해서 인코더 설계\n",
    "        - 무조건 LSTM 의 capacity 값이 크다고 해서 성능이 반드시 올라가는 것은 아니다.\n",
    "        - 총 3개의 층 => 모델 용량 증가\n",
    "        - overfitting 방지: dropout + recurrent_dropout => Variational Dropout\n",
    "        - recurrent_dropout 은 cuDNN 을 사용할 수 없음 => 학습 시간 오래 걸림\n",
    "    - 디코더 설계\n",
    "        - 디코더의 LSTM의 입력을 정의할 때, initial_state 인자값으로 인코더의 hidden state 와 cell state 값을 넣어야 한다.\n",
    "        - 디코더 출력층에서는 tar_vocab 의 수많은 단어 중 하나를 선택하는 다중 클래스 분류 문제를 푼다.\n",
    "        - => Dense(tar_vocab, activation='softmax')\n",
    "        - 디코더의 출력층 설계에 어텐션 메커니즘을 적용하면 성능을 높일 수 있다!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Input, LSTM, Embedding, Dense, Concatenate, TimeDistributed\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "인코더 설계"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer lstm_2 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n"
     ]
    }
   ],
   "source": [
    "# 인코더 설계 시작\n",
    "embedding_dim = 128\n",
    "hidden_size = 256  # LSTM의 capacity == LSTM의 용량 크기(뉴런 개수)\n",
    "\n",
    "# 인코더\n",
    "encoder_inputs = Input(shape=(text_max_len,))\n",
    "\n",
    "# 인코더의 임베딩 층\n",
    "enc_emb = Embedding(src_vocab, embedding_dim)(encoder_inputs)\n",
    "\n",
    "# 인코더의 LSTM 1\n",
    "encoder_lstm1 = LSTM(hidden_size, return_sequences=True, return_state=True ,dropout = 0.4, recurrent_dropout = 0.4)\n",
    "encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)\n",
    "\n",
    "# 인코더의 LSTM 2\n",
    "encoder_lstm2 = LSTM(hidden_size, return_sequences=True, return_state=True, dropout=0.4, recurrent_dropout=0.4)\n",
    "encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)\n",
    "\n",
    "# 인코더의 LSTM 3\n",
    "encoder_lstm3 = LSTM(hidden_size, return_state=True, return_sequences=True, dropout=0.4, recurrent_dropout=0.4)\n",
    "encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "디코더 설계"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm_3 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n"
     ]
    }
   ],
   "source": [
    "# 디코더 설계\n",
    "decoder_inputs = Input(shape=(None,))\n",
    "\n",
    "# 디코더의 임베딩 층\n",
    "dec_emb_layer = Embedding(tar_vocab, embedding_dim)\n",
    "dec_emb = dec_emb_layer(decoder_inputs)\n",
    "\n",
    "# 디코더의 LSTM: initial_state 인자값으로 인코더의 hidden state, cell state 값을 넣어야 함!\n",
    "decoder_lstm = LSTM(hidden_size, return_sequences=True, return_state=True, dropout=0.4, recurrent_dropout=0.2)\n",
    "decoder_outputs, _, _ = decoder_lstm(dec_emb, initial_state=[state_h, state_c])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_1 (InputLayer)           [(None, 40)]         0           []                               \n",
      "                                                                                                  \n",
      " embedding (Embedding)          (None, 40, 128)      2560000     ['input_1[0][0]']                \n",
      "                                                                                                  \n",
      " lstm (LSTM)                    [(None, 40, 256),    394240      ['embedding[0][0]']              \n",
      "                                 (None, 256),                                                     \n",
      "                                 (None, 256)]                                                     \n",
      "                                                                                                  \n",
      " input_2 (InputLayer)           [(None, None)]       0           []                               \n",
      "                                                                                                  \n",
      " lstm_1 (LSTM)                  [(None, 40, 256),    525312      ['lstm[0][0]']                   \n",
      "                                 (None, 256),                                                     \n",
      "                                 (None, 256)]                                                     \n",
      "                                                                                                  \n",
      " embedding_1 (Embedding)        (None, None, 128)    1280000     ['input_2[0][0]']                \n",
      "                                                                                                  \n",
      " lstm_2 (LSTM)                  [(None, 40, 256),    525312      ['lstm_1[0][0]']                 \n",
      "                                 (None, 256),                                                     \n",
      "                                 (None, 256)]                                                     \n",
      "                                                                                                  \n",
      " lstm_3 (LSTM)                  [(None, None, 256),  394240      ['embedding_1[0][0]',            \n",
      "                                 (None, 256),                     'lstm_2[0][1]',                 \n",
      "                                 (None, 256)]                     'lstm_2[0][2]']                 \n",
      "                                                                                                  \n",
      " dense (Dense)                  (None, None, 10000)  2570000     ['lstm_3[0][0]']                 \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 8,249,104\n",
      "Trainable params: 8,249,104\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 디코더의 출력층\n",
    "# tar_vocab 의 여러 단어 중 하나를 선택해야 하는 다중 클래스 분류 문제라서 Dense(tar_vocab, activation='softmax') 사용\n",
    "decoder_softmax_layer = Dense(tar_vocab, activation='softmax')\n",
    "decoder_softmax_outputs = decoder_softmax_layer(decoder_outputs) \n",
    "\n",
    "# 모델 정의\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_softmax_outputs)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3. 어텐션 메커니즘 사용하기 (추상적 요약)\n",
    "일반적인 seq2seq보다는 어텐션 메커니즘을 사용한 seq2seq를 사용하는 것이 더 나은 성능을 얻을 수 있어요. 실습 내용을 참고하여 어텐션 메커니즘을 사용한 seq2seq를 설계해 보세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 어텐션 층을 만들어서 앞서 설계한 디코더 출력층을 수정함\n",
    "    - 인코더, 디코더의 hidden state 들을 어텐션 함수의 입력으로 사용\n",
    "    - 어텐션 함수가 리턴한 값을 predict 할 때 디코더의 hidden state 와 함께 활용함\n",
    "    - 어텐션 함수 설계 == 다른 새로운 신경망 설계"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_1 (InputLayer)           [(None, 40)]         0           []                               \n",
      "                                                                                                  \n",
      " embedding (Embedding)          (None, 40, 128)      2560000     ['input_1[0][0]']                \n",
      "                                                                                                  \n",
      " lstm (LSTM)                    [(None, 40, 256),    394240      ['embedding[0][0]']              \n",
      "                                 (None, 256),                                                     \n",
      "                                 (None, 256)]                                                     \n",
      "                                                                                                  \n",
      " input_2 (InputLayer)           [(None, None)]       0           []                               \n",
      "                                                                                                  \n",
      " lstm_1 (LSTM)                  [(None, 40, 256),    525312      ['lstm[0][0]']                   \n",
      "                                 (None, 256),                                                     \n",
      "                                 (None, 256)]                                                     \n",
      "                                                                                                  \n",
      " embedding_1 (Embedding)        (None, None, 128)    1280000     ['input_2[0][0]']                \n",
      "                                                                                                  \n",
      " lstm_2 (LSTM)                  [(None, 40, 256),    525312      ['lstm_1[0][0]']                 \n",
      "                                 (None, 256),                                                     \n",
      "                                 (None, 256)]                                                     \n",
      "                                                                                                  \n",
      " lstm_3 (LSTM)                  [(None, None, 256),  394240      ['embedding_1[0][0]',            \n",
      "                                 (None, 256),                     'lstm_2[0][1]',                 \n",
      "                                 (None, 256)]                     'lstm_2[0][2]']                 \n",
      "                                                                                                  \n",
      " attention_layer (AdditiveAtten  (None, None, 256)   256         ['lstm_3[0][0]',                 \n",
      " tion)                                                            'lstm_2[0][0]']                 \n",
      "                                                                                                  \n",
      " concat_layer (Concatenate)     (None, None, 512)    0           ['lstm_3[0][0]',                 \n",
      "                                                                  'attention_layer[0][0]']        \n",
      "                                                                                                  \n",
      " dense_1 (Dense)                (None, None, 10000)  5130000     ['concat_layer[0][0]']           \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 10,809,360\n",
      "Trainable params: 10,809,360\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import AdditiveAttention\n",
    "\n",
    "# 어텐션 층(어텐션 함수)\n",
    "attn_layer = AdditiveAttention(name='attention_layer')\n",
    "\n",
    "# 인코더와 디코더의 모든 time step의 hidden state를 어텐션 층에 전달하고 결과를 리턴\n",
    "attn_out = attn_layer([decoder_outputs, encoder_outputs])\n",
    "\n",
    "\n",
    "# 어텐션의 결과와 디코더의 hidden state들을 연결\n",
    "decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])\n",
    "\n",
    "# 디코더의 출력층\n",
    "decoder_softmax_layer = Dense(tar_vocab, activation='softmax')\n",
    "decoder_softmax_outputs = decoder_softmax_layer(decoder_concat_input)\n",
    "\n",
    "# 모델 정의\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_softmax_outputs)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 모델 훈련\n",
    "    - EarlyStopping\n",
    "        - callbacks=[es]: 특정 조건이 충족되면 훈련을 멈추는 역할\n",
    "        - es == EarlyStopping(monitor='val_loss', patience=2, verbose=1)\n",
    "        - => val_loss(검증 데이터 손실) 관찰하다가 값이 작아지지 않고 증가하는 현상이 2번(patience=2) 관측되면 학습 종료\n",
    "        - => epoch 값이 커도 최적점에서 훈련 종료 가능"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "269/269 [==============================] - 241s 870ms/step - loss: 6.0500 - val_loss: 5.6038\n",
      "Epoch 2/50\n",
      "269/269 [==============================] - 231s 858ms/step - loss: 5.4760 - val_loss: 5.2548\n",
      "Epoch 3/50\n",
      "269/269 [==============================] - 234s 871ms/step - loss: 5.1222 - val_loss: 4.9711\n",
      "Epoch 4/50\n",
      "269/269 [==============================] - 232s 862ms/step - loss: 4.8447 - val_loss: 4.7775\n",
      "Epoch 5/50\n",
      "269/269 [==============================] - 236s 876ms/step - loss: 4.6289 - val_loss: 4.6148\n",
      "Epoch 6/50\n",
      "269/269 [==============================] - 240s 893ms/step - loss: 4.4502 - val_loss: 4.4999\n",
      "Epoch 7/50\n",
      "269/269 [==============================] - 242s 899ms/step - loss: 4.2972 - val_loss: 4.4008\n",
      "Epoch 8/50\n",
      "269/269 [==============================] - 240s 891ms/step - loss: 4.1616 - val_loss: 4.3263\n",
      "Epoch 9/50\n",
      "269/269 [==============================] - 240s 891ms/step - loss: 4.0400 - val_loss: 4.2537\n",
      "Epoch 10/50\n",
      "269/269 [==============================] - 240s 891ms/step - loss: 3.9299 - val_loss: 4.2071\n",
      "Epoch 11/50\n",
      "269/269 [==============================] - 236s 876ms/step - loss: 3.8313 - val_loss: 4.1503\n",
      "Epoch 12/50\n",
      "269/269 [==============================] - 232s 862ms/step - loss: 3.7432 - val_loss: 4.1314\n",
      "Epoch 13/50\n",
      "269/269 [==============================] - 234s 869ms/step - loss: 3.6621 - val_loss: 4.0900\n",
      "Epoch 14/50\n",
      "269/269 [==============================] - 236s 878ms/step - loss: 3.5850 - val_loss: 4.0557\n",
      "Epoch 15/50\n",
      "269/269 [==============================] - 238s 885ms/step - loss: 3.5151 - val_loss: 4.0315\n",
      "Epoch 16/50\n",
      "269/269 [==============================] - 238s 886ms/step - loss: 3.4493 - val_loss: 4.0183\n",
      "Epoch 17/50\n",
      "269/269 [==============================] - 236s 878ms/step - loss: 3.3935 - val_loss: 3.9978\n",
      "Epoch 18/50\n",
      "269/269 [==============================] - 237s 881ms/step - loss: 3.3359 - val_loss: 3.9939\n",
      "Epoch 19/50\n",
      "269/269 [==============================] - 237s 880ms/step - loss: 3.2834 - val_loss: 3.9671\n",
      "Epoch 20/50\n",
      "269/269 [==============================] - 238s 886ms/step - loss: 3.2302 - val_loss: 3.9624\n",
      "Epoch 21/50\n",
      "269/269 [==============================] - 242s 901ms/step - loss: 3.1800 - val_loss: 3.9540\n",
      "Epoch 22/50\n",
      "269/269 [==============================] - 235s 875ms/step - loss: 3.1388 - val_loss: 3.9366\n",
      "Epoch 23/50\n",
      "269/269 [==============================] - 241s 896ms/step - loss: 3.0938 - val_loss: 3.9329\n",
      "Epoch 24/50\n",
      "269/269 [==============================] - 239s 890ms/step - loss: 3.0510 - val_loss: 3.9355\n",
      "Epoch 25/50\n",
      "269/269 [==============================] - 237s 881ms/step - loss: 3.0138 - val_loss: 3.9425\n",
      "Epoch 00025: early stopping\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy')\n",
    "es = EarlyStopping(monitor='val_loss', patience=2, verbose=1)\n",
    "history = model.fit(x=[encoder_input_train, decoder_input_train], y=decoder_target_train, \\\n",
    "          validation_data=([encoder_input_test, decoder_input_test], decoder_target_test), \\\n",
    "          batch_size=256, callbacks=[es], epochs=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- train_loss, val_loss 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD4CAYAAADmWv3KAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAuSklEQVR4nO3deXyU1b3H8c8vOwnZNwLZIIR9CRCQRRBQQRb31opFq7cVaW1rV+tar721tb2tda2Kra1e99LiAmgBAYEqS9gDBBLWhABJCNnJOuf+8QwQskACmUxm5vd+vfKa5Xlm5vcw+s3Jec5zjhhjUEop5X68nF2AUkopx9CAV0opN6UBr5RSbkoDXiml3JQGvFJKuSkfZ31wVFSUSU5OdtbHK6WUS9q8eXORMSa6Lfs6LeCTk5PJyMhw1scrpZRLEpHDbd1Xu2iUUspNacArpZSb0oBXSik35bQ+eKWUuhR1dXXk5eVRXV3t7FIcKiAggPj4eHx9fS/5PTTglVIuJS8vj+DgYJKTkxERZ5fjEMYYTp48SV5eHr17977k99EuGqWUS6muriYyMtJtwx1ARIiMjLzsv1I04JVSLsedw/2MjjhGlwv47BPl/OqT3dTW25xdilJKdWltCngRCRORhSKSJSJ7RGRck+0iIs+LSI6I7BCRkY4pF/JOneb1/xxkbXahoz5CKaVaVVJSwp///Od2v27mzJmUlJR0fEEX0NYW/HPAZ8aYAcBwYE+T7TOAVPvPPODlDquwiQl9owjt5sviHccc9RFKKdWq1gK+oaHhgq9bunQpYWFhDqqqZRcNeBEJASYBfwUwxtQaY0qa7HYj8KaxrAfCRCSuo4sF8PPx4rrBPVi++wTVdRf+B1VKqY720EMPsX//ftLS0hg9ejRTpkzhjjvuYOjQoQDcdNNNjBo1isGDB7NgwYKzr0tOTqaoqIhDhw4xcOBA7r33XgYPHsy0adM4ffq0Q2ptyzDJPkAh8DcRGQ5sBh4wxlQ22qcXkNvocZ79ufOa2SIyD6uFT2Ji4iUXPXt4HO9n5PLFvkKmD+5xye+jlHJtT36yi935ZR36noN6hvDE9YNb3f7000+TmZnJtm3bWL16NbNmzSIzM/PscMbXX3+diIgITp8+zejRo7n11luJjIw87z2ys7N59913ee2117jtttv45z//ydy5czv0OKBtXTQ+wEjgZWPMCKASeKjJPi2d7m222KsxZoExJt0Ykx4d3abJ0Fo0rk8kkUF+2k2jlHK6MWPGnDdW/fnnn2f48OGMHTuW3NxcsrOzm72md+/epKWlATBq1CgOHTrkkNra0oLPA/KMMRvsjxfSPODzgIRGj+OB/Msvr2U+3l5cN6QHi7Ye5XRtA938vB31UUqpLuxCLe3OEhQUdPb+6tWrWbFiBV999RWBgYFMnjy5xbHs/v7+Z+97e3s7rIvmoi14Y8xxIFdE+tufuhrY3WS3j4G77KNpxgKlxhiHNq9nDYujqraBlVkFjvwYpZQ6T3BwMOXl5S1uKy0tJTw8nMDAQLKysli/fn0nV3e+tk5V8APgbRHxAw4A94jIfABjzCvAUmAmkANUAfc4oNbzXNE7kuhgfxbvyGfWMIecz1VKqWYiIyOZMGECQ4YMoVu3bsTGxp7ddt111/HKK68wbNgw+vfvz9ixY51YKYgxzbrKO0V6erq53AU/nvgok/c25bL58Wvp7q/T6ijlCfbs2cPAgQOdXUanaOlYRWSzMSa9La93uStZG5s9vCc19TY+33PC2aUopVSX49IBPyoxnB4hAXyyXUfTKKVUUy4d8F5ewqxhcazZV0jp6Tpnl6OUUl2KSwc8wOxhcdQ22FixW7tplFKqMZcP+LSEMHqFdWPxDocNu1dKKZfk8gEvIsweFsfa7CJKqmqdXY5SSnUZLh/wALOH9aTeZvj3ruPOLkUp5eYudbpggGeffZaqqqoOrqh1bhHwQ3qFkBQZqHPTKKUczpUC3i2uDjrTTfPKFwc4WVFDZHf/i79IKaUuQePpgq+99lpiYmL44IMPqKmp4eabb+bJJ5+ksrKS2267jby8PBoaGnj88cc5ceIE+fn5TJkyhaioKFatWuXwWt0i4MHqpnlp1X4+zTzO3LFJzi5HKdUZPn0Iju/s2PfsMRRmPN3q5sbTBS9btoyFCxeyceNGjDHccMMNrFmzhsLCQnr27MmSJUsAa46a0NBQnnnmGVatWkVUVFTH1twKt+iiARjQI5iU6CAdTaOU6jTLli1j2bJljBgxgpEjR5KVlUV2djZDhw5lxYoV/OIXv2Dt2rWEhoY6pT63acGLCLOG9eSFldkUlFUTExLg7JKUUo52gZZ2ZzDG8PDDD3Pfffc127Z582aWLl3Kww8/zLRp0/jlL3/Z6fW5TQse4PphcRgDn2bqaBqllGM0ni54+vTpvP7661RUVABw9OhRCgoKyM/PJzAwkLlz5/Kzn/2MLVu2NHttZ3CbFjxAamww/WODWbwjn2+NT3Z2OUopN9R4uuAZM2Zwxx13MG7cOAC6d+/OW2+9RU5ODj//+c/x8vLC19eXl19+GYB58+YxY8YM4uLiOuUkq0tPF9ySFz7P5o/L9/HVw1OJC+3W4e+vlHIunS7YQ6YLbsns4T0BWKJj4pVSHs7tAr53VBCDe4boRU9KKY/ndgEP1pj4bbkl5BZ33hVjSqnO46yu5c7UEcfopgFvrdG6ZKe24pVyNwEBAZw8edKtQ94Yw8mTJwkIuLzh3q43iqbsGGz6C0x+GLxbLj8hIpDh8aEs3pHP/KtSOrlApZQjxcfHk5eXR2FhobNLcaiAgADi4+Mv6z1cL+DzNsLaP0BYAoy6u9XdZg/ryVNL93CoqJLkqKDOq08p5VC+vr707t3b2WW4BNfrohl4AySMhZVPQU3rFwzM0m4apZSHc72AF4HpT0FlAfzn+VZ36xnWjVFJ4XyyXeemUUp5JtcLeID4dBhyK3z5ApQebXW32cPiyDpeTk5BRScWp5RSXUObAl5EDonIThHZJiLNLj8VkckiUmrfvk1EHD+rztVPgLHByl+3usvMoXGIoDNMKqU8Unta8FOMMWkXuER2rX17mjHmVx1R3AWFJ8HY+bD9XTi2vcVdYkMCGJMcweIdx9x6SJVSSrXENbtozpj4UwiMgH8/Cq0E+OxhceQUVLD3ROfN4KaUUl1BWwPeAMtEZLOIzGtln3Eisl1EPhWRwS3tICLzRCRDRDI6ZAxrQKg1Hv7QWtj3WYu7XDckDi+Bxdt1NI1SyrO0NeAnGGNGAjOA+0VkUpPtW4AkY8xw4AXgw5bexBizwBiTboxJj46OvtSazzfqbohMhWWPQ0Nds83Rwf6MT4li0daj1NQ3dMxnKqWUC2hTwBtj8u23BcAiYEyT7WXGmAr7/aWAr4h0zqKD3r5w7a/gZDZs/nuLu8yb1IejJaf5v68Od0pJSinVFVw04EUkSESCz9wHpgGZTfbpISJivz/G/r4nO77cVvSfAckTYfVvobq02eZJ/aKZmBrFCytzKK1q3spXSil31JYWfCywTkS2AxuBJcaYz0RkvojMt+/zNSDTvs/zwO2mM4etiMC0X0NVMax9psVdHpk5kLLqOl5cld1pZSmllDNddC4aY8wBYHgLz7/S6P6LwIsdW1o79UyD4bfD+pch/b+sYZSNDIwL4daR8bzx5WHuGpdMQkSgc+pUSqlO4trDJJua+jiIF3ze8jD8n07rh5cX/O+/93ZyYUop1fncK+BDe8H470PmQsjb3GxzXGg3vn1lbz7ens+OvJLOr08ppTqRewU8wIQHICgG/v1Iixc/zb8qhcggP36zdI9e3aqUcmvuF/D+wTDlEchdD3s+abY5OMCXB65JZf2BYlZmFTihQKWU6hzuF/AAI+6E6IGw4gmor222ec6YRPpEBfHbT7Oob7A5oUCllHI89wx4bx9r2GTxAWt5vyZ8vb148LoB5BRU8EFGnhMKVEopx3PPgAdIvQZSpsIXv7PGxzcxfXAs6UnhPLN8H5U19U4oUCmlHMt9Ax6sVnxNGaz5Q7NNIsIjswZSVFHDgjUHnFCcUko5lnsHfOxgGDEXNi6Ak/ubbR6ZGM6soXEsWHOAgrJqJxSolFKO494BDzDlUfD2g+UtLzL14HX9qbfZ+NOKfZ1cmFJKOZb7B3xwD5j0M8haDDsXNtucFBnE3LFJvL8pl326KIhSyo24f8ADjP8h9EqHJT+FsuYLf/xwaipB/j48/WmWE4pTSinH8IyA9/aBm1+F+hr4+PvNrnAND/Lj/il9WZlVwJf7i5xUpFJKdSzPCHiAqL7WwiA5K1pcGOTu8cn0CuvGb5buwWbTKQyUUq7PcwIeYPR3oM9ka5Hu4vOHRgb4evPTaf3IPFrGJzvynVOfUkp1IM8KeC8vuPEl8PKBRd8F2/lrtN6U1ovBPUP4/Wd7qa7T9VuVUq7NswIeIDQeZv6vNRnZly+ct8nLS3hk5kCOlpzmza8OOac+pZTqIJ4X8ADDboOB18Oqp+DErvM2TegbxeT+0by4MoeSquYTlSmllKvwzIAXgdnPQkAo/Ou+ZjNOPjxjIBU19fzuM135SSnlujwz4AGCouD65+HETvji6fM29e8RzL0T+/DuxiN8vueEkwpUSqnL47kBDzBgpjVXzbo/Qe7G8zb9ZFo/BsaF8ODCHRSW1zipQKWUunSeHfAA038LIfGwaD7UVp592t/Hm+duT6O8pp6H/rlDl/dTSrkcDfiAELjpz1C8H5Y/cd6mfrHBPHTdAD7PKuDdjblOKlAppS5NmwJeRA6JyE4R2SYiGS1sFxF5XkRyRGSHiIzs+FIdqPdEGHs/bHoNcj4/b9Pd45OZmBrF/yzezYHCCicVqJRS7deeFvwUY0yaMSa9hW0zgFT7zzzg5Y4orlNd/ThE9YePvg+nT5192stL+MPXh+Pv68WP399Gna7hqpRyER3VRXMj8KaxrAfCRCSug967c/h2g1tehcoCWPrgeZtiQwL4zc1D2Z5Xygsrc5xUoFJKtU9bA94Ay0Rks4jMa2F7L6BxJ3We/TnX0nMETHoQdn4Auz48b9PMoXHcOjKeF1dms/nwqZZfr5RSXUhbA36CMWYkVlfM/SIyqcl2aeE1zYadiMg8EckQkYzCwsJ2ltpJJv7ECvrFP4by88fA//cNg+gZ1o0fv7+NCl2oWynVxbUp4I0x+fbbAmARMKbJLnlAQqPH8UCzKRmNMQuMMenGmPTo6OhLq9jRvH3h5gVQVwXv3HZef3xwgC9/+kYaeaeq+J9PdjuxSKWUuriLBryIBIlI8Jn7wDQgs8luHwN32UfTjAVKjTHNl05yFdH94LY3oWA3vHkjVBWf3TQ6OYLvTk7h/YxcPss87sQilVLqwtrSgo8F1onIdmAjsMQY85mIzBeR+fZ9lgIHgBzgNeB7Dqm2M/WbDt94Gwr2NAv5B67ux9BeoTz8rx0UlFU7sUillGqdOOsKzfT0dJOR0WxIfdeTvRze+6bVqr/rYwiMACCnoILZL6zlit6R/P2e0Yi0dBpCKaU6lohsbmW4ejN6JevFpF4Lt78DhfvgzRvOtuT7xnTn0ZkD+WJfIf+3/rCTi1RKqeY04Nsi9RqYYw/5N26AypMAzB2bxOT+0Ty1ZA85BeVOLlIppc6nAd9Wfa+BOe/CyWx443qoLEJE+P3XhhHk78OP3t9Gbb1e5aqU6jo04Nuj79Uw5z1rYrI3boDKImKCA/jtLUPJPFrGsyv2ObtCpZQ6SwO+vVKmwB3vQ/EBqyVfUcj0wT24fXQCL3+xXxcIUUp1GRrwl6LPZHvIHzwb8o/PHsTQXqHc/84WncpAKdUlaMBfqj5XwTc/gFOH4I3ZBNUV8/rdo+kREsC339ikJ12VUk6nAX85ek+Cb/4DSo7A32cTZUp487+uwMfLi7v+upHjpXoRlFLKeTTgL1fviVbIl+bCG7NJ9D7J3+8ZTVl1Pd96fSOlVXXOrlAp5aE04DtC8pXwzYVQfhxem8oQ2z4W3DmKA0UV3PtmBtV1Dc6uUCnlgTTgO0ryBPjOCvALhL/PYnzVKp65LY1Nh4v54btbabDpot1Kqc6lAd+RovvDvasgfjT86ztcX/QXnpg1gGW7T/DYh5k4a94fpZRn0oDvaIERcOciGHkXrP0jdx99ggcm9uTdjUd4dkW2s6tTSnkQH2cX4JZ8/OD65yF6ICx7lB/FHqZy2OM893k20cH+zB2b5OwKlVIeQFvwjiIC474Hd3yAFB/k0fzv8+3exfzyo0xdKEQp1Sk04B0t9Vr4znLEJ4DHCn/G/dHb+eF7W9lw4KSzK1NKuTkN+M4QMxDuXYX0HMlPy37HY4GLuPfNjWQdL3N2ZUopN6YB31mCIuGujyBtLnfVvs+f5Dnm/XUteaeqnF2ZUspNacB3Jh8/uPFFmPZrppr1vFz3GD9+7VMNeaWUQ2jAdzYRGP8DZM57DPA5watVD/Dmi79i3/FSZ1emlHIzGvDO0v86vO9bRUCPgTzS8DK1r0whK2Ols6tSSrkRDXhniu5P4H3LKJr2ErFyigGLb+bYG/dARYGzK1NKuQENeGcTIWr8XPh+Bh/4f43IAx9R9+wI+OolaNCZKJVSl04DvouIjozkuh+/woMxr7KuJgX+/Qi8ciUcWO3s0pRSLqrNAS8i3iKyVUQWt7BtsoiUisg2+88vO7ZMzxAS4MvT827hnb5/5Nu1P6WkrBzevBHev9NaVEQppdqhPS34B4A9F9i+1hiTZv/51WXW5bECfL15ee4oIkfeyBWlT7G8x3cw2cvhxTHwxe+hTleJUkq1TZsCXkTigVnAXxxbjgLw8fbid7cO456rBnLvoak8Hv83GlKnwaqn4KUxkPkvsNmcXaZSqotrawv+WeBB4EKpMk5EtovIpyIyuKUdRGSeiGSISEZhYWE7S/UsIsJDMwbw6MyBvJVl41vl93N6ziLwC4KF91j983s+AZ1jXinViosGvIjMBgqMMZsvsNsWIMkYMxx4AfiwpZ2MMQuMMenGmPTo6OhLqdfj3DupD3/4+nC+OnCS25f7UXznSrjlL9BQA+/PhVcnwd5PNeiVUs20pQU/AbhBRA4B7wFTReStxjsYY8qMMRX2+0sBXxGJ6uhiPdXXRsXz6txRZB0v52sLNpCXMAu+twFuegVqyuDd2+G1qZC9QoNeKXXWRQPeGPOwMSbeGJMM3A6sNMbMbbyPiPQQEbHfH2N/X50PtwNdMyiW//v2FRSW13DDi//hy4MlkDYHvp8BN7wIlUXw9q3w12mwf5UGvVLq0sfBi8h8EZlvf/g1IFNEtgPPA7cbXYC0w43pHcGH908gMsiPuX/dwCtf7Md4+cDIO+EHm2H2n6DsKPzfTfD3WXBonbNLVko5kTgrh9PT001GRoZTPtvVVdbU8+DCHSzZeYwZQ3rwv18fTnd/++qL9TWw5U1Y8weoOA69J8HkRyBpnHOLVkp1CBHZbIxJb9O+GvCuyRjDX9Ye5OnPskiODOTVO9PpG9P93A51pyHjb7DuGagshB5DYcSdMPTr1sLgSimXpAHvQb7cX8QP3tlKdV0Df/j6cGYMjTt/h9pK2PYObH0Ljm0Dbz/oP9MK+5Qp4OXtlLqVUpdGA97DHCs9zfy3trA9t4T7rurDz6f1x8e7hdMrx3fC1rdhx/twuhhCesHwOZB2B0SmdH7hSql204D3QDX1DTz5yW7e2XCE8SmRvDBnBJHd/Vveub7GGju/7W3IWQHGBklXwohvwqAbrYuplFJdkga8B/sgI5fHPswkKsiPl+eOYnhC2IVfUJYP29+1unCKD4BfMAy5GdLmQsIYawUqpVSXoQHv4XbmlTL/rc0Ultfw5I2DmTMm8eIvMgaOfGUF/a5FUFcFESlW983w2yE03vGFK6UuSgNeUVxZywPvbWVtdhHfSE/gyRsHE+DbxhOqNeWw60OrZX/4P4BAn6tg+B0w8HrwC3Rk6UqpC9CAVwA02AzPLN/LS6v20zemO89+I40hvULb9ybFB2H7e7D9HWtOer9gGHwjpH0TEsdpF45SnUwDXp1nzb5Cfr5wOycravnxtf2Yf1UK3l7tDGabDY58aQ253PUh1FVCeLLVqh9+O4QnOaJ0pVQTGvCqmZKqWh5dlMmSnccYlRTOn25LIzHyErtaaiqsqYq3vwMH11jPJU+EAbOtK2Zjh+j4eqUcRANetcgYw0fb8nn8o0xsNsMvrx/EbekJyOV0s5Qcge3vW/31xfut5/xDIOEKK+wTx0OvkeDTypBNpVS7aMCrCzpacpqffbCdrw6c5NpBsfz2lqFEtTZmvj1Kcq2ROIe/tH6K9lrPe/tDfLrVZ5803hp+6R98+Z+nlAfSgFcXZbMZXv/PQX7/2V5Cuvnw9C3DuGZQbMd+SOVJK/DPhP6x7WAaQLygxzBImgApUyH5SvAN6NjPVspNacCrNtt7vJwH3ttK1vFy5oxJ4LFZgwg6MzNlR6upgLyNcNge+nmboL4afAOh91WQei2kToOwBMd8vlJuQANetUtNfQPPLN/HgjUHSIwI5Jnb0hiVFO74D647bc1Zn70M9v0bSg5bz8cMtsK+33SIHwPeDvqFo5QL0oBXl2TDgZP85IPtHCs9zXcnp/CDqaltvzjqchkDRfvOhf2Rr8BWDwGhkHK11bJPvRaCdCVI5dk04NUlK6+u48lPdrNwcx69o4J46qYhjO/rhFCtLoUDq2HfMiv0KwsAgZhBENITgmOhew8I7gHdY62fM89pf75yYxrw6rKtzS7ksQ8zOXyyiltG9OLRWQNbn53S0Ww2OL4dspdDXoa1UlX5CSv0ja35/gGh9vC3B35UqjVUs+dIXexEuTwNeNUhqusaeHFlDq+u2U+Qvw+PzBjI19PjL2/cfEeyNViLjVcch4oCKD9+Lvwb35bkAvb/ziNSoNcoa9hmr1HWSlc6Rl+5EA141aH2nSjn0UU72XToFGN6R/Cbm4fQN8aFxrFXl0L+Nji62fo581cAgJevFfK9Rp0L/ogU8Lrk9eiVcigNeNXhbDbDPzbn8pulWVTV1vPdq1L43pS+nXcStqOVHj0X+Ec3Q/5WqK2wtvmHQuxgiB1k9fnHDrZuA0KcW7NSaMArByqqqOGpJXtYtPUoyZGB/PqmoVyZ6gYjW2wN1iievAzI3wIndsGJ3VBbfm6f0MTmoR+VCt6+zqtbeRwNeOVw67KLeOzDnRw6WcXN9pOwHTLdQVdiDJTmWkFfYA/8gt3WLwJbvbWPly9E9YOYARCWZF2kFZZo3Q+NB99uzj0G5XY04FWnqK5r4M+rcnj5i/0E+vnw8+n9uX10QssLfruT+lor5At2Wy39gt1QuBfKjp4L/jO6x0LomdBPPBf+YQnWNv8Q7e9X7eKQgBcRbyADOGqMmd1kmwDPATOBKuBuY8yWC72fBrz7yCko59FFmWw4WEy/2O48NmsQk/pFO7uszmdrgPJj1gybJbn228P22yNQmge2uvNfI17WsM6AMOgWDt3CWr/fLdw+3j8O/Lt39tGpLsJRAf8TIB0IaSHgZwI/wAr4K4DnjDFXXOj9NODdizGGf+86zm+WZnGkuIrJ/aN5bNZA1xpt42i2BmsoZ2kunDoMlYVw+hRUl8DpkpbvtzTOH6yWf3AchMRBcE/7bZz9IjD7bVC0zsvvhjo84EUkHngDeAr4SQsB/yqw2hjzrv3xXmCyMeZYa++pAe+eauobeOPLQ7zweQ5VdQ1884pEfnRNPyKC/Jxdmusxxlof90zYVxVDxQkoy7f+Uig/BmX22/Lj1kydjYk3BEZa4/y9fKyTwV6+1tw+3n7n7nv52rfZ9+kWARG9Ibw3RPSxVu7SdXi7jPYEfFtncXoWeBBorTnWC8ht9DjP/tx5AS8i84B5AImJiW38aOVK/H28mTcphVtHxvPsimze3nCERVuP8sOpqdw1Pgl/H21RtpmINTQzIAS4yJKItgbrL4Iz4X/mtrIIGuqsrqGGuvPv2+qt8wm2ynOPG2qt96kuPf/9u/ewwv5s8Pc+d1+vDu6yLhrwIjIbKDDGbBaRya3t1sJzzf40MMYsABaA1YJve5nK1UR29+d/bhrCXeOSeGrpHp5auoe3Nhzm4RkDmD64R9e5GtZdeHlb8/IE9+iY96sqhlMHrUXXiw+eu79/pfWLo7GAUKvLyNvX+svg7G3Tn0bP+/hZ00T7Blp/Hfh1P3ffN8h+Gwh+Qedu/UN0ZtF2asu/1gTgBns/ewAQIiJvGWPmNtonD2g8iXc8kN9xZSpXlRobzN/vGcMX+wr59eLdzH9rC1f0juDx2YMY0ivU2eWp1gRGWD+9RjXfVlsFpw6dC/1Th6Cuymr9N9Ta/1Kw36+vsbqZGj/XUGetA1BXZf20R7cI6B5jnV/oHgNBMdYMo2fud4+238Y4bwoKm806vvpqa0rs826roK7aWqQ+ur/DS2nXMEl7C/5nLfTBzwK+z7mTrM8bY8Zc6L20D97z1DfYeG9TLs8s38epqlpuGRHPj69NJT5c+3c9ls0G9aetXxp1lfbbKqitbHJbZZ2HqCiwJpmrKDx32/hitMb8Q63rELx8Wj7X4OXT/DyEeFnnPmz11jkNW73V/WVs5+6f3WY7163VOMTrqy9+3BN+BNc+eUn/ZI7og2/pQ+YDGGNeAZZihXsO1jDJey71fZX78vH2Yu7YJG5I68lLK3P4238O8fH2o3xjdAL3T+lLXKheFORxvLys7he/IOASh9bWnbYHf2GTXwCF1i+Phnp7ODc693Dm1lZv/fKw1Z0LcPG26vLysd/3sbrAfLud/9jL23rs7Wtt8+lmTVXdltvQXh36z9gavdBJOc3RktO8tCqHf2TkIghzxiTwvSl9iQ3R+dyVao1eyapcSm5xFS+tymHh5jy8vIRvXpHIdyenEBOsQa9UUxrwyiUdOVnFCyuz+dfWo/h6C3eOTeK+q1Lcb44bpS6DBrxyaYeKKnl+ZTYfbj2Kv483d41P4r5JKXqxlFJowCs3caCwguc/z+aj7fkE+nrzrfHJ3DuxD+Ea9MqDacArt5JTUM5zn+eweIcV9LeNTuC/JvQmIUKHVyrPowGv3NK+E+W88sV+Pt6Wj80YZg6NY96kPgyLD3N2aUp1Gg145daOlZ7m718e4p31RyivqWdM7wjmTezD1AExeHnpFAjKvWnAK49QXl3H+5ty+dt/DnG05DR9ooO4d2Ifbh7Ry3XXilXqIjTglUepb7CxNPM4C9bsJ/NoGZFBftw1Lpk7xyXpyBvldjTglUcyxrD+QDGvrT3AyqwCAny9uHVkPP91ZW9SonUFJOUeOmUuGqW6GhFhXEok41IiySko5y9rD/KPjDze3nCEK/tGcee4JK4eEOP+a8YqZacteOXWCstreH/TEd7ZcIT80mp6hgZwxxWJfGN0ItHBeoWscj3aRaNUE/UNNj7PKuCt9YdZm12Er7cwY0gcd41LYlRSuC5AolyGdtEo1YSPtxfTB/dg+uAe7C+s4O31R/jH5lw+3p7PgB7B3DUumRvTehLkr/9LKPehLXjlsapq6/loWz5vfnWYPcfKCPb34dZR8dw5LklPyqouS7tolGoHYwxbjpziza8Os3TnMeoaDGP7RDBnTCLTB/fQMfWqS9GAV+oSFVXU8P6mXN7flMuR4irCAn25ZUQ8c8YkkBob7OzylNKAV+py2WyGrw6c5J2NR1i26zh1DYZRSeHMGZPIrKFxdPPTVr1yDg14pTrQyYoa/rXlKO9uPMKBokqCA3y4Ka0Xc8YkMqhniLPLUx5GA14pBzDGsPFgMe9tymXJzmPU1tsYHh/K7WMSuX54T7rrCBzVCTTglXKwkqpaFm21WvX7TlQQ6OfN9ME9uGlELyakROrVssphNOCV6iTWCJwSFm7OZcmOY5RV1xPV3Z8bhvfkphE9GdorVC+iUh1KA14pJ6ipb2BVViEfbj3KyqwCahts9IkO4ua0Xtw0opeuQKU6RIcGvIgEAGsAf6wrXxcaY55oss9k4CPgoP2pfxljfnWh99WAV+6stKqOTzOPsWjrUTYcLAYgPSmcm0b0YtbQOF1XVl2yjg54AYKMMRUi4gusAx4wxqxvtM9k4GfGmNltLVIDXnmKvFNVfLw9n0VbjpJdUIGvt3BVvxhuTOvJ1QNjCPTTk7Oq7Tp0Lhpj/QaosD/0tf84p19HKRcUHx7I9yb35btXpbD7WBkfbj3KR9vyWbHnBN18vZk6MIbZQ+OYMiBGr5pVHapNffAi4g1sBvoCLxljftFk+2Tgn0AekI/Vmt/VwvvMA+YBJCYmjjp8+PBllq+Ua2qwWUMul+zM59OdxzlZWUuQnzfXDIpl1tA4JvWL1rBXLXLYSVYRCQMWAT8wxmQ2ej4EsNm7cWYCzxljUi/0XtpFo5SlvsHGhoPFLN6Rz6eZxympqiPY34drB8Uye3gcV/aNxs9Hh10qi0NH0YjIE0ClMeYPF9jnEJBujClqbR8NeKWaq2uw8eX+kyzens+/dx2nrLqekAAfpg/uwaxhcYxPidKw93AdfZI1GqgzxpSISDdgGfA7Y8ziRvv0AE4YY4yIjAEWAknmAm+uAa/UhdXW21iXU8jiHcdYvusE5TX1BPl5M75vFJP7RzO5fwy9wro5u0zVyTp6wY844A17P7wX8IExZrGIzAcwxrwCfA34rojUA6eB2y8U7kqpi/Pz8WLqgFimDoiluq6BddlFrNpbwOq9hSzffQKAfrHdmdw/hsn9o0lPitDWvTqPXuiklIsxxpBTUMHqvYWs3lfAxoPF1DUYgvy8mdA3iikDrMCPC9XWvTvSK1mV8iAVNfV8mVPE6n2FrM4qIL+0GoD+scFMHhDN1P4xjEoK1/lx3IQGvFIeyhhDdkEFq/cWsCqrkE2Hiqm3GUK7+XJVv2imDojhqn7ReiWtC9OAV0oBUF5dx7rsIj7PKmD13gKKKmrxEhiVFM6UATFcPSCWfrHddUI0F6IBr5RqxmYz7Dhayso9J1i5t4DMo2UA9ArrxtQBMUwdGMO4PpF6gVUXpwGvlLqoE2XVrMoq4POsAtZlF3G6roEAXy/Gp0RxZd8oJqZG0TdGW/ddjQa8Uqpdqusa2HCwmJV7TrAmu4iDRZUA9AgJYII97Cf0jSI62N/JlaqOHgevlHJzAb7eXNUvmqv6RQPWDJjrsotYm13E51kn+OeWPAAG9AhmYmoUV6ZGMyY5Qhcf7+K0Ba+UuqAGm2FXfilrs4tYl13E5sOnqG2w4efjRXpSOFemRjEpNZpBcSF4eWl3jqNpF41SymGqauvZeLCYddlFrMspIut4OQARQX5n++4npkbTIzTAyZW6J+2iUUo5TKCfj316hBgACsqqWZdjdeeszS7i4+35gDWNwsTUaCamRnFF70jtznECbcErpTqMMYas4+WszS5kbXYRGw4WU1tvw8/bi/Tk8LOBr905l067aJRSXUJ1XQMbDxafDfwz3TmRQX6MTYlkfEok41OiSI4M1OGYbaRdNEqpLiHA15tJ/aKZZB+dc6Y7Z112Ef/ZX8SSHccAiAsNYJw97MenRNJTp0HuENqCV0o5hTGGg0WVfLn/JF/tP8lXB05SXFkLQHJkIONSopjQN5KxfSKJ6q7j78/QLhqllMux2Qx7T5TbA7+IDQeKKa+pB6zx92N6RzA8Poy0xDB6RwZ5bB++BrxSyuXVN9jIzC/jy/1FfJlzkq1HTlFZ2wBAcIAPw+PDGJ4QaoV+QhgxIZ4xLFMDXinldhpshv2FFWzLLWF7bgnbckvIOl5Og83KsLjQANISwhieEMbw+DCGxofS3d/9TjPqSVallNvx9hL6xQbTLzaY29ITAGuUzq78UrbllrI9t4TteSV8mnkcABFr0ZP05HDSkyIYlRROfHg3jxqtoy14pZRbKa6sZUdeCVuPlLDlyCm2Himhwt6XHxviT3pSxNnQHxgX7HIrXWkLXinlsSKC/M670rbBZsg6Xsbmw6fIOHSKjEPFLNlpDc8M9PMmLSGM9KRwRiVHMDIxjOAAX2eW36G0Ba+U8jj5JafJOHyKzYeKyTh8ij3HyrAZq1unX0wwIxLDGJEYxsjEcFKiu3epETt6klUppdqhoqaebUdKyDhczNYjJWw9coqyaqtbJ9jfh7TEMEYkhDEiKZwRCWGEBTpvTVvtolFKqXbo7u/DlalRXJkaBVhj8g+erDwb9luOlPDiqhzsA3boExVkhX5iOGnxYfTr0R1/n643mZq24JVSqg0qa+rZkVfK1txTZ4O/qMK68tbHS+gb053BPUMZ3DOEwT1DGNQzxCH9+R3agheRAGAN4G/ff6Ex5okm+wjwHDATqALuNsZsaW/hSinVVQX5+zAuJZJxKZGANdVC3qnT7MgrZVd+Kbvyy/hiX+HZ1a8AkiID7YEfyiB78McEd94FWW3poqkBphpjKkTEF1gnIp8aY9Y32mcGkGr/uQJ42X6rlFJuSURIiAgkISKQWcPizj5fUFbNrvwyduWXsvtYGbvyy1i68/jZ7VHd/blvUh/undTH4TVeNOCN1YdTYX/oa/9p2q9zI/Cmfd/1IhImInHGmGMdWq1SSnVxMSEBxIQEMGVAzNnnyqrr2JNfZg/+MmJCOmfytDadZBURb2Az0Bd4yRizockuvYDcRo/z7M+dF/AiMg+YB5CYmHiJJSullGsJCfDlij6RXNEnslM/t02XcBljGowxaUA8MEZEhjTZpaVBos3O3hpjFhhj0o0x6dHR0e0uVimlVNu16xpdY0wJsBq4rsmmPCCh0eN4IP9yClNKKXV5LhrwIhItImH2+92Aa4CsJrt9DNwllrFAqfa/K6WUc7WlDz4OeMPeD+8FfGCMWSwi8wGMMa8AS7GGSOZgDZO8x0H1KqWUaqO2jKLZAYxo4flXGt03wP0dW5pSSqnL4VrzZCqllGozDXillHJTGvBKKeWmnDbZmIgUAocv8eVRQFEHluNqPPn4PfnYwbOPX4/dkmSMadOFRE4L+MshIhltnU3NHXny8XvysYNnH78ee/uPXbtolFLKTWnAK6WUm3LVgF/g7AKczJOP35OPHTz7+PXY28kl++CVUkpdnKu24JVSSl2EBrxSSrkplwt4EblORPaKSI6IPOTsejqTiBwSkZ0isk1E3H7FchF5XUQKRCSz0XMRIrJcRLLtt+HOrNFRWjn2/xaRo/bvf5uIzHRmjY4iIgkiskpE9ojILhF5wP68p3z3rR1/u79/l+qDt89ouQ+4FmsO+k3AHGPMbqcW1klE5BCQbozxiIs9RGQS1nKRbxpjhtif+z1QbIx52v4LPtwY8wtn1ukIrRz7fwMVxpg/OLM2RxOROCDOGLNFRIKxVpO7Cbgbz/juWzv+22jn9+9qLfgxQI4x5oAxphZ4D2s9WOWGjDFrgOImT98IvGG//wbWf/hup5Vj9wjGmGPGmC32++XAHqwlQD3lu2/t+NvN1QK+tbVfPYUBlonIZvv6tp4o9sxiMvbbmIvs726+LyI77F04btlF0ZiIJGNNV74BD/zumxw/tPP7d7WAb9Par25sgjFmJDADuN/+Z7zyHC8DKUAa1oL2f3RqNQ4mIt2BfwI/MsaUObueztbC8bf7+3e1gPfotV+NMfn22wJgEVaXlac5Ye+jPNNXWeDkejqNMeaEMabBGGMDXsONv38R8cUKt7eNMf+yP+0x331Lx38p37+rBfwmIFVEeouIH3A71nqwbk9EguwnXBCRIGAakHnhV7mlj4Fv2e9/C/jIibV0qjPhZnczbvr9i4gAfwX2GGOeabTJI7771o7/Ur5/lxpFA2AfGvQs4A28box5yrkVdQ4R6YPVagdrqcV33P3YReRdYDLWVKkngCeAD4EPgETgCPB1Y4zbnYxs5dgnY/15boBDwH3uuLi9iFwJrAV2Ajb7049g9UN7wnff2vHPoZ3fv8sFvFJKqbZxtS4apZRSbaQBr5RSbkoDXiml3JQGvFJKuSkNeKWUclMa8Eop5aY04JVSyk39P4c8XvPEeQVpAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['loss'], label='train')\n",
    "plt.plot(history.history['val_loss'], label='test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 인퍼런스 모델 구현\n",
    "    - test 할 때, 정수 인덱스 행렬로 존재하던 텍스트 데이터를 실제 데이터로 복원 필요\n",
    "    - => 사전 3개 준비\n",
    "    - seq2seq\n",
    "        - 훈련할 때, 동작(인퍼런스 단계)할 때 방식이 다름 => 모델 설계 별개로 진행해야 함\n",
    "        - 훈련 단계: 인코더와 디코더를 엮은 모델\n",
    "        - 인퍼런스 단계\n",
    "            - 정답 문장 없음\n",
    "            - => 만들어야 할 문장 길이만큼 디코더가 반복 구조로 동작해야 함\n",
    "            - => 인퍼런스를 위한 모델 설계 별도로 진행\n",
    "            - => 인코더 모델, 디코더 모델 분리해서 설계"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "사전 3개 준비"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "# 사전 3개\n",
    "src_index_to_word = src_tokenizer.index_word # 원문 단어 집합에서 정수 -> 단어를 얻음\n",
    "tar_word_to_index = tar_tokenizer.word_index # 요약 단어 집합에서 단어 -> 정수를 얻음\n",
    "tar_index_to_word = tar_tokenizer.index_word # 요약 단어 집합에서 정수 -> 단어를 얻음\n",
    "\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련 단계: 인코더와 디코더를 엮은 모델 설계"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "# 인코더 설계\n",
    "encoder_model = Model(inputs=encoder_inputs, outputs=[encoder_outputs, state_h, state_c])\n",
    "\n",
    "# 이전 시점의 상태들을 저장하는 텐서\n",
    "decoder_state_input_h = Input(shape=(hidden_size,))\n",
    "decoder_state_input_c = Input(shape=(hidden_size,))\n",
    "\n",
    "dec_emb2 = dec_emb_layer(decoder_inputs)\n",
    "\n",
    "# 문장의 다음 단어를 예측하기 위해서 초기 상태(initial_state)를 이전 시점의 상태로 사용. 이는 뒤의 함수 decode_sequence()에 구현\n",
    "# 훈련 과정에서와 달리 LSTM의 리턴하는 은닉 상태와 셀 상태인 state_h와 state_c를 버리지 않음.\n",
    "decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=[decoder_state_input_h, decoder_state_input_c])\n",
    "\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "인퍼런스 단계: 인코더 모델, 디코더 모델 분리해서 설계"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "# 어텐션 함수\n",
    "decoder_hidden_state_input = Input(shape=(text_max_len, hidden_size))\n",
    "attn_out_inf = attn_layer([decoder_outputs2, decoder_hidden_state_input])\n",
    "decoder_inf_concat = Concatenate(axis=-1, name='concat')([decoder_outputs2, attn_out_inf])\n",
    "\n",
    "# 디코더의 출력층\n",
    "decoder_outputs2 = decoder_softmax_layer(decoder_inf_concat) \n",
    "\n",
    "# 최종 디코더 모델\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + [decoder_hidden_state_input,decoder_state_input_h, decoder_state_input_c],\n",
    "    [decoder_outputs2] + [state_h2, state_c2])\n",
    "\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    # 입력으로부터 인코더의 상태를 얻음\n",
    "    e_out, e_h, e_c = encoder_model.predict(input_seq)\n",
    "\n",
    "     # <SOS>에 해당하는 토큰 생성\n",
    "    target_seq = np.zeros((1,1))\n",
    "    target_seq[0, 0] = tar_word_to_index['sostoken']\n",
    "\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition: # stop_condition이 True가 될 때까지 루프 반복\n",
    "\n",
    "        output_tokens, h, c = decoder_model.predict([target_seq] + [e_out, e_h, e_c])\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_token = tar_index_to_word[sampled_token_index]\n",
    "\n",
    "        if (sampled_token!='eostoken'):\n",
    "            decoded_sentence += ' '+sampled_token\n",
    "\n",
    "        #  <eos>에 도달하거나 최대 길이를 넘으면 중단.\n",
    "        if (sampled_token == 'eostoken'  or len(decoded_sentence.split()) >= (headlines_max_len-1)):\n",
    "            stop_condition = True\n",
    "\n",
    "        # 길이가 1인 타겟 시퀀스를 업데이트\n",
    "        target_seq = np.zeros((1,1))\n",
    "        target_seq[0, 0] = sampled_token_index\n",
    "\n",
    "        # 상태를 업데이트 합니다.\n",
    "        e_h, e_c = h, c\n",
    "\n",
    "    return decoded_sentence\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=3\n"
     ]
    }
   ],
   "source": [
    "# 원문의 정수 시퀀스를 텍스트 시퀀스로 변환\n",
    "def seq2text(input_seq):\n",
    "    temp=''\n",
    "    for i in input_seq:\n",
    "        if (i!=0):\n",
    "            temp = temp + src_index_to_word[i]+' '\n",
    "    return temp\n",
    "\n",
    "# 요약문의 정수 시퀀스를 텍스트 시퀀스로 변환\n",
    "def seq2summary(input_seq):\n",
    "    temp=''\n",
    "    for i in input_seq:\n",
    "        if ((i!=0 and i!=tar_word_to_index['sostoken']) and i!=tar_word_to_index['eostoken']):\n",
    "            temp = temp + tar_index_to_word[i] + ' '\n",
    "    return temp\n",
    "\n",
    "print('=3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4. 실제 결과와 요약문 비교하기 (추상적 요약)\n",
    "원래의 요약문(headlines 열)과 학습을 통해 얻은 추상적 요약의 결과를 비교해 보세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* 원문 : india elected un human rights council highest votes india permanent representative un syed akbaruddin said continue work balanced way protect human rights india victory reflection high standing enjoys globally akbaruddin added india secured votes followed fiji secured votes \n",
      "* 실제 요약 : will work in balanced way to protect human rights india \n",
      "* 추상적 요약 :  un un council rejects un rights council\n",
      "\n",
      "\n",
      "* 원문 : air hostess helped pilot land fly flight carrying people october co pilot fell ill passed revealed tuesday following investigation german airline fly said air hostess trained flight security expert would rewarded investigators said safety incident \n",
      "* 실제 요약 : air hostess helped land plane after co pilot passed out \n",
      "* 추상적 요약 :  indigo flight makes emergency landing after cockpit\n",
      "\n",
      "\n",
      "* 원문 : telecom commission tuesday approved proposal allowing phone calls internet service flights within indian airspace telecom regulator trai recommended making calls allowed aircraft reaches metres government intervene pricing mechanism would left airlines flight connectivity providers \n",
      "* 실제 요약 : flights to get internet calling service as govt clears plan \n",
      "* 추상적 요약 :  govt proposes nod to reduce air india to reduce costs\n",
      "\n",
      "\n",
      "* 원문 : rape victim silence cannot taken proof consent sexual relations delhi high court observed upholding man year jail term raping pregnant woman court made observation rejecting rape convict defence victim silence regarding incident proved consensual sexual relations existed \n",
      "* 실제 요약 : victim silence rape not proof of consent hc \n",
      "* 추상적 요약 :  rape victim is not rape victim on rape case\n",
      "\n",
      "\n",
      "* 원문 : co working space giant wework raised million funding series round led temasek softbank vision fund others china business latest funding reportedly values business billion wework entered china two years ago covers beijing shanghai nearly locations \n",
      "* 실제 요약 : co working space wework raises mn for china business \n",
      "* 추상적 요약 :  wework raises million from wework in wework\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print(\"* 원문 :\", seq2text(encoder_input_test[i]))\n",
    "    print(\"* 실제 요약 :\", seq2summary(decoder_input_test[i]))\n",
    "    print(\"* 추상적 요약 :\", decode_sequence(encoder_input_test[i].reshape(1, text_max_len)))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 추상적 요약(Abstractive Summarization)\n",
    "    - 원문으로부터 내용이 요약된 새로운 문장을 생성\n",
    "    - 자연어 생성(Natural Language Generation, NLG)의 영역"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 실제 요약(headlines)과 추상적 요약 결과 비교\n",
    "    - 실제 요약과 추상적 요약을 비교해봤을 때 의미가 비슷한 경우가 있었다.\n",
    "    - 그렇지만 아예 다른 주제의 요약을 한 경우가 있었다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5. Summa을 이용해서 추출적 요약해보기\n",
    "추상적 요약은 추출적 요약과는 달리 문장의 표현력을 다양하게 가져갈 수 있지만, 추출적 요약에 비해서 난이도가 높아요. 반대로 말하면 추출적 요약은 추상적 요약에 비해 난이도가 낮고 기존 문장에서 문장을 꺼내오는 것이므로 잘못된 요약이 나올 가능성이 낮아요.\n",
    "\n",
    "Summa의 summarize를 사용하여 추출적 요약을 해보세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Summa의 summarize()의 인자로 사용되는 값\n",
    "\n",
    ">text (str) : 요약할 테스트.\n",
    "\n",
    ">ratio (float, optional) – 요약문에서 원본에서 선택되는 문장 비율. 0~1 사이값\n",
    "\n",
    ">words (int or None, optional) – 출력에 포함할 단어 수.\n",
    "\n",
    ">만약, ratio와 함께 두 파라미터가 모두 제공되는 경우 ratio는 무시한다.\n",
    "\n",
    ">split (bool, optional) – True면 문장 list / False는 조인(join)된 문자열을 반환"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from summa.summarizer import summarize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = pd.read_csv(DATA_PATH, encoding='iso-8859-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              headlines  \\\n",
      "0     upGrad learner switches to career in ML & Al w...   \n",
      "1     Delhi techie wins free food from Swiggy for on...   \n",
      "2     New Zealand end Rohit Sharma-led India's 12-ma...   \n",
      "3     Aegon life iTerm insurance plan helps customer...   \n",
      "4     Have known Hirani for yrs, what if MeToo claim...   \n",
      "...                                                 ...   \n",
      "1495  Have seen Sachin Tendulkar get angry but not M...   \n",
      "1496  Player ensures wife stays dry in rain during A...   \n",
      "1497  Federer denied entry into locker room by secur...   \n",
      "1498  42/42 seats in West Bengal: CM Mamata's cutout...   \n",
      "1499  Court refuses chargesheet by Delhi Police agai...   \n",
      "\n",
      "                                                   text  \n",
      "0     Saurav Kant, an alumnus of upGrad and IIIT-B's...  \n",
      "1     Kunal Shah's credit card bill payment platform...  \n",
      "2     New Zealand defeated India by 8 wickets in the...  \n",
      "3     With Aegon Life iTerm Insurance plan, customer...  \n",
      "4     Speaking about the sexual harassment allegatio...  \n",
      "...                                                 ...  \n",
      "1495  India head coach Ravi Shastri has said that he...  \n",
      "1496  Italian tennis player Andreas Seppi ensured hi...  \n",
      "1497  World number three Roger Federer was stopped b...  \n",
      "1498  A cutout of TMC chief and West Bengal CM Mamat...  \n",
      "1499  A Delhi court on Saturday pulled up Delhi Poli...  \n",
      "\n",
      "[1500 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "print(text[:1500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* 원문 : Saurav Kant, an alumnus of upGrad and IIIT-B's PG Program in Machine learning and Artificial Intelligence, was a Sr Systems Engineer at Infosys with almost 5 years of work experience. The program and upGrad's 360-degree career support helped him transition to a Data Scientist at Tech Mahindra with 90% salary hike. upGrad's Online Power Learning has powered 3 lakh+ careers.\n",
      "* 실제 요약 : upGrad learner switches to career in ML & Al with 90% salary hike\n",
      "* 추출적 요약 : upGrad's Online Power Learning has powered 3 lakh+ careers.\n",
      "\n",
      "\n",
      "* 원문 : Kunal Shah's credit card bill payment platform, CRED, gave users a chance to win free food from Swiggy for one year. Pranav Kaushik, a Delhi techie, bagged this reward after spending 2000 CRED coins. Users get one CRED coin per rupee of bill paid, which can be used to avail rewards from brands like Ixigo, BookMyShow, UberEats, Cult.Fit and more.\n",
      "* 실제 요약 : Delhi techie wins free food from Swiggy for one year on CRED\n",
      "* 추출적 요약 : Users get one CRED coin per rupee of bill paid, which can be used to avail rewards from brands like Ixigo, BookMyShow, UberEats, Cult.Fit and more.\n",
      "\n",
      "\n",
      "* 원문 : New Zealand defeated India by 8 wickets in the fourth ODI at Hamilton on Thursday to win their first match of the five-match ODI series. India lost an international match under Rohit Sharma's captaincy after 12 consecutive victories dating back to March 2018. The match witnessed India getting all out for 92, their seventh lowest total in ODI cricket history.\n",
      "* 실제 요약 : New Zealand end Rohit Sharma-led India's 12-match winning streak\n",
      "* 추출적 요약 : The match witnessed India getting all out for 92, their seventh lowest total in ODI cricket history.\n",
      "\n",
      "\n",
      "* 원문 : With Aegon Life iTerm Insurance plan, customers can enjoy tax benefits on your premiums paid and save up to Ã¢ÂÂ¹46,800^ on taxes. The plan provides life cover up to the age of 100 years. Also, customers have options to insure against Critical Illnesses, Disability and Accidental Death Benefit Rider with a life cover up to the age of 80 years.\n",
      "* 실제 요약 : Aegon life iTerm insurance plan helps customers save tax\n",
      "* 추출적 요약 : Also, customers have options to insure against Critical Illnesses, Disability and Accidental Death Benefit Rider with a life cover up to the age of 80 years.\n",
      "\n",
      "\n",
      "* 원문 : Speaking about the sexual harassment allegations against Rajkumar Hirani, Sonam Kapoor said, \"I've known Hirani for many years...What if it's not true, the [#MeToo] movement will get derailed.\" \"In the #MeToo movement, I always believe a woman. But in this case, we need to reserve our judgment,\" she added. Hirani has been accused by an assistant who worked in 'Sanju'.\n",
      "* 실제 요약 : Have known Hirani for yrs, what if MeToo claims are not true: Sonam\n",
      "* 추출적 요약 : Speaking about the sexual harassment allegations against Rajkumar Hirani, Sonam Kapoor said, \"I've known Hirani for many years...What if it's not true, the [#MeToo] movement will get derailed.\" \"In the #MeToo movement, I always believe a woman.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print(\"* 원문 :\", text['text'][i])\n",
    "    print(\"* 실제 요약 :\", text['headlines'][i])\n",
    "    print(\"* 추출적 요약 :\", summarize(text['text'][i], ratio=0.4))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 추출적 요약(Extractive Summarization)\n",
    "    - 원문에서 문장을 추출해서 요약하는 방식 => 결과로 나온 문장들 간 호응이 자연스럽지 않을 수 있다.\n",
    "    - 문장 분류(Text Classification) 문제\n",
    "    - cf. ratio 값이 0.3 이하면 추출적 요약 결과가 나오지 않는다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 실제 요약(headlines)과 추출적 요약 결과 비교\n",
    "    - 문장을 추출하는 방식이라 그런지 요약이라기엔 내용이 많은 경우가 있었다.\n",
    "    - 관련 내용의 문장이라서 요약은 잘 됐다고 볼 수 있다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 추상적 요약(Abstractive Summarization) vs 추출적 요약(Extractive Summarization)\n",
    "- Abstractive\n",
    "    - 문장을 새로 만들어낸 Abstractive 는 실제 요약과 다른 주제를 만들어내는 경우가 꽤 있었다.\n",
    "\n",
    "|headlines|Abstractive|\n",
    "|---|---|\n",
    "|will work in balanced way to protect human rights india|un un council rejects un rights council|\n",
    "|air hostess helped land plane after co pilot passed out|indigo flight makes emergency landing after cockpit|\n",
    "|flights to get internet calling service as govt clears plan|govt proposes nod to reduce air india to reduce costs|\n",
    "|victim silence rape not proof of consent hc|rape victim is not rape victim on rape case|\n",
    "|co working space wework raises mn for china business|wework raises million from wework in wework|\n",
    "\n",
    "- Extractive\n",
    "    - 원문에 있는 문장을 그대로 추출한 Extractive 가 아무래도 실제 요약과 핵심단어가 겹친다.\n",
    "    - Extractive 는 요약이라기엔 문장이 길어서 살짝 아쉬웠다.\n",
    "\n",
    "|headlines|Extractive|\n",
    "|---|---|\n",
    "|upGrad learner switches to career in ML & Al with 90% salary hike|upGrad's Online Power Learning has powered 3 lakh+ careers.|\n",
    "|Delhi techie wins free food from Swiggy for one year on CRED|Users get one CRED coin per rupee of bill paid, which can be used to avail rewards from brands like Ixigo, BookMyShow, UberEats, Cult.Fit and more.|\n",
    "|New Zealand end Rohit Sharma-led India's 12-match winning streak|The match witnessed India getting all out for 92, their seventh lowest total in ODI cricket history.|\n",
    "|Aegon life iTerm insurance plan helps customers save tax|Also, customers have options to insure against Critical Illnesses, Disability and Accidental Death Benefit Rider with a life cover up to the age of 80 years.|\n",
    "|Have known Hirani for yrs, what if MeToo claims are not true: Sonam|Speaking about the sexual harassment allegations against Rajkumar Hirani, Sonam Kapoor said, \"I've known Hirani for many years...What if it's not true, the [#MeToo] movement will get derailed.\" \"In the #MeToo movement, I always believe a woman.|\n",
    "\n",
    "- 결론\n",
    "    - Extractive 가 Abstractive 보다 상대적으로 요약을 더 잘 한 것 같다.\n",
    "    - 전처리를 다르게 하거나 원문이 더 길면 달라질 수도 있다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 회고\n",
    "\n",
    "### - 이번 프로젝트에서 **어려웠던 점**.\n",
    "텍스트 전처리 및 모델 학습이 오래 걸려서 기다리는 시간이 길었다. 컴퓨터가 굉음을 내면서 돌아가고 있다. 병렬 프로그래밍을 설정하려다가 끝낼 수 없을 것 같아서 일단 제출했다.\n",
    "\n",
    "### - 프로젝트를 진행하면서 **알아낸 점** 혹은 **아직 모호한 점**.\n",
    "1. 텍스트를 모델 인코더에 입력하면 인코더는 컨텍스트 벡터(context vector)로 변환한다.\n",
    "2. LSTM과 바닐라 RNN과 다른 점은 다음 time step 셀에 hidden state, cell state를 함께 전달한다.\n",
    "3. 어텐션 메커니즘(Attention Mechanism): 인코더의 모든 step의 hidden state 정보가 컨텍스트 벡터에 전부 반영됨(가중합 계산)\n",
    "4. 텍스트는 데이터 전처리와 모델 학습이 매우 오래 걸린다!\n",
    "\n",
    "### - 루브릭 평가 지표를 맞추기 위해 **시도한 것들**.\n",
    "\n",
    ">#### **루브릭**\n",
    ">|번호|평가문항|상세기준|\n",
    ">|:---:|---|---|\n",
    ">|1|Abstractive 모델 구성을 위한 텍스트 전처리 단계가 체계적으로 진행되었다.|분석단계, 정제단계, 정규화와 불용어 제거, 데이터셋 분리, 인코딩 과정이 빠짐없이 체계적으로 진행되었다.|\n",
    ">|2|텍스트 요약모델이 성공적으로 학습되었음을 확인하였다.|모델학습이 안정적으로 수렴되었음을 그래프를 통해 확인하였으며, 실제 요약문과 유사한 요약문장을 얻을 수 있었다.|\n",
    ">|3|Extractive 요약을 시도해 보고 Abstractive 요약 결과과 함께 비교해 보았다.|두 요약 결과를 문법완성도 측면과 핵심단어 포함 측면으로 나누어 비교분석 결과를 제시하였다.|\n",
    "\n",
    "seq2seq를 구현할 때 바닐라 RNN이 아니라 LSTM 을 사용한다. seq2seq는 어텐션 메커니즘을 적용했다. 텍스트 전처리를 위해 중복 샘플 제거, null 값 제거, 불용어 제거를 통해 전처리를 완료했다. 최대 길이를 설정해서 데이터 제거 + 빈도수가 낮은 단어 제거를 통해 단어 집합(=정수 인코딩)을 만들었다. 데이터셋은 train : test = 8 : 2 비율로 분리했다. 인코더와 어텐션 메커니즘을 적용한 디코더를 만들어서 훈련시키고 인퍼런스 모델에 넣어서 실제 데이터로 복원시켰다. 이후 모델 테스트를 통해 추상적 요약(Abstractive Summary)과 추출적 요약(Extractive Summary)을 만들어서 실제 요약(headlines)와 비교할 수 있었다.\n",
    "\n",
    "기존 lms 코드를 사용해서 완성했다. 다른 걸 시도해보려고 했지만 전처리 시간과 학습 시간이 너무 오래 걸려서 기존 코드가 돌아가는 것만 확인했다. epoch=50 으로 학습하던 모델이 Early Stopping으로 인해 25에서 멈췄다. 그래프를 보면 학습할수록 train_loss 와 val_loss 값이 줄어드는 것을 볼 수 있다.\n",
    "\n",
    "요약 문장을 얻어보았다. 아무래도 원문에 있는 문장을 그대로 가져오는 Extractive 가 상대적으로 더 요약을 잘 한 것 같다. 문장을 만들어내는 Abstractive 는 실제 요약과 다른 내용을 도출하는 경우가 있었기 때문이다.\n",
    "\n",
    "### - 만약에 루브릭 평가 관련 지표를 **달성 하지 못했을 때, 이유에 관한 추정**.\n",
    "없음\n",
    "\n",
    "### - **자기 다짐**\n",
    "병렬 프로그래밍을 구현하는 방법을 찾아봐야 겠다."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b3ce9abe337a9e694d01ea52d504102083454ad8bd4b0e3a574e4432f4229329"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 64-bit ('aiffel_3.8': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}