Chapter11.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning with PyTorch Step-by-Step: A Beginner's Guide"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install gensim==3.8.3\n",
    "# !pip install allennlp==0.9.0\n",
    "# !pip install flair==0.8.0.post1 # uses PyTorch 1.7.1\n",
    "# !pip install torchvision==0.8.2\n",
    "# # HuggingFace\n",
    "# !pip install transformers==4.5.1\n",
    "# !pip install datasets==1.6.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "    import requests\n",
    "    url = 'https://raw.githubusercontent.com/dvgodoy/PyTorchStepByStep/master/config.py'\n",
    "    r = requests.get(url, allow_redirects=True)\n",
    "    open('config.py', 'wb').write(r.content)    \n",
    "except ModuleNotFoundError:\n",
    "    pass\n",
    "\n",
    "from config import *\n",
    "config_chapter11()\n",
    "# This is needed to render the plots in this chapter\n",
    "from plots.chapter11 import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import errno\n",
    "import requests\n",
    "import numpy as np\n",
    "from copy import deepcopy\n",
    "from operator import itemgetter\n",
    "\n",
    "import torch\n",
    "import torch.optim as optim\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader, TensorDataset, Dataset\n",
    "\n",
    "from data_generation.nlp import ALICE_URL, WIZARD_URL, download_text\n",
    "from stepbystep.v4 import StepByStep\n",
    "# These are the classes we built in Chapter 10\n",
    "from seq2seq import *\n",
    "\n",
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "import gensim\n",
    "from gensim import corpora, downloader\n",
    "from gensim.parsing.preprocessing import *\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.models import Word2Vec\n",
    "\n",
    "from flair.data import Sentence\n",
    "from flair.embeddings import ELMoEmbeddings, WordEmbeddings, \\\n",
    "    TransformerWordEmbeddings, TransformerDocumentEmbeddings\n",
    "\n",
    "from datasets import load_dataset, Split\n",
    "from transformers import (\n",
    "    DataCollatorForLanguageModeling,\n",
    "    BertModel, BertTokenizer, BertForSequenceClassification,\n",
    "    DistilBertModel, DistilBertTokenizer, \n",
    "    DistilBertForSequenceClassification,\n",
    "    AutoModelForSequenceClassification,\n",
    "    AutoModel, AutoTokenizer, AutoModelForCausalLM,\n",
    "    Trainer, TrainingArguments, pipeline, TextClassificationPipeline\n",
    ")\n",
    "from transformers.pipelines import SUPPORTED_TASKS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Down the Yellow Brick Rabbit Hole"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/alice_dorothy.png)\n",
    "\n",
    "*Left: \"Alice and the Baby Pig\" illustration by John Tenniel's, from \"Alice's Adventure's in Wonderland\" (1865).*\n",
    "\n",
    "*Right: \"Dorothy meets the Cowardly Lion\" illustration by W.W. Denslow, from \"The Wonderful Wizard of Oz\" (1900)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "localfolder = 'texts'\n",
    "download_text(ALICE_URL, localfolder)\n",
    "download_text(WIZARD_URL, localfolder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(localfolder, 'alice28-1476.txt'), 'r') as f:\n",
    "    alice = ''.join(f.readlines()[104:3704])\n",
    "    \n",
    "with open(os.path.join(localfolder, 'wizoz10-1740.txt'), 'r') as f:\n",
    "    wizard = ''.join(f.readlines()[310:5100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                ALICE'S ADVENTURES IN WONDERLAND\n",
      "\n",
      "                          Lewis Carroll\n",
      "\n",
      "               THE MILLENNIUM FULCRUM EDITION 2.8\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "                            CHAPTER I\n",
      "\n",
      "                      Down the Rabbit-Hole\n",
      "\n",
      "\n",
      "  Alice was beginning to get very tired of sitting by her sister\n",
      "on the bank, and of having nothing to do:  once or twice she had\n",
      "peeped into the book her sister was reading, but it had no\n",
      "pictures or conversations in it, `and what is the use of a book,'\n",
      "thought Alice `w\n",
      "\n",
      "\n",
      "                    THE WONDERFUL WIZARD OF OZ\n",
      "\n",
      "\n",
      "                          1.  The Cyclone\n",
      "\n",
      "\n",
      "    Dorothy lived in the midst of the great Kansas prairies, with\n",
      "Uncle Henry, who was a farmer, and Aunt Em, who was the farmer's\n",
      "wife.  Their house was small, for the lumber to build it had to be\n",
      "carried by wagon many miles.  There were four walls, a floor and a\n",
      "roof, which made one room; and this room contained a rusty looking\n",
      "cookstove, a cupboard for the dishes, a table, three or four\n",
      "chairs, and th\n"
     ]
    }
   ],
   "source": [
    "print(alice[:500])\n",
    "print('\\n')\n",
    "print(wizard[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_cfg = \"\"\"fname,start,end\n",
    "alice28-1476.txt,104,3704\n",
    "wizoz10-1740.txt,310,5100\"\"\"\n",
    "bytes_written = open(os.path.join(localfolder, 'lines.cfg'), 'w').write(text_cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"I'm\", 'following', 'the', 'white', 'rabbit']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"I'm following the white rabbit\"\n",
    "tokens = sentence.split(' ')\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/dvgodoy/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1612, 2240)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "nltk.download('punkt')\n",
    "corpus_alice = sent_tokenize(alice)\n",
    "corpus_wizard = sent_tokenize(wizard)\n",
    "len(corpus_alice), len(corpus_wizard)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'There was nothing so VERY remarkable in that; nor did Alice\\nthink it so VERY much out of the way to hear the Rabbit say to\\nitself, `Oh dear!'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_alice[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\"There\\'s a cyclone coming, Em,\" he called to his wife.'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_wizard[30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence_tokenize(source, quote_char='\\\\', sep_char=',',\n",
    "                      include_header=True, include_source=True, \n",
    "                      extensions=('txt'), **kwargs):\n",
    "    nltk.download('punkt')\n",
    "    # If source is a folder, goes through all files inside it\n",
    "    # that match the desired extensions ('txt' by default)\n",
    "    if os.path.isdir(source):\n",
    "        filenames = [f for f in os.listdir(source)\n",
    "                     if os.path.isfile(os.path.join(source, f)) and\n",
    "                        os.path.splitext(f)[1][1:] in extensions]\n",
    "    elif isinstance(source, str):\n",
    "        filenames = [source]\n",
    "    \n",
    "    # If there is a configuration file, builds a dictionary with\n",
    "    # the corresponding start and end lines of each text file\n",
    "    config_file = os.path.join(source, 'lines.cfg')\n",
    "    config = {}\n",
    "    if os.path.exists(config_file):\n",
    "        with open(config_file, 'r') as f:\n",
    "            rows = f.readlines()\n",
    "\n",
    "        for r in rows[1:]:\n",
    "            fname, start, end = r.strip().split(',')\n",
    "            config.update({fname: (int(start), int(end))})\n",
    "       \n",
    "    new_fnames = []\n",
    "    # For each file of text\n",
    "    for fname in filenames:\n",
    "        # If there's a start and end line for that file, use it\n",
    "        try:\n",
    "            start, end = config[fname]\n",
    "        except KeyError:\n",
    "            start = None\n",
    "            end = None\n",
    "            \n",
    "        # Opens the file, slices the configures lines (if any)\n",
    "        # cleans line breaks and uses the sentence tokenizer\n",
    "        with open(os.path.join(source, fname), 'r') as f:\n",
    "            contents = (''.join(f.readlines()[slice(start, end, None)])\n",
    "                        .replace('\\n', ' ').replace('\\r', ''))\n",
    "        corpus = sent_tokenize(contents, **kwargs)\n",
    "        \n",
    "        # Builds a CSV file containing tokenized sentences\n",
    "        base = os.path.splitext(fname)[0]\n",
    "        new_fname = f'{base}.sent.csv'\n",
    "        new_fname = os.path.join(source, new_fname)\n",
    "        with open(new_fname, 'w') as f:\n",
    "            # Header of the file\n",
    "            if include_header:\n",
    "                if include_source:\n",
    "                    f.write('sentence,source\\n')\n",
    "                else:\n",
    "                    f.write('sentence\\n')\n",
    "            # Writes one line for each sentence\n",
    "            for sentence in corpus:\n",
    "                if include_source:\n",
    "                    f.write(f'{quote_char}{sentence}{quote_char}{sep_char}{fname}\\n')\n",
    "                else:\n",
    "                    f.write(f'{quote_char}{sentence}{quote_char}\\n')\n",
    "        new_fnames.append(new_fname)\n",
    "        \n",
    "    # Returns list of the newly generated CSV files\n",
    "    return sorted(new_fnames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/dvgodoy/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['texts/alice28-1476.sent.csv', 'texts/wizoz10-1740.sent.csv']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_fnames = sentence_tokenize(localfolder)\n",
    "new_fnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import spacy\n",
    "# nlp = spacy.blank(\"en\")\n",
    "# nlp.add_pipe(nlp.create_pipe(\"sentencizer\"))\n",
    "\n",
    "# sentences = []\n",
    "# for doc in nlp.pipe(corpus_alice):\n",
    "#     sentences.extend(sent.text for sent in doc.sents)\n",
    "    \n",
    "# len(sentences), sentences[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HuggingFace's Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading a Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default-5f3dd563caa0ef2a\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset csv/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/dvgodoy/.cache/huggingface/datasets/csv/default-5f3dd563caa0ef2a/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', layout=Layout(width='20px'), max=1.0), HTML(value=''…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset csv downloaded and prepared to /home/dvgodoy/.cache/huggingface/datasets/csv/default-5f3dd563caa0ef2a/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0. Subsequent calls will reuse this data.\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset, Split\n",
    "\n",
    "dataset = load_dataset(path='csv', data_files=new_fnames, quotechar='\\\\', split=Split.TRAIN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'sentence': Value(dtype='string', id=None),\n",
       "  'source': Value(dtype='string', id=None)},\n",
       " 2,\n",
       " (3852, 2))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.features, dataset.num_columns, dataset.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sentence': 'There was nothing so VERY remarkable in that; nor did Alice think it so VERY much out of the way to hear the Rabbit say to itself, `Oh dear!',\n",
       " 'source': 'alice28-1476.txt'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alice28-1476.txt', 'alice28-1476.txt', 'alice28-1476.txt']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset['source'][:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alice28-1476.txt', 'wizoz10-1740.txt']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.unique('source')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29b4ed6e6f174d5ebe4ca9be7b911055",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=3852.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "def is_alice_label(row):\n",
    "    is_alice = int(row['source'] == 'alice28-1476.txt')\n",
    "    return {'labels': is_alice}\n",
    "\n",
    "dataset = dataset.map(is_alice_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'labels': 1,\n",
       " 'sentence': 'There was nothing so VERY remarkable in that; nor did Alice think it so VERY much out of the way to hear the Rabbit say to itself, `Oh dear!',\n",
       " 'source': 'alice28-1476.txt'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "shuffled_dataset = dataset.shuffle(seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['labels', 'sentence', 'source'],\n",
       "        num_rows: 3081\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['labels', 'sentence', 'source'],\n",
       "        num_rows: 771\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "split_dataset = shuffled_dataset.train_test_split(test_size=0.2)\n",
    "split_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset, test_dataset = split_dataset['train'], split_dataset['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Augmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install textattack\n",
    "# from textattack.augmentation import EmbeddingAugmenter\n",
    "# augmenter = EmbeddingAugmenter()\n",
    "# feynman = 'What I cannot create, I do not understand.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(5):\n",
    "#     print(augmenter.augment(feynman))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"I'm\", 'following', 'the', 'white', 'rabbit']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"I'm following the white rabbit\"\n",
    "tokens = sentence.split(' ')\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['follow', 'white', 'rabbit']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.parsing.preprocessing import *\n",
    "\n",
    "preprocess_string(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'm', 'following', 'the', 'white', 'rabbit']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filters = [lambda x: x.lower(), strip_tags, strip_punctuation, strip_multiple_whitespaces, strip_numeric]\n",
    "preprocess_string(sentence, filters=filters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['following', 'the', 'white', 'rabbit']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.utils import simple_preprocess\n",
    "\n",
    "tokens = simple_preprocess(sentence)\n",
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['and', 'so', 'far', 'as', 'they', 'knew', 'they', 'were', 'quite', 'right']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = train_dataset['sentence']\n",
    "tokens = [simple_preprocess(sent) for sent in sentences]\n",
    "tokens[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary(3704 unique tokens: ['and', 'as', 'far', 'knew', 'quite']...)\n"
     ]
    }
   ],
   "source": [
    "from gensim import corpora\n",
    "\n",
    "dictionary = corpora.Dictionary(tokens)\n",
    "print(dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3081"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary.num_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50802"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary.num_pos # processed words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'and': 0,\n",
       " 'as': 1,\n",
       " 'far': 2,\n",
       " 'knew': 3,\n",
       " 'quite': 4,\n",
       " 'right': 5,\n",
       " 'so': 6,\n",
       " 'they': 7,\n",
       " 'were': 8,\n",
       " 'beginning': 9,\n",
       " 'begun': 10,\n",
       " 'cried': 11,\n",
       " 'cry': 12,\n",
       " 'gryphon': 13,\n",
       " 'had': 14,\n",
       " 'it': 15,\n",
       " 'just': 16,\n",
       " 'mock': 17,\n",
       " 'of': 18,\n",
       " 'repeat': 19,\n",
       " 'the': 20,\n",
       " 'to': 21,\n",
       " 'trial': 22,\n",
       " 'turtle': 23,\n",
       " 'when': 24,\n",
       " 'across': 25,\n",
       " 'at': 26,\n",
       " 'broad': 27,\n",
       " 'but': 28,\n",
       " 'could': 29,\n",
       " 'deep': 30,\n",
       " 'he': 31,\n",
       " 'leap': 32,\n",
       " 'lion': 33,\n",
       " 'not': 34,\n",
       " 'once': 35,\n",
       " 'one': 36,\n",
       " 'that': 37,\n",
       " 'this': 38,\n",
       " 'was': 39,\n",
       " 'are': 40,\n",
       " 'do': 41,\n",
       " 'me': 42,\n",
       " 'seek': 43,\n",
       " 'who': 44,\n",
       " 'why': 45,\n",
       " 'you': 46,\n",
       " 'destroy': 47,\n",
       " 'for': 48,\n",
       " 'her': 49,\n",
       " 'mean': 50,\n",
       " 'perhaps': 51,\n",
       " 'said': 52,\n",
       " 'scarecrow': 53,\n",
       " 'we': 54,\n",
       " 'alice': 55,\n",
       " 'afterward': 56,\n",
       " 'city': 57,\n",
       " 'emerald': 58,\n",
       " 'gate': 59,\n",
       " 'into': 60,\n",
       " 'on': 61,\n",
       " 'passed': 62,\n",
       " 'through': 63,\n",
       " 'added': 64,\n",
       " 'care': 65,\n",
       " 'chin': 66,\n",
       " 'digging': 67,\n",
       " 'duchess': 68,\n",
       " 'is': 69,\n",
       " 'little': 70,\n",
       " 'means': 71,\n",
       " 'moral': 72,\n",
       " 'much': 73,\n",
       " 'same': 74,\n",
       " 'sense': 75,\n",
       " 'sharp': 76,\n",
       " 'she': 77,\n",
       " 'shoulder': 78,\n",
       " 'sounds': 79,\n",
       " 'take': 80,\n",
       " 'themselves': 81,\n",
       " 'thing': 82,\n",
       " 'will': 83,\n",
       " 'arms': 84,\n",
       " 'dorothy': 85,\n",
       " 'down': 86,\n",
       " 'held': 87,\n",
       " 'in': 88,\n",
       " 'middle': 89,\n",
       " 'raft': 90,\n",
       " 'sat': 91,\n",
       " 'toto': 92,\n",
       " 'about': 93,\n",
       " 'his': 94,\n",
       " 'toes': 95,\n",
       " 'both': 96,\n",
       " 'fury': 97,\n",
       " 'go': 98,\n",
       " 'house': 99,\n",
       " 'idea': 100,\n",
       " 'kept': 101,\n",
       " 'law': 102,\n",
       " 'let': 103,\n",
       " 'like': 104,\n",
       " 'met': 105,\n",
       " 'mouse': 106,\n",
       " 'prosecute': 107,\n",
       " 'puzzling': 108,\n",
       " 'something': 109,\n",
       " 'speaking': 110,\n",
       " 'tale': 111,\n",
       " 'us': 112,\n",
       " 'while': 113,\n",
       " 'hastily': 114,\n",
       " 'lory': 115,\n",
       " 'princess': 116,\n",
       " 'king': 117,\n",
       " 'may': 118,\n",
       " 'replied': 119,\n",
       " 'sit': 120,\n",
       " 'then': 121,\n",
       " 'clock': 122,\n",
       " 'day': 123,\n",
       " 'doesn': 124,\n",
       " 'month': 125,\n",
       " 'tell': 126,\n",
       " 'tells': 127,\n",
       " 'what': 128,\n",
       " 'answer': 129,\n",
       " 'no': 130,\n",
       " 'would': 131,\n",
       " 'anything': 132,\n",
       " 'basket': 133,\n",
       " 'cap': 134,\n",
       " 'carry': 135,\n",
       " 'charm': 136,\n",
       " 'did': 137,\n",
       " 'golden': 138,\n",
       " 'know': 139,\n",
       " 'made': 140,\n",
       " 'mind': 141,\n",
       " 'pretty': 142,\n",
       " 'saw': 143,\n",
       " 'sunbonnet': 144,\n",
       " 'up': 145,\n",
       " 'wear': 146,\n",
       " 'after': 147,\n",
       " 'air': 148,\n",
       " 'appearance': 149,\n",
       " 'away': 150,\n",
       " 'be': 151,\n",
       " 'being': 152,\n",
       " 'cat': 153,\n",
       " 'cheshire': 154,\n",
       " 'curious': 155,\n",
       " 'escape': 156,\n",
       " 'first': 157,\n",
       " 'get': 158,\n",
       " 'grin': 159,\n",
       " 'have': 160,\n",
       " 'herself': 161,\n",
       " 'looking': 162,\n",
       " 'minute': 163,\n",
       " 'noticed': 164,\n",
       " 'now': 165,\n",
       " 'or': 166,\n",
       " 'out': 167,\n",
       " 'puzzled': 168,\n",
       " 'seen': 169,\n",
       " 'shall': 170,\n",
       " 'some': 171,\n",
       " 'somebody': 172,\n",
       " 'talk': 173,\n",
       " 'two': 174,\n",
       " 'very': 175,\n",
       " 'watching': 176,\n",
       " 'way': 177,\n",
       " 'whether': 178,\n",
       " 'without': 179,\n",
       " 'wondering': 180,\n",
       " 'again': 181,\n",
       " 'find': 182,\n",
       " 'garden': 183,\n",
       " 'got': 184,\n",
       " 'grow': 185,\n",
       " 'lovely': 186,\n",
       " 'my': 187,\n",
       " 'second': 188,\n",
       " 'size': 189,\n",
       " 've': 190,\n",
       " 'wandered': 191,\n",
       " 'wood': 192,\n",
       " 'politely': 193,\n",
       " 'thank': 194,\n",
       " 'well': 195,\n",
       " 'pig': 196,\n",
       " 'corner': 197,\n",
       " 'couch': 198,\n",
       " 'hurt': 199,\n",
       " 'leg': 200,\n",
       " 'lying': 201,\n",
       " 'man': 202,\n",
       " 'all': 203,\n",
       " 'blue': 204,\n",
       " 'bodices': 205,\n",
       " 'breeches': 206,\n",
       " 'brightly': 207,\n",
       " 'buckles': 208,\n",
       " 'caps': 209,\n",
       " 'cheeks': 210,\n",
       " 'clowns': 211,\n",
       " 'colored': 212,\n",
       " 'crowns': 213,\n",
       " 'doublets': 214,\n",
       " 'dressed': 215,\n",
       " 'ermine': 216,\n",
       " 'frocks': 217,\n",
       " 'funny': 218,\n",
       " 'gold': 219,\n",
       " 'gorgeous': 220,\n",
       " 'gowns': 221,\n",
       " 'heads': 222,\n",
       " 'jeweled': 223,\n",
       " 'knee': 224,\n",
       " 'milkmaids': 225,\n",
       " 'most': 226,\n",
       " 'over': 227,\n",
       " 'pink': 228,\n",
       " 'pointed': 229,\n",
       " 'princes': 230,\n",
       " 'princesses': 231,\n",
       " 'purple': 232,\n",
       " 'red': 233,\n",
       " 'robes': 234,\n",
       " 'round': 235,\n",
       " 'ruffled': 236,\n",
       " 'satin': 237,\n",
       " 'shepherdesses': 238,\n",
       " 'shepherds': 239,\n",
       " 'shoes': 240,\n",
       " 'silver': 241,\n",
       " 'spots': 242,\n",
       " 'stripes': 243,\n",
       " 'tall': 244,\n",
       " 'their': 245,\n",
       " 'them': 246,\n",
       " 'there': 247,\n",
       " 'upon': 248,\n",
       " 'wearing': 249,\n",
       " 'with': 250,\n",
       " 'yellow': 251,\n",
       " 'axe': 252,\n",
       " 'chopped': 253,\n",
       " 'cleared': 254,\n",
       " 'entire': 255,\n",
       " 'party': 256,\n",
       " 'passage': 257,\n",
       " 'set': 258,\n",
       " 'soon': 259,\n",
       " 'tin': 260,\n",
       " 'woodman': 261,\n",
       " 'work': 262,\n",
       " 'courage': 263,\n",
       " 'give': 264,\n",
       " 'oz': 265,\n",
       " 'sort': 266,\n",
       " 'tomorrow': 267,\n",
       " 'alone': 268,\n",
       " 'shoulders': 269,\n",
       " 'where': 270,\n",
       " 'really': 271,\n",
       " 'began': 272,\n",
       " 'chance': 273,\n",
       " 'everything': 274,\n",
       " 'pass': 275,\n",
       " 'seemed': 276,\n",
       " 'time': 277,\n",
       " 'here': 278,\n",
       " 'too': 279,\n",
       " 'anger': 280,\n",
       " 'bank': 281,\n",
       " 'beat': 282,\n",
       " 'brushing': 283,\n",
       " 'came': 284,\n",
       " 'dead': 285,\n",
       " 'face': 286,\n",
       " 'fluttered': 287,\n",
       " 'flying': 288,\n",
       " 'found': 289,\n",
       " 'fright': 290,\n",
       " 'from': 291,\n",
       " 'gave': 292,\n",
       " 'gently': 293,\n",
       " 'half': 294,\n",
       " 'head': 295,\n",
       " 'lap': 296,\n",
       " 'leaves': 297,\n",
       " 'off': 298,\n",
       " 'pack': 299,\n",
       " 'rose': 300,\n",
       " 'scream': 301,\n",
       " 'sister': 302,\n",
       " 'trees': 303,\n",
       " 'tried': 304,\n",
       " 'whole': 305,\n",
       " 'attended': 306,\n",
       " 'exclaimed': 307,\n",
       " 'last': 308,\n",
       " 'remark': 309,\n",
       " 'vegetable': 310,\n",
       " 'beautiful': 311,\n",
       " 'body': 312,\n",
       " 'cut': 313,\n",
       " 'halves': 314,\n",
       " 'kill': 315,\n",
       " 'love': 316,\n",
       " 'maiden': 317,\n",
       " 'munchkin': 318,\n",
       " 'new': 319,\n",
       " 'slip': 320,\n",
       " 'splitting': 321,\n",
       " 'thought': 322,\n",
       " 'back': 323,\n",
       " 'hear': 324,\n",
       " 'left': 325,\n",
       " 'room': 326,\n",
       " 'sorrowfully': 327,\n",
       " 'throne': 328,\n",
       " 'waiting': 329,\n",
       " 'went': 330,\n",
       " 'hurried': 331,\n",
       " 'low': 332,\n",
       " 'rabbit': 333,\n",
       " 'tone': 334,\n",
       " 'hope': 335,\n",
       " 'won': 336,\n",
       " 'bigger': 337,\n",
       " 'crossed': 338,\n",
       " 'forward': 339,\n",
       " 'him': 340,\n",
       " 'kalidahs': 341,\n",
       " 'next': 342,\n",
       " 'only': 343,\n",
       " 'remembering': 344,\n",
       " 'rushed': 345,\n",
       " 'see': 346,\n",
       " 'seeing': 347,\n",
       " 'than': 348,\n",
       " 'tree': 349,\n",
       " 'turned': 350,\n",
       " 'brain': 351,\n",
       " 'going': 352,\n",
       " 'more': 353,\n",
       " 'splendid': 354,\n",
       " 'surely': 355,\n",
       " 'think': 356,\n",
       " 'thoughts': 357,\n",
       " 'turn': 358,\n",
       " 'should': 359,\n",
       " 'hanging': 360,\n",
       " 'shame': 361,\n",
       " 'bottom': 362,\n",
       " 'couldn': 363,\n",
       " 'living': 364,\n",
       " 'sea': 365,\n",
       " 'wanted': 366,\n",
       " 'by': 367,\n",
       " 'confusing': 368,\n",
       " 'ever': 369,\n",
       " 'heard': 370,\n",
       " 'people': 371,\n",
       " 'stop': 372,\n",
       " 'comrades': 373,\n",
       " 'fate': 374,\n",
       " 'sad': 375,\n",
       " 'stood': 376,\n",
       " 'thinking': 377,\n",
       " 'anxious': 378,\n",
       " 'been': 379,\n",
       " 'greatly': 380,\n",
       " 'heart': 381,\n",
       " 'interested': 382,\n",
       " 'story': 383,\n",
       " 'easily': 384,\n",
       " 'remarked': 385,\n",
       " 'demand': 386,\n",
       " 'gates': 387,\n",
       " 'great': 388,\n",
       " 'guardian': 389,\n",
       " 'strangers': 390,\n",
       " 'afraid': 391,\n",
       " 'forget': 392,\n",
       " 'makes': 393,\n",
       " 'unhappy': 394,\n",
       " 'unless': 395,\n",
       " 'before': 396,\n",
       " 'dear': 397,\n",
       " 'fit': 398,\n",
       " 'fits': 399,\n",
       " 'never': 400,\n",
       " 'feel': 401,\n",
       " 'girl': 402,\n",
       " 'how': 403,\n",
       " 'sorry': 404,\n",
       " 'truly': 405,\n",
       " 'understand': 406,\n",
       " 'consider': 407,\n",
       " 'jury': 408,\n",
       " 'verdict': 409,\n",
       " 'your': 410,\n",
       " 'asked': 411,\n",
       " 'offered': 412,\n",
       " 'piece': 413,\n",
       " 'refused': 414,\n",
       " 'any': 415,\n",
       " 'don': 416,\n",
       " 'longer': 417,\n",
       " 'stay': 418,\n",
       " 'sure': 419,\n",
       " 'want': 420,\n",
       " 'can': 421,\n",
       " 'fast': 422,\n",
       " 'hold': 423,\n",
       " 'if': 424,\n",
       " 'pull': 425,\n",
       " 'shore': 426,\n",
       " 'swim': 427,\n",
       " 'tail': 428,\n",
       " 'tip': 429,\n",
       " 'answered': 430,\n",
       " 'child': 431,\n",
       " 'indeed': 432,\n",
       " 'yes': 433,\n",
       " 'alarmed': 434,\n",
       " 'proposal': 435,\n",
       " 'rather': 436,\n",
       " 'am': 437,\n",
       " 'straw': 438,\n",
       " 'stuffed': 439,\n",
       " 'books': 440,\n",
       " 'flowers': 441,\n",
       " 'green': 442,\n",
       " 'row': 443,\n",
       " 'shelf': 444,\n",
       " 'windows': 445,\n",
       " 'cross': 446,\n",
       " 'due': 447,\n",
       " 'except': 448,\n",
       " 'must': 449,\n",
       " 'order': 450,\n",
       " 'other': 451,\n",
       " 'place': 452,\n",
       " 'side': 453,\n",
       " 'south': 454,\n",
       " 'strange': 455,\n",
       " 'unwise': 456,\n",
       " 'big': 457,\n",
       " 'fond': 458,\n",
       " 'glad': 459,\n",
       " 'grown': 460,\n",
       " 'rescued': 461,\n",
       " 'along': 462,\n",
       " 'door': 463,\n",
       " 'hole': 464,\n",
       " 'knelt': 465,\n",
       " 'larger': 466,\n",
       " 'led': 467,\n",
       " 'looked': 468,\n",
       " 'loveliest': 469,\n",
       " 'opened': 470,\n",
       " 'rat': 471,\n",
       " 'small': 472,\n",
       " 'animals': 473,\n",
       " 'because': 474,\n",
       " 'birds': 475,\n",
       " 'box': 476,\n",
       " 'creatures': 477,\n",
       " 'jurors': 478,\n",
       " 'obliged': 479,\n",
       " 'say': 480,\n",
       " 'suppose': 481,\n",
       " 'those': 482,\n",
       " 'twelve': 483,\n",
       " 'confused': 484,\n",
       " 'dormouse': 485,\n",
       " 'interrupting': 486,\n",
       " 'poor': 487,\n",
       " 'appeared': 488,\n",
       " 'deeply': 489,\n",
       " 'moment': 490,\n",
       " 'evidence': 491,\n",
       " 'feeling': 492,\n",
       " 'fumbled': 493,\n",
       " 'haven': 494,\n",
       " 'list': 495,\n",
       " 'watched': 496,\n",
       " 'white': 497,\n",
       " 'witness': 498,\n",
       " 'yet': 499,\n",
       " 'costume': 500,\n",
       " 'grandfather': 501,\n",
       " 'rich': 502,\n",
       " 'silk': 503,\n",
       " 'velvet': 504,\n",
       " 'players': 505,\n",
       " 'playing': 506,\n",
       " 'quarrelling': 507,\n",
       " 'queen': 508,\n",
       " 'shouting': 509,\n",
       " 'kick': 510,\n",
       " 'll': 511,\n",
       " 'stairs': 512,\n",
       " 'die': 513,\n",
       " 'leave': 514,\n",
       " 'advantage': 515,\n",
       " 'change': 516,\n",
       " 'deal': 517,\n",
       " 'good': 518,\n",
       " 'hare': 519,\n",
       " 'hatter': 520,\n",
       " 'jug': 521,\n",
       " 'march': 522,\n",
       " 'milk': 523,\n",
       " 'plate': 524,\n",
       " 'upset': 525,\n",
       " 'worse': 526,\n",
       " 'advisable': 527,\n",
       " 'atheling': 528,\n",
       " 'crown': 529,\n",
       " 'edgar': 530,\n",
       " 'hurriedly': 531,\n",
       " 'meet': 532,\n",
       " 'notice': 533,\n",
       " 'offer': 534,\n",
       " 'question': 535,\n",
       " 'william': 536,\n",
       " 'these': 537,\n",
       " 'dance': 538,\n",
       " 'join': 539,\n",
       " 'branch': 540,\n",
       " 'caught': 541,\n",
       " 'howled': 542,\n",
       " 'injury': 543,\n",
       " 'ran': 544,\n",
       " 'shaken': 545,\n",
       " 'under': 546,\n",
       " 'until': 547,\n",
       " 'bats': 548,\n",
       " 'cats': 549,\n",
       " 'dreamy': 550,\n",
       " 'eat': 551,\n",
       " 'saying': 552,\n",
       " 'sleepy': 553,\n",
       " 'already': 554,\n",
       " 'confusion': 555,\n",
       " 'executed': 556,\n",
       " 'game': 557,\n",
       " 'having': 558,\n",
       " 'look': 559,\n",
       " 'missed': 560,\n",
       " 'sentence': 561,\n",
       " 'such': 562,\n",
       " 'things': 563,\n",
       " 'three': 564,\n",
       " 'turns': 565,\n",
       " 'come': 566,\n",
       " 'even': 567,\n",
       " 'wicked': 568,\n",
       " 'witch': 569,\n",
       " 'beau': 570,\n",
       " 'oop': 571,\n",
       " 'ootiful': 572,\n",
       " 'soo': 573,\n",
       " 'done': 574,\n",
       " 'however': 575,\n",
       " 'wonder': 576,\n",
       " 'helped': 577,\n",
       " 'put': 578,\n",
       " 'pole': 579,\n",
       " 'river': 580,\n",
       " 'stuck': 581,\n",
       " 'use': 582,\n",
       " 'downwards': 583,\n",
       " 'haste': 584,\n",
       " 'its': 585,\n",
       " 'lizard': 586,\n",
       " 'melancholy': 587,\n",
       " 'move': 588,\n",
       " 'unable': 589,\n",
       " 'waving': 590,\n",
       " 'anybody': 591,\n",
       " 'aunt': 592,\n",
       " 'dried': 593,\n",
       " 'em': 594,\n",
       " 'eyes': 595,\n",
       " 'try': 596,\n",
       " 'clearer': 597,\n",
       " 'nothing': 598,\n",
       " 'beast': 599,\n",
       " 'demanded': 600,\n",
       " 'ease': 601,\n",
       " 'edge': 602,\n",
       " 'farther': 603,\n",
       " 'four': 604,\n",
       " 'travelers': 605,\n",
       " 'walked': 606,\n",
       " 'probably': 607,\n",
       " 'cheers': 608,\n",
       " 'journey': 609,\n",
       " 'many': 610,\n",
       " 'prepared': 611,\n",
       " 'started': 612,\n",
       " 'winkies': 613,\n",
       " 'wishes': 614,\n",
       " 'loudly': 615,\n",
       " 'rate': 616,\n",
       " 'matter': 617,\n",
       " 'opinion': 618,\n",
       " 'help': 619,\n",
       " 'mad': 620,\n",
       " 'oh': 621,\n",
       " 're': 622,\n",
       " 'flat': 623,\n",
       " 'hill': 624,\n",
       " 'lightning': 625,\n",
       " 'neck': 626,\n",
       " 'quick': 627,\n",
       " 'sent': 628,\n",
       " 'shot': 629,\n",
       " 'stretched': 630,\n",
       " 'struck': 631,\n",
       " 'top': 632,\n",
       " 'tumbling': 633,\n",
       " 'bees': 634,\n",
       " 'black': 635,\n",
       " 'buzzing': 636,\n",
       " 'forthwith': 637,\n",
       " 'swarm': 638,\n",
       " 'toward': 639,\n",
       " 'explain': 640,\n",
       " 'hard': 641,\n",
       " 'word': 642,\n",
       " 'bye': 643,\n",
       " 'country': 644,\n",
       " 'dangerous': 645,\n",
       " 'end': 646,\n",
       " 'places': 647,\n",
       " 'pleasant': 648,\n",
       " 'reach': 649,\n",
       " 'rough': 650,\n",
       " 'angry': 651,\n",
       " 'became': 652,\n",
       " 'caused': 653,\n",
       " 'dashed': 654,\n",
       " 'echoed': 655,\n",
       " 'giving': 656,\n",
       " 'laughter': 657,\n",
       " 'loud': 658,\n",
       " 'mishap': 659,\n",
       " 'roar': 660,\n",
       " 'thunder': 661,\n",
       " 'another': 662,\n",
       " 'fill': 663,\n",
       " 'float': 664,\n",
       " 'hot': 665,\n",
       " 'make': 666,\n",
       " 'which': 667,\n",
       " 'nor': 668,\n",
       " 'allow': 669,\n",
       " 'friend': 670,\n",
       " 'introduce': 671,\n",
       " 'mine': 672,\n",
       " 'bring': 673,\n",
       " 'each': 674,\n",
       " 'long': 675,\n",
       " 'possible': 676,\n",
       " 'send': 677,\n",
       " 'string': 678,\n",
       " 'sh': 679,\n",
       " 'sing': 680,\n",
       " 'calling': 681,\n",
       " 'doing': 682,\n",
       " 'eating': 683,\n",
       " 'forest': 684,\n",
       " 'free': 685,\n",
       " 'fruit': 686,\n",
       " 'happily': 687,\n",
       " 'leader': 688,\n",
       " 'master': 689,\n",
       " 'nuts': 690,\n",
       " 'pleased': 691,\n",
       " 'believe': 692,\n",
       " 'best': 693,\n",
       " 'desert': 694,\n",
       " 'followed': 695,\n",
       " 'others': 696,\n",
       " 'turning': 697,\n",
       " 'case': 698,\n",
       " 'peep': 699,\n",
       " 'sleep': 700,\n",
       " 'supper': 701,\n",
       " 'taking': 702,\n",
       " 'woman': 703,\n",
       " 'almost': 704,\n",
       " 'brains': 705,\n",
       " 'instead': 706,\n",
       " 'suspect': 707,\n",
       " 'danger': 708,\n",
       " 'faces': 709,\n",
       " 'didn': 710,\n",
       " 'knave': 711,\n",
       " 'majesty': 712,\n",
       " 'name': 713,\n",
       " 'please': 714,\n",
       " 'prove': 715,\n",
       " 'signed': 716,\n",
       " 'write': 717,\n",
       " 'beg': 718,\n",
       " 'hearted': 719,\n",
       " 'kindly': 720,\n",
       " 'ladies': 721,\n",
       " 'lady': 722,\n",
       " 'natural': 723,\n",
       " 'quietly': 724,\n",
       " 'arm': 725,\n",
       " 'carried': 726,\n",
       " 'cornfield': 727,\n",
       " 'farmer': 728,\n",
       " 'stick': 729,\n",
       " 'around': 730,\n",
       " 'besides': 731,\n",
       " 'dare': 732,\n",
       " 'direction': 733,\n",
       " 'extend': 734,\n",
       " 'fear': 735,\n",
       " 'getting': 736,\n",
       " 'lost': 737,\n",
       " 'pity': 738,\n",
       " 'respect': 739,\n",
       " 'treated': 740,\n",
       " 'course': 741,\n",
       " 'fish': 742,\n",
       " 'porpoise': 743,\n",
       " 'told': 744,\n",
       " 'pardon': 745,\n",
       " 'every': 746,\n",
       " 'everyone': 747,\n",
       " 'riding': 748,\n",
       " 'rising': 749,\n",
       " 'shouted': 750,\n",
       " 'sky': 751,\n",
       " 'upward': 752,\n",
       " 'wizard': 753,\n",
       " 'blacking': 754,\n",
       " 'crows': 755,\n",
       " 'enough': 756,\n",
       " 'fellow': 757,\n",
       " 'scare': 758,\n",
       " 'hardly': 759,\n",
       " 'person': 760,\n",
       " 'respectable': 761,\n",
       " 'agree': 762,\n",
       " 'large': 763,\n",
       " 'mustard': 764,\n",
       " 'near': 765,\n",
       " 'ready': 766,\n",
       " 'conqueror': 767,\n",
       " 'daresay': 768,\n",
       " 'english': 769,\n",
       " 'french': 770,\n",
       " 'croquet': 771,\n",
       " 'play': 772,\n",
       " 'bad': 773,\n",
       " 'breaking': 774,\n",
       " 'church': 775,\n",
       " 'cow': 776,\n",
       " 'harm': 777,\n",
       " 'lucky': 778,\n",
       " 'has': 779,\n",
       " 'morning': 780,\n",
       " 'soldier': 781,\n",
       " 'whiskers': 782,\n",
       " 'clumsy': 783,\n",
       " 'dropped': 784,\n",
       " 'hands': 785,\n",
       " 'padded': 786,\n",
       " 'desire': 787,\n",
       " 'mercy': 788,\n",
       " 'promises': 789,\n",
       " 'roll': 790,\n",
       " 'breathe': 791,\n",
       " 'might': 792,\n",
       " 'talking': 793,\n",
       " 'few': 794,\n",
       " 'kansas': 795,\n",
       " 'wants': 796,\n",
       " 'above': 797,\n",
       " 'clouds': 798,\n",
       " 'current': 799,\n",
       " 'miles': 800,\n",
       " 'hurry': 801,\n",
       " 'interrupted': 802,\n",
       " 'changed': 803,\n",
       " 'mabel': 804,\n",
       " 'behind': 805,\n",
       " 'ears': 806,\n",
       " 'groan': 807,\n",
       " 'reached': 808,\n",
       " 'sound': 809,\n",
       " 'life': 810,\n",
       " 'saving': 811,\n",
       " 'continued': 812,\n",
       " 'secret': 813,\n",
       " 'clinging': 814,\n",
       " 'swept': 815,\n",
       " 'center': 816,\n",
       " 'exactly': 817,\n",
       " 'ruled': 818,\n",
       " 'bill': 819,\n",
       " 'bottle': 820,\n",
       " 'guinea': 821,\n",
       " 'pigs': 822,\n",
       " 'geography': 823,\n",
       " 'multiplication': 824,\n",
       " 'signify': 825,\n",
       " 'table': 826,\n",
       " 'hate': 827,\n",
       " 'history': 828,\n",
       " 'offended': 829,\n",
       " 'promised': 830,\n",
       " 'whisper': 831,\n",
       " 'sadly': 832,\n",
       " 'evening': 833,\n",
       " 'night': 834,\n",
       " 'rest': 835,\n",
       " 'tired': 836,\n",
       " 'walk': 837,\n",
       " 'always': 838,\n",
       " 'blew': 839,\n",
       " 'given': 840,\n",
       " 'mice': 841,\n",
       " 'since': 842,\n",
       " 'whistle': 843,\n",
       " 'balloon': 844,\n",
       " 'picked': 845,\n",
       " 'towards': 846,\n",
       " 'chattering': 847,\n",
       " 'crowd': 848,\n",
       " 'dark': 849,\n",
       " 'immense': 850,\n",
       " 'laughing': 851,\n",
       " 'monkeys': 852,\n",
       " 'pair': 853,\n",
       " 'powerful': 854,\n",
       " 'rushing': 855,\n",
       " 'show': 856,\n",
       " 'sun': 857,\n",
       " 'surrounded': 858,\n",
       " 'wings': 859,\n",
       " 'explained': 860,\n",
       " 'also': 861,\n",
       " 'an': 862,\n",
       " 'belonged': 863,\n",
       " 'clothes': 864,\n",
       " 'faded': 865,\n",
       " 'figure': 866,\n",
       " 'hat': 867,\n",
       " 'old': 868,\n",
       " 'perched': 869,\n",
       " 'suit': 870,\n",
       " 'worn': 871,\n",
       " 'encouraging': 872,\n",
       " 'wine': 873,\n",
       " 'hush': 874,\n",
       " 'able': 875,\n",
       " 'bar': 876,\n",
       " 'fell': 877,\n",
       " 'floor': 878,\n",
       " 'full': 879,\n",
       " 'length': 880,\n",
       " 'stumbled': 881,\n",
       " 'banks': 882,\n",
       " 'between': 883,\n",
       " 'brook': 884,\n",
       " 'dry': 885,\n",
       " 'grateful': 886,\n",
       " 'gray': 887,\n",
       " 'lived': 888,\n",
       " 'murmuring': 889,\n",
       " 'prairies': 890,\n",
       " 'sparkling': 891,\n",
       " 'voice': 892,\n",
       " 'ask': 893,\n",
       " 'forms': 894,\n",
       " 'frighten': 895,\n",
       " 'grant': 896,\n",
       " 'himself': 897,\n",
       " 'loudest': 898,\n",
       " 'wondered': 899,\n",
       " 'spectacles': 900,\n",
       " 'unlocked': 901,\n",
       " 'speak': 902,\n",
       " 'choked': 903,\n",
       " 'sobs': 904,\n",
       " 'thick': 905,\n",
       " 'traveled': 906,\n",
       " 'brutes': 907,\n",
       " 'carrying': 908,\n",
       " 'crash': 909,\n",
       " 'gulf': 910,\n",
       " 'nearly': 911,\n",
       " 'pieces': 912,\n",
       " 'rocks': 913,\n",
       " 'snarling': 914,\n",
       " 'ugly': 915,\n",
       " 'sight': 916,\n",
       " 'battered': 917,\n",
       " 'dented': 918,\n",
       " 'distance': 919,\n",
       " 'lay': 920,\n",
       " 'neither': 921,\n",
       " 'beside': 922,\n",
       " 'fight': 923,\n",
       " 'dwelt': 924,\n",
       " 'east': 925,\n",
       " 'killed': 926,\n",
       " 'land': 927,\n",
       " 'lives': 928,\n",
       " 'west': 929,\n",
       " 'witches': 930,\n",
       " 'growing': 931,\n",
       " 'meekly': 932,\n",
       " 'angrily': 933,\n",
       " 'beasts': 934,\n",
       " 'keep': 935,\n",
       " 'welcome': 936,\n",
       " 'wild': 937,\n",
       " 'elephant': 938,\n",
       " 'strong': 939,\n",
       " 'weight': 940,\n",
       " 'alas': 941,\n",
       " 'speed': 942,\n",
       " 'beheaded': 943,\n",
       " 'shan': 944,\n",
       " 'grinned': 945,\n",
       " 'flew': 946,\n",
       " 'forehead': 947,\n",
       " 'grinning': 948,\n",
       " 'hairy': 949,\n",
       " 'kiss': 950,\n",
       " 'mark': 951,\n",
       " 'motioning': 952,\n",
       " 'short': 953,\n",
       " 'stopped': 954,\n",
       " 'terribly': 955,\n",
       " 'touch': 956,\n",
       " 'winged': 957,\n",
       " 'ladder': 958,\n",
       " 'waked': 959,\n",
       " 'bells': 960,\n",
       " 'brims': 961,\n",
       " 'foot': 962,\n",
       " 'hats': 963,\n",
       " 'moved': 964,\n",
       " 'point': 965,\n",
       " 'sweetly': 966,\n",
       " 'tinkled': 967,\n",
       " 'wore': 968,\n",
       " 'built': 969,\n",
       " 'myself': 970,\n",
       " 'palace': 971,\n",
       " 'shut': 972,\n",
       " 'trap': 973,\n",
       " 'destroyed': 974,\n",
       " 'carefully': 975,\n",
       " 'castle': 976,\n",
       " 'doorstep': 977,\n",
       " 'front': 978,\n",
       " 'lifted': 979,\n",
       " 'swiftly': 980,\n",
       " 'clown': 981,\n",
       " 'completely': 982,\n",
       " 'covered': 983,\n",
       " 'cracks': 984,\n",
       " 'jolly': 985,\n",
       " 'mended': 986,\n",
       " 'plainly': 987,\n",
       " 'running': 988,\n",
       " 'showing': 989,\n",
       " 'spite': 990,\n",
       " 'walking': 991,\n",
       " 'engaged': 992,\n",
       " 'sitting': 993,\n",
       " 'window': 994,\n",
       " 'ear': 995,\n",
       " 'holding': 996,\n",
       " 'pocket': 997,\n",
       " 'shaking': 998,\n",
       " 'taken': 999,\n",
       " ...}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary.token2id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['and', 'as', 'far', 'knew', 'quite']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = list(dictionary.token2id.keys())\n",
    "vocab[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 2024,\n",
       " 6: 362,\n",
       " 2: 29,\n",
       " 1: 473,\n",
       " 7: 443,\n",
       " 3: 36,\n",
       " 8: 231,\n",
       " 4: 65,\n",
       " 5: 45,\n",
       " 11: 36,\n",
       " 20: 3725,\n",
       " 13: 46,\n",
       " 17: 46,\n",
       " 23: 49,\n",
       " 14: 360,\n",
       " 16: 72,\n",
       " 10: 8,\n",
       " 21: 1481,\n",
       " 19: 5,\n",
       " 15: 811,\n",
       " 24: 172,\n",
       " 12: 12,\n",
       " 18: 1107,\n",
       " 22: 9,\n",
       " 9: 14,\n",
       " 28: 369,\n",
       " 38: 230,\n",
       " 36: 189,\n",
       " 39: 693,\n",
       " 27: 5,\n",
       " 30: 12,\n",
       " 37: 550,\n",
       " 33: 140,\n",
       " 26: 383,\n",
       " 35: 77,\n",
       " 31: 466,\n",
       " 29: 161,\n",
       " 34: 297,\n",
       " 32: 2,\n",
       " 25: 15,\n",
       " 44: 144,\n",
       " 40: 143,\n",
       " 46: 706,\n",
       " 45: 67,\n",
       " 41: 160,\n",
       " 43: 6,\n",
       " 42: 220,\n",
       " 51: 22,\n",
       " 52: 613,\n",
       " 53: 173,\n",
       " 48: 390,\n",
       " 54: 154,\n",
       " 50: 10,\n",
       " 47: 6,\n",
       " 49: 502,\n",
       " 55: 305,\n",
       " 56: 3,\n",
       " 62: 20,\n",
       " 61: 278,\n",
       " 63: 56,\n",
       " 59: 13,\n",
       " 60: 121,\n",
       " 58: 52,\n",
       " 57: 66,\n",
       " 71: 9,\n",
       " 73: 82,\n",
       " 74: 33,\n",
       " 82: 59,\n",
       " 68: 34,\n",
       " 67: 4,\n",
       " 76: 17,\n",
       " 70: 199,\n",
       " 66: 8,\n",
       " 78: 4,\n",
       " 77: 749,\n",
       " 64: 27,\n",
       " 72: 7,\n",
       " 69: 292,\n",
       " 80: 37,\n",
       " 65: 12,\n",
       " 75: 4,\n",
       " 79: 3,\n",
       " 83: 158,\n",
       " 81: 10,\n",
       " 85: 293,\n",
       " 91: 37,\n",
       " 86: 145,\n",
       " 88: 681,\n",
       " 89: 19,\n",
       " 90: 12,\n",
       " 87: 7,\n",
       " 92: 72,\n",
       " 84: 34,\n",
       " 93: 104,\n",
       " 94: 254,\n",
       " 95: 5,\n",
       " 101: 18,\n",
       " 108: 4,\n",
       " 113: 50,\n",
       " 106: 40,\n",
       " 110: 3,\n",
       " 100: 13,\n",
       " 111: 3,\n",
       " 109: 22,\n",
       " 104: 122,\n",
       " 97: 3,\n",
       " 105: 11,\n",
       " 99: 43,\n",
       " 103: 43,\n",
       " 112: 66,\n",
       " 96: 24,\n",
       " 98: 98,\n",
       " 102: 2,\n",
       " 107: 1,\n",
       " 115: 7,\n",
       " 114: 13,\n",
       " 116: 9,\n",
       " 121: 184,\n",
       " 118: 28,\n",
       " 120: 10,\n",
       " 117: 66,\n",
       " 119: 68,\n",
       " 127: 2,\n",
       " 123: 56,\n",
       " 125: 2,\n",
       " 124: 18,\n",
       " 126: 49,\n",
       " 128: 193,\n",
       " 122: 6,\n",
       " 131: 163,\n",
       " 129: 14,\n",
       " 130: 165,\n",
       " 137: 117,\n",
       " 139: 109,\n",
       " 132: 35,\n",
       " 136: 14,\n",
       " 138: 30,\n",
       " 134: 26,\n",
       " 143: 55,\n",
       " 142: 24,\n",
       " 140: 95,\n",
       " 145: 165,\n",
       " 141: 24,\n",
       " 146: 8,\n",
       " 135: 22,\n",
       " 144: 2,\n",
       " 133: 20,\n",
       " 162: 36,\n",
       " 171: 73,\n",
       " 177: 99,\n",
       " 156: 6,\n",
       " 180: 7,\n",
       " 178: 15,\n",
       " 158: 107,\n",
       " 150: 68,\n",
       " 179: 30,\n",
       " 152: 34,\n",
       " 169: 24,\n",
       " 164: 11,\n",
       " 155: 13,\n",
       " 149: 1,\n",
       " 148: 44,\n",
       " 168: 12,\n",
       " 175: 175,\n",
       " 157: 87,\n",
       " 147: 93,\n",
       " 176: 3,\n",
       " 163: 23,\n",
       " 166: 102,\n",
       " 174: 48,\n",
       " 167: 182,\n",
       " 151: 260,\n",
       " 159: 3,\n",
       " 161: 80,\n",
       " 154: 6,\n",
       " 153: 36,\n",
       " 165: 119,\n",
       " 170: 85,\n",
       " 160: 196,\n",
       " 172: 7,\n",
       " 173: 14,\n",
       " 190: 34,\n",
       " 184: 52,\n",
       " 191: 2,\n",
       " 192: 13,\n",
       " 185: 11,\n",
       " 187: 198,\n",
       " 189: 13,\n",
       " 181: 134,\n",
       " 188: 8,\n",
       " 182: 46,\n",
       " 186: 17,\n",
       " 183: 15,\n",
       " 195: 99,\n",
       " 194: 11,\n",
       " 193: 8,\n",
       " 196: 10,\n",
       " 202: 58,\n",
       " 199: 21,\n",
       " 200: 9,\n",
       " 201: 19,\n",
       " 198: 1,\n",
       " 197: 14,\n",
       " 247: 173,\n",
       " 225: 1,\n",
       " 238: 1,\n",
       " 250: 364,\n",
       " 207: 5,\n",
       " 212: 2,\n",
       " 205: 1,\n",
       " 242: 2,\n",
       " 203: 334,\n",
       " 227: 96,\n",
       " 245: 121,\n",
       " 221: 3,\n",
       " 231: 1,\n",
       " 226: 19,\n",
       " 220: 3,\n",
       " 217: 1,\n",
       " 241: 25,\n",
       " 219: 4,\n",
       " 232: 4,\n",
       " 239: 1,\n",
       " 215: 15,\n",
       " 224: 6,\n",
       " 206: 1,\n",
       " 228: 3,\n",
       " 251: 25,\n",
       " 204: 17,\n",
       " 243: 1,\n",
       " 246: 258,\n",
       " 208: 1,\n",
       " 240: 37,\n",
       " 230: 1,\n",
       " 223: 1,\n",
       " 213: 1,\n",
       " 248: 93,\n",
       " 222: 17,\n",
       " 249: 3,\n",
       " 216: 1,\n",
       " 234: 1,\n",
       " 237: 3,\n",
       " 214: 1,\n",
       " 218: 4,\n",
       " 211: 2,\n",
       " 236: 1,\n",
       " 235: 38,\n",
       " 233: 9,\n",
       " 210: 4,\n",
       " 244: 7,\n",
       " 229: 6,\n",
       " 209: 1,\n",
       " 260: 115,\n",
       " 261: 140,\n",
       " 258: 29,\n",
       " 262: 22,\n",
       " 252: 25,\n",
       " 253: 8,\n",
       " 259: 62,\n",
       " 254: 1,\n",
       " 257: 4,\n",
       " 255: 2,\n",
       " 256: 13,\n",
       " 264: 43,\n",
       " 266: 21,\n",
       " 263: 24,\n",
       " 267: 7,\n",
       " 265: 131,\n",
       " 268: 7,\n",
       " 270: 81,\n",
       " 269: 5,\n",
       " 271: 27,\n",
       " 276: 45,\n",
       " 273: 6,\n",
       " 272: 69,\n",
       " 274: 28,\n",
       " 275: 12,\n",
       " 277: 84,\n",
       " 278: 77,\n",
       " 279: 46,\n",
       " 305: 12,\n",
       " 299: 6,\n",
       " 300: 8,\n",
       " 284: 124,\n",
       " 288: 8,\n",
       " 292: 34,\n",
       " 301: 1,\n",
       " 294: 20,\n",
       " 290: 3,\n",
       " 280: 1,\n",
       " 304: 25,\n",
       " 282: 9,\n",
       " 298: 88,\n",
       " 289: 63,\n",
       " 281: 5,\n",
       " 295: 119,\n",
       " 296: 3,\n",
       " 302: 5,\n",
       " 293: 8,\n",
       " 283: 1,\n",
       " 285: 11,\n",
       " 297: 9,\n",
       " 287: 3,\n",
       " 291: 124,\n",
       " 303: 35,\n",
       " 286: 34,\n",
       " 307: 13,\n",
       " 306: 2,\n",
       " 308: 52,\n",
       " 309: 9,\n",
       " 310: 1,\n",
       " 322: 104,\n",
       " 319: 19,\n",
       " 315: 12,\n",
       " 316: 11,\n",
       " 311: 43,\n",
       " 318: 9,\n",
       " 317: 2,\n",
       " 320: 2,\n",
       " 313: 17,\n",
       " 312: 19,\n",
       " 321: 1,\n",
       " 314: 1,\n",
       " 327: 5,\n",
       " 325: 32,\n",
       " 328: 26,\n",
       " 326: 63,\n",
       " 330: 96,\n",
       " 323: 108,\n",
       " 329: 14,\n",
       " 324: 17,\n",
       " 333: 39,\n",
       " 332: 20,\n",
       " 331: 9,\n",
       " 334: 30,\n",
       " 335: 10,\n",
       " 336: 26,\n",
       " 347: 10,\n",
       " 337: 6,\n",
       " 348: 58,\n",
       " 344: 2,\n",
       " 343: 76,\n",
       " 340: 179,\n",
       " 341: 7,\n",
       " 345: 4,\n",
       " 339: 7,\n",
       " 338: 5,\n",
       " 349: 32,\n",
       " 350: 28,\n",
       " 346: 125,\n",
       " 342: 52,\n",
       " 355: 10,\n",
       " 356: 72,\n",
       " 353: 71,\n",
       " 354: 2,\n",
       " 357: 2,\n",
       " 351: 3,\n",
       " 352: 40,\n",
       " 358: 10,\n",
       " 359: 71,\n",
       " 360: 7,\n",
       " 361: 2,\n",
       " 363: 10,\n",
       " 366: 12,\n",
       " 364: 11,\n",
       " 362: 11,\n",
       " 365: 12,\n",
       " 367: 121,\n",
       " 368: 2,\n",
       " 369: 40,\n",
       " 370: 42,\n",
       " 372: 9,\n",
       " 371: 61,\n",
       " 376: 34,\n",
       " 375: 7,\n",
       " 374: 1,\n",
       " 373: 4,\n",
       " 377: 16,\n",
       " 379: 80,\n",
       " 380: 12,\n",
       " 382: 4,\n",
       " 383: 15,\n",
       " 378: 10,\n",
       " 381: 56,\n",
       " 384: 12,\n",
       " 385: 19,\n",
       " 390: 11,\n",
       " 389: 11,\n",
       " 387: 12,\n",
       " 386: 2,\n",
       " 388: 144,\n",
       " 394: 10,\n",
       " 395: 8,\n",
       " 393: 13,\n",
       " 392: 3,\n",
       " 391: 29,\n",
       " 396: 83,\n",
       " 398: 4,\n",
       " 400: 90,\n",
       " 399: 1,\n",
       " 397: 36,\n",
       " 406: 12,\n",
       " 403: 105,\n",
       " 401: 14,\n",
       " 402: 82,\n",
       " 405: 3,\n",
       " 404: 12,\n",
       " 407: 6,\n",
       " 410: 102,\n",
       " 409: 4,\n",
       " 408: 17,\n",
       " 411: 112,\n",
       " 412: 2,\n",
       " 413: 8,\n",
       " 414: 2,\n",
       " 419: 40,\n",
       " 416: 73,\n",
       " 420: 19,\n",
       " 418: 8,\n",
       " 415: 64,\n",
       " 417: 9,\n",
       " 421: 111,\n",
       " 427: 6,\n",
       " 426: 6,\n",
       " 425: 3,\n",
       " 424: 179,\n",
       " 423: 14,\n",
       " 422: 19,\n",
       " 429: 3,\n",
       " 428: 13,\n",
       " 433: 25,\n",
       " 432: 34,\n",
       " 430: 56,\n",
       " 431: 21,\n",
       " 436: 27,\n",
       " 434: 1,\n",
       " 435: 1,\n",
       " 437: 89,\n",
       " 439: 24,\n",
       " 438: 25,\n",
       " 442: 97,\n",
       " 441: 17,\n",
       " 445: 1,\n",
       " 444: 4,\n",
       " 443: 7,\n",
       " 440: 3,\n",
       " 449: 95,\n",
       " 446: 14,\n",
       " 455: 19,\n",
       " 452: 25,\n",
       " 450: 6,\n",
       " 451: 72,\n",
       " 453: 34,\n",
       " 456: 1,\n",
       " 448: 12,\n",
       " 447: 2,\n",
       " 454: 14,\n",
       " 460: 9,\n",
       " 458: 6,\n",
       " 457: 37,\n",
       " 459: 23,\n",
       " 461: 1,\n",
       " 470: 18,\n",
       " 463: 47,\n",
       " 467: 14,\n",
       " 472: 33,\n",
       " 466: 8,\n",
       " 471: 1,\n",
       " 464: 13,\n",
       " 465: 1,\n",
       " 468: 84,\n",
       " 462: 31,\n",
       " 469: 1,\n",
       " 476: 12,\n",
       " 482: 20,\n",
       " 483: 4,\n",
       " 477: 10,\n",
       " 479: 6,\n",
       " 480: 51,\n",
       " 474: 24,\n",
       " 473: 10,\n",
       " 475: 19,\n",
       " 481: 19,\n",
       " 478: 3,\n",
       " 484: 4,\n",
       " 487: 29,\n",
       " 485: 33,\n",
       " 486: 2,\n",
       " 488: 9,\n",
       " 489: 4,\n",
       " 490: 34,\n",
       " 496: 6,\n",
       " 497: 34,\n",
       " 493: 1,\n",
       " 495: 3,\n",
       " 492: 8,\n",
       " 498: 7,\n",
       " 494: 7,\n",
       " 491: 4,\n",
       " 499: 22,\n",
       " 502: 8,\n",
       " 500: 1,\n",
       " 503: 12,\n",
       " 504: 5,\n",
       " 501: 5,\n",
       " 506: 2,\n",
       " 508: 79,\n",
       " 507: 2,\n",
       " 505: 3,\n",
       " 509: 2,\n",
       " 511: 56,\n",
       " 510: 4,\n",
       " 512: 2,\n",
       " 514: 12,\n",
       " 513: 2,\n",
       " 520: 41,\n",
       " 515: 2,\n",
       " 516: 11,\n",
       " 518: 102,\n",
       " 517: 12,\n",
       " 526: 5,\n",
       " 522: 29,\n",
       " 519: 26,\n",
       " 525: 2,\n",
       " 523: 3,\n",
       " 521: 1,\n",
       " 524: 4,\n",
       " 533: 4,\n",
       " 535: 15,\n",
       " 531: 3,\n",
       " 527: 2,\n",
       " 530: 1,\n",
       " 528: 1,\n",
       " 532: 3,\n",
       " 536: 7,\n",
       " 534: 2,\n",
       " 529: 3,\n",
       " 537: 44,\n",
       " 539: 10,\n",
       " 538: 12,\n",
       " 544: 30,\n",
       " 546: 23,\n",
       " 543: 1,\n",
       " 541: 10,\n",
       " 540: 3,\n",
       " 545: 2,\n",
       " 547: 54,\n",
       " 542: 3,\n",
       " 553: 3,\n",
       " 552: 22,\n",
       " 550: 1,\n",
       " 549: 7,\n",
       " 551: 23,\n",
       " 548: 3,\n",
       " 554: 3,\n",
       " 561: 4,\n",
       " 564: 47,\n",
       " 556: 4,\n",
       " 558: 21,\n",
       " 560: 1,\n",
       " 565: 3,\n",
       " 559: 39,\n",
       " 563: 29,\n",
       " 557: 8,\n",
       " 562: 43,\n",
       " 555: 4,\n",
       " 566: 95,\n",
       " 567: 34,\n",
       " 568: 63,\n",
       " 569: 106,\n",
       " 570: 3,\n",
       " 572: 3,\n",
       " 573: 5,\n",
       " 571: 5,\n",
       " 575: 30,\n",
       " 574: 28,\n",
       " 576: 21,\n",
       " 577: 7,\n",
       " 578: 59,\n",
       " 582: 30,\n",
       " 581: 6,\n",
       " 579: 13,\n",
       " 580: 19,\n",
       " 584: 1,\n",
       " 586: 6,\n",
       " 583: 1,\n",
       " 590: 5,\n",
       " 585: 66,\n",
       " 587: 4,\n",
       " 589: 3,\n",
       " 588: 9,\n",
       " 593: 9,\n",
       " 595: 52,\n",
       " 596: 23,\n",
       " 591: 2,\n",
       " 592: 24,\n",
       " 594: 24,\n",
       " 598: 54,\n",
       " 597: 1,\n",
       " 600: 2,\n",
       " 599: 19,\n",
       " 604: 22,\n",
       " 605: 21,\n",
       " 606: 49,\n",
       " 601: 2,\n",
       " 603: 14,\n",
       " 602: 9,\n",
       " 607: 4,\n",
       " 611: 1,\n",
       " 609: 27,\n",
       " 612: 22,\n",
       " 613: 23,\n",
       " 608: 1,\n",
       " 610: 49,\n",
       " 614: 5,\n",
       " 615: 6,\n",
       " 616: 10,\n",
       " 617: 15,\n",
       " 618: 2,\n",
       " 621: 67,\n",
       " 619: 45,\n",
       " 622: 34,\n",
       " 620: 14,\n",
       " 627: 7,\n",
       " 625: 1,\n",
       " 629: 4,\n",
       " 626: 18,\n",
       " 630: 7,\n",
       " 632: 21,\n",
       " 623: 5,\n",
       " 631: 6,\n",
       " 628: 12,\n",
       " 633: 2,\n",
       " 624: 8,\n",
       " 637: 1,\n",
       " 636: 1,\n",
       " 638: 1,\n",
       " 635: 7,\n",
       " 634: 7,\n",
       " 639: 19,\n",
       " 641: 25,\n",
       " 642: 13,\n",
       " 640: 9,\n",
       " 643: 21,\n",
       " 644: 55,\n",
       " 648: 4,\n",
       " 650: 5,\n",
       " 645: 3,\n",
       " 647: 5,\n",
       " 649: 8,\n",
       " 646: 33,\n",
       " 652: 18,\n",
       " 651: 14,\n",
       " 657: 2,\n",
       " 653: 3,\n",
       " 659: 3,\n",
       " 656: 5,\n",
       " 658: 14,\n",
       " 660: 7,\n",
       " 655: 1,\n",
       " 661: 2,\n",
       " 654: 5,\n",
       " 662: 41,\n",
       " 666: 62,\n",
       " 664: 4,\n",
       " 667: 87,\n",
       " 663: 3,\n",
       " 665: 10,\n",
       " 668: 21,\n",
       " 670: 13,\n",
       " 672: 10,\n",
       " 669: 5,\n",
       " 671: 3,\n",
       " 677: 14,\n",
       " 676: 6,\n",
       " 674: 21,\n",
       " 673: 9,\n",
       " 675: 82,\n",
       " 678: 3,\n",
       " 679: 2,\n",
       " 680: 6,\n",
       " 688: 7,\n",
       " 685: 10,\n",
       " 687: 2,\n",
       " 684: 40,\n",
       " 683: 2,\n",
       " 690: 5,\n",
       " 686: 8,\n",
       " 682: 8,\n",
       " 691: 18,\n",
       " 681: 3,\n",
       " 689: 4,\n",
       " 692: 14,\n",
       " 693: 19,\n",
       " 694: 11,\n",
       " 696: 14,\n",
       " 695: 18,\n",
       " 697: 14,\n",
       " 703: 28,\n",
       " 702: 10,\n",
       " 699: 1,\n",
       " 698: 7,\n",
       " 701: 6,\n",
       " 700: 12,\n",
       " 704: 8,\n",
       " 707: 1,\n",
       " 705: 39,\n",
       " 706: 9,\n",
       " 709: 8,\n",
       " 708: 7,\n",
       " 714: 24,\n",
       " 712: 11,\n",
       " 711: 8,\n",
       " 710: 13,\n",
       " 717: 5,\n",
       " 715: 1,\n",
       " 713: 15,\n",
       " 716: 2,\n",
       " 722: 13,\n",
       " 718: 9,\n",
       " 721: 2,\n",
       " 720: 6,\n",
       " 719: 4,\n",
       " 724: 5,\n",
       " 723: 3,\n",
       " 728: 15,\n",
       " 726: 27,\n",
       " 725: 17,\n",
       " 727: 6,\n",
       " 729: 7,\n",
       " 730: 35,\n",
       " 734: 1,\n",
       " 731: 12,\n",
       " 732: 10,\n",
       " 733: 7,\n",
       " 735: 14,\n",
       " 736: 19,\n",
       " 737: 13,\n",
       " 738: 3,\n",
       " 740: 7,\n",
       " 739: 3,\n",
       " 741: 41,\n",
       " 742: 7,\n",
       " 744: 24,\n",
       " 743: 3,\n",
       " 745: 5,\n",
       " 750: 11,\n",
       " 747: 9,\n",
       " 752: 1,\n",
       " 753: 29,\n",
       " 748: 3,\n",
       " 749: 3,\n",
       " 746: 32,\n",
       " 751: 13,\n",
       " 754: 1,\n",
       " 757: 7,\n",
       " 758: 4,\n",
       " 755: 12,\n",
       " 756: 26,\n",
       " 759: 13,\n",
       " 761: 1,\n",
       " 760: 9,\n",
       " 766: 20,\n",
       " 762: 2,\n",
       " 763: 31,\n",
       " 764: 3,\n",
       " 765: 29,\n",
       " 769: 6,\n",
       " 768: 1,\n",
       " 770: 4,\n",
       " 767: 2,\n",
       " 772: 6,\n",
       " 771: 7,\n",
       " 773: 8,\n",
       " 778: 2,\n",
       " 777: 7,\n",
       " 774: 1,\n",
       " 776: 5,\n",
       " 775: 1,\n",
       " 780: 29,\n",
       " 781: 22,\n",
       " 782: 11,\n",
       " 779: 36,\n",
       " 786: 1,\n",
       " 785: 20,\n",
       " 783: 3,\n",
       " 784: 8,\n",
       " 788: 1,\n",
       " 790: 2,\n",
       " 789: 3,\n",
       " 787: 2,\n",
       " 792: 40,\n",
       " 793: 14,\n",
       " 791: 4,\n",
       " 795: 41,\n",
       " 796: 2,\n",
       " 794: 22,\n",
       " 797: 8,\n",
       " 798: 4,\n",
       " 799: 3,\n",
       " 800: 9,\n",
       " 802: 8,\n",
       " 801: 6,\n",
       " 803: 7,\n",
       " 804: 3,\n",
       " 807: 3,\n",
       " 808: 14,\n",
       " 806: 9,\n",
       " 809: 8,\n",
       " 805: 26,\n",
       " 811: 5,\n",
       " 810: 26,\n",
       " 813: 2,\n",
       " 812: 17,\n",
       " 815: 3,\n",
       " 814: 1,\n",
       " 817: 11,\n",
       " 816: 3,\n",
       " 818: 5,\n",
       " 819: 17,\n",
       " 821: 6,\n",
       " 822: 6,\n",
       " 820: 9,\n",
       " 824: 1,\n",
       " 826: 17,\n",
       " 825: 1,\n",
       " 823: 1,\n",
       " 830: 8,\n",
       " 828: 7,\n",
       " 827: 2,\n",
       " 831: 4,\n",
       " 829: 7,\n",
       " 832: 8,\n",
       " 833: 6,\n",
       " 836: 14,\n",
       " 837: 14,\n",
       " 834: 22,\n",
       " 835: 21,\n",
       " 839: 7,\n",
       " 843: 6,\n",
       " 838: 25,\n",
       " 842: 12,\n",
       " 841: 23,\n",
       " 840: 7,\n",
       " 845: 11,\n",
       " 846: 2,\n",
       " 844: 20,\n",
       " 855: 2,\n",
       " 859: 8,\n",
       " 847: 3,\n",
       " 851: 5,\n",
       " 857: 20,\n",
       " 849: 15,\n",
       " 856: 8,\n",
       " 858: 2,\n",
       " 848: 6,\n",
       " 852: 34,\n",
       " 853: 5,\n",
       " 850: 1,\n",
       " 854: 12,\n",
       " 860: 5,\n",
       " 862: 75,\n",
       " 868: 48,\n",
       " 867: 2,\n",
       " 863: 2,\n",
       " 869: 4,\n",
       " 866: 7,\n",
       " 870: 2,\n",
       " 864: 11,\n",
       " 871: 4,\n",
       " 865: 3,\n",
       " 861: 25,\n",
       " 873: 1,\n",
       " 872: 2,\n",
       " 874: 4,\n",
       " 878: 13,\n",
       " 881: 3,\n",
       " 876: 2,\n",
       " 875: 7,\n",
       " 877: 29,\n",
       " 879: 20,\n",
       " 880: 3,\n",
       " 884: 4,\n",
       " 891: 3,\n",
       " 883: 11,\n",
       " 882: 2,\n",
       " 889: 1,\n",
       " 892: 66,\n",
       " 886: 6,\n",
       " 888: 20,\n",
       " 885: 9,\n",
       " 887: 12,\n",
       " 890: 2,\n",
       " 899: 5,\n",
       " 894: 2,\n",
       " 897: 23,\n",
       " 898: 1,\n",
       " 895: 2,\n",
       " 896: 4,\n",
       " 893: 25,\n",
       " 901: 3,\n",
       " 900: 11,\n",
       " 902: 21,\n",
       " 904: 4,\n",
       " 903: 3,\n",
       " 906: 3,\n",
       " 905: 12,\n",
       " 911: 23,\n",
       " 909: 6,\n",
       " 910: 4,\n",
       " 908: 3,\n",
       " 915: 5,\n",
       " 914: 2,\n",
       " 907: 1,\n",
       " 912: 8,\n",
       " 913: 6,\n",
       " 916: 15,\n",
       " 919: 9,\n",
       " 920: 19,\n",
       " 917: 2,\n",
       " 918: 2,\n",
       " 921: 7,\n",
       " 922: 21,\n",
       " 923: 8,\n",
       " 924: 2,\n",
       " 925: 17,\n",
       " 929: 22,\n",
       " 930: 13,\n",
       " 926: 16,\n",
       " 927: 30,\n",
       " 928: 8,\n",
       " 932: 5,\n",
       " 931: 12,\n",
       " 933: 9,\n",
       " 936: 5,\n",
       " 935: 29,\n",
       " 937: 9,\n",
       " 934: 16,\n",
       " 938: 3,\n",
       " 939: 7,\n",
       " 940: 2,\n",
       " 942: 2,\n",
       " 941: 5,\n",
       " 944: 6,\n",
       " 943: 3,\n",
       " 945: 2,\n",
       " 957: 24,\n",
       " 946: 22,\n",
       " 949: 1,\n",
       " 948: 2,\n",
       " 955: 5,\n",
       " 951: 8,\n",
       " 950: 5,\n",
       " 947: 4,\n",
       " 954: 14,\n",
       " 953: 16,\n",
       " 952: 1,\n",
       " 956: 4,\n",
       " 959: 1,\n",
       " 958: 9,\n",
       " 968: 8,\n",
       " 963: 4,\n",
       " 965: 2,\n",
       " 962: 19,\n",
       " 960: 2,\n",
       " 961: 1,\n",
       " 967: 1,\n",
       " 966: 3,\n",
       " 964: 9,\n",
       " 971: 21,\n",
       " 969: 9,\n",
       " 972: 10,\n",
       " 970: 16,\n",
       " 973: 4,\n",
       " 974: 4,\n",
       " 975: 15,\n",
       " 979: 6,\n",
       " 980: 4,\n",
       " 976: 18,\n",
       " 978: 6,\n",
       " 977: 2,\n",
       " 985: 1,\n",
       " 981: 3,\n",
       " 991: 19,\n",
       " 990: 5,\n",
       " 982: 1,\n",
       " 983: 8,\n",
       " 984: 1,\n",
       " 988: 16,\n",
       " 989: 2,\n",
       " 987: 1,\n",
       " 986: 4,\n",
       " 993: 18,\n",
       " 994: 12,\n",
       " 992: 2,\n",
       " 999: 9,\n",
       " 1001: 6,\n",
       " 997: 7,\n",
       " 1000: 2,\n",
       " 998: 7,\n",
       " ...}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary.cfs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 1306,\n",
       " 6: 351,\n",
       " 2: 27,\n",
       " 1: 338,\n",
       " 7: 342,\n",
       " 3: 34,\n",
       " 8: 201,\n",
       " 4: 61,\n",
       " 5: 45,\n",
       " 11: 36,\n",
       " 20: 1884,\n",
       " 13: 45,\n",
       " 17: 46,\n",
       " 23: 48,\n",
       " 14: 315,\n",
       " 16: 72,\n",
       " 10: 8,\n",
       " 21: 1072,\n",
       " 19: 5,\n",
       " 15: 609,\n",
       " 24: 165,\n",
       " 12: 12,\n",
       " 18: 833,\n",
       " 22: 9,\n",
       " 9: 14,\n",
       " 28: 358,\n",
       " 38: 225,\n",
       " 36: 175,\n",
       " 39: 550,\n",
       " 27: 5,\n",
       " 30: 12,\n",
       " 37: 491,\n",
       " 33: 138,\n",
       " 26: 345,\n",
       " 35: 75,\n",
       " 31: 366,\n",
       " 29: 153,\n",
       " 34: 267,\n",
       " 32: 2,\n",
       " 25: 14,\n",
       " 44: 140,\n",
       " 40: 132,\n",
       " 46: 549,\n",
       " 45: 67,\n",
       " 41: 157,\n",
       " 43: 6,\n",
       " 42: 193,\n",
       " 51: 22,\n",
       " 52: 605,\n",
       " 53: 173,\n",
       " 48: 358,\n",
       " 54: 131,\n",
       " 50: 10,\n",
       " 47: 6,\n",
       " 49: 371,\n",
       " 55: 295,\n",
       " 56: 3,\n",
       " 62: 20,\n",
       " 61: 260,\n",
       " 63: 55,\n",
       " 59: 12,\n",
       " 60: 115,\n",
       " 58: 52,\n",
       " 57: 65,\n",
       " 71: 9,\n",
       " 73: 81,\n",
       " 74: 33,\n",
       " 82: 57,\n",
       " 68: 32,\n",
       " 67: 4,\n",
       " 76: 17,\n",
       " 70: 188,\n",
       " 66: 7,\n",
       " 78: 4,\n",
       " 77: 498,\n",
       " 64: 27,\n",
       " 72: 7,\n",
       " 69: 263,\n",
       " 80: 36,\n",
       " 65: 11,\n",
       " 75: 4,\n",
       " 79: 3,\n",
       " 83: 138,\n",
       " 81: 10,\n",
       " 85: 288,\n",
       " 91: 36,\n",
       " 86: 135,\n",
       " 88: 590,\n",
       " 89: 19,\n",
       " 90: 11,\n",
       " 87: 7,\n",
       " 92: 70,\n",
       " 84: 33,\n",
       " 93: 103,\n",
       " 94: 206,\n",
       " 95: 5,\n",
       " 101: 18,\n",
       " 108: 4,\n",
       " 113: 50,\n",
       " 106: 39,\n",
       " 110: 3,\n",
       " 100: 13,\n",
       " 111: 3,\n",
       " 109: 22,\n",
       " 104: 116,\n",
       " 97: 3,\n",
       " 105: 11,\n",
       " 99: 41,\n",
       " 103: 43,\n",
       " 112: 60,\n",
       " 96: 24,\n",
       " 98: 97,\n",
       " 102: 2,\n",
       " 107: 1,\n",
       " 115: 6,\n",
       " 114: 13,\n",
       " 116: 9,\n",
       " 121: 174,\n",
       " 118: 28,\n",
       " 120: 10,\n",
       " 117: 63,\n",
       " 119: 68,\n",
       " 127: 2,\n",
       " 123: 51,\n",
       " 125: 2,\n",
       " 124: 18,\n",
       " 126: 49,\n",
       " 128: 187,\n",
       " 122: 5,\n",
       " 131: 143,\n",
       " 129: 14,\n",
       " 130: 155,\n",
       " 137: 113,\n",
       " 139: 104,\n",
       " 132: 35,\n",
       " 136: 14,\n",
       " 138: 29,\n",
       " 134: 25,\n",
       " 143: 55,\n",
       " 142: 24,\n",
       " 140: 92,\n",
       " 145: 157,\n",
       " 141: 24,\n",
       " 146: 8,\n",
       " 135: 22,\n",
       " 144: 2,\n",
       " 133: 20,\n",
       " 162: 36,\n",
       " 171: 70,\n",
       " 177: 93,\n",
       " 156: 6,\n",
       " 180: 7,\n",
       " 178: 15,\n",
       " 158: 107,\n",
       " 150: 67,\n",
       " 179: 30,\n",
       " 152: 34,\n",
       " 169: 24,\n",
       " 164: 11,\n",
       " 155: 13,\n",
       " 149: 1,\n",
       " 148: 42,\n",
       " 168: 12,\n",
       " 175: 165,\n",
       " 157: 85,\n",
       " 147: 92,\n",
       " 176: 3,\n",
       " 163: 23,\n",
       " 166: 90,\n",
       " 174: 47,\n",
       " 167: 176,\n",
       " 151: 239,\n",
       " 159: 3,\n",
       " 161: 75,\n",
       " 154: 6,\n",
       " 153: 36,\n",
       " 165: 117,\n",
       " 170: 85,\n",
       " 160: 182,\n",
       " 172: 7,\n",
       " 173: 14,\n",
       " 190: 33,\n",
       " 184: 51,\n",
       " 191: 2,\n",
       " 192: 13,\n",
       " 185: 8,\n",
       " 187: 177,\n",
       " 189: 13,\n",
       " 181: 130,\n",
       " 188: 8,\n",
       " 182: 46,\n",
       " 186: 16,\n",
       " 183: 15,\n",
       " 195: 97,\n",
       " 194: 11,\n",
       " 193: 8,\n",
       " 196: 10,\n",
       " 202: 56,\n",
       " 199: 21,\n",
       " 200: 9,\n",
       " 201: 18,\n",
       " 198: 1,\n",
       " 197: 13,\n",
       " 247: 169,\n",
       " 225: 1,\n",
       " 238: 1,\n",
       " 250: 322,\n",
       " 207: 5,\n",
       " 212: 2,\n",
       " 205: 1,\n",
       " 242: 1,\n",
       " 203: 309,\n",
       " 227: 93,\n",
       " 245: 107,\n",
       " 221: 2,\n",
       " 231: 1,\n",
       " 226: 19,\n",
       " 220: 3,\n",
       " 217: 1,\n",
       " 241: 25,\n",
       " 219: 4,\n",
       " 232: 4,\n",
       " 239: 1,\n",
       " 215: 15,\n",
       " 224: 6,\n",
       " 206: 1,\n",
       " 228: 3,\n",
       " 251: 25,\n",
       " 204: 12,\n",
       " 243: 1,\n",
       " 246: 231,\n",
       " 208: 1,\n",
       " 240: 37,\n",
       " 230: 1,\n",
       " 223: 1,\n",
       " 213: 1,\n",
       " 248: 88,\n",
       " 222: 17,\n",
       " 249: 3,\n",
       " 216: 1,\n",
       " 234: 1,\n",
       " 237: 3,\n",
       " 214: 1,\n",
       " 218: 4,\n",
       " 211: 2,\n",
       " 236: 1,\n",
       " 235: 37,\n",
       " 233: 9,\n",
       " 210: 4,\n",
       " 244: 7,\n",
       " 229: 6,\n",
       " 209: 1,\n",
       " 260: 114,\n",
       " 261: 137,\n",
       " 258: 29,\n",
       " 262: 21,\n",
       " 252: 23,\n",
       " 253: 8,\n",
       " 259: 61,\n",
       " 254: 1,\n",
       " 257: 3,\n",
       " 255: 2,\n",
       " 256: 13,\n",
       " 264: 43,\n",
       " 266: 21,\n",
       " 263: 22,\n",
       " 267: 7,\n",
       " 265: 130,\n",
       " 268: 7,\n",
       " 270: 81,\n",
       " 269: 5,\n",
       " 271: 27,\n",
       " 276: 45,\n",
       " 273: 6,\n",
       " 272: 68,\n",
       " 274: 28,\n",
       " 275: 12,\n",
       " 277: 81,\n",
       " 278: 76,\n",
       " 279: 44,\n",
       " 305: 12,\n",
       " 299: 6,\n",
       " 300: 8,\n",
       " 284: 120,\n",
       " 288: 8,\n",
       " 292: 30,\n",
       " 301: 1,\n",
       " 294: 18,\n",
       " 290: 3,\n",
       " 280: 1,\n",
       " 304: 25,\n",
       " 282: 9,\n",
       " 298: 85,\n",
       " 289: 61,\n",
       " 281: 5,\n",
       " 295: 113,\n",
       " 296: 3,\n",
       " 302: 4,\n",
       " 293: 8,\n",
       " 283: 1,\n",
       " 285: 11,\n",
       " 297: 9,\n",
       " 287: 3,\n",
       " 291: 121,\n",
       " 303: 33,\n",
       " 286: 33,\n",
       " 307: 13,\n",
       " 306: 2,\n",
       " 308: 52,\n",
       " 309: 9,\n",
       " 310: 1,\n",
       " 322: 104,\n",
       " 319: 19,\n",
       " 315: 12,\n",
       " 316: 9,\n",
       " 311: 42,\n",
       " 318: 9,\n",
       " 317: 2,\n",
       " 320: 2,\n",
       " 313: 16,\n",
       " 312: 19,\n",
       " 321: 1,\n",
       " 314: 1,\n",
       " 327: 5,\n",
       " 325: 32,\n",
       " 328: 25,\n",
       " 326: 59,\n",
       " 330: 94,\n",
       " 323: 107,\n",
       " 329: 14,\n",
       " 324: 17,\n",
       " 333: 35,\n",
       " 332: 20,\n",
       " 331: 9,\n",
       " 334: 30,\n",
       " 335: 10,\n",
       " 336: 20,\n",
       " 347: 10,\n",
       " 337: 5,\n",
       " 348: 57,\n",
       " 344: 2,\n",
       " 343: 76,\n",
       " 340: 151,\n",
       " 341: 7,\n",
       " 345: 4,\n",
       " 339: 7,\n",
       " 338: 5,\n",
       " 349: 30,\n",
       " 350: 28,\n",
       " 346: 120,\n",
       " 342: 50,\n",
       " 355: 10,\n",
       " 356: 72,\n",
       " 353: 68,\n",
       " 354: 2,\n",
       " 357: 2,\n",
       " 351: 3,\n",
       " 352: 38,\n",
       " 358: 10,\n",
       " 359: 68,\n",
       " 360: 7,\n",
       " 361: 2,\n",
       " 363: 10,\n",
       " 366: 12,\n",
       " 364: 11,\n",
       " 362: 11,\n",
       " 365: 12,\n",
       " 367: 116,\n",
       " 368: 2,\n",
       " 369: 40,\n",
       " 370: 42,\n",
       " 372: 9,\n",
       " 371: 61,\n",
       " 376: 34,\n",
       " 375: 7,\n",
       " 374: 1,\n",
       " 373: 4,\n",
       " 377: 16,\n",
       " 379: 75,\n",
       " 380: 12,\n",
       " 382: 4,\n",
       " 383: 15,\n",
       " 378: 10,\n",
       " 381: 51,\n",
       " 384: 12,\n",
       " 385: 19,\n",
       " 390: 11,\n",
       " 389: 11,\n",
       " 387: 12,\n",
       " 386: 2,\n",
       " 388: 140,\n",
       " 394: 9,\n",
       " 395: 8,\n",
       " 393: 10,\n",
       " 392: 3,\n",
       " 391: 29,\n",
       " 396: 82,\n",
       " 398: 4,\n",
       " 400: 87,\n",
       " 399: 1,\n",
       " 397: 35,\n",
       " 406: 12,\n",
       " 403: 97,\n",
       " 401: 14,\n",
       " 402: 81,\n",
       " 405: 3,\n",
       " 404: 12,\n",
       " 407: 6,\n",
       " 410: 99,\n",
       " 409: 4,\n",
       " 408: 17,\n",
       " 411: 112,\n",
       " 412: 2,\n",
       " 413: 8,\n",
       " 414: 2,\n",
       " 419: 39,\n",
       " 416: 70,\n",
       " 420: 19,\n",
       " 418: 8,\n",
       " 415: 61,\n",
       " 417: 9,\n",
       " 421: 106,\n",
       " 427: 5,\n",
       " 426: 6,\n",
       " 425: 3,\n",
       " 424: 175,\n",
       " 423: 14,\n",
       " 422: 19,\n",
       " 429: 3,\n",
       " 428: 13,\n",
       " 433: 25,\n",
       " 432: 34,\n",
       " 430: 56,\n",
       " 431: 21,\n",
       " 436: 27,\n",
       " 434: 1,\n",
       " 435: 1,\n",
       " 437: 82,\n",
       " 439: 24,\n",
       " 438: 25,\n",
       " 442: 72,\n",
       " 441: 17,\n",
       " 445: 1,\n",
       " 444: 4,\n",
       " 443: 7,\n",
       " 440: 3,\n",
       " 449: 91,\n",
       " 446: 14,\n",
       " 455: 19,\n",
       " 452: 24,\n",
       " 450: 6,\n",
       " 451: 69,\n",
       " 453: 31,\n",
       " 456: 1,\n",
       " 448: 12,\n",
       " 447: 2,\n",
       " 454: 14,\n",
       " 460: 9,\n",
       " 458: 6,\n",
       " 457: 37,\n",
       " 459: 23,\n",
       " 461: 1,\n",
       " 470: 18,\n",
       " 463: 46,\n",
       " 467: 14,\n",
       " 472: 33,\n",
       " 466: 8,\n",
       " 471: 1,\n",
       " 464: 13,\n",
       " 465: 1,\n",
       " 468: 81,\n",
       " 462: 31,\n",
       " 469: 1,\n",
       " 476: 12,\n",
       " 482: 20,\n",
       " 483: 4,\n",
       " 477: 9,\n",
       " 479: 6,\n",
       " 480: 49,\n",
       " 474: 21,\n",
       " 473: 10,\n",
       " 475: 18,\n",
       " 481: 19,\n",
       " 478: 3,\n",
       " 484: 4,\n",
       " 487: 29,\n",
       " 485: 30,\n",
       " 486: 2,\n",
       " 488: 9,\n",
       " 489: 4,\n",
       " 490: 32,\n",
       " 496: 6,\n",
       " 497: 32,\n",
       " 493: 1,\n",
       " 495: 3,\n",
       " 492: 8,\n",
       " 498: 6,\n",
       " 494: 7,\n",
       " 491: 4,\n",
       " 499: 21,\n",
       " 502: 8,\n",
       " 500: 1,\n",
       " 503: 12,\n",
       " 504: 5,\n",
       " 501: 5,\n",
       " 506: 2,\n",
       " 508: 76,\n",
       " 507: 2,\n",
       " 505: 3,\n",
       " 509: 2,\n",
       " 511: 51,\n",
       " 510: 4,\n",
       " 512: 2,\n",
       " 514: 12,\n",
       " 513: 2,\n",
       " 520: 41,\n",
       " 515: 2,\n",
       " 516: 9,\n",
       " 518: 95,\n",
       " 517: 12,\n",
       " 526: 5,\n",
       " 522: 26,\n",
       " 519: 25,\n",
       " 525: 2,\n",
       " 523: 3,\n",
       " 521: 1,\n",
       " 524: 4,\n",
       " 533: 4,\n",
       " 535: 15,\n",
       " 531: 3,\n",
       " 527: 2,\n",
       " 530: 1,\n",
       " 528: 1,\n",
       " 532: 3,\n",
       " 536: 7,\n",
       " 534: 2,\n",
       " 529: 3,\n",
       " 537: 43,\n",
       " 539: 10,\n",
       " 538: 12,\n",
       " 544: 29,\n",
       " 546: 23,\n",
       " 543: 1,\n",
       " 541: 10,\n",
       " 540: 3,\n",
       " 545: 2,\n",
       " 547: 54,\n",
       " 542: 3,\n",
       " 553: 3,\n",
       " 552: 22,\n",
       " 550: 1,\n",
       " 549: 7,\n",
       " 551: 22,\n",
       " 548: 3,\n",
       " 554: 3,\n",
       " 561: 4,\n",
       " 564: 46,\n",
       " 556: 4,\n",
       " 558: 21,\n",
       " 560: 1,\n",
       " 565: 3,\n",
       " 559: 38,\n",
       " 563: 28,\n",
       " 557: 8,\n",
       " 562: 40,\n",
       " 555: 4,\n",
       " 566: 93,\n",
       " 567: 34,\n",
       " 568: 60,\n",
       " 569: 106,\n",
       " 570: 3,\n",
       " 572: 3,\n",
       " 573: 5,\n",
       " 571: 5,\n",
       " 575: 30,\n",
       " 574: 28,\n",
       " 576: 21,\n",
       " 577: 7,\n",
       " 578: 59,\n",
       " 582: 30,\n",
       " 581: 6,\n",
       " 579: 13,\n",
       " 580: 19,\n",
       " 584: 1,\n",
       " 586: 6,\n",
       " 583: 1,\n",
       " 590: 4,\n",
       " 585: 54,\n",
       " 587: 4,\n",
       " 589: 3,\n",
       " 588: 9,\n",
       " 593: 9,\n",
       " 595: 52,\n",
       " 596: 23,\n",
       " 591: 2,\n",
       " 592: 24,\n",
       " 594: 23,\n",
       " 598: 54,\n",
       " 597: 1,\n",
       " 600: 2,\n",
       " 599: 19,\n",
       " 604: 19,\n",
       " 605: 21,\n",
       " 606: 49,\n",
       " 601: 2,\n",
       " 603: 12,\n",
       " 602: 9,\n",
       " 607: 4,\n",
       " 611: 1,\n",
       " 609: 27,\n",
       " 612: 22,\n",
       " 613: 22,\n",
       " 608: 1,\n",
       " 610: 48,\n",
       " 614: 5,\n",
       " 615: 6,\n",
       " 616: 10,\n",
       " 617: 15,\n",
       " 618: 2,\n",
       " 621: 67,\n",
       " 619: 44,\n",
       " 622: 32,\n",
       " 620: 13,\n",
       " 627: 7,\n",
       " 625: 1,\n",
       " 629: 4,\n",
       " 626: 18,\n",
       " 630: 7,\n",
       " 632: 21,\n",
       " 623: 5,\n",
       " 631: 6,\n",
       " 628: 12,\n",
       " 633: 2,\n",
       " 624: 8,\n",
       " 637: 1,\n",
       " 636: 1,\n",
       " 638: 1,\n",
       " 635: 6,\n",
       " 634: 6,\n",
       " 639: 19,\n",
       " 641: 24,\n",
       " 642: 13,\n",
       " 640: 9,\n",
       " 643: 21,\n",
       " 644: 53,\n",
       " 648: 4,\n",
       " 650: 5,\n",
       " 645: 3,\n",
       " 647: 5,\n",
       " 649: 8,\n",
       " 646: 32,\n",
       " 652: 18,\n",
       " 651: 14,\n",
       " 657: 2,\n",
       " 653: 3,\n",
       " 659: 3,\n",
       " 656: 5,\n",
       " 658: 14,\n",
       " 660: 7,\n",
       " 655: 1,\n",
       " 661: 2,\n",
       " 654: 5,\n",
       " 662: 41,\n",
       " 666: 59,\n",
       " 664: 4,\n",
       " 667: 85,\n",
       " 663: 3,\n",
       " 665: 10,\n",
       " 668: 17,\n",
       " 670: 13,\n",
       " 672: 10,\n",
       " 669: 5,\n",
       " 671: 3,\n",
       " 677: 14,\n",
       " 676: 6,\n",
       " 674: 20,\n",
       " 673: 9,\n",
       " 675: 80,\n",
       " 678: 3,\n",
       " 679: 2,\n",
       " 680: 6,\n",
       " 688: 7,\n",
       " 685: 10,\n",
       " 687: 2,\n",
       " 684: 38,\n",
       " 683: 2,\n",
       " 690: 5,\n",
       " 686: 8,\n",
       " 682: 8,\n",
       " 691: 18,\n",
       " 681: 3,\n",
       " 689: 4,\n",
       " 692: 14,\n",
       " 693: 19,\n",
       " 694: 11,\n",
       " 696: 14,\n",
       " 695: 18,\n",
       " 697: 14,\n",
       " 703: 28,\n",
       " 702: 10,\n",
       " 699: 1,\n",
       " 698: 7,\n",
       " 701: 6,\n",
       " 700: 10,\n",
       " 704: 8,\n",
       " 707: 1,\n",
       " 705: 37,\n",
       " 706: 9,\n",
       " 709: 8,\n",
       " 708: 7,\n",
       " 714: 24,\n",
       " 712: 11,\n",
       " 711: 8,\n",
       " 710: 13,\n",
       " 717: 5,\n",
       " 715: 1,\n",
       " 713: 15,\n",
       " 716: 2,\n",
       " 722: 12,\n",
       " 718: 9,\n",
       " 721: 2,\n",
       " 720: 6,\n",
       " 719: 4,\n",
       " 724: 5,\n",
       " 723: 3,\n",
       " 728: 14,\n",
       " 726: 27,\n",
       " 725: 16,\n",
       " 727: 6,\n",
       " 729: 4,\n",
       " 730: 34,\n",
       " 734: 1,\n",
       " 731: 12,\n",
       " 732: 10,\n",
       " 733: 6,\n",
       " 735: 14,\n",
       " 736: 18,\n",
       " 737: 13,\n",
       " 738: 3,\n",
       " 740: 6,\n",
       " 739: 3,\n",
       " 741: 39,\n",
       " 742: 7,\n",
       " 744: 24,\n",
       " 743: 3,\n",
       " 745: 5,\n",
       " 750: 11,\n",
       " 747: 9,\n",
       " 752: 1,\n",
       " 753: 29,\n",
       " 748: 3,\n",
       " 749: 3,\n",
       " 746: 32,\n",
       " 751: 13,\n",
       " 754: 1,\n",
       " 757: 7,\n",
       " 758: 4,\n",
       " 755: 12,\n",
       " 756: 26,\n",
       " 759: 13,\n",
       " 761: 1,\n",
       " 760: 9,\n",
       " 766: 20,\n",
       " 762: 2,\n",
       " 763: 30,\n",
       " 764: 3,\n",
       " 765: 29,\n",
       " 769: 6,\n",
       " 768: 1,\n",
       " 770: 4,\n",
       " 767: 2,\n",
       " 772: 6,\n",
       " 771: 7,\n",
       " 773: 8,\n",
       " 778: 2,\n",
       " 777: 7,\n",
       " 774: 1,\n",
       " 776: 5,\n",
       " 775: 1,\n",
       " 780: 29,\n",
       " 781: 22,\n",
       " 782: 11,\n",
       " 779: 35,\n",
       " 786: 1,\n",
       " 785: 20,\n",
       " 783: 3,\n",
       " 784: 8,\n",
       " 788: 1,\n",
       " 790: 2,\n",
       " 789: 3,\n",
       " 787: 2,\n",
       " 792: 38,\n",
       " 793: 14,\n",
       " 791: 3,\n",
       " 795: 41,\n",
       " 796: 2,\n",
       " 794: 22,\n",
       " 797: 8,\n",
       " 798: 4,\n",
       " 799: 3,\n",
       " 800: 8,\n",
       " 802: 8,\n",
       " 801: 6,\n",
       " 803: 7,\n",
       " 804: 3,\n",
       " 807: 3,\n",
       " 808: 14,\n",
       " 806: 9,\n",
       " 809: 8,\n",
       " 805: 26,\n",
       " 811: 5,\n",
       " 810: 26,\n",
       " 813: 2,\n",
       " 812: 17,\n",
       " 815: 3,\n",
       " 814: 1,\n",
       " 817: 11,\n",
       " 816: 3,\n",
       " 818: 5,\n",
       " 819: 15,\n",
       " 821: 6,\n",
       " 822: 6,\n",
       " 820: 8,\n",
       " 824: 1,\n",
       " 826: 17,\n",
       " 825: 1,\n",
       " 823: 1,\n",
       " 830: 8,\n",
       " 828: 7,\n",
       " 827: 2,\n",
       " 831: 4,\n",
       " 829: 7,\n",
       " 832: 8,\n",
       " 833: 6,\n",
       " 836: 14,\n",
       " 837: 14,\n",
       " 834: 22,\n",
       " 835: 21,\n",
       " 839: 7,\n",
       " 843: 6,\n",
       " 838: 25,\n",
       " 842: 12,\n",
       " 841: 20,\n",
       " 840: 7,\n",
       " 845: 11,\n",
       " 846: 2,\n",
       " 844: 20,\n",
       " 855: 2,\n",
       " 859: 7,\n",
       " 847: 3,\n",
       " 851: 5,\n",
       " 857: 20,\n",
       " 849: 14,\n",
       " 856: 8,\n",
       " 858: 2,\n",
       " 848: 6,\n",
       " 852: 33,\n",
       " 853: 5,\n",
       " 850: 1,\n",
       " 854: 12,\n",
       " 860: 5,\n",
       " 862: 74,\n",
       " 868: 48,\n",
       " 867: 2,\n",
       " 863: 2,\n",
       " 869: 4,\n",
       " 866: 7,\n",
       " 870: 2,\n",
       " 864: 11,\n",
       " 871: 4,\n",
       " 865: 3,\n",
       " 861: 25,\n",
       " 873: 1,\n",
       " 872: 2,\n",
       " 874: 4,\n",
       " 878: 13,\n",
       " 881: 3,\n",
       " 876: 2,\n",
       " 875: 7,\n",
       " 877: 28,\n",
       " 879: 19,\n",
       " 880: 3,\n",
       " 884: 4,\n",
       " 891: 3,\n",
       " 883: 11,\n",
       " 882: 2,\n",
       " 889: 1,\n",
       " 892: 66,\n",
       " 886: 6,\n",
       " 888: 20,\n",
       " 885: 9,\n",
       " 887: 11,\n",
       " 890: 2,\n",
       " 899: 5,\n",
       " 894: 2,\n",
       " 897: 22,\n",
       " 898: 1,\n",
       " 895: 2,\n",
       " 896: 4,\n",
       " 893: 25,\n",
       " 901: 3,\n",
       " 900: 11,\n",
       " 902: 21,\n",
       " 904: 4,\n",
       " 903: 3,\n",
       " 906: 3,\n",
       " 905: 12,\n",
       " 911: 23,\n",
       " 909: 6,\n",
       " 910: 4,\n",
       " 908: 3,\n",
       " 915: 5,\n",
       " 914: 2,\n",
       " 907: 1,\n",
       " 912: 8,\n",
       " 913: 6,\n",
       " 916: 15,\n",
       " 919: 9,\n",
       " 920: 18,\n",
       " 917: 2,\n",
       " 918: 2,\n",
       " 921: 7,\n",
       " 922: 21,\n",
       " 923: 8,\n",
       " 924: 2,\n",
       " 925: 17,\n",
       " 929: 20,\n",
       " 930: 10,\n",
       " 926: 16,\n",
       " 927: 30,\n",
       " 928: 7,\n",
       " 932: 5,\n",
       " 931: 11,\n",
       " 933: 9,\n",
       " 936: 5,\n",
       " 935: 28,\n",
       " 937: 9,\n",
       " 934: 16,\n",
       " 938: 3,\n",
       " 939: 7,\n",
       " 940: 2,\n",
       " 942: 2,\n",
       " 941: 5,\n",
       " 944: 6,\n",
       " 943: 3,\n",
       " 945: 2,\n",
       " 957: 24,\n",
       " 946: 21,\n",
       " 949: 1,\n",
       " 948: 2,\n",
       " 955: 5,\n",
       " 951: 8,\n",
       " 950: 5,\n",
       " 947: 4,\n",
       " 954: 14,\n",
       " 953: 16,\n",
       " 952: 1,\n",
       " 956: 4,\n",
       " 959: 1,\n",
       " 958: 9,\n",
       " 968: 8,\n",
       " 963: 4,\n",
       " 965: 2,\n",
       " 962: 19,\n",
       " 960: 2,\n",
       " 961: 1,\n",
       " 967: 1,\n",
       " 966: 3,\n",
       " 964: 8,\n",
       " 971: 21,\n",
       " 969: 9,\n",
       " 972: 10,\n",
       " 970: 16,\n",
       " 973: 4,\n",
       " 974: 4,\n",
       " 975: 15,\n",
       " 979: 6,\n",
       " 980: 4,\n",
       " 976: 18,\n",
       " 978: 6,\n",
       " 977: 2,\n",
       " 985: 1,\n",
       " 981: 3,\n",
       " 991: 19,\n",
       " 990: 5,\n",
       " 982: 1,\n",
       " 983: 8,\n",
       " 984: 1,\n",
       " 988: 16,\n",
       " 989: 2,\n",
       " 987: 1,\n",
       " 986: 4,\n",
       " 993: 18,\n",
       " 994: 12,\n",
       " 992: 2,\n",
       " 999: 8,\n",
       " 1001: 5,\n",
       " 997: 6,\n",
       " 1000: 2,\n",
       " 998: 7,\n",
       " ...}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary.dfs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['follow', 'the', 'white', 'rabbit']\n",
      "[1482, 20, 497, 333]\n"
     ]
    }
   ],
   "source": [
    "sentence = 'follow the white rabbit'\n",
    "new_tokens = simple_preprocess(sentence)\n",
    "ids = dictionary.doc2idx(new_tokens)\n",
    "print(new_tokens)\n",
    "print(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "special_tokens = {'[PAD]': 0, '[UNK]': 1}\n",
    "dictionary.patch_with_special_tokens(special_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_rare_ids(dictionary, min_freq):\n",
    "    rare_ids = [t[0] for t in dictionary.cfs.items() if t[1] < min_freq]\n",
    "    return rare_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_vocab(sentences, folder=None, special_tokens=None, vocab_size=None, min_freq=None):\n",
    "    if folder is not None:\n",
    "        if not os.path.exists(folder):\n",
    "            os.mkdir(folder)\n",
    "            \n",
    "    # tokenizes the sentences and create a Dictionary\n",
    "    tokens = [simple_preprocess(sent) for sent in sentences]\n",
    "    dictionary = corpora.Dictionary(tokens)\n",
    "    # keeps only the most frequent words (vocab size)\n",
    "    if vocab_size is not None:\n",
    "        dictionary.filter_extremes(keep_n=vocab_size)\n",
    "    # removes rare words (in case the vocab size still\n",
    "    # includes words with low frequency)\n",
    "    if min_freq is not None:\n",
    "        rare_tokens = get_rare_ids(dictionary, min_freq)\n",
    "        dictionary.filter_tokens(bad_ids=rare_tokens)\n",
    "    # gets the whole list of tokens and frequencies\n",
    "    items = dictionary.cfs.items()\n",
    "    # sorts the tokens in descending order\n",
    "    words = [dictionary[t[0]] for t in sorted(dictionary.cfs.items(), key=lambda t: -t[1])]\n",
    "    # prepends special tokens, if any\n",
    "    if special_tokens is not None:\n",
    "        to_add = []\n",
    "        for special_token in special_tokens:\n",
    "            if special_token not in words:\n",
    "                to_add.append(special_token)\n",
    "        words = to_add + words\n",
    "                \n",
    "    with open(os.path.join(folder, 'vocab.txt'), 'w') as f:\n",
    "        for word in words:\n",
    "            f.write(f'{word}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_vocab(train_dataset['sentence'], 'our_vocab/', special_tokens=['[PAD]', '[UNK]', '[SEP]', '[CLS]', '[MASK]'], min_freq=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HugginFace's Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer\n",
    "\n",
    "tokenizer = BertTokenizer('our_vocab/vocab.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['follow', 'the', 'white', 'rabbit', '[UNK]']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_sentence = 'follow the white rabbit neo'\n",
    "new_tokens = tokenizer.tokenize(new_sentence)\n",
    "new_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1219, 5, 229, 200, 1]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_ids = tokenizer.convert_tokens_to_ids(new_tokens)\n",
    "new_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3, 1219, 5, 229, 200, 1, 2]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_ids = tokenizer.encode(new_sentence)\n",
    "new_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[CLS]', 'follow', 'the', 'white', 'rabbit', '[UNK]', '[SEP]']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.convert_ids_to_tokens(new_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1219, 5, 229, 200, 1]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.encode(new_sentence, add_special_tokens=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[1219,    5,  229,  200,    1]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(new_sentence, add_special_tokens=False, return_tensors='pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [3, 1219, 5, 229, 200, 1, 2, 51, 42, 78, 32, 307, 41, 5, 1, 30, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence1 = 'follow the white rabbit neo'\n",
    "sentence2 = 'no one can be told what the matrix is'\n",
    "joined_sentences = tokenizer(sentence1, sentence2)\n",
    "joined_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'follow', 'the', 'white', 'rabbit', '[UNK]', '[SEP]', 'no', 'one', 'can', 'be', 'told', 'what', 'the', '[UNK]', 'is', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.convert_ids_to_tokens(joined_sentences['input_ids']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [[3, 1219, 5, 229, 200, 1, 2, 0, 0, 0, 0], [3, 51, 42, 78, 32, 307, 41, 5, 1, 30, 2]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "separate_sentences = tokenizer([sentence1, sentence2], padding=True)\n",
    "separate_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'follow', 'the', 'white', 'rabbit', '[UNK]', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']\n",
      "[1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.convert_ids_to_tokens(separate_sentences['input_ids'][0]))\n",
    "print(separate_sentences['attention_mask'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'follow', 'the', 'white', 'rabbit', '[UNK]', '[SEP]', 'no', 'one', 'can', 'be', 'told', 'what', 'the', '[UNK]', 'is', '[SEP]']\n",
      "['[CLS]', 'another', 'first', 'sentence', '[SEP]', '[UNK]', 'second', 'sentence', 'here', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "first_sentences = [sentence1, 'another first sentence']\n",
    "second_sentences = [sentence2, 'a second sentence here']\n",
    "batch_of_pairs = tokenizer(first_sentences, second_sentences)\n",
    "first_input = tokenizer.convert_ids_to_tokens(batch_of_pairs['input_ids'][0])\n",
    "second_input = tokenizer.convert_ids_to_tokens(batch_of_pairs['input_ids'][1])\n",
    "print(first_input)\n",
    "print(second_input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   3,   27,    1,  ...,   86,  870,    2],\n",
       "        [   3,   24,   10,  ..., 1573,    5,    2],\n",
       "        [   3,   49,   12,  ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [   3,    1,    6,  ...,    0,    0,    0],\n",
       "        [   3,    6,  132,  ...,    0,    0,    0],\n",
       "        [   3,    1,    1,  ...,    0,    0,    0]])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_dataset = tokenizer(dataset['sentence'], \n",
    "                              padding=True, \n",
    "                              return_tensors='pt', \n",
    "                              max_length=50, \n",
    "                              truncation=True)\n",
    "tokenized_dataset['input_ids']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Before Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One-Hot Encoding (OHE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/ohe1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/ohe2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of Words (BoW)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'white', 'rabbit', 'is', 'rabbit']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = 'the white rabbit is a rabbit'\n",
    "bow_tokens = simple_preprocess(sentence)\n",
    "bow_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(20, 1), (69, 1), (333, 2), (497, 1)]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bow = dictionary.doc2bow(bow_tokens)\n",
    "bow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language Models\n",
    "\n",
    "![](images/blank1.png)\n",
    "\n",
    "![](images/blank2.png)\n",
    "\n",
    "![](images/fill1.png)\n",
    "\n",
    "![](images/fill2.png)\n",
    "\n",
    "## N-grams\n",
    "\n",
    "![](images/ngrams.png)\n",
    "\n",
    "## Continuous Bag-of-Words (CBoW)\n",
    "\n",
    "![](images/blank_end.png)\n",
    "\n",
    "![](images/blank_center.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings\n",
    "\n",
    "## Word2Vec\n",
    "\n",
    "![](images/cbow.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CBOW(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_size):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_size)\n",
    "        self.linear = nn.Linear(embedding_size, vocab_size)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        embeddings = self.embedding(X)\n",
    "        bow = embeddings.mean(dim=1)\n",
    "        logits = self.linear(bow)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('weight',\n",
       "              tensor([[ 0.3367,  0.1288,  0.2345],\n",
       "                      [ 0.2303, -1.1229, -0.1863],\n",
       "                      [ 2.2082, -0.6380,  0.4617],\n",
       "                      [ 0.2674,  0.5349,  0.8094],\n",
       "                      [ 1.1103, -1.6898, -0.9890]]))])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "dummy_cbow = CBOW(vocab_size=5, embedding_size=3)\n",
    "dummy_cbow.embedding.state_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/w2v_embed.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 2.2082, -0.6380,  0.4617],\n",
       "        [ 0.2674,  0.5349,  0.8094]], grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tokens: ['is', 'barking']\n",
    "dummy_cbow.embedding(torch.as_tensor([2, 3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "tiny_vocab = ['the', 'small', 'is', 'barking', 'dog']\n",
    "context_words = ['the', 'small', 'is', 'barking']\n",
    "target_words = ['dog']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_context = torch.as_tensor([[0, 1, 2, 3]]).long()\n",
    "batch_target = torch.as_tensor([4]).long()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/w2v_cbow.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.7606, -0.2743,  0.3298]], grad_fn=<MeanBackward1>)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cbow_features = dummy_cbow.embedding(batch_context).mean(dim=1)\n",
    "cbow_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/w2v_logits.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.3542,  0.6937, -0.2028, -0.5873,  0.2099]],\n",
       "       grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits = dummy_cbow.linear(cbow_features)\n",
    "logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is an Embeddings Anyway?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/rest_discrete.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/rest_continuous.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.0000, -0.4318,  0.9976, -0.2974],\n",
       "        [-0.4318,  1.0000, -0.4270,  0.3581],\n",
       "        [ 0.9976, -0.4270,  1.0000, -0.3598],\n",
       "        [-0.2974,  0.3581, -0.3598,  1.0000]])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings = torch.as_tensor([[.7, -.4, .7],\n",
    "                           [.3, .7, -.5],\n",
    "                           [.9, -.55, .8],\n",
    "                           [-.3, .8, .34]]).float()\n",
    "sims = torch.zeros(4, 4)\n",
    "for i in range(4):\n",
    "    for j in range(4):\n",
    "        sims[i, j] = F.cosine_similarity(ratings[i], ratings[j], dim=0)\n",
    "sims"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-trained Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global Vectors (GloVe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "400000"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim import downloader\n",
    "\n",
    "glove = downloader.load('glove-wiki-gigaword-50')\n",
    "len(glove.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.16386 ,  0.57795 , -0.59197 , -0.32446 ,  0.29762 ,  0.85151 ,\n",
       "       -0.76695 , -0.20733 ,  0.21491 , -0.51587 , -0.17517 ,  0.94459 ,\n",
       "        0.12705 , -0.33031 ,  0.75951 ,  0.44449 ,  0.16553 , -0.19235 ,\n",
       "        0.065533, -0.12394 ,  0.61446 ,  0.89784 ,  0.17413 ,  0.41149 ,\n",
       "        1.191   , -0.39461 , -0.459   ,  0.022161, -0.50843 , -0.44464 ,\n",
       "        0.68721 , -0.7167  ,  0.20835 , -0.23437 ,  0.02604 , -0.47993 ,\n",
       "        0.31873 , -0.29135 ,  0.50273 , -0.55144 , -0.066692,  0.43873 ,\n",
       "       -0.24293 , -1.0247  ,  0.029375,  0.068499,  0.25451 , -1.9663  ,\n",
       "        0.26673 ,  0.88486 ], dtype=float32)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glove['alice']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "synthetic_queen = glove['king'] - glove['man'] + glove['woman']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/dvgodoy/projects/PyTorchStepByStep/plots/chapter11.py:22: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "  axs[i].set_yticklabels(['', word, ''])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABQAAAADZCAYAAAB/7Z23AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAevklEQVR4nO3de5CU5Z0v8N/ADCAgl0VguCMDoly8gQR0NIu36JKUblxBGTVeEhOIIUgiOSMuF/GUOUexygUdL8VCDKklEogYMMesCAiIqKsLJiqsEQIKIhgxDiMzwMz5I2snE1Qk4WGYdz+fqqnq9/J8+3mb7p6eL2935+3atasmAAAAAIBMalDXEwAAAAAA0lEAAgAAAECGKQABAAAAIMMUgAAAAACQYQpAAAAAAMgwBSAAAAAAZJgCEAAAAAAyLL+uJ1Afrf9gf5Lcd8urkuR+7OImW5Lm727fL2n+i1vLk+af2alp0vyavLR9+6s79yTNP7VqQ9L8/a26JM3fe0zrpPlfnfVSsux5156eLDsiotGqnyTNf7rrV5Lmv//R3qT5I9q8nzS/umna+2bD8h1J85/c0zFpfv92zZLmt22SlzQ/Ej/3N6hIe/9cX5n29t/8hbOT5g/9l+uS5qeWX3xZ0vyN1a2S5n/ljqeT5r/0f76UNL/p1rVJ839R2S1pfucWTZLmFzYvSJpfXlWdLLt7o4+SZUdE5L22Imn+jpMuSpr//p40f+9+bM5LbyfNnxTLk+Y3KB6eNL8mv1HS/Mff+EPS/EveX5o0f8uCRUnze4wZmzS/ulG611a7W3b9zO3OAAQAAACADFMAAgAAAECGKQABAAAAIMMUgAAAAACQYQpAAAAAAMiwelUADhs2LG655ZZP3HbnnXfGkCFDjvCMAAAAAODoVq8KwM/yne98JxYvXlzX0wAAAACAo0p+XU/gcGnevHldTwEAAAAAjjr1+gzA5cuXR9euXWPWrFkHvAV41KhRMWLEiCgrK4uTTjopunXrFqNHj46KiorcPrt3745vfvOb0alTp+jVq1fcc889MWLEiBg1alRdHA4AAAAAHHb1tgBcuHBhXHXVVXHvvffGdddd94n7rF69Ol577bV47LHHYtasWbFo0aJ44IEHcttvu+22WLVqVcyZMycef/zx+PWvfx2rV68+UocAAAAAAMnVywJw9uzZ8Z3vfCd+9KMfxT/+4z9+6n7HHnts3HPPPdG7d+8499xz49JLL43ly5dHRER5eXnMmTMnpkyZEkOHDo2TTjoppk+fHnl5eUfqMAAAAAAguXr3GYBPPPFEzJo1K5544okYNGjQZ+7bu3fvyM//0yEWFhbGiy++GBERGzdujL1798aAAQNy25s1axZ9+vRJM3EAAAAAqAP17gzAvn37RmFhYfz4xz+Ompqaz9y3oKCg1nJeXl5uzMHGAgAAAEAW1LsCsFu3brFo0aJYunRpfPe73/2ri7wePXpEQUFBvPTSS7l1FRUV8eqrrx6uqQIAAABAnat3BWBERPfu3eMXv/hFLFmy5K8uAZs3bx5XXXVVTJo0KZYvXx6vv/56jBkzJmpqanwOIAAAAACZUe8+A/Bjxx9/fCxatCi+/OUvx9ixY6N9+/aHnDF16tTYvXt3XHnlldGsWbMYPXp0vPvuu9GkSZMEMwYAAACAI69eFYCLFy+utXz88cfHb37zm9zyrbfemrtcVlZ2wPjS0tIoLS3NLTdv3jweeuih3HJlZWWUlZXFBRdccDinDQAAAAB1pl4VgIfb2rVrY8OGDTFgwID48MMP4957743y8vL46le/WtdTAwAAAIDD4n90ARgRcd9998Ubb7wRDRs2jP79+8cTTzwRnTp1qutpAQAAAMBh8T+6ADzllFNi2bJldT0NAAAAAEgmb9euXYf+FboAAAAAQL3QoK4nAAAAAACkowAEAAAAgAxTAAIAAABAhikAAQAAACDDFIAAAAAAkGEKQAAAAADIMAUgAAAAAGSYAhAAAAAAMkwBCAAAAAAZpgAEAAAAgAxTAAIAAABAhikAAQAAACDDFIAAAAAAkGEKQAAAAADIMAUgAAAAAGSYAhAAAAAAMkwBCAAAAAAZpgAEAAAAgAxTAAIAAABAhikAAQAAACDDFIAAAAAAkGEKQAAAAADIMAUgAAAAAGRYfl1PoD5q1KhRktw33q9KkvuxXi2SxkfDD7YlzR86Z2vS/AeuGZA0/4SG7yfNf7tBm6T5HZqlfbr4oKo6aX7LRmn/v+P3e/Yny27/7tpk2RERzzU6MWn+3uqapPnFjXckzd/ZvGvS/LXby5Pmn7fv1aT5bxUOTJrf8b+eSpr/3S3HJ83/3xedkDT/9x/tS5q/Z3/ax++Mjqekzf/1rKT5r7U+PWl+Xl7S+Dih0e6k+c3Pvy1pfvmvJifNrz6mZdL8bXvSvjZJ/fs3tRaNGibLPiY/7YPr/cp0rwsjIrr8YUPS/KrCPknzV275MGn+mZ2PTZq/rXxv0vzfvv9R0vzzmryTND8apP27cd9rq5Pm/+cJX02af2r7Y5JlV1Z9dqfkDEAAAAAAyDAFIAAAAABkmAIQAAAAADJMAQgAAAAAGaYABAAAAIAMUwACAAAAQIYpAAEAAAAgwxSAAAAAAJBh9aYAHDZsWIwbNy4mTJgQ3bt3j6KioigrK4vKysr4/ve/H127do1+/frF3Llzc2MmT54cAwcOjMLCwujfv39MnDgx9uzZk9t+5513xpAhQ2L+/Plx6qmnRufOnWPkyJHx3nvv1cUhAgAAAMBhV28KwIiIefPmRfPmzWPJkiUxduzYKC0tjZKSkigqKoply5bFFVdcEWPGjIlt27ZFRETTpk1jxowZsWbNmpg2bVosWLAg7r777lqZmzdvjgULFsScOXNiwYIFsW7dupg6dWpdHB4AAAAAHHb1qgA88cQTo7S0NIqKiuKmm26KNm3aRH5+fowaNSp69OgRP/jBD6Kmpiaef/75iIgYP358DB48OLp16xYXXnhhjBs3LubPn18rc9++fXH//fdHv379YtCgQXHttdfG8uXL6+LwAAAAAOCwy6/rCRyKvn375i7n5eVF27Zta60rKCiIVq1axY4dOyIiYuHChVFWVhZvvvlm7N69O/bv3x/79++vldmlS5do2bJlbrmwsDB27tyZ+EgAAAAA4MioV2cAFhQU1FrOy8uL/Pz8A9ZVV1fHCy+8ENdff32ce+65MXfu3HjmmWdiwoQJsXfv3oNmVldXpzkAAAAAADjC6tUZgIfiueeeiw4dOsT48eNz67Zs2VKHMwIAAACAIy+zBWDPnj1j27Zt8eijj8agQYNiyZIlB3z+HwAAAABkXb16C/ChuPjii2PMmDFRWloaZ511VixdujRuvfXWup4WAAAAABxR9eYMwMWLFx+wbvXq1Qes27BhQ+7ypEmTYtKkSbW233DDDbnLpaWlUVpaWmt7SUlJlJSU/K3TBQAAAICjQmbPAAQAAAAAFIAAAAAAkGkKQAAAAADIsLxdu3bV1PUkAAAAAIA0nAEIAAAAABmmAAQAAACADFMAAgAAAECGKQABAAAAIMMUgAAAAACQYQpAAAAAAMgwBSAAAAAAZJgCEAAAAAAyTAEIAAAAABmmAAQAAACADFMAAgAAAECGKQABAAAAIMMUgAAAAACQYQpAAAAAAMgwBSAAAAAAZJgCEAAAAAAyTAEIAAAAABmmAAQAAACADFMAAgAAAECGKQABAAAAIMMUgAAAAACQYQpAAAAAAMgwBSAAAAAAZFh+XU+gPmrcME3u8rf3pAn+b+c3eitpfk1+k6T5F/7890nzO7Y+Jmn+rPNaJM1f9dHfJc1v36xR0vwejT9Kmr+9ulnS/I5b1yTLfrP9GcmyIyK2l1clzd+0K+2/7RWd9ibNr37t2aT560/4StL87i3TPnZ/v2df0vzO5W8mzR88852k+cv/1zlJ85du+iBpfmqrTj0zaf7ta+5Lmv9mx7OS5jctSPt/9e33p31t1fkbP0ua/9a/jkyaHzXVSeM/atwqaX753rTzb9QwL2l+44bp7v/7qmuSZUdE7KhI+7vx+I1LkubX9E773Lzho7R/N1btT3vf79G6cdL8V3ekfe18xralSfNr+nwxaX75nP+bNH/TJbcmze97XLr7f2XVZ/9d5wxAAAAAAMgwBSAAAAAAZJgCEAAAAAAyTAEIAAAAABmmAAQAAACADFMAAgAAAECGKQABAAAAIMOSFID//u//Hp07d459+/ZFRMRvf/vbaNWqVYwbNy63z9SpU+PSSy+NiIhVq1bFeeedF+3bt49evXpFaWlpVFVV5fYdNmxYjBs3LiZMmBDdu3ePoqKiKCsri8rKyvj+978fXbt2jX79+sXcuXNrzWPy5MkxcODAKCwsjP79+8fEiRNjz549ue133nlnDBkyJObPnx+nnnpqdO7cOUaOHBnvvfdeipsFAAAAAI64JAXgkCFDYs+ePfHyyy9HRMTKlSujTZs2sWLFitw+K1eujOLi4ti6dWtcfvnlcfLJJ8czzzwT06dPj/nz58eUKVNqZc6bNy+aN28eS5YsibFjx0ZpaWmUlJREUVFRLFu2LK644ooYM2ZMbNu2LTemadOmMWPGjFizZk1MmzYtFixYEHfffXet3M2bN8eCBQtizpw5sWDBgli3bl1MnTo1xc0CAAAAAEdckgKwefPmccopp+QKv5UrV8Y3vvGN2LJlS7zzzjtRUVERL730UhQXF8fMmTOjffv2MW3atOjdu3dcdNFFMWnSpHj44YejoqIil3niiSdGaWlpFBUVxU033RRt2rSJ/Pz8GDVqVPTo0SN+8IMfRE1NTTz//PO5MePHj4/BgwdHt27d4sILL4xx48bF/Pnza8113759cf/990e/fv1i0KBBce2118by5ctT3CwAAAAAcMQl+wzA4uLiWLlyZUT88S2+F1xwQZx++umxcuXKWLNmTRQUFMSAAQNi/fr1ccYZZ0SDBn+aypAhQ6KqqirefPPN3Lq+ffvmLufl5UXbtm1rrSsoKIhWrVrFjh07cusWLlwYF110UZxwwgnRqVOnuPXWW+Ott96qNc8uXbpEy5Ytc8uFhYWxc+fOw3dDAAAAAEAdSloArlmzJl5//fUoLy+PU089NYqLi2PFihWxcuXKGDRoUBQUFERNTU3k5eV9Ysafry8oKDhgW35+/gHrqqurIyLihRdeiOuvvz7OPffcmDt3bjzzzDMxYcKE2Lt3b60xn5T7cQYAAAAA1Hf5B9/lrzNkyJCorKyMe++9NwYPHhwNGzaM4uLiGDt2bLRt2zYuuOCCiPjjW3t//vOfR3V1de4swNWrV0ejRo3i+OOP/6uv/7nnnosOHTrE+PHjc+u2bNnytx0UAAAAANQzyc4A/PhzAB999NE4++yzIyJi0KBB8fbbb8eLL74YxcXFERFxww03xDvvvBPf+973Yv369fHkk0/GlClT4hvf+EY0bdr0r77+nj17xrZt2+LRRx+NTZs2xcyZMw/4/D8AAAAAyLpkBWBExNlnnx379+/PlX1NmjSJgQMHRuPGjWPAgAEREdGxY8eYN29erFu3Ls4+++y46aab4rLLLouJEyf+Tdd98cUXx5gxY6K0tDTOOuusWLp0adx6661/8zEBAAAAQH2S7C3AERGTJ0+OyZMn11q3ePHiA/Y766yzYsmSJZ+a80ljVq9efcC6DRs21FqeNGlSTJo0qda6G264IXe5tLQ0SktLa20vKSmJkpKST50LAAAAANQnSc8ABAAAAADqlgIQAAAAADJMAQgAAAAAGaYABAAAAIAMy9u1a1dNXU8CAAAAAEjDGYAAAAAAkGEKQAAAAADIMAUgAAAAAGSYAhAAAAAAMkwBCAAAAAAZpgAEAAAAgAxTAAIAAABAhikAAQAAACDDFIAAAAAAkGEKQAAAAADIMAUgAAAAAGSYAhAAAAAAMkwBCAAAAAAZpgAEAAAAgAxTAAIAAABAhikAAQAAACDDFIAAAAAAkGEKQAAAAADIMAUgAAAAAGSYAhAAAAAAMkwBCAAAAAAZpgAEAAAAgAxTAAIAAABAhuXX9QTqo/Lqhkly2+7flSQ3540Xksb/8tjBSfO/3OC/kuY/vr9X0vzjmhYkzW/+/ZKk+fEvP00af8mEx5PmF53WPWn+v151WrLsjo33JcuOiNjXsEnS/IZRnTT/P9+tTJpf0DAvaf7qLbuS5n/9tMKk+Q1eXpw0P3oOTBqft68qaf7TH7ZMmn/evleT5u/fviVp/r7BlyfN/6Byf9L8h198O2n+4K6tkuafW/FS0vyzfpHmNfPHVvz9e0nz9w68NGl+fnXa55/ntu9Nmv93x6R9bdv52HT5Nz/+WrLsiIjpl/ZJmr/pg7T3nWYFac8T6vbRpqT5K/d2SJr//kdpH1s7K9L++17b6aOk+YveT/vaZ1CnY5Pmt6t4K2l+g/J0v7vKC/t99nUnu2YAAAAAoM4pAAEAAAAgwxSAAAAAAJBhCkAAAAAAyLCjtgAcNmxY3HLLLUmyR40aFSNGjEiSDQAAAABHkzovAFesWBGtWrWK9947/N+E8mnZP/zhD+PBBx887NcHAAAAAEeb/LqeQF1o2TLt11IDAAAAwNHic50BuGrVqjj//POjU6dO0bVr1zjvvPPilVdeiS5dusTChQtr7bt06dI47rjj4t13343f/e530apVq1i4cGFceuml0aFDh/jCF74QS5cujYiI3/3ud/GVr3wlIiKKioqiVatWMWrUqFxWdXV13H777dGjR4/o2bNn3HbbbVFdXZ3bXlVVFZMmTYo+ffpEx44dY+jQobFkyZKDZv/lW4Brampi+vTpcfrpp0e7du2iT58+MWXKlEO+MQEAAADgaHPQAnDfvn0xcuTIGDx4cKxcuTKeeuqp+Na3vhWNGjWKyy67LObMmVNr/zlz5sSXvvSlaNeuXW7dHXfcEd/85jdj5cqVcdppp8X1118f5eXl0blz53jkkUciIuK5556L9evXxw9/+MPcuHnz5kXDhg3jV7/6Vdx1111RVlYWCxYsyG3/9re/HatWrYqHH344nn322bjyyivjiiuuiFdeeeWg2X/u9ttvj7vuuituvvnmeO6552L27NnRqVOnQ7gZAQAAAODodNC3AH/44YfxwQcfxEUXXRTHH398RESccMIJERHxta99Lc4///zYunVrdOzYMXbt2hWLFy+O2bNn18oYPXp0XHzxxRERMXHixJg7d2688sorMWTIkGjdunVERLRt2zbatGlTa1zv3r1jwoQJERHRs2fP+NGPfhTLly+Pf/qnf4qNGzfGz372s1i3bl106dIlIiJuvPHGWLZsWcyePTumTZv2mdkfKy8vj/vvvz/uvPPOuPrqqyMiokePHjFo0KCD33oAAAAAcJQ76BmArVu3jpEjR8Zll10Ww4cPjxkzZsRbb70VERGnnXZa9OnTJ/7t3/4tIv54xl6rVq3iggsuqJXRt2/f3OUOHTpERMSOHTsOOrk/HxcRUVhYmBu3du3aqKmpicGDB0enTp1yP7/61a9i48aNB83+2Pr166OysjK++MUvfu4xAAAAAFBffK7PALz//vvjqaeeijPPPDN++ctfxsCBA3OftXfNNdfET37yk4j449t/R44cGQ0bNqw1vqCgIHc5Ly8vIv74uXsH8+fjPh778bjq6urIy8uLp59+OlasWJH7ef7552PGjBmf57A+9zwAAAAAoL76XAVgRET//v1j7NixsXjx4iguLs6d9Td8+PDYtm1bPPTQQ7F27dooKSk5pAk0atQoIiL2799/SONOPvnkqKmpie3bt0ePHj1q/XTs2PFzZ/fu3TsaN24cy5cvP6TrBwAAAID64KAF4KZNm2Ly5MmxZs2a2Lx5czzzzDPxm9/8Jnr37h0RES1btoxLLrkkbrvttjjzzDOjqKjokCbQpUuXyMvLiyeffDJ27twZ5eXln2tcz549Y/jw4TF69OhYuHBhbNq0KV5++eWYPn16PP744587+9hjj41vfetbMWXKlJgzZ05s3Lgx/uM//iNmzpx5SMcBAAAAAEejgxaATZs2jTfeeCOuvfbaGDhwYIwePTouv/zyGDt2bG6fq6++OqqqqnJfonEoOnbsGKWlpXHHHXdEr1694pZbbvncY++7774oKSmJiRMnxhlnnBEjRoyIVatWRdeuXQ8pe9KkSTF27Ni46667YtCgQXHNNdfE1q1bD/lYAAAAAOBoc9BvAW7Xrl3MmTPnM/fZvn17tGjRIi655JJa67t16xa7du06YP+/XDd+/PgYP358rXWLFy8+YFxZWVmt5YKCgigtLY3S0tJPndsnZf9lToMGDeLmm2+Om2+++VNzAAAAAKA+OmgB+FkqKipi8+bNMW3atPja174WTZs2PVzzAgAAAAAOg8/9JSCf5N57743i4uJo3br1Ib11FwAAAAA4Mv6mArC0tDR27twZixYtihYtWhyuOQEAAAAAh8nfVAACAAAAAEe3vF27dtXU9SQAAAAAgDScAQgAAAAAGaYABAAAAIAMUwACAAAAQIYpAAEAAAAgwxSAAAAAAJBhCkAAAAAAyDAFIAAAAABkmAIQAAAAADJMAQgAAAAAGaYABAAAAIAMUwACAAAAQIYpAAEAAAAgwxSAAAAAAJBhCkAAAAAAyDAFIAAAAABkmAIQAAAAADJMAQgAAAAAGaYABAAAAIAMUwACAAAAQIYpAAEAAAAgwxSAAAAAAJBhCkAAAAAAyDAFIAAAAABkWH5dT6A+evujNLkl059NE/zfnu76TNL8Y66ZmDR/4wd7k+Zv312VNH9w+4Kk+fM2/CFp/ldPPC5pfqP1y5Lmv921OGn+wtd3JMvu3+7YZNkREcWtK5Pmb65ukTR/W3nix+4HLyTN31V0TtL8v3vv9aT5b7fonTT/dx/sSZrfpmna5+YeLdK+1Hp7d3XS/NTKR1+eNP+Ea76cNL/B35ckzc9/b1PS/I/anZg0/ysPpX3+nD7ilKT5vZqkff7ZVHVM0vyKvWmfH9o1S/v8tuWDdL/f+7dLe9sXrPt/SfP373g7af7rp12VNL9Vk4ZJ8zs0SXvfz//tmqT5+3uckTR/W1Xax26XD99Imr/wC2nvn0N/m/Z3V4t3f5Msu7zNCZ+53RmAAAAAAJBhCkAAAAAAyDAFIAAAAABkmAIQAAAAADJMAQgAAAAAGaYABAAAAIAMUwACAAAAQIbVWQFYUVERo0aNik6dOkWvXr1i2rRpMWLEiBg1alRERPTv3z+mT59ea8ywYcPilltuyS1XVVXFpEmTok+fPtGxY8cYOnRoLFmypNaY119/PYYPHx6dO3eOnj17xg033BDbt2/PbR81alSMGDEiysrK4qSTTopu3brF6NGjo6KiIuHRAwAAAMCRUWcF4D//8z/HsmXL4pFHHomFCxfGunXr4tlnnz2kjG9/+9uxatWqePjhh+PZZ5+NK6+8Mq644op45ZVXIiLinXfeiX/4h3+Ik046KZYsWRKPPfZYlJeXx5VXXhnV1dW5nNWrV8drr70Wjz32WMyaNSsWLVoUDzzwwGE9XgAAAACoC/l1caXl5eXx4x//OGbMmBHnnXdeRETcd9990adPn8+dsXHjxvjZz34W69atiy5dukRExI033hjLli2L2bNnx7Rp02LmzJnRr1+/mDJlSm7cgw8+GN27d4+XX345BgwYEBERxx57bNxzzz2Rn58fvXv3jksvvTSWL18e48aNO4xHDQAAAABHXp0UgBs3boyqqqoYNGhQbl3z5s2jb9++nztj7dq1UVNTE4MHD661vrKyMs4555zcPs8++2x06tTpE+fwcQHYu3fvyM//001RWFgYL7744iEdEwAAAAAcjeqkAKypqTnoPg0aNDhgv3379uUuV1dXR15eXjz99NNRUFBQa78mTZrk9rnwwgvjjjvuOCC/bdu2uct/OT4vL+9zzREAAAAAjnZ1UgD26NEjCgoK4oUXXoju3btHRMTu3bvj1VdfzS0fd9xx8c477+TG7NmzJzZs2BAnn3xyREScfPLJUVNTE9u3b8+d8feXTjnllPj5z38eXbp0OaDkAwAAAID/CerkS0CaN28eV199dUyePDmWLl0ar732Wtx00021vpjjnHPOiXnz5sWKFSty2//8DMCePXvG8OHDY/To0bFw4cLYtGlTvPzyyzF9+vR4/PHHIyLi61//evzhD3+I6667Ll588cXYtGlTLFu2LL773e/Ghx9+eMSPGwAAAACOtDo5AzAiYurUqbF79+646qqr4phjjokbb7wxKioqcttvvvnm2Lx5c5SUlESzZs3ie9/7Xmzbtq1Wxn333Rd33313TJw4MbZu3RqtW7eO008/Pc4+++yIiOjQoUM8+eSTMWXKlLjsssuisrIyOnfuHEOHDo3GjRsf0eMFAAAAgLpQZwVgs2bN4sEHH6y1bsSIEbnLLVq0iJkzZ9ba/vWvf73WckFBQZSWlkZpaemnXk9RUVE88sgjn7q9rKzsgHUHywQAAACA+qJO3gIMAAAAABwZCkAAAAAAyLA6ewvwJ/npT39a11MAAAAAgExxBiAAAAAAZFjerl27aup6EgAAAABAGs4ABAAAAIAMUwACAAAAQIYpAAEAAAAgwxSAAAAAAJBhCkAAAAAAyDAFIAAAAABkmAIQAAAAADLs/wMOb5WTh9NgXgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1296x252 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plot_word_vectors(glove, \n",
    "                        ['king', 'man', 'woman', 'synthetic', 'queen'],\n",
    "                        other={'synthetic': synthetic_queen})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('king', 0.8859835863113403),\n",
       " ('queen', 0.8609581589698792),\n",
       " ('daughter', 0.7684512138366699),\n",
       " ('prince', 0.7640699148178101),\n",
       " ('throne', 0.7634971141815186)]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glove.similar_by_vector(synthetic_queen, topn=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/embed_arithmetic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\Large\n",
    "w_{\\text{king}} - w_{\\text{man}}\\approx w_{\\text{queen}}-w_{\\text{woman}} \\implies w_{\\text{king}} - w_{\\text{man}} + w_{\\text{woman}} \\approx w_{\\text{queen}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocabulary Coverage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3706"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = list(dictionary.token2id.keys())\n",
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44\n",
      "['[PAD]', '[UNK]', 'arrum', 'barrowful', 'beauti']\n"
     ]
    }
   ],
   "source": [
    "unknown_words = sorted(list(set(vocab).difference(set(glove.vocab))))\n",
    "print(len(unknown_words))\n",
    "print(unknown_words[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(82, 50802)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unknown_ids = [dictionary.token2id[w] for w in unknown_words if w not in ['[PAD]', '[UNK]']]\n",
    "unknown_count = np.sum([dictionary.cfs[idx] for idx in unknown_ids])\n",
    "unknown_count, dictionary.num_pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vocab_coverage(gensim_dict, pretrained_wv, special_tokens=('[PAD]', '[UNK]')):\n",
    "    vocab = list(gensim_dict.token2id.keys())\n",
    "    unknown_words = sorted(list(set(vocab).difference(set(pretrained_wv.vocab))))\n",
    "    unknown_ids = [gensim_dict.token2id[w] for w in unknown_words if w not in special_tokens]\n",
    "    unknown_count = np.sum([gensim_dict.cfs[idx] for idx in unknown_ids])\n",
    "    cov = 1 - unknown_count / gensim_dict.num_pos\n",
    "    return cov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9983858903192788"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_coverage(dictionary, glove)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_vocab_from_wv(wv, folder=None, special_tokens=None):\n",
    "    if folder is not None:\n",
    "        if not os.path.exists(folder):\n",
    "            os.mkdir(folder)\n",
    "\n",
    "    words = wv.index2word\n",
    "    if special_tokens is not None:\n",
    "        to_add = []\n",
    "        for special_token in special_tokens:\n",
    "            if special_token not in words:\n",
    "                to_add.append(special_token)\n",
    "        words = to_add + words\n",
    "                \n",
    "    with open(os.path.join(folder, 'vocab.txt'), 'w') as f:\n",
    "        for word in words:\n",
    "            f.write(f'{word}\\n')    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_vocab_from_wv(glove, 'glove_vocab/', special_tokens=['[PAD]', '[UNK]'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_tokenizer = BertTokenizer('glove_vocab/vocab.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7101, 930, 2, 300, 12427]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glove_tokenizer.encode('alice followed the white rabbit', add_special_tokens=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400002, 400000)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(glove_tokenizer.vocab), len(glove.vectors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Special Tokens' Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "special_embeddings = np.zeros((2, glove.vector_size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400002, 50)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extended_embeddings = np.concatenate([special_embeddings, glove.vectors], axis=0)\n",
    "extended_embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alice_idx = glove_tokenizer.encode('alice', add_special_tokens=False)\n",
    "np.all(extended_embeddings[alice_idx] == glove['alice'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model I - GloVe + Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_sentences = train_dataset['sentence']\n",
    "train_labels = train_dataset['labels']\n",
    "\n",
    "test_sentences = test_dataset['sentence']\n",
    "test_labels = test_dataset['labels']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ids = glove_tokenizer(train_sentences, \n",
    "                            truncation=True,\n",
    "                            padding=True, \n",
    "                            max_length=60, \n",
    "                            add_special_tokens=False, \n",
    "                            return_tensors='pt')['input_ids']\n",
    "train_labels = torch.as_tensor(train_labels).float().view(-1, 1)\n",
    "\n",
    "test_ids = glove_tokenizer(test_sentences, \n",
    "                           truncation=True,\n",
    "                           padding=True, \n",
    "                           max_length=60, \n",
    "                           add_special_tokens=False, \n",
    "                           return_tensors='pt')['input_ids']\n",
    "test_labels = torch.as_tensor(test_labels).float().view(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_tensor_dataset = TensorDataset(train_ids, train_labels)\n",
    "generator = torch.Generator()\n",
    "train_loader = DataLoader(train_tensor_dataset, batch_size=32, shuffle=True, generator=generator)\n",
    "test_tensor_dataset = TensorDataset(test_ids, test_labels)\n",
    "test_loader = DataLoader(test_tensor_dataset, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-Trained PyTorch Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "extended_embeddings = torch.as_tensor(extended_embeddings).float()\n",
    "torch_embeddings = nn.Embedding.from_pretrained(extended_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  36,   63,    1,  ...,    0,    0,    0],\n",
       "        [ 934,   16,   14,  ...,    0,    0,    0],\n",
       "        [  57,  311,    8,  ...,  140,    3,   83],\n",
       "        ...,\n",
       "        [7101,   59, 1536,  ...,    0,    0,    0],\n",
       "        [  43,   59, 1995,  ...,    0,    0,    0],\n",
       "        [ 102,   41,  210,  ...,  685,    3,    7]])"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_ids, labels = next(iter(train_loader))\n",
    "token_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([32, 60, 50])"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_embeddings = torch_embeddings(token_ids)\n",
    "token_embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0665, -0.0071, -0.0534,  ..., -0.1906, -0.0202, -0.1432],\n",
       "        [ 0.0514,  0.0495,  0.0083,  ..., -0.0272,  0.0162,  0.0687],\n",
       "        [ 0.2431,  0.1981, -0.0282,  ..., -0.2559, -0.0252,  0.0176],\n",
       "        ...,\n",
       "        [-0.0018,  0.0598,  0.0083,  ..., -0.0792,  0.0222,  0.0261],\n",
       "        [ 0.0516,  0.1091,  0.0917,  ..., -0.0851,  0.0037,  0.0553],\n",
       "        [ 0.1972,  0.1069, -0.2049,  ..., -0.4969, -0.1026, -0.3731]])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_embeddings.mean(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0665, -0.0071, -0.0534,  ..., -0.1906, -0.0202, -0.1432],\n",
       "        [ 0.0514,  0.0495,  0.0083,  ..., -0.0272,  0.0162,  0.0687],\n",
       "        [ 0.2431,  0.1981, -0.0282,  ..., -0.2559, -0.0252,  0.0176],\n",
       "        ...,\n",
       "        [-0.0018,  0.0598,  0.0083,  ..., -0.0792,  0.0222,  0.0261],\n",
       "        [ 0.0516,  0.1091,  0.0917,  ..., -0.0851,  0.0037,  0.0553],\n",
       "        [ 0.1972,  0.1069, -0.2049,  ..., -0.4969, -0.1026, -0.3731]])"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "boe_mean = nn.EmbeddingBag.from_pretrained(extended_embeddings, mode='mean')\n",
    "boe_mean(token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Configuration & Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "extended_embeddings = torch.as_tensor(extended_embeddings).float()\n",
    "boe_mean = nn.EmbeddingBag.from_pretrained(\n",
    "    extended_embeddings, mode='mean'\n",
    ")\n",
    "torch.manual_seed(41)\n",
    "model = nn.Sequential(\n",
    "    # Embeddings\n",
    "    boe_mean,\n",
    "    # Classifier\n",
    "    nn.Linear(boe_mean.embedding_dim, 128), \n",
    "    nn.ReLU(),\n",
    "    nn.Linear(128, 1)\n",
    ")\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "sbs_emb = StepByStep(model, loss_fn, optimizer)\n",
    "sbs_emb.set_loaders(train_loader, test_loader)\n",
    "sbs_emb.train(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAEQCAYAAAC++cJdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABwc0lEQVR4nO3dd3hT1RvA8e/N7KbMll2hrCKVvZQhW5EhQ6YgiGwVf6CCLEUUEAcqQ2UJAoJlyBIREGQjKktQhlCG7EKhK826vz8iKemiLel+P8+TJ73nrpPDTXhz8t5zlMjISBUhhBBCCCHyCU12V0AIIYQQQoisJAGwEEIIIYTIVyQAFkIIIYQQ+YoEwEIIIYQQIl+RAFgIIYQQQuQrEgALIYQQQoh8RQJgIYQQQgiRr0gALIQQQggh8hUJgPO406dPZ3cVcjRpn9RJ+zyYtFHqpH1SJ+3zYNJGqZP2yRgJgIUQQgghRL4iAbAQQgghhMhXJAAWQgghhBD5igTAQgghhBAiX9FldwWEEEIIkfdYrVZiYmIe+jgeHh7cuXPHDTXKm/Jz+3h7e6PTZSyUlQBYCCGEEG5ltVqJiorC398fRVEe6lhGoxEPDw831Szvya/to6oqkZGR+Pr6ZigIlhSIPCw6GjZtKsTixfrsrooQQoh8JCYmxi3BrxApURQFf3//DP/KID3AedDVqwpvveXBpk164uIKEBhop1cvC1ptdtdMCCFEfiHBr8hsD3ONSQ9wHlSggMqWLXri4hwXxtWrGnbvluhXCCGEEAIkAHar7t27U7ZsWfr06ZOt9fD0hLZtLS5lK1casqk2QgghhBA5iwTAbjR06FC++OKL7K4GAF27ugbAa9fqiY/PpsoIIYQQ+dSQIUPo1q1buvZp27Ytr7/+eibVSIDkALtV48aN2bVrV3ZXA4AmTawUKWLn5k3Hd5y7dxW2btXRtq01m2smhBBC5Dz+/v6pru/Rowdz5sxJ93GnTp2Kqqrp2mfJkiUZHt4rPaZMmcK6devYt29fpp8rp8nWHuApU6bg7+/v8qhYsaLbz7Nnzx66d+9OlSpV8Pf3Z+nSpcluN2/ePEJDQwkICKBJkybs3bvX7XXJKjodPPts4jQIGQ1CCCGESM7Jkyedj88++yxJ2dSpU122t1gsyR0miQIFCjwwuE6sYMGC+Pr6pmsfkT7ZngJRoUIFlwsstaDzwIEDxCfzO354eDjnz59Pcb+YmBhCQkKYOnUqnp6eyW6zevVqRo8ezciRI9m5cyd169ala9euXLx40blNgwYNkn1cunQpHa8463Tp4vrm3LRJT1RUNlVGCCGEyMECAgKcjwIFCriUmUwmypYty8qVK2nXrh2BgYEsXLiQW7du8eKLLxISEkJgYCD169dnyZIlLsdNnALRtm1bRo4cyaRJkyhXrhzBwcGMGzcOu93uss39KRDVqlVj+vTpjBgxgtKlSxMSEuIM0u85c+YMTz/9NAEBAdSuXZuffvqJkiVLptjplxbHjx+nQ4cOBAYGEhQUxJAhQ1wm3Th+/Djt27endOnSlCpViscff5ydO3cCji8Ib7zxBpUrV6ZYsWJUrVqVt99+O8N1cbdsT4HQ6XQEBAQ8cDtVVXnjjTcIDAxkyZIl6PWO3syLFy/Srl07nn32WSZNmpTsvq1ataJVq1aAI083ObNmzaJnz5707dsXgOnTp7Nt2zYWLFjAxIkTAXLdTwR169ooUSKey5eNAJhMChs36unePW3fWoUQQgh38fcvkME9M7ZfZKT7Z0d75513mDx5Mp9//jl6vR6TycRjjz3Gq6++ip+fHzt27OC1116jdOnSNGnSJMXjhIWFMWjQIH766SeOHTvGgAEDqF69Ol26dElxn9mzZzNmzBheeeUVtmzZwptvvkn9+vUJDQ3FbrfTu3dvihUrxpYtWzCZTIwZMybZTsO0io2NpUuXLtSoUYNt27Zx+/ZtXn31VYYPH84333wDwEsvvcSjjz7Ktm3b0Ol0HD9+3DkpxxdffMHGjRuZP38+ZcqU4fLly5w+fTrD9XG3bO8BDg8Pp0qVKoSGhtK/f3/Cw8OT3U5RFMLCwjh79iz9+vXDarXy77//0q5dO+rUqeMMUjPCbDZz+PBhmjVr5lLerFkzDhw4kOHjZjdFgVatbrmUSRqEEEIIkTEDBw6kQ4cOBAUFUbJkSUqUKMErr7xCaGgoQUFBvPDCC7Rr146VK1emepxKlSoxduxYgoODefbZZ2nUqBG//PJLqvs0a9aMgQMHUq5cOQYNGkS5cuWc+2zfvp3Tp0/zxRdfEBoaSt26dXn//fexWjN+309YWBgxMTF8+eWXVK1alSeeeIIZM2awfv16zp49Czg6IZs2bUrFihUpV64c7dq1o27dus515cuXp2HDhpQuXZp69erRu3fvDNfH3bI1AK5duzazZ88mLCyMzz77jGvXrtGqVStu3bqV7PbFihVj3bp1nDhxghdeeIH27dtTrVo1vvrqK7QPMctDREQENpuNokWLupQXLVqU69evp/k4HTp04IUXXmDLli2EhITw66+/ZrhO7tK6dYTL8vbtOm7elMHJhRBCiPSqUaOGy7LNZuPDDz+kYcOGPPLII5QsWZL169c/MDWyatWqLsuBgYHcuHEjw/ucOnWK4sWLU6JECef6mjVrotFkPMw7efIkVatWdclFrlevHhqNhr///htw/Kr+yiuv0K5dOz788ENOnTrl3LZnz54cO3aMWrVqMWrUKDZv3uyS5pHdsjUAbtmyJc8++yyPPvooTZs2ZcWKFdjtdpYtW5biPsWLF2fBggVs2LCBmJgY5s6d67Y7JRPPKKKqarpmGVm7di3//PMPV65c4cSJE85vQdkpONhESIjNuWyzKXz/vfQCCyGEEOnl7e3tsvz5558zc+ZMXnnlFdauXcuuXbto27YtZrM51ePcS+O8R1GUB44Ukdo+6R1lIi1SO+a92GjMmDEcOHCAtm3b8uuvv/L444870yOqV6/O0aNHmTBhAna7nSFDhtCxY8ccEwRnew7w/Xx8fKhcubKzaz05ERERDB06lBYtWvD333/z2muvMWvWrIf6llO4cGG0Wm2S3t6bN28m6RXOjbp0sTBpUkIP+cqVegYMSP3NKYQQQrhTRnNyTSaTM680p9m3bx9t2rShe/fugCNoPHPmjPMmuqxSqVIlrly5wpUrVyhevDgAhw4deqhgs3LlyixdupSoqChnL/CBAwew2+1UqlTJuV358uUpX748gwcP5n//+x/ffPMNzz//PAC+vr507NiRjh070rNnT1q0aMHZs2cJDg5+iFfrHtmeA3w/k8nE6dOnU7wp7vbt23Ts2JHixYuzdOlSNmzYwM6dO3n11Vcf6tuPwWCgevXqbN++3aV8+/bt1KtXL8PHzSk6dXINdvfv13H+vKRBCCGEEA8jODiYnTt3sm/fPk6dOsXrr7/OhQsXsrweTz75JBUqVGDIkCEcO3aMgwcPMnbsWHQ63QN/yTaZTBw9etTlcebMGbp27YqXlxeDBw/m+PHj7Nmzh9dee4127dpRrlw54uLiGDVqFLt27eL8+fP89ttv7N+/3xkcz5w5k5UrV3Ly5EnOnj1LWFgYfn5+Lmka2SlbA+Bx48axe/duwsPD+e233+jbty+xsbH06NEjybaqqvLcc89RqFAhli5ditFoJCgoiHXr1rF161YmT56c4nmio6Od/6h2u51Lly5x9OhRlyHOhg0bxrJly1i8eDEnT57kzTff5OrVq/Tr1y9TXntWCgpSqVfPNRF+9WqZGlkIIYR4GK+//jo1a9aka9euPP3003h5edG1a9csr4dGo2HJkiXEx8fTvHlzhgwZwsiRI1EU5YG95+fOnaNx48YujwEDBuDl5cWqVauIioqiefPm9OzZkzp16jBz5kwAtFotkZGRDBkyhDp16tC7d2/q1KnDe++9Bzh6fz/77DOaN29OkyZNOHbsGGFhYXh5eWV6e6SFEhkZ6f7EkTTq378/e/fuJSIigiJFilC7dm3Gjh1L5cqVk91+x44d1K1bN0njnTp1Co1Gk2KX+q5du2jXrl2S8sSzusybN49PP/2Ua9euUaVKFd5//30ef/zxh3iF2e/06dNUqFCBuXMNvP56whjIVava2LMnOhtrljPcax+RPGmfB5M2Sp20T+ryavvcuXPHbWkAOTkFIidIqX2OHTtGo0aN2LFjB9WrV8/6imWRjF5r2RoAi8x378P1xg2FypV9sdkSfgrZuzeKkJCckYyeXfLqfz7uIu3zYNJGqZP2SV1ebR8JgLPOvfZZv3493t7elCtXjgsXLjB27FhUVWXXrl3puqE/t8notZajcoBF5ilaVKVpU9c0iFWrZDQIIYQQIi+Ijo7m9ddfp379+gwcOJBKlSqxevXqPB38PowcNQqEyFxduljYti0h6A0LMzBuXDzy3hBCCCFytx49eiR7D5VInvQA5yNt21rw8EjIeLlwQcPBgxmfQEQIIYQQIjeSADgf8fODNm0sLmUyNbIQQggh8hsJgPOZzp1dA+A1a/Q8xFThQgghhBC5jgTA+UzLllb8/BLSIG7c0LBzp6SCCyGEECL/kAA4n/HwgPbtXXuBw8IkDUIIIYQQ+YcEwPlQ166uUyNv2KAnLi6bKiOEEEIIkcUkAM6HnnjCRkBAwgQYUVEKP/0kaRBCCCHEw5oyZQoNGjRIcTk5r7/+Om3btnX7uUXKJADOh7RaePbZxKNBGLKpNkIIIUT269atGx06dEh23cmTJ/H392f79u3pPu7LL7/Mxo0bH7Z6Ls6fP4+/vz+HDh3K9HMlZ8iQIXTr1i3Tz5OZJADOp7p2dQ2Af/pJR2Rk9tRFCCGEyG59+vRh586dnD9/Psm6b775htKlS9OkSZN0H9fHx4dChQq5o4o56ly5nQTA+VTNmjYeecTmXI6PV9iwQW6GE0IIkT+1bt2aYsWKsXTpUpdyi8XCihUr6N27N6qqMnz4cEJDQwkMDKRmzZp8+umn2O32FI6aNC3BZrMxbtw4ypYtS9myZRk9ejQ2m81ln61bt/LUU09RtmxZgoKC6NSpEydPnnSuf+yxxwB48sknCQwMdKZPJD6X3W7ngw8+oGrVqhQrVoyGDRu69BDf60leu3YtHTt2pHjx4tSrVy9DPd3327NnD82bNycgIIAKFSowZswYzGazy/oWLVpQsmRJypQpQ/PmzTlx4gQAd+7cYeDAgQQHBxMQEMBjjz3G7NmzH6o+yZEAOJ9SFMfUyPeTSTGEEEJklgL+/hl6BAQGZmi/9NLpdPTo0YNly5a5BLSbNm0iIiKCXr16YbfbKV68OF9//TUHDhxg/PjxfPTRRyxZsiTN55k5cyaLFy9mxowZbNmyBZvNRlhYmMs2MTExDB48mJ9//pkNGzbg5+dH9+7dnUHkzz//DMCqVas4evRoiuefM2cOn3/+OW+//TZ79+6lbdu2PP/88xw9etRlu8mTJzNo0CB2795NjRo16N+/P9HR0Wl+Tfe7fPkyXbt2JTQ0lJ07d/L555+zatUq3nnnHQCsVis9e/akfv367N69m61btzJ48GC0Wq2zLidOnGDFihX8+uuvzJw5kxIlSmSoLqmRO5/ysS5dLEyf7uFc3rlTx7VrCgEBaip7CSGEEHnT888/z4wZM9ixYwfNmjUDYMmSJTRr1oxSpUoBMHbsWOf2ZcuW5ciRI6xatYo+ffqk6Rxz5szhlVde4dlnnwVg2rRpzoD2nsS5yLNmzaJ06dL8/vvvNGjQgMKFCwNQqFAhihUrhoeHB8mZOXMmw4cPp2vXrs667927l5kzZ/LVV185txs6dChPPfUUABMmTGD58uUcO3YsQzfUzZ8/n4CAAD766CM0Gg2VKlVi4sSJvPbaa4wdO5b4+Hju3LlDmzZteOSRRwCoWLGic/+LFy8SGhpKrVq1AEcbZwbpAc7HKlWyU61aws8udrvCmjXSCyyEECJ/Kl++PA0bNnT2qF65coVt27bx/PPPO7dZsGABTZs2pXz58pQsWZLZs2dz6dKlNB3/zp07XL16lTp16jjLNBqNM9i759y5cwwYMIDq1atTunRpKlasiN1uT/N5AO7evcuVK1eoX7++S3mDBg34+++/XcqqVq3q/Lt48eIA3LhxI83nut/JkyepU6cOGk1CiNmgQQPMZjNnz56lYMGC9OzZk86dO/Pcc88xc+ZMl9f14osv8v333/P4448zbtw4du/enaF6PIgEwPlc4jGBJQ1CCCFEftanTx82btzI7du3WbZsGQULFuTpp58GYPXq1YwZM4aePXuyatUqdu3axYsvvuiS3+oO3bt35+bNm8yYMYOtW7eyc+dOdDqd286jKIrLsl6vT7JOVTP2a7CqqkmOn/jYs2fPZuvWrTRs2JBNmzZRu3Zttm3bBkDLli05duwYL7/8MhEREXTr1o2hQ4dmqC6pkQA4n+vUyTUP+LffdJw7J5eFEEII97oTGZmhx7WrVzO0X0Z16NABo9HIihUrWLJkCd27d3cGiPv27aNWrVoMHDiQ6tWrU65cOc6dO5fmYxcoUIDAwEB+++03Z5mqqvzxxx/O5Vu3bnHy5En+97//0bRpUypVqkRUVBRWq9W5jcHgGLo08c1z9/Pz86N48eLs37/fpXzfvn1UqlQpzXVOr8qVK3Pw4EGXPOp9+/ZhMBicKQ8A1apVY8SIEWzcuJEnnniCb7/91rmucOHCdO/e3ZnD/O233xIfH+/WekoOcD5XqpRKw4ZW9u5NuBRWrtTz+uvuvdCEEEKI3MDT05OuXbsydepUIiMjXdIfgoOD+fbbb9myZQvlypVj1apV7N27lwIFCqT5+IMHD+bjjz8mODiYkJAQ5s2bx7Vr1wgICADA39+fwoULs3jxYkqVKsXly5eZMGECOl3C/9NFixbF09OTbdu2ERAQgJ+fX7J1ePnll5kyZQrly5enevXqrFixgn379rFjx46MN9B/7t69m+RmugIFCvDiiy8yZ84cRo4cyeDBgwkPD+edd97hpZdewsvLi/DwcL7++mueeuopihcvTnh4OMePH6d///4AvPfeezz22GNUqVIFq9XK+vXrCQoKwmg0PnSd7ycBsKBrV0uSAHjUqHhS+AVDCCGEyNOef/555s+fT7169Vx6S/v168exY8cYMGAAqqrSvn17hg0blq5RIIYPH861a9d4+eWXAccEHF27dnUOc6bRaFiwYAGjR4+mQYMGlCtXjsmTJ7vcZKfT6Zg2bRoffPAB06ZNo0GDBslOgDF48GCio6OZOHEi169fp0KFCixevJjQ0NCMNo3Tvn37aNy4sUtZ+/btWbx4MWFhYUyYMIFGjRpRoEABunTpwoQJEwDw8vLizJkzvPDCC0RERFCsWDG6du3KiBEjADAajUyePJnz589jNBqpU6cOy5cvf+j6JqZERkbKLf952OnTp6lQoUKq29y6pVCxoi9Wa0LEu2tXFNWqpTyuYV6RlvbJz6R9HkzaKHXSPqnLq+1z586ddPWKpsZkMqU4yoGQ9snotSbJnoJChVSaN7e6lMnUyEIIIYTIqyQAFkDSSTFWrdKTysQ2QgghhBC5lgTAAoCnnrLg5ZWQDXPpkob9+7XZWCMhhBBCiMwhAbAAwMcHnn46aS+wEEIIIUReIwGwcOrc2TUAXrNGj8WSwsZCCCGEELmUBMDCqXlzK/7+CYm/t25p2L5dRsoTQgiRfhmdSUyItHqYa0wCYOFkMEDHjq5dvjI1shBCiPTy9vYmMjJSgmCRaVRVJTIyEm9v7wztL917wkWXLha+/jphtpWNG/XExsbh5ZWNlRJCCJGr6HQ6fH19uXv37kMf6+7du/j5+bmhVnlTfm4fX19flxny0kMCYOGiYUMbJUrYuXzZ8eNATIzCjz/q6dRJkoGFEEKknU6nc8tkGNevX6d06dJuqFHeJO2TMZICIVxoNCQJdsPCJA1CCCGEEHmHBMAiiS5dzC7LW7fquH1bSWFrIYQQQojcRQJgkcRjj9mpUMHmXLZYFNatk2wZIYQQQuQNEgCLJBQl6dTIYWGGbKqNEEIIIYR7SQAskpU4AN6zR8vly5IGIYQQQojcTwJgkazy5e3UqGF1LquqwurVcjOcEEIIIXI/CYBFihL3AsukGEIIIYTICyQAFinq1MmCoiTM4nP4sI7Tp+WSEUIIIUTuJtGMSFHx4iqNGtlcyqQXWAghhBC5nQTAIlWJxwRetUqPTO0uhBBCiNxMAmA36t69O2XLlqVPnz7ZXRW3ad/egl6fEPGeOaPlyBG5bIQQQgiRe0kk40ZDhw7liy++yO5quJW/P7RsaXUpkzGBhRBCCJGbSQDsRo0bN8bHxye7q+F2Xbu6jgaxerUemy2FjYUQQgghcrgcEwB/9NFH+Pv78/rrr7v92Hv27KF79+5UqVIFf39/li5dmux28+bNIzQ0lICAAJo0acLevXvdXpfcqHVrCz4+CWkQV65o2LNHm401EkIIIYTIuBwRAB88eJBFixZRtWrVVLc7cOAA8fHxScrDw8M5f/58ivvFxMQQEhLC1KlT8fT0THab1atXM3r0aEaOHMnOnTupW7cuXbt25eLFi85tGjRokOzj0qVLaXyluZOXF7Rt69oLvGqVpEEIIYQQInfK9gD4zp07vPTSS3z++ef4+/unuJ2qqrzxxhv06dMHiyUhGLt48SLt2rVj/vz5Ke7bqlUrJkyYQIcOHdBokn/Js2bNomfPnvTt25dKlSoxffp0AgICWLBggXObffv2JfsoVapU+l94LpN4Uoy1a3Uk811ECCGEECLHy/YAeMSIEXTo0IEmTZqkup2iKISFhXH27Fn69euH1Wrl33//pV27dtSpU4eJEydmuA5ms5nDhw/TrFkzl/JmzZpx4MCBDB83L2na1ErhwnbncmSkhm3bdNlYIyGEEEKIjMnWAHjRokWcPXuWsWPHpmn7YsWKsW7dOk6cOMELL7xA+/btqVatGl999RVabcZzUiMiIrDZbBQtWtSlvGjRoly/fj3Nx+nQoQMvvPACW7ZsISQkhF9//TXDdcpp9Hp49lmZGlkIIYQQuV+2deGdPn2aSZMmsWnTJgyGtOeTFi9enAULFtC0aVMCAwOZO3cuOp17XoaiKC7LqqomKUvN2rVr3VKPnKpLFwvz5hmdy5s26YmOjiMPDnwhhBBCiDws23qAf/31VyIiImjQoAGFCxemcOHC7Nmzh3nz5lG4cOFkb3YDR2/t0KFDadGiBTqdjtdeew273Z7stmlVuHBhtFptkt7emzdvJukVzs/q1rVRunRCW8fFKfzwg/QCCyGEECJ3ybYAuG3btuzdu5ddu3Y5HzVq1KBz587s2rUr2V7h27dv07FjR4oXL87SpUvZsGEDO3fu5NVXX0V9iPl5DQYD1atXZ/v27S7l27dvp169ehk+bl6j0UDnzq5TI0sahBBCCCFym2xLgfD3908y6oOXlxcFCxYkJCQkyfaqqvLcc89RqFAhli5ditFoJCgoiHXr1vHMM88wefJkxo8fn+y5oqOjOXv2LAB2u51Lly5x9OhRChYsSOnSpQEYNmwYgwYNolatWtSrV48FCxZw9epV+vXr594Xnst16WJhxgwP5/K2bTpu3lQoUiTjX0CEEEIIIbJSrrmNX1EUxo4dS926dfHwSAjAypcvz9q1a1Mc3gzg0KFDtGvXzrk8ZcoUpkyZQo8ePZgzZw4AnTp14tatW0yfPp1r165RpUoVvvvuO8qUKZN5LyoXqlrVTpUqNv76y3HToc2msHatnhdfND9gTyGEEEKInCFHBcAbN25MdX3Tpk2TLa9YsWKq+zVq1IjIyMgHnn/AgAEMGDDggdvlZ4ri6AV+992EUTdWrpQAWAghhBC5R7aPAyxyn8R5wPv26bh4Me2jZQghhBBCZCcJgEW6BQWp1KljdSlbvVpuhhNCCCFE7iABsMiQxFMjh4WlfSxnIYQQQojsJAFwHqekMJ7yw3r2WQsaTcLID3/+qeWvv+RyEkIIIUTOJxFLXqaqBI8ahefAgZCGmwDTo1gxlaZNXdMgVq2SNAghhBBC5HwSAOdhhgULKLB/P4bvvsO3YUN0iSb6eFiJ0yBWrtTzEPORCCGEEEJkCQmA8yjNP//gcd/EIJrLl/F+9lk8Xn8dYmLcco5nnrFgNCZEvOHhWn7/XZvKHkIIIYQQ2U8C4LzKYsEeFJSk2Dh3Lj6NG6M9ePChT+HnB61bu6ZBhIVJGoQQQgghcjYJgPMoe+XKRG/fzpU+fVATzZKn/ecfvFu3xjh5MpgfbgKLLl1c91+zRo/VmsLGQgghhBA5gATAeZnRyL8vv0zMDz9gS9QbrNjteHz4IT4tWqA5cSLDp2jVyoqfX0IaxPXrGnbtylETDAohhBBCuJAAOB+w1a9P9O7dxPfvn2Sd9uhRfJo2xfD552CzpfvYHh7Qrl3Sm+GEEEIIIXIqCYDzCx8fTB9/TExYGPbAQJdVitmM5/jxeLdrhxIenu5DJx4NYv16PSbTw1RWCCGEECLzSACcz1hbtiR63z7MnTolWafbuxffJ55Av3gx6RnPrFEjK8WK2Z3Ld+8qPP+8F2fPyuUlhBBCiJxHIpR8SC1YkLgFC4idPx+7v7/LOiU6Gq9XXsGre3eUa9fSdDydzjEz3P22bNFTv74P775rdNeoa0IIIYQQbiEBcD5m6dyZ6H37sLRokWSdfvNmfBo0QLd2bZqO9eKLZjw8XHuNzWaFjz7yoG5dX1avlkkyhBBCCJEzSACcz6nFixMbFkbcJ5+genm5rNPcuoV3375pmkq5YkU7W7ZEU79+0jHQ/v1XQ//+XrRr583x43LJCSGEECJ7pTsa2bNnD1988YVLWVhYGLVr1yY4OJg333wTu92ewt4iR1IUzP36Eb17N9Z69ZKsTutUytWq2dm0KYYvv4wlMDDpNbB7t47GjX14802PB8XTQgghhBCZJt0B8LRp0zhw4IBz+dSpUwwdOhSNRkONGjWYO3dukgBZ5A72cuWI+eEH4t5+G1XvOpSZy1TKsbEpHkNRoFs3CwcPRvHKK/HodK55DzabwpdfGqld25fFi/XIdyUhhBBCZLV0B8B///03tWrVci5/9913eHp6snXrVsLCwujWrRtLlixxayVFFtJqMY8YQfTPP2OrWjXJaudUyr/9luphfH1h0iQTe/dG06yZJcn6mzc1vPKKFy1aePP771q3VV8IIYQQ4kHSHQDfvXsX//tGDti2bRtPPvkkfn5+ADRo0IALFy64rYIie9irVSP6558xvfZa0qmUz5zBu1WrNE2lXLGinVWrYlmyJIYyZZJ29/7xh47mzX0YPtyTGzcUt74GIYQQQojkpDsADggI4OTJkwBcuXKFo0eP0qxZM+f6u3fvotVKj16eYDQSP3GiYyrlRx5xWeUylfJff6V6GEWBZ56xcuBAFGPGmJKMFgGwZImBWrV8mTPHgDXpfXRCCCGEEG6T7gC4Xbt2zJ07lzfeeIPnn38eo9HIU0895Vz/559/EhQU5M46imxmq1+f6F27HnoqZU9PePPNeH79NYr27ZOmRdy9qzBmjCeNGvmwc6d8iRJCCCFE5kh3ADxmzBjat2/Pd999x7Vr15g5cybFihUDHL2/69ev58knn3R7RUU2S20q5fj4dE2lXKaMyuLFsXz/fTSVKiUNmv/6S0v79j706+fJpUuSFiGEEEII90p3AOzt7c1XX31FeHg4x44do9N9U+r6+Phw4sQJxo4d69ZKipzDOZVy585J1jmnUk7jTZBNm9rYvTuayZPj8PVNmhaxZo2BunV9+fBDIybTQ1ddCCGEEAJw40QYV69e5dSpUxQoUAB9oiG0RN6iFixI3Pz5KU+lPHw4hrlz03QsvR6GDzfz229R9OiR9Ia62FiFyZM9qF/fhx9/1Lmj+kIIIYTI59IdAC9cuJBBgwa5lI0cOZKQkBAaNmxIo0aNiIiIcFsFRc6V2lTKHm++iW7r1jQfKyBAZc6cOH76KZrHHkuaFhEerqV7d2+ee86Lf/6R2eSEEEIIkXHpjiQWLVqEr6+vc3nnzp0sWLCALl26MGHCBM6dO8eHH37o1kqKnMtlKmVPT2e5Yrfj1b//A0eISKxuXRs//xzNjBmxFCqUdNi0n37S06CBD++8YyQ6+qGrL4QQQoh8KN0B8Pnz56lcubJz+fvvv6dkyZJ88cUXjBgxgpdeeolNmza5tZIih/tvKuXYhQtRlYSb1pS7d/Hu1g3lxo10HU6rhRdesPD779G89FI8Go1rfrDZrPDJJx7UrevLypV61KTpw0LkbFYrxMRkdy2EECLfSncAbDabXXJ8t2/fTosWLdD8N1lCuXLluHr1qvtqKHINa5s2mCZPdinTXLiAV69eZOQutoIFVaZPN/HLL9E0aJB0cODLlzUMGOBF27be/PGHDJsmcgftL7/gV748fkFBGGbMyO7qCCFEvpTuALhs2bLs2LEDgD/++IPw8HCXiTCuX7/ukiIh8hfz0KHE9+vnUqb79Vc8hw8no1211arZ+eGHGObOjaV48aRpEXv36mjWzIfGjX2YN89AZGSGTiNE5ouKwuull1Du3EGxWPB8+200R45kd62EECLfSXcA3L9/f77//nsaNmxIp06dKFmyJC1btnSu379/v0uKhMhnFAXTBx9gadrUpdiwciXGadMe5rB07Wrh4MEoRowwodcnDaaPHtUyapQnlSv7MXCgJ7t3ayU9QuQoxs8+Q3P9ukuZx3vvZVNthBAi/0p3ADxgwAA+/fRTypUrx1NPPcWqVavw/O/mp9u3b3Pjxg26du3q9oqKXESvJ/brr7FVrOhS7DF1KvqVKx/q0D4+8Pbb8ezbF03LlklnkwMwmRS++87AM8/4UKuWD598YuTqVZlQQ2Qv5fJljDNnJinX//QT2n37sqFGQgiRf2VoPKk+ffqwZMkS5syZQ6VKlZzlBQsWZMeOHfTp08dtFRS5lL8/sStWYC9UyKXYc9gwtL/++tCHDw628913sYSFxdCypSXJjXL3nD2r5Z13PKha1ZcePbzYtEmHNWk6sRCZzuP991Hi4pJfN2lShlOEhBBCpN9DDah6/PhxNm3axKZNmzh+/Li76iTyCPsjjxC7dCmqweAsU+Lj8erZE+X8+Yc+vqJAy5ZWwsJiOXo0irfeMlG6dNIcYQCbTWHTJj09enjz6KO+TJpk5OxZGU9YZA3Nn3+iX7o0xfW6ffvSNW62EEKIh5OhCGDjxo2EhobSqFEjevXqRa9evWjUqBGPPfYYGzdudHcdRS5ma9CAuM8+cynT3LyJd/fucOeO285TqpTKG2/Ec+RIFN9/H02nTmYMhuR71K5e1fDxxx7UrOnL4MEV+e47PSl0zAnhFh4TJ6Lc18Nrq1gRy303D8N/vcD25L/ACSGEcK90B8Bbt26lT58+qKrK+PHjWbJkCd988w3jx49HVVX69u3Ltm3bMqOuIpeydO+OadQolzLtX3/h9eKLuDsfQaOBpk1tLFgQx99/RzFlShwhIUlnlrvn99/9GDjQi8qV/Xj9dQ+OHJFeYeFeup9/Rp/oM9H0zjuYJk50KdMeO4Z+7dqsrJoQQuRbSmRkZLoSz1q1akV0dDSbN29OMtxZVFQUrVu3xs/Pjx9//NGtFRUZc/r0aSpUqJDd1QC7Hc/+/TF8/71LcfxLL2GaPj1TT62q8McfWhYvNrBqlZ7o6NRviHvsMRt9+pjp3NmMv3+mVi3HyzHXTw6WahvZbPg0boz2vhQx6xNPELN+PSgKnv36YVizJmHz4GCi9+8HnS6zq51l5BpKnbTPg0kbpU7aJ2PS3d31559/0qtXr2TH+vX19aVXr14cPXrULZUTeYhGQ9ycOVhr1XIpNs6di+GrrzL11IoCtWrZ+PTTOP7++y4zZ8ZSr17KPc9HjmgZOdIxnNqgQTKcmsg4/fLlLsEvQNzkyY6LEogfOxZVmzCJi/bMGfTLlmVpHYUQIj9KdwCs1+uJjY1NcX1MTIzLTHFCOHl6ErtsGfZSpVyKPUaPzrIbgHx8oHdvC5s3x3DgQBS9el2lcOHk8y5NJoUVKxzDqdWu7cOMGQauXZPh1EQaxcbikWhmRPNzz2GvXt25bA8OxtKzp8s2HtOmZWjmRCGEEGmX7gC4QYMGzJ07l3/++SfJurNnzzJv3jwaNmzolsqJvEcNCCBm+XJUHx9nmWK349WvH5oTJ7K0LpUq2Rkx4hJ//RXFokUxtGhhQVGS7+r95x8tb7/tSUiILz17ynBq4sGMs2ejuXLFuawajZjGjUuynenNN11GStH8+y+G+fOzpI5CCJFfpTsAnjhxIiaTiQYNGtC3b18mT57M5MmT6dOnDw0aNMBkMjEx0c0dQtzP/uijxM6fj6pJuPyUqCi8u3VDSTRLVlYwGKBDBysrVzqGUxszJvXh1H74wTGcWmioL1OnGrlyRXqFhSvl+nWMM2a4lJkHD0YtUybJtmqpUphffNGlzPjxxxAVlZlVFEKIfC3dAXCVKlXYvn07bdq0Ydu2bXz00Ud89NFH/Pzzz7Rp04Y1a9Zgla4x8QDW1q0xJZoCVnPxIl69epGdY5KVLq3y5puO4dTWrInh2WfNyU67DHD5soapUz149FFf+vTx4pdfJFdYOBinTUOJjnYu2wsVwvTaayluHz9ypMuvIpqICIyzZ2dqHYUQIj/L0JhP5cuXZ/HixVy4cIGTJ09y8uRJLly4wKJFi9i+fTuNGzd2dz1FHmQePJj4/v1dynQHD+I5fHi2z4ql0cCTT1pZuNAxnNr778dRpUryw6nZbArr1unp0MGHOnV8mDXLwO3b0iucX2lOncLw9dcuZfFvvEFqQ4qoRYoQP3SoS5lx5kyUW7cyoYZCCCEeatBTjUZDsWLFKFasGBqNjJ8q0klRME2bhuXJJ12KDatWYZw6NZsqlVThwipDh5rZuzearVujef55M15eyQfoZ85oGTvWkypVfBk61JPff5de4fzGY+JEFFvClyVbuXKYE33RS078sGHYCxZ0LitRURg/+SRT6iiEEPmdRK0ie+n1xC5ciK1SJZdij2nT0IeFZVOlkqcoULu2jc8/j+Ovv+7ywQdxVK6cfK+wyaSwbJmB5s19aNrUh8WL9cTEZHGFRZbT7t6NftMmlzLTxImORPMHKVCA+P/9z6XIMHcuyuXL7qyiEEIIJAAWOYG/PzErVmAvXNil2HPYMLQHDmRTpVJXoAAMHGhm375oNm6MpnPnlHOFjxzR8sorXlSp4scbb3jw99/ytsuT7HY8xo93KbLWrYu1ffs0H8I8YAD24sWdy4rJhDGTJ4oRQoj8SP4nFjmCGhRE7NKlLsNBKWYzXr16oYSHZ1/FHkBR4PHHbcyfH8fx41FMmJDyCBJ37yp89ZWR+vV9advWm9Wr9ZjNWVxhkWn0q1ejO3TIpcz07rvOSS/SxNPTkS98H8M336A5e9YdVRRCCPGfNM23+fvvv6f5gJfl5zqRQbb69YmbOROvgQOdZZqbN/Hu3p3ozZsd3a45WLFiKv/7XzyvvhrP1q065s83sGWLDlVNGgDt2aNjzx4dxYrZef55M337milTRpKFcy2TCY933nEpsnTogK1evXQfyty7N4bPPkN77hwAitWKccoU4ubOdUtVhRBCpDEAbtGiBUoaezFUVU3ztkIkZnnuOUynT+Nx38++2r//xqt/f2JXrABdmi7ZbKXVQuvWVlq3thIerrBokYFvvjFw82bSH1yuX9fw0UcefPKJkZYtrbz4opnmza3cNzuuyAUMc+eiuXjRuazq9Y7c34zQ64l/6y28XnopoWjlSuJffRX7o48+bFWFEEKQxgB41qxZmV0PIZzix4xBc+YMhjVrnGX6bdvwGDMGUy7LhwwKUpk4MZ7Ro+NZv17P/PkG9u1L+raz2xU2b9azebOeMmXs9OtnpndvM0WLSq9wTqeNjHT5wgZgfvFF7OXKZfiYls6dsX3yCdr/ZkdUVBWPyZOJXb78oeoqhBDCIU0BcM9Ec9ULkak0GuJmz0Zz8SK6335zFhvnzsUeHIx50KBsrFzGGI3QpYuFLl0snDihYeFCA8uXG4iKSvpryYULGt55x4P33zfSoYOF/v3NNGhgS1cqqcg6JRYsQLl717ms+vklyeNNN40G0/jxePfo4SzS//gj2gMHMpRWIYQQwpXcBCdyJk9PYpctw16qlEuxx5gx6LZsyaZKuUdIiJ3p00389dddZsyIpVq15IdSs1gUVq408PTTPjRs6MOXXxrYtk3Hnj1afv9dy59/ajhzRsOlSwo3bypERYHFksUvJp/TnDtH0UTD9ZlGjUItVOihj21t0wZr3bouZR6TJmX7JDFCCJEX5PyESpFvqcWKEbNiBT6tWzunlVXsdrz69yd682bsISHZXMOH4+MDL7xgoW9fC7/9pmX+fANr1uiJj0/a1fvXX1refNMzTcfValU8PMBodDx7eCRdNhoTnj097y07tvH0TNjWZCpI4cIKhQpJ0JUc4zvvoLlv6nd76dKY77uJ86EoCqbx4/Fp185ZpNuzB9327VibNXPPOYQQIp+SAFjkaPaqVYldsACv7t1R7I7hxZSoKLy7dSN62zbUYsWyuYYPT1GgTh0bderE8f77JpYu1bNwoYGzZzN2J5zNphATAzEx7siZKM+776r06GFm6FAzwcHJD/GWH2l//RXD99+7lJkmTHB8k3ATW6NGWJo1Q//zz84y46RJWJ98Mn3DqwkhhHAhKRAix7O2aoXp/fddyjQXL+LVsyfExWVTrTJHoUIqL79s5rffolmzJoZnnrGg1WZv72tcnMKCBUbq1PGhe3cvdu+W6Z1R1aSTXlSvjqVzZ7efKj7ReXSHD6Nbt87t5xFCiPxEAmA36t69O2XLlqVPnz7ZXZU8xzxoEPEDBriU6X77Dc9hw/JkTqRGA08+aWXJkliOHYti7FgTTz1loWlTCw0aWKlRw0pIiI1y5WyUKGGnUCE73t4qGk3mtYWqKvz4o55nnvHhySe9CQvT59ucY926degSzVJoevddxz+cm9lq1MCSaDY5j/feg/tSL4QQQqSPpEC40dChQ+nbty/ffvttdlcl71EUTFOnojl71uXnYMPq1diDg4l/661srFzmKlFC5fXX49O8vcUCJhPExyuYTGAyKS7L8fEKcXGpLTt6fePiFH78Ea5fNyQ5x+HDOl56Scfbb9sZNCiePn3M+Pu78UXnZGZz0kkv2rTB1qhRpp3SNHYsug0bnGlA2lOn0C9fjqV370w7pxBC5GUSALtR48aN2bVrV3ZXI+/S6YhduBCf1q3R/v23s9jjgw+wlymDpVcvyYsE9HrHw9f3Xm9wxnuF//rrDH/+WYWZM40cOZI0J/nffzVMmODJBx948PzzZgYPjqds2bzXI38/w4IFaO+bmljVajElCojdzV6pEpbu3TEsW+Ys85g6FUvXro4x9oQQQqRLtqZAzJ07l4YNG1K6dGlKly5Ny5Yt2bx5s9vPs2fPHrp3706VKlXw9/dn6dKlyW43b948QkNDCQgIoEmTJuzdu9ftdREPqUABYpYvx16kiEux1/Dh+NSsiXHSJDRHj+bJtIjsoNOpdO1qYceOaDZsiOappywoStK2jY5WmDPHSI0avrzwgie//ZZHp7KLjMT4wQcuRTc6dsReqVKmn9o0ejSqXu9c1ly6hGHhwkw/rxBC5EXZGgCXKFGCd955h19++YXt27fTuHFjevXqxZ9//pns9gcOHCA+PulPweHh4Zw/fz7F88TExBASEsLUqVPx9Ex+KKnVq1czevRoRo4cyc6dO6lbty5du3bl4n3TmzZo0CDZx6VLl9L5ysXDUIOCiF26FNXg+tO89tw5PD7+GN/GjfGpUwfj5Mlojh+XYNgNFAWeeMLGt9/GcvBgNP37x+PpmbRd7XaF77830KKFD61be7NunQ5b8sMc5wh2O5w9q+HUKU2aLhOPTz5Bc+uWc1n18eHyfVMWZya1TBnM/fq5lBk/+gj+GyJQCCFE2imRkZE5KjoICgpi4sSJ9Ev0Qa+qKk2bNiUwMJAlS5ag/68n5OLFizz99NM8++yzTJo06YHHL1myJB988AG9evVyKW/evDlVq1bls88+c5bVrFmTDh06MHHixDTXf9euXcydO5fFixeneZ/MdPr0aSpUqJDd1cgU+rAwPAcORHlA5GKrVAlLx45YOnVK0lOXl9vHHVJrn4gIhQULDMyda+D69ZS/SwcF2RgyxEyvXmZ8fDKrpg+mqnDunIbDh7UcOqTl8GEtR45ouXvXkTbTurWFuXNj8fNLfn/lwgV869RBue9LuGnsWP7s2DHLriHl+nV8q1dHiY1NqMO4ccSPGpUl588IeY+lTtrnwaSNUiftkzE5ZhQIm83GqlWriImJoW6i2Y8AFEUhLCyMs2fP0q9fP6xWK//++y/t2rWjTp066QpSEzObzRw+fJhmiQaXb9asGQcS3ektcg5L167EbNyI5emnk/QG30978iQe06bhW68ePg0bYpw+Hc2ZM1lY07ypcGHHzXnHjkUxc2YsISHJd/WGhzsm8aha1Y+33zZy+XLm52nfC3bXrNEzYYIH7dt7U7asHzVr+tK/vxeff25k1y6dM/gF2LxZT5s2Ply6lHz9PCZPdgl+7cWLEz9sWKa/lvupxYoRP2SIS5nxs89Qbt/O0noIIURul+03wR0/fpxWrVphMpnw9vZmyZIlVK1aNdltixUrxrp162jbti0vvPACf/31F9WqVeOrr75Cq814zmFERAQ2m42iRYu6lBctWpTr16+n+TgdOnTgzz//JDY2lpCQEL7++utkg3nhPraGDYlt2BDu3EG/aRP6NWvQ/fwzSgrjc2lPnEB74gQe772HrVo1Ahs1QvPSS9gfeSSLa553GI3Qu7eFXr0sbN+uY+ZMAz//rE+y3Z07CjNmeDBzppFOnSwMHx5PaOjDT6yhqnD+vMLhw9r/end1HDmiITIy/d/vT5zQ0rKlD8uXx/DYYwl10xw+jOG771y2NY0dC15eD13/9Ip/+WUM8+ejiYwEQLl7F8OnnxL/9ttZXhchhMitsj0ArlChArt27eLOnTusW7eOIUOGsGHDBkJSmOa2ePHiLFiwwJkOMXfuXHQ697wMJdEIAqqqJilLzdq1a91SD5EBBQpg6d4dS/fuEBmJfuNG9N9/j277dpQUxkvVHjtGqWPHYPZsxyQGzz6LpWNH1LJls7jyeYOiQLNmVpo1s3LihIZZs4yEhekxm13fQ1arwnffGfjuOwONGlkZPjyeli2taRpC916we+SI1iWVISPB7j06nYrVmlDHK1c0PP20DwsXxtKqlRVUFc9x41z2sVWtiqVHjwyf86H4+xM/YgSe9wW8xi+/xDx4MGpgYPbUSQghcplsT4EwGAyUK1eOGjVqMHHiRKpVq8bs2bNT3D4iIoKhQ4fSokULdDodr732Gnb7w/UiFS5cGK1Wm6S39+bNm0l6hUUu4O+PpVcvYsPCiDp1itjPPsPy5JOoqfxKoDt8GM+JE/F77DG8mzfHMHMmitzcmGEhIXZmzYrj2LEoRo0yUahQ8u/RXbt0dOvmTf36Pnz9td5lYj9VhQsXFNau1TFpkpFnn/WiXDlfqlf3o29fbz75xIMdO/TpCn79/FQaNbLy8svxzJ8fyx9/RHH+/F2eecb1F4OYGIXu3b2YP9+AbvNmdLt3u6w3vfsuPMSvTg/LPHAg9oAA57ISF4fxww+zrT5CCJHbZHsAnJjdbsdsNie77vbt23Ts2JHixYuzdOlSNmzYwM6dO3n11VdRH+JOf4PBQPXq1dm+fbtL+fbt26lXr16Gjyuyn1qoEJY+fYhds4aokyeJ++QTrI0bo6bS3aj7/Xc8x43D79FH8W7dGsMXX6BcuZKFtc47AgJUxo2L588/o/j44ziCg5PPEz51SsuIEV48+qgvr73mQefOXpQv70toqCPY/fhjD7Zv13P7dto/snx9VR5/3NHDPG9eLL//HkV4+F3Wr4/h3XdNdO5soVw5O97esGhRLMOGuY4wY7crvDFST9SQt13KLc2bY010v0CW8/Ii/vXXXYoMX3+NEh6ePfURQohcJltTIN5++21atWpFyZIliY6OZuXKlezevZvvEuXagSMd4bnnnqNQoUIsXboUo9FIUFAQ69at45lnnmHy5MmMHz8+2fNER0dz9r+B6+12O5cuXeLo0aMULFiQ0qVLAzBs2DAGDRpErVq1qFevHgsWLODq1atJRqMQuZdapAjmfv0w9+uHcu0a+vXrsS5dis/hwymOJKE7cADdgQN4jBmDrX59LJ06YWnfHvW+3jfxYF5e0L+/mRdeMLN5s46ZM43s2ZP04yciQsPChemf2MHHRyU01EaNGjaqV3c8lytnT/PMxFotvPeeiaAgO2++6YHd7kiJeJH5BN5OmHRFVZRMn/Qircx9+mD8/HM0/w0BqViteEyZQtyXX2ZzzYQQIufL1mHQhgwZwq5du7h+/Tp+fn5UrVqVV155hebNmye7/Y4dO6hbty5eiW48OXXqFBqNhuDg4GT327VrF+3atUtS3qNHD+bMmeNcnjdvHp9++inXrl2jSpUqvP/++zz++OMP8QqznwyPkrrTp09T0ccH/bp1jhvo9u9/4D6qRoOtYUNsVauiBgZiDwx0PAcEoAYGohYsmLtnpFNVuHsXTUQEF06epHTduqiFCrn9NR06pGX2bAOrV+ux2dJ+bB8flWrVEoLd6tVtBAenPdh9kB9/1NG/vxea2GhOU4FArjnX3e3SG3XeTJfts/M9pl++HK/Bg53LqqIQvWcP9hTuocgO8hmUOmmfB5M2Sp20T8bkuHGAhXvJGyN1idtH+fdf9N9/77iB7uDBDB1TNRhQixVzBMYBAUmf7wXKRYtmTR6p3Y5y+zZKRATKzZsoN2+iue/ve+Wae39HRKAkSkNSPT2xlyyJWrIk9pIlsZcqhb1UqYTlkiXB1zdD1bt0SeHLL40sWmRwGZYMwNvb0bP72GMJAW9wsD3Tm+3wYQ2/t/2A/8VMdpbF4kmzUn8zc01BKlRIyGnO1veYzYbPE0+g/esvZ5GlbVtiU5jtMjvk6c8gsxnDokXoV61C9fPDNG4c9tDQdB0iT7ePm0gbpU7aJ2MkAM7j5I2RutTaR7lwAf3atehXr0Z36JDbz61qNKhFiyYbHLs8BwQ4xhq7x2JBuXUrxWBWkyiwVW7dQnnIG0XT9HoKFEgxOLaXLo1avLjr60gkKgq+/17PxYsaypWzU726jQoVMj/YTY5y+TI+NWuhMSXclfcu45jAu/j721m2LJaGDR35zNn9HtNt2IB3794uZdFbt2KrXdul7OJFBZNJcQnes0J2t0+msNvRr1qF8b330N6Xd616ehL7xRdYO3RI86HyZPu4mbRR6qR9MkYC4DxO3hipS2v7KOHhGNasQb9mDdqjR7OgZq7sBQuiFiiAEhnpHP81N7IXK5bQk1yqlOPv/57tpUo5gv1sHF3hHs/hwzEsWeJcvkYxgjlDNI5eboNBZdasOLp2tWT/e0xV8W7RAt3vvzuLrI0bE7NuHeBINXn3XaNzbOaOHc3MnBmXZbPyZXv7uJOqotu2DY933kF77FiKm5lGjyb+jTdIS15OnmqfTCJtlDppn4zJ9nGAhcgN1KAg4l97jfjXXkPzzz9of/8d5epVNFevoly7lvB87RpKVJTbz6+5fRuycLYv1csLtXBhzIqCMSICJSbGLcfVXL+O5vp1SKFHXdXpsJcqhbVpU6zPPIO1cWNIZZa/zKD580/0iVIINtYZT/TBhBQPs1nhpZe8uHDBRDK3F2QtRcE0YQI+9/U66nbu5PI3O3nzpzasX+86Kcn33xs4c0bLt9/GULq09H+klfb33/GYODHJkHjJ8Zg6Fe1ffxE7ezZ4e2dB7YQQ6SU9wHmcfDNMXaa0T0yMIxC+etX5rFy7hubKlYQg+epVR1CbRVQ/P+yFC6MWKYL637P9vr/vPe5tc2+Gs9OnT1MhOBju3EFz6ZLj8e+/KP/+67p8+XKKs+89bL0trVphbdsWS4sWGc4zTg+vzp3Rb9vmXLZVrEjUnr18OMOb997zSLJ9hw43mDfPgD7p5HdZyrtDB3S//OJcPkBd6rMfSP4Gw6JF7SxZEku9eskPTecuuf0zSHPqFB6TJ6P/r0c9MVWnw/r00+g2bUryHrCFhhKzbBlqqVIpHj+3t09WkDZKnbRPxkgPsBDu5u2NvVw5KFeOVEOL+HjXgPj+5/t6lZUbN5Lk8NoLFkwIXAsXdgSziQLa+wPe1HJvH0hRwN8fu78/9kcfTX4bux3lxg00ly6h/BcUa/4LkpX//lauXk1xuLkUT333LoaVKzGsXIlqNGJt2hRL27ZYn37aEai7me7nn12CXwDTO++g6HW8/no8ZcvaGT7c02V2u7Vri3L3roWvv46lQAG3VynNLg2ZQNAvCSPo1ONXOrCWtXRMdvsbNzS0a+fNjBlx9Ozp/i8vuZ1y+TIeU6eiX7oUxZb8O9ncqRPx48ZhL1cO7Z49ePXpgyYiwrlee/QoPs2aEbtkCba6dbOq6kKINJAe4DxOvhmmLle0j9XquJnt7l1Uf3/HkGRumv77QdzaPhYLyuXLzuD4XmCsuXgxYfnWrTQdStVoHOMyt22L5Zln3DN9tc2GT+PGaI8fdxZZn3iCmPXrXYaA271bS+/eXklmoAsJsbFiRdanFUREKMyYYWTuXAPLTJ14lu+d644TQihHCa2u8uab8SxdamDDhqRd1a+8Es/EiaZMSb/OFe+x+0VGYpwxA+MXX6CYTMluYmnWDNOECdirV3cpV86fx7tHD7QnTriUqwYDcZ9+muz02bmufbKBtFHqpH0yRgLgPE7eGKmT9kldlrdPTAy6vXvRb9iA7ocf0Ny4kabdbNWqYXnmGSzPPOMYAzcDYxbrly7Fa9gwl7KoHTuSBDkAp05p6NrVm/PnXYPgwEA7y5fHUL165o+0cPcuzJ5tZNYsI1FRjtcbwnGOUQ0NCR/rewZ+RdVpz6EoYLfD++8b+fDDpKkcrVtbmDs3Fj8/99Yz17zH4uIwfPUVHh9/jHLnTrKbWGvUwPT229iaNEn5OFFReA0ciH7TpiSr4l95BdPEiS43euaa9slG0kapk/bJmBw3FbIQIh/z9sbasiVxn35K1N9/E71pE/HDhmF/QA+v9tgxPKZMwffxx/GpWROPcePQ7t/viPjSIjYWj8mTXYrMzz2XbPALULGina1bo6ld2+pSfvWqhrZtfdi8OfN66OPi4PPPDVSv7svUqR7O4BfgBFVZguuQaA1+nIxicYzrrNHAuHGOqaE9PFz7PjZv1tOqlQ/h4bl4EpeMsFrRL1qEb82aeE6cmGzwaytfnpivvybm559TD34BfH2JXboU0//+l2SV8bPP8OrZ0/HtRQiRrSQAFkLkTFottgYNML33HlGHDxO1ezem0aOxpZSHfG+3c+cwzpyJT5s2+FaujMeIEei2boVEk3vczzh7NporV5zLqtGIady4VM9TtKjKunUxPPmk682MMTEKPXp4MW+ee0evMJthwQIDNWv6Mn68J7duJf34Dgy0Y35rDOp9d+RpLlzAsGiRy3Zdulj44YcYAgNdvyD8/beWZs182L07+4eiy3Sqim7dOnwaNMDr1Vdd/v3vsQcGEvfJJ0Tv34+1Y8e0/7Kg0RA/YQKxc+eiJsq/12/ejE+rVmjOnXPDixBCZJQEwEKInE9RsD/6KPGjRxO9ezd3Dx8m7r33sDZogJpKUKK5fh3j11/j3aULfsHBeL74Ivo1axyzbtw79PXrGGfMcNnPPHgwapkyD6yWlxdMnfoPw4fHu5Tb7QqjRnkybpxHmjuhU2KzwfLleurW9eF///PkypWkH9uFCtl59904Dh2KousbJTD37euy3jh9OiQayq5mTRs//xxNjRquvdi3bmno2NGbRYuyeViLTKTduRPvFi3w7tMH7enTSdarfn6YJk4k6o8/MPfrR0aH+LB07UrMDz9gDwx0Pf/ff+PdrBnanTszdFwhxMOTHOA8TnKDUiftk7rc0D7KjRvoNm1Cv3Ejuu3bk0zjnJz7R5TQ7duH4dtvnevshQoR9ccf4O+fpvPfa6O5cw28+aYHdrtrQN6+vYUvv4zF0zNdLwtVhQ0bdLz3ngd//518j6yvr8qwYfEMHRrvkrurXL2Kb40aKHEJM9mZJk4k/rXXkhwjNhaGD/dk9eqkPdaDBsXz3num1O+5tFodI3+Eh6M9dw7NuXNowsMdz+fPY1NVqFQJe8WK2CpWxF6xIvZKlRxpLVl0M+c9mqNH8XjnnSQjfdyjGo2YBw4k/n//Qy1Y0G3nVS5fxqtXryQzSqo6HRdGjcJ/9Gi3nSsvyg2fQ9lJ2idjJADO4+SNkTppn9TluvaJikK/dSu6DRvQ//RThiYliZs6FfPgwWne/v42+vFHHf37exEb6xoE16ljZdmyWIoWffDHrarC9u063n3XyKFDyQeIHh4qL71kZsSIeAoXTv6YxrffxuO+nm21QAHuHjmSbGCvqjB9upH33096c9yTT1r4euZ1Ct76L7C9F9zeC3QvXkSxWpPs98DXaTBgL1/eERhXqIC9UiXHc4UKbp88QgkPx+O99zCEhSVfF40GS8+emEaPTnXM3ocSF+eYYXDVqiSr4gcMwDRlSoZ7mvO6XPc5lMWkfTJGAuA8Tt4YqZP2SV2ubp/4eHQ7d6LbuBH9Dz84ZqB7AFu5ckTv35+u2ecSt9Hhwxq6dfPm2jXXVIWyZe2sXBlDhQop50Ts36/l3Xc92LMn+cBXp1Pp29fMqFHxFC/+gI/uyEj8QkNR7rvhyjRyJPHjxyfdVlVRrl1j7zcXWDX9MqXMZynPP5TnH8pxlgAe3HbuZC9dOqG3+L6eY7VIkXSN8KFcv45x+nQMX3+d4kQtlrZtHUOaVarkruqnTFUxfvwxHu++m2SVtXFjYhctcmvPc16Rqz+HsoC0T8ZIAJzHyRsjddI+qcsz7WOzoT140DG82oYNaMPDk90sZtEirPdNKZwWybXRxYsKzz3nzV9/uaYu+PvbWbo0lscfd51Y4ehRDe+958Hmzcn3ACqKynPPWRgzxkRQUNo/so0ffeQSbKleXsR+8QWaq1cTenL/69W9P10ip7IXLOhIn6hQwRkU2ypVQi1d2mVoMe7exThzJsZZs1KcxtvasKFjSLNsmKBCt349XoMHJ6mb7ZFHiF2+PGuC8Vwkz3wOZRJpn4yRADiPkzdG6qR9Upcn20dV0Zw4gX7DBvQbNqA9dgyA+P79MX30UbrHEE6pje7cgb59vdixwzWoNRhUZs2Ko2tXC6dPa3j/fSNr1qTc49yunYW33jJRpUoG7qaLjsa3Ro00j6ecUfaiRbE/8gj2oCDH45FHnMvnTp8m2GpFc/IkmlOn0J46hebUqTT1yKeV6uGBvXx5RzAcGIh+xQqXGdnuZ6taFdPEiVhbtszQeNHuovnzT7x79EBz8aJLuernR+y8eVhbtcqmmuU8efJzyI2kfTJGAuA8Tt4YqZP2SV1+aB/l2jWwWFBLlsxQQJRaG1ksMGKEJ0uXJg1wmza1sHOnLslNc/c0a2Zh3Lh4atZMdULtBzJ8+SWeb775UMewoOM8Zf9LiihP4TpleWpYSZRgR5CLj0+K+6bYPpGRjmD45Em0p087AuTTpx290Q87dEYy7GXKYBo7FkvXro4BkXMA5cYNlK5d8T182KVcVRRMkyZhHj48W4P0nCI/fA49DGmfjMnaW3CFECKHUQMCMu3Yej3MnBnHI4/YmTzZ9QazxD3D99Svb2XcOBNPPPFwge895hdewDhrFpoLF1LdTvXzc+nBtT3yCPayQSz4pSIjZwRj5b76HoQn5llZvDiWQj4Z7EPx98dWty62unVxyc41mdD88w+a06fR/hcUa0+eRHPmTIbSNOxFihA/apRjOLNEY/JmN7VoUU7Pnk21OXMwfPONs1xRVTzHj0f711/EffJJjqu3EHmBBMBCCJGJFAVGjYqnTBk7w4d7YjYn36MXGmpj/HgTLVpY3dvpZzQS8913eL72GppLl7CXLu2apvBfqoJaqFCyvY19n4Ri9cy89JKO6OiE9bt362jWzJvly2OpXNmNPbYeHtirVsVetSouY0vY7SgXLzpTKJypFKdOJZvuoPr4ED9sGPHDh4Ovr/vq52aqXk/cZ59hCwnBY+xYl95vw7JlaP75h9hvvkEtViwbaylE3iMBsBBCZIHnnrNQooSd3r29iIxM+Am+QgUbY8eaaN/emmm/zNsrVyZm06YM7//UU1Z++ima7t29uXAhoZLh4VpatvRh/vxYWrVK/1Bo6aLRoJYti7VsWWjZ0mWVEhGREAyfO4davDiWTp1QixbN3Dq5i6JgHjIEe8WKePXr5zJyh+7AAXyaNSNm6VLsjz2WjZUUIm/JGYlQQgiRDzzxhI2tW2Po3NlMy5YWZs6MZd++aDp2zLzg111CQuz8/HM0DRu6BrpRUQrdunnx+ecG1Gy6o0QtXBhbgwZY+vYl/u23MQ8alHuC3/tYmzcnets2bOXLu5RrLl3C56mn0K1dm001EyLvyeEfuUIIkbcEB9uZPz+OsLBYeve2ZPVkaA+lSBGV77+P4fnnXWfbU1WF8eM9GT7ck/j4FHYWaWKvUIHobduwNG3qUq7ExuLdty/GadPItm8aQuQhueijVwghRHYzGOCzz+IICbExdqzr1M9Llxr45x8N33yTtlnvMio6Gi5c0HDhgobz5x2Pe3//+69CyZKOCUP69DGnZ06TnMPfn9iVK/EYOxbjl1+6rPKYMgXN338TN2sWeHm575yqCnFxKNHRKNHREBXl+Pu/Z6KjUeLisFesiLVpU9dxl4XIhSQAFkIIkS6KAkOGmKlY0U6/fl7cvZsQBO/fr+PJJ31YvjyGRx/N2M1xJhNcvOga2F64oDiD3YiI1H+8vHULRo3yZOZMA2PHxtO5syXHp5gkodNhmjYNW0gIniNHukw3bVizBu3Zs8QsXoxaoIAzSHUGr3fvui7f+/v+YPa+ZSUqCmJiUGxpG3nEFhSEecgQzL16pToEnhA5mQTAQgghMqR5cytbt0bTvbsXZ88m9AheuqShdWsfvvwyluQmNbNY4N9/FZfe2/t7c69edU+0Gh6u5aWXvJgxw8aECSZatXLzCBtZwNK3L/by5fHq0wfNrVvOcu2RI/hl001x2vBwPN98E4/33ye+f3/ML72EWqJEttRFiIySAFgIIUSGVaxoZ9u2GF54wYtffkn4LyUmRqF3b2969SpF6dJGlwD38mUlxQlAMsPx41q6dfOmfn0rEyaYaNjQPWMsZxXbE08Q/fPPePfsifbEieyujpNy5w4en3yC8fPPsXTuTPywYdhDQ7O7WkKkiQTAQgghHkrBgiorV8YwZowH8+a5TtqwdGmg28+n06mUKmWnbFmVMmXslC3reJQpY8ffX2XuXANff23AanUNsvfv1/H00z60amVh3DgToaHun3Eus6hBQURv3ozXSy+h//HHzDmH0Yjq64vq4wM+Ps6/7y0TG4t+/XoUs+tNkIrVimHFCgwrVmBt3Jj4YcMcU03nurwTkZ9IACyEEOKh6fXw4YcmKle28+abHthsGe/hVRSVkiUdwW3iALdsWTvFi6upjp7x4Ycmhg0zM2WKkbAwParqWpefftLz0096unQx89Zb8ZQrl0sCYV9fYpctw/jBBxg//xxiY+G+INUZuN4LXu8PZu8rT7LPve30yc9OeD/TtWsY5s3DMH++S0rGPbqdO9Ht3ImtYkXihw3D8txz4OmZGa0hxENRIiMjZTyVPEzmCE+dtE/qpH0eTNooqV9+0dK3r+uEH4kFBLgGtQmBrkrJkna3jd7w558a3n3Xg82bkw/udDqVPn3MvP56PMWLZ/1/hxm+fu4NhZZdSc2xsRiWL8cwezbaM2dS3MxepAjmF1/EPGBAhsdmlvdY6qR9MkYC4DxO3hipk/ZJnbTPg0kbJS88XGHGDCOXLsUQEuLt0pNburQ9yzsF9+/X8s47Huzbl3zXsaenyqBB8YwYEY+/f9bVK9dfP3Y7us2bMc6ahW737hQ3U41GLN26OfKEk7szMhW5vo0ymbRPxkiCjhBCCLcLClKZMcPElClnmTTJxIABZlq2tFKxYtYHvwD169v44YcYwsJiePTRpDfBxcUpzJjhQWioHx9/bCQmJuvrmCtpNFifeoqYDRuI2rEDc9euqMmMEazEx2NYvBjfevXw6toV7S+/yIQeIltJACyEECJfUBRo2dLKzp3RzJ8fyyOPJA2E795VmDTJgxo1fJk710Ci+71EKuzVqxM3dy5RR44Q/8orqH5+yW6n37IFnw4d8GnUCP233yKNnMXsdjRHj6JfsQLd99+j274d7R9/oPnnH5SbN/PNv4fcBCeEECJf0Wigc2cL7dtbWLLEwLRpxiRjD1+/ruH11z2ZOdPIW2+Z6NLFIpOfpZFaqhSmSZMwvf46hiVLMM6Zg+bChSTbaf/8E68hQ7C/8w7mgQMx9+uHWrBgNtQ4j1NVNP/847hB8Zdf0O7alewNjC67eHmhFijwwAd+fsmX54IpGCUHOI+T3KDUSfukTtrnwaSNUpcb2ic2FubONfDJJ8YUb9wLCXFMptG6tXsn08gN7fPQrFZ0GzdinDkT3cGDKW6menlh7tUL85Ah2MuVc5bnizZ6CMm1j3L5MrpffnE8du1C8++/WVon1dMz2cDY+tRTWDp3ztK6pERSIIQQQuRrXl7w6qtmDh+OYuRIE15eSfuFTpzQ0r27N23aeLNnj3QFp4tOh7VDB2K2bCF682Ys7dujJjNGsBIbi3HuXHxq1cKrd2+0+/ZJnnAaKbduoVu7Fo+RI/GpXRu/kBC8hgzBsHx5lge/AEpcHJqrV9GePInu11/Rb9mCYeVKNMePZ3ldUiIpEEIIIQTg7w/jx8czcKCZDz80snBh0sk0DhzQ0batDy1bWhg/PndNppET2OrVI7ZePTTnzmGYMwfD0qUoie44VFQV/YYN6DdswFqrFkVatULTujX2KlXAaEzhyPlMdDS6ffvQ/fILVbZswevUKZR0fFlQ/fywNmgABgPKnTsod+7Af8/KnTso9sy5rtUCBTLluBkhAbAQQghxn4AAlenTTQwbFs/773skO5nGli16tmzR07mzYzKN8uUlEE4P+yOPYPrgA0xvvYXh668xfvklmitXkmyn+/13gn7/HaZMQdXpsFeqhC00NOFRrRqkcLNdnhIfj/bgQedEI9rffkOxWgFIy1cC1dMTa/36WJs0wda4MbbHHiPFpHZVhejohGA4jY+0BNA5KQCWHOA8TnKnUiftkzppnweTNkpdXmif48cdk2n8+GPyk2lotSrPPmuhcmXHGMdlyjieixdXH3jjXF5oH7cwm9GvWYNx5ky0x46la1fbI49gvz8oDg1FDQjIpIpmEZsN7ZEjaP+7cU23fz9KXFyad1d1Omy1a2Nt1MgR9Napk3W956kE0LbatbEHB2dNPR5AeoCFEEKIVFStamf58tgUJ9Ow2RRWrkx617tOp1KihErp0naXhyNAVilVSnqNnQwGLN26YXnuObQ7d2KcPRv95s1p2lV77hzac+fQr13rLLMHBLgExPbQUOxlyzqGAMmJVBXNyZOOYHfnTnS7dzt6VdPBVq0a1iZNHI/69cHXN5Mq+wCK4phe29cXtVSp7KlDGkgALIQQQqTBvck0tm7VMWmSB8eOpd69a7UqXLigcOFCykFX4cKhPPKI5r/AOGmwnF0xTLZRFGxNmhDbpAmakyfRr11L3N69FPjnHzQXL6b5MJpr19Bs2YJ+yxZnmernh+3RR7FVq5YQGFeuDPrke/YzzGpFuXsXJTLS8bhzJ+HvyEhHqkCidZpLl9DcuJGu09iCg7E2acKlChUo2rUrauHC7n0deZwEwEIIIUQa3ZtMo3nzaNas0fPee0bOns34qBAREQYiIuC335Jf7+/v6C1O2ousUrCgHQ8P8PBQMRodv3Dn1A7OjLBXqkT8G2/wz39pIsrt22iOHkV79CjaY8fQHj2K5tSpNN+wpdy9i27vXnR79zrLVIMBe5UqrnnFjz4KWm3yAWxKf99bvnMHJSoqc9qjRAmsjRs7engbN0YtWRKA26dPU0SC33STAFgIIYRIp/sn09i1S8fJkxouXrz3ULh4UUNExMNHo5GRGiIjeWBv8z16vYqHBxgMjmejMfGyo8xodATOBkNCAJ14+d62945jNELBgiqBgXaKFVOzfK4DtWBBbE2aYGvSJKEwNhbtiROOYPhecHziBIrJlKZjKmazI9f2yJFMqnXG2QsVwtaokTPotZcvj1sHoc7nJAAWQgghMkivh2bNrDRrlnRdTAxcupQ0ML73uHxZSTK6xMOyWBQsFoDMD5QKF7YTEOAIiAMDHc/3lu8v9/DIxEp4eWGrXRtb7doJZVYrmlOnHMHwfT3G6c2pzWqqtzfWhg0TAt5HH81bXfo5jATAQgghRCbw9oZKlexUqpT8T/QWC+zZcx6NppxLYHwvWL50SYPFknN7/CIiNEREOCYJSU2BAkmD4oCAhOfixR3PPj5uqphOhz0kBHtICJbu3R1lqopy/rxLQKw9ejTZodfcQfXzQ/X3dzwKFEj+70TL9qAg9+cjixRJACyEEEJkA70eSpY0U6GCDbAlWW+3w7VrSrI9yJcuaYiKUoiPB5NJwWx2POdEd+4o3Lmj5eTJ1Lfz8XEEwgEBKsWLJwTMhQr5UbYsD5dyoSioQUFYg4Kwtm+fUHzjhiMQ/i8g1h49iuaff0CrfXAAm8Lf+PqmPMauyDEkABZCCCFyII0GihdXKV7cRt26SQPkxFSV/wJhMJsVTCaIj1eSLMfH4wycHX+nbdu4OLh5U8O1awo3b7o/fSM6WiE6Wss//yReU5G33lJp08bCM89YaN7cire3e86pFi2KtXlzaN48odBmczS+5NvmaRIACyGEEHmAouC8eQ3uzXGVOXNdWSxw44bCtWsarl5VuHrV8XztmuPva9cc665dU7DZHj6QvHtX4bvvDHz3nQFPT5Xmza0884yFNm0s+Ps//OtxIb23+YIEwEIIIYRIF70eSpRQKVEi9Z5pmw0iIpT/gmON8/naNYUrVzQuAbPZnLZAOS5OYcMGPRs26NHpVBo1stKunZWnn7YQGCiT24q0kQBYCCGEEJlCq4VixVSKFVOBlMfrVVWIjFScvcj3guPTp7X88IPC7dvJ3xxmtSps365n+3Y9I0d6ULeujWeesdCunYWgIAmGRcokABZCCCFEtlIUxxjDBQuqVKkC998U+Pffp4mIqMz69Y5e30uXkh8aTFUVDhzQceCAjvHjPXn00YRgOCTELim9woUMMCeEEEKIHEurhccftzF1qoljx6LYsSOKkSNNVKqUevrFn39qmTrVg8cf96VWLR8mTPDg4EEtaZw4TuRx0gMshBBCiFxBUaB6dTvVq8czfnw8p05p2LBBz/r1Og4dSjmkOXtWy2efafnsMyPFi9tp29YxosTjj9uyZehdsxlu31a4dUvh9m2FmBiF4GA7QUHSU51VJAAWQgghRK5UsaKd//0vnv/9L56LFxU2bnSkSezdq8VuTz6SvHJFw7x5RubNM+Lvb+eppxwjSjRrZsXTM33nt9kc4xzfC2Rv3XL9+/bt+8s1zuXo6OTrVqaMnaZNrTRtaqVxYytFikgec2aRAFgIIYQQuV7p0iqDB5sZPNjMzZsKP/ygY+NGPdu361IcYSIyUsO33xr49lsDXl4qLVo4guEiRdRUg9l7z3fuuHc85AsXNCxebGDxYsesH9Wq2ZwBcYMGVry83HaqfE8CYCGEEELkKUWKqPTpY6FPHwt378LWrY40iS1b9Cn2vsbGKqxbp2fdupwzHfGxY1qOHdPy+edGDAaVunUTAuLq1W3oJIrLMGk6IYQQQuRZfn7QqZOFTp0smExx7NihY/16PZs26bh1K3vGAtBoHCNeFCrkeNZq4Y8/tMTHp9ybbDYr7N6tY/duHZMng5+fYwzkkJCiPPechuBgyR9ODwmAhRBCCJEveHhAmzZW2rSxYrXC3r1aNmzQs3Gjnn//zVgw7OenUqiQ3RnM3nu+/+9Che4vt+Pn55ht+X5xcXDggJYdO3Ts2KHjyBFtqukVd+86cp43bizL9OlQsqSdJk0cvcNNmlgJCMhZ+cM2m2O855zSa51DqiGEEEIIkXV0Omjc2EbjxjamTTNx6JCW9et1/PqrDq0Wl6A2cTB779nfX3VbQOfpCU2b2mja1AbEc+uWwq5dCQHxuXOpT9H8778ali0zsGyZI384JMTmDIgbNrTi6+ueet6jqnDnDty4oeHGDYWbNxVu3rz/b4UbNzRERCjcuOHImZ47N47OnS3urUgGSQAshBBCiHxNUaBmTRs1azqCz5ygUCGVDh2sdOhgBSA8XGHnTkcw/MsvOiIiUu+xPnFCy4kTWubMMaLTqdSpkxAQ16qVdPg3VYWoKIiISBrEJvztCHLvLVut6cu5uHEj5+RoSAAshBBCCJHDBQWpBAU5buyz2+HPPzX88ouOjRvNHDniR1xcysGl1aqwb5+Offt0TJ0KPj4q9es7Auv7e2lTy0F2h5s3JQAWQgghhBAZoNFAaKid0FAzbdqcpkyZCvz6q5ZffnH0EP/xR8rjIANERyts3Zr1o11ERUkALIQQQggh3MBohEaNbDRqZGPcuHgiI2HXLp0zID5zJvX84Yzy8VEpUsRO0aIqhQurFC2qUrSo/b6/VQoXTlhvMGRKNTJEAmAhhBBCiDzE3x/atbPSrp0jzeHiRYVffkkIiG/cSD5/2MNDpUgRRxBbpMi9vx1BbsLfCcvpnTkvJ5EAWAghhBAiDytdWqV3bwu9e1tQVThxQsNff2nx9nYNeL29yTdjCUsALIQQQgiRTygKVK1qp2pVe3ZXJVtlzxQoQgghhBBCZBMJgIUQQgghRL4iAbAQQgghhMhXJAAWQgghhBD5igTAQgghhBAiX1EiIyPV7K6EEEIIIYQQWUV6gIUQQgghRL4iAbAQQgghhMhXJAAWQgghhBD5igTAQgghhBAiX5EAWAghhBBC5CsSAOdi8+bNIzQ0lICAAJo0acLevXtT3f748eM8/fTTBAYGUqVKFaZNm4aq5s1BQD7++GOefPJJSpcuTfny5enWrRsnTpxIdZ/z58/j7++f5LF169YsqnXWmTJlSpLXWbFixVT3yU/XD0C1atWSvR6ee+65ZLfP69fPnj176N69O1WqVMHf35+lS5e6rFdVlSlTplC5cmUCAwNp27Ytf/311wOPu3v3bpo0aUJAQACPPfYYCxYsyKyXkKlSax+LxcLEiRNp2LAhJUqUoFKlSgwYMICLFy+mesxdu3Yle02dOnUqs19OpnjQNTRkyJAkr7VFixYPPG5+uIaAZK8Ff39/Ro0aleIx89o15E667K6AyJjVq1czevRoPvroI+rXr8+8efPo2rUr+/fvp3Tp0km2v3v3Ls8++ywNGzbk559/5vTp0wwbNgwvLy9efvnlbHgFmWv37t28+OKL1KxZE1VVef/99+nYsSMHDhygYMGCqe67atUqHn30Uefyg7bPrSpUqMCGDRucy1qtNsVt89v1A7B9+3ZsNptz+erVqzRt2pSOHTumul9evX5iYmIICQmhR48eDB48OMn6Tz/9lFmzZjFr1iwqVKjABx98wLPPPsvBgwfx9fVN9pjh4eE899xz9OrVi6+++or9+/czcuRIChcuTIcOHTL7JblVau0TGxvLkSNHGDVqFNWqVePu3buMGzeOLl26sGfPHnS61P8r3r9/v8t1VKRIkUx5DZntQdcQQNOmTfnyyy+dywaDIdVj5pdrCODkyZMuy4cOHaJ79+4P/EyCvHMNuZMEwLnUrFmz6NmzJ3379gVg+vTpbNu2jQULFjBx4sQk24eFhREXF8ecOXPw9PQkJCSEU6dOMXv2bIYPH46iKFn9EjLV6tWrXZa//PJLypQpw/79+3nqqadS3bdQoUIEBARkZvVyBJ1Ol+bXmd+uH0j6H8Q333yDr6/vA/+zyavXT6tWrWjVqhUAQ4cOdVmnqipz5sxhxIgRzqBjzpw5VKhQgZUrV9KvX79kj7lw4UICAwOZPn06AJUqVeK3335j5syZuS54Sa19ChQowPfff+9S9sknn1C/fn1OnjxJ1apVUz120aJFKVy4sFvrmx1Sa6N7jEZjut4/+eUaApK0yw8//EBwcDBPPPHEA4+dV64hd5IUiFzIbDZz+PBhmjVr5lLerFkzDhw4kOw+v/76Kw0aNMDT09NZ1rx5c65cucL58+cztb45QXR0NHa7HX9//wdu+/zzzxMcHEzr1q1Zu3Zt5lcum4SHh1OlShVCQ0Pp378/4eHhKW6b368fVVX55ptv6NatG15eXqlum1+un/udP3+ea9euuXwmeXp60rBhwxQ/k8BxXSX+HGvevDmHDh3CYrFkWn1zgqioKIA0fSY1bdqUSpUq0b59e3bu3JnJNcte+/btIzg4mFq1avHKK69w48aNVLfPr9dQdHQ0q1evdnaCPUh+uobSSgLgXCgiIgKbzUbRokVdyosWLcr169eT3ef69evJbn9vXV43evRoqlWrRt26dVPcxsfHh3fffZeFCxcSFhZG48aN6devHytWrMjCmmaN2rVrM3v2bMLCwvjss8+4du0arVq14tatW8lun9+vn+3bt3P+/Hmef/75FLfJT9dPYteuXQNI12cSpHxdWa1WIiIi3F/RHMJsNjNu3DjatGlDyZIlU9wuMDCQjz/+mG+++YZvvvmGChUq0KFDB/bs2ZOFtc06LVq04IsvvmDt2rVMnjyZ33//nfbt2xMfH5/iPvn1Glq5ciXx8fH06NEj1e3y2zWUHpICkYsl/tlZVdVUf4pObvvkyvOat956i/379/Pjjz+mmudauHBhl3zWGjVqcOvWLT799FO6deuWFVXNMi1btnRZrl27NtWrV2fZsmUMHz482X3y6/UDsGjRImrWrEloaGiK2+Sn6ycl6f1MSmmf5MrzCqvVysCBA7lz5w7ffvttqttWqFCBChUqOJfr1q3LhQsX+Pzzz3n88cczu6pZrnPnzs6/q1atSvXq1alWrRqbN2+mffv2Ke6X364hcHwmtW3b9oG5vPntGkoP6QHOhQoXLoxWq03Ss3Lz5s0k34TvKVasWLLbQ9Jem7xkzJgxrFq1inXr1hEUFJTu/WvVqsXZs2fdX7EcxsfHh8qVK6f4WvPr9QNw48YNfvjhhzT/1Hi//HL93MtNTM9nEqR8Xel0OgoVKuT+imYzq9XKiy++yPHjx1m7dm2GXmN+uaYAihcvTokSJVJ9vfntGgI4evQohw4dytBnEuSvayg1EgDnQgaDgerVq7N9+3aX8u3bt1OvXr1k96lbty779u3DZDK5bF+8eHHKli2bqfXNLm+++SYrV65k3bp1DxziKyXHjh3Lkzc0JWYymTh9+nSKrzU/Xj/3LFu2DKPRSKdOndK9b365fsqWLUtAQIDLZ5LJZGLfvn0pfiaB47rasWOHS9n27dupUaMGer0+s6qbLSwWC/369eP48eOsX78+w9dFfrmmwJHud+XKlVRfb366hu5ZtGgRZcqUoWnTphnaPz9dQ6mRFIhcatiwYQwaNIhatWpRr149FixYwNWrV513W7/zzjv8/vvvrFu3DoAuXbowbdo0hg4dyqhRozhz5gwzZszgjTfeyJM/E40aNYoVK1awZMkS/P39nTmK3t7e+Pj4AEnbaNmyZej1ekJDQ9FoNPz444/MmzePt99+O7teRqa5l39YqlQpbt68yfTp04mNjXXmk+X36+ceVVVZvHgxnTp1SjKUV367fqKjo529Rna7nUuXLnH06FEKFixI6dKlGTJkCB999BEVKlQgODiYDz/8EG9vb7p06eI8xqBBgwCcw1z169ePuXPnMnr0aPr168eBAwdYtmwZ8+bNy/oX+JBSa5/ixYvTt29fDh06xLfffouiKM7PJD8/P+fNpYnbZ/bs2ZQpU4YqVapgNpv57rvv2LhxI4sXL86GV/jwUmujggULMnXqVNq3b09AQAAXLlxg0qRJFC1alGeeecZ5jPx6Dd0b3jQ2NpawsDBeeeWVZD978/o15E4SAOdSnTp14tatW0yfPp1r165RpUoVvvvuO8qUKQM4xiw9d+6cc/sCBQqwZs0aRo0axZNPPom/vz/Dhg1LMd8zt7v34Zd4GJw333yTMWPGAEnbCODDDz/k4sWLaLVaypcvz8yZM/Nk/ubly5cZMGAAERERFClShNq1a7Nlyxa5fhLZtWsX//zzD1999VWSdfnt+jl06BDt2rVzLk+ZMoUpU6bQo0cP5syZw6uvvkpcXByvv/46kZGR1KpVi9WrV7t8cbh06ZLLMYOCgvjuu+946623WLBgAYGBgUybNi3XDV8FqbfP6NGj+eGHHwCS9NrNmjWLXr16AUnbx2KxMH78eK5cuYKHh4fzc/7eUFm5TWpt9PHHH3PixAmWL1/OnTt3CAgIoFGjRixcuFCuof/eY+AY4jMmJsZ5zSSW168hd1IiIyPz7lROQgghhBBCJCI5wEIIIYQQIl+RAFgIIYQQQuQrEgALIYQQQoh8RQJgIYQQQgiRr0gALIQQQggh8hUJgIUQQgghRL4iAbAQQgi38vf357XXXsvuagghRIokABZCiFxm6dKl+Pv7p/j48ccfs7uKQgiRo8lMcEIIkUuNHj2aRx55JEl5aGhoNtRGCCFyDwmAhRAil2revDl16tTJ7moIIUSuIykQQgiRR93LxV29ejX16tUjICCAhg0bsnnz5iTbXrx4kZdeeoly5coREBDAE088wbfffptkO1VVmTt3Lk888QSBgYGUK1eOjh07snfv3iTbbtmyhUaNGhEQEEDNmjVZuXKly3qr1cr06dOpVauW81itWrVi7dq17msEIYRIhvQACyFELnX37l0iIiKSlBcuXNj594EDB1izZg2DBg3Cx8eHRYsW0atXL9auXcvjjz8OQEREBG3atOH27dsMHDiQwMBAVq9ezZAhQ4iMjGTIkCHO47366qssXryYpk2b0rNnT1RV5ddff2Xfvn00bNjQud3BgwfZuHEj/fr14/nnn2fx4sUMHDiQatWqUalSJQCmTp3KRx99xPPPP0+tWrWIiYnh6NGj/Pbbb3To0CGzmk0IIVAiIyPV7K6EEEKItFu6dCnDhg1Lcf2lS5fw8fHB398fgM2bN1OvXj0Abt26Rc2aNalYsSI//fQTAOPGjWPmzJmsXbuWJk2aAGA2m3nqqaf4+++/OXHiBAUKFGDXrl20a9eOvn378umnn7qcU1VVFEUBHD3POp2OPXv2OIPd69ev8+ijjzJo0CDeffddABo1akSJEiVYsWKF+xpHCCHSQHqAhRAil5o2bZozwLyfp6en8+8aNWo4g1+AQoUK0bVrV+bOnUtkZCT+/v5s3ryZ0NBQZ/ALYDAYGDJkCAMGDGD37t20bduWdevWAY6AObF7we89jRo1cqlbsWLFqFChAuHh4c4yX19f/vrrL86cOUNwcHD6G0AIITJIAmAhhMilatas+cCb4MqXL59i2cWLF/H39+fChQu0a9cuyXb3AtgLFy4AcO7cOYoWLUrRokUfWLfSpUsnKfP39+f27dvO5TFjxtC7d29q165N5cqVadasGV26dKFmzZoPPL4QQjwMuQlOCCHysMQ9s+BIV0iLxNvdn+bwIFqt9oHHbNSoEUeOHGHOnDmEhoayfPlymjdvzscff5ymcwghREZJACyEEHnYmTNnkpSdPXsWSOilLVOmDKdOnUqy3enTp53rAcqVK8f169e5ceOG2+rn7+9Pjx49+Oqrrzh+/DgNGzZk2rRp2Gw2t51DCCESkwBYCCHysEOHDvHrr786l2/dukVYWBh16tRx3iTXunVrjh49ys6dO53bWSwWvvjiC7y8vHjiiScAaN++PQDvv/9+kvOktVf5frdu3XJZ9vT0pFKlSsTHxxMbG5vu4wkhRFpJDrAQQuRS27Ztc/bm3q969erO/N2QkBC6devGwIEDncOgRUVFMWHCBOf298YK7tGjB4MGDSIgIIA1a9Zw8OBB3n//fQoUKAA4UhZ69uzJwoULCQ8Pp1WrVoBjyLOqVasycuTIdNW/bt26NGzYkJo1a1KoUCH+/PNPFi9eTOvWrfH19c1oswghxANJACyEELnU1KlTky1/9913nQFwvXr1aNSoEVOnTiU8PJzy5cuzZMkSGjVq5Ny+cOHCbN68mXfeeYeFCxcSGxtLcHAwc+bMoUePHi7HnjlzJlWrVuWbb75h4sSJ+Pj48NhjjznHFE6PIUOGsGnTJnbu3InJZKJkyZKMGDGCESNGpPtYQgiRHjIOsBBC5FH+/v7069ePTz75JLurIoQQOYrkAAshhBBCiHxFAmAhhBBCCJGvSAAshBBCCCHyFbkJTggh8qjIyMjsroIQQuRI0gMshBBCCCHyFQmAhRBCCCFEviIBsBBCCCGEyFckABZCCCGEEPmKBMBCCCGEECJfkQBYCCGEEELkK/8Hx3dniod5sHUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = sbs_emb.plot_losses()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[380, 440],\n",
       "        [311, 331]])"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "StepByStep.loader_apply(test_loader, sbs_emb.correct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model II - GloVe + Transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransfClassifier(nn.Module):\n",
    "    def __init__(self, embedding_layer, encoder, n_outputs):\n",
    "        super().__init__()\n",
    "        self.d_model = encoder.d_model\n",
    "        self.n_outputs = n_outputs\n",
    "        self.encoder = encoder\n",
    "        self.mlp = nn.Linear(self.d_model, n_outputs)\n",
    "\n",
    "        self.embed = embedding_layer\n",
    "        self.cls_token = nn.Parameter(torch.zeros(1, 1, self.d_model))\n",
    "            \n",
    "    def preprocess(self, X):\n",
    "        # N, L -> N, L, D\n",
    "        src = self.embed(X)\n",
    "        # Special classifier token\n",
    "        # 1, 1, D -> N, 1, D\n",
    "        cls_tokens = self.cls_token.expand(X.size(0), -1, -1)\n",
    "        # Concatenates CLS tokens -> N, 1 + L, D\n",
    "        src = torch.cat((cls_tokens, src), dim=1)\n",
    "        return src\n",
    "    \n",
    "    def encode(self, source, source_mask=None):\n",
    "        # Encoder generates \"hidden states\"\n",
    "        states = self.encoder(source, source_mask)\n",
    "        # Gets state from first token only: [CLS]\n",
    "        cls_state = states[:, 0]  # N, 1, D\n",
    "        return cls_state\n",
    "    \n",
    "    @staticmethod\n",
    "    def source_mask(X):\n",
    "        cls_mask = torch.ones(X.size(0), 1).type_as(X)\n",
    "        pad_mask = torch.cat((cls_mask, X > 0), dim=1).bool()\n",
    "        return pad_mask.unsqueeze(1)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        src = self.preprocess(X)\n",
    "        # Featurizer\n",
    "        cls_state = self.encode(src, self.source_mask(X))\n",
    "        # Classifier\n",
    "        out = self.mlp(cls_state) # N, 1, outputs\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(33)\n",
    "# Loads the pretrained GloVe embeddings into an embedding layer\n",
    "torch_embeddings = nn.Embedding.from_pretrained(extended_embeddings)\n",
    "# Creates a Transformer Encoder\n",
    "layer = EncoderLayer(n_heads=2, d_model=torch_embeddings.embedding_dim, ff_units=128)\n",
    "encoder = EncoderTransf(layer, n_layers=1)\n",
    "# Uses both layers above to build our model\n",
    "model = TransfClassifier(torch_embeddings, encoder, n_outputs=1)\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "sbs_transf = StepByStep(model, loss_fn, optimizer)\n",
    "sbs_transf.set_loaders(train_loader, test_loader)\n",
    "sbs_transf.train(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAEQCAYAAAC++cJdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABgt0lEQVR4nO3dd1hT1/8H8PdNQsJU1MpQFApOrFtA3HW3fhU7sIp11b1aR121ah1Vq7VWq7V1tQ7qqtb5s1atFgci2lqsG3FA3SgiYYQk9/dHSuDKEDAQSN6v5+GRnNzxCdfAm8O55wgJCQkiiIiIiIishMzcBRARERERFScGYCIiIiKyKgzARERERGRVGICJiIiIyKowABMRERGRVWEAJiIiIiKrwgBMRERERFaFAZiIiIiIrAoDsIW7du2auUugYsDrbB14na0Dr7N14HU2LwZgIiIiIrIqDMBEREREZFUYgImIiIjIqjAAExEREZFVUZi7ACIiIrI8Wq0WarXa3GWUWLa2tnj69Km5yyjVHBwcoFAULsoyAFuwhw8FdOlSDwEBAvz8tGjSRIcGDXSwtzd3ZUREZMm0Wi2ePXsGZ2dnCIJg7nJKJJVKBVtbW3OXUWqJooiEhAQ4OTkVKgQzAFuwM2fkePBAiT17gD17bAAAcrmIOnX08PPTonFjHfz8dPDx0UPGwTBERGQiarWa4ZeKlCAIcHZ2RmJiIsqWLVvg/RmALdiZM/JsbTqdgKgoOaKi5FizxtBWtqyIJk0MPcQZH+XKicVcLRERWRKGXypqL/N/jAHYgp07lz0A5+TpUwGHD9vg8GEbY1u1ajpjD3GTJlrUqaOHjU0eByEiIiIqJRiALdjmzcmInrgY0Y5tsed+U5w9K8f16/kLxdHRckRHy7Fli+Gxra2Ihg0zQrGht7hyZfYSExERUenDAGzBVHE34L9xLpqmz0SP4GCk7pyOR/ZVcfasHGfOZHwokJj44j8hpKYKCA9XIDxcAUAFAHB316NJE51xPHGDBjo4OBTxiyIiIipFhg8fjsePH2NLRo9SPnTp0gW+vr5YuHBhEVZm3YSEhAR241ko+379YLNrl/GxaGuLtJEjkTZmDODkBADQ64HoaBkiIzMD8YULMuj1BR9Xk3GDXdbxxNWq8Qa74nDt2jVUr17d3GVQEeN1tg6WcJ2fPn1aqBuTzMnZ2TnP53v16oUVK1YU+LhPnz6FKIrZjp+amprrLBBPnjyBQqGA038/q4vKvHnzsHv3boSHhxfpeYpSYf+vsQfYQskuXpSEXwAQUlNhu2gRlBs2IHXqVKS//z5kcjlq1NCjRg09evdOBwAkJRnGD589K0dkpAJnzshx796LU2zWG+zWrjW0ZdxglzGeuHFjHcqX5+9cRERUsly5csX4+YEDB/Dhhx9K2p4Pq+np6bDJx80xhQln5cqVK/A+VDDsm7NQel9fJO3cieQcehFkDx7A/qOP4NiyJRS//57teUdHoEULHT76SIONG5Nx6dIz/PNPIn78UY2RI9PQtKkWtrb5C7EZN9gtWGCL4GAHeHuXQePGjhg61A6rVilx7pwM6ekv/XKJiIheiqurq/EjI7RmPE5NTYWnpyd+/vlndO3aFW5ubvjhhx/w+PFjDBw4EL6+vnBzc0PTpk2xceNGyXGHDx+O9957z/i4S5cuGD9+PObOnQtvb29Uq1YNn376KfR6vWSbCRMmGB/XrVsXCxcuxJgxY1ClShX4+vpi6dKlkvNER0fjzTffhKurK5o0aYLffvsNlStXRmhoaKG/JhcuXEBQUBDc3Nzg5eWF4cOHSxbvuHDhArp164YqVarAw8MDzZs3R1hYGADDLwgTJ05ErVq14OLigjp16uCzzz4rdC2mxh5gC6Zr0waXN2yA7+nTsP38c8ju35c8L794EQ5vv430Dh2QOns29LVq5XgcQQA8PER4eGjRvbsWAJCeDly4IENkpAKRkfIC3WB3/bph26w32DVooJOMJ65cWQRn0CEishzOzsU7JCIhwfSrrM2cORNz5szBN998AxsbG6SmpqJ+/fr46KOPUKZMGRw9ehRjx45FlSpV0Lp161yPs23bNgwcOBC//fYbzp8/j0GDBqFBgwZ49913c93n22+/xZQpU/Dhhx/i4MGDmDRpEpo2bQp/f3/o9Xq8//77cHFxwcGDB5GamoopU6YgLS2t0K81OTkZ7777Lho2bIjDhw/jyZMn+OijjzBq1Chs2LABADB48GC89tprOHz4MBQKBS5cuGDsKf/uu++wb98+rFmzBlWrVsWdO3dw7dq1QtdjagzAlk4uR3rfvkh/6y2oliyBatkyCKmpkk1sDh6E4vffoenfH2mTJ0OsWPGFh7WxARo00KNBAw0GDza0xccLhb7B7tQpBU6dyn6DXcZ4Yt5gR0RE5jZkyBAEBQVJ2j788EPj5/3790dYWBh+/vnnPANwzZo1MWnSJNja2qJatWpYt24d/vjjjzwDcNu2bTFkyBAAwNChQ/H999/jjz/+gL+/P44cOYJr165hx44dqFSpEgBg7ty56NSpU6Ff67Zt26BWq/H9998bxyJ//fXX6Nq1K2JiYuDt7Y3Y2FiMGjUKNWrUAAB4e3sb94+NjYWPjw+aNWsGQRBQpUoVBAQEFLoeU2MAthZOTkj79FNo+veH7ezZUD53N6qg00G1Zg2U27Yhddw4aIYNAwq4RGOFCiI6dtSiY0dDL/HL3GB3964Me/bIJCvY+fpKV7DjDXZERFScGjZsKHms0+mwePFi7NixA3fv3oVGo4FGo0GLFi3yPE6dOnUkj93c3PDw4cNC73P16lW4u7sbwy8ANGrUCLKX+CF55coV1KlTR3IjXkBAAGQyGS5fvgxvb2+MGDECH374ITZt2oTWrVujW7duxjAcEhKCt956C40bN0bbtm3RoUMHdOjQ4aVqMqWSUQUVG9HDAynff4+kI0egDQzM9ryQmAi7zz6Dk58fbHbsAMTC37Amk8F4c93ixak4diwJsbGJ2LcvCTNnpuB//0uHm5v+xQeC4Qa78+flWLtWhZEj7eHv74RXXy2Dt9+2x9y5Khw8qMDjxxwzQURERcfhuT9FfvPNN1i2bBk+/PBD7Nq1C8eOHUOXLl2g0WjyPM7zN88JggDxBT9v89rnRfsWRl7HzFiBbcqUKYiIiECXLl1w+vRpNG/e3Dg8okGDBoiKisL06dOh1+sxfPhwdO/eXTLW2ZzYA2yldA0bQv1//wfFnj2wnTED8hs3JM/LYmNh/8EH0H73HVI//xw6Pz+TnNfBAWjeXIfmzXUANBBF4N9/BWMP8Zkzcpw7J0dq6ovD7NOnAn7/3Qa//575TcHHR4f33kvHhx+mFbQDm4iIilBRjMk1t/DwcHTu3Bk9e/YEYAiN0dHRxT4FXM2aNXH37l3cvXsX7u7uAIC//vrrpcJmrVq1EBoaimfPnhl7gSMiIqDX61GzZk3jdj4+PvDx8cGwYcMwbtw4bNiwAX369AEAODk5oXv37ujevTtCQkLQvn17xMTEoFq1ai/xak2DPcAm1LNnT3h6eqJv377mLiV/BAHabt2QdOoUUubMgZjDG1Zx+jQcO3SA3cCBEG7dKooS4OEhont3LebMScWvv6oRG5uIo0efYeHCFPTooYGPjy7fx7t+XY65c23Rv789dPnfjYiIqMCqVauGsLAwhIeH4+rVq5gwYQJu375d7HW8/vrrqF69OoYPH47z588jMjISU6dOhUKhMPbW5iY1NRVRUVGSj+joaAQHB8Pe3h7Dhg3DhQsXcOLECYwdOxZdu3aFt7c3UlJS8PHHH+PYsWO4desWzpw5g1OnThnD8bJly/Dzzz/jypUriImJwbZt21CmTBnJMA1zYgA2oREjRuC7774zdxkFp1JBM2oUnv31F9KGDoWoyP6HAeX27XDy94fqs8+AxMQiLSfjBrvBgzVYuTIFZ88m4fr1RGzdqsbEialo2zYdZcrk/eeeX3+1weTJti8zgoOIiChPEyZMQKNGjRAcHIw333wT9vb2CA4OLvY6ZDIZNm7ciLS0NLRr1w7Dhw/H+PHjIQhCrottZLhx4wZatWol+Rg0aBDs7e2xfft2PHv2DO3atUNISAj8/PywbNkyAIBcLkdCQgKGDx8OPz8/vP/++/Dz88Pnn38OwND7u3TpUrRr1w6tW7fG+fPnsW3bNtjb2xf51yM/uBKciR07dgyrVq3C+vXrzV0KgMKtKCS7dg2206fDZv/+HJ/Xv/IK0j75BJq+fYEcwnJxyO8NdnPnpmDEiLzHYlkCS1g5il6M19k6WMJ1Lo0rwRW3vFaCM4Xz58+jZcuWOHr0KBo0aFBk5zG3wv5fM3sP8L179zBs2DD4+PjA1dUVAQEBOH78uEnPceLECfTs2RO1a9eGs7NzrpNCr169GvXq1YOrqytat26NkydPmrSO0kJfvTqSN21C0u7d0NWtm+152aNHsBs3Do4tWkBx6JAZKsz5Brvz55+hcmXpeKepU22xezeHuhMRkWXbs2cPfv/9d9y8eRNhYWEYMWIEXnvtNdSvX9/cpZVIZg3ACQkJ6NSpE0RRxNatWxEREYEFCxagYi7z0EZEROQ4qfPNmzdxK4/xqWq1Gr6+vpg/fz7s7Oxy3GbHjh2YPHkyxo8fj7CwMPj7+yM4OBixsbHGbQIDA3P8iIuLK+ArLx10rVoh6ehRJC9fDr2bW7bn5Zcvw+Hdd2H/zjuQXbxohgqlKlcWsWWLGk5OmX/UEEUBQ4bY48yZ/C3SQUREVBolJSVhwoQJaNq0KYYMGYKaNWtix44dLxwDbK3MOgRi1qxZOHHiBA4cOPDCbUVRRJs2beDm5oaNGzcapwOJjY3Fm2++ibfeeguzZs164XEqV66MBQsWoHfv3pL2du3aoU6dOpKlBRs1aoSgoCDMmDEj36/JEoZA5EithmrpUqiWLoWQkpLtaVEmg6ZvX6R98glEF5eXP99L+P13BYKD7aHTZb7pX3lFj4MH1Xj11ZIx/YqpWcKfTOnFeJ2tgyVcZw6BeLGiHgJhLUrlEIh9+/ahcePGGDBgAKpVq4YWLVpg5cqVOc49JwgCtm3bhpiYGAwYMABarRb//vsvunbtCj8/vwKF1OdpNBqcO3cObdu2lbS3bdsWERERhT6uRXFwQNqUKXh29iw0vXpBfO43SkGvh+rHH+HUqBFUX30F5BCSi0vbtlosXiw9/6NHMgQH2+PJE/4mTEREZO3MGoBv3ryJNWvWwMvLC9u3b8ewYcMwc+ZMrFq1KsftXVxcsHv3bly8eBH9+/dHt27dULduXaxcuRJyeeH/xB0fHw+dTpdt6EXFihXx4MGDfB8nKCgI/fv3x8GDB+Hr64vTp08XuqaSSqxUCSkrVhgW0shhpRshKQm2s2YZFtL4+eeXWkjjZfTtm46PP5Yu+RwdLUdIiD2eWwmaiIiIrIxZA7Ber0f9+vUxY8YM1K9fH++//z6GDh2K1atX57qPu7s71q5di71790KtVmPVqlVQmGgmgufHyYiiWKCxM7t27cL169dx9+5dXLx4Ef7+/iapqyTSN2gA9Z49UIeGQufjk+15WVwc7AcNgkP79pCfOmWGCoGpU9MQHCydASI8XIGRI+1QQhaiISIiIjMwawB2dXWVrCYCADVq1MjzprL4+HiMGDEC7du3h0KhwNixY196Wb0KFSpALpdn6+199OhRrjfkEQwLaXTpgqTwcKTMmwe9s3O2TRRnz8Kxc2fY9e8P4ebN4i4Py5aloFkzraR9+3YlPv9cVay1EBERUclh1gDctGlTREdHS9qio6NRpUqVHLd/8uQJunfvDnd3d4SGhmLv3r0ICwvDRx999FLrYCuVSjRo0ABHjhyRtB85cgQBAQGFPq7VUCqhGT4cSX/9hbQRIyA+t145ACh37oSTvz9sp08HEhKKrTSVCggNTUb16tJl4RYtssW6ddnrJCIiIstn1gA8YsQIREZG4ssvv0RMTAx27tyJlStXYtCgQdm2FUURPXr0QPny5REaGgqVSgUvLy/s3r0bhw4dwpw5c3I9T1JSknF5P71ej7i4OERFRUmmOBs5ciR++uknrF+/HleuXMGkSZNw7949DBgwoEheuyUSy5VD6ty5SDp1Cun/+1+25wWNBqqlS+HUuDGUq1YB6enFUle5ciK2bVOjYkXpXwrGjbPD4cOcI5iIiMjamH0luAMHDmDWrFmIjo6Gh4cHBg8ejKFDh+Y49vbo0aPw9/fPtoze1atXIZPJUK1atRzPcezYMXTt2jVbe69evbBixQrj49WrV2PJkiW4f/8+ateujblz56J58+Yv+QrNy5zT6ciPH4fd1KmQ//13js/ratRA6uzZ0HbsaBivUMTOnpXjf/9zQEpK5rkcHUXs35+EunVL96BgS5g2iV6M19k6WMJ1tuZp0ObNm4fdu3cjPDw8x8cZsk6DNmHCBFy8eBH79u0z6bmtQWH/r5k9AFPRMvs3Ur0eNlu3wnbWLMju3Mlxk/Q2bZA6Zw70r71W5OXs3atAnz72EMXMEFypkh4HDyahcuXS+1Yw+3WmYsHrbB0s4TqXxgD83nvvITU1Fbt27cr23JUrVxAQEIBffvkFr7/+ep7HeT6EJiUlQaPRoHz58pLtXiYA37p1C/Xr18eRI0fQsGFDY3tu5zK14cOH4/Hjx9iyZUuRnic/SuU8wGQFZDKk9+yJZ2fOIPWTTyA+13sPADZHj8KxZUvYjR4N4d69Ii3nf//TYu5c6Txod+7I0KOHAxITi/TURERUgvXt2xdhYWE5riy7YcMGVKlSBa1bty7wcR0dHYs8kJrjXKUdAzAVD3t7pE2caFhI4/33sy+kIYpQbtgAp8aNoVq4EEhOLrJShg/XYOhQ6ZLaFy7IMWCAfXENSyYiohKmU6dOcHFxQWhoqKQ9PT0dW7Zswfvvvw9RFDFq1CjUq1cPbm5uaNSoEZYsWZLnbFTz5s1DYGCg8bFOp8Onn36KmjVrwtPTE5MnT4ZOJ71R+9ChQ3jjjTfg6ekJLy8vvP3227hy5Yrx+fr16wMAXn/9dTg7O6NLly45nkuv12PBggWoU6cOXFxc0KxZM0kv861bt+Ds7Ixdu3YZJxkICAjINilAQZ04cQLt2rWDq6srqlevjilTpkCj0Uieb9++PSpXroyqVauiXbt2uHjxIgBDj+6QIUNQrVo1uLq6on79+vj2229fqp6cMABTsRLd3ZGybBmS/vgD2latsj0vqNWw/fxzw0IaW7agqCbsnTs3FW++KU27hw/bYPx4O3Ot3UFEZNHKOjsX60dBKRQK9OrVCz/99JMk0O7fvx/x8fHo3bs39Ho93N3d8eOPPyIiIgLTpk3DokWLsHHjxnyfZ9myZVi/fj0WLFiAgwcPQqfTYdu2bZJt1Go1hg0bht9//x179+5FmTJl0LNnT2OI/P333wEA27dvx5UrV3I9/4oVK/DNN9/gs88+w8mTJ9GlSxf06dMHUVFRku3mzJmDoUOH4vjx42jYsCE++OADJCUl5fs1ZXXnzh0EBwejXr16CAsLwzfffIPt27dj5syZAACtVouQkBA0bdoUx48fx6FDhzBs2DDjgmZz5szBxYsXsWXLFpw+fRrLli1DpUqVClVLXhiAySz09epBvWsX1Js3Q5fDWDfZv//CfuhQOLRrB/nJkyY/v1wOrF6djEaNpHMEr1+vxOLFnCOYiMga9enTB3FxcTh69KixbePGjWjbti08PDxgY2ODqVOnolGjRvD09MRbb72FDz74ANu3b8/3OVasWIEPP/wQQUFBqFGjBr744gu4uLhItgkKCkJQUBB8fHzw2muvYfny5bh16xbOnj0LwLB+AQCUL18erq6uKFeuXI7nWrZsGUaNGoXg4GBUq1YNU6dORWBgIJYtWybZbsSIEXjjjTfg4+OD6dOn48mTJzh//ny+X1NWa9asgaurKxYtWoSaNWuic+fOmDFjBlatWoXk5GQ8e/YMT58+RefOnfHqq6+iRo0aCA4ONq4LERsbi3r16qFx48bw9PREy5Yt0b1790LVkhcGYDIfQYC2c2cknTyJlAULoM9h3JLir7/g+OabsO/TB7KYGJOe3t4e2Lw5GVWrSnuZZ82yxc8/c45gIiJr4+Pjg2bNmhl7VO/evYvDhw+jT58+xm3Wrl2LNm3awMfHB5UrV8a3336b5wJeWT19+hT37t2Dn5+fsU0mk6Fx48aS7W7cuIFBgwahQYMGqFKlCmrUqGGcxjW/EhMTcffuXTRt2lTSHhgYiMuXL0va6tSpY/zc3d0dAPDw4cN8nyurK1euwM/PDzJZZsQMDAyERqNBTEwMypUrh5CQELzzzjvo0aMHli1bJnldAwcOxM6dO9G8eXN8+umnOH78eKHqeBEGYDI/GxtohgzBsz//RNqoUTkupGGzZw8cAwJgO3WqSRfScHExzBFctqx03MOIEXY4cUJusvMQEVHp0LdvX+zbtw9PnjzBTz/9hHLlyuHNN98EAOzYsQNTpkxBSEgItm/fjmPHjmHgwIGS8a2m0LNnTzx69Ahff/01Dh06hLCwMCgUCpOd5/mpZm2y/NzNeK6wC4yJopjjVLZZj/3tt9/i0KFDaNasGfbv348mTZrg8OHDAIAOHTrg/PnzGD16NOLj4/Hee+9hxIgRhaolLwzAVHI4OyN1zhwknT6N9KCgbE8L6elQLV8Op4YNofzuO5MtpFGzph6hoWrY2GS+2TUaAb172+PaNb5FiIhM4WlCQrF+FFZQUBBUKhW2bNmCjRs3omfPnsaAGB4ejsaNG2PIkCFo0KABvL29cePGjXwfu2zZsnBzc8OZM2eMbaIo4s8//zQ+fvz4Ma5cuYJx48ahTZs2qFmzJp49ewatNnPInlKpBIBsN89lVaZMGbi7u+PUqVOS9vDwcONwg6JQq1YtREZGSsZRh4eHQ6lU4tVXXzW21a1bF2PGjMG+ffvQokULbNq0yfhchQoV0LNnT+MY5k2bNiEtTXrz+sviT3cqcfSvvorkdeuQtH8/tI0aZXte9uQJ7CZPhmNgIBT/938wxV1rLVrosHx5iqQtIUGG4GB7PHxY9It0EBFRyWBnZ4fg4GDMnz8fN27ckAx/qFatGqKionDw4EFcv34dCxYswMkC3qcybNgwLFmyBHv27MG1a9cwefJk3L9/3/i8s7MzKlSogPXr1yMmJgbHjx/HuHHjoFBkrlxasWJF2NnZ4fDhw3jw4AGePn2a47lGjx6NZcuW4eeff0Z0dDQ+//xzhIeHY9SoUQX8qmSXmJhoXGU34+PWrVsYOHAg7t27h/Hjx+PKlSs4cOAAZs6cicGDB8Pe3h43b97EZ599hoiICNy+fRthYWG4cOGCMZR//vnn2Lt3L65fv44rV65gz5498PLygkpl2vtzGICpxNIFBkJ96BCSV66EvnLlbM/Lo6PhEBICh27dIMtltbmC6NEjHVOnSucIvnlTjl697ItyVjYiIiph+vTpg4SEBAQEBEh6SwcMGIDu3btj0KBBeP3113H79m2MHDmyQMceNWoUevfujfHjx6Ndu3bQ6/UIDg42Pi+TybB27VpcuHABgYGBmDBhAqZOnSoJgAqFAl988QU2bNiAWrVqISQkJMdzDRs2DKNHj8aMGTMQGBiIffv2Yf369ahXr14BvyLZhYeHo1WrVpKPadOmoVKlSti2bRuioqLQsmVLjBo1Cu+88w6mT58OALC3t0d0dDT69++PJk2aYMSIEQgODsaYMWMAACqVCnPmzEGLFi3QqVMnJCUlYfPmzS9d7/O4EpyFs4QVhQAAyclQffstVF9/DSGHqVlEQUB6SAhSP/0U4n8D+AtDFIHRo+2wcaNS0v6//6Vj3bpkyEvosGCLuc6UJ15n62AJ17k0rgRX3LKuBEeFx5XgyLLZ2yPt448NC2n06wdRJv2vK4gilKGhhoU0Fi0q9LAIQQAWL05BmzbS8cV799pg2jR+oyIiIrIEDMBUqoiurkhZsgRJYWFIb9Mm2/NCcjJsZ8+Gau7cQp/DxgZYty4Zvr7Smwu+/VaF779X5rIXERERlRYMwFQq6V97Dcm//AL11q3Q5XA3q2rxYsguXSr08cuWBbZuVcPNTTpH8JQptvi//1PkshcRERGVBgzAVHoJArQdOyLpxAmkLFoEMcsYIEGrhd3YsS+1lLKHh4gtW9RwdMwcTqHXCxg40B5//llCBwMTERHRCzEAU+mnUEAzcCCSly6VNp86BZsCrM+ek/r19fjhh2TI5ZkhOCVFQM+e9rh1i9OjERERlUYMwGQxtN26Ib1TJ0mb7fTpEAq5nGOGDh20+PJL6fRoDx7I0KOHgykXpSMisiiFXUmMKL9e5v8YAzBZDkFAyoIFEO3tjU2yhATD8skvacAADcaMkYbgK1fkeP99B5h4BUwiolLPwcEBCQkJDMFUZERRREJCAhwcHAq1P+/mIYsienoidfJk2P034TYAKLduhaZ3b+hat36pY0+fnoZbt2T45ZfMmSCOH1dg9Gg7fPddCnJZ+pyIyOooFAo4OTkhMTHR3KWUWImJiShTpoy5yyjVnJycJCvkFQQDMFkczfDhUG7ZAvmFC8Y2u3HjkHTiBPASk47LZMCKFSm4e1eGU6cy3zpbtijh6anHJ5+Ydp1yIqLSTKFQcDGMPDx48ABVqlQxdxlWi0MgyPLY2CDl668hZumSlV+/DtVXX730oW1tgdDQZPj4SOcIXrDAFhs32rz08YmIiKjoMQCTRdL5+UHzwQeSNtXixZBdvfrSx65QQcS2bcmoUEE6xdqYMXY4epTToxEREZV0DMBksVKnTYPe1dX4WEhPN8wNbIKbMry99di0KRkqVeaxtFoBffs64OJFvq2IiIhKMv6kJsvl7IzUefMkTYoTJ2Dz008mOby/vw4rVyZL2hITBfTo4YC7d3lHHBERUUnFAEwWLf2tt5Derp2kzXbaNAjx8SY5flCQFrNnp0ja4uJkeO89ByQlmeQUREREZGIMwGTZBMGwTHKW2R9kjx/Ddto0k51i1CgNBg2SzgARFSXHBx/YQ6s12WmIiIjIRBiAyeKJXl5ImzhR0qb86SfIjx0zyfEFAZg/PxWdOqVL2n/7zQYTJ9qaYsgxERERmRADMFmFtFGjoKtVS9JmN24ckGaauXsVCmDNmmTUry+dHm3tWhWWLVPmshcRERGZAwMwWQelEimLF0ua5NeuQbVkiclO4egIbNmihoeHdHq0adPssHMn15whIiIqKRiAyWroAgOh6dtX0qZatAiy69dNdg43NxHbtqlRpox03MPQofaIiOAcwURERCUBAzBZldSZM6F/5RXjYyEtzTAUwoQDdWvX1mPDBjUUisxjpqUJ6NXLHtev8y1HRERkbvxpTFZFLFcOqZ9/LmlT/PEHbLZtM+l5WrfWYelS6fRojx/LEBxsj/h4zhFMRERkTgzAZHXSe/SAtnVrSZvtJ59AePLEpOcJCUnHpEmpkraYGDlCQuyRkpLLTkRERFTkGIDJ+ggCUr76CqJKZWySPXoE288+M/mpJk9OQ8+eGklbRIQCw4fbQa/PZSciIiIqUgzAZJX0Pj5IGz9e0qZctw7y8HCTnkcQgKVLU9CypXRFjJ07lZg50zaXvYiIiKgoMQCT1Ur76CPoqleXtNmNGwdoNLnsUThKJbBhgxq1aknnCF6yRIW1azlHMBERUXFjACbrpVIh5auvJE3yS5egWrbM5KdydjbMEeziIh338PHHtvjtN84RTEREVJwYgMmq6Vq2hCYkRNKmWrAAws2bJj+Xp6eILVuSYW+fOT2aXi9gwAB7nDvHtyIREVFx4U9dsnqps2dDX7688bGQmgq78eNNOjdwhoYNdVizJhkyWeax1WoBPXs6IDaW06MREREVBwZgsnpihQpInT1b0mZz+DBsfvmlSM73xhtafPGFdHq0e/dkeO89Bzx9WiSnJCIioiwYgIkApIeEQNu8uaTNdsoUICGhSM43eLAGI0emSdouXpSjb18HU9+DR0RERM9hACYCDHMDL14M0cbG2CS7fx+2z/UMm9Ls2ano1i1d0vbHHwqMHWtXFKMviIiI6D8MwET/0deogbQxYyRtyrVrIY+MLJLzyWTA998nw89POkdwaKgSCxeqctmLiIiIXhYDMFEWaePHQ+ftbXwsiCLsxowB0tNz3+kl2NkBP/2UDC8v6RzBc+faYssWm1z2IiIiopfBAEyUla1t9rmBL1yA8rvviuyUFSuK+PnnZJQrJ50jeNQoO4SFyYvsvERERNaKAZjoObo2baDp0UPSZjtvHoTbt4vsnNWq6REamgylMnPwb3q6gPffd8Dly3ybEhERmRJ/shLlIPXzz6F3djY+FpKTYTdhQpHMDZyhWTMdVqxIkbQlJgoIDnbA/fucI5iIiMhUGICJciBWrIjUWbMkbTYHDkCxe3eRnvedd9IxY4Z0juDYWBl69rSHWl2kpyYiIrIaDMAm1LNnT3h6eqJv377mLoVMIP3996Ft2lTSZjd5MpCYWKTnHTMmDf36SScD/usvBQYNsodOl8tORERElG8MwCY0YsQIfFeEN0tRMZPJDHMDKxSZTXfvwnbOnCI9rSAAixaloH176cwT+/fbYMoUW84RTERE9JIYgE2oVatWcHR0NHcZZEL62rWR9uGHkjblqlWQ//VXkZ5XoQB++CEZr70m7fJduVKFFSuURXpuIiIiS1diAvCiRYvg7OyMCRMmmPzYJ06cQM+ePVG7dm04OzsjNDQ0x+1Wr16NevXqwdXVFa1bt8bJkydNXguVPmkffwy9p6fxsXFuYK02951MwMkJ2LpVjUqVpNOjTZ1qi927FbnsRURERC9SIgJwZGQk1q1bhzp16uS5XUREBNLS0rK137x5E7du3cp1P7VaDV9fX8yfPx92dnY5brNjxw5MnjwZ48ePR1hYGPz9/REcHIzY2FjjNoGBgTl+xMXF5fOVUqlkb4+URYskTfK//4Zy5coiP3WlSiK2blXDySlz3IMoChgyxB5nznCOYCIiosIwewB++vQpBg8ejG+++QbOWaadep4oipg4cSL69u2L9CyrcsXGxqJr165Ys2ZNrvt27NgR06dPR1BQEGSynF/y8uXLERISgn79+qFmzZpYuHAhXF1dsXbtWuM24eHhOX54eHgU/IVTqaJt3x6at9+WtNnOnQuhGH75ee01PdatS4ZcnhmCU1MF9Oxpj5s3OT0aERFRQZk9AI8ZMwZBQUFo3bp1ntsJgoBt27YhJiYGAwYMgFarxb///ouuXbvCz88PM2bMKHQNGo0G586dQ9u2bSXtbdu2RURERKGPS5Ylde5ciGXKGB8LSUmwmzSpWM7dtq0WixdL5wh+9EiG4GAHPHnCEExERFQQZg3A69atQ0xMDKZOnZqv7V1cXLB7925cvHgR/fv3R7du3VC3bl2sXLkScnnh/xwcHx8PnU6HihUrStorVqyIBw8e5Ps4QUFB6N+/Pw4ePAhfX1+cPn260DVRySO6uSH1uV+0bPbtg2LfvmI5f9++6Rg/XjpH8LVrcoSE2EOjYQgmIiLKL7MF4GvXrmHWrFlYtWoVlMr839Xu7u6OtWvXYu/evVCr1Vi1ahUUCtPcECQI0hAhimK2trzs2rUL169fx927d3Hx4kX4+/ubpC4qOTQDBkDbpImkzW7SJCApqVjO/+mnaQgOls4RHB6uwKxZXtDrc9mJiIiIJMwWgE+fPo34+HgEBgaiQoUKqFChAk6cOIHVq1ejQoUKOd7sBhh6a0eMGIH27dtDoVBg7Nix0L/kT/4KFSpALpdn6+199OhRtl5hsnIyGVK+/hpilr84yOLiYDtvXrGcXhCAZctS0KyZdAaKAwcqYNo0zhFMRESUH2YLwF26dMHJkydx7Ngx40fDhg3xzjvv4NixYzn2Cj958gTdu3eHu7s7QkNDsXfvXoSFheGjjz6C+BI/+ZVKJRo0aIAjR45I2o8cOYKAgIBCH5csk/6116AZOVLSplyxArK//y6W86tUQGhoMqpXl84RvHy5CvPnq4qlBiIiotLMbAHY2dkZvr6+kg97e3uUK1cOvr6+OQ5H6NGjB8qXL4/Q0FCoVCp4eXlh9+7dOHToEObksTpXUlISoqKiEBUVBb1ej7i4OERFRUmmOBs5ciR++uknrF+/HleuXMGkSZNw7949DBgwoMi+BlR6pU6aBH2VKsbHgl4Pu7FjUVxrFZcrJ2LbNjVeeUX6148vvrDF0qVcKIOIiCgvBQ7AJ06cyLbc77Zt29CkSRNUq1YNkyZNeukhCTkRBAFTp07Fpk2bYGtra2z38fHBrl270KtXr1z3/euvv9CqVSu0atUKKSkpmDdvHlq1aoW5c+cat3n77bcxb948LFy4EC1btsSpU6ewdetWVK1a1eSvhSyAgwNSFi6UNCn+/BPKLNPmFTUvLxHbt6tRtqz0rx/Tp9th9WqGYCIiotwICQkJBRo70K1bN1SoUAE//PADAODq1ato3rw5Xn31VXh6euLw4cOYM2cORowYUSQFU8Fcu3YN1atXN3cZFsu+b1/Y7N5tfCw6OeHZ6dMQ3d2LrYbISDm6dbNDSop0JpRvv01GSEh6LntRacT3s3XgdbYOvM7mVeAe4MuXL6Nx48bGx1u3boWdnR0OHTqEbdu24b333sPGjRtNWiRRSZUyfz5EJyfjY+HZM9hOmVKsNfj56bB4cTRsbaW/y44aZYedO7lkMhER0fMKHIATExMlK7YdPnwYr7/+Osr8t0BAYGAgbt++bbICiUoysVIlpH76qaRNuXMnFL/9Vqx1NG78DBs2JMPGJjME6/UCBg2yx4EDDMFERERZFTgAu7q64sqVKwCAu3fvIioqSrKCWmJi4kstSkFU2mgGDYK2YUNJm9348YBaXax1dOigxZo10iWTtVoBffva448/+J4kIiLKUOAA3LVrV6xatQoTJ05Enz59oFKp8MYbbxif/+eff+Dl5WXKGolKNrkcKYsXQ5Rlvp1ksbGwXbCg2Evp1k2Lb79NgSBkhuC0NAG9ejng1CmGYCIiIqAQAXjKlCno1q0btm7divv372PZsmVwcXEBYOj93bNnD15//XWTF0pUkukbNIBm6FBJm3L5csguXCj2Wt57Lx1ffSVdMjk5WUCPHg44d86sq58TERGVCAUeHOjg4ICVK1fm+JyjoyMuXrwIe3v7ly6MqLRJ/eQT2OzeDdm//wIABK0WdmPGQH3gACAr3uA5YIAGycnA1Kl2xrbERAFvv+2AvXvV8PXluslERGS9TPZT+d69e7h69SrKli0LGxsbUx2WqPRwckLKF19ImhSRkVCuW2eWckaO1OCTT6Q9wY8fy/DWWw64fp09wUREZL0K/FPwhx9+wNDn/tQ7fvx4+Pr6olmzZmjZsiXi4+NNViBRaaL93/+Q/uabkjbbGTMg3L9vlnomTEjDRx+lSdru35chKMgBt28LuexFRERk2QocgNetWwenLPOehoWFYe3atXj33Xcxffp03LhxA19++aVJiyQqTVIWLIDo4GB8LCQmwnbqVLPUIgjAZ5+lYvBgaQiOi5Ohe3cH3LvHEExERNanwAH41q1bqFWrlvHxzp07UblyZXz33XcYM2YMBg8ejP3795u0SKLSRPTwQOpzi2Eof/4Zit9/N0s9ggB88UUqQkI0kvaYGDm6d3dAfDxDMBERWZcCB2CNRiMZ43vkyBG0b98esv9u8vH29sa9e/dMVyFRKaQZNgy6unUlbbbjxgEpKWapRyYDvvkmBW+9JQ3Bly/L8dZbDkhIMEtZREREZlHgAOzp6YmjR48CAP7880/cvHlTshDGgwcPJEMkiKySQoGUr7+GKGT2rspv3oRq0SKzlSSXA99/n4JOndIl7VFRcvTo4YCkJDMVRkREVMwKHIA/+OAD7Ny5E82aNcPbb7+NypUro0OHDsbnT506JRkiQWStdI0bQzNokKRNtWQJZJcvm6kiQKkE1q1LRuvWWkn76dMK9OrlYK4OaiIiomJV4AA8aNAgLFmyBN7e3njjjTewfft22NkZ5hp98uQJHj58iODgYJMXSlQapX76KfRubsbHQno67MaOBfTmm4fX1hb46Sc1mjaVhuBjxxTo398eGk0uOxIREVkIISEhQXzxZlRaXbt2DdWrVzd3GVZNsWsXHPr1k7QlL12K9L59TXaOwlznp0+BoCAHnDsnXQ8nKCgda9YkQ1HgZXKoqPH9bB14na0Dr7N5vdRs+BcuXMD+/fuxf/9+XDDDkq9EpYG2Wzekd+woabOdPh3Cw4dmqsigbFlgx45k+PrqJO27dtlg1Cg7c3ZSExERFalCBeB9+/ahXr16aNmyJXr37o3evXujZcuWqF+/Pvbt22fqGolKN0FAysKFEO0ylyWWJSTA9tNPzViUQfnyIn75RQ1vb2kI3rxZiQkTbCHy70NERGSBChyADx06hL59+0IURUybNg0bN27Ehg0bMG3aNIiiiH79+uHw4cNFUStRqSV6eiJ18mRJm3LLFsj/+MNMFWVydRWxa5caHh7SLt81a1SYPp0hmIiILE+BxwB37NgRSUlJOHDgQLbpzp49e4ZOnTqhTJky+PXXX01aKBUOxxiVIOnpcGzdGvKLF41NOh8fJJ04Ybgz7SWY4jrHxMjw5psOuHdP+nvx5MmpmDw5LZe9qDjx/WwdeJ2tA6+zeRW4B/iff/5B7969c5zr18nJCb1790ZUVJRJiiOyKDY22ecGvn4dqsWLzVhUJm9vPXbuVKN8eWlP8Pz5tvjmG6WZqiIiIjK9AgdgGxsbJCcn5/q8Wq2WrBRHRJl0/v7QDBggaVMtXgzZtWtmqkiqVi09duxQo0wZ6R+Gpk2zw5o1DMFERGQZChyAAwMDsWrVKly/fj3bczExMVi9ejWaNWtmkuKILFHq9OnQu7gYHwsajWFu4BIy2LZBAz1+/lkNBwdpPePH22HTJv5yS0REpV+BZ/qcMWMGOnXqhMDAQLzxxhvG8StXr17FgQMHoFKpMGPGDJMXSmQxnJ2ROncu7LOsEqc4fhw2mzcjvVcvMxaWyd9fh02b1OjRwwGpqZlDNkaOtIO9vYigIG0eexMREZVsBe4Brl27No4cOYLOnTvj8OHDWLRoERYtWoTff/8dnTt3xi+//AKtlj8cifKS/s47SG/bVtJm++mnEB4/NlNF2bVqpcP69cmwscnsCdbrBQwcaI8DB7hKBhERlV6FmgfYx8cH69evx+3bt3HlyhVcuXIFt2/fxrp163DkyBG0atXK1HUSWRZBQOqiRRCzzP4gi4+H7fTpZiwqu44dtVi9OhkyWWYI1moF9O1rjz/+kJuxMiIiosJ7qZXgZDIZXFxc4OLiApnspQ5FZHX0r76KtAkTJG3KjRshP3HCTBXlLChIi2+/TZG0paUJCAlxQEQEQzAREZU+TK1EZpQ2ejR0tWpJ2uzGjgXSSta8uz17puOrr6QhWK0WEBzsgHPn+G2EiIhKF/7kIjInpRIpX30laZJfvQrV0qVmKih3H3ygwezZ0hCcmCjg7bcdcOkSv5UQEVHpwZ9aRGama9YMmj59JG2qL7+ELCbGTBXlbvRoDaZMSZW0PX4sQ/fuDoiJ4bcTIiIqHfJ1K/fZs2fzfcA7d+4Uuhgia5U6axYU+/dD9ugRAEBIS4PtuHFI/uUXIMvKcSXBxIlpUKsFLF2qMrbdvy9Dt24O2L8/CVWqlIz5jImIiHKTrwDcvn17CPn8ISyKYr63tTQ9e/ZEeHg4WrdujfXr15u7HCpFxHLlkDpnDuyHDTO22Rw9Cpuff0Z6cLAZK8tOEICZM1ORnAysXp0ZguPiZAgKcsD//Z8abm4MwUREVHLlKwAvX768qOuwCCNGjEC/fv2wadMmc5dCpVD6e+9B+9NPUISFGdtsP/kE6R06AM7O5issB4IALFiQCrVawKZNmUskx8TI8dZbDti7V40KFRiCiYioZMpXAA4JCSnqOixCq1atcOzYMXOXQaWVICDlq6/g2KwZBI0GACB7+BC2n32G1K+/Nm9tOZDJgG++SUFKCrBzZ2YIvnRJjrffdsDu3UkoW9aMBRIREeXCrHetrFq1Cs2aNUOVKlVQpUoVdOjQAQcOHDD5eU6cOIGePXuidu3acHZ2RmhoaI7brV69GvXq1YOrqytat26NkydPmrwWorzoq1VD2rhxkjbVjz9CHhFhporyplAAK1emoFOndEn733/L0aOHA5KSzFQYERFRHswagCtVqoSZM2fijz/+MK4g17t3b/zzzz85bh8REYG0HOZHvXnzJm7dupXredRqNXx9fTF//nzY2dnluM2OHTswefJkjB8/HmFhYfD390dwcDBiY2ON2wQGBub4ERcXV8BXTpS7tLFjoatWTdJmN3YskJ6eyx7mpVQC69Ylo1Ur6RLoEREKhIQ4IDU1lx2JiIjMxKwBuEuXLujQoQO8vb1RrVo1TJs2DY6OjoiMjMy2rSiKmDhxIvr27Yv0LEEgNjYWXbt2xZo1a3I9T8eOHTF9+nQEBQXlumLd8uXLERISgn79+qFmzZpYuHAhXF1dsXbtWuM24eHhOX54eHi8xFeB6DkqVfa5gS9ehLIEj8W3tQV++kmNgABpCA4LU6BfP3v8N6KDiIioRCgxE3fqdDps374darUa/v7+2Z4XBAHbtm1DTEwMBgwYAK1Wi3///Rddu3aFn58fZsyYUehzazQanDt3Dm3btpW0t23bFhEl9E/PZNl0rVpB06uXpM32iy8g3LxpnoLywdER2LJFjfr1dZL2AwdsMGSIHbTaXHYkIiIqZmYPwBcuXEDlypXh4uKCsWPHYuPGjahTp06O27q4uGD37t24ePEi+vfvj27duqFu3bpYuXIl5HJ5oWuIj4+HTqdDxYoVJe0VK1bEgwcP8n2coKAg9O/fHwcPHoSvry9Onz5d6JqIUufMgb5cOeNjISUFdh9/DIgld3YFZ2dgxw41ateWhuCdO5UYPdoOer156iIiIsrK7AG4evXqOHbsGA4dOoSBAwdi+PDhuHjxYq7bu7u7Y+3atdi7dy/UajVWrVoFhSJfk1m80PPzFxd0TuNdu3bh+vXruHv3Li5evJhjTzZRfokVKiB19mxJm82hQ7DZudM8BeVThQoifvlFDW9vaQjetEmJiRNtS3J+JyIiK2H2AKxUKuHt7Y2GDRtixowZqFu3Lr799ttct4+Pj8eIESPQvn17KBQKjB07FvqX7FaqUKEC5HJ5tt7eR48eZesVJipO6b17Q9usmaTNdvJk4OlTM1WUP25uInbtUsPDQ/reXL1ahRkzGIKJiMi8zB6An6fX66HJ5Y6ZJ0+eoHv37nB3d0doaCj27t2LsLAwfPTRRxBf4ieqUqlEgwYNcOTIEUn7kSNHEBAQUOjjEr00QUDK4sUQbWyMTbL792E7Z44Zi8qfKlVE7N6thqurNAQvXarCggWqXPYiIiIqemYNwJ999hlOnjyJW7du4cKFC5g5cyaOHz+O4ByWfhVFET169ED58uURGhoKlUoFLy8v7N69G4cOHcKcPAJBUlISoqKiEBUVBb1ej7i4OERFRUmmOBs5ciR++uknrF+/HleuXMGkSZNw7949DBgwoEheO1F+6WvWRNpHH0nalKtXQ372rJkqyj9vbz127lSjfHlpCJ43zxbffKPMZS8iIqKiJSQkJJjtj5HDhw/HsWPH8ODBA5QpUwZ16tTBhx9+iHbt2uW4/dGjR+Hv7w97e3tJ+9WrVyGTyVDtublTMxw7dgxdu3bN1t6rVy+sWLHC+Hj16tVYsmQJ7t+/j9q1a2Pu3Llo3rz5S7xC87t27RqqV69u7jLoZaWkwLF5c8hjYoxNutdeQ9LRo4BCUeKv87lzMnTr5ojEROmY+q++SsEHH3COtPwq6deZTIPX2TrwOpuXWQMwFT2+wSyH/OhROHbvLmlLmT0bmtGjS8V1jogwLJGsVktD8IoVyejVq2Qu8lHSlIbrTC+P19k68DqbV4kbA0xEOdO1aQPNc8ODbOfNg3D7tpkqKpiAAB1++kkNlUr6O/fIkXbYtcs0M7kQERHlBwMwUSmS+vnnEMuWNT4WkpNhN3FiiZ4bOKvWrXVYvz4ZCkVmvXq9gEGD7PHbbwzBRERUPBiAiUoR0cUFKTNnStpsfv0VzkePmqegQujUSYvVq5Mhk2WG4PR0AX362OOPPwq/oA0REVF+MQATlTLpfftC+9z0fFW//BKyy5dLTU9w9+5aLFuWImlLSxMQEuKA06cZgomIqGgxABOVNjKZYW7gLCsgKh88gFPTpnCqVw+248ZBsW8f8OyZGYt8sZCQdCxaJA3BarWAd991wLlz/NZERERFhz9liEohva8v0kaPztYui42Fau1aOPTujTLe3nDo2hXKJUsgu3ChRPYODxyowezZ0hCcmCjg7bcdcOkSvz0REVHR4E8YolIqbcIE6GrVyvV5IT0dimPHYDdjBpyaN4dTnTqwGz0ail27gISE4iv0BUaP1mDSpFRJ2+PHMrz1lgNiYvgtioiITI8/XYhKK3t7JB04gJR58/C0aVOIqryXF5bduQPlhg1w6NcPZXx84PDGG1AtWgTZuXOAXp/nvkVt8uQ0jB6dJmm7d0+Gbt0cEBsr5LIXERFR4TAAE5VmZctCM3w4rn3zDRJv3IB62zakDRkCnY9PnrsJOh0U4eGwnT0bTm3awKlWLdgNGwabn3+G8PhxMRWfpR4BmDUrFR98IA3BcXEyBAU54N49hmAiIjIdTrxJZCns7aHt0AHaDh0AALIbN6A4dMjwcewYhOTkXHeVPXgA5ebNUG7eDFEQoGvcGNr27aFt3x66hg0BedHPzCAIwJdfpiI5WcDmzUpje0yMHG+95YB9+9QoX77kjWMmIqLShz3ARBZK/+qr0AwejOQtW5AYE4OknTuRNnJknuOGAUAQRSjOnIHt/PlwbN8eTtWrw27QINhs3gzhwYMirVkmA5YtS0FQkHRp5EuX5Hj7bXs8fVqkpyciIivBAExkDWxtoWvTBqmff46kU6eQGBWF5K+/RnqXLhAdHfPcVfb4MZQ//wz7YcNQpkYNOLRpA9WcOZCfOgVotSYvVaEAVq1KRseO0hB87pwCPXo4QK02+SmJiMjKMAATWSGxalWk9++P5NBQQ+/wnj1IHTMGutdee+G+inPnYPvll3Ds3BllfHxg178/bDZsgHDnjsnqUyqBdeuS0aqVNGBHRCgQEuKA1NRcdiQiIsoHBmAia6dUQteyJdI++wxJx48j8dIlJC9bBk337hDLls1zV+HpUyh37oT96NEo4+sLx+bNYTtjBuTHjgEazUuVZWcH/PSTGv7+0hD8xx8K9Otnj/T0XHYkIiJ6AQZgIpIQ3d2R/v77SPnxRyRev46kX39F6scfQ9ugwQv3lV+4ANWSJXDs2hVlfHxg37s3lD/8AOH27ULV4ugIbN2qRr16Okn7gQM2GDzYjrNDEBFRoQgJCQm8rdqCXbt2DdWrVzd3GVTEius6Cw8fQvH774aZJQ4fhqwAU6bpatY0ziyhDQwEbG3zvW98vIAuXRxw+XL22ShefVWHwEAdAgO1aNZMB29vPQQLzcV8P1sHXmfrwOtsXgzAFo5vMOtgluus00F+7pxxqjX52bMQ8rmghmhvD23LltC2awdthw7Qv/rqC/e5d0/AG2844MaNvKdkc3HRGwNxYKAWr72mL45Z3IoF38/WgdfZOvA6mxcDsIXjG8w6lITrLDx+DMWRI5m9wwWYMk3n7Z3ZO9yiBWBvn+N2t28LePNNR8TF5X/0VpkyIvz9tcZQ3KiRriCdzyVKSbjOVPR4na0Dr7N5MQBbOL7BrEOJu856PWTnz8Pm8GEoDh6E/PRpCDrdi/cDIKpU0DZvbgzE+urVkXVMw5MnApYvV+LoUQXOnZNDqy3YeAelUkSjRhk9xDoEBGjxgnv9SowSd52pSPA6WwdeZ/NiALZwfINZhxJ/nZ8+heLoUUMgPnQIsgJMmaavWhXp7dsbhku0agU4ORmfU6uBs2flCA9XIDxcjshIBdTqggViQRBRp47eOIY4MFALN7eS+W2xxF9nyp1eDyQlQUhMNHw8e5b5eWIikOXzhKQkOLVpA21AAERPT1jsoHYrx/ezeTEAWzi+waxDqbrOogjZpUtQHDoEm0OHIA8Ph5DPOc1EGxvomjZFeocO0DVuDNHZGWLZsobp2hwdka4VcP68HCdPGkLxqVNyxMcXfLKbknpjXam6zpZEpwOePYPw9Gm+Aqzx+Szb49kzCGLBf9zqXVyg8/eHNiAAOn9/6OrXL9ANpFRy8f1sXgzAFo5vMOtQqq9zUhIUYWFQHD4Mm4MHISvklGmiTAaxTBngv0Asli0LfZkyeApnxCU5I/phOVy8Ux43EsohAc54irJIgLPx86coCz1yvluupNxYV6qvs7mkp2eG0KwB9kUhNutzSUnmfhVGolIJXYMG0AUEQOvvD52/P0RXV3OXRYXA97N5MQBbOL7BrIPFXGdRhCw6GoqDB6E4fBiK48chpKUV2+kT4ZQtHOcUljV2ZeFWywmvNnJC7QAH1A50gsq1rGEJuyJkMdf5RUQRSE8HNBoIycnSgFrQXtiUFHO/miKn8/Iy9A7/F4r1vr6wmKlPLJjVvJ9LKAZgC8c3mHWw2OucnAzFiROZU61dv27uivKULreF1qksFBXKQihfFmKZMsbeaLFsWSDr4+eeE8uUMcx+kcdYC5NeZ53OsFpfWpphCEqWf6HRQNBoDM9nfJ71+fR0wy8mWbfL0pa1PWNf4/MZ53puu2z7WCDRwQGik5Ph2mf5QNbHTk54cu0aXKKjDVMLqtUFP4+TE7SNGxtCcdOm0DZujFJzp6cVsdjv26WEwtwFEBHlyt4e2g4doO3QAQAgu3HDEIbDwiDcv2/oDczoEUxONnOxgI0uFTYJqUDCfaAQWV1UKKSB+LmwXPnZM9g6OuYaLHMKoNmCZcbn+ZyVgwzE50JqrgE2p+fLloXo5AQo8vcj9861a3CoXh3QaiG7cAGKiAjIT5+GIiICstjYF+4vPHsGm6NHYXP0qKF2QYC+du3MccQBAYa5t0vCwHYiM2EPsIXjb5jWgdcZhlCX5U/kyAjHz39kbJP1ccaf0MniGMeGPx9QM36xyKUXVvKckxMgK/jNlIWV1/tZuHMH8shIYyiW//13vm8izUr/yivSm+saNuTNdUXp2TPIbt3K/Lh5E+obN+Do5GT4RSS3D5lM8lh87rHx/2VO22f5XHz+OSDbsV94rjy2zfFcOdSlq1/fMESnBGAPMBFZBqUS4iuvQHzllcLtn/VO/5zC8n+f4+lTqP9NhPrfRKQ/SoQ88SkcdQkoi6dQgL2qpiIqFIBKBdHWNu+AmtFbnsvzcHCwqJ5OsVIlaIOCoA0KMjSkpEB+7hzkERHGUCyLj3/hcWSPHkH2f/8Hm//7P8NxbWwMN9f5+xturgsIgOjmVpQvxbJoNJDFxhoDrvBc2M1p2XiVGco0t9Tp05HGAExEVILI5YCzs2FqtRdtCqBMlsdxcQJ+OSnHn8dScSk8CQ+ik7LcQmcIxzl9nvVxOTyBCsU39lUUBEClMvzioFQabuCzsYGoUmX++1+7+Ny/UCqN20Glgpjxb9bn89gux2Nmfd7Gplh7XEs1OzvoAgOhCww0/O8RRchiYiDPGDZx+jRkly69cAo2IT0dishIKCIjoVq+HIBhDu6MHmJtQICh5y6fwzgsjl4P4d49aajN+Pz2bQj//luoae6sTgn6ZdRK/ycTEZmOh4eI4B5aBPdQAHDG48flcOqUl3GBjvyuWKdCqjEQV1Q8QZPqT9DIOx51Kj+Bd4UnUD+9jwru7rmGTUkYzSlsZgmjkMtL1A8jMhFBgN7HB3ofH6SHhBjaEhKgOHPG0Et8+jTkZ87k6+Y62e3bUN6+DWzbBgAQHR2ha9zY2EOsbdIEcHYuwhdTzBISpME2a9iNjS3WGWkslViCfrHlGGALx7Gh1oHXuWRTq4EzZzJWrFMgMlKO5OSCr1hXvXoK2rWTo3lzLZo316FcOX77tkRF/n7WaiG7eNEQhjNurrt1q8CHEQUB+lq1JMMm9D4+JfcXq5QUyG7fzt6DmzFsoQjvAxAVCuirVoXe0xN6T0+Inp64I5fDrXJlw7R/er3h34yP5x//9yHkZ1u9/r+T/rdP1ufz2heQPM7XuZ57TnhBXenvvANtp05F9nUuCAZgC8dgZB14nUuX9HQgKkq6Yt3jxwXrGclYwrl5cy1atDAE4vLl+e3cEpjj/SzcvWscMiE/fRryc+cKd3NdhQrZb66zsyuCinOg1UL4919psL19OzPs3r9fpKfXu7sbAm7VqtB7eRnDrt7TE2KlStnmZub3bfNiALZwfINZB17n0k2vB65elRmHTJw8qUBcXMH/VOjrq/svDBsC8Suv8Nt7aVQi3s+pqYab6/7rIZZHRED26FGBDyMqFNDVry8JxWKlSoWrSRQhPHqUYw+ucOsWZHFxELTawh07H/TOzhCzhFq9p2dm0K1SpcCzaJSI62zFGIAtHN9g1oHX2fLExgo4dcoQiMPDFbh0qeAre/n66ow9xM2a6VCxIr/dlwYl8v0sipDduJF5c11ERL5ursuJ3sMD2qZNjUMn9K+9lnlz3fPThWWE3du3DTebFWJhkPwSbW0zhyl4eUmGLOg9PU0+3rlEXmcrwgBs4fgGsw68zpYvPl7A9u0PEBNTBcePK3DhggyiWLCxlrVq6YzDJZo318LFhd/+S6JS835++hSKs2cNoTgiAoozZyAkJRX4MKK9PfTe3hDu3s3XFG6FJcpkECtXzrkH19MTootLiZnvmYoeA7CF4xvMOvA6W4es1/nJEwEnT8px/LgCJ04ocP58wQNxzZoZPcSGf11d+eOgJCi172edLvPmuoyFOm7eLNYS9K+8km38bcZNZ3oPD8MsKCVEqb3OFoLToBERlULlyono0kWLLl0MYx4TEoCTJw1h+PhxBaKiXhyIr1yR48oVOdauNTyuXl3aQ+zuzkBMBSCXQ1+3LjR16wIDBwIAhPv3M8cRZ9xcpyn8fNeio6N0aELWsFu1KuDoaKpXQxaOAZiIyAI4OwNvvqnFm29mBuJTpxT/9RDL8fffcuj1eQfia9fkuHZNjh9+MDyuVk2H5s0zb6yrVImBmApGdHWFtmtXaLt2NTSkpkL+99/Sm+sePszc3sYG+ipVpD23WYKuWL58yZ1mjUoVBmAiIgvk7Ax07qxF586GQPz0qSEQG3qIDYtzvCgQR0fLER0tx7p1SgCAt7cOLVpkBuLKlRmIqYBsbaELCIAuIACa0aMNMzvcugXZgwfQu7vnOF0YUVFgACYisgJlywKdOmnRqZMhECcmAhERhh7ijECs0+UdiGNi5IiJkWP9ekMgfvXVzB7iFi208PBgIKYCEgSIXl7QeXmZuxKyMgzARERWqEwZoEMHLTp0MATiZ88MgfjECcONdX/++eJAfOOGHDduyLFxoyEQe3rqjWG4eXMtqlZlICaikokBmIiI4OQEtG+vRfv2WgBpSEoCTp829A5nBGKtNu9AfOuWDLduKREaagjEVatmrlTXooUWnp4MxERUMjAAExFRNo6OQNu2WrRtawjEarUhEGf0EJ89K0d6et6B+PZtGW7fVmLTJkMg9vDI7CHOCMS8n4mIzIEBmIiIXsjBAXj9dS1ef90QiJOTgchIOY4dM9xYd+bMiwNxXJwMmzcrsXlzZiA2LNusRcuWOnh56RmIiahYMAATEVGB2dsDrVvr0Lq1DlkDccY8xGfOyKHRvDgQb9mixJYthkBcqVLWMcQ6eHszEBNR0WAAJiKil/Z8IE5JAc6ckf83y4QhEKel5Z1m79yRYetWJbZuNQRid3c96tXTQaUC5HIRcrlhpVq5POtHTu25b5u9LXP7/GwrkxV+W4Uir22L4SIRkREDMBERmZydHdCypQ4tWxoCcWqqIRBn9BBHRsqRmpp3IL57V4a7d60nGWYEcZWqATw8BFSqpEflyiIqV9b/95H5uYODuaslKt0YgImIqMjZ2uK/RTR0mDQpDWlpwNmzmT3Ep0+/OBBbOp1OgE4HaDQKXLoEXLqU+4IQzs56VKokwsMjMxwbArMeHh6Gz+3sirF4olKGAZiIiIqdSgU0a6ZDs2Y6TJxoCMR//ik3rlQXEaFASop1B+K8JCTIkJAAXLyYe0guXz6z19jDwxCYM3qQPTz0cHcXoVIVX81EJQkDsAn17NkT4eHhaN26NdavX2/ucoiISg2VCggM1CEwUIePPwY0GuDvv+W4d0+AXp/ZOyr9yN6W07aGNml7Tm0v3hbQaoVsbRnbStsM7flrA0TR9GH/8WMZHj8Gzp/PPSRXrCjtQTb0KBuCcqVKhtBsY2Py0ojMjgHYhEaMGIF+/fph06ZN5i6FiKhUUyoBPz+ducsoNqKYGYbPnbsBlcoH//4ry/IhGD+/e1d44ZRz+fXwoQwPH8pw7lzOzwuCCFdXMc/xyG5uIhRME1TK8L+sCbVq1QrHjh0zdxlERFTKCIJhlgiFAihfXovq1fVo0ECf47Z6PfDggYA7d2SIi8sMxnfuSEPyi5ayzg9RFHDvnoB792T488+ct5HJRLi5ScPx873JLi6GG/yISgqzBuCvvvoKe/bsQXR0NJRKJZo0aYIZM2bA19fXpOc5ceIEvvnmG/z999+4e/culi9fjt69e2fbbvXq1Vi6dCnu37+PWrVqYd68eWjWrJlJayEiInoZMhng5ibCzU2HRo1y3kanA+7fF4y9x3FxMty5I+1NvndPMMnQC71ewJ07hkAeGZnzNgqFISRn3LQnHY8sokoVPV55hUtlU/ExawA+fvw4Bg4ciEaNGkEURcydOxfdu3dHREQEypUrl237iIgINGjQAKrnRu3fvHkTgiDA09Mzx/Oo1Wr4+vqiV69eGDZsWI7b7NixA5MnT8aiRYvQtGlTrF69GsHBwTh16hSqVKkCAAgMDMxx323btsHDw6MgL52IiKjIyOVApUoiKlXSwc8v523S04F796Q9yHFx0pD84IFppqHTagXExRmOnxsPDz38/LRo0kQHf3+dcQ5ooqJg1gC8Y8cOyePvv/8eVatWxalTp/DGG29InhNFERMnToSbmxs2btwIm/9G5cfGxqJr16546623MGvWrBzP07FjR3Ts2BGAYZxuTpYvX46QkBD069cPALBw4UIcPnwYa9euxYwZMwAA4eHhhX+xREREJYiNDVCliogqVXQAch5vrdFAMrQi4/OMoHznjoBHj0wTkuPiZIiLU+KXXwyPlUoR9evrjIG4SRMtPDxErg5IJlGixgAnJSVBr9fD2dk523OCIGDbtm3o0qULBgwYgB9//BH3799H165d4efnZwyphaHRaHDu3DmMHj1a0t62bVtEREQU+rhERESlmVIJeHmJ8PLKPSSnpsI4Hvn5YRYZnz95UvCQrNEIiIxUIDJSgRUrDG3u7vr/ArGhp7hBAx3nO6ZCKVEBePLkyahbty78/f1zfN7FxQW7d+9Gly5d0L9/f1y6dAl169bFypUrIX+J0fXx8fHQ6XSoWLGipL1ixYp48OBBvo8TFBSEf/75B8nJyfD19cWPP/6Y62shIiKyBLa2gLe3Ht7eQG4hWa02hGTpMIvMHuXoaBk0mhd37d69K8OePTLs2WP4K7BCIaJu3cxeYj8/LTw92UtML1ZiAvAnn3yCU6dO4ddff80zzLq7u2Pt2rVo06YN3NzcsGrVKihMNP+K8Nw7RhTFbG152bVrl0nqICIisiQODkD16npUrw7kFJLT0oCoKDkiIzM+FHmOF86g1Qr46y8F/vpLgVWrDG0VK+olwyYaNdJx6WjKpkQE4ClTpmDHjh3Ys2cPvLy88tw2Pj4eI0aMQPv27XH58mWMHTsWy5cvh0xW+DFIFSpUgFwuz9bb++jRo2y9wkRERGRaKpVh3uescz/fvSsYw/CZM3L89Vf+lst++FCG/ftl2L/f0Essl4vw9dUbh034++vg7a1nL7GVM3sAnjRpEnbs2IG9e/eiRo0aeW775MkTdO/eHe7u7ggNDcXdu3fRpUsXfPTRR1i6dGmBemuzUiqVaNCgAY4cOYLu3bsb248cOYJu3boV6phERERUeO7uIrp106JbNy0Aww15Fy5k7SWW4+bNFw9/1OkEnD8vx/nzcqxZY2grX14PPz+dcTxxw4Y6lClTlK+GShqzBuCPP/4YW7ZswcaNG+Hs7Iz79+8DABwcHODo6CjZVhRF9OjRA+XLl0doaChUKhW8vLywe/du/O9//8OcOXMwbdq0HM+TlJSEmJgYAIBer0dcXByioqJQrlw54xRnI0eOxNChQ9G4cWMEBARg7dq1uHfvHgYMGFCEXwEiIiLKD6USaNhQh4YNdRgyxND24IGAM2cyh038+accyckv7gx7/FiGAwdkOHDA0EssCCJq19b/1wuthZ+fDtWr6/ESf1ymEk5ISEgw28zTOc32ABh6hadMmZKt/ejRo/D394e9vb2k/erVq5DJZKhWrVqOxzt27Bi6du2arb1Xr15YkXFrKQwLYSxZsgT3799H7dq1MXfuXDRv3rwAr6jkuXbtGqobBl2RBeN1tg68ztaB17nwtFrg4kXZf7NHGILx9euFu0m+bFkRTZpojUMzGjfWIpfYUii8zuZl1gBMRY9vMOvA62wdeJ2tA6+zacXHZ+8lfvascEMma9bUSXqJa9bUF3qJZ15n8zL7GGAiIiKiolKhgohOnbTo1EkLIA06HXD5sgxnzshx+rThBrsrV/KXYq9cMWy7caMSAODkJKJxY22WWSd0KF+e/YqlAQMwERERWQ25HKhTR486dfTo1y8dAJCQAJw9q8Dp0/L/eosVSEx8cS/xs2cCjh61wdGjNsa2atWkq9f5+uphotlayYR4SYiIiMiqOTsD7dpp0a6dYcYJvR64dk1mHDYRGSnHpUsyiOKLQ3F0tBzR0XJs3mx47OAgomHDzNXr/Px0qFjR9L3EOp1hDHR6esa/AtLTMx5LP8/YLqfnMj7X6fI6RuZzWbfLPK4g2U6rNXw+dKgG77yTbvLXXhgMwERERERZyGRAzZp61Kypx/vvGwJbYiLw11+ZwyYiI+V4/PjF00So1QKOH1fg+PHMyOXlpYOPjzfKlbOThMSM8Ph8kDSEzLy2Q77Cubl17Voywi/AAExERET0QmXKAK1b69C6tWGxDlEEYmJkxmETp08rcOGCDHr9i4PozZty3LxZvqhLLnHS00tOSGcAJiIiIiogQQB8fPTw8dGjVy9Dz2ZSkqGX+MyZzPHEDx9yMuEMWq25K8jEAExERERkAo6OQMuWOrRsmdlLfOuWgMjIzEB8/rwcWm3R9ITa2IiwsQEUitw/z+s5w7+i8XMbm6yfSx9LtzM8fv7zrNva2ABVquiL5HUXBgMwERERUREQBMDLS4SXVzqCgw29xCkpwLlzckREPIS7u2u28Ph86JTLswfLnIKmXG44H+UPAzARERFRMbGzAwIDdXjllXhUr25944BLCg5MISIiIiKrwgBMRERERFaFAZiIiIiIrAoDMBERERFZFQZgIiIiIrIqQkJCgukXpCYiIiIiKqHYA0xEREREVoUBmIiIiIisCgMwEREREVkVBmAiIiIisioMwERERERkVRiALdTq1atRr149uLq6onXr1jh58qS5SyIT+uqrr/D666+jSpUq8PHxwXvvvYeLFy+auywqYosWLYKzszMmTJhg7lKoCNy7dw/Dhg2Dj48PXF1dERAQgOPHj5u7LDIhnU6HOXPmGH8+16tXD3PmzIFWqzV3aVaHAdgC7dixA5MnT8b48eMRFhYGf39/BAcHIzY21tylkYkcP34cAwcOxIEDB7B7924oFAp0794dT548MXdpVEQiIyOxbt061KlTx9ylUBFISEhAp06dIIoitm7dioiICCxYsAAVK1Y0d2lkQl9//TVWr16NL774AqdPn8b8+fOxatUqfPXVV+YuzepwHmAL1K5dO9SpUwdLly41tjVq1AhBQUGYMWOGGSujopKUlISqVasiNDQUb7zxhrnLIRN7+vQpWrdujSVLlmDBggXw9fXFwoULzV0WmdCsWbNw4sQJHDhwwNylUBF67733UK5cOXz33XfGtmHDhuHJkyfYsmWLGSuzPuwBtjAajQbnzp1D27ZtJe1t27ZFRESEmaqiopaUlAS9Xg9nZ2dzl0JFYMyYMQgKCkLr1q3NXQoVkX379qFx48YYMGAAqlWrhhYtWmDlypUQRfZRWZKmTZvi+PHjuHr1KgDg8uXLOHbsGDp06GDmyqyPwtwFkGnFx8dDp9Nl+7NZxYoV8eDBAzNVRUVt8uTJqFu3Lvz9/c1dCpnYunXrEBMTg++//97cpVARunnzJtasWYMRI0ZgzJgxOH/+PCZNmgQAGDJkiJmrI1MZM2YMkpKSEBAQALlcDq1Wi48//hiDBg0yd2lWhwHYQgmCIHksimK2NrIMn3zyCU6dOoVff/0Vcrnc3OWQCV27dg2zZs3C/v37oVQqzV0OFSG9Xo+GDRsah6nVr18fMTExWL16NQOwBdmxYwc2b96M1atXo1atWjh//jwmT56MqlWrom/fvuYuz6owAFuYChUqQC6XZ+vtffToEW+msEBTpkzBjh07sGfPHnh5eZm7HDKx06dPIz4+HoGBgcY2nU6HkydPYu3atbhz5w5UKpUZKyRTcXV1Rc2aNSVtNWrUQFxcnJkqoqIwffp0jBo1Cu+88w4AoE6dOoiNjcXixYsZgIsZA7CFUSqVaNCgAY4cOYLu3bsb248cOYJu3bqZrzAyuUmTJmHHjh3Yu3cvatSoYe5yqAh06dIFDRs2lLSNHDkSPj4+GDduHHuFLUjTpk0RHR0taYuOjkaVKlXMVBEVheTk5Gx/qZPL5dDr9WaqyHoxAFugkSNHYujQoWjcuDECAgKwdu1a3Lt3DwMGDDB3aWQiH3/8MbZs2YKNGzfC2dkZ9+/fBwA4ODjA0dHRzNWRqTg7O2e7sdHe3h7lypWDr6+veYqiIjFixAh07NgRX375Jd5++21ERUVh5cqVmDZtmrlLIxPq3Lkzvv76a3h6eqJWrVqIiorC8uXL0bNnT3OXZnU4DZqFWr16NZYsWYL79++jdu3amDt3Lpo3b27usshEcpvtYdKkSZgyZUrxFkPFqkuXLpwGzUIdOHAAs2bNQnR0NDw8PDB48GAMHTqU929YkGfPnuHzzz/H3r178ejRI7i6uuKdd97BxIkTYWtra+7yrAoDMBERERFZFc4DTERERERWhQGYiIiIiKwKAzARERERWRUGYCIiIiKyKgzARERERGRVGICJiIiIyKowABMRkUk5Oztj7Nix5i6DiChXDMBERKVMaGiocZW4nD5+/fVXc5dIRFSicSlkIqJSavLkyXj11VeztderV88M1RARlR4MwEREpVS7du3g5+dn7jKIiEodDoEgIrJQGWNxd+zYgYCAALi6uqJZs2Y4cOBAtm1jY2MxePBgeHt7w9XVFS1atMCmTZuybSeKIlatWoUWLVrAzc0N3t7e6N69O06ePJlt24MHD6Jly5ZwdXVFo0aN8PPPP0ue12q1WLhwIRo3bmw8VseOHbFr1y7TfRGIiHLAHmAiolIqMTER8fHx2dorVKhg/DwiIgK//PILhg4dCkdHR6xbtw69e/fGrl270Lx5cwBAfHw8OnfujCdPnmDIkCFwc3PDjh07MHz4cCQkJGD48OHG43300UdYv3492rRpg5CQEIiiiNOnTyM8PBzNmjUzbhcZGYl9+/ZhwIAB6NOnD9avX48hQ4agbt26qFmzJgBg/vz5WLRoEfr06YPGjRtDrVYjKioKZ86cQVBQUFF92YiIICQkJIjmLoKIiPIvNDQUI0eOzPX5uLg4ODo6wtnZGQBw4MABBAQEAAAeP36MRo0aoUaNGvjtt98AAJ9++imWLVuGXbt2oXXr1gAAjUaDN954A5cvX8bFixdRtmxZHDt2DF27dkW/fv2wZMkSyTlFUYQgCAAMPc8KhQInTpwwht0HDx7gtddew9ChQzF79mwAQMuWLVGpUiVs2bLFdF8cIqJ8YA8wEVEp9cUXXxgDZlZ2dnbGzxs2bGgMvwBQvnx5BAcHY9WqVUhISICzszMOHDiAevXqGcMvACiVSgwfPhyDBg3C8ePH0aVLF+zevRuAITA/LyP8ZmjZsqWkNhcXF1SvXh03b940tjk5OeHSpUuIjo5GtWrVCv4FICIqJAZgIqJSqlGjRi+8Cc7HxyfXttjYWDg7O+P27dvo2rVrtu0yAuzt27cBADdu3EDFihVRsWLFF9ZWpUqVbG3Ozs548uSJ8fGUKVPw/vvvo0mTJqhVqxbatm2Ld999F40aNXrh8YmIXgZvgiMismDP98wChuEK+fH8dlmHObyIXC5/4TFbtmyJv//+GytWrEC9evWwefNmtGvXDl999VW+zkFEVFgMwEREFiw6OjpbW0xMDIDMXtqqVavi6tWr2ba7du2a8XkA8Pb2xoMHD/Dw4UOT1efs7IxevXph5cqVuHDhApo1a4YvvvgCOp3OZOcgInoeAzARkQX766+/cPr0aePjx48fY9u2bfDz8zPeJNepUydERUUhLCzMuF16ejq+++472Nvbo0WLFgCAbt26AQDmzp2b7Tz57VXO6vHjx5LHdnZ2qFmzJtLS0pCcnFzg4xER5RfHABMRlVKHDx829uZm1aBBA+P4XV9fX7z33nsYMmSIcRq0Z8+eYfr06cbtM+YK7tWrF4YOHQpXV1f88ssviIyMxNy5c1G2bFkAhiELISEh+OGHH3Dz5k107NgRgGHKszp16mD8+PEFqt/f3x/NmjVDo0aNUL58efzzzz9Yv349OnXqBCcnp8J+WYiIXogBmIiolJo/f36O7bNnzzYG4ICAALRs2RLz58/HzZs34ePjg40bN6Jly5bG7StUqIADBw5g5syZ+OGHH5CcnIxq1aphxYoV6NWrl+TYy5YtQ506dbBhwwbMmDEDjo6OqF+/vnFO4YIYPnw49u/fj7CwMKSmpqJy5coYM2YMxowZU+BjEREVBOcBJiKyUM7OzhgwYAAWL15s7lKIiEoUjgEmIiIiIqvCAExEREREVoUBmIiIiIisCm+CIyKyUAkJCeYugYioRGIPMBERERFZFQZgIiIiIrIqDMBEREREZFUYgImIiIjIqjAAExEREZFVYQAmIiIiIqvy//SZU0wR6vgcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = sbs_transf.plot_losses()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[410, 440],\n",
       "        [300, 331]])"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "StepByStep.loader_apply(test_loader, sbs_transf.correct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[    2,   300, 12427,     7,  7101,  1423,   422],\n",
       "        [    2,  6659,   811, 11238,    15,     2,   588]], device='cuda:0')"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = ['The white rabbit and Alice ran away', 'The lion met Dorothy on the road']\n",
    "inputs = glove_tokenizer(sentences, add_special_tokens=False, return_tensors='pt')['input_ids']\n",
    "inputs = inputs.to(sbs_transf.device)\n",
    "inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.9888],\n",
       "        [0.0101]], device='cuda:0', grad_fn=<SigmoidBackward>)"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sbs_transf.model.eval()\n",
    "out = sbs_transf.model(inputs)\n",
    "# our model outputs logits, so we turn them into probs\n",
    "torch.sigmoid(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[2.6334e-01, 6.9912e-02, 1.6958e-01, 1.6574e-01, 1.1365e-01,\n",
       "          1.3449e-01, 6.6508e-02, 1.6772e-02],\n",
       "         [2.7878e-05, 2.5806e-03, 2.9353e-03, 1.3467e-01, 1.7490e-03,\n",
       "          8.5641e-01, 7.3843e-04, 8.8371e-04]],\n",
       "\n",
       "        [[6.8102e-02, 1.8080e-02, 1.0238e-01, 6.1889e-02, 6.2652e-01,\n",
       "          1.0388e-02, 1.6588e-02, 9.6055e-02],\n",
       "         [2.2783e-04, 2.1089e-02, 3.4972e-01, 2.3252e-02, 5.2879e-01,\n",
       "          3.5840e-02, 2.5432e-02, 1.5650e-02]]], device='cuda:0')"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alphas = sbs_transf.model.encoder.layers[0].self_attn_heads.alphas\n",
    "alphas[:, :, 0, :].squeeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-107-a1f395f91ca0>:23: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "  ax.set_yticklabels(['', f'Head #{i}', ''])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABBoAAACUCAYAAAA9KUrkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABSdklEQVR4nO3ddVxU2f8/8JeUa60ISAtIKAKKgYRiLe7agSI2ayKY2IhrB7YYGGthf1TWVgwwFsQELFCwsFBBBQsT+P3Bj/t1mAFB74DMvp6PB4+H3jl35rzn3DjzvueeWyotLS0LREREREREREQiUCruChARERERERGR4mCigYiIiIiIiIhEw0QDEREREREREYmGiQYiIiIiIiIiEg0TDUREREREREQkGiYaiIiIiIiIiEg0TDSIoE2bNlBXV4e6ujouXrxYpJ/t7e0tfPa+ffuK9LO/Fh4eDnV1dbx48aLY6iCWgsRSEuPdunUrDAwM8i3j7+8PJyenHy5TksTExEBdXR33798v7qoAgNS+XNz7NhGVDOyLlMxzcw5vb2907dpV6t9UdLp27Qpvb+9i+eyaNWti2bJlxfLZ/yUl+RghTy9evIC6ujrCw8NFfV8mGkTSs2dPxMfHo3bt2sKy8PBwdO3aFaamptDV1UX9+vUxduxY4QfNtzb29PR0TJ8+HXXq1IGOjg5MTU3RokULBAcHC2XmzJmD+Ph4ucYmS5s2bTB27Ngi/9yfhYODA+Lj46GhoQGgYD/iFcWwYcNw6NAh4f/sEMlXfHw8WrZsWdzVIKISgH0RxTBnzhysXr26uKvx01GECx3/pf7iz0BRjxElBRMNIilbtix0dHSgqqoKANiwYQM6dOiASpUqISgoCOfPn8eyZcuQlZWFBQsWFOg9R44cid27d8Pf3x8XLlzA7t274e7ujtTUVKFMxYoVoaOjI5eYKG9qamrQ0dFBqVKlirsqAIBPnz4V2WeVL19eSLCQ/Ono6KB06dLFXY0fVpTbKNF/FfsiiqFixYpQV1cv7mpQIWRmZiIjI6O4q0EKrqT1pZhokIPHjx9j/PjxGDBgAFatWoXGjRvD2NgYjo6OWLBgAWbMmFGg9wkJCcGoUaPQsmVLGBsbo3bt2ujfvz8GDhwo5wjy5+3tjTNnzmDNmjXCUMkHDx4AAK5fvw4XFxfo6emhadOmuHz5ssS658+fR+vWraGnp4caNWpg1KhReP36tdzrfPz4cRgaGuLLly8AgDt37kBdXR2jRo0SysyYMQMdO3YU/p9fLF9fAQoPD8eQIUPw7t074fvw9/cHkH1AmDJlCqysrKCvr49mzZohLCzsh+Np06YNRo0ahb/++gtmZmZo0aIFli9fjgYNGkBfXx81atTAsGHDkJaWJrVuSEgI6tWrBx0dHbRt2xaJiYlSZTZt2gQbGxvo6uqiR48eEle6vr6i4O/vj+3bt+Po0aNC7GIPu/paaGgoWrVqBWNjY5iYmKBTp07CVbT79+8Lw3Y7duwIPT09ODg44OTJk1LvUb9+fejo6KBVq1a4ffu23Opb2BhkyT0U+cmTJxg4cCCqVq0KPT09ODs7499//xVeDwkJQZMmTaCjo4NatWphxowZxXJi+p5tNOdKz+nTp+Hk5AR9ff08t1F5ya99+vbtK3XMUFdXx6VLl4RlVlZW2LlzJwAgOjoarq6uMDU1RZUqVdCyZUtcuHBBKDtkyBCp0UCZmZmwsbHB8uXL5Rkm/QewL/Lz9UUKKvdIwY8fP8LX1xcWFhbQ0dFB8+bNcfbsWeH1nD7J6dOn8427KOWcAyZOnAgTExOYmZlh5cqV+PjxI8aMGQMjIyPY2Njgf//7n7BOUlIS+vXrB2NjYxgbG8Pd3R137twBkH1+mDt3Lm7cuCG099atW7+7funp6fD29oaBgQEsLCywcOFCidfT0tLg5eUFY2Nj6OrqokOHDrhx44bwes756tixY3ByckLlypURHx+f73r59RcB4MOHD/Dx8UGVKlVgZWWFpUuXCq8V9fkiv23uZ9zeZFGEY4SsvtSZM2fg4uICHR0dWFhYYMKECRL9vIL0M6Ojo4W+YqNGjST6MWJiokEO9u7di0+fPmHkyJEyXy9ollpHRwehoaF49eqViLX7cXPmzIG9vb0wRDM+Ph6GhoYAgGnTpmHKlCk4ffo0NDQ04OnpiaysLABAbGwsOnXqhFatWiEiIgKbN2/GtWvXMHToULnX2cnJCR8+fEBMTAwAICIiApqamhI/iiMiIuDs7Cz8P79Yvubg4AB/f3+ULVtW+D6GDRsGIPvEkHOQi4yMRPfu3dGtWzdcu3bth2PauXMnsrKyEBISglWrVkFJSQn+/v44e/Ys1qxZg6ioKIwbN05inY8fP2Lu3LkIDAzEsWPHkJGRgZ49e0rE9eDBA+zYsQNbt27F3r17cffuXQwZMkRmHYYNGwZXV1c0bdpUiN3BweGHY8vLu3fv4OXlhRMnTuDgwYP49ddf0a1bN4kD7MyZMzFo0CBERESgTp066NevH96+fQsAePToEXr27ImmTZsiPDwcnp6emDJlitzq+70x5LdumzZt8ODBA2zZsgWRkZESbRwWFgZPT08MHDgQ586dw/Lly7Fv3z5Mnz5dniHl6Xu30UWLFmH58uU4duwYXr16JfHjXt7yax9nZ2epY8bXx5E7d+4gKSlJOI68efMGXbt2RUhICMLCwlCzZk106dJFSNz9+eefCA0NxdOnT4X3PHnyJJ49e4Zu3boVWcykmNgX+fn6It9r8uTJ2LNnD5YvX45///0XVlZWcHNzkzh2AAXvtxSVXbt2oXz58ggLC4OPjw8mTJiAnj17wszMDKdOnUK3bt0wfPhwPHnyBOnp6WjXrh1Kly6NQ4cO4fjx49DR0UGHDh2Qnp6OTp06YejQobCwsBDau1OnTt9dt0mTJuHUqVPYtGkT9u3bh6tXryIyMlJ43dvbG1FRUdi2bRvCwsJQpkwZuLm54f3790KZDx8+YMGCBVi8eDHOnz+PKlWq5Ltefv1FAFixYgWsrKxw+vRpjBgxApMnTxaS00V9vijINvezbW+5Kcox4uu+lL+/P7p06YJatWrh33//xbJly/DPP/9g2rRpQvlv9TPfvXsHd3d3mJiY4OTJk5g6dSomTZokl7oz0SAHd+/exa+//go9Pb0fep+AgABERUXBzMwMjRs3xtixY6WuzhaHihUrQlVVVRiiqaOjAyWl7E1p4sSJaNy4MapVq4Zx48YhISEBSUlJAIClS5fC1dUVw4YNg5mZGezs7LBw4ULs378fKSkpcq1z+fLlYWtrK/wgiIiIwMCBA/Hw4UM8ffoU6enpiI6Olkg05BfL19TU1PDrr7+iVKlSwvdRvnx53Lt3D8HBwdiwYQMaNmwIExMTeHp64vfff0dQUNAPx2RkZIRZs2ahWrVqqF69OgYPHowmTZrA2NgYzs7OmD59Ovbu3YvMzExhnS9fvmDOnDlwdHSEra0tVq9ejRs3buD06dNCmffv32PVqlWwtbWFo6MjFi9ejCNHjghXFXJ/r7/88gtKly4txK6mpvbDseWlQ4cO6NChA8zMzGBjY4PAwEDcv38fUVFRQpnBgwejVatWMDMzw+TJk5GamiokdtavXw9DQ0PMmzcP1apVg6urK/r27Su3+n5vDHkJDg5GcnIytm3bhoYNG6Jq1apo3749GjduDABYsGABhg0bhl69eqFq1apo3Lgxpk6dig0bNhTLyf97t9EFCxagXr16sLGxwbBhwxAeHi5RRp7yax9nZ2fcunVLOGbExMRg6NChEscVU1NT6OvrAwCaNGmCbt26oXr16qhWrRrmzZuHX375BaGhoQAAe3t7VKtWDdu3bxc+f8uWLWjVqhW0tLSKJF5SXOyL/Hx9ke/x7t07rF+/HlOnTkWLFi1QvXp1LF68GJUrV8batWslyha031JULC0tMWHCBJiZmWHo0KHQ1NSEiooKvL29YWpqivHjxyMrKwsXLlzAP//8g6ysLKxYsQI2NjaoVq0aAgIC8O7dOxw9ehRlypRBuXLloKKiIrR3mTJlvqteb9++xebNmzFt2jS4uLjAysoKgYGBwu2wd+7cQUhICAICAtCwYUNYW1tj9erVePPmDXbt2iW8T0ZGBubNmwdHR0eYm5sjOTk53/Xy6i/m+O233+Dp6QlTU1MMGjQIpqamQv+sKM8XBd3mfrbtLTdFOUZ83ZfKScAtXLgQ1atXR8uWLTFlyhSsWbMG6enpAL7dz9y1axc+ffqEwMBAWFlZwcXFBaNHj5ZL3ZlokAOxOvQNGzbE5cuXsX//fri6uuL27dtwdXWFj4+PKO8vD9bW1sK/dXV1AUDYKa9cuYKdO3fCwMBA+MuZ5O7evXtyr5uzszMiIiIAAGfOnMHvv/+OunXrIiIiAufPn4eqqirq1atXoFgK4sqVK8jKyoKjo6NEzMeOHRMl3q8n+wKA06dPo2PHjrCysoKhoSF69+6NT58+4dmzZ0IZJSUliRiNjIygp6eHmzdvCsv09fVRpUoV4f92dnZQUlIqlom+crt37x4GDBiA2rVro0qVKqhWrRoyMzPx6NEjoczX7ZbTwc5pt/j4eNjZ2UnMrWFvb19Etc9WkBjycvXqVVhbW0NTU1Pm61euXMHChQsltreBAwfi3bt3EttBUfmebbR06dKwsLAQ/q+rq4vPnz8X2dXU/NqnevXq0NbWFo4ZVatWRadOnXD+/Hl8/vxZalRUSkoKfHx8UK9ePRgZGcHQ0BApKSkSbe3h4SEM/01NTcXhw4fRu3fvIomVFBv7Itl+tr5IYd27dw+fP3+Go6OjsExZWRn29vYS527gx/stYvu6PqVKlULlypUllqmqqkJdXR0pKSm4cuUK7t+/D0NDQ6FdjIyMkJaWJnq73Lt3D58+fZI4/5cvX16oW3x8PJSUlCRer1ixIqysrCS+cxUVFdSsWVP4f0HXy8vX3w2Q3YZft19RnS8Kus39bNtbYZSkY8TXfan4+HjUr19fSJgA2aO2P336hLt37wL4dj8zPj4e1tbWEkkuefWFVeTyrv9xZmZmeP36NZ48efLDVxJUVVXRoEEDNGjQACNHjsT8+fMxa9YsjBw5EsbGxiLVWDw5E1ABEH7M5XR2MjMz4eHhgcGDB0ut96PfU0E4Oztj7dq1uHnzJt6+fYvatWsLQ6G1tLRgb28vUf/8YimIzMxMlCpVCidOnJB4LwD45ZdffjAaoFy5csK/Hzx4gK5du8LDwwN+fn7Q0NDAlStX0L9//xI3cUx+unXrBj09PQQEBEBPTw8qKipwcHCQiDG/dvsZhvQVJIa8fKv+mZmZGD9+vMRcIzmK4wr592yjKiqSp6WcNiyqEQ3fap+GDRsKx4xGjRrB2NgYGhoaiI6OxpkzZzB16lThvby9vZGcnIzZs2fDyMgIpUuXRvv27SXi7datG6ZOnYqzZ8/i6tWr0NTUxG+//VYksZJiY18k28/WFymsnHrLmnw697If7beILXffp1SpUjKP8ZmZmcjMzETNmjWxfv16qfepVKmSqPX61neS3+tff+elS5eGsrJyodfLi6zv6+v3LKrzRUG3uZ9teyuMknSM+LovlZWVlee2lLP8W/2YomwjjmiQgw4dOkBNTQ2LFy+W+bqsCfoKqnr16gCyhzUVJzU1tULPrmtra4sbN27A1NRU6u97h78VhpOTEz5+/IglS5bA0dERysrKQqIh95XIwpL1fdSqVQtZWVl49uyZVLw5Q6vFEhMTg0+fPsHf3x/29vYwNzfHkydPpMplZmYiOjpa+P/Dhw/x5MkTYbsCsidj+vqKa1RUFDIzMyXKfO17toXv8fLlS8THx2PUqFFo2rQpqlevjjdv3ggTfBaEpaUloqKiJA6yRfm8+R+NwdbWFrGxsXk+hs7W1hYJCQky97HcnbuiVtBttDgVpH1kHTMaNmyIoKAgifkZAODcuXPw9PREixYtUKNGDZQvX15qZEmlSpXQrl07bNmyBVu2bEGPHj0kOq5E34t9EdmKuy9SWKamplBTU5OY/DEjIwMXLlzI87xcEtna2uLu3bvQ0NCQapecRINY/Q1TU1OoqqpKnP/fvXuHuLg4ANl9hczMTInJe1+/fo24uLh8v/OCrPcjMRTV+UKRtjlFO0ZYWlri4sWLEhdfzp49CzU1NVStWrVA/RhLS0vExcVJHL/l1RdmokEODA0NMXv2bKxZswZeXl4IDw/HgwcPcOHCBYwdOxaTJ0+WKB8XF4erV69K/GVmZqJNmzbYsGEDLl++jPv37+PYsWOYMWMGLCwsin1HNzIyQlRUFO7fv48XL14U6GrjiBEjEB0djZEjR+LKlSu4e/cujhw5UmTDL3Pmadi5cycaNWoEIHuo0OPHj3Hp0qUfSjQYGRnhw4cPOHnyJF68eIH09HSYm5vD3d0dgwcPxr59+5CYmIiYmBgsW7YM+/fvFyssANlXrjIzM7FixQokJiYiODgYq1atkiqnoqKCCRMm4MKFC7h69Sq8vb1haWmJpk2bCmXKlCkDb29vXL16FRcuXMCoUaPQokULmJmZ5Rn7jRs3cOvWLbx48QKfP38WNbYc6urq0NTUxKZNm3D37l1ERERg1KhRhfoB3bdvXzx48AC+vr64desW9u3bhw0bNsilvrL8aAxubm7Q0tJCz549ERkZicTERBw+fFh46sS4ceMQHByMWbNmIS4uDgkJCdi3b5/UMac4FHQbLU4FaR9nZ2fcvXtXmLMhZ9nOnTulkohmZmbYuXMnbt68iejoaPTr10/mHCYeHh7YtWsXrl+/jp49e8o/UPpPYF9EtuLuixRWuXLl0K9fP0ybNg3Hjh0TfkSkpKRgwIABxV090XTp0gXa2tro0aMHIiIikJiYiDNnzmDixInCHFFGRkZ4+PAhLl++jBcvXuDjx4/f9Vnly5dH7969MXXqVJw8eRI3btzA0KFDhe3HzMwMrVu3xsiRIxEZGYnY2Fh4enqiQoUK6NKlS57vW5D1ZPUXC6MozheKtM0p2jGif//+ePr0KUaPHo34+HgcPXoU06ZNw8CBA1G2bNkC9WPc3NygoqKCoUOH4saNGzh58qTUU1fEwkSDnAwYMAB79+7Fixcv4OHhgfr16wtDcMaMGSNRtl27dmjcuLHEX3p6OlxcXLBjxw506tQJ9vb2GD16NJycnLBnz55iv+I1bNgwqKmpwdHREWZmZgW6v9zGxgaHDx/GgwcP0LZtW2EyuMqVKxdBjbM1atQIGRkZwg+EX375BXZ2dihdurTE3AWF5eDggH79+qF///4wMzPDkiVLAACBgYHo2bMnJk+ejPr166Nr1644c+YMjIyMRIknh42NDebMmYMVK1bA0dERmzZtkvnostKlS2P06NHw8vJC8+bNkZmZiS1btkgMwzIyMkLnzp3RvXt3tG/fHsbGxggMDMzzs//8809Uq1YNzZo1g5mZGc6dOydqbDmUlJSwfv16xMbGwsnJCWPHjsXEiRNRunTpAr9HlSpVsHnzZoSFhcHZ2RkrVqwo0qdO/GgM5cqVw6FDh6Cnp4du3brByckJ/v7+Qvu5uLhg586diIiIgIuLC1xcXLB48WJhluXiVNBttDgVpH2qV68uPFIq53aU3MeVHMuXL8e7d+/QtGlT9OvXD7169ZK57zdq1Aj6+vpwdnZG1apV5Rsk/aewLyLtZ+iLFNa0adPQsWNHDBkyBI0aNUJsbCyCg4OFe8sVQdmyZXH48GGYmJigT58+sLe3h7e3N9LS0oQnpLRv3x6///67MNFdcHDwd3/ejBkz4OzsjF69eqFdu3aoUaMGGjRoILy+YsUK1K1bF927d4eLiwvev3+P4ODgb17R/tZ6efUXC6qozheKss0p2jFCX18fu3btwtWrV9GoUSMMHToUnTt3FhLHBenHlC9fHjt27MCdO3fQpEkT/PXXXxK3fYqpVFpaWsm4meYn1qZNG1hZWWH+/PnFVgd1dXVs3LgRHTp0KLY6EBFR4b1//x41atTAvHnz4O7uXtzVoRKKfREixcfzBZUkHNEgkqCgIBgYGEjc/14URo4cCQMDgyL9TCIi+nGZmZl4+vQp/P39UaZMGZmTeBIVBvsiRIqJ5wsqiTiiQQRJSUn48OEDAMDAwKBQQ7l/VEpKCt68eQMA0NHRkZiZlIiIfl7379+Hra0tDAwMsGzZMj5tgn4I+yJEiovnCyqJmGggIiIiIiIiItHw1gkiIiIiIiIiEg0TDUREREREREQkGiYaiIiIiIiIiEg0TDQQERERERERkWhUirsCJUXnzp2LuwqiKerHXhUFfX394q6C6LKyFG+e1pxZyRWNqqpqcVdBdKmpqcVdBdEp2vb3/Pnz4q4CFbGqVasWdxVEo2j7IwBoa2sXdxVE9+7du+KuguiWLFlS3FWQi379+hV3FUSnpKR418S1tLSKuwqiuXnzZr6vK17rEREREREREVGxYaKBiIiIiIiIiETDRAMRERERERERiYaJBiIiIiIiIiISDRMNRERERERERCQaJhqIiIiIiIiISDRMNBARERERERGRaJhoICIiIiIiIiLRMNFARERERERERKJhooGIiIiIiIiIRMNEAxERERERERGJhokGIiIiIiIiIhINEw1EREREREREJBomGoiIiIiIiIhINEw0EBEREREREZFomGggIiIiIiIiItEw0UBEREREREREomGigYiIiIiIiIhEw0SDnLRv3x6bN2/G4cOHsWLFCtjY2ORZ1tbWFtOnT8eOHTtw8OBB/P3332jZsqVUORUVFfz555/C+27btg0dO3aUYxSS+vXrh5iYGCQlJeHEiRNwdHTMt3yNGjVw4MABPH78GNevX8fYsWMlXl++fDlevnwp9ffw4UN5hiGha9euOHLkCKKiorBjxw7UrVs3z7JqamqYOXMmdu/ejZiYGGzYsEGqzMyZM3H9+nWpvwsXLsgzDAndunXD0aNHER0djZ07d34zplmzZmH37t24fPmyzJhmzZqF2NhYqb+LFy/KMwwJvXv3RkREBBISEnDo0CHY29vnWbZ06dJYuHAhjh49ijt37mDHjh1SZbS1tbF06VKcOHEC9+7dw8KFC+VZfZl69uyJU6dOIS4uDvv27YOdnV2eZdXU1DBv3jwcOnQIN2/exNatW6XK2NvbY9euXbh06RJiY2Nx7NgxDBgwQJ4hSFHEY4SXlxcSEhLw5s0bnD9/Hg0bNsy3vI2NDcLCwvD69WskJiZi4sSJEq937NgRhw8fRlJSEl6+fIkzZ86gbdu28gyB/sO4T5aMffLPP//E2bNncefOHYSEhHzzHLd48WIcP34ciYmJ2LVrl8xyjo6OCAkJwZ07dxAZGYnevXvLq/oy9e/fH1euXMHTp09x6tQpODk55VveysoKhw4dwpMnTxAXF4dx48ZJlVFVVYWfnx+uXLmCZ8+e4dq1axg0aJC8QpASFhaGMWPGYMCAAZgyZQri4+PzLZ+VlYWjR4/C19cXAwYMwIgRI7Bz507h9Zs3b2LmzJkYMmQIBg4cCF9fX4SEhMg7DAleXl64ffs23r17hwsXLsDZ2Tnf8jY2Njhx4gTevn2LBw8e4K+//pJ4XVdXF1u2bEFsbCw+ffqE9evXy7P6MinqMeL8+fO4d+8ejh49CgcHhzzLli5dGgEBAQgLC8ODBw/wzz//yCzn5OSEo0eP4t69ezh37hw8PDxEr3eJSDQYGBjI7Fj/rJo2bYrBgwdj+/bt8PLyQlxcHPz9/aGtrS2zvJWVFe7du4fp06dj4MCBOHDgAEaOHInffvtNotzEiRNRv359LF68GH379sX06dNx7969oggJrq6u8Pf3x+LFi9G0aVNcuHABO3fuhIGBgczyFSpUwO7du5GSkoLmzZtjwoQJGDp0KIYMGSKUmTBhAiwtLSX+7t27h3379hVJTC1btoSvry/WrFmDLl264PLly1i1ahV0dXVllldWVsanT5+wbds2/PvvvzLLzJkzB02aNJH4e/jwIY4ePSrPUARfx+Tm5obLly9j9erV0NPTk1leWVkZHz9+xPbt2/OMyd/fXyqmBw8eFFlM7dq1w9SpUxEYGIjWrVvj0qVL2LhxI/T19WWWV1JSwsePH7Fx40acOHFCZhk1NTWkpqZixYoViImJkWf1ZWrTpg0mTZqElStXol27doiOjsb69eu/2U6bN2/GqVOnZJZJT0/Hxo0b0b17d7Ro0QKBgYEYMWIEevbsKcdI/o8iHiO6dOmCRYsWYe7cuahfvz7Onj2LgwcPokqVKnnGFBISgmfPnsHJyQkjR47E6NGj4ePjI5Rp3LgxTp48iQ4dOqB+/foICQlBcHDwNztCVPxKWl+E+2TJ2Cfbt2+PadOmYdmyZWjRogUuXbqELVu2fPMcFxQUhLCwMJllqlSpgs2bN+PSpUto0aIFli9fjhkzZqB169byDEXg6uqKOXPmYOHChWjcuDEuXLiAXbt2wdDQUGb5ChUqYM+ePUhOTsZvv/0GX19fDBs2DEOHDpUot27dOri4uGDEiBGoX78++vTpg9jY2KIICefPn8e2bdvQtm1bTJ8+Hebm5li0aBFevHiR5zr/+9//cOLECbi7u2P27NkYNWoUqlevLrz+yy+/oHnz5vDz88Ps2bPRvn177NmzJ892FZu7uzsCAgIwZ84c1KtXD2fPnsWhQ4fy3Z+OHj2K5ORkODg4wMfHB2PGjMHIkSOFMqVLl8bz588xb948nD9/vkji+JqiHiNmzJiBpUuX4o8//sDFixexdevWPI/lOceIDRs25HuM2LJlCy5evIg//vgDy5Ytw8yZM9GmTRtR614qLS0tK78C3t7eePnypdRVwZiYGDRr1gxXrlyBsbGxqJXKzcDAAPPmzStQhzkzMxPGxsY4efIkzM3NUbduXSxbtkxiY8jKysKcOXOwceNGpKWloV69eliwYAFq1KiR5/t27ty5wPVdtmwZ7t27h0WLFgnLgoKCEB4ejnXr1hXoPSZNmgQlJSVMmzYNAFCvXj1MnjwZvXv3xuvXrwtcF1mio6MLvc7x48cRGxsrseNdvHgR+/fvx4wZM6TK9+3bF1OnTkX16tXx4cMHAMDo0aPRt2/fPEd3ODg4ICQkBC1btiz0CIC8Tsj52bZtGxISEjB16lRh2aFDh3D8+HEEBATku66fnx8sLCzQt2/ffMvVqVMHmzdvRq9evXD58uVC1S8rK99dU6bt27cjISEBU6ZMEZYdPnwYx44d+2ZMEydOhLm5eYFi2rJlC3r27FnomN68eVOo8gCwb98+3Lx5E+PHjxeWnT59GocPH8bcuXPzXXf69OmoXr06unbtmmeZDRs24OXLlxg9enSh65ZDVVW1UOX/+ecfxMfHw8/PT1gWFhaGkJAQLFiwIN91p0yZgmrVqhXoeLhixQp8+vRJYr8tqNTU1EKV/9mPEUDht78zZ87g2rVr8PLyEpbFxcVh9+7dUldxAGDQoEGYPXs2DAwMhJgmTJiAQYMGwcTEJM/PiYyMREREhMwrePl5/vx5ocqXZOyLZKtatWqB6/uz75Pfcz742ffJvC4o5efAgQO4ceOGxGdFRETg4MGDmDNnTr7rzpw5E9WrV0eXLl0klvv5+aF169YSV6fnz5+P6tWro3379oWq37t37wpVHgBCQ0MRGxuLESNGCMuioqKwb98+TJ8+Xap8v379MHXqVFSrVk1opzFjxqBfv36wsrICADRr1gxBQUGoU6cOXr58Weg6fW3JkiWFXmf69OkwNDREv379hGXjx4+HnZ2d1PcPAE+ePMFff/2FGTNmFKqPumzZMqioqMDb27vQdfy6bgURGRkpNSrk5s2b+Oeff6Su6gPZIwX8/f2hp6cntJOfnx+8vLxgZGQkVX7//v14/vx5oev1NSWlwl0T/9mPEQCgpaVVqPKHDh3CjRs3MGbMGGHZmTNncOjQIcyePTvfdWfNmgVLS0up37ETJ05E69atJc5JCxYsQPXq1dGuXbsC1+3mzZv5vl4iRjQURlxcHNTU1GBubo6UlBQ8evRIauj4kiVLEBgYiLlz5+LEiROoXLkyXF1dv+ukl5uKigqqVauGS5cuSSyPiooSDpYFUbZsWYn6NGzYEPHx8XBzc8P27dsRFBSEIUOG4JdffvnhOn+LqqoqbG1tcfLkSYnlJ0+ezHN4X04WMWenBYATJ05AX19f5sEIyB4ef+PGjSK5zUBFRQVWVlaIjIyUWB4ZGQlbW1vRPqdz5864detWoX+Qfw9VVVVYWVnhzJkzEssjIyNRu3Zt0T7Hzc2tSGOqWbOm1GiLf//9F/Xq1ZP758uDqqoqbGxsEB4eLrE8PDw839tcCsvKygp169YtkisKiniMUFVVRd26dXH8+HGJ5aGhoXkOAXZ0dERERIRETMePH4eBgUG+HZby5csXOrFDP7fi7otwn8z2s++TqqqqqFWrltQ57vTp0/neTvct9erVw+nTp6Xes1atWlBRUfnu9y0IVVVV1K5dW2pE4YkTJ/Ic7m1vby+17YWFhUFfX19IILZp0wYxMTEYMmQIYmNjERUVhblz56JcuXLyC+b/+/LlCxITE6USbtbW1rh9+7bMdWJiYlC5cmVcu3YNY8eOxejRo7FmzZp8Lxbev38ft27dkhj1IC+qqqqoV6+e1P50/PjxQu1Px44d++b+VFQU+RiRezTrjx4j7OzspI4Rp06dgq2trajHCFETDTdv3oS7uzsMDQ1hbm6O/v3749mzZ8Lr0dHRcHV1hampKapUqSIzA3737l20adMGOjo6sLOzw5EjRwpVhwsXLggHssjISNSqVQtlypQRXs/KysLKlSvh4+ODDh06wMrKCitXrsTbt28RHBz8A9Fnq1ixIpSVlaU2vtTUVGhoaBToPRwcHFCnTh0cOnRIWKanpwcbGxuYmZlh2rRpWL58OerXr/9dmbTC0tTUhIqKCpKTkyWWp6Sk5Jm919HRQUpKisSynPV1dHSkyleoUAEdOnTApk2bRKp1/ipVqgQVFRWpIW8vXrwodKYxL+XLl8cff/yR571RYlNXVy+ymMTYVwpCQ0MDKioqUtvS8+fPUbly5SKpg9hytr3cV6PFiikiIgJxcXHYu3cvtmzZgu3bt//we36LIh4jtLS0oKKiInEOA4Bnz57JrB+QXe/c30HO+nndkuXt7Q1DQ8MSNST/Z8e+CPfJHD/7PpnXOS6/dioIbW1tme+pqqpa4L7o98rZ9goTU171zXkNAExMTODo6AgbGxt4eHhg7NixcHFxwYoVK+QQhaQ3b94gMzMTFStWlFhesWJFvHr1SuY6ycnJeP78Oc6fP48BAwbA09MTT548QUBAADIzMyXKjhw5EgMGDMDUqVPh4uIideu0POS3P+W1b+jq6sosn/NacVPkY0TuPmNKSsoP9RkrV64ss28t9jFCtJTF06dP0bp1a/Tu3RszZszA58+fMWPGDHTv3h2hoaFQUlLCmzdv0LVrV8yZMwelSpUS7o2Pjo6GpqYmMjMz0atXL1SsWBHHjh3D+/fv4evri48fP37z83Oy7R8/fkRWVhaMjIzw6dMnZGRkwMjICE5OTtixYwfu37+PZ8+eSezEZcqUQYMGDXD+/PlvDhsvqNzD3kuVKlWgofDW1tbw8/NDYGCgxCQzOevPnj1bGMa2bNkyzJ07F+rq6khLSxOl3vkpbEyyystaDmTfJ6asrCxz4j55+t52Koi2bdtCWVkZ+/fvF+X9CkpW/cWKqV27dlBWVsaBAwdEeb/vJWY7FRd5bXvdunVD2bJlUadOHYwbNw6PHj3C3r17f/h9C4LHiMLFlHMfc8+ePfHgwQMRakvsi0jiPlky9kl59kW+fk9ZnyUvYreTkpISsrKyMHDgQGFUwNixY7Fnzx6ZP5qKwrfi+fLlCzw9PYUfrJ6envD19cW9e/dgZmYmlPXz88OHDx9w584d7Nq1C1paWkV2/78896fi8l85RohNHm1ZoERDaGio1IQTubNx69atg42NjTCnAACsXr0aJiYmiImJQb169dCkSROJdebNm4f9+/cjNDQUXbt2xalTp3Dz5k1cuXJFmLTD398frVq1+mYdw8PDkZWVhaZNm2Lt2rWwsLBAp06d4OvrC3t7e+EWg5wsVe4sUOXKlfHkyZOCfB35evXqFTIyMqSyQerq6t8cYmNjY4NZs2Zh48aNUj/kXr58iefPn0vcK5ezgevo6Mg10fDixQt8+fJFKhuopaWV54H92bNnUpnrnO88d+YQADw8PHDgwIEiSZgA2SNMvnz5InWlX0NDI9+JfQrDzc0Nx48f/+E5NQoqLS1NZkyampqix5RXBl9sL1++xJcvX6T2V01NzRJ7f3rOtievmB49egQASEhIgJaWFoYPHy73RIMiHiOeP3+OL1++SF3R0NbWllk/QPZVk5wYc19hcXV1RVBQEPr27YuDBw+KWHPFxb5IwXGfzPaz75M557jc33t+7VQQycnJUtuWlpYWPn/+LPfh3jnbXmFiSk5Ollke+L+RDc+ePcOTJ08k+lQJCQkAAENDQ7kmGipUqAAlJSWpvs/r16+lRjnkUFdXh7KyssT2qqOjA2VlZbx48UIi0ZDTVlWqVMHr16+xd+9euSca8tufcu8bOZ4+fSqzPCC9PxUHRT5GyNqff2SblzUiQlNTU/RjRIFunWjQoAHCw8Ml/tasWSNR5sqVK4iMjISBgYHwZ21tDQDCkxFSUlLg4+ODevXqwcjISDgw5HSM4+Pjoa+vLzEzqJ2dXYEmAjE2NsaLFy9QtmxZNG/eHKqqqnj69Cnat28PY2NjqY0odyYoKytLlOzQly9fkJCQIHX/eL169RAXF5fnejVr1sTs2bOxefNm7N69W+r12NhYaGpqSszJkDN7r7x37s+fP+PKlSto2rSpxPKcWaxluXjxIpycnFC6dGmJ8klJSVIZwLp166JmzZpFNvwSyG6nuLg4qXu2nJyccOXKlR9+fxsbG1haWhbZbRNAdjvFxcWhQYMGEsudnJxEmU+hZs2asLS0LLLbJoDsmK5du4ZGjRpJLG/UqBGioqKKrB5i+vz5M65fvy71CClnZ+fvmqg1P6VKlYKampqo7ymLIh4jPn/+jOjoaDRv3lxiuYuLC86ePStznXPnzsHZ2VkiJhcXFzx+/BiJiYnCMjc3N2zcuBH9+/eXebwn2dgXKTjuk9l+9n3y8+fPuHr1qtQ5rnHjxlJzfRVGVFSUzPPm1atX8eXLl+9+34L4/PkzLl++jGbNmkksb9asWZ5zBl24cEFq22vWrBmSkpJw//59ANltqaurKzEnQ86PdXk/XlVFRQUmJiZST7iIjY2Fubm5zHUsLCyQkZEh8QM3JSUFGRkZ+d7OmjMSQt4+f/6MqKgoqf2pefPmhdqfmjdvLrU/FRdFPkbkTpD/6DHi0qVLUseIJk2a4MqVK6JufwVKNJQtWxampqYSf7KuKvzxxx9SnYDo6Gi0aNECQPY9LdHR0Zg9ezaOHj2K8PBw6Ovr49OnTwC+f6iGm5sbDAwM0Lp1azx79gwGBgaoW7cu0tPTpeqac5LPndkS857vf/75B3/88QdatWoFIyMjDB48GJqamsIohf79+2PevHlCeVtbW8yePRsHDx5EWFgYKlWqhEqVKklkSXOe7zp27FgYGxvD2toagwcPxunTp4vkasKKFSvQvXt39O7dG9WqVYO/vz90dXWxYcMGANlPydizZ49QPjg4GOnp6QgMDESNGjXQtm1b+Pj4YOXKlVLv/eeff+L27dtSkxjK26ZNm9CxY0d07twZpqam8PX1hba2tjAM1MfHB2vXrpVYx9TUFNWrV0elSpVQtmxZVK9eXeakPV26dEFiYiIuXrxYJLHk2Lhx4zdjyv3kEzMzM1haWkJdXR1ly5YVHlmWm5ubW7HEtHbtWri5uaFbt24wNzfH1KlToaOjgy1btgDInvU59zwEFhYWsLKygoaGBsqVKwcrKyupyVhzlpUvXx7q6uqwsrKChYVFkcS0fv16dOrUCe7u7jAzM8OkSZOgra2Nbdu2AciebXvz5s0S65ibm6NGjRpCTDVq1JCYnd7DwwPNmjWDiYkJTExM0KVLFwwYMKDIHjuniMeIgIAAeHh4oF+/frC0tMSiRYugr6+Pv//+G0D2jO9fP+Z1+/btSE9Px7p162BtbY2OHTti3LhxEk98cXd3x6ZNmzBx4kSEh4dDR0cHOjo6qFSpUpHGVhKxL1I43CdLxj65Zs0auLu7o3v37jA3N8e0adOgo6MjnAN8fX2lbk+xsLCAtbW1cD6wtrYWEmoAsHnzZujp6WHatGkwNzdH9+7d4e7ujlWrVhVJTIGBgejRo4ew7c2ZM0di25s8ebLEuSk4OBjv37/HihUrUKNGDbRr1w4+Pj4S8y8EBwfj5cuXCAwMhKWlJRwcHDBnzhzs3bu3SEY4tmjRAhERETh9+jSSkpKwdetWpKWlCQmVXbt2STwJy8rKCsbGxli3bh3u37+P+/fvY926dTA1NRUmGTx+/DguX76Mp0+f4unTpzh9+jRCQkLynLhQbAEBAfjzzz/Rv39/WFpaYvHixdDX18fq1asBZD+x4NixY0L5bdu2IT09HRs2bIC1tTVcXV0xfvx4LF68WOJ9bW1tYWtriwoVKkBDQwO2trb5Pk1H7JgU7RixevVquLu7o0ePHrCwsMCMGTOgq6srJHr9/Pywc+dOiXWqVauW7zFi06ZN0NfXx/Tp02FhYYEePXrI5Rgh2hwNtra22LNnD6pUqZLno97OnTuHOXPmCCf75ORkiavxlpaWSEpKwqNHj4Sr9VFRUVJDI3NbunQpPnz4AG9vb7Rt2xZt2rTBX3/9hWrVqsHDw0OibM4VhZMnTwozQH/48AFnz56V+cid73Hq1Cn8+uuv6NmzJzQ0NJCYmAg/Pz+hQ6GhoSHxqJs//vgDZcqUgbu7O9zd3YXlT58+Ra9evYQ6jhs3DkOHDkVgYCDevn2LM2fOSP0Qlpc9e/agUqVKGD16NHR0dHDjxg107dpVuAKko6Mj8ditN2/eoFOnTpg/fz7CwsKQlpaGwMBABAYGSrxv+fLl4erqivnz5xdJHF87cuQIKlasCE9PT1SuXBm3bt2Ct7e3MGxVS0tL6rm7K1eulOgs5oxY+Hom4rJly6JVq1YyO2fyduTIEairq2PQoEFCTF5eXkJMlStXLnBMXx+QypYti9atWxdLTAcOHIC6ujqGDRsGbW1tJCQk4M8//8Tjx48BZA9xyz0jelBQkEScORO5fV0u9+Ruv//+Ox4+fFgk90UeOnQI6urqGDJkiNBO/fv3R1JSEgDZMa1bt07iGeQ5w/ZyrugoKSlh3LhxMDQ0REZGBu7fv4/58+cLyQt5U8RjxK5du6CpqYkJEyZAT08PsbGxaNeunXB1V09PD6ampkL5169fo1WrVli6dCnOnTuH1NRULF68WKLD4unpCVVVVSxatEjiEcinT5+WugpDhce+yP/hPlky9sn9+/ejUqVKGDFiBLS1tREfH4/evXsL5zgdHR2pR7du3rxZ4hyX82Mw51z+8OFD9O7dG1OnTkXv3r3x7NkzTJ48GYcPH5Z7PED2tqehoYGxY8cK2567u7sw8kBXV1di23v9+jVcXV2xYMECnDx5Utj2li9fLpR59+4dOnbsiHnz5uHEiRNIS0vDoUOHJG6TkicHBwe8ffsW+/fvx6tXr2BgYIBRo0YJoxPS0tIkEodKSkoYOXIktmzZAn9/f6iqqsLa2hrdu3cXRkdlZmZi586deP78OZSVlaGtrY0uXbpIjQaRl507d0JDQwN+fn7Q09PD9evX0bZtW4n96etbPF6/fo0WLVpg2bJluHDhAlJTU7Fo0SKpREPu0Znt2rVDYmKixHvJiyIfI3x8fIRjRK9evYRjuba2ttQTMrZs2SJxjAgNDQWQHT+QfYzo1asXpk2bBg8PDzx79gyTJk2SeBCBGEqlpaXlm7ov6LOrnzx5gkaNGsHR0RE+Pj7Q0tJCYmIi9uzZg5kzZ6JChQpo0qQJKlasiHnz5iE9PR2TJ09GTEwMhg4digkTJiAzMxMNGjSAlpYWZs+ejQ8fPsDX1xdXr17FkiVL8n129ZcvX2BiYoKIiAiYmJjA1tYWa9askfkYp4CAACxcuBCBgYEwNzfHggULEBkZiYsXL6JChQoy3z/380dLMrGHZ/8MCvOM4pLiZ5pYRyxiPLbtZ5TXD5qSTBEft6ho219Jnafke7Avku3rH2clnaLtjwB+6EkRP6uv5wZTFEuWLCnuKshFv379irsKoivILWsljVhPgvsZ3Lx5M9/XRWs9PT09HD16FEpKSujcuTMcHR0xZswYqKmpCfe9LF++HO/evUPTpk3Rr18/9OrVS+KqnZKSErZs2YLMzEw0b94cXl5eGDNmjMR9M3mJiYlBxYoVYWJigsePHyMlJQV16tSRWXbEiBEYPHgwxo4di2bNmuHp06fYvXt3nid2IiIi+vmxL0JERPRz+OaIBsrGEQ0/N45oKBkU8QoWwBENJYWibX//pRENlI0jGn5uHNFQMnBEQ8nBEQ0/tyIb0UBERERERERExEQDEREREREREYmGiQYiIiIiIiIiEg0TDUREREREREQkGiYaiIiIiIiIiEg0TDQQERERERERkWiYaCAiIiIiIiIi0TDRQERERERERESiYaKBiIiIiIiIiETDRAMRERERERERiYaJBiIiIiIiIiISDRMNRERERERERCQaJhqIiIiIiIiISDRMNBARERERERGRaJhoICIiIiIiIiLRMNFARERERERERKJhooGIiIiIiIiIRMNEAxERERERERGJplRaWlpWcVeCiIiIiIiIiBQDRzQQERERERERkWiYaCAiIiIiIiIi0TDRQERERERERESiYaKBiIiIiIiIiETDRAMRERERERERiYaJBiIiIiIiIiISDRMNRERERERERCQaJhqIiIiIiIiISDRMNBARERERERGRaJhoICIiIiIiIiLRMNFARERERERERKJhooGIiIiIiIiIRMNEAxERERERERGJRqW4K1BSqKurF3cV6D/G1NS0uKsgurt37xZ3FeTi1q1bxV0F0VlYWBR3Fegb0tLSirsKVMQUqS+iqqpa3FUQnZubW3FXQXTBwcHFXQXRTZ8+vbirIBeKGFdGRkZxV0F0mZmZxV0F0Tx//jzf1zmigYiIiIiIiIhEw0QDEREREREREYmGiQYiIiIiIiIiEg0TDUREREREREQkGiYaiIiIiIiIiEg0TDQQERERERERkWiYaCAiIiIiIiIi0TDRQERERERERESiYaKBiIiIiIiIiETDRAMRERERERERiYaJBiIiIiIiIiISDRMNRERERERERCQaJhqIiIiIiIiISDRMNBARERERERGRaJhoICIiIiIiIiLRMNFARERERERERKJhooGIiIiIiIiIRMNEAxERERERERGJpkQkGgwMDLB169birkaheHt74+7du3j//j0uXboEZ2fnfMvb2Njg1KlTSE9Px6NHjzBp0iSpMo0bN8alS5fw/v173LlzB4MGDZJX9WViTCUjpp49e+LUqVOIi4vDvn37YGdnl2dZNTU1zJs3D4cOHcLNmzdl7mf29vbYtWsXLl26hNjYWBw7dgwDBgyQZwhSFLGdtm7dimbNmsHa2hodO3bExYsX8y0fHh6OLl26oHbt2rC3t4eXlxfu3bsnUebTp08ICAhAs2bNYGVlhcaNG2Pjxo3yDEOCIraTIsZE34d9kWzFvf0OGjQI8fHxeP36Nc6dO4eGDRvmW97GxgahoaF49eoV7t27h4kTJ0q83rFjRxw6dAiPHz/GixcvEBERgbZt28ozBCnNmzdHQEAAgoKCMGvWLFSvXj3PsgYGBpg4cSJWrlyJoKAgBAQEoGvXrlBWVhbK1KhRA9u2bZP609fXL4pwAChmO509exbz5s3DpEmTsGzZMqlz8NdSU1MxYcIEqb/4+HihzN27d7Fy5UpMnz4dkyZNwqJFi/Dvv/8WRSgCT09PxMXF4eXLlzhz5gwaNGiQb3lra2scPXoUL168wO3btzFhwoQ8yzo5OeH169ff7N+ILWfbe/XqFc6ePfvNbc/a2hrHjx9HWloa7t69Cz8/P4nXGzVqhFOnTiEpKQlpaWm4evUqRo4cKc8QpHh5eSEhIQFv3rzB+fPnC7Q/hYWF4fXr10hMTJS5Px0+fBhJSUlC28tjf/pmosHb2xtdu3aVWh4TEwN1dXXcv39f9Er9iMzMTFSpUgW3b98GANStWxdnzpyRKLN//3506tQJZmZmUFdXR3h4uKh1cHd3x5IlSzB79mzUqVMHkZGRCAkJQZUqVWSWr1ChAo4fP45nz56hfv36GD58OMaOHYtRo0YJZUxMTHD48GFERkaiTp068Pf3x7Jly9CpUydR686YSnZMbdq0waRJk7By5Uq0a9cO0dHRWL9+PfT09GSWV1ZWxsePH7F582acOnVKZpn09HRs3LgR3bt3R4sWLRAYGIgRI0agZ8+ecozk/yhiOx06dAgzZ86El5cX9u3bh7p162LAgAFISkqSWf7hw4fw8vKCnZ0d9u3bh6CgIHz48EEq4TNy5EiEh4djxowZOHbsGJYuXQpLS8uiCEkh20kRYyqp2BcpPEXcfrt06YJFixZh7ty5sLe3x9mzZ3HgwIF8Yzp8+DCSk5PRoEEDjBo1CqNGjYKPj49QJudHRIcOHWBvb48jR45g165d3+zIi8XR0REeHh7Yt28f/Pz8kJCQgPHjx0NTU1Nm+S9fviA8PBz+/v4YM2YMNm3ahKZNm8Ld3V2q7NixY+Ht7S38PXnyRN7hAFDMdrp69SoOHjyIpk2bYtiwYTA2NkZQUBDS0tLyXa9v377w8/MT/szMzITXSpcujQYNGmDQoEEYOXIkmjVrhtDQUJw9e1bO0WTr3Lkz5s+fj/nz58PJyQnnzp3D3r17YWhoKLN8hQoVcPDgQSQnJ6NRo0YYPXo0fHx8MHz4cKmy6urqWLt2LU6ePCnvMCS4ublh4cKFmDt3LhwcHHDu3Dns37//h7a9t2/fIjAwEC4uLqhduzb8/f0xadKkIkuyfr0/1a9fH2fPnsXBgwfzjSkkJATPnj2Dk5MTRo4cKbRVjsaNG+PkyZPo0KED6tevj5CQEAQHB4u+P5VKS0vLyq+At7c3Xr58iR07dkgsj4mJQbNmzXDlyhUYGxuLWqncDAwMMG/evAL9sLl+/To6dOiAO3fuICUlBdbW1rh//z7KlCkjlPnf//6HxMREmJiYwMvLCwcOHECjRo3yfV91dfUC1/fcuXO4evUqPD09hWUJCQkIDg6WypIB2VmquXPnQkdHBx8+fAAATJw4Ed7e3sLOPmfOHHTq1AnVqlUT1luzZg2sra2/mX0UA2Mq+phMTU0LHdM///yD+Ph4ifqHhYUhJCQECxYsyHfdKVOmoFq1agXaz1asWIFPnz5JHLQK4u7du4UqD/z87QQAt27dKlT5zp07w9LSErNmzRKWNW/eHC1btsSYMWOkyoeEhMDHxwdxcXHCVatz586hd+/eOH/+PDQ0NBAeHo7hw4cjLCwMGhoahY4hNwsLi0KVLwntVFg/e0zf6uAqEvZFsilSX0RVVbVQ5QEgIiIC165dg7e3t7AsNjYWe/bswV9//SVV3tPTE7Nnz4ahoaEQ04QJE+Dp6YmqVavm+TlnzpxBREQExo8fX6j6ubm5Fao8AEyfPh0PHjzA2rVrhWWLFi3C+fPnpbb3vPTq1QsWFhaYMmUKgOwRDTk/hN68eVPoOn0tODi40Ov87O00ffr0QpUHgMDAQOjp6Ukk1RYsWAAbGxu0bNlSqnxqairmzZuHIUOG5PnDXZYtW7ZAWVkZ3bt3L3QdCxvX6dOncf36dQwZMkRYdvXqVezZs0fYlr42cOBAzJgxAyYmJkI7jR8/HgMHDoS5ublE2e3bt+PatWsoVaoUOnbsiPr16xc6HgDIyMgoVPnw8HBcu3YNgwcPFpbFxsZi9+7dMkdoeXp6YtasWahSpYoQk6+vLzw9PfPth+/YsQMfP36Eh4dHoeoHZCeiC+PMmTO4du0avLy8hGVxcXHYvXu3zP1p0KBBmD17NgwMDCT2p0GDBsHExCTPz4mMjERERATGjRtX4Lo9f/4839dFvXXi5s2bcHd3h6GhIczNzdG/f388e/ZMeD06Ohqurq4wNTVFlSpV0LJlS1y4cEHiPe7evYs2bdpAR0cHdnZ2OHLkSKHqcOHCBTg4OADI/sJq1aolcWIHgG7dusHX1xe///77d0aaN1VVVdSrVw/Hjh2TWH7s2LE8T8JOTk4IDw8XNgYAOHr0KAwMDIQNwsnJSeo9jx49Cjs7O6ioqIgbRC6MKVtJiMnGxkbqqlh4eDjq1q0r2udYWVmhbt26OH/+vGjvmRdFbKdPnz4hNjZWagizs7MzoqOjZa5Ts2ZNqKioYOfOncjIyMDbt2+xe/du1KpVS0gqhIaGolatWtiwYQOcnZ3RvHlzTJ8+He/evZNrPIBitpMixvRfwb6IYm6/qqqqqFu3LkJDQyWWh4aGwtHRUeY6jo6OiIiIkIjp2LFjEjHJUqFChSJJ5CkrK6Nq1aq4du2axPKrV69KJHPyo6Ojg1q1auHGjRtSr82cOROBgYHw8/ODlZWVKHX+FkVspy9fviApKUkqAW9hYYEHDx7ku+6WLVswc+ZMrFq1Sqqdc0tKSsL9+/e/60JTYamqqqJOnTpS7RQWFpZnO9nb2yMyMlKinUJDQ6Gvry+R6PX09ISOjg7mzJkjn8rn4Xu2PQcHB5w5c0YipuPHj+e77dna2sLR0VH0UWiy5MR0/PhxieWhoaFwcnKSuY6s/elbMQFA+fLlkZqaKkq9c4iWaHj69Clat26NGjVqICwsDHv37sXbt2/RvXt3IXPz5s0bdO3aFSEhIQgLC0PNmjXRpUsXvHjxAkB2hqdXr17IzMzEsWPHsHz5csyZMwcfP3785ucbGRnByMgIEyZMQGhoKIyMjDBo0CBcuXIFRkZGModcyoOWlhZUVFQkOjUA8OzZM+jq6spcR1dXV2b5nNfyK6OqqgotLS2xqi8TY8r2s8dUqVIlqKioSGUXnz9/jsqVK//w+0dERCAuLg579+7Fli1bsH379h9+z29RxHZKTU1FRkaG1LBYTU3NPDPDhoaGCAoKwtKlS2FtbY26desiISEBf//9t1Dm4cOHuHTpEm7cuIHly5djypQpCA8PL/SVnu+hiO2kiDH9F7Avkk0Rt9+8YkpOTs4zJh0dHSQnJ0uVz3lNFi8vryKbj6NChQpQVlbGq1evJJa/evUKFStWzHfdqVOnIigoCIsXL0ZCQoLE6Ie0tDSsW7cOAQEBCAgIwJMnT+Dn51ckt9IpYjulp6cjMzMT5cuXl1hevnz5PEeMqKmpoXXr1ujRowf69OkDMzMzbN++HTExMVJl/f398ddff2H58uVwdHQUkpTylNNOsr73vL5zWe2U084561hbW2PChAno27dvoa/c/6i8YirscS+vbe/OnTt4/fo1zp49i9WrV2PNmjUi1l62/I7l39NOeX0POSPXxN6fCpR+Dg0NhYGBgcSy3BvPunXrYGNjg2nTpgnLVq9eDRMTE8TExKBevXpo0qSJxDrz5s3D/v37ERoaiq5du+LUqVO4efMmrly5Itx34u/vj1atWn2zjuHh4cjKykLTpk2xdu1aWFhYoFOnTvD19YW9vT1++eWXgoQqmqwsyTtSSpUqJbXsW+VzLy9IGXliTIoZU0F169YNZcuWRZ06dTBu3Dg8evQIe/fu/eH3LQhFbKecz/vWMgBISUmBn58fOnbsiLZt2+Ldu3dYsmQJhg8fjs2bN0NJSQmZmZkoVaoUFi9ejAoVKgAAJk+ejH79+uH58+dF8iNWEdtJEWMqqdgXKTxF3H7lEVMOV1dXzJkzB7169frmlWox5VXH/CxduhRlypSBkZERevTogXbt2mH//v0AgCdPnkjMx3Dr1i1UrlwZbdq0wc2bN8WtfB4UsZ1yy8rKyrOtypUrJ3ErlKGhId69e4d///0XderUkSg7aNAgfPr0CQ8ePMCRI0dQqVIlUUei5kfMdlJTU8PGjRvh5+dXrPPmyGvbc3FxQbly5eDg4IBZs2YhMTER27ZtE6nW+SuK/alnz56i708FSjQ0aNAAS5YskVgWFxeHXr16Cf+/cuUKIiMjpToBAHDv3j3Uq1cPKSkpmDVrFsLDw5GSkoKMjAy8f/8ejx49AgDEx8dDX19fYnILOzs7KCl9e+CFsbExoqKiULZsWTRv3hxJSUl4+vQp2rdvj9KlSxckTFE8f/4cX758kcoYaWtrS2Wjcjx9+lRmeeD/MlB5lfn8+bNwFUZeGFO2nz2m1NRUfPnyRWr0Qn5XygsjZz9NSEiAlpYWhg8fLvdEgyK2U6VKlaCsrCzVJi9evMhz8q8tW7agTJkyEqMTFixYgMaNGyM6Ohp2dnbQ1taGjo6OkGQAINwzmZSUJNdEgyK2kyLGVNKxL1Jwirj95hVT5cqV84xJ1lW/nHNk7it+rq6u2LBhA/r164eDBw+KWPO8vXnzBhkZGVJzb/z6669Soxxye/nyJQDg8ePHUFJSwsCBA3Hw4ME8ryLfvn07z6HWYlLEdipbtiyUlJTw9u1bieXv3r2TGuWQnypVqiAqKkpqec4tkLq6unj79i3CwsLknmjIaSdZ33vu7zyHrHbKOUbkjFixsrLC6tWrsXr1agCAkpISlJSU8Pr1a7i6uiIsLEwO0WTLK6bCHvfy2vYSExMBZM/5oK2tjUmTJsk90ZDfsfx72in39+Dq6oqgoCD07dtXLvtTgW6dKFu2LExNTSX+ZF1V+OOPPxAeHi7xFx0djRYtWgDIHpYRHR2N2bNn4+jRowgPD4e+vj4+ffoE4Puz4W5ubjAwMEDr1q3x7NkzGBgYoG7dukhPT5dZV3n6/PkzoqKipO65/P333xEZGSlznbNnz6JRo0YSnZDff/8djx8/Fjbqs2fPonnz5lLveenSJXz58kXcIHJhTNlKQkzXr18v1L3/36tUqVJQU1MT9T1lUcR2UlNTg7W1tdQM9GfOnMmzY/HhwweJR5cBEP6f06msW7cukpOTJeZkyHn0lryPgYrYTooYU0nHvkjBKeL2+/nzZ0RHR8PFxUViefPmzXHu3DmZ65w7dw7Ozs4SMTVv3lwiJiC77YKCgjBgwADs3r1bLvWXJSMjA/fu3UPNmjUlltesWRMJCQkFfh8lJSUoKyvnmwwzNjYukvkMFLGdVFRUoK+vLzXx861bt2BkZFTg93ny5InExQBZsrKyiuRc8PnzZ8TExEi102+//ZZnO124cAENGjSQaKfffvtNmFsiKSkJdnZ2cHR0FP7Wrl2L27dvw9HRMc/3FTOm6OhoqWOUi4tLnp+d86jIr2NycXGR2vZyU1JSKrJ+cF4x5fV0Eln7k6yY3NzcsHHjRvTv319u+5NoczTY2tri5s2bqFKlilRHIGenOnfuHDw9PdGiRQvUqFED5cuXl8isWFpaIikpSbiqAABRUVHfvMdn6dKlCA8PR+3atTF16lSEh4fjt99+w4gRI4RORlFatGgR+vTpg/79+8PS0hIBAQHQ19fHqlWrAACzZ8+WmKhk27ZtSE9PR1BQEKytreHq6gpfX18sWrRIKLNq1SoYGhpi8eLFsLS0RP/+/dGnT59vPkmAMf23Ylq/fj06deoEd3d3mJmZYdKkSdDW1hYyrmPGjMHmzZsl1jE3N0eNGjWgoaGBcuXKoUaNGqhRo4bwuoeHB5o1awYTExOYmJigS5cuGDBgAPbt21ckMSliO/Xr1w+7d+/Gzp07cfv2bcyYMQPJycnCLNMLFiyQmMm4adOmiI2NxbJly5CYmIjY2Fj4+vpCT08PNjY2AIB27dpBXV0dvr6+uHXrFqKiojBz5ky0bNkyz5ESYlLEdlLEmBQd+yL/RxG33yVLlsDDwwN9+/aFpaUlFi5cCD09PWG+mpkzZ0pM3Pm///0P6enpWLduHaytrdGxY0eMHTtWYmSMu7s7Nm7ciIkTJyI8PBw6OjrQ0dFBpUqViiSmw4cPo3HjxmjatCn09fXh4eGBSpUqCVd+u3btKvGUEGdnZzg4OEBfXx/a2tpwcHBA165dceHCBeEHasuWLWFnZwddXV0YGBiga9euqF+/vtREnvKiiO3UqFEjREdH4+LFi0hOTsaBAwfw5s0bYT6FI0eOSDw5JCoqCpcvX0ZycjJSUlLw77//4ty5cxKTsUZGRuLGjRt4/vw5nj9/josXLyI8PFzq1gp5Wbp0KXr16oU+ffqgevXqmD9/PvT09IQ4pk2bhkOHDgnld+zYgffv3+Pvv/+GlZUVOnTogNGjR2Pp0qUAsifNjIuLk/hLSUnBx48fERcXVySTUy9ZsgS9e/eW2vZy5lOYMWOGzG1v7dq1Qkxjx44VYgKAwYMHo3Xr1jA3N4e5uTn69OmDkSNHFslcZQAQEBAADw8P9OvXD5aWlli0aBH09fUl9qejR48K5bdv3y61P40bNw4BAQFCGXd3d2zatEnu+5NoUwQPGDAAGzduRN++feHj4wMtLS0kJiZiz549mDlzJipUqAAzMzPs3LkTdnZ2SE9Px+TJkyWyQU2bNkW1atXg5eWF2bNn48OHD/Dz8/vmTMb6+vr48uULYmNjhXsxY2Nj4ePjI3Pm1tTUVDx8+FAYlnbv3j1UrFhR+JJ/1M6dO6GpqYm//voLenp6uH79Olq3bi3c96KnpyfxHN3Xr1/j999/R2BgIC5duoTU1FQsXLhQ4uSemJiI1q1bY/HixfD29kZSUhKGDx9eZBldxlQyYjp06BDU1dUxZMgQVK5cGbdu3UL//v2RlJQEIHvoVO7s+7p16yQevZQzdCondiUlJYwbNw6GhobIyMjA/fv3MX/+/CK7L00R26lNmzZIS0vDihUrkJycjGrVqmHNmjXCFc/k5GSJ++ScnJywaNEirFmzBmvXrkXp0qVRu3ZtrFu3DmXLlgWQfT/oxo0bMX36dHTq1AkVK1ZE8+bNZT4uUx4UsZ0UMSZFx77I/1HE7XfXrl3Q0NDAhAkToKenh9jYWLRv316ISVdXV+K7fv36NVq3bo0lS5bg7NmzSE1NFSZIzDFw4ECoqqpi0aJFErGePn1aLk8Eye3cuXMoX748XF1doa6ujkePHmHevHnC7XXq6uoS20NGRgbat28PXV1dlCpVCs+fP8fx48dx+PBhoYyKigp69OgBDQ0NfPr0SXjPy5cvyz0eQDHbqVatWnj37h1OnDiBN2/eQEdHB3369BF+mL1580bq9qGTJ08iNTUVSkpK0NLSQufOnSWSCJmZmThy5IhQRlNTEy1atCiSySCB7Eeia2pqYvz48dDV1UVcXBxcXV3x8OFDALLbqW3btli8eDEiIiKQlpaGpUuXSvwoL27BwcHQ1NQULsbExsaiQ4cOEtve149MLci2p6ysjFmzZsHY2BhfvnzB3bt38ddff0lMyC1Pu3btgqampsT+1K5dO4ljee52atWqFZYuXYpz584hNTUVixcvlojJ09Mzz/0p9+iJH1EqLS0t3zGChXl29Z07dzBt2jScPn0aHz9+hKGhIZo1a4ZZs2ZBTU0N165dg4+PD2JjY6GrqwtfX18sWbIE7du3x4QJEwBk30M2fPhwXLp0CYaGhpg5cyYGDhz4zWdXX7x4EX369EFsbCweP36M+vXr4/79+zKf07x161aJZ8bmGD9+vFCP3Arz7GoiMRTF442K2t27d4u7CnKRezilIsj9GC/6+RTFMOifBfsi2RSpLyLrOynp3NzcirsKogsODi7uKohu+vTpxV0FuVDEuDIyMoq7CqIr6qdxyNO35oD7ZqKBsinSyZ1KBiYaSg4mGqg4/JcSDZRNkfoiTDSUDEw0lByKGBcTDT+3byUaRJujgYiIiIiIiIiIiQYiIiIiIiIiEg0TDUREREREREQkGiYaiIiIiIiIiEg0TDQQERERERERkWiYaCAiIiIiIiIi0TDRQERERERERESiYaKBiIiIiIiIiETDRAMRERERERERiYaJBiIiIiIiIiISDRMNRERERERERCQaJhqIiIiIiIiISDRMNBARERERERGRaJhoICIiIiIiIiLRMNFARERERERERKJhooGIiIiIiIiIRMNEAxERERERERGJhokGIiIiIiIiIhINEw1EREREREREJJpSaWlpWcVdCSIiIiIiIiJSDBzRQERERERERESiYaKBiIiIiIiIiETDRAMRERERERERiYaJBiIiIiIiIiISDRMNRERERERERCQaJhqIiIiIiIiISDRMNBARERERERGRaP4fOM3E0rM3+BgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1152x144 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokens = [['[CLS]'] + glove_tokenizer.tokenize(sent) for sent in sentences]\n",
    "fig = plot_attention(tokens, alphas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contextual Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ELMo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "watch1 = \"\"\"\n",
    "The Hatter was the first to break the silence. `What day of the month is it?' he said, turning to Alice:  he had taken his watch out of his pocket, and was looking at it uneasily, shaking it every now and then, and holding it to his ear.\n",
    "\"\"\"\n",
    "\n",
    "watch2 = \"\"\"\n",
    "Alice thought this a very curious thing, and she went nearer to watch them, and just as she came up to them she heard one of them say, `Look out now, Five!  Don't go splashing paint over me like that!\n",
    "\"\"\"\n",
    "\n",
    "sentences = [watch1, watch2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sentence: \"The Hatter was the first to break the silence . ` What day of the month is it ? ' he said , turning to Alice : he had taken his watch out of his pocket , and was looking at it uneasily , shaking it every now and then , and holding it to his ear .\"   [− Tokens: 58]"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from flair.data import Sentence\n",
    "\n",
    "flair_sentences = [Sentence(s) for s in sentences]\n",
    "flair_sentences[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Token: 32 watch"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flair_sentences[0].get_token(32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Token: 32 watch"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flair_sentences[0].tokens[31]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import ELMoEmbeddings\n",
    "elmo = ELMoEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence: \"The Hatter was the first to break the silence . ` What day of the month is it ? ' he said , turning to Alice : he had taken his watch out of his pocket , and was looking at it uneasily , shaking it every now and then , and holding it to his ear .\"   [− Tokens: 58],\n",
       " Sentence: \"Alice thought this a very curious thing , and she went nearer to watch them , and just as she came up to them she heard one of them say , ` Look out now , Five ! Do n't go splashing paint over me like that !\"   [− Tokens: 48]]"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elmo.embed(flair_sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Token: 32 watch, Token: 14 watch)"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_watch1 = flair_sentences[0].tokens[31]\n",
    "token_watch2 = flair_sentences[1].tokens[13]\n",
    "token_watch1, token_watch2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([-0.5047, -0.4183,  0.0910,  ...,  2.4156, -0.2228,  0.7794],\n",
       "        device='cuda:0'),\n",
       " tensor([-0.5047, -0.4183,  0.0910,  ...,  0.2134,  0.8352, -0.5018],\n",
       "        device='cuda:0'))"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_watch1.embedding, token_watch2.embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where do ELMo Embeddings come from?\n",
    "\n",
    "![](images/elmo_lstm.png)\n",
    "\n",
    "![](images/elmo_embed.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(-0.5047, device='cuda:0'), tensor(-0.5047, device='cuda:0'))"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_watch1.embedding[0], token_watch1.embedding[512]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True, device='cuda:0')"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(token_watch1.embedding[:1024] == token_watch2.embedding[:1024]).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.5949, device='cuda:0')"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity = nn.CosineSimilarity(dim=0, eps=1e-6)\n",
    "similarity(token_watch1.embedding, token_watch2.embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embeddings(embeddings, sentence):\n",
    "    sent = Sentence(sentence)\n",
    "    embeddings.embed(sent)\n",
    "    return torch.stack([token.embedding for token in sent.tokens]).float()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.3288,  0.2022, -0.5940,  ..., -0.0425,  1.0605,  0.2637],\n",
       "        [-0.7142,  0.4210, -0.9504,  ...,  0.3226, -0.6685,  1.7245],\n",
       "        [ 0.2981, -0.0738, -0.1319,  ...,  0.3250,  1.1165,  0.6453],\n",
       "        ...,\n",
       "        [ 0.0475,  0.2325, -0.2013,  ...,  0.7386, -0.5294, -0.8543],\n",
       "        [ 0.1599,  0.6898,  0.2946,  ...,  1.5656,  0.9583,  1.0337],\n",
       "        [-0.8872, -0.2004, -1.0601,  ..., -0.0166, -0.0841,  0.0618]],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_embeddings(elmo, watch1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GloVe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import WordEmbeddings\n",
    "glove_embedding = WordEmbeddings('glove')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence: \"The Hatter was the first to break the silence . ` What day of the month is it ? ' he said , turning to Alice : he had taken his watch out of his pocket , and was looking at it uneasily , shaking it every now and then , and holding it to his ear .\"   [− Tokens: 58],\n",
       " Sentence: \"Alice thought this a very curious thing , and she went nearer to watch them , and just as she came up to them she heard one of them say , ` Look out now , Five ! Do n't go splashing paint over me like that !\"   [− Tokens: 48]]"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_flair_sentences = [Sentence(s) for s in sentences]\n",
    "glove_embedding.embed(new_flair_sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True, device='cuda:0')"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.all(new_flair_sentences[0].tokens[31].embedding == new_flair_sentences[1].tokens[13].embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import TransformerWordEmbeddings\n",
    "bert_flair = TransformerWordEmbeddings('bert-base-uncased', layers='-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.6554, -0.3799, -0.2842,  ...,  0.0601,  0.8865,  0.4760],\n",
       "        [-0.1459, -0.0204, -0.0615,  ..., -0.0257,  0.5052,  0.3324],\n",
       "        [-0.0436, -0.0401, -0.0135,  ...,  0.3920,  0.5231,  0.9067],\n",
       "        ...,\n",
       "        [-0.2582,  0.6933,  0.2688,  ..., -0.0325,  0.0772,  0.2187],\n",
       "        [-0.1868,  0.6398, -0.8127,  ..., -0.5957,  0.2793,  0.1880],\n",
       "        [-0.1021,  0.5222, -0.7142,  ...,  0.3644,  0.0600, -0.1419]],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embed1 = get_embeddings(bert_flair, watch1)\n",
    "embed2 = get_embeddings(bert_flair, watch2)\n",
    "embed2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 8.5760e-01,  3.5888e-01, -3.7825e-01, -8.3564e-01,  1.3528e+00,\n",
       "          1.0204e-01, -8.2618e-01,  3.1570e-01,  3.1091e-01, -4.2653e-01,\n",
       "          1.6632e-01,  7.2439e-02, -1.0276e+00,  4.4680e-01, -1.4549e-01,\n",
       "          1.8315e-01,  3.7484e-01, -4.0245e-01, -1.4190e-01, -5.1596e-01,\n",
       "          1.1984e+00,  6.8202e-01,  1.1028e+00, -4.3955e-02,  1.6932e-02,\n",
       "          7.4420e-01, -1.0604e-01,  3.8719e-01,  7.3662e-02,  2.2424e-01,\n",
       "         -5.0281e-02,  1.9586e-01,  1.0773e+00,  1.0341e+00,  4.4217e-01,\n",
       "         -6.5567e-01, -5.6308e-01, -1.1827e-01,  1.2862e-01,  4.6581e-01,\n",
       "         -7.1305e-01, -1.5682e-01, -1.3552e-01,  4.5852e-02,  8.2853e-03,\n",
       "          5.2762e-01,  1.6906e+00,  5.5388e-01, -7.3778e-02,  5.1504e-01,\n",
       "         -2.0619e-01, -2.9505e-03,  2.5389e-01, -1.5379e-01,  8.4900e-01,\n",
       "          6.5440e-01, -3.2925e-01, -1.1198e+00,  2.4251e-01,  1.5586e-01,\n",
       "          5.7514e-01,  8.5832e-02,  3.3306e-01, -3.9344e-01, -4.1766e-01,\n",
       "          1.0790e+00,  1.1719e-02,  7.0214e-01, -6.0665e-01,  7.3702e-01,\n",
       "          6.8223e-02, -3.7110e-01, -7.7589e-01, -1.3057e-01,  1.7179e-01,\n",
       "          6.2051e-01, -6.4163e-01,  4.2881e-01,  5.7445e-01,  3.9801e-02,\n",
       "         -2.1279e-01, -5.1984e-01,  7.5919e-01,  3.0600e-01, -3.3987e-01,\n",
       "         -3.0427e-01, -5.7486e-01, -5.2273e-02, -6.8957e-01,  7.3935e-01,\n",
       "         -1.6545e-01, -8.2786e-01,  3.4169e-02,  7.1547e-01, -5.6962e-01,\n",
       "         -7.8686e-01,  3.6913e-01, -1.9961e-02, -5.7287e-01, -2.6407e-01,\n",
       "          8.5581e-01, -6.1418e-02,  3.8184e-01, -9.1563e-03, -1.0273e+00,\n",
       "         -5.3303e-01,  4.4014e-01,  1.7875e-01,  9.1851e-01,  5.9216e-01,\n",
       "         -4.8853e-01, -3.9106e-01, -8.8284e-02, -7.1936e-01, -3.8647e-01,\n",
       "          9.9550e-02,  2.2983e-01, -1.5325e+00,  6.5410e-02, -1.0896e+00,\n",
       "          1.2108e-01, -1.9921e-01, -2.5195e-01,  1.4421e-01,  2.7530e-02,\n",
       "         -3.0479e-01, -3.8533e-01, -1.1540e-01, -8.3979e-01, -7.7467e-01,\n",
       "          6.5181e-01,  3.1980e-01,  4.7616e-01, -8.2181e-01, -4.4323e-01,\n",
       "          1.2445e-01, -2.9082e-01, -4.2669e-01, -6.8031e-01,  1.1736e+00,\n",
       "          1.0199e+00,  7.3959e-01, -6.1510e-01,  1.0691e+00,  5.6806e-01,\n",
       "          9.4541e-01, -9.8110e-01, -7.1101e-01,  3.6151e-01,  2.0443e-01,\n",
       "         -5.0706e-02, -2.8713e-01,  1.2332e-01,  3.9614e-01, -4.4268e-01,\n",
       "         -5.5477e-01,  2.8325e-01,  5.5901e-01, -3.8944e-01, -4.2003e-01,\n",
       "         -1.2470e-01, -9.5830e-01, -3.4774e-02,  1.0381e+00, -2.2606e-02,\n",
       "          4.7267e-01,  8.1840e-01,  9.0778e-01, -1.9271e-02,  9.3007e-03,\n",
       "         -2.9097e-01,  5.1579e-01,  6.5791e-01,  3.1353e-01,  4.7983e-01,\n",
       "          6.2282e-01,  1.4813e-01, -3.8600e-01,  1.8680e-01, -7.3365e-01,\n",
       "         -5.2868e-01, -3.5756e-01,  1.4778e+00,  1.7065e+00,  5.2843e-01,\n",
       "          4.3788e-01, -6.0783e-01, -2.0891e-01, -2.6378e-01, -3.5563e-01,\n",
       "         -1.9136e-01,  4.4895e-01, -2.1980e-01, -2.2114e-01, -6.3080e-01,\n",
       "         -1.1537e+00,  3.8770e-01, -6.5095e-01, -1.4381e-01, -3.3387e-01,\n",
       "         -4.8165e-01, -2.2852e-01,  4.4550e-01, -1.4106e-01, -1.9820e-01,\n",
       "          1.2682e-01,  5.7340e-01,  5.2960e-01,  3.8535e-01, -1.8421e-01,\n",
       "         -2.7734e-01,  2.1106e-02, -5.1269e-01,  1.2724e-01, -5.2737e-01,\n",
       "          1.9794e-01,  6.6517e-01, -4.9908e-01, -5.2315e-01, -3.3081e-01,\n",
       "         -6.9502e-01, -2.1738e-01, -9.1546e-01,  8.4446e-01, -8.6271e-01,\n",
       "          9.8701e-01, -5.3786e-01,  2.2177e-01, -7.9477e-01,  1.6505e-01,\n",
       "         -3.9847e-01, -2.9883e-01,  2.0877e-01,  2.4684e-01,  1.0068e-01,\n",
       "          1.6014e-01, -1.1922e+00, -1.5679e-01, -2.0230e-03, -5.7467e-01,\n",
       "          4.6132e-03, -2.6884e-01,  4.2084e-01, -7.2796e-03,  3.4695e-01,\n",
       "          5.3609e-01,  2.1878e-01,  3.3520e-01, -2.8570e-01, -5.0958e-01,\n",
       "         -3.9454e-01, -8.9165e-01,  1.2181e+00, -6.8443e-01,  1.9280e-01,\n",
       "         -2.1449e-01,  4.6195e-02,  5.2594e-01, -4.3565e-02,  2.4555e-01,\n",
       "          1.9387e+00,  1.5278e-01,  9.9872e-01,  4.8577e-01, -4.1082e-01,\n",
       "         -8.3016e-01,  2.7045e-01,  4.8655e-01, -2.4722e-01, -1.8512e-01,\n",
       "          2.0524e-01, -6.6170e-02,  2.3728e-01,  1.2404e+00, -2.6077e-01,\n",
       "         -1.6077e-01,  1.3507e-01,  1.1847e-01, -2.9235e-01,  5.5437e-01,\n",
       "         -1.8559e-01,  1.3344e+00, -3.6667e-01, -3.7478e-01, -5.5664e-02,\n",
       "         -3.4930e-01,  8.5271e-01,  1.5741e+00,  6.0562e-01,  4.3779e-01,\n",
       "         -1.0298e+00,  4.0720e-01, -8.2617e-02,  2.7544e-02, -4.9199e-01,\n",
       "          7.3797e-01,  2.4101e-01,  9.6176e-01,  6.4367e-01,  1.1653e-01,\n",
       "          9.4047e-01, -2.6082e-01,  1.0976e+00, -1.1490e-01, -9.5253e-01,\n",
       "          5.8644e-02, -2.3060e-01, -6.7886e-02, -3.0703e+00,  2.4538e-02,\n",
       "          2.7582e-01, -4.3797e-01, -7.8420e-02,  4.2382e-01,  1.0763e+00,\n",
       "          1.8140e-01, -1.0572e+00, -3.9120e-01, -9.3492e-02, -3.7970e-01,\n",
       "          5.3946e-02,  1.1576e+00,  8.4130e-01,  1.8007e-01, -5.2631e-01,\n",
       "         -2.0291e-01, -1.9092e-02,  6.1720e-01,  1.1918e-01, -1.4457e+00,\n",
       "         -2.5592e-01, -1.2389e-01,  2.3928e-01,  1.1959e+00, -5.4284e-01,\n",
       "         -8.6952e-01, -4.8750e-01,  2.3500e-01,  1.3829e-02, -2.6659e-01,\n",
       "          9.7897e-01, -8.5489e-01,  3.1946e-01, -4.9998e-01, -3.2241e-01,\n",
       "         -1.3864e-01,  1.2900e-01, -7.5765e-01, -4.5315e-01,  2.9473e-01,\n",
       "          8.7803e-02, -1.3806e-01,  4.5430e-01,  8.0649e-02, -4.1994e-01,\n",
       "          5.8681e-03, -7.0910e-02,  3.8368e-01, -1.3059e-01,  8.7690e-01,\n",
       "         -6.1343e-01, -6.1015e-01, -7.0236e-01,  3.7512e-01,  3.3982e-01,\n",
       "          8.0857e-01, -1.6385e-01, -1.0569e-01,  6.1968e-01, -5.8364e-02,\n",
       "          2.3294e-01, -2.7713e-01,  5.2177e-01, -1.7158e-01, -1.0602e+00,\n",
       "         -4.3479e-01, -6.6953e-02,  3.6794e-01, -2.3135e-01, -2.0400e-01,\n",
       "         -5.3282e-01, -1.2286e+00, -2.4327e-01, -2.8658e-01,  1.3142e+00,\n",
       "          3.1902e-01, -9.1673e-02, -6.9225e-01, -3.0026e-01, -9.0491e-01,\n",
       "          8.4393e-02,  2.1298e-01,  8.2160e-01, -6.7558e-01, -8.9888e-01,\n",
       "          3.5113e-01, -5.2840e-01, -3.1060e-02, -2.8948e-01,  5.1492e-01,\n",
       "         -2.2328e-02,  1.6455e-01, -1.3406e-01,  7.9522e-01, -1.0633e-01,\n",
       "          4.0473e-01,  3.6492e-01, -6.1566e-01, -5.1250e-01, -5.4742e-01,\n",
       "          3.2135e-02, -4.5447e-02,  5.3155e-02,  7.0305e-01, -1.5488e+00,\n",
       "          6.2800e-01, -4.1181e-01,  3.1976e-01,  3.9906e-01, -1.7447e-01,\n",
       "          1.0357e+00, -5.4064e-01,  2.1086e-01, -5.4963e-01, -3.4768e-02,\n",
       "         -1.6814e-01,  4.8931e-01,  1.6012e-01, -6.6077e-01, -6.7637e-02,\n",
       "          9.2604e-02, -1.1114e+00,  4.6336e-01, -3.5745e-01, -3.2055e-02,\n",
       "          3.4265e-01,  3.5538e-01,  6.7498e-01, -2.0282e-01,  2.1987e-02,\n",
       "         -1.0548e+00,  9.8446e-01, -5.6733e-01,  6.9381e-02, -4.7736e-01,\n",
       "         -9.9546e-01, -3.5255e-02,  4.6173e-01,  7.1046e-01,  3.3707e-01,\n",
       "         -8.1136e-02, -4.8355e-01,  5.4515e-01,  3.7670e-02,  5.2528e-01,\n",
       "          4.1853e-02, -1.2563e+00, -3.2226e-01, -8.2649e-01, -7.5457e-02,\n",
       "         -7.5999e-01, -1.8107e-01,  1.1781e+00,  7.7896e-01,  6.6844e-01,\n",
       "         -7.6050e-01, -4.5029e-01, -5.4186e-01,  2.7287e-03,  2.3229e-01,\n",
       "         -6.9320e-01, -4.4791e-02, -7.4108e-02,  9.3096e-01, -4.1993e-01,\n",
       "         -7.3685e-01,  6.3003e-02,  4.0879e-01, -7.2932e-01, -1.6402e-01,\n",
       "         -2.7400e-01,  7.5048e-01, -6.5255e-01, -4.3226e-01, -6.2632e-02,\n",
       "         -8.2434e-01, -9.1996e-01,  5.1996e-01,  3.3342e-01,  3.0552e-03,\n",
       "         -3.7381e-01,  8.1910e-01, -2.5230e-01, -8.2635e-01,  3.8007e-01,\n",
       "         -1.0642e+00, -7.9937e-01,  1.0933e-01, -2.3652e-01,  4.5540e-01,\n",
       "         -2.9788e-01, -1.5564e+00, -2.3166e-01, -1.1862e+00, -2.9165e-01,\n",
       "         -2.1934e-01,  1.1169e-01,  1.3712e-01, -5.1983e-01, -2.5509e-01,\n",
       "         -4.4754e-01, -4.7948e-01,  4.6833e-01, -1.8625e-01,  6.9716e-01,\n",
       "         -1.1866e-01,  6.1084e-01,  6.3114e-01, -8.4579e-01, -4.6913e-01,\n",
       "          9.2668e-02,  2.9337e-02,  2.5508e-01,  4.5543e-01,  1.8251e-01,\n",
       "         -1.1855e-01, -1.2805e-01,  8.3519e-02,  1.7325e-01,  6.1112e-01,\n",
       "         -3.9921e-01,  1.6798e-01, -8.4275e-02,  3.9789e-01,  9.2255e-01,\n",
       "          7.7275e-02, -4.0163e-01,  3.1146e-01, -3.0563e-01, -5.8679e-01,\n",
       "          4.1130e-01, -4.3299e-01, -1.2496e+00, -8.5653e-01, -2.3101e-01,\n",
       "         -1.2980e+00, -2.7850e-01,  1.3634e-01,  3.4895e-01,  4.5002e-01,\n",
       "         -6.9828e-01,  6.2787e-01, -2.2598e-01, -1.7732e-01, -3.2117e-02,\n",
       "          4.8136e-01,  5.0901e-01,  5.0199e-01,  3.6993e-01, -4.5981e-01,\n",
       "         -6.7486e-02, -8.6722e-01, -5.7723e-01, -3.2765e-01,  7.8836e-01,\n",
       "          4.0974e-01, -1.8077e-01, -8.4645e-01, -2.5588e-01,  1.0655e+00,\n",
       "         -3.7329e-01, -4.9543e-01, -2.4806e-01, -3.7839e-02,  9.1718e-02,\n",
       "         -1.5725e-01,  2.4216e-01,  6.5864e-02,  1.3641e+00, -1.2707e+00,\n",
       "          7.8174e-02, -1.0401e+00, -3.9388e-01, -2.5627e-01, -4.9158e-01,\n",
       "         -4.0819e-01,  7.3981e-01, -4.4651e-01, -3.1208e-01,  6.2193e-01,\n",
       "          1.1230e+00, -4.4226e-02, -1.2044e+00,  2.2500e-01, -9.1115e-02,\n",
       "          3.5365e-01, -3.7870e-01, -1.0870e-01,  3.2082e-01,  1.2314e-01,\n",
       "         -2.1645e-02,  1.1413e+00,  1.3573e-02, -9.9737e-02,  9.3759e-01,\n",
       "          5.0418e-02, -4.9671e-01,  9.0127e-01,  1.4177e-01,  2.0027e-01,\n",
       "          6.3084e-01, -2.0751e-02,  3.0619e-01,  4.5247e-01, -7.6788e-01,\n",
       "         -7.2624e-01, -5.2419e-01,  6.0564e-01, -1.9887e-02, -2.9045e-01,\n",
       "          4.0384e-01,  9.0467e-01, -9.9711e-01,  1.0205e-02,  1.7958e-01,\n",
       "          3.5785e-02, -2.3696e-01,  6.4812e-01,  1.5352e-01,  3.1153e-02,\n",
       "          1.3805e-01, -1.4471e-01, -9.9713e-01,  1.7288e-01,  1.8073e-01,\n",
       "         -2.8294e-02,  7.8877e-01,  3.7700e-01, -7.0564e-02,  1.0712e+00,\n",
       "         -4.2392e-01,  4.1656e-01, -6.8028e-01,  1.6911e-01,  2.6617e-01,\n",
       "         -1.1312e+00,  1.6047e-01, -4.5070e-01,  4.9935e-01,  3.4413e-01,\n",
       "          6.4540e-01, -2.6726e-01,  1.5033e-01,  4.2149e-01,  3.4961e-01,\n",
       "         -1.5505e-01,  1.5471e+00,  7.7871e-01, -2.7912e-01, -4.5703e-01,\n",
       "         -7.9149e-01, -5.7845e-01, -2.4879e-01, -6.0028e-01,  6.9574e-02,\n",
       "         -1.9024e-02,  2.0336e-01,  1.3373e+00,  1.7257e-01, -3.3320e-01,\n",
       "          1.4024e-01, -2.2573e-02, -2.2282e-01,  3.4986e-02, -1.4985e-01,\n",
       "          5.2162e-02,  1.5562e-01, -2.3856e-01, -7.4213e-02, -2.1547e-01,\n",
       "          7.6899e-01,  1.1663e-01,  5.3913e-02, -2.6869e-01, -2.3951e-01,\n",
       "          8.8453e-01,  3.2750e-01, -7.8229e-02,  4.5901e-01, -4.3255e-01,\n",
       "         -4.4316e-01, -3.2504e-01, -1.4550e+00, -8.6594e-02, -7.3577e-02,\n",
       "         -2.0595e-01, -6.1714e-01, -7.1308e-01,  8.2567e-01,  1.6622e-01,\n",
       "         -8.5806e-01, -6.5291e-01, -1.5517e-02, -6.2837e-01,  5.6276e-01,\n",
       "          9.6670e-02,  4.9524e-01,  3.4259e-01, -6.7065e-01, -4.1591e-01,\n",
       "         -3.3399e-01, -7.4808e-01,  3.2285e-01, -5.1326e-02,  4.1144e-01,\n",
       "          2.6481e-01, -9.0313e-01, -4.4464e-01,  6.6253e-02,  4.1635e-01,\n",
       "         -7.5472e-01,  1.2165e+00, -4.2961e-01,  6.3139e-01, -2.1191e-02,\n",
       "         -4.6616e-01, -1.4575e-01, -7.4167e-01, -2.7592e-01, -4.6199e-02,\n",
       "          4.5393e-01,  4.7207e-01, -1.7622e-01, -6.7738e-01,  3.7787e-01,\n",
       "          5.3374e-01,  3.6832e-01,  1.3724e-01, -1.1172e+00, -2.5947e-01,\n",
       "          1.1888e+00, -1.0117e-01, -4.8335e-01,  5.8898e-02,  1.6610e-01,\n",
       "          6.6885e-01, -9.0577e-01, -2.3182e-01,  5.9258e-01,  1.1081e-01,\n",
       "          1.8942e-01,  6.5041e-01,  4.6973e-01, -3.1615e-01, -5.9470e-01,\n",
       "         -5.1061e-01, -9.7604e-01,  6.0229e-01, -6.3250e-01, -4.2595e-01,\n",
       "         -5.7413e-01, -7.1769e-01,  9.2507e-01, -6.2844e-01,  1.8694e-01,\n",
       "          2.0768e-01,  1.1880e-01,  4.1445e-01], device='cuda:0'),\n",
       " tensor([-9.8449e-02,  1.4698e+00,  2.8573e-01, -3.9569e-01,  9.6890e-02,\n",
       "          8.6436e-01, -3.7401e-01,  2.1752e-01, -2.5937e-01, -4.2487e-01,\n",
       "         -6.1411e-02,  3.3272e-01, -1.9102e-01, -2.9803e-02, -1.7782e-01,\n",
       "          7.1223e-01, -3.6092e-01,  4.7531e-01,  1.8901e-01, -1.9464e-01,\n",
       "          5.5041e-01, -2.0075e-01, -4.9742e-01, -1.3700e-02,  6.5577e-02,\n",
       "          2.2761e-01,  7.6017e-01, -7.9264e-02,  2.7941e-01, -5.4700e-02,\n",
       "         -2.6595e-01,  5.5095e-01, -2.1662e-01,  1.4397e-01, -3.8731e-01,\n",
       "         -1.9042e-01,  1.3155e-01,  2.2447e-01,  2.9488e-01,  2.7523e-01,\n",
       "         -2.7458e-01, -5.0479e-01, -3.9631e-01,  3.0837e-01, -6.3027e-01,\n",
       "         -3.5416e-01,  1.1855e+00,  8.9831e-02,  4.4822e-01,  4.0406e-01,\n",
       "         -3.4759e-01,  5.6113e-01,  1.0385e+00,  1.5012e-01,  3.7633e-01,\n",
       "         -3.9084e-01,  1.0627e-01, -5.0344e-02,  2.2779e-01, -5.2758e-01,\n",
       "         -1.9242e-01,  1.8374e-02,  5.2802e-01,  1.6686e-01, -7.3694e-01,\n",
       "          4.7303e-01, -4.7016e-01,  1.6289e-01,  9.7774e-02,  1.2824e-01,\n",
       "         -2.8870e-01, -3.9116e-01, -1.8324e-01,  5.2460e-01, -1.9830e-01,\n",
       "          5.4971e-01,  4.1160e-02, -3.3559e-01,  2.1169e-02, -8.5710e-02,\n",
       "          1.3109e-01, -3.6348e-01,  2.7382e-01,  1.1263e+00, -2.5038e-01,\n",
       "         -6.4161e-01, -3.2357e-01, -1.3938e-01, -1.0110e+00,  7.3331e-01,\n",
       "          1.0102e-01,  1.5387e-01, -2.9724e-01,  2.5424e-01,  3.2296e-01,\n",
       "         -3.5495e-01,  1.2272e-01,  9.6730e-02,  2.1487e-02,  9.1426e-02,\n",
       "         -1.4177e-01,  5.2628e-01,  2.1227e-01, -1.7098e-01, -3.8353e-01,\n",
       "          1.6601e-01,  4.4298e-01,  6.3222e-03, -5.9510e-01,  3.5965e-01,\n",
       "         -7.1791e-01, -3.6388e-01, -7.0434e-02,  1.8896e-01,  5.4333e-03,\n",
       "          1.1766e-02,  1.0752e-01, -5.5094e-01, -6.7836e-02,  3.5240e-01,\n",
       "         -2.6229e-01,  3.7230e-01, -8.1172e-02,  4.8515e-01,  1.0484e-02,\n",
       "         -2.9454e-01, -3.1111e-01, -2.6310e-01,  6.3381e-02, -1.0854e-01,\n",
       "          1.3688e+00,  2.9465e-01,  2.0623e-01,  3.8318e-02,  5.4437e-02,\n",
       "         -1.5726e+00,  3.7239e-01, -6.0964e-01, -8.8024e-01,  2.4682e-01,\n",
       "          6.7070e-01, -5.6562e-01,  2.1737e-01,  8.4505e-02, -2.6882e-01,\n",
       "          1.4112e-01, -8.0701e-01, -1.1845e-01, -3.2562e-01,  7.8287e-01,\n",
       "         -1.8123e-01, -7.1260e-01,  2.0073e-01, -2.2902e-01, -1.7190e-01,\n",
       "          8.2328e-02,  3.2728e-01,  3.3844e-01, -4.3359e-01, -2.0109e-01,\n",
       "          5.2177e-01, -2.1286e-01, -5.4271e-01,  4.2771e-01, -2.8439e-02,\n",
       "          6.8534e-01,  3.0166e-01,  3.3880e-01, -7.2538e-01,  4.7892e-02,\n",
       "         -5.7754e-01,  4.3418e-01,  6.2832e-01,  1.1230e-01,  1.6312e-02,\n",
       "         -3.1558e-01,  3.0021e-01, -3.4793e-01, -4.2287e-01,  8.9215e-02,\n",
       "         -1.7020e-01,  5.5911e-01,  4.3507e-01,  1.0352e+00, -6.0532e-02,\n",
       "          6.4033e-02,  6.3265e-01,  5.8860e-01, -6.2739e-01, -3.6926e-01,\n",
       "         -1.3001e-01,  3.0337e-01, -7.3679e-01, -1.0968e+00, -4.9620e-01,\n",
       "         -2.4886e-02,  6.0554e-02, -2.8982e-01, -5.5470e-01,  9.2083e-02,\n",
       "         -3.5928e-01, -6.8506e-01,  7.0260e-01,  7.4057e-01,  4.1953e-02,\n",
       "         -2.2261e-01,  3.1581e-02, -5.7034e-01, -1.5258e-01, -1.9746e-01,\n",
       "         -5.7486e-01, -6.6855e-01, -5.3475e-01,  2.1482e-01, -8.8653e-01,\n",
       "          9.2317e-02, -2.2076e-02,  9.8558e-01, -8.5093e-01, -1.6057e-02,\n",
       "         -1.1169e-01, -2.8778e-01, -4.2301e-01,  7.7859e-02,  6.9498e-02,\n",
       "          4.2390e-01, -7.0423e-02,  7.7604e-02, -4.0021e-01, -2.4131e-01,\n",
       "          1.3328e-01, -1.1587e-01,  9.0171e-01, -3.4312e-01,  6.0788e-01,\n",
       "          3.1653e-01, -2.4758e-01,  4.5197e-02,  4.9370e-01,  3.8995e-01,\n",
       "         -3.4692e-01,  3.3056e-02,  2.0106e-01,  4.0234e-01,  1.2993e-01,\n",
       "          2.9119e-01,  9.2159e-01, -4.1758e-01,  1.5004e-02,  5.2865e-02,\n",
       "          6.0922e-01,  7.6300e-02,  2.6940e-01, -6.3987e-01,  1.7187e-01,\n",
       "         -1.4997e-01, -2.2630e-01, -5.7578e-01, -5.4688e-02, -1.4096e-01,\n",
       "         -1.8318e-01,  1.6508e-01,  5.1326e-01,  4.4380e-01, -2.8080e-01,\n",
       "         -4.5417e-01, -3.0488e-01,  4.8723e-01, -1.7721e-01, -2.2288e-01,\n",
       "          6.0853e-01, -2.2009e-04,  5.6217e-01,  1.1224e+00, -1.4180e+00,\n",
       "         -3.6117e-01,  1.5190e-01, -1.7284e-01, -1.7632e-02, -2.1651e-01,\n",
       "         -8.6195e-01,  4.2032e-02, -4.6597e-01,  7.2990e-02, -6.2120e-02,\n",
       "         -3.9781e-01,  3.5995e-01, -3.4646e-01, -3.0336e-01, -1.5005e-02,\n",
       "          3.3628e-01,  5.7878e-02, -3.3315e-01,  1.1097e-01,  3.0777e-02,\n",
       "         -7.1115e-01,  3.6039e-01,  1.6007e-01,  3.4098e-01, -5.3309e-01,\n",
       "          5.6319e-01,  2.1157e-01,  2.6252e-01,  4.2321e-01, -5.2220e-01,\n",
       "          2.4814e-01, -3.6991e-01, -5.4816e-01, -5.3290e+00,  1.0130e-01,\n",
       "         -2.4280e-01,  2.6960e-01,  6.2188e-01,  3.4420e-01,  2.2724e-01,\n",
       "          1.2104e-01, -3.0778e-01, -1.3375e-01,  1.2070e-01,  6.9458e-01,\n",
       "         -1.7730e-01,  6.3572e-01,  3.9321e-01,  1.0833e-03, -4.1992e-01,\n",
       "         -3.1437e-01,  4.6219e-01,  3.0782e-01, -2.4236e-01, -6.3911e-01,\n",
       "         -7.2801e-03, -2.1637e-02,  2.9671e-01,  8.6041e-01, -1.6743e-01,\n",
       "         -5.2726e-01, -7.3630e-01, -4.9699e-01, -4.0285e-02, -5.5160e-02,\n",
       "         -2.8419e-01,  3.4066e-01,  1.9508e-01, -3.1847e-01,  1.1065e-01,\n",
       "         -1.3114e+00, -6.5144e-02, -5.2608e-01, -2.2926e-01, -4.9479e-01,\n",
       "         -3.6531e-01, -1.5899e-01,  9.3910e-02,  1.2114e-01, -1.5592e-01,\n",
       "          1.9144e-01, -7.1318e-01, -1.7203e-01, -6.8007e-02,  4.2680e-01,\n",
       "          2.7249e-01, -8.1867e-02, -7.7034e-01,  4.4867e-02,  6.1248e-01,\n",
       "          6.4211e-01, -4.2664e-01, -4.1450e-01, -4.6396e-02, -5.7337e-01,\n",
       "          4.2681e-01, -4.0819e-02,  2.8339e-02, -3.5921e-01, -2.4894e-01,\n",
       "         -4.9876e-02,  5.8249e-01, -1.4033e-02, -3.2829e-01,  1.0102e-01,\n",
       "         -4.5791e-01, -1.0023e+00, -2.6306e-01, -2.5233e-01,  6.0858e-01,\n",
       "          2.0588e-01, -2.5912e-01, -1.0158e-01, -3.4643e-01, -1.6082e-01,\n",
       "         -3.1902e-01,  5.3895e-01,  2.7880e-01, -6.8466e-01, -7.6329e-02,\n",
       "          9.4315e-02,  1.3531e-01, -8.8651e-01, -5.7966e-01, -3.3632e-01,\n",
       "          3.1987e-01, -1.9380e-01,  2.8006e-01,  5.9293e-01,  5.4404e-01,\n",
       "          3.7746e-01,  1.5074e-01,  1.3050e-01,  1.8917e-01, -8.2221e-03,\n",
       "         -3.1581e-01,  3.0006e-01,  2.2615e-01, -4.7198e-01,  2.9255e-01,\n",
       "          3.9098e-01,  7.1997e-01, -6.5656e-01, -5.7878e-02, -7.4665e-01,\n",
       "          4.5998e-01, -1.7215e-01,  3.4801e-01, -3.4560e-01,  3.4529e-01,\n",
       "          4.9162e-01,  3.4105e-01,  5.2297e-01, -5.5147e-01,  2.9069e-02,\n",
       "         -1.3466e+00,  1.6773e-01, -2.8781e-01, -6.5206e-01, -2.8571e-01,\n",
       "          4.4820e-01, -8.2600e-01,  2.7299e-01, -1.6052e-01, -4.9491e-01,\n",
       "         -4.4123e-01, -3.0832e-02,  3.5858e-01, -3.2000e-01, -3.6482e-01,\n",
       "         -4.6250e-01,  6.8682e-01,  6.2212e-01, -6.7616e-01,  4.4652e-01,\n",
       "          6.3600e-02,  5.6310e-01,  9.1847e-01,  4.4499e-01,  4.0972e-01,\n",
       "          6.3157e-01, -4.8508e-01, -4.0152e-01, -5.0366e-01, -4.5478e-02,\n",
       "         -4.5590e-01, -7.5448e-01,  6.8782e-01, -3.8171e-01,  2.5105e-01,\n",
       "          1.3606e-01,  1.9530e-01, -1.6731e-01,  1.8509e-01,  2.4151e-01,\n",
       "         -1.2497e-01,  7.3940e-02,  8.0915e-02,  4.0503e-01, -4.4470e-01,\n",
       "          1.4211e-01, -6.2766e-02, -1.0847e+00,  1.5061e-02,  3.1443e-01,\n",
       "          6.1792e-02, -2.7084e-01, -9.5111e-02,  8.0670e-01,  3.4657e-01,\n",
       "          1.7020e-01, -3.4658e-01,  6.2307e-01, -3.1217e-01,  6.7439e-02,\n",
       "         -4.2302e-01,  6.6040e-01,  2.8155e-01, -3.0294e-02,  1.3720e-01,\n",
       "         -4.0739e-01,  4.6180e-01,  7.8493e-01, -2.0656e-01, -6.3213e-02,\n",
       "         -8.2126e-01,  1.9584e-01, -8.8633e-02, -1.0430e+00,  2.6545e-01,\n",
       "          1.2379e+00,  6.7183e-01, -6.8933e-01,  2.3559e-01,  6.4715e-01,\n",
       "          1.2309e-01, -2.8892e-01,  2.7236e-01, -4.1748e-01,  9.3375e-01,\n",
       "          1.2763e-01, -4.8758e-01,  4.1386e-01, -5.7510e-01,  3.0247e-02,\n",
       "         -5.2484e-01, -3.5384e-01, -4.3309e-01, -5.0379e-01, -1.4411e-01,\n",
       "         -9.1641e-02,  1.3801e-01, -2.2797e-01,  4.0613e-01, -5.3039e-01,\n",
       "          2.8264e-01,  1.4536e-01, -3.8260e-01, -1.0636e-01, -1.9395e-01,\n",
       "          2.4888e-01,  3.9480e-01, -1.9706e-01, -9.8293e-02, -5.3219e-01,\n",
       "          3.9783e-01, -3.8129e-01,  1.8971e-01, -3.3756e-01, -6.2739e-01,\n",
       "         -7.0887e-01,  3.7720e-01,  2.1101e-01, -5.8501e-01, -4.8129e-01,\n",
       "          2.2947e-01, -1.3340e-01,  1.1180e+00,  6.6438e-02,  4.8484e-01,\n",
       "         -2.2995e-01, -3.2569e-01,  4.2933e-02,  3.4589e-01, -2.4830e-01,\n",
       "          1.0948e-01, -1.3069e-01, -5.7944e-01, -1.9196e-01, -7.8060e-02,\n",
       "          1.1934e-01, -2.3422e-01, -2.1563e-01, -3.3932e-01,  8.7396e-02,\n",
       "          2.8107e-01, -4.7971e-01,  2.1479e-01, -1.1446e-01, -5.0093e-01,\n",
       "         -1.2180e-01,  5.6778e-01, -1.7505e-01,  5.0499e-01, -6.7217e-01,\n",
       "          3.6737e-01, -1.8358e-01,  2.3660e-01,  2.9737e-02, -3.5624e-01,\n",
       "         -1.1451e-01,  6.5315e-01, -4.0913e-01, -5.6405e-01,  7.2596e-01,\n",
       "         -2.3398e-01,  1.7279e-01, -4.2138e-01,  1.0075e-01,  1.4012e-01,\n",
       "          1.6682e-01,  2.0844e-01,  7.5970e-02,  2.4352e-01,  1.3456e-01,\n",
       "          1.4141e-01,  1.7959e-01,  1.4143e+00,  1.2107e-01,  3.6410e-01,\n",
       "         -5.9567e-01,  4.5287e-02,  4.0947e-02, -6.3151e-03, -1.5476e-02,\n",
       "         -5.4966e-02,  4.0419e-02,  5.3073e-01,  3.6719e-01,  5.9932e-03,\n",
       "         -6.0457e-01, -4.0461e-02,  3.0526e-01,  2.2857e-01,  3.1699e-01,\n",
       "         -6.1115e-01, -3.5432e-01, -1.7680e-01,  3.0425e-01,  4.4631e-01,\n",
       "         -8.0750e-02,  3.3444e-02, -1.7067e-01,  2.9058e-01,  1.1439e-01,\n",
       "         -1.2962e-02,  2.6377e-01, -2.9394e-01, -2.3483e-01,  2.1922e-01,\n",
       "          7.9876e-01,  9.3168e-01, -1.6078e-01, -6.8044e-02, -2.8835e-01,\n",
       "         -3.7442e-02,  8.7288e-01, -3.0851e-02, -2.6066e-01,  8.7559e-01,\n",
       "          5.2805e-02,  4.2036e-01, -4.4657e-01, -6.0128e-01,  2.4042e-01,\n",
       "          1.0409e+00,  4.5662e-02, -5.9405e-01, -7.7239e-02,  4.4467e-01,\n",
       "          2.8607e-01,  3.1795e-01,  4.0720e-01, -4.4101e-01,  1.0156e-01,\n",
       "          8.4665e-02,  2.2385e-01,  3.2971e-02,  8.5590e-01, -8.3134e-02,\n",
       "          2.4421e-01, -2.5320e-01,  3.0961e-01,  3.2430e-01,  7.6288e-01,\n",
       "         -4.1389e-01, -7.4261e-01, -1.0888e-01,  3.0744e-02, -8.4961e-02,\n",
       "         -3.3006e-01, -9.4410e-02, -1.7684e-01,  9.1215e-02,  3.9247e-01,\n",
       "         -1.6165e-01, -2.5071e-01,  3.2119e-01, -5.8943e-01,  5.3371e-01,\n",
       "         -3.9494e-01,  4.5296e-01, -3.2192e-02,  2.9046e-02,  9.0094e-02,\n",
       "          8.0394e-02, -3.8424e-01, -2.4358e-01, -5.8893e-01, -1.6014e-01,\n",
       "         -5.0848e-01, -2.1314e-02, -5.0818e-01, -9.5969e-02,  4.0968e-01,\n",
       "          3.0001e-01, -5.3867e-01,  2.3648e-01, -5.9909e-01,  1.8138e-01,\n",
       "          5.6676e-01,  3.3343e-01,  3.6125e-02, -7.1760e-01, -2.8207e-01,\n",
       "         -1.7438e-01, -9.5262e-02,  2.1486e-01, -5.2020e-02, -1.4080e-04,\n",
       "         -1.5751e-01, -6.5504e-02,  3.1234e-01, -1.2235e-01, -1.2310e-01,\n",
       "         -2.6430e-01,  6.9013e-01, -8.8553e-01,  1.4780e-01, -4.3434e-01,\n",
       "          9.2149e-03, -5.1354e-01, -4.0582e-01,  4.0951e-02, -9.0218e-01,\n",
       "          1.7678e-01,  8.2160e-01,  9.0611e-02,  1.0791e-01, -5.0325e-01,\n",
       "          5.2987e-01, -4.0699e-01,  1.9170e-01, -2.1431e-01,  4.5255e-01,\n",
       "          5.6901e-01,  2.7348e-01, -4.4278e-01,  1.8978e-01,  1.3641e-03,\n",
       "          2.6983e-01,  6.8033e-01, -4.7854e-01,  8.3501e-01,  5.7284e-01,\n",
       "          5.5623e-01, -3.8540e-01, -1.7700e-02,  3.0426e-01,  2.9132e-01,\n",
       "         -2.5081e-01, -5.4555e-01,  4.0296e-02, -3.4137e-01,  4.2379e-01,\n",
       "         -5.2939e-01,  1.7819e-01, -1.2489e-02,  5.5839e-01,  1.0675e-01,\n",
       "          3.1746e-01, -2.8264e-01, -2.1325e-01], device='cuda:0'))"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bert_watch1 = embed1[31]\n",
    "bert_watch2 = embed2[13]\n",
    "bert_watch1, bert_watch2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.3504, device='cuda:0')"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity = nn.CosineSimilarity(dim=0, eps=1e-6)\n",
    "similarity(bert_watch1, bert_watch2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = [Sentence(watch1), Sentence(watch2)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence: \"The Hatter was the first to break the silence . ` What day of the month is it ? ' he said , turning to Alice : he had taken his watch out of his pocket , and was looking at it uneasily , shaking it every now and then , and holding it to his ear .\"   [− Tokens: 58],\n",
       " Sentence: \"Alice thought this a very curious thing , and she went nearer to watch them , and just as she came up to them she heard one of them say , ` Look out now , Five ! Do n't go splashing paint over me like that !\"   [− Tokens: 48]]"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from flair.embeddings import TransformerDocumentEmbeddings\n",
    "bert_doc = TransformerDocumentEmbeddings('bert-base-uncased')\n",
    "bert_doc.embed(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-6.4245e-02,  3.5365e-01, -2.4962e-01, -5.3912e-01, -1.9917e-01,\n",
       "        -2.7712e-01,  1.6942e-01,  1.0867e-01, -4.0799e-01, -1.4945e-01,\n",
       "         4.4678e-02, -1.9687e-01, -3.0574e-01,  9.6215e-02,  2.8053e-01,\n",
       "         9.0462e-01, -3.0250e-01,  1.7854e-01, -1.6856e-01, -1.7227e-01,\n",
       "        -1.8270e-01,  4.5121e-01,  3.6128e-01,  1.2310e-01, -1.2047e-02,\n",
       "        -5.0255e-01,  4.2645e-01, -3.7184e-02, -1.7052e-01, -4.8920e-02,\n",
       "         1.0503e-01,  3.3456e-01, -3.8483e-02, -4.7287e-01, -7.4431e-02,\n",
       "         2.3990e-01, -8.3232e-02,  1.7974e-01,  4.8939e-01,  9.3019e-03,\n",
       "        -4.2722e-01,  2.6337e-01, -1.3216e-01, -1.3950e-01, -1.1859e-01,\n",
       "         4.5834e-02, -3.9031e+00, -1.0726e-01,  5.1103e-02, -9.9710e-02,\n",
       "         1.5913e-01, -9.5331e-02,  2.1513e-01,  4.7707e-01,  2.1226e-01,\n",
       "         4.3450e-01, -4.5154e-01,  1.8344e-01, -1.0171e-01, -3.6373e-01,\n",
       "         8.9560e-01,  3.5931e-01, -2.8519e-01,  2.9427e-02,  3.3968e-01,\n",
       "         4.1062e-01, -2.1050e-02,  2.7802e-02,  2.8888e-01,  4.5761e-01,\n",
       "        -3.8254e-01, -1.9211e-01, -2.6551e-01,  1.7064e-01, -2.0920e-01,\n",
       "        -3.3608e-01, -1.0044e-01, -2.9611e-01, -5.6573e-01,  4.3841e-01,\n",
       "        -8.9238e-02, -1.4644e-02,  2.2009e-01,  2.5755e-01, -2.6711e-01,\n",
       "        -1.4440e-01, -2.8944e-01,  2.2546e-01,  6.0303e-01,  1.0250e-01,\n",
       "        -2.0606e-01, -1.5392e-01, -3.2990e-03,  7.7604e-02,  1.8428e-01,\n",
       "         1.6114e-01,  2.5668e-01, -8.3616e-02,  2.7078e-01, -5.4945e-02,\n",
       "         4.9633e-01, -2.7734e-01,  4.1486e-01, -5.3347e-01, -7.8131e-01,\n",
       "         1.7047e-01,  2.8687e-01,  4.9673e-02,  4.6922e-01, -1.6500e+00,\n",
       "         4.6991e-01, -1.5328e-01,  2.7729e-01, -1.0157e+00,  1.5022e-01,\n",
       "        -1.7466e-01,  4.1232e-02, -4.4218e-01, -5.0299e-01,  1.1937e-01,\n",
       "         1.8505e-01,  1.7673e-01,  2.1935e-01, -1.3052e+00, -3.4139e-01,\n",
       "        -3.8844e-02, -2.4191e-01,  2.5906e-01,  1.2551e-01, -1.7339e-02,\n",
       "         4.4672e-01,  3.3219e-01, -2.1076e-01, -2.6042e-01,  2.1486e-01,\n",
       "         1.6804e-01, -1.6398e-01,  1.1147e-01, -1.4731e-01,  1.0370e-01,\n",
       "        -2.0025e-01, -1.7985e-01, -2.5822e+00, -9.3411e-02,  5.8352e-01,\n",
       "         4.2851e-01,  1.3719e-01, -4.4423e-01, -2.4911e-03,  6.3892e-01,\n",
       "         4.1029e-01, -3.2187e-01,  5.8879e-02, -2.1344e-01, -4.0473e-02,\n",
       "        -4.0414e-01,  3.8056e-01,  1.7888e-01, -4.1893e-01,  1.6880e-01,\n",
       "        -4.6561e-02, -3.3560e-01,  1.3599e-02,  2.4541e-01, -1.9910e-01,\n",
       "         5.3978e-01,  5.7401e-01,  5.7995e-01, -3.8400e-02, -4.4134e-02,\n",
       "         3.8312e-01, -1.6357e-02,  7.0812e-01, -1.6653e-01, -4.0108e-01,\n",
       "         8.2581e-02,  4.2481e-01, -1.2611e-01,  4.2772e-01, -7.9697e-02,\n",
       "        -7.4779e-01,  7.4372e-02,  2.0130e-01,  8.0436e-01,  8.2707e-01,\n",
       "         3.8594e-01, -2.6690e-02, -2.8838e-01,  2.1367e-01,  2.3883e-01,\n",
       "         1.4772e-01, -1.7737e-01, -1.6458e-01, -4.7632e-01, -9.9866e-02,\n",
       "        -7.7609e-02,  3.6381e-01, -5.1024e-01, -3.6723e-01, -2.4667e-01,\n",
       "        -2.0106e-01,  1.8970e-01,  1.2936e-01, -7.0922e-01,  1.8065e-01,\n",
       "         4.2441e+00,  3.2519e-01, -3.5622e-01,  7.9890e-02,  4.0457e-01,\n",
       "        -5.2027e-01, -4.0209e-01,  1.1543e-01,  5.4013e-01, -8.5186e-01,\n",
       "         5.7481e-01,  3.0116e-01,  3.0603e-01,  3.0002e-02, -3.7785e-01,\n",
       "        -2.3895e-01,  1.0855e-01, -7.3347e-02, -2.2568e-02, -1.8636e-01,\n",
       "         5.5137e-01, -2.5551e-01,  5.6109e-01,  8.9961e-02, -1.4174e+00,\n",
       "         1.9548e-01,  1.8128e-01,  5.7270e-01,  3.7286e-01,  2.1191e-02,\n",
       "         2.0575e-01, -2.7981e-01, -4.0404e-02, -5.4553e-02, -2.4785e-01,\n",
       "        -2.4466e-01, -4.0419e-03,  8.9292e-02,  5.5506e-01, -1.7438e-01,\n",
       "         4.4969e-01,  3.9400e-01, -5.3704e-01,  6.6374e-01,  3.8592e-01,\n",
       "         6.9357e-01,  6.0498e-01, -2.1726e-01, -1.0124e-01, -2.2627e-01,\n",
       "         5.7905e-01,  4.2297e-01, -1.2389e-01,  2.0031e-01,  4.5129e-01,\n",
       "         1.7262e-01,  1.0963e-02,  3.6436e-01,  5.7858e-01, -5.3210e-01,\n",
       "        -1.4754e-02,  6.1678e-01, -3.0254e-01, -3.7331e-01,  4.2933e-02,\n",
       "        -2.8411e-01,  2.5481e-01, -3.1135e-01, -2.4889e+00,  1.2208e-01,\n",
       "        -8.3915e-02, -1.0063e-01,  1.6028e-01,  9.8307e-02, -1.5816e-01,\n",
       "        -3.4994e-02, -1.4911e-01, -8.0463e-01,  5.4030e-01,  7.1562e-02,\n",
       "         1.7398e-01,  1.4491e-01, -7.0773e-01,  3.9191e-01,  1.0117e-01,\n",
       "        -2.8370e-01,  2.8936e-01,  1.5133e-01,  5.0763e-02, -2.4023e-01,\n",
       "         4.4110e-01, -3.9492e-02,  2.6435e-01,  3.1356e-01, -1.8297e-01,\n",
       "        -2.4273e-01, -1.7542e-01,  1.4501e-01,  3.0055e-01, -2.8845e-01,\n",
       "        -1.5901e-01,  1.7493e-01, -2.9559e-01, -3.4490e+00,  1.3099e-01,\n",
       "        -3.8233e-01, -1.0336e-01,  6.8291e-01,  6.9417e-01,  6.1574e-01,\n",
       "        -8.2423e-02, -1.2499e-01, -1.2243e-01,  2.8974e-01, -4.1564e-01,\n",
       "         6.8799e-02,  6.7222e-01,  5.1716e-01, -3.0252e-02,  3.6390e-01,\n",
       "        -5.9201e-01,  2.2436e-01, -2.7201e-01, -8.8383e-02, -5.5831e-01,\n",
       "         1.0460e-01, -3.3686e-01,  4.6233e-01,  6.3347e-01, -5.7720e-01,\n",
       "        -4.1045e-01, -6.3033e-02, -2.6181e-01,  2.5220e-01, -5.7010e-01,\n",
       "        -4.4242e-02, -2.5702e-03, -2.6010e-01, -2.0902e-01,  2.1671e-01,\n",
       "        -1.5338e-01,  2.3135e-02,  8.3758e-02,  2.4785e-01,  8.1829e-01,\n",
       "         4.6562e-01,  1.6161e-01,  1.5305e-01, -2.2000e-01, -2.5913e-01,\n",
       "        -3.0355e-01,  2.0841e-01, -8.6139e-02, -1.0182e-01,  7.8481e-01,\n",
       "         1.0315e+00, -5.8092e-01, -2.5592e-01,  1.4235e-01,  2.2627e-01,\n",
       "        -3.8135e-01, -1.7177e-01,  6.8986e-01,  9.5218e-01, -4.2518e-01,\n",
       "         2.9856e-01, -7.7871e-01,  4.6342e-01,  3.8082e-01,  3.3461e-01,\n",
       "        -3.6359e-01,  1.3449e-01,  6.0971e-02, -6.3682e-01,  6.5791e-01,\n",
       "        -3.4900e-01, -1.6919e+00, -1.1290e-01, -8.9330e-03,  7.0164e-01,\n",
       "        -2.4298e-01, -5.0134e-01, -9.4679e-02,  9.2998e-02, -6.9514e-01,\n",
       "         1.6732e-01,  1.0738e-01, -3.5135e-01, -4.1886e-01, -5.6835e-02,\n",
       "         1.8974e-01, -6.7744e-01, -5.0675e-02, -2.8237e-01, -9.4652e-02,\n",
       "         2.3333e-01,  2.1025e-01, -1.3943e-01,  4.7849e-01, -3.5598e-02,\n",
       "        -5.0288e-02,  2.2644e-01,  1.7870e-01, -8.6450e-02, -4.6139e-01,\n",
       "        -9.7895e-02,  9.5696e-03, -6.0332e-01, -1.0339e-01, -6.9934e-01,\n",
       "         8.6564e-01,  2.4269e-01,  1.2407e-01, -4.4824e-01,  6.2005e-02,\n",
       "         1.1447e-01, -4.3742e-02,  1.5308e+00, -4.7716e-01,  4.9675e-01,\n",
       "         8.2663e-01,  4.4054e-01, -8.6616e-02,  2.1151e-01, -1.6980e-02,\n",
       "         3.6449e-02,  5.2057e-02, -4.3879e-01, -2.0526e-01, -1.0826e-01,\n",
       "        -2.0655e-01,  5.4324e-02, -2.3006e-01, -2.3910e-01,  3.5886e-01,\n",
       "        -3.6865e-01, -4.1345e-01,  3.2852e-01, -2.1309e-01, -7.7967e-01,\n",
       "        -4.0635e-01,  4.1312e-01,  4.5644e-01,  5.6297e-01, -4.0113e-02,\n",
       "        -7.8325e-02,  3.5196e-01, -4.0508e-01,  4.9119e-01,  2.8871e-01,\n",
       "         1.2004e-02,  2.1122e-02,  5.8201e-01,  7.2554e-02, -1.2406e-01,\n",
       "        -2.2747e-01, -3.6875e-01,  1.2822e-01, -1.3220e-02, -2.5432e-01,\n",
       "        -5.4382e-01, -1.6104e-01, -4.9271e-01, -2.7111e-05, -1.6866e-01,\n",
       "        -2.0367e+00, -1.8150e-01,  7.2432e-01,  2.6644e-01,  2.3441e-01,\n",
       "         1.3906e-01, -7.0425e-03,  4.3320e-01, -3.9089e-01,  3.8913e-01,\n",
       "        -1.3505e-01,  1.8608e-01, -2.7714e-01,  7.7001e-02, -3.5178e-01,\n",
       "         3.1395e-01, -6.1126e-01, -1.9384e-01,  3.4077e-02,  7.6536e-02,\n",
       "        -6.3849e-01,  9.1557e-02,  6.1446e-01, -9.3747e-02,  2.7309e-02,\n",
       "        -2.9999e-01,  8.1415e-02,  2.3125e-01,  4.6290e-02,  1.2134e-01,\n",
       "         1.3592e-01, -5.6097e-01, -2.1794e-01, -6.8162e-01,  6.4019e-01,\n",
       "        -1.8568e-01, -1.6626e-01,  3.4131e-01,  1.8143e-01,  1.4511e-01,\n",
       "         4.1716e-02,  3.5290e-01,  2.6542e-01, -3.3044e-01,  6.5737e-01,\n",
       "        -4.8462e-01, -4.2369e-01,  5.4556e-01, -4.1632e-01, -7.4869e-02,\n",
       "        -2.7194e-01, -5.6730e-02, -1.9846e-01, -2.5328e-01, -8.4373e-01,\n",
       "        -2.5077e-01, -7.6245e-02,  1.3954e-01, -7.3810e-01, -7.1694e-02,\n",
       "         3.7180e-01, -6.6641e-01, -7.3949e-02,  1.0741e-02,  6.6874e-02,\n",
       "        -5.4790e-01, -2.3494e-01, -5.4339e-02, -2.7594e-01,  2.3640e-01,\n",
       "         9.0829e-02, -2.4924e-02, -6.5618e-01, -1.5407e-01,  6.5142e-03,\n",
       "        -1.8896e-01, -4.8536e-01,  4.1967e-01,  4.9990e-01, -2.3470e-01,\n",
       "         2.3999e-01,  2.2489e-02,  7.1898e-02,  3.7770e-01,  2.2666e-01,\n",
       "        -4.6463e-02,  7.9462e-02, -1.3600e-01,  2.6624e-01, -1.0557e-01,\n",
       "        -5.1542e-01, -2.1026e-01, -3.6529e-01,  3.7681e-01,  4.2558e-01,\n",
       "        -2.0673e-01,  1.7708e-01, -9.5000e-01,  3.5931e-01, -6.9485e-01,\n",
       "         2.6662e-01, -3.8393e-01, -1.4876e-01,  3.5606e-01, -1.9180e-01,\n",
       "         5.2491e-02,  6.9162e-01,  1.2743e-01, -2.9465e-01, -6.5748e-01,\n",
       "        -1.3319e-01, -1.6700e-01,  3.4043e-01,  1.0655e-01,  1.4711e-01,\n",
       "        -5.7975e-01, -1.9442e-01, -5.5820e-01,  1.5113e+00,  2.5854e-01,\n",
       "        -7.7181e-02,  1.9113e-01, -2.0155e-01,  5.2040e-02, -6.6809e-02,\n",
       "         7.7022e-01,  4.3337e-01,  2.1713e-01, -1.9559e-01,  1.1622e-01,\n",
       "         6.9701e-02,  1.9887e-01,  4.4560e-01, -2.0377e-01,  9.3891e-02,\n",
       "        -3.8858e-01, -1.0287e+00, -8.7257e-02,  3.0796e-01,  1.1579e-01,\n",
       "         2.2186e-01,  2.4396e-02, -2.2758e-01,  7.7906e-01,  8.9298e-02,\n",
       "        -5.7492e-01, -2.8990e-01,  5.5056e-01, -3.1180e-01,  5.1760e-01,\n",
       "        -2.1880e-01,  3.5772e-01, -9.8653e-01,  3.4400e-02, -1.5934e-01,\n",
       "        -2.6761e-01,  4.0133e-02,  7.4575e-02,  3.0935e-01,  2.3990e-02,\n",
       "        -2.4070e-01,  1.9046e-01, -3.6814e-01,  3.0127e-01,  1.0484e-01,\n",
       "         3.0938e-02,  3.2499e-01,  4.0934e-01, -1.7192e-01,  5.4189e-01,\n",
       "         3.5965e-01,  1.4977e-01, -2.0299e-01,  2.5858e-02, -2.3124e-01,\n",
       "        -5.2681e-01, -5.0654e-02, -4.0077e-01,  3.5185e-02,  6.1388e-01,\n",
       "         1.7713e-01, -5.8146e-01,  1.2276e-01,  2.9924e-02, -5.3960e-02,\n",
       "        -4.8978e-01,  8.5426e-02, -1.6308e-01,  2.2882e-02,  2.6261e-01,\n",
       "         2.0515e-01,  3.4557e-01,  2.7707e-01,  3.1900e-01,  6.0790e-01,\n",
       "        -5.5657e-01, -4.6023e-01, -1.5954e+00,  1.5275e-01,  3.5691e-01,\n",
       "         4.9637e-01, -1.7432e-01,  1.8391e-01,  5.2575e-01,  1.6172e-01,\n",
       "        -1.6158e-01,  1.2285e-01,  1.7329e-01,  1.0693e+00,  4.1782e-01,\n",
       "         2.6315e-01, -2.6413e-02, -3.4535e-01,  3.1312e-01,  3.1305e-03,\n",
       "        -1.3553e-01,  9.2501e-02,  1.8213e-01,  1.0432e-01, -5.5064e-02,\n",
       "        -5.2151e-01, -5.9766e-01, -3.0186e-01,  1.1255e-02,  6.2411e-01,\n",
       "        -2.4277e-01,  2.0318e-01, -1.6011e-01,  8.6397e-01, -2.2691e-02,\n",
       "        -4.4046e-01, -6.2857e-02, -3.9069e-01,  1.0784e-01,  1.1981e-01,\n",
       "         4.9630e-01, -3.8374e-03, -4.2250e-01,  1.0451e+00, -2.6349e-01,\n",
       "        -1.4386e-01,  2.8545e-01,  2.6442e-01,  5.0848e-01, -3.4544e-01,\n",
       "         2.0938e-01,  3.9765e-01, -2.8668e-01, -2.9289e-01,  1.0621e-01,\n",
       "         1.2800e-01,  4.2874e-01, -7.5396e-02,  2.4798e-01,  8.1016e-02,\n",
       "        -2.1906e-02, -2.7718e-01, -7.3759e-02, -5.9758e-02, -1.1127e-01,\n",
       "        -1.2804e-01,  2.8441e-01, -1.5104e-01, -2.0100e-01,  7.6934e-01,\n",
       "         3.0968e-01, -5.9233e-01, -3.4167e-01, -1.5913e-01, -5.3265e-02,\n",
       "         2.1583e-01, -3.5234e-01, -1.2197e-01,  3.0294e-01,  7.3882e-01,\n",
       "         1.3297e-01, -3.4018e-01, -8.1062e-02, -2.5973e-01,  1.7507e-01,\n",
       "         2.3974e-02,  4.1746e-01, -6.2070e+00,  1.1619e-01, -5.7523e-01,\n",
       "        -2.6148e-01, -3.7661e-01, -2.8330e-01,  1.3297e-01,  3.6957e-01,\n",
       "         7.4661e-02, -3.4777e-01,  1.5740e-01,  3.4407e-01, -5.0272e-01,\n",
       "         1.7432e-01,  7.9398e-01,  7.3562e-01], device='cuda:0',\n",
       "       grad_fn=<CatBackward>)"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents[0].embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([], device='cuda:0')"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents[0].tokens[31].embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embeddings(embeddings, sentence):\n",
    "    sent = Sentence(sentence)\n",
    "    embeddings.embed(sent)\n",
    "    if len(sent.embedding):\n",
    "        return sent.embedding.float()\n",
    "    else:\n",
    "        return torch.stack([token.embedding for token in sent.tokens]).float()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-6.4245e-02,  3.5365e-01, -2.4962e-01, -5.3912e-01, -1.9917e-01,\n",
       "        -2.7712e-01,  1.6942e-01,  1.0867e-01, -4.0799e-01, -1.4945e-01,\n",
       "         4.4678e-02, -1.9687e-01, -3.0574e-01,  9.6215e-02,  2.8053e-01,\n",
       "         9.0462e-01, -3.0250e-01,  1.7854e-01, -1.6856e-01, -1.7227e-01,\n",
       "        -1.8270e-01,  4.5121e-01,  3.6128e-01,  1.2310e-01, -1.2047e-02,\n",
       "        -5.0255e-01,  4.2645e-01, -3.7184e-02, -1.7052e-01, -4.8920e-02,\n",
       "         1.0503e-01,  3.3456e-01, -3.8483e-02, -4.7287e-01, -7.4431e-02,\n",
       "         2.3990e-01, -8.3232e-02,  1.7974e-01,  4.8939e-01,  9.3019e-03,\n",
       "        -4.2722e-01,  2.6337e-01, -1.3216e-01, -1.3950e-01, -1.1859e-01,\n",
       "         4.5834e-02, -3.9031e+00, -1.0726e-01,  5.1103e-02, -9.9710e-02,\n",
       "         1.5913e-01, -9.5331e-02,  2.1513e-01,  4.7707e-01,  2.1226e-01,\n",
       "         4.3450e-01, -4.5154e-01,  1.8344e-01, -1.0171e-01, -3.6373e-01,\n",
       "         8.9560e-01,  3.5931e-01, -2.8519e-01,  2.9427e-02,  3.3968e-01,\n",
       "         4.1062e-01, -2.1050e-02,  2.7802e-02,  2.8888e-01,  4.5761e-01,\n",
       "        -3.8254e-01, -1.9211e-01, -2.6551e-01,  1.7064e-01, -2.0920e-01,\n",
       "        -3.3608e-01, -1.0044e-01, -2.9611e-01, -5.6573e-01,  4.3841e-01,\n",
       "        -8.9238e-02, -1.4644e-02,  2.2009e-01,  2.5755e-01, -2.6711e-01,\n",
       "        -1.4440e-01, -2.8944e-01,  2.2546e-01,  6.0303e-01,  1.0250e-01,\n",
       "        -2.0606e-01, -1.5392e-01, -3.2990e-03,  7.7604e-02,  1.8428e-01,\n",
       "         1.6114e-01,  2.5668e-01, -8.3616e-02,  2.7078e-01, -5.4945e-02,\n",
       "         4.9633e-01, -2.7734e-01,  4.1486e-01, -5.3347e-01, -7.8131e-01,\n",
       "         1.7047e-01,  2.8687e-01,  4.9673e-02,  4.6922e-01, -1.6500e+00,\n",
       "         4.6991e-01, -1.5328e-01,  2.7729e-01, -1.0157e+00,  1.5022e-01,\n",
       "        -1.7466e-01,  4.1232e-02, -4.4218e-01, -5.0299e-01,  1.1937e-01,\n",
       "         1.8505e-01,  1.7673e-01,  2.1935e-01, -1.3052e+00, -3.4139e-01,\n",
       "        -3.8844e-02, -2.4191e-01,  2.5906e-01,  1.2551e-01, -1.7339e-02,\n",
       "         4.4672e-01,  3.3219e-01, -2.1076e-01, -2.6042e-01,  2.1486e-01,\n",
       "         1.6804e-01, -1.6398e-01,  1.1147e-01, -1.4731e-01,  1.0370e-01,\n",
       "        -2.0025e-01, -1.7985e-01, -2.5822e+00, -9.3411e-02,  5.8352e-01,\n",
       "         4.2851e-01,  1.3719e-01, -4.4423e-01, -2.4911e-03,  6.3892e-01,\n",
       "         4.1029e-01, -3.2187e-01,  5.8879e-02, -2.1344e-01, -4.0473e-02,\n",
       "        -4.0414e-01,  3.8056e-01,  1.7888e-01, -4.1893e-01,  1.6880e-01,\n",
       "        -4.6561e-02, -3.3560e-01,  1.3599e-02,  2.4541e-01, -1.9910e-01,\n",
       "         5.3978e-01,  5.7401e-01,  5.7995e-01, -3.8400e-02, -4.4134e-02,\n",
       "         3.8312e-01, -1.6357e-02,  7.0812e-01, -1.6653e-01, -4.0108e-01,\n",
       "         8.2581e-02,  4.2481e-01, -1.2611e-01,  4.2772e-01, -7.9697e-02,\n",
       "        -7.4779e-01,  7.4372e-02,  2.0130e-01,  8.0436e-01,  8.2707e-01,\n",
       "         3.8594e-01, -2.6690e-02, -2.8838e-01,  2.1367e-01,  2.3883e-01,\n",
       "         1.4772e-01, -1.7737e-01, -1.6458e-01, -4.7632e-01, -9.9866e-02,\n",
       "        -7.7609e-02,  3.6381e-01, -5.1024e-01, -3.6723e-01, -2.4667e-01,\n",
       "        -2.0106e-01,  1.8970e-01,  1.2936e-01, -7.0922e-01,  1.8065e-01,\n",
       "         4.2441e+00,  3.2519e-01, -3.5622e-01,  7.9890e-02,  4.0457e-01,\n",
       "        -5.2027e-01, -4.0209e-01,  1.1543e-01,  5.4013e-01, -8.5186e-01,\n",
       "         5.7481e-01,  3.0116e-01,  3.0603e-01,  3.0002e-02, -3.7785e-01,\n",
       "        -2.3895e-01,  1.0855e-01, -7.3347e-02, -2.2568e-02, -1.8636e-01,\n",
       "         5.5137e-01, -2.5551e-01,  5.6109e-01,  8.9961e-02, -1.4174e+00,\n",
       "         1.9548e-01,  1.8128e-01,  5.7270e-01,  3.7286e-01,  2.1191e-02,\n",
       "         2.0575e-01, -2.7981e-01, -4.0404e-02, -5.4553e-02, -2.4785e-01,\n",
       "        -2.4466e-01, -4.0419e-03,  8.9292e-02,  5.5506e-01, -1.7438e-01,\n",
       "         4.4969e-01,  3.9400e-01, -5.3704e-01,  6.6374e-01,  3.8592e-01,\n",
       "         6.9357e-01,  6.0498e-01, -2.1726e-01, -1.0124e-01, -2.2627e-01,\n",
       "         5.7905e-01,  4.2297e-01, -1.2389e-01,  2.0031e-01,  4.5129e-01,\n",
       "         1.7262e-01,  1.0963e-02,  3.6436e-01,  5.7858e-01, -5.3210e-01,\n",
       "        -1.4754e-02,  6.1678e-01, -3.0254e-01, -3.7331e-01,  4.2933e-02,\n",
       "        -2.8411e-01,  2.5481e-01, -3.1135e-01, -2.4889e+00,  1.2208e-01,\n",
       "        -8.3915e-02, -1.0063e-01,  1.6028e-01,  9.8307e-02, -1.5816e-01,\n",
       "        -3.4994e-02, -1.4911e-01, -8.0463e-01,  5.4030e-01,  7.1562e-02,\n",
       "         1.7398e-01,  1.4491e-01, -7.0773e-01,  3.9191e-01,  1.0117e-01,\n",
       "        -2.8370e-01,  2.8936e-01,  1.5133e-01,  5.0763e-02, -2.4023e-01,\n",
       "         4.4110e-01, -3.9492e-02,  2.6435e-01,  3.1356e-01, -1.8297e-01,\n",
       "        -2.4273e-01, -1.7542e-01,  1.4501e-01,  3.0055e-01, -2.8845e-01,\n",
       "        -1.5901e-01,  1.7493e-01, -2.9559e-01, -3.4490e+00,  1.3099e-01,\n",
       "        -3.8233e-01, -1.0336e-01,  6.8291e-01,  6.9417e-01,  6.1574e-01,\n",
       "        -8.2423e-02, -1.2499e-01, -1.2243e-01,  2.8974e-01, -4.1564e-01,\n",
       "         6.8799e-02,  6.7222e-01,  5.1716e-01, -3.0252e-02,  3.6390e-01,\n",
       "        -5.9201e-01,  2.2436e-01, -2.7201e-01, -8.8383e-02, -5.5831e-01,\n",
       "         1.0460e-01, -3.3686e-01,  4.6233e-01,  6.3347e-01, -5.7720e-01,\n",
       "        -4.1045e-01, -6.3033e-02, -2.6181e-01,  2.5220e-01, -5.7010e-01,\n",
       "        -4.4242e-02, -2.5702e-03, -2.6010e-01, -2.0902e-01,  2.1671e-01,\n",
       "        -1.5338e-01,  2.3135e-02,  8.3758e-02,  2.4785e-01,  8.1829e-01,\n",
       "         4.6562e-01,  1.6161e-01,  1.5305e-01, -2.2000e-01, -2.5913e-01,\n",
       "        -3.0355e-01,  2.0841e-01, -8.6139e-02, -1.0182e-01,  7.8481e-01,\n",
       "         1.0315e+00, -5.8092e-01, -2.5592e-01,  1.4235e-01,  2.2627e-01,\n",
       "        -3.8135e-01, -1.7177e-01,  6.8986e-01,  9.5218e-01, -4.2518e-01,\n",
       "         2.9856e-01, -7.7871e-01,  4.6342e-01,  3.8082e-01,  3.3461e-01,\n",
       "        -3.6359e-01,  1.3449e-01,  6.0971e-02, -6.3682e-01,  6.5791e-01,\n",
       "        -3.4900e-01, -1.6919e+00, -1.1290e-01, -8.9330e-03,  7.0164e-01,\n",
       "        -2.4298e-01, -5.0134e-01, -9.4679e-02,  9.2998e-02, -6.9514e-01,\n",
       "         1.6732e-01,  1.0738e-01, -3.5135e-01, -4.1886e-01, -5.6835e-02,\n",
       "         1.8974e-01, -6.7744e-01, -5.0675e-02, -2.8237e-01, -9.4652e-02,\n",
       "         2.3333e-01,  2.1025e-01, -1.3943e-01,  4.7849e-01, -3.5598e-02,\n",
       "        -5.0288e-02,  2.2644e-01,  1.7870e-01, -8.6450e-02, -4.6139e-01,\n",
       "        -9.7895e-02,  9.5696e-03, -6.0332e-01, -1.0339e-01, -6.9934e-01,\n",
       "         8.6564e-01,  2.4269e-01,  1.2407e-01, -4.4824e-01,  6.2005e-02,\n",
       "         1.1447e-01, -4.3742e-02,  1.5308e+00, -4.7716e-01,  4.9675e-01,\n",
       "         8.2663e-01,  4.4054e-01, -8.6616e-02,  2.1151e-01, -1.6980e-02,\n",
       "         3.6449e-02,  5.2057e-02, -4.3879e-01, -2.0526e-01, -1.0826e-01,\n",
       "        -2.0655e-01,  5.4324e-02, -2.3006e-01, -2.3910e-01,  3.5886e-01,\n",
       "        -3.6865e-01, -4.1345e-01,  3.2852e-01, -2.1309e-01, -7.7967e-01,\n",
       "        -4.0635e-01,  4.1312e-01,  4.5644e-01,  5.6297e-01, -4.0113e-02,\n",
       "        -7.8325e-02,  3.5196e-01, -4.0508e-01,  4.9119e-01,  2.8871e-01,\n",
       "         1.2004e-02,  2.1122e-02,  5.8201e-01,  7.2554e-02, -1.2406e-01,\n",
       "        -2.2747e-01, -3.6875e-01,  1.2822e-01, -1.3220e-02, -2.5432e-01,\n",
       "        -5.4382e-01, -1.6104e-01, -4.9271e-01, -2.7111e-05, -1.6866e-01,\n",
       "        -2.0367e+00, -1.8150e-01,  7.2432e-01,  2.6644e-01,  2.3441e-01,\n",
       "         1.3906e-01, -7.0425e-03,  4.3320e-01, -3.9089e-01,  3.8913e-01,\n",
       "        -1.3505e-01,  1.8608e-01, -2.7714e-01,  7.7001e-02, -3.5178e-01,\n",
       "         3.1395e-01, -6.1126e-01, -1.9384e-01,  3.4077e-02,  7.6536e-02,\n",
       "        -6.3849e-01,  9.1557e-02,  6.1446e-01, -9.3747e-02,  2.7309e-02,\n",
       "        -2.9999e-01,  8.1415e-02,  2.3125e-01,  4.6290e-02,  1.2134e-01,\n",
       "         1.3592e-01, -5.6097e-01, -2.1794e-01, -6.8162e-01,  6.4019e-01,\n",
       "        -1.8568e-01, -1.6626e-01,  3.4131e-01,  1.8143e-01,  1.4511e-01,\n",
       "         4.1716e-02,  3.5290e-01,  2.6542e-01, -3.3044e-01,  6.5737e-01,\n",
       "        -4.8462e-01, -4.2369e-01,  5.4556e-01, -4.1632e-01, -7.4869e-02,\n",
       "        -2.7194e-01, -5.6730e-02, -1.9846e-01, -2.5328e-01, -8.4373e-01,\n",
       "        -2.5077e-01, -7.6245e-02,  1.3954e-01, -7.3810e-01, -7.1694e-02,\n",
       "         3.7180e-01, -6.6641e-01, -7.3949e-02,  1.0741e-02,  6.6874e-02,\n",
       "        -5.4790e-01, -2.3494e-01, -5.4339e-02, -2.7594e-01,  2.3640e-01,\n",
       "         9.0829e-02, -2.4924e-02, -6.5618e-01, -1.5407e-01,  6.5142e-03,\n",
       "        -1.8896e-01, -4.8536e-01,  4.1967e-01,  4.9990e-01, -2.3470e-01,\n",
       "         2.3999e-01,  2.2489e-02,  7.1898e-02,  3.7770e-01,  2.2666e-01,\n",
       "        -4.6463e-02,  7.9462e-02, -1.3600e-01,  2.6624e-01, -1.0557e-01,\n",
       "        -5.1542e-01, -2.1026e-01, -3.6529e-01,  3.7681e-01,  4.2558e-01,\n",
       "        -2.0673e-01,  1.7708e-01, -9.5000e-01,  3.5931e-01, -6.9485e-01,\n",
       "         2.6662e-01, -3.8393e-01, -1.4876e-01,  3.5606e-01, -1.9180e-01,\n",
       "         5.2491e-02,  6.9162e-01,  1.2743e-01, -2.9465e-01, -6.5748e-01,\n",
       "        -1.3319e-01, -1.6700e-01,  3.4043e-01,  1.0655e-01,  1.4711e-01,\n",
       "        -5.7975e-01, -1.9442e-01, -5.5820e-01,  1.5113e+00,  2.5854e-01,\n",
       "        -7.7181e-02,  1.9113e-01, -2.0155e-01,  5.2040e-02, -6.6809e-02,\n",
       "         7.7022e-01,  4.3337e-01,  2.1713e-01, -1.9559e-01,  1.1622e-01,\n",
       "         6.9701e-02,  1.9887e-01,  4.4560e-01, -2.0377e-01,  9.3891e-02,\n",
       "        -3.8858e-01, -1.0287e+00, -8.7257e-02,  3.0796e-01,  1.1579e-01,\n",
       "         2.2186e-01,  2.4396e-02, -2.2758e-01,  7.7906e-01,  8.9298e-02,\n",
       "        -5.7492e-01, -2.8990e-01,  5.5056e-01, -3.1180e-01,  5.1760e-01,\n",
       "        -2.1880e-01,  3.5772e-01, -9.8653e-01,  3.4400e-02, -1.5934e-01,\n",
       "        -2.6761e-01,  4.0133e-02,  7.4575e-02,  3.0935e-01,  2.3990e-02,\n",
       "        -2.4070e-01,  1.9046e-01, -3.6814e-01,  3.0127e-01,  1.0484e-01,\n",
       "         3.0938e-02,  3.2499e-01,  4.0934e-01, -1.7192e-01,  5.4189e-01,\n",
       "         3.5965e-01,  1.4977e-01, -2.0299e-01,  2.5858e-02, -2.3124e-01,\n",
       "        -5.2681e-01, -5.0654e-02, -4.0077e-01,  3.5185e-02,  6.1388e-01,\n",
       "         1.7713e-01, -5.8146e-01,  1.2276e-01,  2.9924e-02, -5.3960e-02,\n",
       "        -4.8978e-01,  8.5426e-02, -1.6308e-01,  2.2882e-02,  2.6261e-01,\n",
       "         2.0515e-01,  3.4557e-01,  2.7707e-01,  3.1900e-01,  6.0790e-01,\n",
       "        -5.5657e-01, -4.6023e-01, -1.5954e+00,  1.5275e-01,  3.5691e-01,\n",
       "         4.9637e-01, -1.7432e-01,  1.8391e-01,  5.2575e-01,  1.6172e-01,\n",
       "        -1.6158e-01,  1.2285e-01,  1.7329e-01,  1.0693e+00,  4.1782e-01,\n",
       "         2.6315e-01, -2.6413e-02, -3.4535e-01,  3.1312e-01,  3.1305e-03,\n",
       "        -1.3553e-01,  9.2501e-02,  1.8213e-01,  1.0432e-01, -5.5064e-02,\n",
       "        -5.2151e-01, -5.9766e-01, -3.0186e-01,  1.1255e-02,  6.2411e-01,\n",
       "        -2.4277e-01,  2.0318e-01, -1.6011e-01,  8.6397e-01, -2.2691e-02,\n",
       "        -4.4046e-01, -6.2857e-02, -3.9069e-01,  1.0784e-01,  1.1981e-01,\n",
       "         4.9630e-01, -3.8374e-03, -4.2250e-01,  1.0451e+00, -2.6349e-01,\n",
       "        -1.4386e-01,  2.8545e-01,  2.6442e-01,  5.0848e-01, -3.4544e-01,\n",
       "         2.0938e-01,  3.9765e-01, -2.8668e-01, -2.9289e-01,  1.0621e-01,\n",
       "         1.2800e-01,  4.2874e-01, -7.5396e-02,  2.4798e-01,  8.1016e-02,\n",
       "        -2.1906e-02, -2.7718e-01, -7.3759e-02, -5.9758e-02, -1.1127e-01,\n",
       "        -1.2804e-01,  2.8441e-01, -1.5104e-01, -2.0100e-01,  7.6934e-01,\n",
       "         3.0968e-01, -5.9233e-01, -3.4167e-01, -1.5913e-01, -5.3265e-02,\n",
       "         2.1583e-01, -3.5234e-01, -1.2197e-01,  3.0294e-01,  7.3882e-01,\n",
       "         1.3297e-01, -3.4018e-01, -8.1062e-02, -2.5973e-01,  1.7507e-01,\n",
       "         2.3974e-02,  4.1746e-01, -6.2070e+00,  1.1619e-01, -5.7523e-01,\n",
       "        -2.6148e-01, -3.7661e-01, -2.8330e-01,  1.3297e-01,  3.6957e-01,\n",
       "         7.4661e-02, -3.4777e-01,  1.5740e-01,  3.4407e-01, -5.0272e-01,\n",
       "         1.7432e-01,  7.9398e-01,  7.3562e-01], device='cuda:0',\n",
       "       grad_fn=<CatBackward>)"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_embeddings(bert_doc, watch1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model III - Preprocessing Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aac06999ef7f4ed398eb3849201585b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=3081.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80060133f2e44a45b5d444392d156fea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=771.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "train_dataset_doc = train_dataset.map(lambda row: {'embeddings': get_embeddings(bert_doc, row['sentence'])})\n",
    "test_dataset_doc = test_dataset.map(lambda row: {'embeddings': get_embeddings(bert_doc, row['sentence'])})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset_doc.set_format(type='torch', columns=['embeddings', 'labels'])\n",
    "test_dataset_doc.set_format(type='torch', columns=['embeddings', 'labels'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.2932,  0.2595, -0.1252,  ..., -0.2407,  0.2998,  0.1157],\n",
       "        [ 0.4934,  0.0129, -0.1991,  ...,  0.6354,  0.6320,  0.7036],\n",
       "        [-0.6256, -0.3536, -0.4682,  ...,  0.5507,  0.2467,  0.6108],\n",
       "        ...,\n",
       "        [-0.5786,  0.0274, -0.1081,  ..., -0.0108,  0.0329,  0.9563],\n",
       "        [ 0.1244,  0.3181,  0.0352,  ...,  0.3957,  0.6648,  0.9231],\n",
       "        [ 0.2124,  0.6195, -0.2281,  ..., -0.0835,  0.4346,  0.6358]])"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset_doc['embeddings']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset_doc = TensorDataset(train_dataset_doc['embeddings'].float(), \n",
    "                                  train_dataset_doc['labels'].view(-1, 1).float())\n",
    "generator = torch.Generator()\n",
    "train_loader = DataLoader(train_dataset_doc, batch_size=32, shuffle=True, generator=generator)\n",
    "\n",
    "test_dataset_doc = TensorDataset(test_dataset_doc['embeddings'].float(), \n",
    "                                 test_dataset_doc['labels'].view(-1, 1).float())\n",
    "test_loader = DataLoader(test_dataset_doc, batch_size=32, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Configuration & Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(41)\n",
    "model = nn.Sequential(\n",
    "    # Classifier\n",
    "    nn.Linear(bert_doc.embedding_length, 3), \n",
    "    nn.ReLU(),\n",
    "    nn.Linear(3, 1)\n",
    ")\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "sbs_doc_emb = StepByStep(model, loss_fn, optimizer)\n",
    "sbs_doc_emb.set_loaders(train_loader, test_loader)\n",
    "sbs_doc_emb.train(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAEQCAYAAAC++cJdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABu0UlEQVR4nO3dd3gUVd/G8e9sSw+hJnQMCSVICy303oUg0hEFC0gRGygoiiiP2PCxUFQQX6kPVap0Qg8EBKWH0INAgEAC6Vvm/WNlw5IeEtJ+n+vay+yZ2dkzx8ly5+yZc5SoqCgVIYQQQgghighNXldACCGEEEKIJ0kCsBBCCCGEKFIkAAshhBBCiCJFArAQQgghhChSJAALIYQQQogiRQKwEEIIIYQoUiQACyGEEEKIIkUCsBBCCCGEKFIkABdyYWFheV2FfE3aJ33SPhmTNkqftE/6pH0yJm2UPmmf7JEALIQQQgghihQJwEIIIYQQokiRACyEEEIIIYoUCcBCCCGEEKJI0eV1BYQQQghR+JhMJmJjYx/7OI6OjkRHR+dAjQqnotw+Li4u6HTZi7ISgAs5VYX798HNLa9rIoQQoqgwmUzcv38fDw8PFEV5rGM5ODjg6OiYQzUrfIpq+6iqSlRUFG5ubtkKwTIEohCLjFR45x0fnn/eBYslr2sjhBCiqIiNjc2R8CtEWhRFwcPDI9vfMkgALqT27NHSooUre/Z4sGuXjm+/dcjrKgkhhChCJPyK3PY415gE4EJIVWH6dAeuX0/+3/uf/zgQEqLNw1oJIYQQQuQPEoALIUWBWbPiKVEiedyD2azw8svOREXlXb2EEEIIIfIDCcCFVLlyKrNmxduVhYdrePNNJ1Q1jyolhBBCFDEjR46kf//+WXpN9+7dGT9+fC7VSIDMAlGodeliYuDACJYs8bSVrV5toG1bEy++aMzDmgkhhBD5i4eHR7rbBw4cyOzZs7N83M8//xw1iz1PCxcuzPb0Xlkxbdo01q5dS3BwcK6/V34jAbiQGzPmKidPluLYseTxvxMmONGkiZkaNWRqCCGEEAIgNDTU9vPmzZsZO3asXdmjU40ZjUb0en2Gxy1WrFiW61K8ePEsv0ZkjQyBKOQMBpV58+JwcUn+6zM+XuGll5yJj0/nhUIIIUQR4unpaXs8CK0PnickJFC5cmVWrFhBjx498PLy4tdff+XOnTu8/PLL+Pn54eXlRUBAAAsXLrQ77qNDILp3784777zDJ598gre3Nz4+PkyaNAnLQ/OVPjoEonbt2nz11Ve8+eabVKxYET8/P77//nu79zl37hzdunXD09OThg0bsmXLFsqXL8+iRYuy3SYnT54kMDAQLy8vqlSpwsiRI+0W3Th58iQ9e/akYsWKVKhQgebNm7N7927A+gfCu+++S40aNShTpgy1atXi448/znZdcpr0ABcBPj4Wvv46npEjnW1lp05pmTTJkenTE/KwZkIIIYoKD4+s94RaZe91UVE5vzralClTmDp1Kj/88AN6vZ6EhATq1q3LG2+8gbu7Ozt37uStt96iYsWKtG7dOs3jLF++nBEjRrBlyxaOHz/OK6+8Qr169ejTp0+ar5k1axYTJ05k7NixbN26lffee4+AgADq1KmDxWLh+eefp0yZMmzdupWEhAQmTpxIYmJits81Li6OPn36UL9+fbZv387du3d54403GDNmDAsWLADg1Vdf5emnn2b79u3odDpOnjxp6yn/8ccf2bBhA7/88guVKlXi2rVrhIWFZbs+OU0CcBExcKCRoKAkli0z2Mp++cWB1q1N9OxpysOaCSGEEAXD8OHDCQwMtCsbO3as7eehQ4eye/duVqxYkW4Arl69Oh988AEAPj4+/Pbbb+zatSvdANyuXTuGDx8OwIgRI/jpp5/YtWsXderUISgoiLCwMFatWkW5cuUA+Oyzz+jcuXO2z3X58uXExsby008/4fbvcrLffvstPXr04MKFC3h7exMeHs6YMWOoVq0aAN7e3rbXh4eHU7VqVZo1a4aiKFSsWJEmTZpkuz45TYZAFCHTp8fj7W22K3v9dWeuXJHJyoUQQoiM1K9f3+652Wzm66+/plmzZjz11FOUL1+edevWcfXq1XSPU6tWLbvnXl5e3Lp1K9uvOXv2LGXLlrWFXwB/f380muzHvNDQUGrVqmULvwBNmjRBo9Fw5swZAEaNGsXYsWPp0aMHX3/9NWfPnrXtO2jQII4fP06DBg0YN24cmzdvthvmkdckABchbm4wb14cen3yeODoaIXhw50xSSewEEIIkS4XFxe75z/88AMzZsxg7NixrFmzhj179tC9e3eSkpLSPc6jN88pipLhTBHpvSars0xkRnrHfLAC28SJEzl48CDdu3cnJCSE5s2b24ZH1KtXj2PHjvHRRx9hsVgYOXIkvXr1yjchWIZAFDH16lmYPDmBSZOcbGUHDuj4/HMHJk3K/lghIYQQIj3ZHZObkJCQYgaG/CI4OJguXbowYMAAwBoaz507l62ZHx5H9erVuX79OtevX6ds2bIAHD169LHCZo0aNVi0aBH379+39QIfPHgQi8VC9erVbftVrVqVqlWr8tprr/H222+zYMEChgwZAoCbmxu9evWiV69eDBo0iA4dOnDhwgV8fHwe42xzhvQAF0GjRiXRsaP9PMDTpzuwe7cslSyEEEJklo+PD7t37yY4OJizZ88yfvx4rly58sTr0bZtW3x9fRk5ciTHjx/n0KFDfPDBB+h0OltvbVoSEhI4duyY3ePcuXP07dsXZ2dnXnvtNU6ePMm+fft466236NGjB97e3sTHxzNu3Dj27NnD5cuXOXz4MAcOHLCF4xkzZrBixQpCQ0O5cOECy5cvx93d3W6YRl6SHuAiSKOxLpXcooWWiAjr30CqqjBihDN798ZQsqQsFSeEEEJkZPz48Vy+fJm+ffvi6OjIoEGD6Nu3r22M7JOi0WhYuHAhr7/+Ou3bt6dSpUpMnTqVIUOGZNh7fvHiRVq1amVXVq9ePXbu3MnKlSuZOHEi7du3x8HBgW7duvH5558DoNVqiYqKYuTIkdy8eZMSJUrQuXNnPv30U8Da+/v9999z4cIFFEWhdu3aLF++HGdn5xR1yAtKVFSUpJ1CLCwsDF9f31S37dqlpVcvF1Q1+a/Dzp2N/O9/cWTwB2OhkV77CGmfzJA2Sp+0T/oKa/tER0fn2DCA/DwEIj9Iq32OHz9Oy5Yt2blzJ/Xq1XvyFXtCsnutyRCIIqx1azNvvWU/7nfzZj0//mhI4xVCCCGEyI/WrVvHjh07uHTpErt372bUqFE8/fTT1K1bN6+rli9JAC7iJk5MpHFj+ykgJk925O+/5dIQQgghCoqYmBjGjx9PQEAAw4cPp3r16qxatSrDMcBFlYwBLuL0epgzJ46WLd24d8/6S5KUZF0qedeuGFxd87iCQgghhMjQwIEDGThwYF5Xo8CQbj5B5coqP/wQZ1d2/ryW8eOd0niFEEIIIUTBJQFYABAYaGLoUPvxwEuWGFi6VJ/GK4QQQgghCiYJwMLms88SqFnTfqnkd95x4vx5uUyEEEIIUXhIshE2zs7wyy9xODomz4wXE6Pw8stOZLCqoxBCCCFEgSEBWNjx87Pw2WcJdmV//aVjyhSZg1EIIYQQhYME4EJOk5CQ8U6PGDYsiZ497ZdKnjnTga1bZdIQIYQQQhR8EoALM4uF6q++itPQoWguXsz0yxQFvv8+jgoVLHblI0c6ceOGzCcohBBCpGXatGk0bdo0zeepGT9+PN27d8/x9xZpkwBciOmXLcPlzBkMq1fj2rgxju+/j3L3bqZe6+EBc+fGodUmjwe+fVvDiBHOWCxpv04IIYQoiPr3709gYGCq20JDQ/Hw8CAoKCjLx3399dfZsGHD41bPzuXLl/Hw8ODo0aO5/l6pGTlyJP3798/198lNEoBz0IABA6hcuTIvvPBCXlcF4uNx/PRT21PFaMRh1ixc69fHMHMmmbmrLSDAzIQJ9lOj7dql49tvHXK8ukIIIUReeuGFF9i9ezeXL19OsW3BggVUrFiR1q1bZ/m4rq6ulChRIieqmK/eq6CTAJyDRo0axY8//pjX1QBAiYjAUrp0inJNVBROH3yAa5Mm6NasAVVN5dXJ3n47kRYt7JdK/s9/HAgJ0eZofYUQQoi81LlzZ8qUKcOiRYvsyo1GI0uXLuX5559HVVXGjBlDnTp18PLywt/fn++++w5LOl+NPjoswWw2M2nSJCpXrkzlypWZMGECZrP9FKTbtm2ja9euVK5cmSpVqtC7d29CQ0Nt2+vWrQtA27Zt8fLysg2fePS9LBYLX375JbVq1aJMmTI0a9bMrof4QU/ymjVr6NWrF2XLlqVJkybZ6ul+2L59+2jfvj2enp74+voyceJEkh7qeNu3bx8dOnSgfPnyVKpUifbt23Pq1CkAoqOjGT58OD4+Pnh6elK3bl1mzZr1WPVJjQTgHNSqVStc88nawWqVKsTu2MGFKVOwVKiQYrv24kVcXnwRly5d0B46lOZxtFr4+ec4SpRI/uU2mxVeftmZqKjcqLkQQojCqJiHR7Yenl5e2XpdVul0OgYOHMjixYvtAu3GjRuJjIxk8ODBWCwWypYty//93/9x8OBBPvzwQ6ZPn87ChQsz/T4zZsxg/vz5fPvtt2zduhWz2czy5cvt9omNjeW1115jx44drF+/Hnd3dwYMGGALkTt27ABg5cqVHDt2LM33nz17Nj/88AMff/wx+/fvp3v37gwZMoRjx47Z7Td16lRGjBjB3r17qV+/Pi+99BIxMTGZPqeHXbt2jb59+1KnTh12797NDz/8wMqVK5kyZQoAJpOJQYMGERAQwN69e9m2bRuvvfYaWq3WVpdTp06xdOlSQkJCmDFjBuXKlctWXdKTbwLw9OnT8fDwYPz48Tl+7H379jFgwABq1qyJh4dHir/uHpg7dy516tTB09OT1q1bs3///hyvyxOl0XCnWzfuHzpEwkcfobq5pdhFd/Agrh074vTSSyiXLqV6mHLlVGbPjrcrCw/X8OabThl1IAshhBAFxpAhQ7h69So7d+60lS1cuJB27dpRoUIF9Ho9H3zwAf7+/lSuXJlnn32Wl156iZUrV2b6PWbPns3YsWN59tlnqVatGl988QVlypSx2ycwMJDAwECqVq3K008/zcyZM7l8+TJ//vknACVLlgSgRIkSlClThuLFi6f6XjNmzGDMmDH07dsXHx8fPvjgA5o2bcqMGTPs9hs1ahRdu3alatWqfPTRR9y9e5fjx49n+pwe9ssvv+Dp6cn06dOpXr06Xbp0YfLkycyZM4e4uDju379PdHQ0Xbp04amnnqJatWr07duX6tWrAxAeHk6dOnVo0KABlStXpmXLlvTq1StbdUlPvgjAhw4d4rfffqNWrVrp7nfw4EESExNTlF+6dCnVMTsPxMbG4ufnx+eff46Tk1Oq+6xatYoJEybwzjvvsHv3bho3bkzfvn0JDw+37dO0adNUH1evXs3kmeYRJycS336b+0eOkPjKK6jalMMXDKtW4da4MY4ffkhqXbudO5sYOdK+7VevNvDbb7JUshBCiMKhatWqNGvWzNajev36dbZv386QIUNs+8ybN482bdpQtWpVypcvz6xZszKdA6Kjo7lx4waNGjWylWk0Gho0aGC338WLF3nllVeoV68eFStWpFq1algslizljXv37nH9+nUCAgLsyps2bcqZM2fsyh7OX2XLlgXg1q1bmX6vh4WGhtKoUSM0muSI2bRpU5KSkrhw4QLFixdn0KBBPPfcc/Tr148ZM2bYndfLL7/M6tWrad68OZMmTWLv3r3ZqkdG8jwAR0dH8+qrr/LDDz/gkc5XFqqq8u677/LCCy9gNCbPURseHk6PHj345Zdf0nxtp06d+OijjwgMDLT7H/KwmTNnMmjQIF588UWqV6/OV199haenJ/PmzbPtExwcnOqjQipDDPIjtXRpEr7+mpj9+zF26ZJiu5KUhMMPP+BWvz6GH39McaPcxx8nUKeO/TilCROcOH06zy8jIYQQIke88MILbNiwgbt377J48WKKFy9Ot27dAGtn2cSJExk0aBArV65kz549vPzyy3bjW3PCgAEDuH37Nt9++y3btm1j9+7d6HS6HHsfRbGf0lSv16fYpmbzK15VVVMc/9Fjz5o1i23bttGsWTM2btxIw4YN2b59OwAdO3bk+PHjvP7660RGRtK/f39GjRqVrbqkJ8+Ty5tvvklgYGCGd1YqisLy5cu5cOECw4YNw2Qy8c8//9CjRw8aNWrE5MmTs12HpKQk/vrrL9q1a2dX3q5dOw4ePJjt4+ZXlurVifvf/4hZswZznToptmvu3sVpwgRcAwLQrVtnu1HOwQHmzYvDxSX5lyIhQeGll5yJj09xGCGEEMImOioqW4+IGzey9brsCgwMxMHBgaVLl7Jw4UIGDBhgC4jBwcE0aNCA4cOHU69ePby9vbmYhXn2ixUrhpeXF4cPH7aVqarKkSNHbM/v3LlDaGgob7/9Nm3atKF69ercv38fkyn5hnSDwQCQ4ua5h7m7u1O2bFkOHDhgVx4cHGwbbpAbatSowaFDh+zGUQcHB2MwGHjqqadsZbVr1+bNN99kw4YNtGjRgiVLlti2lSxZkgEDBtjGMC9ZsiTVEQCPI08D8G+//caFCxf44IMPMrV/mTJlWLt2LadOnWLo0KH07NmT2rVr8/PPP9sGT2dHZGQkZrOZ0o/MmlC6dGlu3ryZ6eMEBgYydOhQtm7dip+fHyEhIdmu05Ngbt2amJ07iZs9G0sqA8y1Fy7gMmQILt26of133JGPj4Wvv7ZPu6dPa/ngA1kqWQghRMHn5ORE3759+fzzz7l48aLd8AcfHx+OHTvG1q1bOX/+PF9++WWW7xd67bXX+O6771izZg1hYWFMmDCBiIgI23YPDw9KlizJ/PnzuXDhAnv37uXtt99Gp0tejbV06dI4OTmxfft2bt26RXR0dKrv9frrrzNjxgxWrFjBuXPn+M9//kNwcDBjxozJYqukdO/ePY4dO2b3uHz5Mi+//DI3btzgnXfeITQ0lM2bNzNlyhReffVVnJ2duXTpEh9//DEHDx7kypUr7N69m5MnT9pC+X/+8x/Wr1/P+fPnCQ0NZd26dVSpUgUHh5ydgjXP1rYNCwvjk08+YePGjba/ZDKjbNmytvE3Xl5ezJkzx+6ieByPdtmn142fmjVr1uRIPZ4ojQbjwIEYAwNxmDULh2+/RXnkzk9dcDCu7duT1KcPCR9+yMCBlQkKSmLZsuT/b/PmOdC6tYnAQNOj7yCEEEIUKEOGDOGXX36hSZMmdr2lw4YN4/jx47zyyiuoqkrPnj0ZPXp0lmaBGDNmDBEREbz++uuAdQGOvn372qY502g0zJs3jwkTJtC0aVO8vb2ZOnWq3RoDOp2OL774gi+//JIvvviCpk2bproAxmuvvUZMTAyTJ0/m5s2b+Pr6Mn/+fOqk8u1vVgUHB9OqVSu7sp49ezJ//nyWL1/ORx99RMuWLSlWrBh9+vTho48+AsDZ2Zlz584xdOhQIiMjKVOmDH379uXNN98EwMHBgalTp3L58mUcHBxo1KgR//vf/x67vo9SoqKi8uQ+/kWLFjF69Gi7nluz2YyiKGg0Gq5du5Zq2o+MjKRHjx6UK1eOM2fO0LJlS2bOnJnm2N5HlS9fni+//JLBgwfbypKSkihbtiy//PKL3Z2G48aN49SpU/zxxx/ZP9E8FhYWhq+vb6b3V27exGHaNAy//YaSyryGqoMDSa+9xu1X36JVzwpcuJD8/69YMZU9e+5TqVLBmRoiq+1T1Ej7ZEzaKH3SPukrrO0THR1NsWLFcuRYCQkJODrKt4xpKertk91rLc+GQHTv3p39+/ezZ88e26N+/fo899xz7NmzJ9Ve4bt379omal60aBHr169n9+7dvPHGG9kerA3WsTT16tVLMfFzUFAQTZo0yfZxCyK1TBkS/vtf641ynTql2K4kJuLw3XeUbeXPpmf+i5MueUB+dLTC8OHOmKQTWAghhBD5WJ4FYA8PD/z8/Owezs7OFC9eHD8/v1SHI/Tr148SJUqwaNEiHBwcqFKlCmvXrmXbtm1MnTo1zfeKiYmxjU95MI3IsWPH7KY4Gz16NIsXL2b+/PmEhoby3nvvcePGDYYNG5ZrbZCfWWrUIG7ZMmJWr8b89NMptmvu3MH3+/GEezxNT9YA1j9ADhzQ8fnnslSyEEIIIfKvPBsDnFWKovDBBx/QuHFju67+qlWrsmbNmnSHQBw9epQePXrYnk+bNo1p06YxcOBAZs+eDUDv3r25c+cOX331FREREdSsWZNly5ZRqVKl3DupAsDcpg0xu3ah/9//cJw6Fc3163bbS94OYw292ElrxvE1f9KQr7925MoVDR99lECFCgVnOIQQQgghioY8GwMsnowcHV8WG4vDzJk4fPcdSmxsqrssZDDv8xnhVMLRUWXMmETeeCORVBahyxcK6/i7nCLtkzFpo/RJ+6SvsLaPjAF+cop6+xS4McCiAHJxIfHdd7l/5AhJL76Imkqv+/Ms4gw1eJ//YElI4uuvHWnQwI358/WkM12hEEIIIcQTIwFYZJnq6Un8d98Rs3cvxg4dUmx3Jp7/MInj1KYjW7h5U8PYsc60bOlKUFCBGXUjhBDiMTzOzelCZMbjXGMSgEW2Wfz8iFuxgthVqzD7+aXYXo0wttCZZfSlPFc5dUrLs8+60K+fM2fOyKUnhBCFlYuLC1FRURKCRa5RVZWoqChcXFyy9XrpjhOPzdSuHTF79qBftAjHKVPQREbabe/LCrqykU/4iG95ky1bDGzfrmPo0CQmTEikdGn5gBRCiMJEp9Ph5ubGvXv3HvtY9+7dw93dPQdqVTgV5fZxc3PL9mJoEoBFztBqMb7wAqYePXD49FMMv/6K8tBf/q7E8iXvMZT/YzQz2Wluyy+/OLB8uYG3307gtdeSKMJj+IUQotDR6XQ5ciPczZs3qVixYg7UqHCS9ske+R5a5Ci1eHESvvmG2O3bMdWvn2K7H6cJoh2LGIQX17l3T+Hjj51o1MiNVav0yLdlQgghhMhtEoBFrjD7+xO7bRvx33yDxcMjxfZBLCGU6rzBt2gxER6u4aWXnOnUyYWQEG3KAwohhBBC5BAJwCL3aLUkvfQSMYcPk/T88yk2u3Ofb3mLI/jTnL0AHDqko1MnV156yYnLl5UUrxFCCCGEeFwSgEWuU0uVIn7GDGI2b051WeU6HGcvLfmVoZQhAoBVqww0buzGxx87EB39pGsshBBCiMJMArB4YsxNmhCzcyfxX3yBmsodq0P5jVCqM4qZaDCTmKjw7beO+Pu7MXeuAZMpDyothBBCiEJHArB4snQ6kkaM4P6hQyT1759iswfRzGQMITSmMQcBiIzUMG6cE82bu7Jli05ulBNCCCHEY5EALPKE6ulJ/E8/EbNhA+aaNVNsb8ARDhLAz7xKSW4DEBqqpV8/F3r3dubECbl0hRBCCJE9kiJEnjI3b07M7t3Ef/opqqtriu2vMpdQqvMKc1CwABAUpKdVK1fGjnUiIkJulBNCCCFE1kgAFnlPryfp9de5HxJCUu/eKTaX5A5zGE4wTfHnTwAsFoX58w34+7vx1VcOxMU96UoLIYQQoqCSACzyDbVcOeLnzSNmzRrMvr4ptjchhEM0Ygaj8eAuALGxCv/5j/VGuRkzDMTG5kLF4uJQLl1Ce+gQug0bMPz6K4aZM9GEheXCmwkhhBAit8lSyCLfMbduTcy+fTjMnInDV1+hPNS9q0FlNLPoy3Le5Uvm8wIqGm7c0DBpkhPffOPAqFFJvPJKIqmsv5EsLg7l5k1cjh9Hd/Ysyq1baCIirP+9eRPl34fm1i2U+/dTPYT68cckfPQRSaNHg0b+lhRCCCEKCgnAIn8yGEh86y2S+vTB6f330a9bZ7e5DLf4P4bxCnMZxSyOUwdH4nG7E8HWqTc4N/0Gz7X4h3a1ruMSE2ENtbduoURE2IXalJOxZZ5iNOL04YfoduwgfvZsVC+vxziaEEIIIZ4UCcAiX1MrViRuwQJ027bhOH482osX7ba3YB9HqU8sLrjzUE9tPLD130cu0wcFoW3WjPgZMzB165b7byiEEEKIxyLf24oCwdShAzHBwSS8/z6qo6PdNi0W+/Cbw1SdDku5cpjr1sXYsSNJAwagFitmt4/mzh1cBg3C8Z13kDvyhBBCiPxNeoBFweHoSOK775LUrx9O772HfvPmbB9K1epQPcsQ7+6OoWJF1NKlsXh6opYujVqmDJYyZVAfPDw8UozxTbh6FecRI9Dt22dX7vDLL+j27iVu7lwstWtnu35CCCGEyD0SgEWBo1apQtzSpeg2bsRx0iS0589by3U61NKlMZUqw5VETw6Hl+VSvBc38CICTyLwtP0cbfGgV1MzffqE0bVrhazXoUIFYteuxeHbb3GYNg3loXWataGhuLZvT8LkySSNHCk3yAkhhBD5jARgUWCZunYlpmtXlBs3wGCw66ktDbRNgMWLDaz6rwPh4Y+EUBVWrtSycmUtunUzMm5cIv7+5qxVQKsl8Z13MLVpg9Mrr9iNT1aSknD64APrDXKzZqF6ej7eyQohhBAix0jXlCjwVC8v1BIlUvS0OjrCSy8lceTIfWbOjMPHJ/WA+8cfetq1c6V3b2f279dm+f3NDRoQs3s3SYMGpdim374d1+bN0T3GcA0hhBBC5CwJwKLQ0+th8GAjBw/GMG9eHH5+qQfhHTv0dOvmSteuLuzYoUNVs/Ambm7Ez5pF3Lx5qO72k6tpbt/GpX9/HMePh/j4xzgTIYQQQuQECcCiyNBqoXdvI3v3xrB4cSz+/qZU9wsO1tG7twvt2rmwYYMOiyXz72Hs3Zv7e/diato0xTaHOXNwbdcOzcmT2T0FIYQQQuQACcCiyNFooFs3E9u3x/LDD2dp1iz1IHz0qI7Bg11o0cKVlSv1mDM5RFitVInY9etJ+OADVK39kArt6dO4tmuH4ccfyVoXsxBCCCFyigRgUWQpCgQE3OOPP2L5448Y2rc3prrfqVNaXn7ZmSZNXFm0SI8x9d3sabUkjh9P7KZNWCpXtn/fxEScJkzAuV8/lJs3c+BMhBBCCJEVEoBz0IABA6hcuTIvvPBCXldFZFGzZmZWroxjx44YundPPeGeO6dl9Ghn/P3dmDvXQEJCxsc1N2rE/T17SOrfP8U2/dat1hvktmx53OoLIYQQIgskAOegUaNG8eOPP+Z1NcRj8Pc3s2hRHPv23ee555LQaFIOUwgP1zBunBM1argxbpwjR49q0x/N4O5O/E8/ETd3bsob5G7dwqVfPxzfe49MJWohhBBCPDYJwDmoVatWuLq65nU1RA6oVcvCL7/EExISw+DBSeh0KRNuVJSGuXMdaNvWlebNXZkxw8DNm0qaxzT26cP9PXswBQSk2Obw00/WG+ROncrR8xBCCCFESnkagOfMmUOzZs2oWLEiFStWpGPHjmzOhflS9+3bx4ABA6hZsyYeHh4sWrQo1f3mzp1LnTp18PT0pHXr1uzfvz/H6yIKFh8fCzNnxnPkyH1eeSURB4fUu3pPndIyaZITfn5uDBzozPr1ulTHCquVK1tvkJswAfWReYu1p05Zb5D7+We5QU4IIYTIRXkagMuVK8eUKVPYtWsXQUFBtGrVisGDB3PixIlU9z948CCJiYkpyi9dusTly5fTfJ/Y2Fj8/Pz4/PPPcXJySnWfVatWMWHCBN555x12795N48aN6du3L+Hh4bZ9mjZtmurj6tWrWTxzUdBUqqTy9dcJ/P33fV5/PZESJVKfG81kUti4Uc/zz7tQs6Yb77/vyIkTj/ya6XQkTphA7MaNWCpVstukJCTg9O67OA8YgHLrVm6djhBCCFGk5WkA7t69Ox07dsTb2xsfHx8+/PBDXF1dOXToUIp9VVXl3Xff5YUXXsD4UNdaeHg4PXr04JdffknzfTp16sRHH31EYGAgGk3qpzxz5kwGDRrEiy++SPXq1fnqq6/w9PRk3rx5tn2Cg4NTfVSoUOExWkEUJF5eKp9+msCZM/eZPz+Wzp2NaLWp99bevq1h1iwHWrRwo3VrV37+2cDdu8lDJMxNmlhvkOvXL8Vr9Zs3W2+Q2749185FCCGEKKryzRhgs9nMypUriY2NpXHjxim2K4rC8uXLuXDhAsOGDcNkMvHPP//Qo0cPGjVqxOTJk7P93klJSfz111+0a9fOrrxdu3YcPHgw28cVhZfBAD17mli6NI5Tp+7z6afx1KiR9kTBf/+t5d13nahe3Y2hQ53YskWHyQQUK0b8zz8T99NPqG5udq/R3LyJy3PP4ThxIqTyzYcQQgghsifPA/DJkycpX748ZcqU4a233mLhwoXUqlUr1X3LlCnD2rVrOXXqFEOHDqVnz57Url2bn3/+Ge0jCw5kRWRkJGazmdKlS9uVly5dmptZmKc1MDCQoUOHsnXrVvz8/AgJCcl2nUTB4emp8vrrSQQHx7B9ewwvvZSIu3vqvcJJSQqrVxvo18+F2rXd+PhjB8LCNBj797feINeoUYrXOMyejWv79mjOnMntUxFCCCGKhDwPwL6+vuzZs4dt27bx8ssvM3LkSE6lcyd82bJlmTdvHuvXryc2NpY5c+ag0+lypC6KYn8Hv6qqKcrSs2bNGs6fP8/169c5depUqj3ZovBSFGjQwMw33yQQGnqPuXPjaNvWiKKkHoavX9fw7beONGrkRqdOLvzfLl+uL91IwrvvprxB7sQJXNu0wfHDD9Hu3Anx8bl/QkIIIUQhlecB2GAw4O3tTf369Zk8eTK1a9dm1qxZae4fGRnJqFGj6NChAzqdjrfeeguLJfUbkjKrZMmSaLXaFL29t2/fTtErLERmODlBnz5Gfv89juPH7zNpUgLe3mkPkQgJ0fHGG85Ur1WCFy99yr7/bMT8yNhyJSEBhx9+wLVXL9yrVMElMBCH//4X7ZEjZHqdZpHnlLt3cfjiCxwnTkRJ5+ZdIYQQuSfPA/CjLBYLSUlJqW67e/cuvXr1omzZsixatIj169eze/du3njjDdTHmDbKYDBQr149goKC7MqDgoJo0qRJto8rBECFCirjxiXy558x/PGHdV5hF5fUr9f4eIVlywy0nNiJuurf/FWjb6r7KYmJ6HbtwnHKFFzbtcPd2xvnIUMw/PILmvPnZRq1fEq7axeuzZvjOG2adWhLu3YytEUIIfJAngbgjz/+mP3793P58mVOnjzJlClT2Lt3L337pvxHX1VV+vXrR4kSJVi0aBEODg5UqVKFtWvXsm3bNqZOnZrm+8TExHDs2DGOHTuGxWLh6tWrHDt2zG6Ks9GjR7N48WLmz59PaGgo7733Hjdu3GDYsGG5cu6i6FEU65LLM2fGExp6j5kz42je3JTm/if/KUH9M0t5gd+I0bqnuR+AEh2Nft06nN55B7cGDXCrXRunMWPQr1iBkoVx7CKXJCbi+OGHuPTqhebaNVuxJjISl2efRbl0Ke/qJoQQRZASFRWVZ11FI0eOZM+ePdy8eRN3d3dq1arF2LFjad++far779y5k8aNG+Ps7GxXfvbsWTQaDT4+Pqm+bs+ePfTo0SNF+cCBA5k9e7bt+dy5c/nuu++IiIigZs2afPbZZzRv3vwxzjDvhYWF4evrm9fVyLfyQ/tcvKhh8WI9S5YYuHo19b9JixFFFzbRVb+V7oZtlIq9kqX3MNeqhalNG+ujaVPI5IqF+aF98ruM2khz5gzOr76K9vjxNPexVKpEzMaNqOXL50YV85RcQ+mT9smYtFH6pH2yJ08DsMh98ouRvvzUPhYL7N6tZfFiA2vX6klISOsGTJWqnGeI1xb6ldxGtatBaKOjMv0+ql6PuVEjaxhu3Rqzvz/o9anum5/aJ79Ks41UFcOcOTh+9BFKQkKGxzH7+hL7xx+ohey+A7mG0iftkzFpo/RJ+2RPvhsDLERRpdFAmzZmfv7ZOkTi22/jaNQotSESCufx4eMbo/A7uQqX2Fu83eoAxwd/grF1G1QHh3TfRzEa0e3fj+Nnn+HaubN1/PCAARh+/NE6HlXGDz82JSIC5379cHr33RTh11KhAjFr16ZYAEUbFobLs89CVNQTrKkQQhRNOTN/mBAiRxUrBkOHGhk61EhoqIbFiw387396IiJS/s2aaNLx391N+C9NKF/+A4a8Fs0rfnsof2Ynup070f71F0o6oVa5fx/9pk3oN20CwOLlhal1a0xt2qCvVAmkZyFLdBs34vT662hu306xLal3b+K/+QY8PIhv1gwlLg79+vW27doTJ3Dp25fYVavgkYVRhBBC5BwZAlHIyVcj6StI7WM0wtatOhYsMLBliw6zOf05qlu2NDFkSBI9W9zE9dBudLt2WQPxhQtZel/zU09hDgjA1LQp5oAALL6+1jv6BPDQNRQXh+OkSTg8tHz6A6qbG/FffYWxf3/7tktMxHnQIPSPLHltatmS2GXLrPPpFXAF6XcsL0j7ZEzaKH3SPtkjAbiQk1+M9BXU9rlxQ2HpUj0LFhg4dy79VRDd3VX69k1iyJAk6ta1oLly2RqG/32k1lOZHkvJkpibNLEFYnPduta1oYuosLAwqsfGWm90CwtLsd0UEEDcjz+iVqmS+gHi4nB57jl0wcF2xcbOnYlbsKDAt21B/R17UqR9MiZtlD5pn+zJ8hjgffv28eOPP9qVLV++nIYNG+Lj48N777332AtTCCHS5+Wl8sYbSRw6FMPGjTEMGpSEs3Pqf8veu6fwyy8OtGnjRsuWrvy4yZebPV4k/pdfuH/2LPf37iV+6lSMHTuiPjLDSmo0kZHo//gDpw8/xLVjR9wrVcKle3ccpk5Ft20bREfn9OnmX2YzXv/3f7h26JAi/KpaLQkffEDs+vVph18AZ2dily7FVL++XbF+82acRoyQRU6EECIXZLkHuGfPnpQsWZJff/0VsE5B1rx5c5566ikqV67M9u3bmTp1KqNGjcqVCouskb8M01eY2ufePfj9d2uv8OHD6Q/vd3BQeeYZI0OGJNGqlRnbystJSWgPHUK30zp+WHPkCJosBjBVo8FSqxamgADMTZtiCghALVcum2eVfynh4Ti/9hq6fftSbDN7exM/Zw7mBg0yf7w7d3B55hm0jywFnzR4MPE//ACagnnPcmH6HcsN0j4ZkzZKn7RP9mT5E/XMmTM0eOhDfdmyZTg5ObFt2zaWL19O//79WbhwYY5WUgiRMXd3ePFFI9u2xRIcfJ/RoxMpWTL1b2MSExVWrjTQq5crdeu68fnnDoSHK2AwYG7enMQPPiB261b+CgoiZu1aEt5/H2O7dqiZmD9YsVjQHj+Ow5w5OL/0Eu5+frjVqYPT8OEYfv0VzenT1jnfCjD9ihW4NW+eavhNGjKEmN27sxR+AdQSJYj9/XfM3t525YZFi3CcMEFm5xBCiByU5Vkg7t27h4eHh+359u3badu2Le7u1pWqmjZtyrp163KsgkKIrKtZ08J//pPA5MkJbNyoY+FCA9u367BYUt68Fh6u4fPPHfniCwfatjUxZIiRbt2MODiAxckJc506mFu1su5sMqE5eRLdgQNoDxxAFxyM5saNDOujuXIFw5UrsGwZABYPD/txxPXrQwbTt+UL0dE4jR+P4d/zeJileHHiv/8eUyqL7mSW6ulJ7Jo1uHbtiubqVVu5w88/o7q5kfjhh9k+thBCiGRZDsCenp6EhoYCcP36dY4dO8bQoUNt2+/du4dWm/5NOUKIJ8NggMBAE4GBJv75R2HJEgMLF+q5dCnl76iqKuzYoWfHDj3Fi1vo189I69ZO9rOg6XRY6tYlqW5dGDECVBXl8uXkQHzgANozZzKslyYqCs3mzeg3b7a+t4MDZn9/67CJgABMTZrAQ39o5wfa4GCchw9H89AS6g8Y27YlftYs1LJlH/t91IoViV2zBpeuXdE8tIy14/Tp4OpK4ltvPfZ7CCFEUZflANyjRw/mzJlDYmIiR44cwcHBga5du9q2nzhxgirp3fAhhMgT5curjBuXyNtvJ7J3r5aFC9Nece7uXQ0//eTATz/Vws/PTGCgkcBAIzVqPDJ0QVFQq1TBWKUKxgEDrEV37qA9eNAWirVHjqAYjenWTUlMRBccbJsJQVUULLVrY2reHFPz5pibNUMtUSJnGiKrjEYcvvgCh2++QXlk6Ibq4ED4qFEU+/DDHB2ja6laldjVq3Hp3h3N3bu2cscpU1BdXEgaPjzH3ksIIYqiLAfgiRMncvPmTZYtW4abmxszZsygTJkygLX3d926dbz66qs5XlEhRM7QaKBVKzOtWsXz5ZfxrFxpYMECPX/9lfrHwalTWk6d0jJtmiM1apjp2dNIr15Gata0pDodsFqiBKauXTE9+MM4Ph7t0aN2vcTKvXvp1lFRVbTHjqE9dgyH2bMBMPv5WQNxixbWQPwElgzWnD+P06uvojtyJMU2s58fcXPmcNNgoFgu3KBm8fMjbtUqXHr2RLl/31bu9O67qC4uGAcPzvH3FEKIoiJH5wG2WCzcv38fZ2dn9Hp9Th1WPAa5OzR90j7Jjh/XsGCBgWXL9ERFZRzofH2Te4affjr1MJwqiwXN6dP244gfGu+aWebq1a29w//2EqteXlk+RppUFf38+ThNnIgSF5dic+LIkSRMngyOjrl+DWn378fluedQ4uOTq6fREDdvHqZevXLtfXOK/I6lT9onY9JG6ZP2yZ4cC8A3btwgKiqKGjVq5MThRA6RX4z0SfuklJAAf/yhZ+FCPTt3pn7j3KO8vc306mWkZ08jdetmIQz/SwkPR3fwINrgYHT796M9fTrL9TZXrWoLw6bmzVErVMjyMQCUyEicxo5Fv2FDim0WT0/iZ8/G1K6drexJXEO67dtxHjgQJSnJVqbqdMQtXoypU6dcfe/HJb9j6ZP2yZi0UfqkfbInywH4119/5cCBA/z000+2snfeecc2L3CtWrVYvXo1JUuWzNmaimyRX4z0SfukLyTkIqdPV2PNGj27dmW8/DJAlSrmf2+8M1K/vjlbqyYrkZFo9+1D9+9Dc/IkShanAbNUrmwLw6bmzVErV85wCWfdjh04jRyJJiIixTZj9+7Ef/896iOfbU/qGtKtW4fz0KEoD83LrDo6Ert8OeaWLXP9/bNLfsfSJ+2TMWmj9En7ZE+WB6799ttvuLm52Z7v3r2befPm0adPHz766CMuXrzI119/naOVFELkjeLFTbz4opFVq+IIC7vPDz/E0bGjEZ0u7TB66ZKW775zoF07V+rUcWPSJEcOH9ZmaRpbtWRJTD17kvDFF8Ts3cu9ixeJXbyYxNGjMdWrh5qJMbeay5cxLF6M8+jRuNerh1vt2jgNH45+/nw058/bz6ubkIDjhAm49O6dIvyqzs7Eff89cQsXpgi/T5KpRw/rTBMPhXglIQGXgQPRHj6cZ/USQoiCKMs3wV2+fJnnn3/e9nz16tWUL1+eH3/8EY1GQ3R0NL///jvTpk3L0YoKIfJWiRIqQ4YYGTLESFSUdZjEmjV6duzQYTSm3rMaHq5hxgwHZsxwoEIFCz16WG+ga9TInLVJEzw8MHXrhqlbN+vz6GjrkIn9+9Ht24f26FEUkyndQ2iuXsWwbJltDl+Ll5d1DHGjRhjmz0+xAhuAyd+f+DlzsFStmoXK5h5j//4ocXE4PTQVmhITg8tzzxGzfj2W2rXzsHZCCFFwZDkAJyUl2d3gFhQURIcOHdD8+6+Zt7c3NzIxMb4QouDy8IBBg4wMGmQkOho2bdKzerU1DCcmph6Gr17VMHu2A7NnO1CunDUMBwYaadLETJanDi9WDFOnTpg6dSIRICYG3aFDtmET2sOHM5x6TXPjBoaVK2HlyhTbVI2GxLffJvG99yCf3dCbNGwYxMTg9NCiGEp0NC7PPkvsxo1Y5KtQIYTIUJaHQFSuXJmdO3cCcOTIES5dukS7h24IuXnzpt0QCSFE4VasGPTvb2TJkjjCwu4xd24czzxjxNEx7TEP165Z5xnu1s0VPz83xo93ZM8eLQ8Nb80aV1dMbduSOGkSsRs3cu/KFesSzu+9h6lFC9QsrDJnqViR2A0bSJw0Kd+F3weSXn+dhHfftSvT3L6NS69eKJcv51GthBCi4MhyD/BLL73E+PHjCQ0N5dq1a5QvX56OHTvath84cEBmghCiiHJ3hz59jPTpYyQmBrZssQ6T2LJFR3x86j3DEREa5sxxYM4cB0qXTu4Zbt7cjC7Ln1D/cnLC3KoV5latrD3EiYlo//zT2ju8bx+6kJBUpzdL6teP+K++sqb6fC5x4kSUmBgcZs2ylWn++QeXwEBiN27MkVXphBCisMryPy+vvPIKBoOBLVu2ULduXd58802cnJwAuHv3Lrdu3eKll17K8YoKIQoWV1fo3dtI795GYmNh2zYda9bo2bxZT2xs6mH41i0N8+Y5MG+eA6VKJY8ZfqwwDODggLlZM8zNmsH48ZCUhPavv6yBODgYJT6exFdewRQY+Bhv8oQpCgn/+Q9KbCyG336zFWsvXbIOh9iwIU9v2hNCiPwsRxfCEPmPTI+SPmmf9OVG+8TFwfbtOtau1bNxo56YmIznSStVysIzz1jDcIsWjxmGc1ieX0NmM06vvYZh+XL74rp1iVm7Ns97s/O8ffI5aZ+MSRulT9onex7rn5GTJ09y5coVACpVqkStWrVypFJCiMLL2Rl69DDRo4eJhIR4duyw9gxv3Kjn3r3Uw/Dt2xr+7/8c+L//c6BkSQvPPGOiVy8jLVua8lUYzhNaLfGzZqHExqL/44/k4r//xqV/f2JXrgQXlzysoBBC5D/ZWsB+w4YN1KlTh5YtWzJ48GAGDx5My5YtqVu3LhtSWT1JCCFS4+gI3bqZ+OmneMLC7vG//8UycGAS7u5pfzEVGanht98MPPusC9WquTF2rNO/U7E9wYrnN3o9cfPmYWzTxq5Yd+AAzoMHW5f3E0IIYZPlALxt2zZeeOEFVFXlww8/ZOHChSxYsIAPP/wQVVV58cUX2b59e27UVQhRiDk4QJcuJmbPtobhpUutYbhYsbTD8J07GubPN9C7twvVq7vx+utFOAw7OhK3aBGmgAC7Yv3OnTgPG0bRbBQhhEhdlscAd+rUiZiYGDZv3pxiurP79+/TuXNn3N3d2bRpU45WVGSPjA1Kn7RP+vJD+yQlwc6dOlav1rNhg57o6IzHDBcvnjxMolUrU67OZpYf2shOdDSuPXui/ftvu+Kk554j/uefyfqky48n37VPPiPtkzFpo/RJ+2RPlnuAT5w4weDBg1Od69fNzY3Bgwdz7NixHKmcEEIYDNCpk4lZs6w9w8uXxzJ4cBIeHpY0X3P3roYFCww895x1mMSYMU5s21ZEeoaLFSN21SrMj0xHaVi50rqCXGJiHlVMCCHyjyzfPqLX64lLZf7MB2JjY+1WihNCiJxiMEDHjiY6djTx3//C7t3WnuH163VERaX+9/zduxoWLjSwcKEBDw8L3btbe4ZbtzZhMDzhE3hC1JIlif39d1y6dUN78aKt3DB/Pob587GULo2lQgXU8uWxlC9v/3P58qheXsjdhUKIwizLn3BNmzZlzpw59O7dm6pVq9ptu3DhAnPnzqVZs2Y5VkEhhEiNwQAdOpjo0CFlGL57N/UwHBWlYdEiA4sWGShWTKV7d+vUam3aFL4wrJYtS+zq1bh264bmn3/stmlu3UJz6xYcPZr6a7Va1LJlkwPxw+G4QgXrf0uXBiXj4ShCCJEfZTkAT548mc6dO9O0aVO6du1qG3dy9uxZNm/ejIODA5MnT87xigohRFr0emjf3kT79ia++Qb27EkOw3fupB6Go6MVFi82sHixNQy3bWu0HaNcucIxPbpauTKxq1fj0q2bNfBmkmI2o1y9iubq1bSPbTDYh+NHepEtFSrk+RzEQgiRliwH4Jo1axIUFMSUKVPYvn07a9euBcDFxYUuXbowduxYTCZTjldUCCEyQ6+Hdu1MtGtnYvp02LtXx++/ZxyGV682sHq1tRvYz89Mu3bWMNy0qQlHxyd5BjnL4utL7IYNOH7wAdqTJ1Fu3ECxpD1+OrOUpCTr8IqHhlg8SnVxwc/LC12dOliqV8dSvTrm6tWxVK1qnfZDCCHyyGOtBGexWLh9+zYApUqVQqPR8PXXX/PZZ59x586dHKukyD65OzR90j7pK0ztYzRaw/CDnuHIyMzdA+zkpNKihcnWO+zjY7H75r/AtZHRiHL9Opp//kHzzz8o//yD5urV5OdXr6KJjMzVKqhaLRZvb2sgrlEjORj7+oKTU66+d35T4K6fPCBtlD5pn+x5rLscNBoNZcqUyam6CCFErtHroW1bE23bPugZ1rJ6tZ516/TphuH4eIWtW/Vs3Wq9ubdiRQsdOhhp185E69YF8NsuvR61UiXMlSphTmuf+Hg0165Zw3BaQfnevWxXQTGb0YaFoQ0LQ79+va1cVRQslStbe4tr1LCG4ho1MPv6QiozDwkhRHbJbb5CiCJHp4M2bcy0aWNm+vQE/vpLy7ZtOnbs0HHokBaLJe2bu8LDNfz6qwO//uqATqdSu3Z1unfX06GDkTp1LGiytb5mPuPkZB2mULVq2iH53j37cBwebv/8n39QsrgCnaKqaC9dQnvpEmzebLfNUqGCNRA/0muMh0d2zlAIUcRJABZCFGlaLTRoYKZBAzPvvZdIVBTs2qVj+3Y927fr+OeftBOtyaRw9KgbR4/C1KmOlCplsY0/btfORJkyheNmulS5u2Nxd8dSs2bq21UVJTKSazt3Uik2Fu2ZM2jOnkUbGpruzXVp0Ty4Ke+RlUYtXl5YqlVL7i3+979qqVLZOSshRBEhAVgIIR7i4QGBgSYCA02oKoSGamy9w/v26UhMTLt3+PZtDcuWGVi2zHozXZ06Ztq3tw6XaNLEXOimWkuXoqCWKkVM3boYfX2xW4Pk3j20YWFozpyxBuIHj8uXUdSs/dGguXEDzY0b6Hbvtiu3lC5tDcQ1amDx88NcowbmmjWlx1gIAWQyAP/555+ZPuC1a9eyXRkhhMhPFAVq1LBQo0YSY8YkERcH+/fr2L7dGohDQ9NfVvjYMS3Hjmn573/B1VWlZUvrvMXt2xupUqUQ9w5nxN0dc4MGmBs0sA/GcXFowsJsoVh75ow1GF+8iGJOczBGqh7Mdazbs8eu3FK2rDUU16xp918ZYyxE0ZKpANyhQweUTE54rqpqpvcVQoiCxNk5efENgPBwhSVL7nDiRHl27tRx717an30xMQobN+rZuFEPOPHUU2YCAsw0bmymUSMTNWta0Kafpws/Z2csdetiqVvXvjwxEc3589Zg/G8o1oaGojl3DiWL61trrl9Hc/06BAXZlVsqVMDs52frNTb7+WGpVs36P10IUehkKgDPnDkzt+shhBAFTsWKKs8+e5t33y2OyQSHDyffTHf0qBZVTTsQX7yo5eJFLUuWWJ+7uan4+1vDsDUUmylevAj3Ej/MwQGLnx8WPz/7cqMRzaVLdkMptKdPowkLQ0lMzNJb2MYYb9liK1MVBUuVKtZQXLNmcq+xry8FenJoIUTmAvCgQYNyux5CCFGg6XQQEGDt1Z00KZHISIWgoOThEhER6U8Pcf++wq5dOnbtSv5Y9vW1BuHGjU00amSmRg3pJbaj12Px9cXi64upR4/kcrMZzcWLaE6ftg6jePDfsLAs9Rgrqor24kW0Fy+i37jRVq5qNFiqVrULxqb69VGrVMnBkxNC5Ca5CU4IIXJByZIqffoY6dPHiKrCiRMaduzQsW2bngMHtBiNGQ8VCwvTEhamZfFi691zbm4qDRqY/g3F1t5iuacrFVotFh8fLD4+9sHYaERz/ry1x/jUKWsoPnMGzfnzWRpjrFgsyfMYr1tnKzf7+mLq0gVjly6YmzSx/lUkhMiX5LdTCCFymaJA7doWatdO4o03koiNhSNHtBw6pCMkRMuhQ9pMrUx3/77Czp16du7U28qqVUvZS1wo5iLODXo9lho1sNSogalXr+TyxETrzXenT1vD8enTaE6fRnPpUpZmpXgQih1++AFL8eKYOna0BuL27aFYsZw/HyFEtkkAFkKIJ8zFBVq2NNOypbXXUVXhwgWNLQyHhOg4dUqT7oIcD5w9q+XsWS2LFll7id3d7XuJGzaUXuIMOThgefppLE8/bV8eF2edu/jhoRSnT6MJD8/wkJq7dzEsW4Zh2TJUnQ5zs2YYu3TB1KULFm/vXDoRIURmSQAWQog8pihQtaqFqlUtDBxoHaN6/35yL/GhQ9ZgfOdOxl279+4pBAXpCQpK7iWuXt3aSxwQYKJbNxMlSsjNdZni7IylXj0s9erZT9d2/771prt/g7H277/RHjyY5vhixWRCt3u3da7i99/HXK2abagExYs/kVMRQtiTACyEEPmQmxu0bm2mdevkXuLz5+17iU+fzlwvcWioltBQLQsXGtDpVNq3N9Gvn5EuXYy4uOT2mRRCbm6YGzbE3LBhcjCOjkYXFIR+40Z0W7eiuXMnzZdrz55Fe/YsDt9/T71ixbB07oypa1eM7drJUAkhnhAJwEIIUQAoCvj4WPDxsTBokH0vcUhIci/x3bvp9xKbTAqbN+vZvFmPi4tK9+5G+vY10qaNCb0+3ZeK9BQrhqlXL+vYYrMZbUgI+k2b0G3ahDY0NM2X6aKj4eGhEs2bY+zSBWPXrjKrhBC5SImKipLvwnLIgAEDCA4OpnXr1syfPz+vqwNAWFgYvr6+eV2NfEvaJ33SPhnLT22kqnDuXMpe4vTmI36gVCkLzz5rnbWicWMzObWeUX5qn7yiuXgR3caN6DdtQrt/P4rJlKnXmWvUsI4b7twZc+PGFNU58OQaSp+0T/ZIAM5Bu3fvJjY2liVLlkgALiCkfdIn7ZOx/N5G9+5Ze4n37dOxerWesLCMQ1SlShb69k2iTx8jNWtaHuv983v7PHFRUeh37EC3aRO6LVvQREVl6mWWEiUwdeyIsWtXTO3agbt77tRPVcFkgqQkMBqtYf2hny0lS/Kk76qUayh90j7ZI0MgclCrVq3Y88i680IIkZfc3aFNGzNt2ph5//1Ejh3TsHy5gZUr9Vy/nvpwiStXNEyf7sj06Y48/bSZvn2T6N3bSMWK0l/y2Dw8MPbujbF3bzCZuL5yJU+dOGEdKhEWlubLNHfuYFi6FMPSpah6PabmzbHUrAkmE8q/AfVBcFWMRutzo9H+56Qk6z5p/ZyUlKneaYuXF5bq1TFXq2ZdDKR6dSzVq6OWKkWOfXUgRC7L0wD8zTffsG7dOs6dO4fBYKBhw4ZMnjwZv0eXu3xM+/bt44cffuDvv//m+vXrzJw5k8GDB6fYb+7cuXz//fdERERQo0YNpk2bRrNmzXK0LkIIkVcUBerWtVC3bgJTpiSwb5+WFSsMrFmjJzo69eBy4oSWEyecmDzZiWbNTPTta6RXL6Ms05wTdDpi/P1J6N8fPv0UzfnzyUMlgoPTXJxDMRrR79wJO3c+0eo+oLlxA82NG+h27bIrt5QoYQ3G/wbiBz+r5cpJMBb5Tp4G4L179/Lyyy/j7++Pqqp89tln9OrVi4MHD1I8lalhDh48SL169XBwcLArv3TpEoqiULly5VTfJzY2Fj8/PwYOHMhrr72W6j6rVq1iwoQJTJ8+nYCAAObOnUvfvn05cOAAFStWBKBp06apvnb58uVUqFAhK6cuhBB5SquFVq3MtGoVz1dfxbN1q44VK/Rs2qQnISH1sLJ/v479+3W8+64j7dtbw3DXrkacnZ9w5QspS9WqJI0ZQ9KYMdahEtu2odu0Cf3WrSjR0XldvQxp7txBExyMLjjYrlx1c7P2Fj8Sji2VKhXZcc0i7+WrMcAxMTFUqlSJRYsW0bVrV7ttqqrSpk0bvLy8WLhwIfp/b1cODw+nW7duPPvss3zyyScZvkf58uX58ssvU/QAt2/fnlq1avH999/byvz9/QkMDGTy5MmZPoc9e/YwZ84cGQNcQEj7pE/aJ2OFrY3u3YP16/WsWKFn505dhtOsZTSTRGFrn5yWqfYxmdAeOJA8q8S5c7laJ1WrBYMB9HpUnc76s04HGg3KtWuZvokvw/dxdMTi62sLxQ+GVFi8vXn4QpJrKH3SPtmTr8YAx8TEYLFY8EhlgL2iKCxfvpzu3bszbNgw/u///o+IiAh69OhBo0aNshRSH5WUlMRff/3F66+/blferl07Dh48mO3jCiFEQePuDoMGGRk0yEhEhMLvv+tZvlzPn3+m/s9FbKzCsmUGli0z5NpMEkWeToe5RQvMLVrA1Klozp2zLqoRF2cNino96r//xWCwhtaHf340zD68v16P+iDgPihLby1toxHNhQtoQkOti4GcPWtdJS8sDCUhIUunpSQkoD1+HO3x43blqk6HpWpVWygu4eaGtnlzLFWrosrCISKH5KsAPGHCBGrXrk3jxo1T3V6mTBnWrl1L9+7dGTp0KKdPn6Z27dr8/PPPaB/ja5TIyEjMZjOlS5e2Ky9dujQ3b97M9HECAwM5ceIEcXFx+Pn58X//939pnosQQuR3np4qr72WxGuvJXHhgoYVK6xhOK2ZJG7f1jBnjgNz5jjYZpJo2NARHx8ZApqTLD4+JPn45M2b6/W2IQx2/cBmM0p4uDUMPwjFZ8+iDQ1FuX8/S2+hmExo/w3YeuDhhaMtxYtbw7G3t/VRtSqWqlUxe3s/8dkpRMGWbwLw+++/z4EDB9i0aVO6YbZs2bLMmzfPNhxizpw56HQ5cxrKI5/QqqqmKEvPmjVrcqQeQgiR33h7W3j33UTGj0/k77+tM0msWpXxTBLwNOXLW2jTxkTbtibatDFRqlS+GXkncopWi1qlCqYqVaBLl+RyVUW5ft3aW/xwOA4NTXe1vLRo7t5Fc/gwHD6cYpulZMkU4dj87/NcmzZOFFj5IgBPnDiRVatWsW7dOqpksPJNZGQko0aNokOHDpw5c4a33nqLmTNnoknvK5sMlCxZEq1Wm6K39/bt2yl6hYUQoihTFKhXz0K9egl88ol1Jonly60zSdy7l3qHwT//aFi0yMCiRQYAatc207atNRAHBJhwcnqSZyCeKEVBLVcOU7ly0Lat/abbt5OHUvz70IaGorl+PVtvpYmMRBMZCSEhKbZZSpdOGY7//Rk3t2y9nyjY8jwAv/fee6xatYr169dTrVq1dPe9e/cuvXr1omzZsixatIjr16/TvXt33njjDb7//vss9dY+zGAwUK9ePYKCgujVq5etPCgoiJ49e2brmEIIUdg9PJPE11/Hs2WLjhUrDGzapCMxMe3P4+PHtRw/ruX77x1wdFQJCDDZAvHTT1vSHYIqCg+1VCnMpUphbt7cfkN0NNqzZ22BOO74cdwjItBcuICSmJit99LcuoXm1i04cCDFNounZ/JwCm9va69x7dpYnnpKxu4UYnkagMeNG8fSpUtZuHAhHh4eREREAODi4oKrq6vdvqqq0q9fP0qUKMGiRYtwcHCgSpUqrF27lmeeeYapU6fy4Ycfpvo+MTExXLhwAQCLxcLVq1c5duwYxYsXt01xNnr0aEaMGEGDBg1o0qQJ8+bN48aNGwwbNiwXW0AIIQoHBwfo0cNEjx4moqNhwwY9q1bp2bNHQ2Ji2sPaEhIUdu7Us3OnnsmTrUsyt2ljsg2ZKF9ehksUOcWKYW7UCHOjRhiB8w9mObBYUP75B82FC2jPn0dz/rz1hrwLF9BcvGhd0CMbNBERaCIi4JHp2yylSmFu3BhTkyaYGzfGXK8e8nVF4ZGn06ClNtsDWHuFJ06cmKJ8586dNG7cGOdHJp08e/YsGo0GnzRuCtizZw89evRIUT5w4EBmz55tez537ly+++47IiIiqFmzJp999hnNH/3LtICR6VHSJ+2TPmmfjEkbpe/EiXNERlYnKEhHUJCeY8c0qGrme9WqVTPbwnCLFqZC9221XD8Zy1Qbmc0oV6+ivXAhORg/+O+lS9YV8R6TqtdjrlvXGoobN8bcuLF1kY88JtdQ9uSreYBFzpNfjPRJ+6RP2idj0kbpe7R9IiMVdu3S/RuIdVy9mvnxDjqdSqNGyeOH69c3k0P3QOcZuX4y9thtZDJZw/Gjwfj8eTSXLz/WvMaWChVsPcSmJk2w1KpFismwc5lcQ9lTwD86hBBCFCQlS6r07m2kd28jqgrnz2tsYXjvXl2aN9IBmEwKwcE6goN1fPYZuLurtGqVPH74qacsMmRTpKTTJc9Q0b69/TaTCc2VK8mB+Px5tGfOoD1yBCUmJsNDa65exXD1KqxcCYDq7IzZ3z952ESjRqglSuTCSYnHJQFYCCFEnlAU8PGx4OOTxKuvJmE0wp9/agkK0rFzp47Dh7WYzWkn2nv3FNav17N+vbXHrVIlC23bmmjd2kTjxiYqVJAvOEUGdDrbzBB06JBcbjajOXUKXUgI2oMH0YaEoL10KcPDKXFx6PbuRbd3b/KhqlVLHjbRpAkWX9/0Fxt5HBYL3LuHEhWF5u5dlPQe8fGYfXwwN22KKSAAtUKF3KlTPiVDIAo5+WokfdI+6ZP2yZi0Ufoep32io2HvXmsYDgrSce5c1hY8KlfOQqNGZho1MtGokZm6dc04OmarKrlGrp+M5Zc2UiIi0IaEWENxSAjao0ezdeOdxcPD2jv8IBT7+8MjN/5jMqFERaUfYKOjUe7eJfH6dZzi461lUVEoFku2zs9SsSKmpk2tgbhpUyzVquVeUM8HJAAXcvnlgyO/kvZJn7RPxqSN0peT7XPlimILw7t26bhzJ2v/OBsMKnXqmP8NxdZgXKGCmqfDJuT6yVi+baPERLR//4324EFbT7EmC6vHPqBqtVhq1gStNjnE3ruXCxXOGkvx4pgDAmyh2Fy3rnUp7UJChkAIIYQoECpVUnnhBSMvvGDEYoFjxzQEBekJCtJx4ICWpKT0k2xSksLhwzoOH9bxYAKgsmXte4nr1ct/vcQin3JwsPXkJoF11bvLl209xLqDB9GcPJlhj6xiNqM9ceKJVDkrNHfvotm4Ef3GjQCoTk6YGzTAFBCAuVkzTI0aFehFRCQACyGEKHA0mgcr0iXy1luJxMVBcLB1uMShQ1qOHtWmuxjHA9eva1i7VsPatdZxxHq9tZe4YUMzjRtbg3HFinnbSywKCEVBrVIFY5UqGPv1s5bdv4/2yJHkUBwSghIdnavVUF1dUT08UIsXtz0sD/38YBuKgvbPP9EdOID2zz8zHM6hxMfbjW9WNRostWtj+nfIhDkgANXTM1fPLSdJABZCCFHgOTtD+/Ym2re3TmmVlGRdcS4kRMvhw1pCQnSEh2c8ZMJoVPjzTx1//qnjp5+sZZ6e1l7ixo2Te4llPQSRKW5umFu3xty6tfW5xYLm7FlbD7E2JARtWFiKl6mKglqsmF2IfTTAPnhciYmhQp061ufFimVpmILpmWdIBEhIQHv0KLrgYLQHDqA7cCDDYRiKxWIdAvL33zj8+CMAZm9v20115mbNrDcX5tO/HiUACyGEKHQMBmjQwEyDBmZb2Y0byr+BOLmXOCEh43+cIyI0rF+vsc02odOp1K5tP5a4cmXpJRaZoNFgqVEDS40aGF94AQAlMhLNmTPg6JgcbN3drWuNZ0JsWJh1ZonH4ehoHefbtKn1+YNZMA4cQBscjC44GM316xkeRnvhAtoLFzAsWgSApUwZ6zjigABMzZphefpp8svk3fmjFkIIIUQu8/JS6dnTRM+eyb3EJ05oOXTI+ggJ0XHlSsa9xCaTwtGjOo4e1fHzz9YyT08LTZqYadLERECAmTp1zE96PQRRQKklS2LOb6vOarVYatcmqXZtePXV5PHNwcG2UKw9ezbDw2hu3kSzdi36tWsBMNWrR+zOnblc+cyRACyEEKJIMhjA39+Mv7+ZESOsZRERyr+BWEdIiJa//tISH5+5XuKHxxI7Oak0aGAmIMAaiBs1MlGsWG6ejRC56OHxzQMHWotu37YOl/h32IT2778zXFXPUqfOk6htpkgAFkIIIf7l6anyzDMmnnnG+g+50QgnT2oICdHZeokvX864lzg+XmHvXuvqdgCKolKzpsUWiJs0MVGpksxCKgoutVQpTM88g+mZZ6wFsbFoDx9OHkd86BBKbKzda0wPhljkAxKAhRBCiDTo9Q9mm0hi+HBr2c2bim3YxKFDOo4cybiXWFUVTp3ScuqUlnnzrGVly1qoVcubDh0MBASYePppS34ZHilE1rm42N/wZzSiPX7cNoZYe+CABGAhhBCioCpTRqV7dxPdu5uARIxG64wTBw5oOXjQOidxRETGvcTXr2u4fr0E27ZZn7u4qDRsmDyOuGFDE+7uuXsuQuQavR6zvz9mf3+SRo8GNX994yEBWAghhHgMen3yWOJRo5JQVbh8WeHAAR0HD2o5cEDH6dMZ39EfG6uwa5d1lTsAjUalVi3rsIkmTazjiStUyF8hQohMy2fTpEgAFkIIIXKQokCVKipVqhgZMMAIQFQUhIRYA3FwsHXYREZTsFksCsePazl+XMucOdayChUsth7iZs1M+PlZ8luuEKJAkAAshBBC5DIPD+jUyUSnTtZhE0lJcOyYlvXr73LunBcHD2q5dSvjYRNXr2q4etXAypXW52XLWmjXzroASNu2JooXlx5iITJDArAQQgjxhBkM0LChmWLFIvD1dUdV4eJFDQcOaG1DJ0JDMx42cf26hkWLDCxaZECjsU691q6diQ4dTPj7mzO7loIQRY4EYCGEECKPKQp4e1vw9rYwaJB12MSdOwoHD2r/fViHTSQmpj3ewWJROHRIx6FDOr74Ajw8rL3DD3qIy5aV3mEhHpAALIQQQuRDJUqodO1qomtX67CJxET46y9rIN63zzrHcGxs2oE4KkrDqlUGVq0yAFCrlpn27U20b28kIMCMg8MTOhEh8iEJwEIIIUQB4ODAv8stmxk7NonERDh4UMv27Xq2bdNx8mT64x1OntRy8qSW7793wMVFpUUL61CJDh1MPPWU5QmdhRD5gwRgIYQQogBycIBWrcy0amVmyhS4fl1hxw4d27fr2LFDR1RU2jfVxcYqbN6sZ/Nm69LN3t4PeodNtGxpwsXlSZ2FEHlDArAQQghRCJQtqzJ4sJHBg42YzXD0qJZt26yB+M8/tVgsaQ+XuHBBy4ULWubMccBgUGna1Ez79kbat5ep1kThJAFYCCGEKGS0WussEw0bmpkwIZG7dxV27tTZAvGNG2n3DiclJS/I8dFHyVOtdehgonlzE2XKyM10ouCTACyEEEIUcsWLqzz7rJFnnzWiqnDqlIbt23Vs26YnOFiL0Zh2F+/DU60BlCxpoWZNCzVrmvHzs1CjhpmaNc14eDyhkxEiB0gAFkIIIYoQRYFatSzUqpXE2LFJxMTA3r26fwOxjosX07+ZLjJSw969GvbutY8Q5cpZQ3HNmtZQ7OdnoXp1s4wnFvmSBGAhhBCiCHN1hS5dTHTpYgLgwgWNLQxnNNXaw65d03Dtmobt25PLFEWlcuXkHuMH//X1tcg0bCJPSQAWQgghhI11QY4kXn3VOtXagQPWqdZ279Zy5oyWhITM3xGnqgqXLmm5dEnLxo16W7lWq+LjY6FGjQfB2Npj/NRTFnSSTMQTIJeZEEIIIVLl4ACtW5tp3doMgNkMly9rOHVKw+nTWk6ftv43LEyDyZT5YGw2K4SGWpd7XrMmORg7OKj4+lrw80vuLS5eXKKKyHlyVQkhhBAiU7Ta5CWbn3nGZCtPSoLz561h+EE4PnNGw4ULGlQ188E4MVHhxAktJ04kj0PWaOrSrJmZXr2M9OhhxNNTZqEQj08CsBBCCCEei8HAvz22Fnr3Ti6Pi4OzZx/0FltD8alTWq5eTXsatkdZLAp791rHI48f72gXhr28JAyL7JEALIQQQohc4ewM9epZqFfPAhht5dHREBqaPITiwXCKmzfTD8aqqrBvn459+3S8+64jAQHWMNyzp5GyZSUMi8yTACyEEEKIJ6pYMWjc2EzjxmYeDsa3byucPq3hzBlrIA4J0dkNh3iYqioEB+sIDtYxYYI1DAcGWsNwuXIShkX6JAALIYQQIl8oVUqlZUszLVuabWU7dlzhr798Wb1az7FjmQnDTgQEmGxhuHx5CcMipcwPwhFCCCGEeMIqVkzk7bcT2b07hiNH7jN5cgJ165rTfc2BAzomTnSiVi13Ond2YdYsA1evZv5mPFH4SQAWQgghRIHg7W3hrbcS2bUrhr/+useUKfHUr29K9zUHD+p4/30nnn7anY4dXZgxw0B4uIThok4CsBBCCCEKnCpVVN54I4mgoFj++usen3wSj79/+mH40CEdkyY5Ubu2Ox06uPDDDwauXJEwXBRJABZCCCFEgValisrYsUns2BHL33/f49NP42nQIP0wfPiwjg8/dKJOHXfat7eG4cuXJQwXFRKAhRBCCFFoVK6s8vrrSWzfHsuxY/eYOjWeRo3SD8N//mkNw3XrutOunQvTpjmwebOOW7ckEBdWMguEEEIIIQqlSpVUxoxJYsyYJMLDFdau1bNmjZ6QkLTjz5EjOo4cSd5eoYIFf38z/v4m6tc3U6+emWLFnkTtRW6SACyEEEKIQq9iRZXRo5MYPTqJq1eTw/DBg+lHoatXNVy9qmHtWr2tzNfXTP365n+DsZnatc04OeX2GYicJAFYCCGEEEVKhQoqo0YlMWpUEv/8kxyGDxzIXCwKC9MSFqZl2TLrc51OpWZN+57imjUt6PXpH0fkHQnAQgghhCiyypdXGTkyiZEjk7h2TWHHDh1Hj2o5ckTLiRNajMaMxwGbTArHj2s5flzLb78ZAHB0VKlTx76nuGpVCxq5+ypfkAAshBBCCAGUK6fy/PNGnn/eujxzYiKcPGkNw0eOaDl6VEtoqAaLJeNQnJCgEBKisxtv7O6uUreu2a6nuGJFFUXutXviJAALIYQQQqTCwQFb7+0DMTHw99/JgfjIES2XLqW+RPOj7t1T2LNHx549OsABgNKlLdSvb8bX10LFig8/VDw8JBznFgnAQgghhBCZ5OoKzZubad48ORTfuaPYwvCDYHzjRubGOty6pWHLFg1btqT2XqotEFeoYA3FyT9b8PJSc+q0ihwJwEIIIYQQj6FECZX27U20b5883/C1a4pdL/HRo1qiorI2ADgmRuH0aS2nT6few6zTqZQpU5unntKm6D1+EJQdHR/r1AotCcBCCCGEEDmsXDmVcuVMPPOMNRSrKly8qLHrJf77by1xcdkf42AyKVy75sC1a2nvU6aMfe/xg2BcqZKFGjWK7kwVEoCFEEIIIXKZooC3twVvbwt9+lhvsjObITRUw/HjWsLDNYSHa7h6VbH9HB//+AOAb97UcPOmhiNHUm4rVkylWzcjvXoZadPGhIPDY79dgSEBWAghhBAiD2i14Odnwc/PkmKbqkJk5IMwnByKrSHZWnbnzuPNqRYdrbBkiYElSwy4u6t07WokMNBIu3amQj90QgKwEEIIIUQ+oyhQqpRKqVJm6tdPfZ+YGNi3LxyN5ilbKH44JF+7pmRqyjawzlCxdKmBpUsNuLmpdOlipGdPIx06mArlKncSgIUQQgghCiBXV/D2TsDX15TqdqPRejNecq+xxtaj/Pff2jR7kO/fV1i+3MDy5QZcXFQ6d7b2DHfsaMLZOTfP6MmRACyEEEIIUQjp9VC5skrlymbAbLfNZIJ9+7SsWaNn3To9t26lHoZjYxVWrTKwapUBZ2eVTp2MBAaa6NTJiIvLEziJXCIL8gkhhBBCFDE6HbRubeabbxI4c+Y+69bF8MoriXh6phyP/EBcnMLq1QaGDXPGx8edIUOcWblSz/37T7DiOUQCsBBCCCFEEabVQsuWZr7+OoFTp+7zxx8xDB+eSNmyaYfh+HiFdev0vPyyNQwPGuTMsmV67t17ghV/DBKAhRBCCCEEYA3DzZqZ+fLLBE6evM+mTTGMHJlI+fJph+HERIU//tAzfLg1DA8Y4MySJXqiop5cvbNKArAQQgghhEhBo4GAADPTpiVw/Ph9tm6NYcyYRCpWTDsMJyUpbNqkZ+RIZ3x93enXz5mFC/Xcvfv4cxrnJAnAQgghhBAiXRoNNGpkZurUBI4du8+OHTGMHZtI5cpph2GjUWHLFj1jxjjj6+tG377OmFKfsOKJkwAshBBCCCEyTVHA39/MJ58k8Ndf99m58z5vvZXAU0+Z03yNyaQQE6Ogyyfzj0kAFkIIIYQQ2aIoUK+ehcmTEzlyJIbdu+8zblwCPj4pw3CvXsY8qGHqJAALIYQQQojHpihQp46FSZMSOXQohn377vPuuwlUr24Nwz175p8AnE86ooUQQgghRGGhKFCrloVatRJ5//1ELl1SKFtWzetq2UgPsBBCCCGEyFVVquSf8AsSgIUQQgghRBEjAVgIIYQQQhQpEoCFEEIIIUSRIgFYCCGEEEIUKRKAhRBCCCFEkaJERUXlr9vyhBBCCCGEyEXSAyyEEEIIIYoUCcBCCCGEEKJIkQAshBBCCCGKFAnAQgghhBCiSJEALIQQQgghihQJwAXY3LlzqVOnDp6enrRu3Zr9+/enu//Jkyfp1q0bXl5e1KxZky+++AJVLZyTgHzzzTe0bduWihUrUrVqVfr378+pU6fSfc3ly5fx8PBI8di2bdsTqvWTM23atBTnWa1atXRfU5SuH4DatWunej3069cv1f0L+/Wzb98+BgwYQM2aNfHw8GDRokV221VVZdq0adSoUQMvLy+6d+/O6dOnMzzu3r17ad26NZ6entStW5d58+bl1inkqvTax2g0MnnyZJo1a0a5cuWoXr06r7zyCuHh4ekec8+ePaleU2fPns3t08kVGV1DI0eOTHGuHTp0yPC4ReEaAlK9Fjw8PBg3blyaxyxs11BO0uV1BUT2rFq1igkTJjB9+nQCAgKYO3cuffv25cCBA1SsWDHF/vfu3ePZZ5+lWbNm7Nixg7CwMEaPHo2zszOvv/56HpxB7tq7dy8vv/wy/v7+qKrKZ599Rq9evTh48CDFixdP97UrV67k6aeftj3PaP+CytfXl/Xr19uea7XaNPctatcPQFBQEGaz2fb8xo0btGnThl69eqX7usJ6/cTGxuLn58fAgQN57bXXUmz/7rvvmDlzJjNnzsTX15cvv/ySZ599lkOHDuHm5pbqMS9dukS/fv0YPHgwP//8MwcOHOCdd96hZMmSBAYG5vYp5aj02icuLo6///6bcePGUbt2be7du8ekSZPo06cP+/btQ6dL/5/iAwcO2F1HpUqVypVzyG0ZXUMAbdq04aeffrI9NxgM6R6zqFxDAKGhoXbPjx49yoABAzL8TILCcw3lJAnABdTMmTMZNGgQL774IgBfffUV27dvZ968eUyePDnF/suXLyc+Pp7Zs2fj5OSEn58fZ8+eZdasWYwZMwZFUZ70KeSqVatW2T3/6aefqFSpEgcOHKBr167pvrZEiRJ4enrmZvXyBZ1Ol+nzLGrXD6T8B2LBggW4ubll+I9NYb1+OnXqRKdOnQAYNWqU3TZVVZk9ezZvvvmmLXTMnj0bX19fVqxYwbBhw1I95q+//oqXlxdfffUVANWrV+fw4cPMmDGjwIWX9NqnWLFirF692q7sv//9LwEBAYSGhlKrVq10j126dGlKliyZo/XNC+m10QMODg5Z+v0pKtcQkKJd/vjjD3x8fGjRokWGxy4s11BOkiEQBVBSUhJ//fUX7dq1sytv164dBw8eTPU1ISEhNG3aFCcnJ1tZ+/btuX79OpcvX87V+uYHMTExWCwWPDw8Mtx3yJAh+Pj40LlzZ9asWZP7lcsjly5dombNmtSpU4eXXnqJS5cupblvUb9+VFVlwYIF9O/fH2dn53T3LSrXz8MuX75MRESE3WeSk5MTzZo1S/MzCazX1aOfY+3bt+fo0aMYjcZcq29+cP/+fYBMfSa1adOG6tWr07NnT3bv3p3LNctbwcHB+Pj40KBBA8aOHcutW7fS3b+oXkMxMTGsWrXK1gmWkaJ0DWWWBOACKDIyErPZTOnSpe3KS5cuzc2bN1N9zc2bN1Pd/8G2wm7ChAnUrl2bxo0bp7mPq6srn376Kb/++ivLly+nVatWDBs2jKVLlz7Bmj4ZDRs2ZNasWSxfvpzvv/+eiIgIOnXqxJ07d1Ldv6hfP0FBQVy+fJkhQ4akuU9Run4eFRERAZClzyRI+7oymUxERkbmfEXziaSkJCZNmkSXLl0oX758mvt5eXnxzTffsGDBAhYsWICvry+BgYHs27fvCdb2yenQoQM//vgja9asYerUqfz555/07NmTxMTENF9TVK+hFStWkJiYyMCBA9Pdr6hdQ1khQyAKsEe/dlZVNd2volPbP7Xywub999/nwIEDbNq0Kd1xriVLlrQbz1q/fn3u3LnDd999R//+/Z9EVZ+Yjh072j1v2LAh9erVY/HixYwZMybV1xTV6wfgt99+w9/fnzp16qS5T1G6ftKS1c+ktF6TWnlhYTKZGD58ONHR0SxZsiTdfX19ffH19bU9b9y4MVeuXOGHH36gefPmuV3VJ+65556z/VyrVi3q1atH7dq12bx5Mz179kzzdUXtGgLrZ1L37t0zHMtb1K6hrJAe4AKoZMmSaLXaFD0rt2/fTvGX8ANlypRJdX9I2WtTmEycOJGVK1eydu1aqlSpkuXXN2jQgAsXLuR8xfIZV1dXatSokea5FtXrB+DWrVv88ccfmf6q8WFF5fp5MDYxK59JkPZ1pdPpKFGiRM5XNI+ZTCZefvllTp48yZo1a7J1jkXlmgIoW7Ys5cqVS/d8i9o1BHDs2DGOHj2arc8kKFrXUHokABdABoOBevXqERQUZFceFBREkyZNUn1N48aNCQ4OJiEhwW7/smXLUrly5Vytb1557733WLFiBWvXrs1wiq+0HD9+vFDe0PSohIQEwsLC0jzXonj9PLB48WIcHBzo3bt3ll9bVK6fypUr4+npafeZlJCQQHBwcJqfSWC9rnbu3GlXFhQURP369dHr9blV3TxhNBoZNmwYJ0+eZN26ddm+LorKNQXW4X7Xr19P93yL0jX0wG+//UalSpVo06ZNtl5flK6h9MgQiAJq9OjRjBgxggYNGtCkSRPmzZvHjRs3bHdbT5kyhT///JO1a9cC0KdPH7744gtGjRrFuHHjOHfuHN9++y3vvvtuofyaaNy4cSxdupSFCxfi4eFhG6Po4uKCq6srkLKNFi9ejF6vp06dOmg0GjZt2sTcuXP5+OOP8+o0cs2D8YcVKlTg9u3bfPXVV8TFxdnGkxX16+cBVVWZP38+vXv3TjGVV1G7fmJiYmy9RhaLhatXr3Ls2DGKFy9OxYoVGTlyJNOnT8fX1xcfHx++/vprXFxc6NOnj+0YI0aMALBNczVs2DDmzJnDhAkTGDZsGAcPHmTx4sXMnTv3yZ/gY0qvfcqWLcuLL77I0aNHWbJkCYqi2D6T3N3dbTeXPto+s2bNolKlStSsWZOkpCSWLVvGhg0bmD9/fh6c4eNLr42KFy/O559/Ts+ePfH09OTKlSt88sknlC5dmmeeecZ2jKJ6DT2Y3jQuLo7ly5czduzYVD97C/s1lJMkABdQvXv35s6dO3z11VdERERQs2ZNli1bRqVKlQDrnKUXL1607V+sWDF+//13xo0bR9u2bfHw8GD06NFpjvcs6B58+D06Dc57773HxIkTgZRtBPD1118THh6OVqulatWqzJgxo1CO37x27RqvvPIKkZGRlCpVioYNG7J161a5fh6xZ88ezp8/z88//5xiW1G7fo4ePUqPHj1sz6dNm8a0adMYOHAgs2fP5o033iA+Pp7x48cTFRVFgwYNWLVqld0fDlevXrU7ZpUqVVi2bBnvv/8+8+bNw8vLiy+++KLATV8F6bfPhAkT+OOPPwBS9NrNnDmTwYMHAynbx2g08uGHH3L9+nUcHR1tn/MPpsoqaNJro2+++YZTp07xv//9j+joaDw9PWnZsiW//vqrXEP//o6BdYrP2NhY2zXzqMJ+DeUkJSoqqvAu5SSEEEIIIcQjZAywEEIIIYQoUiQACyGEEEKIIkUCsBBCCCGEKFIkAAshhBBCiCJFArAQQgghhChSJAALIYQQQogiRQKwEEKIHOXh4cFbb72V19UQQog0SQAWQogCZtGiRXh4eKT52LRpU15XUQgh8jVZCU4IIQqoCRMm8NRTT6Uor1OnTh7URgghCg4JwEIIUUC1b9+eRo0a5XU1hBCiwJEhEEIIUUg9GIu7atUqmjRpgqenJ82aNWPz5s0p9g0PD+fVV1/F29sbT09PWrRowZIlS1Lsp6oqc+bMoUWLFnh5eeHt7U2vXr3Yv39/in23bt1Ky5Yt8fT0xN/fnxUrVthtN5lMfPXVVzRo0MB2rE6dOrFmzZqcawQhhEiF9AALIUQBde/ePSIjI1OUlyxZ0vbzwYMH+f333xkxYgSurq789ttvDB48mDVr1tC8eXMAIiMj6dKlC3fv3mX48OF4eXmxatUqRo4cSVRUFCNHjrQd74033mD+/Pm0adOGQYMGoaoqISEhBAcH06xZM9t+hw4dYsOGDQwbNowhQ4Ywf/58hg8fTu3atalevToAn3/+OdOnT2fIkCE0aNCA2NhYjh07xuHDhwkMDMytZhNCCJSoqCg1ryshhBAi8xYtWsTo0aPT3H716lVcXV3x8PAAYPPmzTRp0gSAO3fu4O/vT7Vq1diyZQsAkyZNYsaMGaxZs4bWrVsDkJSURNeuXTlz5gynTp2iWLFi7Nmzhx49evDiiy/y3Xff2b2nqqooigJYe551Oh379u2zhd2bN2/y9NNPM2LECD799FMAWrZsSbly5Vi6dGnONY4QQmSC9AALIUQB9cUXX9gC5sOcnJxsP9evX98WfgFKlChB3759mTNnDlFRUXh4eLB582bq1KljC78ABoOBkSNH8sorr7B37166d+/O2rVrAWtgftSD8PtAy5Yt7epWpkwZfH19uXTpkq3Mzc2N06dPc+7cOXx8fLLeAEIIkU0SgIUQooDy9/fP8Ca4qlWrplkWHh6Oh4cHV65coUePHin2exBgr1y5AsDFixcpXbo0pUuXzrBuFStWTFHm4eHB3bt3bc8nTpzI888/T8OGDalRowbt2rWjT58++Pv7Z3h8IYR4HHITnBBCFGKP9syCdbhCZjy638PDHDKi1WozPGbLli35+++/mT17NnXq1OF///sf7du355tvvsnUewghRHZJABZCiELs3LlzKcouXLgAJPfSVqpUibNnz6bYLywszLYdwNvbm5s3b3Lr1q0cq5+HhwcDBw7k559/5uTJkzRr1owvvvgCs9mcY+8hhBCPkgAshBCF2NGjRwkJCbE9v3PnDsuXL6dRo0a2m+Q6d+7MsWPH2L17t20/o9HIjz/+iLOzMy1atACgZ8+eAHz22Wcp3iezvcoPu3Pnjt1zJycnqlevTmJiInFxcVk+nhBCZJaMARZCiAJq+/bttt7ch9WrV882ftfPz4/+/fszfPhw2zRo9+/f56OPPrLt/2Cu4IEDBzJixAg8PT35/fffOXToEJ999hnFihUDrEMWBg0axK+//sqlS5fo1KkTYJ3yrFatWrzzzjtZqn/jxo1p1qwZ/v7+lChRghMnTjB//nw6d+6Mm5tbdptFCCEyJAFYCCEKqM8//zzV8k8//dQWgJs0aULLli35/PPPuXTpElWrVmXhwoW0bNnStn/JkiXZvHkzU6ZM4ddffyUuLg4fHx9mz57NwIED7Y49Y8YMatWqxYIFC5g8eTKurq7UrVvXNqdwVowcOZKNGzeye/duEhISKF++PG+++SZvvvlmlo8lhBBZIfMACyFEIeXh4cGwYcP473//m9dVEUKIfEXGAAshhBBCiCJFArAQQgghhChSJAALIYQQQogiRW6CE0KIQioqKiqvqyCEEPmS9AALIYQQQogiRQKwEEIIIYQoUiQACyGEEEKIIkUCsBBCCCGEKFIkAAshhBBCiCJFArAQQgghhChS/h8DsUJduZiQ5gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = sbs_doc_emb.plot_losses()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[424, 440],\n",
       "        [310, 331]])"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "StepByStep.loader_apply(test_loader, sbs_doc_emb.correct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'transformers.models.bert.modeling_bert.BertModel'>\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModel\n",
    "auto_model = AutoModel.from_pretrained('bert-base-uncased')\n",
    "print(auto_model.__class__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertModel\n",
    "bert_model = BertModel.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertConfig {\n",
       "  \"_name_or_path\": \"bert-base-uncased\",\n",
       "  \"architectures\": [\n",
       "    \"BertForMaskedLM\"\n",
       "  ],\n",
       "  \"attention_probs_dropout_prob\": 0.1,\n",
       "  \"gradient_checkpointing\": false,\n",
       "  \"hidden_act\": \"gelu\",\n",
       "  \"hidden_dropout_prob\": 0.1,\n",
       "  \"hidden_size\": 768,\n",
       "  \"initializer_range\": 0.02,\n",
       "  \"intermediate_size\": 3072,\n",
       "  \"layer_norm_eps\": 1e-12,\n",
       "  \"max_position_embeddings\": 512,\n",
       "  \"model_type\": \"bert\",\n",
       "  \"num_attention_heads\": 12,\n",
       "  \"num_hidden_layers\": 12,\n",
       "  \"pad_token_id\": 0,\n",
       "  \"position_embedding_type\": \"absolute\",\n",
       "  \"transformers_version\": \"4.5.1\",\n",
       "  \"type_vocab_size\": 2,\n",
       "  \"use_cache\": true,\n",
       "  \"vocab_size\": 30522\n",
       "}"
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bert_model.config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30522"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import BertTokenizer\n",
    "bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "len(bert_tokenizer.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  5650,  2003,  1999, 10288, 24759,  5555,  6321,  2206,  1996,\n",
       "          2317, 10442,   102,  3582,  1996,  2317, 10442,  1010,  9253,   102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence1 = 'Alice is inexplicably following the white rabbit'\n",
    "sentence2 = 'Follow the white rabbit, Neo'\n",
    "tokens = bert_tokenizer(sentence1, sentence2, return_tensors='pt')\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'alice', 'is', 'in', '##ex', '##pl', '##ica', '##bly', 'following', 'the', 'white', 'rabbit', '[SEP]', 'follow', 'the', 'white', 'rabbit', ',', 'neo', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "print(bert_tokenizer.convert_ids_to_tokens(tokens['input_ids'][0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'transformers.models.bert.tokenization_bert_fast.BertTokenizerFast'>\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "auto_tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')\n",
    "print(auto_tokenizer.__class__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input Embeddings\n",
    "\n",
    "![](images/bert_input_embed.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertEmbeddings(\n",
       "  (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "  (position_embeddings): Embedding(512, 768)\n",
       "  (token_type_embeddings): Embedding(2, 768)\n",
       "  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "  (dropout): Dropout(p=0.1, inplace=False)\n",
       ")"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_embeddings = bert_model.embeddings\n",
    "input_embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(30522, 768, padding_idx=0)"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_embeddings = input_embeddings.word_embeddings\n",
    "token_embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 1.3630e-02, -2.6490e-02, -2.3503e-02,  ...,  8.6805e-03,\n",
       "           7.1340e-03,  1.5147e-02],\n",
       "         [-6.9710e-02, -8.8202e-02,  5.0619e-03,  ...,  1.4105e-02,\n",
       "           2.1815e-02, -1.3769e-02],\n",
       "         [-3.6044e-02, -2.4606e-02, -2.5735e-02,  ...,  3.3691e-03,\n",
       "          -1.8300e-03,  2.6855e-02],\n",
       "         ...,\n",
       "         [ 5.2089e-05, -1.0468e-02, -9.9103e-03,  ...,  1.4558e-02,\n",
       "           1.3217e-02,  2.2406e-02],\n",
       "         [-3.5037e-02, -7.2933e-02, -3.6124e-02,  ..., -5.7723e-02,\n",
       "          -5.5074e-03,  7.2688e-03],\n",
       "         [-1.4521e-02, -9.9615e-03,  6.0263e-03,  ..., -2.5035e-02,\n",
       "           4.6379e-03, -1.5378e-03]]], grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_token_emb = token_embeddings(tokens['input_ids'])\n",
    "input_token_emb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(512, 768)"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "position_embeddings = input_embeddings.position_embeddings\n",
    "position_embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,\n",
       "          14,  15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,  26,  27,\n",
       "          28,  29,  30,  31,  32,  33,  34,  35,  36,  37,  38,  39,  40,  41,\n",
       "          42,  43,  44,  45,  46,  47,  48,  49,  50,  51,  52,  53,  54,  55,\n",
       "          56,  57,  58,  59,  60,  61,  62,  63,  64,  65,  66,  67,  68,  69,\n",
       "          70,  71,  72,  73,  74,  75,  76,  77,  78,  79,  80,  81,  82,  83,\n",
       "          84,  85,  86,  87,  88,  89,  90,  91,  92,  93,  94,  95,  96,  97,\n",
       "          98,  99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111,\n",
       "         112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125,\n",
       "         126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139,\n",
       "         140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153,\n",
       "         154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167,\n",
       "         168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,\n",
       "         182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195,\n",
       "         196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209,\n",
       "         210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223,\n",
       "         224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237,\n",
       "         238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251,\n",
       "         252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265,\n",
       "         266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279,\n",
       "         280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293,\n",
       "         294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307,\n",
       "         308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321,\n",
       "         322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335,\n",
       "         336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349,\n",
       "         350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363,\n",
       "         364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377,\n",
       "         378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391,\n",
       "         392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405,\n",
       "         406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419,\n",
       "         420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433,\n",
       "         434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447,\n",
       "         448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461,\n",
       "         462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475,\n",
       "         476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489,\n",
       "         490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503,\n",
       "         504, 505, 506, 507, 508, 509, 510, 511]])"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "position_ids = torch.arange(512).expand((1, -1))\n",
    "position_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 1.7505e-02, -2.5631e-02, -3.6642e-02,  ...,  3.3437e-05,\n",
       "           6.8312e-04,  1.5441e-02],\n",
       "         [ 7.7580e-03,  2.2613e-03, -1.9444e-02,  ...,  2.8910e-02,\n",
       "           2.9753e-02, -5.3247e-03],\n",
       "         [-1.1287e-02, -1.9644e-03, -1.1573e-02,  ...,  1.4908e-02,\n",
       "           1.8741e-02, -7.3140e-03],\n",
       "         ...,\n",
       "         [-9.2809e-03,  8.3268e-03, -4.1643e-03,  ...,  3.4903e-02,\n",
       "          -1.8319e-02, -2.9017e-03],\n",
       "         [-8.5999e-03,  3.2205e-04, -2.1249e-03,  ...,  2.7744e-02,\n",
       "          -7.2760e-03, -2.0280e-03],\n",
       "         [-3.4622e-04, -8.3709e-04, -2.2228e-02,  ...,  2.3493e-02,\n",
       "          -4.5198e-04, -5.7741e-04]]], grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seq_length = tokens['input_ids'].size(1)\n",
    "input_pos_emb = position_embeddings(position_ids[:, :seq_length])\n",
    "input_pos_emb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(2, 768)"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "segment_embeddings = input_embeddings.token_type_embeddings\n",
    "segment_embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.0004,  0.0110,  0.0037,  ..., -0.0066, -0.0034, -0.0086],\n",
       "         [ 0.0004,  0.0110,  0.0037,  ..., -0.0066, -0.0034, -0.0086],\n",
       "         [ 0.0004,  0.0110,  0.0037,  ..., -0.0066, -0.0034, -0.0086],\n",
       "         ...,\n",
       "         [ 0.0011, -0.0030, -0.0032,  ...,  0.0047, -0.0052, -0.0112],\n",
       "         [ 0.0011, -0.0030, -0.0032,  ...,  0.0047, -0.0052, -0.0112],\n",
       "         [ 0.0011, -0.0030, -0.0032,  ...,  0.0047, -0.0052, -0.0112]]],\n",
       "       grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_seg_emb = segment_embeddings(tokens['token_type_ids'])\n",
    "input_seg_emb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.0316, -0.0411, -0.0564,  ...,  0.0021,  0.0044,  0.0219],\n",
       "         [-0.0615, -0.0750, -0.0107,  ...,  0.0364,  0.0482, -0.0277],\n",
       "         [-0.0469, -0.0156, -0.0336,  ...,  0.0117,  0.0135,  0.0109],\n",
       "         ...,\n",
       "         [-0.0081, -0.0051, -0.0172,  ...,  0.0542, -0.0103,  0.0083],\n",
       "         [-0.0425, -0.0756, -0.0414,  ..., -0.0252, -0.0180, -0.0060],\n",
       "         [-0.0138, -0.0138, -0.0194,  ...,  0.0032, -0.0011, -0.0133]]],\n",
       "       grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_emb = input_token_emb + input_pos_emb + input_seg_emb\n",
    "input_emb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pretraining Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Masked Language Model (MLM)\n",
    "\n",
    "![](images/bert_mlm.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[101, 5650, 2003, 1999, 10288, 24759, 5555, 6321, 2206, 1996, 2317, 10442, 102]"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = 'Alice is inexplicably following the white rabbit'\n",
    "tokens = bert_tokenizer(sentence)\n",
    "tokens['input_ids']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  5650,  2003,  1999, 10288, 24759,   103,  6321,  2206,  1996,\n",
       "          2317, 10442,   102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]), 'labels': tensor([[-100, -100, -100, -100, -100, -100, 5555, -100, -100, -100, -100, -100,\n",
       "         -100]])}"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import DataCollatorForLanguageModeling\n",
    "torch.manual_seed(41)\n",
    "data_collator = DataCollatorForLanguageModeling(tokenizer=bert_tokenizer, mlm_probability=0.15)\n",
    "mlm_tokens = data_collator([tokens])\n",
    "mlm_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'alice', 'is', 'in', '##ex', '##pl', '##ica', '##bly', 'following', 'the', 'white', 'rabbit', '[SEP]']\n",
      "['[CLS]', 'alice', 'is', 'in', '##ex', '##pl', '[MASK]', '##bly', 'following', 'the', 'white', 'rabbit', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "print(bert_tokenizer.convert_ids_to_tokens(tokens['input_ids']))\n",
    "print(bert_tokenizer.convert_ids_to_tokens(mlm_tokens['input_ids'][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next Sentence Prediction (NSP)\n",
    "\n",
    "![](images/bert_nsp.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertPooler(\n",
       "  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "  (activation): Tanh()\n",
       ")"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bert_model.pooler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  5650,  4076,  1996,  2317, 10442,   102,  3582,  1996,  2317,\n",
       "         10442,  9253,   102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence1 = 'alice follows the white rabbit'\n",
    "sentence2 = 'follow the white rabbit neo'\n",
    "bert_tokenizer(sentence1, sentence2, return_tensors='pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'And, so far as they knew, they were quite right.'"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = train_dataset[0]['sentence']\n",
    "sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[ 101, 1998, 1010, 2061, 2521, 2004, 2027, 2354, 1010, 2027, 2020, 3243,\n",
       "         2157, 1012,  102,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "            0,    0,    0,    0,    0,    0]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0]])}"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = bert_tokenizer(sentence,\n",
    "                        padding='max_length',\n",
    "                        max_length=30, \n",
    "                        truncation=True, \n",
    "                        return_tensors=\"pt\")\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['last_hidden_state', 'pooler_output', 'hidden_states', 'attentions'])"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bert_model.eval()\n",
    "out = bert_model(input_ids=tokens['input_ids'], \n",
    "                 attention_mask=tokens['attention_mask'], \n",
    "                 output_attentions=True,\n",
    "                 output_hidden_states=True,\n",
    "                 return_dict=True)\n",
    "out.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/bert_embeddings.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0100,  0.8575, -0.5429,  ...,  0.1661,  0.4241, -0.2035],\n",
       "        [-0.3705,  1.1001,  0.3326,  ..., -0.4595,  0.0656, -0.5644],\n",
       "        [-0.2947,  0.5797,  0.1997,  ..., -0.2929, -0.3062,  0.6690],\n",
       "        ...,\n",
       "        [ 0.0691,  0.7393,  0.0553,  ..., -0.1853, -0.4896, -0.4832],\n",
       "        [-0.1566,  0.6177,  0.1536,  ..., -0.1396,  0.0904, -0.4917],\n",
       "        [ 0.7511,  0.3110, -0.3116,  ...,  0.3498, -0.1740, -0.2337]],\n",
       "       grad_fn=<SliceBackward>)"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_hidden_batch = out['last_hidden_state']\n",
    "last_hidden_sentence = last_hidden_batch[0]\n",
    "# Removes hidden states for [PAD] tokens using the mask\n",
    "mask = tokens['attention_mask'].squeeze().bool()\n",
    "embeddings = last_hidden_sentence[mask]\n",
    "# Removes embeddings for the first [CLS] and last [SEP] tokens\n",
    "embeddings[1:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0100,  0.8575, -0.5429,  ...,  0.1661,  0.4241, -0.2035],\n",
       "        [-0.3705,  1.1001,  0.3326,  ..., -0.4595,  0.0656, -0.5644],\n",
       "        [-0.2947,  0.5797,  0.1997,  ..., -0.2929, -0.3062,  0.6690],\n",
       "        ...,\n",
       "        [ 0.0691,  0.7393,  0.0552,  ..., -0.1853, -0.4896, -0.4832],\n",
       "        [-0.1566,  0.6177,  0.1536,  ..., -0.1396,  0.0904, -0.4917],\n",
       "        [ 0.7511,  0.3110, -0.3116,  ...,  0.3498, -0.1740, -0.2337]],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_embeddings(bert_flair, sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13\n",
      "torch.Size([1, 30, 768])\n"
     ]
    }
   ],
   "source": [
    "print(len(out['hidden_states']))\n",
    "print(out['hidden_states'][0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True)"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(out['hidden_states'][0] == bert_model.embeddings(tokens['input_ids'])).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True)"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(out['hidden_states'][-1] == out['last_hidden_state']).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True)"
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(out['pooler_output'] == bert_model.pooler(out['last_hidden_state'])).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12\n",
      "torch.Size([1, 12, 30, 30])\n"
     ]
    }
   ],
   "source": [
    "print(len(out['attentions']))\n",
    "print(out['attentions'][0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model IV - Classifying using BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BERTClassifier(nn.Module):\n",
    "    def __init__(self, bert_model, ff_units, n_outputs, dropout=0.3):\n",
    "        super().__init__()\n",
    "        self.d_model = bert_model.config.dim\n",
    "        self.n_outputs = n_outputs\n",
    "        self.encoder = bert_model\n",
    "        self.mlp = nn.Sequential(\n",
    "            nn.Linear(self.d_model, ff_units),\n",
    "            nn.ReLU(),\n",
    "            nn.Dropout(dropout),\n",
    "            nn.Linear(ff_units, n_outputs)\n",
    "        )\n",
    "        \n",
    "    def encode(self, source, source_mask=None):\n",
    "        states = self.encoder(input_ids=source, \n",
    "                              attention_mask=source_mask)[0]\n",
    "        cls_state = states[:, 0]\n",
    "        return cls_state\n",
    "\n",
    "    def forward(self, X):\n",
    "        source_mask = (X > 0)\n",
    "        # Featurizer\n",
    "        cls_state = self.encode(X, source_mask)\n",
    "        # Classifier\n",
    "        out = self.mlp(cls_state)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_dataset(hf_dataset, sentence_field, label_field, tokenizer, **kwargs):\n",
    "    sentences = hf_dataset[sentence_field]\n",
    "    token_ids = tokenizer(sentences, return_tensors='pt', **kwargs)['input_ids']\n",
    "    labels = torch.as_tensor(hf_dataset[label_field])\n",
    "    dataset = TensorDataset(token_ids, labels)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [],
   "source": [
    "auto_tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')\n",
    "tokenizer_kwargs = dict(truncation=True, padding=True, max_length=30, add_special_tokens=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3668556b9d86415c98ee39326fd90b63",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=3081.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "be3af643eece4a8db360e7fa2e18d8df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=771.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "train_dataset_float = train_dataset.map(lambda row: {'labels': [float(row['labels'])]})\n",
    "test_dataset_float = test_dataset.map(lambda row: {'labels': [float(row['labels'])]})\n",
    "\n",
    "train_tensor_dataset = tokenize_dataset(train_dataset_float, 'sentence', 'labels', auto_tokenizer, **tokenizer_kwargs)\n",
    "test_tensor_dataset = tokenize_dataset(test_dataset_float, 'sentence', 'labels', auto_tokenizer, **tokenizer_kwargs)\n",
    "\n",
    "generator = torch.Generator()\n",
    "train_loader = DataLoader(train_tensor_dataset, batch_size=4, shuffle=True, generator=generator)\n",
    "test_loader = DataLoader(test_tensor_dataset, batch_size=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Configuration & Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(41)\n",
    "bert_model = AutoModel.from_pretrained(\"distilbert-base-uncased\")\n",
    "model = BERTClassifier(bert_model, 128, n_outputs=1)\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=1e-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [],
   "source": [
    "sbs_bert = StepByStep(model, loss_fn, optimizer)\n",
    "sbs_bert.set_loaders(train_loader, test_loader)\n",
    "sbs_bert.train(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "66461441"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sbs_bert.count_parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[424, 440],\n",
       "        [317, 331]])"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "StepByStep.loader_apply(test_loader, sbs_bert.correct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-Tuning with HuggingFace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequence Classification (or Regression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']\n",
      "- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import DistilBertForSequenceClassification\n",
    "torch.manual_seed(42)\n",
    "bert_cls = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']\n",
      "- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification'>\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForSequenceClassification\n",
    "auto_cls = AutoModelForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=2)\n",
    "print(auto_cls.__class__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenized Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [],
   "source": [
    "auto_tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')\n",
    "def tokenize(row):\n",
    "    return auto_tokenizer(row['sentence'], \n",
    "                          truncation=True, \n",
    "                          padding='max_length',\n",
    "                          max_length=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bef768a50eeb4fba808a0b9cd37369b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=4.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f937f4ab79bc4b8c8cabbfee6bd46d38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenized_train_dataset = train_dataset.map(tokenize, batched=True)\n",
    "tokenized_test_dataset = test_dataset.map(tokenize, batched=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'input_ids': [101, 1998, 1010, 2061, 2521, 2004, 2027, 2354, 1010, 2027, 2020, 3243, 2157, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'labels': 0, 'sentence': 'And, so far as they knew, they were quite right.', 'source': 'wizoz10-1740.txt'}\n"
     ]
    }
   ],
   "source": [
    "print(tokenized_train_dataset[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_train_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask', 'labels'])\n",
    "tokenized_test_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask', 'labels'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0]),\n",
       " 'input_ids': tensor([ 101, 1998, 1010, 2061, 2521, 2004, 2027, 2354, 1010, 2027, 2020, 3243,\n",
       "         2157, 1012,  102,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "            0,    0,    0,    0,    0,    0]),\n",
       " 'labels': tensor(0)}"
      ]
     },
     "execution_count": 217,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_train_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import Trainer\n",
    "trainer = Trainer(model=bert_cls, train_dataset=tokenized_train_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TrainingArguments(output_dir=tmp_trainer, overwrite_output_dir=False, do_train=False, do_eval=None, do_predict=False, evaluation_strategy=IntervalStrategy.NO, prediction_loss_only=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, lr_scheduler_type=SchedulerType.LINEAR, warmup_ratio=0.0, warmup_steps=0, logging_dir=runs/Apr21_20-33-20_MONSTER, logging_strategy=IntervalStrategy.STEPS, logging_first_step=False, logging_steps=500, save_strategy=IntervalStrategy.STEPS, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level=O1, fp16_backend=auto, fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name=tmp_trainer, disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name=length, report_to=['tensorboard'], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, _n_gpu=1, mp_parameters=)"
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.args"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    "training_args = TrainingArguments(\n",
    "    output_dir='output',\n",
    "    num_train_epochs=1,             \n",
    "    per_device_train_batch_size=1,\n",
    "    per_device_eval_batch_size=8,\n",
    "    evaluation_strategy='steps',\n",
    "    eval_steps=300,\n",
    "    logging_steps=300,\n",
    "    gradient_accumulation_steps=8,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(eval_pred):\n",
    "    predictions = eval_pred.predictions\n",
    "    labels = eval_pred.label_ids\n",
    "    predictions = np.argmax(predictions, axis=1)\n",
    "    return {\"accuracy\": (predictions == labels).mean()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = Trainer(model=bert_cls, \n",
    "                  args=training_args, \n",
    "                  train_dataset=tokenized_train_dataset, \n",
    "                  eval_dataset=tokenized_test_dataset,\n",
    "                  compute_metrics=compute_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='385' max='385' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [385/385 01:19, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>0.194600</td>\n",
       "      <td>0.159694</td>\n",
       "      <td>0.953307</td>\n",
       "      <td>1.648500</td>\n",
       "      <td>467.697000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=385, training_loss=0.17661244776341822, metrics={'train_runtime': 80.0324, 'train_samples_per_second': 4.811, 'total_flos': 37119857544000.0, 'epoch': 1.0, 'init_mem_cpu_alloc_delta': 0, 'init_mem_gpu_alloc_delta': 0, 'init_mem_cpu_peaked_delta': 0, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 5025792, 'train_mem_gpu_alloc_delta': 806599168, 'train_mem_cpu_peaked_delta': 0, 'train_mem_gpu_peaked_delta': 96468992})"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='97' max='97' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [97/97 00:01]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.142591193318367,\n",
       " 'eval_accuracy': 0.9610894941634242,\n",
       " 'eval_runtime': 1.6634,\n",
       " 'eval_samples_per_second': 463.51,\n",
       " 'epoch': 1.0,\n",
       " 'eval_mem_cpu_alloc_delta': 0,\n",
       " 'eval_mem_gpu_alloc_delta': 0,\n",
       " 'eval_mem_cpu_peaked_delta': 0,\n",
       " 'eval_mem_gpu_peaked_delta': 8132096}"
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['training_args.bin', 'config.json', 'pytorch_model.bin']"
      ]
     },
     "execution_count": 225,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.save_model('bert_alice_vs_wizard')\n",
    "os.listdir('bert_alice_vs_wizard')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_model = AutoModelForSequenceClassification.from_pretrained('bert_alice_vs_wizard')\n",
    "loaded_model.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda', index=0)"
      ]
     },
     "execution_count": 227,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "loaded_model.to(device)\n",
    "loaded_model.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  2091,  1996,  3756,  5318, 10442,  4920,   102]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = 'Down the yellow brick rabbit hole'\n",
    "tokens = auto_tokenizer(sentence, return_tensors='pt')\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'transformers.tokenization_utils_base.BatchEncoding'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  2091,  1996,  3756,  5318, 10442,  4920,   102]],\n",
       "       device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0')}"
      ]
     },
     "execution_count": 229,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(type(tokens))\n",
    "tokens.to(loaded_model.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SequenceClassifierOutput(loss=None, logits=tensor([[ 2.7745, -2.5539]], device='cuda:0', grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_model.eval()\n",
    "logits = loaded_model(input_ids=tokens['input_ids'], attention_mask=tokens['attention_mask'])\n",
    "logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0], device='cuda:0')"
      ]
     },
     "execution_count": 233,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits.logits.argmax(dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TextClassificationPipeline\n",
    "device_index = loaded_model.device.index if loaded_model.device.type != 'cpu' else -1\n",
    "classifier = TextClassificationPipeline(model=loaded_model,\n",
    "                                        tokenizer=auto_tokenizer, \n",
    "                                        device=device_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'LABEL_0', 'score': 0.9951714277267456},\n",
       " {'label': 'LABEL_1', 'score': 0.9985325336456299}]"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier(['Down the Yellow Brick Rabbit Hole', 'Alice rules!'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_model.config.id2label = {0: 'Wizard', 1: 'Alice'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'Wizard', 'score': 0.9951714277267456},\n",
       " {'label': 'Alice', 'score': 0.9985325336456299}]"
      ]
     },
     "execution_count": 237,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier(['Down the Yellow Brick Rabbit Hole', 'Alice rules!'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More Pipelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f7a51b4024854a9198898c4ca0d1bc1f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=48.0, style=ProgressStyle(description_w…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from transformers import pipeline\n",
    "sentiment = pipeline('sentiment-analysis')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "And, so far as they knew, they were quite right.\n",
      "[{'label': 'POSITIVE', 'score': 0.9998356699943542}]\n"
     ]
    }
   ],
   "source": [
    "sentence = train_dataset[0]['sentence']\n",
    "print(sentence)\n",
    "print(sentiment(sentence))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'impl': transformers.pipelines.text_classification.TextClassificationPipeline,\n",
       " 'tf': None,\n",
       " 'pt': types.AutoModelForSequenceClassification,\n",
       " 'default': {'model': {'pt': 'distilbert-base-uncased-finetuned-sst-2-english',\n",
       "   'tf': 'distilbert-base-uncased-finetuned-sst-2-english'}}}"
      ]
     },
     "execution_count": 240,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers.pipelines import SUPPORTED_TASKS\n",
    "SUPPORTED_TASKS['sentiment-analysis'] # 'text-classification' from v4.6.0 on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'impl': transformers.pipelines.text_generation.TextGenerationPipeline,\n",
       " 'tf': None,\n",
       " 'pt': types.AutoModelForCausalLM,\n",
       " 'default': {'model': {'pt': 'gpt2', 'tf': 'gpt2'}}}"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SUPPORTED_TASKS['text-generation']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GPT-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_generator = pipeline(\"text-generation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text-generation': {'do_sample': True, 'max_length': 50}}"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_generator.model.config.task_specific_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_text = \"\"\"\n",
    "Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do:  once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, `and what is the use of a book,'thought Alice `without pictures or conversation?' So she was considering in her own mind (as well as she could, for the hot day made her feel very sleepy and stupid), whether the pleasure of making a daisy-chain would be worth the trouble of getting up and picking the daisies, when suddenly a White Rabbit with pink eyes ran close by her.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do:  once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, `and what is the use of a book,'thought Alice `without pictures or conversation?' So she was considering in her own mind (as well as she could, for the hot day made her feel very sleepy and stupid), whether the pleasure of making a daisy-chain would be worth the trouble of getting up and picking the daisies, when suddenly a White Rabbit with pink eyes ran close by her.\n",
      "\n",
      "After about a week, when Alice had finished reading about animals and people, the White Rabbit went up to the dining-room door. 'Sir,'she said, `you have to see a White Rabbit!' she said, `and what was it about it that struck you?'\n",
      "\n",
      "Alice smiled. 'It was just to tell you.'\n",
      "\n",
      "'Then I had no idea,' said Alice, `that it was like a book. But it does seem to me (like a book,)a book of dreams. I have tried it for the night. If\n"
     ]
    }
   ],
   "source": [
    "result = text_generator(base_text, max_length=250)\n",
    "print(result[0]['generated_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Putting It All Together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default-b2e45ba3a816baa2\n",
      "Reusing dataset csv (/home/dvgodoy/.cache/huggingface/datasets/csv/default-b2e45ba3a816baa2/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0)\n"
     ]
    }
   ],
   "source": [
    "dataset = load_dataset(path='csv', data_files=['texts/alice28-1476.sent.csv'], quotechar='\\\\', split=Split.TRAIN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "shuffled_dataset = dataset.shuffle(seed=42)\n",
    "split_dataset = shuffled_dataset.train_test_split(test_size=0.2, seed=42)\n",
    "train_dataset, test_dataset = split_dataset['train'], split_dataset['test']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "auto_tokenizer = AutoTokenizer.from_pretrained('gpt2')\n",
    "def tokenize(row):\n",
    "    return auto_tokenizer(row['sentence'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac3386d9ffcb4ce19dd2a5ce7ae5a628",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=2.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eaab2355206949bf8e99462698ecc013",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenized_train_dataset = train_dataset.map(tokenize, remove_columns=['source', 'sentence'], batched=True)\n",
    "tokenized_test_dataset = test_dataset.map(tokenize, remove_columns=['source', 'sentence'], batched=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[9, 28, 20, 9, 34, 29]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(map(len, tokenized_train_dataset[0:6]['input_ids']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"Packed\" Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/block_tokens.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adapted from https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm.py\n",
    "def group_texts(examples, block_size=128):\n",
    "    # Concatenate all texts.\n",
    "    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n",
    "    total_length = len(concatenated_examples[list(examples.keys())[0]])\n",
    "    # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n",
    "    # customize this part to your needs.\n",
    "    total_length = (total_length // block_size) * block_size\n",
    "    # Split by chunks of max_len.\n",
    "    result = {\n",
    "        k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n",
    "        for k, t in concatenated_examples.items()\n",
    "    }\n",
    "    result[\"labels\"] = result[\"input_ids\"].copy()\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d09af828b9c74e8db7e3d770ed7b1111",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=2.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6af3f3e1c9ca4213b1123f0ffb900cb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "lm_train_dataset = tokenized_train_dataset.map(group_texts, batched=True)\n",
    "lm_test_dataset = tokenized_test_dataset.map(group_texts, batched=True)\n",
    "lm_train_dataset.set_format(type='torch')\n",
    "lm_test_dataset.set_format(type='torch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([   63,  2437,   466,   345,   760,   314,  1101,  8805,  8348,   464,\n",
      "         2677,  3114,  7296,  6819,   379,   262,  2635, 25498,    11,   508,\n",
      "          531,   287,   257,  1877,  3809,    11,  4600,  7120, 25788,  1276,\n",
      "         3272,    12,  1069,  9862, 12680,  4973,  2637,  1537,   611,   314,\n",
      "         1101,   407,   262,   976,    11,   262,  1306,  1808,   318,    11,\n",
      "         5338,   287,   262,   995,   716,   314,    30,   464,   360,   579,\n",
      "         1076,  6364,  4721,   465,  2951,    13,    63,  1026,   373,   881,\n",
      "        21289,   272,   353,   379,  1363,  4032,  1807,  3595, 14862,    11,\n",
      "         4600, 12518,   530,  2492,   470,  1464,  3957,  4025,   290,  4833,\n",
      "           11,   290,   852,  6149,   546,   416, 10693,   290, 33043,    13,\n",
      "         1870, 14862,   373,   523,   881, 24776,   326,   673,  4966,   572,\n",
      "          379,  1752,   287,   262,  4571,   340,  6235,   284,    11,  1231,\n",
      "         2111,   284,  4727,   262,  7457,   340,   550,   925])\n"
     ]
    }
   ],
   "source": [
    "print(lm_train_dataset[0]['input_ids'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(239, 56)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(lm_train_dataset), len(lm_test_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Configuration & Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'transformers.models.gpt2.modeling_gpt2.GPT2LMHeadModel'>\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForCausalLM\n",
    "model = AutoModelForCausalLM.from_pretrained('gpt2')\n",
    "print(model.__class__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(50257, 768)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.resize_token_embeddings(len(auto_tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args = TrainingArguments(\n",
    "    output_dir='output',\n",
    "    num_train_epochs=1,             \n",
    "    per_device_train_batch_size=1,\n",
    "    per_device_eval_batch_size=8,\n",
    "    evaluation_strategy='steps',\n",
    "    eval_steps=50,\n",
    "    logging_steps=50,\n",
    "    gradient_accumulation_steps=4,\n",
    "    prediction_loss_only=True,\n",
    ")\n",
    "\n",
    "trainer = Trainer(model=model, \n",
    "                  args=training_args, \n",
    "                  train_dataset=lm_train_dataset, \n",
    "                  eval_dataset=lm_test_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='59' max='59' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [59/59 00:22, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>3.587500</td>\n",
       "      <td>3.327199</td>\n",
       "      <td>0.891100</td>\n",
       "      <td>62.845000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=59, training_loss=3.5507330167091498, metrics={'train_runtime': 22.6958, 'train_samples_per_second': 2.6, 'total_flos': 22554466320384.0, 'epoch': 0.99, 'init_mem_cpu_alloc_delta': 1316954112, 'init_mem_gpu_alloc_delta': 511148032, 'init_mem_cpu_peaked_delta': 465375232, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 13103104, 'train_mem_gpu_alloc_delta': 1499219456, 'train_mem_cpu_peaked_delta': 0, 'train_mem_gpu_peaked_delta': 730768896})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='7' max='7' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [7/7 00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 3.320632219314575,\n",
       " 'eval_runtime': 0.9266,\n",
       " 'eval_samples_per_second': 60.438,\n",
       " 'epoch': 0.99,\n",
       " 'eval_mem_cpu_alloc_delta': 151552,\n",
       " 'eval_mem_gpu_alloc_delta': 0,\n",
       " 'eval_mem_cpu_peaked_delta': 0,\n",
       " 'eval_mem_gpu_peaked_delta': 730768896}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "device_index = model.device.index if model.device.type != 'cpu' else -1\n",
    "gpt2_gen = pipeline('text-generation', model=model, tokenizer=auto_tokenizer, device=device_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do:  once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, `and what is the use of a book,'thought Alice `without pictures or conversation?' So she was considering in her own mind (as well as she could, for the hot day made her feel very sleepy and stupid), whether the pleasure of making a daisy-chain would be worth the trouble of getting up and picking the daisies, when suddenly a White Rabbit with pink eyes ran close by her.\n",
      "`Oh, the way you were,'said Alice; `you've done your best to put it right--'and the Rabbit began to laugh, and it stopped.`Oh!--that's a bad feeling, indeed,'said Alice, `and that's certainly no good, or at all pleasant, to you.`You know, there is just an hour!'The Rabbit said timidly--as she stood up a bit, and looked at the room again.`As you like it,'said Alice again, `you've been talking!'--and she rose\n"
     ]
    }
   ],
   "source": [
    "result = gpt2_gen(base_text, max_length=250)\n",
    "print(result[0]['generated_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}