4 - Packed Padded Sequences, Masking, Inference and BLEU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 - Packed Padded Sequences, Masking, Inference and BLEU\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook we will be adding a few improvements - packed padded sequences and masking - to the model from the previous notebook. Packed padded sequences are used to tell our RNN to skip over padding tokens in our encoder. Masking explicitly forces the model to ignore certain values, such as attention over padded elements. Both of these techniques are commonly used in NLP. \n",
    "\n",
    "We will also look at how to use our model for inference, by giving it a sentence, seeing what it translates it as and seeing where exactly it pays attention to when translating each word.\n",
    "\n",
    "Finally, we'll use the BLEU metric to measure the quality of our translations.\n",
    "\n",
    "## Preparing Data\n",
    "\n",
    "First, we'll import all the modules as before, with the addition of the `matplotlib` modules used for viewing the attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from torchtext.datasets import TranslationDataset, Multi30k\n",
    "from torchtext.data import Field, BucketIterator\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "\n",
    "import spacy\n",
    "import numpy as np\n",
    "\n",
    "import random\n",
    "import math\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll set the random seed for reproducability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "SEED = 1234\n",
    "\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.cuda.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we'll import spaCy and define the German and English tokenizers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "spacy_de = spacy.load('de')\n",
    "spacy_en = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_de(text):\n",
    "    \"\"\"\n",
    "    Tokenizes German text from a string into a list of strings\n",
    "    \"\"\"\n",
    "    return [tok.text for tok in spacy_de.tokenizer(text)]\n",
    "\n",
    "def tokenize_en(text):\n",
    "    \"\"\"\n",
    "    Tokenizes English text from a string into a list of strings\n",
    "    \"\"\"\n",
    "    return [tok.text for tok in spacy_en.tokenizer(text)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using packed padded sequences, we need to tell PyTorch how long the actual (non-padded) sequences are. Luckily for us, TorchText's `Field` objects allow us to use the `include_lengths` argument, this will cause our `batch.src` to be a tuple. The first element of the tuple is the same as before, a batch of numericalized source sentence as a tensor, and the second element is the non-padded lengths of each source sentence within the batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "SRC = Field(tokenize = tokenize_de, \n",
    "            init_token = '<sos>', \n",
    "            eos_token = '<eos>', \n",
    "            lower = True, \n",
    "            include_lengths = True)\n",
    "\n",
    "TRG = Field(tokenize = tokenize_en, \n",
    "            init_token = '<sos>', \n",
    "            eos_token = '<eos>', \n",
    "            lower = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, valid_data, test_data = Multi30k.splits(exts = ('.de', '.en'), \n",
    "                                                    fields = (SRC, TRG))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And build the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "SRC.build_vocab(train_data, min_freq = 2)\n",
    "TRG.build_vocab(train_data, min_freq = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we handle the iterators.\n",
    "\n",
    "One quirk about packed padded sequences is that all elements in the batch need to be sorted by their non-padded lengths in descending order, i.e. the first sentence in the batch needs to be the longest. We use two arguments of the iterator to handle this, `sort_within_batch` which tells the iterator that the contents of the batch need to be sorted, and `sort_key` a function which tells the iterator how to sort the elements in the batch. Here, we sort by the length of the `src` sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "train_iterator, valid_iterator, test_iterator = BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data), \n",
    "     batch_size = BATCH_SIZE,\n",
    "     sort_within_batch = True,\n",
    "     sort_key = lambda x : len(x.src),\n",
    "     device = device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Model\n",
    "\n",
    "### Encoder\n",
    "\n",
    "Next up, we define the encoder.\n",
    "\n",
    "The changes here all within the `forward` method. It now accepts the lengths of the source sentences as well as the sentences themselves. \n",
    "\n",
    "After the source sentence (padded automatically within the iterator) has been embedded, we can then use `pack_padded_sequence` on it with the lengths of the sentences. `packed_embedded` will then be our packed padded sequence. This can be then fed to our RNN as normal which will return `packed_outputs`, a packed tensor containing all of the hidden states from the sequence, and `hidden` which is simply the final hidden state from our sequence. `hidden` is a standard tensor and not packed in any way, the only difference is that as the input was a packed sequence, this tensor is from the final **non-padded element** in the sequence.\n",
    "\n",
    "We then unpack our `packed_outputs` using `pad_packed_sequence` which returns the `outputs` and the lengths of each, which we don't need. \n",
    "\n",
    "The first dimension of `outputs` is the padded sequence lengths however due to using a packed padded sequence the values of tensors when a padding token was the input will be all zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, input_dim, emb_dim, enc_hid_dim, dec_hid_dim, dropout):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.GRU(emb_dim, enc_hid_dim, bidirectional = True)\n",
    "        \n",
    "        self.fc = nn.Linear(enc_hid_dim * 2, dec_hid_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, src, src_len):\n",
    "        \n",
    "        #src = [src len, batch size]\n",
    "        #src_len = [batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(src))\n",
    "        \n",
    "        #embedded = [src len, batch size, emb dim]\n",
    "                \n",
    "        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, src_len)\n",
    "                \n",
    "        packed_outputs, hidden = self.rnn(packed_embedded)\n",
    "                                 \n",
    "        #packed_outputs is a packed sequence containing all hidden states\n",
    "        #hidden is now from the final non-padded element in the batch\n",
    "            \n",
    "        outputs, _ = nn.utils.rnn.pad_packed_sequence(packed_outputs) \n",
    "            \n",
    "        #outputs is now a non-packed sequence, all hidden states obtained\n",
    "        #  when the input is a pad token are all zeros\n",
    "            \n",
    "        #outputs = [src len, batch size, hid dim * num directions]\n",
    "        #hidden = [n layers * num directions, batch size, hid dim]\n",
    "        \n",
    "        #hidden is stacked [forward_1, backward_1, forward_2, backward_2, ...]\n",
    "        #outputs are always from the last layer\n",
    "        \n",
    "        #hidden [-2, :, : ] is the last of the forwards RNN \n",
    "        #hidden [-1, :, : ] is the last of the backwards RNN\n",
    "        \n",
    "        #initial decoder hidden is final hidden state of the forwards and backwards \n",
    "        #  encoder RNNs fed through a linear layer\n",
    "        hidden = torch.tanh(self.fc(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)))\n",
    "        \n",
    "        #outputs = [src len, batch size, enc hid dim * 2]\n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        \n",
    "        return outputs, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention\n",
    "\n",
    "The attention module is where we calculate the attention values over the source sentence. \n",
    "\n",
    "Previously, we allowed this module to \"pay attention\" to padding tokens within the source sentence. However, using *masking*, we can force the attention to only be over non-padding elements.\n",
    "\n",
    "The `forward` method now takes a `mask` input. This is a **[batch size, source sentence length]** tensor that is 1 when the source sentence token is not a padding token, and 0 when it is a padding token. For example, if the source sentence is: [\"hello\", \"how\", \"are\", \"you\", \"?\", `<pad>`, `<pad>`], then the mask would be [1, 1, 1, 1, 1, 0, 0].\n",
    "\n",
    "We apply the mask after the attention has been calculated, but before it has been normalized by the `softmax` function. It is applied using `masked_fill`. This fills the tensor at each element where the first argument (`mask == 0`) is true, with the value given by the second argument (`-1e10`). In other words, it will take the un-normalized attention values, and change the attention values over padded elements to be `-1e10`. As these numbers will be miniscule compared to the other values they will become zero when passed through the `softmax` layer, ensuring no attention is payed to padding tokens in the source sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Attention(nn.Module):\n",
    "    def __init__(self, enc_hid_dim, dec_hid_dim):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.attn = nn.Linear((enc_hid_dim * 2) + dec_hid_dim, dec_hid_dim)\n",
    "        self.v = nn.Linear(dec_hid_dim, 1, bias = False)\n",
    "        \n",
    "    def forward(self, hidden, encoder_outputs, mask):\n",
    "        \n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        #encoder_outputs = [src len, batch size, enc hid dim * 2]\n",
    "        \n",
    "        batch_size = encoder_outputs.shape[1]\n",
    "        src_len = encoder_outputs.shape[0]\n",
    "        \n",
    "        #repeat decoder hidden state src_len times\n",
    "        hidden = hidden.unsqueeze(1).repeat(1, src_len, 1)\n",
    "  \n",
    "        encoder_outputs = encoder_outputs.permute(1, 0, 2)\n",
    "        \n",
    "        #hidden = [batch size, src len, dec hid dim]\n",
    "        #encoder_outputs = [batch size, src len, enc hid dim * 2]\n",
    "        \n",
    "        energy = torch.tanh(self.attn(torch.cat((hidden, encoder_outputs), dim = 2))) \n",
    "        \n",
    "        #energy = [batch size, src len, dec hid dim]\n",
    "\n",
    "        attention = self.v(energy).squeeze(2)\n",
    "        \n",
    "        #attention = [batch size, src len]\n",
    "        \n",
    "        attention = attention.masked_fill(mask == 0, -1e10)\n",
    "        \n",
    "        return F.softmax(attention, dim = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoder\n",
    "\n",
    "The decoder only needs a few small changes. It needs to accept a mask over the source sentence and pass this to the attention module. As we want to view the values of attention during inference, we also return the attention tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, output_dim, emb_dim, enc_hid_dim, dec_hid_dim, dropout, attention):\n",
    "        super().__init__()\n",
    "\n",
    "        self.output_dim = output_dim\n",
    "        self.attention = attention\n",
    "        \n",
    "        self.embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.GRU((enc_hid_dim * 2) + emb_dim, dec_hid_dim)\n",
    "        \n",
    "        self.fc_out = nn.Linear((enc_hid_dim * 2) + dec_hid_dim + emb_dim, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, input, hidden, encoder_outputs, mask):\n",
    "             \n",
    "        #input = [batch size]\n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        #encoder_outputs = [src len, batch size, enc hid dim * 2]\n",
    "        #mask = [batch size, src len]\n",
    "        \n",
    "        input = input.unsqueeze(0)\n",
    "        \n",
    "        #input = [1, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        \n",
    "        #embedded = [1, batch size, emb dim]\n",
    "        \n",
    "        a = self.attention(hidden, encoder_outputs, mask)\n",
    "                \n",
    "        #a = [batch size, src len]\n",
    "        \n",
    "        a = a.unsqueeze(1)\n",
    "        \n",
    "        #a = [batch size, 1, src len]\n",
    "        \n",
    "        encoder_outputs = encoder_outputs.permute(1, 0, 2)\n",
    "        \n",
    "        #encoder_outputs = [batch size, src len, enc hid dim * 2]\n",
    "        \n",
    "        weighted = torch.bmm(a, encoder_outputs)\n",
    "        \n",
    "        #weighted = [batch size, 1, enc hid dim * 2]\n",
    "        \n",
    "        weighted = weighted.permute(1, 0, 2)\n",
    "        \n",
    "        #weighted = [1, batch size, enc hid dim * 2]\n",
    "        \n",
    "        rnn_input = torch.cat((embedded, weighted), dim = 2)\n",
    "        \n",
    "        #rnn_input = [1, batch size, (enc hid dim * 2) + emb dim]\n",
    "            \n",
    "        output, hidden = self.rnn(rnn_input, hidden.unsqueeze(0))\n",
    "        \n",
    "        #output = [seq len, batch size, dec hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, dec hid dim]\n",
    "        \n",
    "        #seq len, n layers and n directions will always be 1 in this decoder, therefore:\n",
    "        #output = [1, batch size, dec hid dim]\n",
    "        #hidden = [1, batch size, dec hid dim]\n",
    "        #this also means that output == hidden\n",
    "        assert (output == hidden).all()\n",
    "        \n",
    "        embedded = embedded.squeeze(0)\n",
    "        output = output.squeeze(0)\n",
    "        weighted = weighted.squeeze(0)\n",
    "        \n",
    "        prediction = self.fc_out(torch.cat((output, weighted, embedded), dim = 1))\n",
    "        \n",
    "        #prediction = [batch size, output dim]\n",
    "        \n",
    "        return prediction, hidden.squeeze(0), a.squeeze(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seq2Seq\n",
    "\n",
    "The overarching seq2seq model also needs a few changes for packed padded sequences, masking and inference. \n",
    "\n",
    "We need to tell it what the indexes are for the pad token and also pass the source sentence lengths as input to the `forward` method.\n",
    "\n",
    "We use the pad token index to create the masks, by creating a mask tensor that is 1 wherever the source sentence is not equal to the pad token. This is all done within the `create_mask` function.\n",
    "\n",
    "The sequence lengths as needed to pass to the encoder to use packed padded sequences.\n",
    "\n",
    "The attention at each time-step is stored in the `attentions` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder, src_pad_idx, device):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.src_pad_idx = src_pad_idx\n",
    "        self.device = device\n",
    "        \n",
    "    def create_mask(self, src):\n",
    "        mask = (src != self.src_pad_idx).permute(1, 0)\n",
    "        return mask\n",
    "        \n",
    "    def forward(self, src, src_len, trg, teacher_forcing_ratio = 0.5):\n",
    "        \n",
    "        #src = [src len, batch size]\n",
    "        #src_len = [batch size]\n",
    "        #trg = [trg len, batch size]\n",
    "        #teacher_forcing_ratio is probability to use teacher forcing\n",
    "        #e.g. if teacher_forcing_ratio is 0.75 we use teacher forcing 75% of the time\n",
    "                    \n",
    "        batch_size = src.shape[1]\n",
    "        trg_len = trg.shape[0]\n",
    "        trg_vocab_size = self.decoder.output_dim\n",
    "        \n",
    "        #tensor to store decoder outputs\n",
    "        outputs = torch.zeros(trg_len, batch_size, trg_vocab_size).to(self.device)\n",
    "        \n",
    "        #encoder_outputs is all hidden states of the input sequence, back and forwards\n",
    "        #hidden is the final forward and backward hidden states, passed through a linear layer\n",
    "        encoder_outputs, hidden = self.encoder(src, src_len)\n",
    "                \n",
    "        #first input to the decoder is the <sos> tokens\n",
    "        input = trg[0,:]\n",
    "        \n",
    "        mask = self.create_mask(src)\n",
    "\n",
    "        #mask = [batch size, src len]\n",
    "                \n",
    "        for t in range(1, trg_len):\n",
    "            \n",
    "            #insert input token embedding, previous hidden state, all encoder hidden states \n",
    "            #  and mask\n",
    "            #receive output tensor (predictions) and new hidden state\n",
    "            output, hidden, _ = self.decoder(input, hidden, encoder_outputs, mask)\n",
    "            \n",
    "            #place predictions in a tensor holding predictions for each token\n",
    "            outputs[t] = output\n",
    "            \n",
    "            #decide if we are going to use teacher forcing or not\n",
    "            teacher_force = random.random() < teacher_forcing_ratio\n",
    "            \n",
    "            #get the highest predicted token from our predictions\n",
    "            top1 = output.argmax(1) \n",
    "            \n",
    "            #if teacher forcing, use actual next token as next input\n",
    "            #if not, use predicted token\n",
    "            input = trg[t] if teacher_force else top1\n",
    "            \n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Seq2Seq Model\n",
    "\n",
    "Next up, initializing the model and placing it on the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DIM = len(SRC.vocab)\n",
    "OUTPUT_DIM = len(TRG.vocab)\n",
    "ENC_EMB_DIM = 256\n",
    "DEC_EMB_DIM = 256\n",
    "ENC_HID_DIM = 512\n",
    "DEC_HID_DIM = 512\n",
    "ENC_DROPOUT = 0.5\n",
    "DEC_DROPOUT = 0.5\n",
    "SRC_PAD_IDX = SRC.vocab.stoi[SRC.pad_token]\n",
    "\n",
    "attn = Attention(ENC_HID_DIM, DEC_HID_DIM)\n",
    "enc = Encoder(INPUT_DIM, ENC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, ENC_DROPOUT)\n",
    "dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, DEC_DROPOUT, attn)\n",
    "\n",
    "model = Seq2Seq(enc, dec, SRC_PAD_IDX, device).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we initialize the model parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Seq2Seq(\n",
       "  (encoder): Encoder(\n",
       "    (embedding): Embedding(7853, 256)\n",
       "    (rnn): GRU(256, 512, bidirectional=True)\n",
       "    (fc): Linear(in_features=1024, out_features=512, bias=True)\n",
       "    (dropout): Dropout(p=0.5, inplace=False)\n",
       "  )\n",
       "  (decoder): Decoder(\n",
       "    (attention): Attention(\n",
       "      (attn): Linear(in_features=1536, out_features=512, bias=True)\n",
       "      (v): Linear(in_features=512, out_features=1, bias=False)\n",
       "    )\n",
       "    (embedding): Embedding(5893, 256)\n",
       "    (rnn): GRU(1280, 512)\n",
       "    (fc_out): Linear(in_features=1792, out_features=5893, bias=True)\n",
       "    (dropout): Dropout(p=0.5, inplace=False)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_weights(m):\n",
    "    for name, param in m.named_parameters():\n",
    "        if 'weight' in name:\n",
    "            nn.init.normal_(param.data, mean=0, std=0.01)\n",
    "        else:\n",
    "            nn.init.constant_(param.data, 0)\n",
    "            \n",
    "model.apply(init_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll print out the number of trainable parameters in the model, noticing that it has the exact same amount of parameters as the model without these improvements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 20,518,405 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define our optimizer and criterion. \n",
    "\n",
    "The `ignore_index` for the criterion needs to be the index of the pad token for the target language, not the source language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRG_PAD_IDX = TRG.vocab.stoi[TRG.pad_token]\n",
    "\n",
    "criterion = nn.CrossEntropyLoss(ignore_index = TRG_PAD_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll define our training and evaluation loops.\n",
    "\n",
    "As we are using `include_lengths = True` for our source field, `batch.src` is now a tuple with the first element being the numericalized tensor representing the sentence and the second element being the lengths of each sentence within the batch.\n",
    "\n",
    "Our model also returns the attention vectors over the batch of source source sentences for each decoding time-step. We won't use these during the training/evaluation, but we will later for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion, clip):\n",
    "    \n",
    "    model.train()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    for i, batch in enumerate(iterator):\n",
    "        \n",
    "        src, src_len = batch.src\n",
    "        trg = batch.trg\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        output = model(src, src_len, trg)\n",
    "        \n",
    "        #trg = [trg len, batch size]\n",
    "        #output = [trg len, batch size, output dim]\n",
    "        \n",
    "        output_dim = output.shape[-1]\n",
    "        \n",
    "        output = output[1:].view(-1, output_dim)\n",
    "        trg = trg[1:].view(-1)\n",
    "        \n",
    "        #trg = [(trg len - 1) * batch size]\n",
    "        #output = [(trg len - 1) * batch size, output dim]\n",
    "        \n",
    "        loss = criterion(output, trg)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        \n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "    \n",
    "        for i, batch in enumerate(iterator):\n",
    "\n",
    "            src, src_len = batch.src\n",
    "            trg = batch.trg\n",
    "\n",
    "            output = model(src, src_len, trg, 0) #turn off teacher forcing\n",
    "            \n",
    "            #trg = [trg len, batch size]\n",
    "            #output = [trg len, batch size, output dim]\n",
    "\n",
    "            output_dim = output.shape[-1]\n",
    "            \n",
    "            output = output[1:].view(-1, output_dim)\n",
    "            trg = trg[1:].view(-1)\n",
    "\n",
    "            #trg = [(trg len - 1) * batch size]\n",
    "            #output = [(trg len - 1) * batch size, output dim]\n",
    "\n",
    "            loss = criterion(output, trg)\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we'll define a useful function for timing how long epochs take."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The penultimate step is to train our model. Notice how it takes almost half the time as our model without the improvements added in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Time: 0m 32s\n",
      "\tTrain Loss: 5.062 | Train PPL: 157.888\n",
      "\t Val. Loss: 4.809 |  Val. PPL: 122.606\n",
      "Epoch: 02 | Time: 0m 32s\n",
      "\tTrain Loss: 4.084 | Train PPL:  59.374\n",
      "\t Val. Loss: 4.108 |  Val. PPL:  60.819\n",
      "Epoch: 03 | Time: 0m 32s\n",
      "\tTrain Loss: 3.293 | Train PPL:  26.919\n",
      "\t Val. Loss: 3.541 |  Val. PPL:  34.504\n",
      "Epoch: 04 | Time: 0m 33s\n",
      "\tTrain Loss: 2.808 | Train PPL:  16.583\n",
      "\t Val. Loss: 3.320 |  Val. PPL:  27.670\n",
      "Epoch: 05 | Time: 0m 33s\n",
      "\tTrain Loss: 2.436 | Train PPL:  11.427\n",
      "\t Val. Loss: 3.242 |  Val. PPL:  25.575\n",
      "Epoch: 06 | Time: 0m 34s\n",
      "\tTrain Loss: 2.159 | Train PPL:   8.659\n",
      "\t Val. Loss: 3.273 |  Val. PPL:  26.389\n",
      "Epoch: 07 | Time: 0m 32s\n",
      "\tTrain Loss: 1.937 | Train PPL:   6.937\n",
      "\t Val. Loss: 3.172 |  Val. PPL:  23.856\n",
      "Epoch: 08 | Time: 0m 31s\n",
      "\tTrain Loss: 1.732 | Train PPL:   5.651\n",
      "\t Val. Loss: 3.231 |  Val. PPL:  25.297\n",
      "Epoch: 09 | Time: 0m 31s\n",
      "\tTrain Loss: 1.601 | Train PPL:   4.960\n",
      "\t Val. Loss: 3.294 |  Val. PPL:  26.957\n",
      "Epoch: 10 | Time: 0m 31s\n",
      "\tTrain Loss: 1.491 | Train PPL:   4.441\n",
      "\t Val. Loss: 3.278 |  Val. PPL:  26.535\n"
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 10\n",
    "CLIP = 1\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss = train(model, train_iterator, optimizer, criterion, CLIP)\n",
    "    valid_loss = evaluate(model, valid_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut4-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we load the parameters from our best validation loss and get our results on the test set.\n",
    "\n",
    "We get the improved test perplexity whilst almost being twice as fast!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Test Loss: 3.154 | Test PPL:  23.441 |\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('tut4-model.pt'))\n",
    "\n",
    "test_loss = evaluate(model, test_iterator, criterion)\n",
    "\n",
    "print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference\n",
    "\n",
    "Now we can use our trained model to generate translations.\n",
    "\n",
    "**Note:** these translations will be poor compared to examples shown in paper as they use hidden dimension sizes of 1000 and train for 4 days! They have been cherry picked in order to show off what attention should look like on a sufficiently sized model.\n",
    "\n",
    "Our `translate_sentence` will do the following:\n",
    "- ensure our model is in evaluation mode, which it should always be for inference\n",
    "- tokenize the source sentence if it has not been tokenized (is a string)\n",
    "- numericalize the source sentence\n",
    "- convert it to a tensor and add a batch dimension\n",
    "- get the length of the source sentence and convert to a tensor\n",
    "- feed the source sentence into the encoder\n",
    "- create the mask for the source sentence\n",
    "- create a list to hold the output sentence, initialized with an `<sos>` token\n",
    "- create a tensor to hold the attention values\n",
    "- while we have not hit a maximum length\n",
    "  - get the input tensor, which should be either `<sos>` or the last predicted token\n",
    "  - feed the input, all encoder outputs, hidden state and mask into the decoder\n",
    "  - store attention values\n",
    "  - get the predicted next token\n",
    "  - add prediction to current output sentence prediction\n",
    "  - break if the prediction was an `<eos>` token\n",
    "- convert the output sentence from indexes to tokens\n",
    "- return the output sentence (with the `<sos>` token removed) and the attention values over the sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate_sentence(sentence, src_field, trg_field, model, device, max_len = 50):\n",
    "\n",
    "    model.eval()\n",
    "        \n",
    "    if isinstance(sentence, str):\n",
    "        nlp = spacy.load('de')\n",
    "        tokens = [token.text.lower() for token in nlp(sentence)]\n",
    "    else:\n",
    "        tokens = [token.lower() for token in sentence]\n",
    "\n",
    "    tokens = [src_field.init_token] + tokens + [src_field.eos_token]\n",
    "        \n",
    "    src_indexes = [src_field.vocab.stoi[token] for token in tokens]\n",
    "    \n",
    "    src_tensor = torch.LongTensor(src_indexes).unsqueeze(1).to(device)\n",
    "\n",
    "    src_len = torch.LongTensor([len(src_indexes)]).to(device)\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        encoder_outputs, hidden = model.encoder(src_tensor, src_len)\n",
    "\n",
    "    mask = model.create_mask(src_tensor)\n",
    "        \n",
    "    trg_indexes = [trg_field.vocab.stoi[trg_field.init_token]]\n",
    "\n",
    "    attentions = torch.zeros(max_len, 1, len(src_indexes)).to(device)\n",
    "    \n",
    "    for i in range(max_len):\n",
    "\n",
    "        trg_tensor = torch.LongTensor([trg_indexes[-1]]).to(device)\n",
    "                \n",
    "        with torch.no_grad():\n",
    "            output, hidden, attention = model.decoder(trg_tensor, hidden, encoder_outputs, mask)\n",
    "\n",
    "        attentions[i] = attention\n",
    "            \n",
    "        pred_token = output.argmax(1).item()\n",
    "        \n",
    "        trg_indexes.append(pred_token)\n",
    "\n",
    "        if pred_token == trg_field.vocab.stoi[trg_field.eos_token]:\n",
    "            break\n",
    "    \n",
    "    trg_tokens = [trg_field.vocab.itos[i] for i in trg_indexes]\n",
    "    \n",
    "    return trg_tokens[1:], attentions[:len(trg_tokens)-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll make a function that displays the model's attention over the source sentence for each target token generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_attention(sentence, translation, attention):\n",
    "    \n",
    "    fig = plt.figure(figsize=(10,10))\n",
    "    ax = fig.add_subplot(111)\n",
    "    \n",
    "    attention = attention.squeeze(1).cpu().detach().numpy()\n",
    "    \n",
    "    cax = ax.matshow(attention, cmap='bone')\n",
    "   \n",
    "    ax.tick_params(labelsize=15)\n",
    "    ax.set_xticklabels(['']+['<sos>']+[t.lower() for t in sentence]+['<eos>'], \n",
    "                       rotation=45)\n",
    "    ax.set_yticklabels(['']+translation)\n",
    "\n",
    "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "\n",
    "    plt.show()\n",
    "    plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll grab some translations from our dataset and see how well our model did. Note, we're going to cherry pick examples here so it gives us something interesting to look at, but feel free to change the `example_idx` value to look at different examples.\n",
    "\n",
    "First, we'll get a source and target from our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "src = ['ein', 'schwarzer', 'hund', 'und', 'ein', 'gefleckter', 'hund', 'kämpfen', '.']\n",
      "trg = ['a', 'black', 'dog', 'and', 'a', 'spotted', 'dog', 'are', 'fighting']\n"
     ]
    }
   ],
   "source": [
    "example_idx = 12\n",
    "\n",
    "src = vars(train_data.examples[example_idx])['src']\n",
    "trg = vars(train_data.examples[example_idx])['trg']\n",
    "\n",
    "print(f'src = {src}')\n",
    "print(f'trg = {trg}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we'll use our `translate_sentence` function to get our predicted translation and attention. We show this graphically by having the source sentence on the x-axis and the predicted translation on the y-axis. The lighter the square at the intersection between two words, the more attention the model gave to that source word when translating that target word.\n",
    "\n",
    "Below is an example the model attempted to translate, it gets the translation correct except changes *are fighting* to just *fighting*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted trg = ['a', 'black', 'dog', 'and', 'a', 'spotted', 'dog', 'fighting', '.', '<eos>']\n"
     ]
    }
   ],
   "source": [
    "translation, attention = translate_sentence(src, SRC, TRG, model, device)\n",
    "\n",
    "print(f'predicted trg = {translation}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAn0AAAJVCAYAAACrjEOWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdedx1c73/8dfHfeM2R4aUKedUKiVFGVIypZSxEJ3iFCcdTadfKkoknUqDOiqpE01oQKYyFBJNhjIVUUIkQweZue/P74/vd7vWvV24h31da+9rvZ6Px37c11573Xt/19577fVe32lFZiJJkqSpbYG2CyBJkqSJZ+iTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH3SJIqIaX333QelcUREtF0GaarxgCNNkoiIzJxZ//5YRCyTmbPaLpc0bCJiWmZmRCweEQu2XR5pqjD0SZMgIhbIes3DiDgU2BNYs91SScOnd3IUEYsD1wP7GPykwTD0SZOgV6MXEc8DVgfeDvyy1UJJQ6ZRw7cAsD1wIXB8Zj7UctGkKWF62wWQuiIivgJsDtwNXJSZD9UaQJt4JaDW8C0CHAS8ALgYuLLdUklThzV90uT5OrA0pVl3Yyg1gHZYl2azBbArsDZwa635cx+RBsDQJ02A8UblZuavgZdRavr2iogX1+Ue1NRZ/d/9zDwR+H/APcA7I+Klvf6wkuaPoU8asNovqdeH7zkRsWEdhbhEZl4GbAo8F/jviFgXDH7qpoiY3vvuR8SCvX0gM78DfAj4J7Bf7wRJ0vwJT6CkwWn20YuI7wAbAisD1wFnAJ/OzGsiYh3gZ8CvgH0y86K2yiy1oZ4c9Ubpfhp4JnAj8IfM/HhdZzdKrd/1wAGZ+Zu2yitNBYY+aQJExBeBVwPvB3q1e3tSatc3zcy/R8QLgbOAPwFvzczftlVeqQ0RsRhlhO4/gUuAJwEvAa4CtsnMeyPizcB7KSdOn8zM89oqrzTqbN6VBiwiVgBeSqm9ODEz/wCcBjyDUrN3V60RvBjYEngK8I+2yitNtkZXhg8CdwC7ZuYemfl64DjKSdIrADLzG8BnKWFw2xaK20n9Vw9qLLcbyghzyhZpPo0z7coywL8Af8zMByLiOcDPgZOAd2bmfRHxmoj4RWb+KiJWz8wH2ii7NJnqxMvZGJjxHOA2Sm03EfF6YG/g/Zl5au0H+8/MPCoibgFOb6fk3dJoel8M2J+SFW7IzEMdVDPaDH3SfOj9ONa/18jMKynNUHcBL4mIyyiB7yfAW2pz1ZbAzsCfKTV8D7ZTemniRcQMYFZmPlgHbUwHFqR87xcC7q5TF70R+Cawb2YeEhELAx+NiAsy8+jM/FF9vkf2OQ1e44ooi1LmSVwASODpEfEKYPfMtGViRNm8K82jvsD3VeCMiHgB5WD2bWA34FpK0+4umXl3RDwZ2AVYAfg7lJG7LRRfmnB1kMZbKfsC9XJq5wIvqvvOScCOEXEwcCTw4cz8RP3vL6JM0DzbccrAN3EaV0SZRple6irKnKIvB15HaWI/NiKWa6+Umh/W9EnzoE418XD9+xTglcAsYLHMfDgivgm8EFgMuKKeOW8E/DvwGmCjzLy9peJLk6Ke6DwbeGNELAu8BbiF2pwLnAi8itK3738z8+A6x+WzgUOAe4FjJr/k3VR/p2YAXwBWBa7NzBvhkd+5XYDvAEdHxC6ZeWt7pdW8cPSuNJf6pmX5IbAOsANwNPCpzPxKfez5wLspo3inAXdSRinulpmXtFF2abL0+u/Vv4+jnOxcB2xdu0H01tuIsp9sBxwOPBV4GqWGb716uUKbdCdJnU7qGMpI6qMz812NxwLYBPgWcCnw5sz8eysF1TyxeVeaS43AdwKwPmVqiV8DD1EOVr1geCnwX8AGlIPajsArDXzqiObozyUoV6J5EvDqiFiy90Bm/hx4O7AXsCLl5Oh7wEtq4Jtu4Js441wR5UJKi8S1wBsiYsfGY0mZZmpXyuXyPjiJRdUAWNMnzaG+Pnyvo0wf8VngktoscgZwZWa+s1fLYQ2FuqLxnW/W8C1BqQX/OXArcASlj9ghwNcy886+55htJLz7z8TqdVOpTeqLAQ8AD9eBNRsDn6AM4jgkM49v/L+gdF/5nZ/PaLGmT5pDjcD3bUoH849k5sWNH71rgbV669dO7B+MiJdPemGlybcUPHJJwV5/8UOAnTPzT5l5V2buDJwD7AO8te4jRMRT636ycPMJDRQTpwbqh2swP5pSg3cG8MWImJGZ5zBWk/e+iNiu93/rrDsX1ZPdcefz03Ay9ElPoPmjFhFbUyaNPZVyaaim24Dl69+LUGoB308dpStNVRHxDODsiNgJoDfIiTJY8J91nYXrYzsDZ1OusrFPRGxKGcV7AHD/5Ja8m2qNam8evl9TBm2cAlwOrAtcFRH/mplnU37DEnhvROzS/1wG89Fi6JOeQKOGb19gdeDYzPxlZj5Ul/f6xFwOLBURKwH/Qxnp9vJmp3VpiloRuA/Yr1kjBMyghj7gwd4JVA1+Z1L68X2HEva2cPqiiRMRL42I99a7vfd5L+AeyoCMAzNzb0oAX5kS/sjMc4H9KFcOstVixBn6pDkQEc+jXPj9s5S+L9R+MM159v4GLE7pt7QLZVqWiye/tNLkqsHgA8BNwMcbnf8XpvQT6zUJzmwEvzdT+vu9kXJy9FCjWVgDFBHrUppunwqz/WY9kzLV1HV1vR2Bj1CuiHJMRCwVEQvWGr/XUwbcaIQZ+qQ5cxXwJsoM9a+MiJVqZ+fmPnQfJfStR5lq4rctlFOaVI0Qdy7wSUqAOLAOBLgOeErts7dkRCwCLBIRC0bEUpl5bmb+pBcGG83CGpA6IfbewC8y870RsVBEvKj3MPBQlstFvgE4FtivXhFlIUqfvv+sg3N+ax++ydc/unp+GfpGUP+XoBc8Bv3l6KrxftQy80HKpdQ+QjkzPjEiFq3Br3fQ+zXwLmBDp2VRF8TYNVoXqiNBz6aM+LweOIwy9ccLgQso87pdQTmBupqyLz3CvmETJoDlgIUjYg3KZ/Dq+tjRwAYR8S3KJfD2a1wRZS3KCeyDzWZ3P6fJU/te9kbCLxMRe0TErvP1nHahGC190yEsDWxFmfvqq5n5QKuFmwL6pmXZEliNMhDjd5l5bZ2tfhPgS8DtwMsy856IWNj3X13SmO5jceDrwM3A/pl5R0RsAryHctmuU4BvAEvX2wxKn7KvWbM3OSJiLeA3lGuC/wV4bWbeXOdLPBjYHTg7M19bB9w8izJR9gPAZga9ydWY/mhBSuvRQZRLd+5A+UyeAdw4L31g7T8xIhphb6GIWAr4KOVLsA3lkkY/ZuzSRpoHvRFt9e9jKNf+XIhyMFswIv49My+JiLMofVu+BJwVEZtk5j2tFVyaZPX3qBf4LgBuBH5Fue40mXlWRMyiTM2yDvD9zPzhOM8z3eA38erv1kPAMpSpWXqtE3dFxFcoIXzvKJdaW4oy+0BS+lrODOdLnFQ18G0M7AxsT6k5/y1lUNQXMvOv8/rcNu+OiPol2BT4PPB7YG3KNSzvoYwmNfDNpxy70sbhwIuBPTNzNUpzyNrAKRGxbmbeT/nh3ItyjdBTbFpXl9TfowWAz1F+g/YEPpeZ9za6O5xDmafvJuCQiHjjOM9j4JtAvd+liNgQ+CLlN2tb4OCIWAUgMy8HPgRsBtxAqRE8grFL4HlFlEkUEXtFxFGU7kSrULpJvJgyEOdSyqj3ee7OZU3fCIiItwMvo1zG68fAoZn5sYh4LfA84Ny63iNNv5o3Ua4D+lzgPZl5Tp3i4E2UJpDNgeMj4jX1zPkcSnX7taP+vnsmr3mwMPB84PzM/HNvYbNmKDPPrjV+nwFeBXy7pbIOzCjsK433PwEy8/yI+EUN67dT+vIRER/OzBsy8y7KiexZ4zyPwXwS1Frzd1Pmr7yAcmz5WWbeUR9/C6V/5s9hthHYc8XQN8Rqe/5ngdcCV1IuWH5+jl26aC9Kv8wzYd6/BF02zg/47cD3gdPqaLYDgN0z89sRcQOln8vxEfHGzPwl9axrFNWRlCcDe9T+ikN/MNNQWYRyTd0HYfZLqNXgtyQwLTN/FhF7Ar9rr6jzZ5T2lcbgmkWBt1Cacm/LzG8DZOZxdTDAd+r6+2XmjX3/N7IYym2cijLz7og4HjgOuDkz/69RU7sd5QTrDb1a9mxcrnBuOJBjyEWZ6X4B4Jb6JVigjhjdmjIB8L9n5k+H+UdoFETEu4EjM/POiHhyZt5e+7fcDLyr12cvIq6gdEa/gzIq8YFRDdt12oYTKN+vDTLzer9HGs9jHWQi4lTgX4B1M/Ofjd+noMxr+VTg/zX6ys7zwapNo7KvNAYALE7pAzaDEs4XpHQLeg9wUW22fR2l5vVo4IDM7L/CkCZJRDytF7wbywLoXTnlc8BLgW3715tb9ukbUhGxYkQslJlXZ+ZVjdTfa8d/GfAP4I/gMPr5ERFPo9SovhugBr7FgadRLj7eC3zrU8Leu4HNM/P+UQ181W+BXSmjky+MiFXSebjUp/bpmhVlbr2VIuJpUS7fBWVC5icD342IJRuB7hmUFoqlKVMcAWP9ZkfQ0O8r9XPq9bV8D2WU7uaU64RvSQmAx1D6J5OZP6BMIr8b8G8tFFlARBwCHFr7XT6iV9MaEWtSWvW+Mr+BD6zpG0oR8XnKMO1vZ5n3qv/x51E6274rM4+Y7PKNuvFGDEbEfwHvpAzeOKMu+z5l5OF/UsLev1EGbmzTaGIfSfWEotcstzlwKOVKIxtk5k3DWIuhyddo7luCMvXKcpRpjM6i/D4dG+UqDv8D3Em5JvVClJPShyk1gA+Pcn/jUdpXahjfmxL0/pCZH208tiSl//e9mblBY/nGwHn23Zt89RjzIsqAp9My89q+xxeizGe5FbBlZt48v69pTd+QiYjvUfruXU6ZxLT/8WmUaVquYoT7k7Wp9+MWEds0Fp9KmTB2x4hYvi7bFbgVOJHSn2cb4N1TIPAt0DiIfYpyFgllpNgvI2LlYavFUDvq92ARSufxoEwVtTdlipajI2L3zPweZe7KK4BXUALHzxkLfNNGOPCN2r6yAfDfwE40aljrie5dlM/uubW/MlBGWdfPyT7+kygi9qfsKztRuhZdW2vTF26sNotSW37BIAIfOJBjqETEvpQZ0F8HXJqZ90ed9LdxtjmDMmL3vP6zAs25iDgUeGdE/IgyC/0lEfFlSvPHicDJmflgRLyEcs3J+4FLMvO69ko9GDk2Nc0RlDPID1CmAtiKMknrbyLixZl5wzDVYmhyNWrntqbU3u0NXFabEJeqqy0LkJlXANvV5fdnnah8vFr1UTJK+0r9vM6MiK0oJ7H/FhE/ysyLG5/BXxkLErMZ5c9pRK0O/DwzLwCIiGcD+wJPjYiLgI/WwR1fBv5Q15nvGnNr+oZEPVN8BiVs/KYGvjWAb0fEacCREbFC7V+2L/D++v+cH24OjPM+nULpn/MK4DMR8d/A+ZRrhx4RESvAI/0qvpeZJ02FwNcTESsCLwe+lJnfynLZuE9QJp2+nVKL8dQhq8WYTURMjzJCUQMQEetExJvGeWhFSnPmDTXw7UyZguX9Wa7RulRErAuQmXc2Al9MhSAxrPtKf81cLwxk5o+B7SjHkw/G2HV2oYy2vo9GLaAmV0RMq822ywBPioitIuKDwEXAM4HbgHdQTjDIzMt6+9EgaswNfUOiniHOBDaLiFdExAGUKQ6Wo+yka1EuYr5QZv6pN7hgVJtNJtNjnB1dTLl01FeAn1FGGf6ask/cCuxXm7WmqgcoI/qmwyN9t2ZR+mp9nvJ+nB0Rqw5jTV9tAjkP+I/a30zzISKeRKnRPmqc4HcvsEwdTPZqymjPfWvgm06p8dqhMbgDmFK/TUO3r9Rm54cjYvGI+HJEHBkR+0bEIvX37kTKvK47UAbZHBARH2Ps8pFfnYxy6tGyzJ/4ILA/ZdLlwykDag7IzJdQrsJxArB2lEE5A+VAjiFSa/a+TPkiXEG5dNEhtZbqBMrv6HZtlnGURcQnKdNL/Ftm3hcRW1D6KB1A6eD8dsqOCGXuse0y8+dtlHWQxgu9UeaA/Amln9amOTbz/sP1+3YpZZLqP1EGr8wctoN4RJxI6Uv2XuCYzPxny0UaSVFGDe4EHEs5+LydMhXUUfXxlSiTwk8D1qD0a/1Cfex5lIPWWZn54ckv/WCN0r5ST0p/RZkk+3bKqNxzKSN3r8oy4nobyrEDyvWPrwI+VR+z68YkijJFziqUY8vZmXlFbVFagpLFrq7rPZmyL/6BMlhzsN+lzPTW0o3yQ/teyo/sSxrLnw+s3Li/FGXHPZxythltl33UbsCSwAcp0xj8kTISdwHgfZS5+J5U11uPUpNxPbBa2+UewHZPa/y9YN9jL6Zcy/ErfcufD5xD6Vu68kSXcR62KRp/f4NSE7Nn7zP0Nlfv5fMpLQyH1PurUGq/ZwG71WXTKSPbr6aEhmUoc7+tR6kd/wUwve1tGcB7MfT7CmXett7fa9bjwsr19219yiXvzqX0+16grvea+nl+CVijLvMYMrnfre9RLnF3c709QOkju2jfes+itED9HXjWhJSl7TejqzfKVR9uqSHkwfqFOHSc9damXAfxtt4O622O3t9p4yybTpkK5zjKRKWn1Pf32Poez6jrLQcs3fY2DOA9aIajQ+r2/hTYCFi8Ln8bpfnuZEqH/VcDR1EuA7Rs29vwGNs1vfH3DEoz73XAHsCSbZdvFG6UWqug9M37RV22RN0fnko5wZxFqfHrvc/vpnQ5uY1y4nRZDRgL1nUetc+Nym0U9pXe957S1Lw0pVb2JGChxjovYCz4rclY8Nuxfp7fAp7b9vvdpRulJelGYFPKnJZrUroFPEypNe/tj5+kzAd5NbDWhJWn7Tekizfgw5SapJdRzphXoVy4/BZKZ+HeentRqu+vmsgvwQRv66SfUfbCW/17a+DNlGapZlh4M/AjSi3Hbym1FS9v+/0a4HvQrBH4Tj0QHAdcQqmxeAclAAdlAtc/UpqI7qSciKzd9jY80feJUiP7Y0oH6DuA/6PU+C3Rdjnnd9sm8TXfU8PAKyl9WX9AqQFfhbHg95a67nRgJeCtwH/U0DOt91jb7998vAdDv68w1hVrCUof5GsoYfMnwMLN7aAEvxuBsykhvrd8u/p5HkFfTaa3CfvcFqKcQHyz//OkDAZ6CFinLnsRJSA+fULL1Pab0sUbcDzww+aOBywPHEwZUr9T/VK8mlIFvFrbZZ6LbWvlbJ8Sno8FVmgs+2790b6XUpv6IWZvNn9yfX9vrz+G/9s8AIzqrS8YLUKpjdiIWiNA6cT9MOUyWUvXZYsDL6E02T2l7W2Yg238ZA0pm1EmC35BDSz31kAy9DV+be0rze8JpVbvjLp/XMDsJ0arUPoYPxL8hnE75vc9aPw9lPsKY8F6AeA0Sug7uP6+zQI+2Vi3F/B6zfZfqv+vFxpfAzyn7fe9K7e6j50O/HCc5U+jzMf7ZcZqyyf8+OM8fZOojsSZTpkC4S9ZOwQDszLzloj4EvBGykzv3wV+FCN0rcpa1t41Nvek9DW5FPhVZt4wwS+/EaVT/6kRsSVl6PvzKGe3f6OMYvsosExEfD4zr8vM24HDIuJiSh+/L4zKe/14sverUq7XuC2lRuKGrJPMZuYeETGTcqZJRHwjM2+l9M8aenXk7nOBMzLzJ42HXhcRR1Oa5zIivpeZd7RSyCfQ8r4ClO9JRPydMrXHP4GnAxsCP6sDGq6vUxkBfDUiZmYd3NH3PCM7GGAU9pUcmyB7M0oL0RGZeWGUSeQvAz5aP5t9swzQWCAzL42IZwHX9pZFBJl5ymSVu+t6g4Ii4hpg24hYMzMvh0e+dzdGxD8pXQMeqssn/vjTdhLuyo3aPFD//n+Ujpwb1vvTG4+dSml2HOWz5+9RmqpvpPT/+TET3ARCOZt9PeXM6QLKAJlP9a3zX5Qz48/S1+maKdDcwezNVIswNpHsjZQpNwAWaaxzOGU6oA8zYoMgKGfPP2vc7/V3WonS/HYN8C6GvKm3jX2l7/WXr5//dpRapFuAjfvWWQU4rO47r277PRvQdo/MvlJ/275Iudb6dTT6G1NaK/atn83Bj7F9I3ssGcUbpVJnBWprA2WOy2soV6lZvbHeUyi1tofQqI2d8PK1/QZ14QZ8qt6eU+8/sx60LmH2UbvLUfrwPWpAxzDfmH3U26Z1GzauP6Zvrfd/DbxoIl+/7jg7Uebgewj4eF3e7OPXC36HAKu2/d5N0Pvx1Prvkygjw/8P+Gnj8eb78W1KM+kybZf7MbZlgb77vZOjfYA/A1v3PT6d0pfpbkoN71CF2SHYV3rhOOjrh0eZgLgX/F7e99jTKSerI9t37zHej6HaVyiXUduqPwBQuvqcWn+73tj32LKU4PcgcFjb72mXb5TuABdSAvqZwFvr8hfX36s/AgdSBkV9v37fJmSU7mOWse03aarfKGfy11Dmf1uxsXw7ylD/O+oO+yHKJcDuYARG6TLWd6TZJ+ZAShPqkczeN2iniTqY0XcWSwl+OzN21r5cXd4c4fau+uN58BQ8iH2EMr9T7wRjSeA/63txUmO95sFsKPvw9QWk1eq29EZYr0KZy/J8YPPGeivWfe5ZwPJtb0Mt01DtK5RWh89SOph/Adizsc7LeIzg11hnSuwzw7avUE5YjqWvhaLx+MspJzR/Brbpe+zJlGvunkcLA4K8JcA3KbNw/Ec9xny6Hmf2q48vR7nE5+8pzfTnAM+f9HK2/UZN5RvwcUp1/EuAxeqyhRuPPxM4lDJvz5/qDj3pX4J52K5FKc2oazSWLU4ZZTyLMqVBfw1N72B2Po3azfksRzMU7EOpKVmKsabeq6lXNRnnvX87U7BDcz1oXUhpNugdzJaiDFj5K+Uyf711e6P+hu4gwewB6cv1h/IqSg3tM+vy5wDXUvo1HV638YeUOa5WansbahmHZV+JxmtfWb8jP6A0J99Kme+wt85L6/KbgC3afg8n8LMZmn2FUhO0JWPzhS5I6arwIhpzuVH69Z1OOeHpD35LNT7Dodunp/Ktfk6/p8wW0RuU8eK6j3+t79izPKVpt5WuJ62/WVP1RjlrPBnYv7Hs6ZSJT48G9mMsCC5HafdfvO1yz+G2PZtSU7Bk3/Ll68HibhrTOTQef3094DwyzcB8lKEZCr5HOfv9NHX0LiX47VgPuL+lzqM1v687TLf+97ex/M2UaUx+Ps7B7Frg3LbLPqfbRQl5N1Bqw4+hhKWfAmvWx1enTEHxh3qgvpghmt5oGPaVxnP25uU7nxqc6/Kv1YPTKxrLXkbpG3tK2+/hoL9Tfctb31fq5/Ir4KB6f4n6Hb+CsRODNzfW35yx4Lf1eM/X9vvdtRuli8bdwPr1/r9Smni/Qw3tDMk0WK0XYKrdaAQ3xgZlvJDSl+xeSrPNhZS2/ffUcDJyHW0ZO6P8InVASr2/HGXOu2vrjtB/MNuOAU5BQ2mivYkyG30vRPea03p9/C4HfsOQNPdNwGexOo+uLdqtcTDrzcK/JKVf1hUM4ZU2xtmuVSnT6OzUWPbWuv/8nLHgtzClT9xKDOFULcOyr9TnPB34TuP+jpT+r++r9xdrPLZW//dq1G/Duq8wNkXMYpTWid7E0M+nTCl1GfDOxvqb1WPLbcBGbb+vXbz1Hes3o4S+Vet+/Q/KlDq9ib23p29KsdbK3XYBptqNMhP6Z+rf2wK/pIS9y4EP1eULUmokjmi7vPOwfc0atlXqD9SdwLqN5cvV7f7LeAezAZZl0XoQO2ycH/Lm4I4dKJ36f8YkjpKapM/joPrDv17/dlGuUPE3St+RXpPokgzp1Ub6vlsfB+6nNJms07fe7o2D9LPbLvccbs+k7yvMfvIznTJR7LnUCeAp00PNAj5Q78+gXCng1eM9z6jfhnVfafxWTaO0VvyU2neQ0k/slvoduRp4W+P/vYYyqf/IVRpMhRuNY329fz6l//7/UeZ7XKIuX4FS43cMQzCbQOtv3FS6AetSzgx7ly6aXn/sN+gdnChV+UtTOksfXO+PRAgZ78eFUo19OmWerxc3lvcOZldTmiPm+8DR/xyUZpjrgc8+zv9ZtB70tgP+pe33cALeg1Xrd+7yxziY/S9wD2Vgy9APEOrbrp/XULInj74W6u6UWvNLgWe0Xd5xyj8U+wqlFvSnwCvr/f0oU4/sQ5l0+EON/7MRcBbwurbfvwF9BkO/rzAW+BaiXNP4Y8Ab6rIjKYNKnl6/OzdRujrsPSffN28T+rn1jvW7M1aT/1pKK8Rd1KtqUPrtH0k5oRiK39/WCzCVbpQRuDc93kGI0vn8CMrZ29AdrB6n3M1+VrsD768HjpUpl/o5aZyD2bKUfkmX0Hdh6fl8/VdRB7xQRqudyTjTKABvAPZq+70b4GfQHIH8ZOpocEqz5qWUfm3r9b1Xn6iB4keMwJVdKFMeHEM5GVqJUqN39WMcpN9GGfy0atvl7itX2/tKb1qWhWpguIdSy7gx8C+UEDiL2Wspnk2pqThtKgSIUdhXGmFhRv181q/lm0EJ4Ncz+8CA/SmD/q4Ddmz7Pe7yjXGO9XV/27l+lv+gtOZdROm+8YK2y/xIOdsuwFS5Ua7t+jfgvfX+o5oRKX34zqpfgqHpbD6X23lc/TH6PaVJ6m/14PsqyhQQdzF789WT5/cHtO+H+Vt1R/pwvb8D5XJDH2L2UW5LUwZ4nECjn9Ko3Sh91TboW9YbuHAzpRl0Rg0Ul9TlG9f/tzjlLHNXYKm2t2VOPmdKB/oHgC/WZStR+jP1DtL9+9TQbldL+0qvhm8JSvPTcfX9u48yQ8DGlBqJMyjN5x+lNCFeQBnwNL35PKN0G4V9hXIlmc37lj2H0uzcnNJrN8oo9NXq/aCMUP8e5RKEIx/MR/XG+Mf6Xo1tUFqX9qac7O3MkPWfbr0Ao35rfNi71h/33uid3o/vUoyNCtuNcrY2ks2MwAcpzQvrMTZK9mTKWc2rKLUYZ1A6Hm8wAa//7Xrgei2N+bIogW8mpd/EG+pn8YNajpGdlqX+gHyl/vhvUZd9nnLS8HFKX8b76rY+td4urNt9NsXYHs0AACAASURBVOUE485h/b6NFywo/V13pwS/L9dlzeD3klEIJG3uK/U9PIsySORFlD5FW1Bqsa5lrMbvQ5Rav+Moc9b1At/IzcM3CvsKpan9x/V7sUWj3GtRWn6Wb6y7GaU2dof6eT6T0if59Y11DH6T+x17omP90qNwvGm9AFPhRkn6VzL7qLglKFMx/LjuvO+oO/jIXu6LMpfXkYyNNFu5/sgezVgTxFqUvlbX05jUdACvvTmltmSTxrInUc6cX1rf6xsoTVk3UKZAGPo5D+dgu59Vw8FllJn6P0Fjfi7glZQaoxMZm4/wUEoT4vHAc9vehjnYxtX77i8E/DuPDn6/pdTYrDPZZZyHbWpzX1mNEnb2biyLWoZfUk6cXlGX9/eVHNkgMQr7CmNXY7qMsX6Wa1ADJyXgBaVG8ov12HEVpSnxYkYwkE+lG098rH8Y+GDjsaHrr996AUb5xlifjH+vIWPten9fSr+QmZT+Sf/R/+M6arf6ZT8XOKHeX50ySum7jM1D9Nb6w/VsYJUBv/4OlLn4lqJMa/AKSl+vv9Yfxr1qGdeoZRjaZr952PZeP6wrauhZqy7vnXluWg9mp1DnI6zLF5rsss7Dth1Ug89L+pYvSBnAMZMyz90ClI745zOkNZeNsre9ryxDCX0f61selFrGB2vo2KBX3rbfswFu+9DvK5RBGT+tn8FmlJrYG2g079b1FqVMqfMlygCcXk3syAbzUb3N5bF+qD+f1gswFW6UM7IrKc0JF1Cq6g+nUSvV/OKM6q1u368oZ9G9eYiWqo+tQWlOesMEvfazKDU/J9Wd7B5KTcq2lGaqB5kCNXuPs/3PoNRizGq+x4w1LWxCaao6h0Y/oLbLPQfbtU49+J3Co4Pf0vXzngV8tS4biZqOlveVRer79kv6RgwCT6uB6DpKMBzZ/q6Ps/1Dv6/U4HdW/SwOpvQR25dy+a53UbqpbEM58Vmp8f9G4vs/VW9T4VjfegFG/UZpXpxVbz+sX4rlGWvCmTKXxQHWpExAOYsyQKJ35rksZVb/S5nATquUvkhnUa4Xumtj+faUWr/V2n6PJvj9X51S03UtsGVjee9gtiWlqXAoLkE2F9u1Vg1+Pxon+PXC00301YQM820I9pU1KSdGx9KYy5DSx/AUYEPKdCCfaPu9mqDtH/p9pQa/MygtGLMoJwZ/rJ/LHynB/EKGvOaoK7epcqzvFVLzKCIWpTTV3EG5VuP/1eULZOasVgs3ASJiM8oX/teUAwqUJqNXUC7QfukEv/50YGb29q6I5Slnys8DtsrM2yfy9dsWEf9CGZG4HLBPZp5Wly+QmbMiYtHMvLfVQs6DiFiLMhDneuCjmfmriFiB0rR7AnBqZt7XZhnn1hDsK1tSBmlcThkEcDPwJkqt0vaU2sYLM3OPiSxHW0ZhX4mIZ1AGmawKvCczf1yPKVAGftyRmTlVjyejZKoc6w19AxAR0zJzZuN+5BR+YyNiXeC/KZ3rZ1I6Gn84M6+Y5HLsQhngsS2TcBAdFhHxr5SRistTpg04o+UiDUREPB/4OmXqkl9QasXWooySu7bNss2rtveViHgupd/kWpQ+fb+n9I9diDL59WmUkcZMxd+sUdhXahkPB55CKePpzWPIqIWKiRQRL83M81p8/ZE/1hv6NE8iYhHKgWMm8FBmPjDJr78epenvQcoEzJdP5uu3rR4ovkhpxntTZv605SINRESsTpnPciPKIJ0PZuZl7ZZq/gzBvjKDUms0IzP/HhGLA/9Dmfpo/cy8ejLLM9lGYV+ptZKHU661u3Vm/rrlIg2diNiUMhH/+zLzM22XZ1QZ+jSSImIByvQHt2fmrW2Xpw0R8SzgU5RmoT+3XZ5BiYigDEaIzLyn7fJMJRGxOXAgZfqW12bm71ou0qQYhX2llvE/KWWc+UTrd01EPAn4L+DozLyy7fKMKkOfNMIiYqHMfLDtcmg01Fq+3YHTpnoNX79R2lf6mxFV2NQ9/wx9kiRJHbBA2wWQJEnSxDP0SZIkdYChT5IkqQMMfZIkSR1g6BsSEbFn22UYlKmyLVNlO8BtGUZTZTvAbRlWU2Vbpsp2QPvbYugbHlPmS83U2Zapsh3gtgyjqbId4LYMq6myLVNlO6DlbTH0SZIkdYDz9D2BiJgyb9ALX/jCSXmd2267jWWXXXbCnv+KKyZnMvaZMx9m2rTpE/oaDzzQ6vXeJUlTz22Zudx4Dxj6nkBEZLni1+i7/8FJveTnhFnzOeu3XYSBueaai9suwsBMrd+SqbQtkjrmosxcZ7wHpkaakSRJ0uMy9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqgM6EvohYPyJOioibIuKeiPhdROzadrkkSZImw/S2CzCJVgXOBw4H7gc2BI6MiFmZeUyrJZMkSZpgnQl9mXls7++ICOBcYCVgD2C20BcRewJ7TmoBJUmSJlBnQl9ELA0cCGwDPA2YVh+6sX/dzDwCOKL+v5ysMkqSJE2UzoQ+4ChgPeAg4PfAXcBelBAoSZI0pXUi9EXEDGArYO/MPLyxvDMDWSRJUrd1JfQsTGnOfaC3ICKWALZurUSSJEmTqBM1fZl5Z0RcAOwfEXcBs4APAHcCS7ZaOEmSpEnQlZo+gF2Aa4FvAp8Hjqt/S5IkTXmdqOkDyMxrgE3GeeiASS6KJEnSpOtSTZ8kSVJnGfokSZI6wNAnSZLUAYY+SZKkDjD0SZIkdYChT5IkqQMMfZIkSR1g6JMkSeoAQ58kSVIHGPokSZI6wNAnSZLUAYY+SZKkDjD0SZIkdYChT5IkqQMMfZIkSR1g6JMkSeoAQ58kSVIHTG+7AKMgc1bbRRiIVVZ6RttFGIibb7627SIMzIwZi7VdhIGZNWtm20UYmIceerDtIuhRsu0CSCPPmj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAyY19EXEURFx4ROskxGx94Bfd+P6vGsO8nklSZJGhTV9kiRJHWDokyRJ6oBWQl9EbBsRV0bE/RFxXkQ853HW3SoizoyIWyLiroj4VURsMc56z4+IkyPijoi4OyJ+ExGbP87z7hwRD0bE2wa1XZIkScOqjdC3KvBZ4CBgF2Ap4PSImPEY6z8dOBn4N2AH4BfAjyNiw94KEbEGcD6wIvA2YDvgBGDl8Z4wInYDvgnsmZmHz/8mSZIkDbfpLbzmssA2mfkLgIi4CPgTsBvwqACWmYf1/o6IBYCzgecCb6EEPYCPAHcCG2XmfXXZmeO9eK3Z+zzwpsw8dgDbI0mSNPTaqOm7pRf4ADLzOuAi4MXjrRwRK0XENyLiRuBh4CFgC+CZjdU2Ab7bCHyP5Z3AocDOjxf4ImLPiLjwiUYaS5IkjYo2avpueYxlK/YvrDV7JwFLAPsD1wD3AB8Flm+s+mTgb3Pw2jvU5/jJ462UmUcAR9Qy5Bw8ryRJ0lBro6Zv+cdYNl5o+1dgbeAdmfm/mfmzzLwQWKRvvdsZJzSOY1dgMeDkiOh/DkmSpCmrldAXERv07kTEKsALgd+Ms24vmD3QWH9VYMO+9X4K7Pg4g0F6/gpsCjwD+EFELDiXZZckSRpJbYS+24BvRcQuEbEdcCqlefeocda9khLUPlOnbtkZOAO4sW+9AymjgM+NiJ0iYrOIeF9E/Hv/E2bmn4HNKH0Iv12bkCVJkqa0NgLPdcD7gAOAY4G7gFdm5v39K2bmA8D2lAEcP6BM8/LfwM/61rsKeCklUH6NMl3L6+prPUpm/oEyGOSVwFcjIgawXZIkSUMrMh2n8Him0kCOFVZYre0iDMTNN1/bdhEGZsaMxdouwsDMmjWz7SIMzEMPPdh2EfQoU+anWJpoF2XmOuM9YNOmJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOmB62wXQ5PnHP/7WdhEG4oVrb952EQbmSyef3HYRBuYju+/VdhEG5r777267CANxxx23tF2EgZk58+G2iyCNPGv6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4YudAXEWtGREbExm2XRZIkaVSMXOiTJEnS3DP0SZIkdcDQh76IeHtE3BAR90TEycCKfY8vGhFfiIibI+L+iLggIrboWyci4qCIuCUi7oqIr0fEzrWZeLVJ3BxJkqRWDHXoi4htgC8CpwDbA5cBX+9b7avA7sDBwHbADcCpEfHSxjrvBvYFDgdeB9wHfGpCCy9JkjREprddgCewH3BaZu5V758eEcsBbwWIiGcDbwB2z8xv1GWnA5cCHwZeGRHTgH2AwzNz//o8Z0TE04GVJ29TJEmS2jO0NX01rK0NnNj30PGNv9cFAvh+b0Fmzqr3ezV9KwNPAU7qe57++83X3jMiLoyIC+et9JIkScNlmGv6lqOU75a+5c37KwJ3Z+a9fev8HVg0IhamBD6AW/vW6b//iMw8AjgCICJyLsstSZI0dIa2po8Syh4Glu9b3rz/N2DxiFi0b50VgHsz8wHg5rpsub51+u9LkiRNWUMb+jJzJvA7YJu+h7Zv/H0BkJTBGUAZqVvvn1cX3UAJfv3Ps/UgyytJkjTMhrl5F+DjwPER8WXgBODlwJa9BzPzDxFxDHBYRCwJXAPsAawB7FXXmRkRhwCHRMStwPmUwPe8+jSzJmtjJEmS2jK0NX0AmXkC8A7gtcAPKQM73tK32h7ANyijdU8EVgVek5nnNdb5HCVAvh04Dli63ge4a6LKL0mSNCyGvaaPzDwMOKxvcTQev5cSDN/xOM+RlFD44UeeIOJrwPWZecdACyxJkjSEhj70DUJErAnsBPyC0pz7KsqEzu9vs1ySJEmTpROhD7iHMm/f3sBiwHWUwPeZNgslSZI0WToR+jLzWuAVbZdDkiSpLUM9kEOSJEmDYeiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA6IzGy7DEMtIqbQGxRtF2Agpk2b1nYRBmbppZ/SdhEG5pNHf6PtIgzM1z7yP20XYSCuufqitoswMLfe9te2izBAU+iwomF0UWauM94D1vRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6oDOhr6IeE1EZESs1nZZJEmSJlpnQ58kSVKXGPokSZI6YKhDX0SsHxEnRcRNEXFPRPwuInZtPL5bbaJ9XkScWde5MiK273ueiIgDIuKWiPhnRHwTWHLSN0iSJKklQx36gFWB84G3Aq8FjgOOjIg39K13NHASsB1wNXBsRKzUePydwP7AEcDrgPuAT01s0SVJkobH9LYL8Hgy89je3xERwLnASsAewDGNVT+XmV+v610E/B14DXB4REwD3g98JTM/VNc/PSLOBJ428VshSZLUvqGu6YuIpSPiCxFxHfBQve0JPLNv1TN6f2Tm7cAtlHAIsDKwInBi3/85/nFed8+IuDAiLpzPTZAkSRoKQ13TBxwFrAccBPweuAvYC9imb707+u4/CMyofz+l/ntL3zr99x+RmUdQmoKJiJzbQkuSJA2boQ19ETED2ArYOzMPbyyf29rJm+u/y/ct778vSZI0ZQ1z8+7CwDTggd6CiFgC2Houn+cGSvDrrx3cfpx1JUmSpqShrenLzDsj4gJg/4i4C5gFfAC4k7mYbiUzZ0bEp4BPR8RtwM+BHYBnT0CxJUmShtIw1/QB7AJcC3wT+DxlypZvzsPzHAp8HHhbfY7FgX0GVEZJkqShN7Q1fQCZeQ2wyTgPHVAfP4oy2KP//63Wdz+BD9db09HzX0pJkqThN+w1fZIkSRoAQ58kSVIHGPokSZI6wNAnSZLUAYY+SZKkDjD0SZIkdYChT5IkqQMMfZIkSR1g6JMkSeoAQ58kSVIHGPokSZI6wNAnSZLUAYY+SZKkDjD0SZIkdYChT5IkqQMMfZIkSR1g6JMkSeqAyMy2yzDUIiIh2i7GgPhZD5uIqXPetfzyq7RdhIE56OtfabsIA3Hy4Se3XYSBOfXUL7ddhIGZNWtW20UYEI8pQ+qizFxnvAemzhFHkiRJj8nQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkd0JnQFxHrR8RJEXFTRNwTEb+LiF3bLpckSdJkmN52ASbRqsD5wOHA/cCGwJERMSszj2m1ZJIkSROsM6EvM4/t/R0RAZwLrATsARj6JEnSlNaZ0BcRSwMHAtsATwOm1YduHGfdPYE9J690kiRJE6szoQ84ClgPOAj4PXAXsBclBM4mM48AjgCIiJy8IkqSJE2MToS+iJgBbAXsnZmHN5Z3ZiCLJEnqtq6EnoUpzbkP9BZExBLA1q2VSJIkaRJ1oqYvM++MiAuA/SPiLmAW8AHgTmDJVgsnSZI0CbpS0wewC3At8E3g88Bx9W9JkqQprxM1fQCZeQ2wyTgPHTDJRZEkSZp0XarpkyRJ6ixDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgdEZrZdhqEWEQnRdjEGxM962ERMnfOuhRZcuO0iDMxaL9ik7SIMxIFf/XjbRRiYnTbatO0iDMy9997VdhEG4uGHH2y7CBrfRZm5zngPTJ0jjiRJkh6ToU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdYOiTJEnqAEOfJElSBxj6JEmSOsDQJ0mS1AGGPkmSpA4w9EmSJHWAoU+SJKkDDH2SJEkdMHShLyL2iYiN+5YtFBEHRMQLBvg6e0dEDur5JEmShtnQhT5gH2DjvmULAR8BBhb6JEmSumQYQ58kSZIGbI5DX0Q8NyJOi4h/RMQ9EfGHiPjP+tg5EfGDiNgzIv4SEfdFxKkR8bS+51g2Ir4REbdHxL31/63TePwvwJOBj0RE1tvGwD/rKkc2lq9W/8+MiPhURNwQEQ9ExCUR8eq+1104Ig6LiDtq+T8HLDjX75YkSdKImj4X654EXAm8EXgAeBawZOPx9euy/wJmAJ8Efgis21jnh8C/Av8PuA14H3B2RKydmdcA2wFnAz8Avlb/z++BTYCzgI8Bp9blf6v//gB4MaX590/AjsBJEbFOZv6urvMJ4K3AfvX59gBePxfbLkmSNNLmKPRFxLLA6sC2mXlZXfzTvtWWBzbIzOvq/7kOOC8itszM0yJiS2BDYOPM/Fld5yzgL5Tw9x+Z+duIeBj4a2b+qvH6F9Q//9S3fFNgq+ZzAmdExDMpAe/1EfFk4G3ARzLzM/X/nU4Jf5IkSZ0wp827/wBuAA6PiJ0iYvlx1rm4F/gAMvN84BZKLRz131sb4YzMvAc4BXjpvBQe2Ay4GTg/Iqb3bpRA2ms2fh6l5vHExuvOat7vV5upL4yIC+exXJIkSUNljkJfDUlbUALW14GbI+LnEbF2Y7VbxvmvtwAr1r9XBP4+zjp/B5aZ4xLPblngKcBDfbcDgJXrOk95jPKNV14AMvOIzFwnM9d5rHUkSZJGyRz36cvMK4EdImJBYCNKn71TI2Klusp4tX/LM9b37m+Psc4KlJrEefEP4EZg28dZ5+ZGWZqvM15ZJEmSpqS5nrIlMx/KzLOAz1Jq755UH3phRKzSWy8iNqQEq9/URb8Glo+IlzXWWZTSJ++8xks8SGmOpW8Z4yz/KaUm7+7MvLD/Vte5DLgf2Kbxugs070uSJE11czqQ4/nAp4HvAn8GlgbeD1ySmf+ICCjNpadExAGMjd69ODNPA8jM0yPifOC7EfEB4HbKKN5FgEMaL3clsFVEnAbcDVyVmf+MiGuBHSPickqIuxQ4EzgdODMiPglcQRlR/AJgRmZ+MDNvj4gjgAPrIJErKKN3F5/7t0uSJGk0zWlN382Uvnf7AT8GvgT8Adi6sc4v6/JDgf8FLufRza7bUYLaocD3gQA2qdO19LwPuIcyNcsFwIvq8rdR+vD9pC5/amYmsD2ln+G7KQHwK5TpY5q1h/vUdfYHjgFuotRUSpIkdUKU3DSfTxJxDnBbZr5uvp9syJTr80bbxRgQLzU8bEpPg6lhoQUXbrsIA7PWCzZpuwgDceBXP952EQZmp402bbsIA3PvvXe1XYSBePjhB594JbXhoscaiDp1jjiSJEl6TIY+SZKkDpiby7A9pszceBDPI0mSpIlhTZ8kSVIHGPokSZI6wNAnSZLUAYY+SZKkDjD0SZIkdYChT5IkqQMMfZIkSR1g6JMkSeoAQ58kSVIHGPokSZI6wNAnSZLUAYY+SZKkDjD0SZIkdYChT5IkqQMMfZIkSR0Qmdl2GYZaRCRE28UYED/r4TNVvluw4IILtV2EgVlmmRXbLsJAvHjdrdouwsCssOrU+EwALv7luW0XYSAuvvjMtougceVFmbnOeI9Y0ydJktQBhj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1wMiFvohYMyIyIjZuuyySJEmjYuRCnyRJkuaeoU+SJKkDhj70RcTbI+KGiLgnIk4GVux7fNGI+EJE3BwR90fEBRGxRd86EREHRcQtEXFXRHw9InauzcSrTeLmSJIktWKoQ19EbAN8ETgF2B64DPh632pfBXYHDga2A24ATo2IlzbWeTewL3A48DrgPuBTE1p4SZKkITK97QI8gf2A0zJzr3r/9IhYDngrQEQ8G3gDsHtmfqMuOx24FPgw8MqImAbsAxyemfvX5zkjIp4OrDx5myJJktSeoa3pq2FtbeDEvoeOb/y9LhDA93sLMnNWvd+r6VsZeApwUt/z9N9vvvaeEXFhRFw4b6WXJEkaLsNc07ccpXy39C1v3l8RuDsz7+1b5+/AohGxMCXwAdzat07//Udk5hHAEQARkXNZbkmSpKEztDV9lFD2MLB83/Lm/b8Bi0fEon3rrADcm5kPADfXZcv1rdN/X5Ikacoa2tCXmTOB3wHb9D20fePvC4CkDM4Aykjdev+8uugGSvDrf56tB1leSZKkYTbMzbsAHweOj4gvAycALwe27D2YmX+IiGOAwyJiSeAaYA9gDWCvus7MiDgEOCQibgXOpwS+59WnmTVZGyNJktSWoa3pA8jME4B3AK8FfkgZ2PGWvtX2AL5BGa17IrAq8JrMPK+xzucoAfLtwHHA0vU+wF0TVX5JkqRhMew1fWTmYcBhfYuj8fi9lGD4jsd5jqSEwg8/8gQRXwOuz8w7BlpgSZKkITT0oW8QImJNYCfgF5Tm3FdRJnR+f5vlkiRJmiydCH3APZR5+/YGFgOuowS+z7RZKEmSpMnSidCXmdcCr2i7HJIkSW0Z6oEckiRJGgxDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAwx9kiRJHWDokyRJ6gBDnyRJUgcY+iRJkjrA0CdJktQBhj5JkqQOMPRJkiR1gKFPkiSpAyIz2y7DUIuIhGi7GAPiZ62JM23a9LaLMDARU+N8eIUVVm27CAOzxhrrtV2EgdnhHW9ouwgDsff2r227CAOzwALT2i7CwDz88IMXZeY64z02NX7ZJEmS9LgMfZIkSR1g6JMkSeoAQ58kSVIHGPokSZI6wNAnSZLUAYY+SZKkDjD0SZIkdYChT5IkqQMMfZIkSR1g6JMkSeoAQ58kSVIHGPokSZI6wNAnSZLUAYY+SZKkDjD0SZIkdYChT5IkqQMMfZIkSR1g6JMkSeqAOQ59EbF/RNwYEbMi4i8RkRGx5ty8WETsVv/f4k+w3p4Rse04y/8SEZ+em9eUJEkSTJ+TlSJiHeBAYF/gHOBeYBHgTxNUrj2By4Ef9i3fDrh9gl5TkiRpypqj0AesUf/9YmbeNVGFeSKZ+du2XluSJGmUPWHzbkQcBXyr3r2zNs9u3N+8GxFLR8SxEXFPRNwUEe+PiE9HxF/GedqnR8SZdd0rI2L7xvOcA7wIeHN9jYyI3epjszXvRsRREXFhRGweEZfW5zsvIp7bKTXrAwAACe9JREFUtw1zUzZJkqQpZ0769B0EfKz+vQmwPrDkOOsdBWzO/2/v7mItO8s6gP+fzsAMNVisQCnW1ih+JMaEmpHEcOEHAjUKiSZASBPaKp0oaqr2Rk1UFI1f0ZJKIozYTFVILRc1LSWCbcULkyqjbVJTKlPoNGrph50BBNsOwzxerH305GTOtJzZZ/Y++/39kp09e613vet5Libzn3fttXZyTabLs69L8pZN5vxgklszXa49nOSmqrpotu8dSR5I8pHZub43ye2nqe/iJH+Q5LeTvDXJS5PcXFW1xdoAAFbOs17e7e5PV9Xad/c+0d1frKrvXz9mtuL3xiRv7u4PzbbdmeTfk3zxFNNe1903zMb9c5LHkvxokvd29/1V9aUkT3T33c+hh/OTvLq7D8/mOyfJLUm+PckDW6gtVbU/UzgEAFgJ83pky77Z+21rG7r7qSR3bDL+Y+vGPZnk8SQXbTL22RxZC3wz98/e1+b7amtLdx/o7n3dvW+zMQAAO8m8Qt/Lkvx3dz+9YfsTm4z/3IbPx5Ps3eK5TzVX1s331dYGALBy5hX6Hk3ywqraGNxeMqf5z8Qy1wYAcFbMK/Qdmr2/cW1DVb0g080TW3EmK38bzbs2AIAd57k+p++0uvtfq+q2JH9SVS/MtLr2i5ke4nxyC1M+kOT1VfX6TA9jfmj23b9lqA0AYMeZ52/vXpnp5ojrk9yQ5O+T/E2SrTzM+beSfDLJzUk+keQNS1QbAMCO85xW+rr7YKZn3a19/niS2jDmaNY9+66qdmf6KbV/3Gyeddu/acPnzyT5oecw7spTjDmyldoAAFbZXC7vJklVvSnJy5Pcl+nhzVcn+dYkb5vXObZqmWsDADgb5hb6knwpyVVJXpFkV6aA9Ybu/qc5nmOrlrk2AIBtN7fQ190fyfTTaUtnmWsDADgb5nkjBwAAS0roAwAYgNAHADAAoQ8AYABCHwDAAIQ+AIABCH0AAAMQ+gAABiD0AQAMQOgDABiA0AcAMAChDwBgAEIfAMAAhD4AgAHsXnQBwGrYvet5iy5hbjq96BLm4ujRRxddwtycPHly0SXMzfGnji+6hLnYs+fcRZcwN3v3fs2iS5ibY8c2/3tvpQ8AYABCHwDAAIQ+AIABCH0AAAMQ+gAABiD0AQAMQOgDABiA0AcAMAChDwBgAEIfAMAAhD4AgAEIfQAAAxD6AAAGIPQBAAxA6AMAGIDQBwAwAKEPAGAAQh8AwACEPgCAAQh9AAADEPoAAAYg9AEADEDoAwAYgNAHADAAoQ8AYAC7F13AMqqq/Un2L7oOAIB5EfpOobsPJDmQJFXVCy4HAOCMubwLADAAoQ8AYADDhr6qeltVnaiqSxZdCwDAdhs29GXqfVeSWnQhAADbbdjQ190Hu7u6+8iiawEA2G7Dhj4AgJEIfQAAAxD6AAAGIPQBAAxA6AMAGIDQBwAwAKEPAGAAQh8AwACEPgCAAQh9AAADEPoAAAYg9AEADEDoAwAYgNAHADAAoQ8AYABCHwDAAIQ+AIABCH0AAAPYvegCll9l9+7nLbqIuThx4suLLmFOetEFzE3V6vy/q87ZtegS5ufkVxZdwVw8//l7F13C3DzyyOFFlzA3D97z4KJLmIuvP//CRZcwN1+3Qr0cO/bopvtW518cAAA2JfQBAAxA6AMAGIDQBwAwAKEPAGAAQh8AwACEPgCAAQh9AAADEPoAAAYg9AEADEDoAwAYgNAHADAAoQ8AYABCHwDAAIQ+AIABCH0AAAMQ+gAABiD0AQAMQOgDABiA0AcAMAChDwBgANse+qrqW7b7HKc458uq6tyzfV4AgGW1LaGvqvZW1eVVdVeSw+u2n1NVv1RVD1bVM1X1qaq64hTH/2xVHZ6NebCqfmHD/ouq6uaqeryqnqqqT1fVu9YNuSzJZ6vqfVX1PdvRIwDATrJ7npNV1SuTvD3J5UnOTXJrkh9ZN+SPk1yR5DeT/EuS1ya5oaqe7O4Pz+a4ejbuj5J8NMkPJPnDqtrT3b87m+fPk7wgyf4kn0vyzUm+Y915bknytUmuSrK/qu5L8mdJ/qK7j86zZwCAneCMQ19VnZcp5P1kku9Ocm+SX0/yl+sDVlW9IslPJ7mqu2+cbb6jqi6cjf9wVZ2T5J1JDnb3tbMxH5ud45er6t3d/XSSVyV5a3ffNhvz8fU1dffnk1yf5PqqujTJTyT5tSS/V1V/neT9Se7s7j7T/gEAdoIzurxbVZcl+WySdyX5hySXdvel3X39KVbUXpPkZJJbqmr32ivJnUleWVW7klyU5OVJPrTh2L/KtHL3XbPP9yb5naq6sqouPl2N3X1Pd//cbN4rkrwo0wriZ07T1/6qOlRVhxK5EADY+c70O33PJPmfJHuTnJfkRVVVm4x9cZJdST6f5MvrXgczrTheOHslyWMbjl37fP7s/S1JDiW5LsnDVXVvVb3mWWpdq/G8TH0f22xgdx/o7n3dvS/ZrB0AgJ3jjC7vdvffVdU3JPmxTJd370pypKoOJrmxux9eN/xokhNJXp1pxW+jx/P/IfSlG/ZdsG6OdPd/Jrlydjn4VZkuCd9aVRd395NrB80C6A9m+m7fjyc5nuSDSd7R3fdspWcAgJ3ojO/e7e5nuvum7n5tphsqPpDk6iQPVdUdVXX5bOhdmVb6zuvuQ6d4HU/yH0keSfKmDad5c5IvJLlvw7lPdvfdSX4j040jlyRJVV1QVe9M8lCSO5JcnOSnklzY3QIfADCcud69291HkvzqLHBdlulO3oNJPtDd/1ZV701yU1X9fqbLs3uTfGeSb+vut3f3ydmx76uqJ5P8bZLvy3QDyK9099Ozmzo+mukO3k8l2ZPk2iSPJvnkrJQfzhTybkzy/u7+v8fGAACMaK6hb013fyXJ7Ulur6oL1u36mUxB7epMj235QpL7Mz1OZe3YP62qPUl+Psk1mVb/ru3u62ZDns604ndNkm/M9J3Cu5O8rrufmo25NdPdwye2oz8AgJ1mW0Lfet392Lo/d5J3z16nO+Y9Sd6zyb5nMoXG0x3vWXwAAOv47V0AgAEIfQAAAxD6AAAGIPQBAAxA6AMAGIDQBwAwAKEPAGAAQh8AwACEPgCAAQh9AAADEPoAAAYg9AEADEDoAwAYgNAHADAAoQ8AYABCHwDAAIQ+AIABCH0AAAOo7l50DUutqp5I8vBZONWLk/zXWTjP2bAqvaxKH4leltGq9JHoZVmtSi+r0kdydnq5pLtfcqodQt+SqKpD3b1v0XXMw6r0sip9JHpZRqvSR6KXZbUqvaxKH8nie3F5FwBgAEIfAMAAhL7lcWDRBczRqvSyKn0kellGq9JHopdltSq9rEofyYJ78Z0+AIABWOkDABiA0AcAMAChDwBgAEIfAMAAhD4AgAH8LxpkEfjRsjvoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_attention(src, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Translations from the training set could simply be memorized by the model. So it's only fair we look at translations from the validation and testing set too.\n",
    "\n",
    "Starting with the validation set, let's get an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "src = ['eine', 'frau', 'spielt', 'ein', 'lied', 'auf', 'ihrer', 'geige', '.']\n",
      "trg = ['a', 'female', 'playing', 'a', 'song', 'on', 'her', 'violin', '.']\n"
     ]
    }
   ],
   "source": [
    "example_idx = 14\n",
    "\n",
    "src = vars(valid_data.examples[example_idx])['src']\n",
    "trg = vars(valid_data.examples[example_idx])['trg']\n",
    "\n",
    "print(f'src = {src}')\n",
    "print(f'trg = {trg}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then let's generate our translation and view the attention.\n",
    "\n",
    "Here, we can see the translation is the same except for swapping *female* with *woman*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted trg = ['a', 'woman', 'playing', 'a', 'song', 'on', 'her', 'violin', '.', '<eos>']\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnoAAAJDCAYAAACc37lpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd7hcZbn38e+dAqFJUUAEwYooFlDALirHXrGh4DmCQmyoeOweG5bXdmzHhqg0CxHrATyiCHZFBVERpYoNBaVIBEwgyf3+8TxjVoYdkh0me808+X6ua669Z82a2ffaa8pvPWVNZCaSJElqz6y+C5AkSdLaYdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZLGRETEKB/PoDchhnd8RMyaarkkSZpMETErM7P+vkVEHBQR+92kx6yPpzEWEdHZ8ZsDjwE2Az6emYt7LU6SJN0kg8/5iJgLbAy8BdgaeDKwGLgjcHGuQWibM9JKNVKdgLdeRGwKvJmy458AXAh8rf6Upq22BkdmLuu7Fklal9WQ92Dg6cCTgD8AZwL/AP4nM/+0po9t1+0Yqzt+L+ADwK+BXYG/AtcACzLTkKdpqwcNZLEsIjaIiK37rkvS2jUY8jO0zOE/PYuI50fEUcA3ge2BDwF7AN8AfgmcXNdbo31l0BtTEfGCiFhA2cG3Bt6fmfcGTgTOAr5b1/NFqtVWuwX2i4gTO4vPBJ7vc0lqV0TMqQd2cyPi9hGxbUTcrDYo+NrvQURsHBGvA/4fcCtKN+2+mfnm2tPyHCCA70E5OF+Tv2PX7ZipH8TvBR4HnAM8FvhBZl5VV3k+pbvtZFjzHa912h+APSLiZ8DmwPnAJ30uSW2KiNmZuSQiNgH+l9JqNA+4LCIOzszv91vhuikzr46ILwFfBC7JzCsHoTsi9gbuDjyjhvFZazrMxskYYygi7khpbf1r3fGz6pHY44EPAs/OzFPqi3dpv9VqEkTETsDczDyrXn8mcAzwd+Dumfknn09SuyJiA+BHwFXAhylB7wHAgcCzgGMzc0l/Fa5bImLbzLx4aFkAszJzaUS8j7J/nji83nTZdTtGImKbiFgvM8/PzHM76X7QrP4g4ArgPAA/lLU6ImIL4OPAgojYpS7eHPgJcB3wlTrxZ2lE2MovVRExu+8aRuiRwGzgkMw8LjOPAX5Yb5s3CHl24659EfFu4P0Rcf/u8jpuemlE3JXSe/exmxrywKA3NiLiA8BbgZXt+LsBLwQ+mpl/7KNGTabMvAI4knIk/9GIuHNmfpAyLOAFwC2B02vL8ZLBh9tUA7eldcXg4Kf+/o6IeHTfNd1Etwa2Af4IEBFPB44AXp2ZH4+IzSNiG4dwrF0R8XnKWLxTgT9Pcft6wDMoQ7dOHL59TfhGPgYi4jjKh+6vgHOnuH025ZQq51Jn30irYxDWMvMI4COUcbmfjIh7ZOZlwP8BL6Gctuendd2lEbEV8Nb6U1qn1GEMg3OXvg94KnDVpLTwraRl/lrgesp27AN8FnhtZr6rbtchwKERMW8GS12nRMQbgF2AfYAjM/OiOjlm/c5qyyg9Lj/NzEtG8XcNej2LiNcC96Ek+I9m5p8HO70meyhjKe4GfD8zL+qn0tW3kin8Ptd6UMd2zq6/f5pyqp45wGERsUtmLgK+CrwY2Coizo6IAyhHko8GLu+pdKk3nZa8HSjdnW8Ffjgpw2Vqy/xGEfHMiNiyLj6Osi0/Bj5Dacl7R71tZ2BPyrjwRTNf8TrjdsD3MvOnmbkoIu5MaVX9v4h4V0RsXLvQP0rpuh1JV7ofvj2qH8B3BE7IzJ/UHb8T8OmIOAk4MiK2zsxrgNcCr6r3G9sxFPVIeFn9/U4RcY+I2GQST8o7HE4nLax2At6/Ppxq2PswJex9tBP2/g94LuUcja+nBLzda+veRG23NAoR8XLgImBf4NJJ6NIcask7kDLhap+I2CozFwIvonyr0gWUz5ctIuK+wMeA9YA31ccZ28+YSRQRs2vDzRbAZhHxmIh4DXAGsCNwGWXfvBogM88ajJkcxfPOWbc9i4hPAA8Enkc5ono1cBpwJSUEfh94cWZe11uRq6k7/TsijqFMHtmcMuD/ncD/Zub5PZa4RiLiY8BxdabzGk9xn0n1nFlLajfMgygTeq7KzNPq7ftRumqWAM/PzJ/XYLgecFvgnNoaOMeZeGtHRMzNzOvr7zEJQWJdEhF3oASffSnvy+8Z5xa9waz5egqVt1IO2l5NmVn/FsqErCXAE4H/oYzZ3Qi4FLgaeGhmXu/s+7WnToY7EUjKfvlU7ToPSivrpsDjRv0ZY9DrWW3B+yjlLNhnA5/PzHfXHf9lSqDfu88ap6sGo0dQWiB/T5lg8grK1P6DRzGLaKZExLbAFyjnM3pEZn5/3MNe53Q8m1BOrL0xZRD2tcDxlH2wqIa9l1DG7Tw/M3851ePMcPnrhCjnyzwd+Fpmvroum8iwV59n/0H5tp6J7OqfKtzU9+AdKKHogZTTXHynj/qmMtXzJcopVM4A/gIcTem1eyLl+9FfBRxez922KeVrtuYCvwW+NZh174Hd6ETEUyjnLLyO8j8+O8q3EG1CyV/n1/VuDiwAfgO8ZNTvA55KYYbVQbDbAf8EzsjMHwMPiYi7A1d2ZtTejJL6L63N8Usn4UMgIrYB7gu8A/hKZi6OiHOBdwO/ozRRj63hN8/MvDgiDgTeDnw9IsY+7NWQtx6lO/YqYD7ljWZr4CRgXkQcmJmfqT00BwNfjIhHZudr9cZ1+xqxGaW1/hUR8Y/MfFtm+YaCSXidD0TERpSvZ/wVZWb3xB0gdENeRPwbsAHldXNaZv4uIp4HHAV8ISKenJnf7a/aFawPDI+nexSle/DZg9Z74KiI+AjlPTkj4jOZ+Vfq/hqo/wdD3ohEmWR5X0qYBnhPRLwMOCIzL+2sdydKCL875SB89K//zPQyQxfg85Tvqv0d5YP3j5SvNhteb1fgcEoo2qnvuqe5jfegzBq6f71+F8p4r+OADeuyPQa/j9OFcqLKwe/rD912V0qT+zXAA4bX77nu9VayH35DOXfWoOV+n7pvXj607vOATwCz+96WdelCCd7vqPvkvzrLo+/aprENh1JC3nr1+pZ91zTN+qPz+3GU011cWvfJ5yit+ADbAl8D/jZ4/fdc926UWbNbDi1/dq19s3p9Tue24ylDgg4GNu97G1q+ULr8Lwb2Am5ePz8+QOk6P6SuE5QhTWdSvp3oHmutnr7/IevKhTLA/Q+U8VIbUJpz31eD30c66z2fMkbv3LW549fC9g3CxGaUroAXATtRTvD8OWDjevuT6/Xt+675Rrbl3fWFuvHQ8p3rm/3VwH3qsl7DHqVb/NjhN27KGdX/Cdy3Xn96/QB4Vb2+BbDPFI83EWFvksLQFLXP7fz+QEoX2zLKWNyJ2j7KOM8rKAd0X6N0G97gwGPcL8B/U4aZPIQyRvVO9X3sLOCedZ0dKAd7ywavqx7rfSnw+imW71IDxiGDkNf5Ob/WfiXw9LpsLA5WW7pQxjmfCBwztDwoB3bXA7vVZfeqnzW3XZs1OZtu5uwK/Az4UWb+MzP/QOkO/Djw+IjYp44J+T3wacqR5C/6K/fGxdD5pLI+a4HFwC8op+v4IXBKZu4DXFvHITyW0i199QyWO133pHwl0HPr+CMAMvNs4F2ULvUTImLP7L+LaiPqdyQOLb+acvS4RZTvTPwspdXonfV59hjgwIi4bfdOOaaDsG/k+TZRatfsYALGMcCbKbPullHOlP9aKNs37jMfa30/pLTofZvl38s59hPHuup4yXsCn6J0114EXEI5kfgPKSeuJTN/Tzlp/Rfp+bRDmfm+zHxLRGwQEe+MiJ3rTb8Ffgn8O/DQoTF3SygHsV+iPNduMQbvXy26ntJde7MpbvsgpRHnOXUy1hnAm3Ntnzat7/Tb+oUyGHY9ykSEY+uyOdQjKUqXwO+BD3Tv03fdq9qmzu/zKTO6ngrcpi67FeVr2hbV24PSXXsEpTt65763oVN/t+tmduf34yhd7K8ANhm6z8nAQsqA5w0Zg9aXWse7gNt3lh1R98EyygDfwfKdKJM0Pj4Otd+E59ut+67tJmzT2yit+f9GmWl3z7o/lgGv66w3Cfvnl5SJPr9ieevX2L6HMdRqTQl0/6C2qFJa865gxeEmjwW2qr/PmalaV2NbnllrPRa4S112q/redQbwMmArSk/SaZSgt1vdX3v3XX9rF5b3bH2Y0rJ61ynW+RFl0uXM1dX3P6blC2W242DHv5zS2jUYuzanc9tXKQPnJ6LbrLN9x9UPq4trgPsa5dxrgzebH1COXhbWD4GzGaPu6Cne8OcMXf9ifcN8ZecN/3aUI+JHA1v3vQ2dWu9XQ8ICajcApaVoAWU86FMo4/YeT/mO29NZ3qUz9mHiRp5vu/Zd1xpuy9cok5W6y25NGcezDHhpZ/lY7Z/u84YSUo+nDE05jXIy3nuNY93DNVGC9T06++NoyuD5yynDS25Wb9sTOAW4X9/1r2Sb/pMyHvdz1IPo+v77f/W1sqy+Xs6ot92vXt+z79pbuVDOarB15zmzEeVchd8DbtdZ75bAdyiBe9ZMvUZ6/we1eqG0rryL5UdZOwJfp3Rr3ruz3pb1DfIGkzLG7cKKLV571bofTBlzeGDnjX4wfm0zSkvF/sDuwC373oaVbMsbKOcwOoPSZXuHzm3HUbpDjqEMdP40ZdzOVn1vQ6fGwQHDwyldtl8AtqvL7kRp2buSct6mMynjR+YO/x/G7TKN59u9+q51Gts0q16+DJxACUvd7dy5fggvo3Tp9F7zjWzLQXQmA1DGgZ5GOZAYu7DHii3Dr6FMrHpQ5/rCuuxLnfW2oExU+kHf71+sGLBj6LZXUrqYu2Fvk/p8+nfgYSzvRVpQ192m733SwoVywHA6pWX1ZODAunyP+tlxHmXS0iGUCZlXAnea0Rr7/ie1eKnh4IIaILbpLN+bMpbl75Rvungdpcn974zp7NrOm0P3SPhQytiiI1lxVtc+nQ/fPfqu/Ua2qbstC+q++hDlfFkLKec1vEdnnXdTwt3llKDea6skN9J1RDm9wjWU1sjtOsvvUd947sDyYDg2XVAjeL6NZdhjJV2YlNMpXMfyyTLd7fpSfZ5dRjkQHJuw1KnxPpQwemT3Q4vyVY5jG/ZqPY+gfBPEc1lxYsxH6za9l/KVk4+ijNu7ErhbzzXPrj83rvWdQGn9fW5nnZezPOzdeYrH2LfedkXf72GtXCgNAH+sz6WXUCb1/GsWfX39/i/lFER/oI5lnfE6+/5HtXYB/h9lzN29gY3qsvU7t+8IvJ8y2PdC4Ft97PjV3JYNKV2uO3WWbUzpjl1W32xmDd1n8OH7Azotl+N4oYz1Opfl3c2DUxNcRjlv1l07696acnTc6+kjht7w/4fSwng4pcV0MLP50ZSw9wVWMptreL+Nw6W15xsrttRtD9x58J5Ql51MGefZPagYnMbj36mnyBjXC7Bf3S9HMHXYG8d98lLgT/X//pC6bF7n9ndTWmeuo3SH/rDv92eWH5htTAlyp9fX9uB0L8d01v1PSqg4lqGx0JQzOnx9eLmXNd4v96r/68ezvIdkj/qa+AQrfu5vRem23aSXWvv+Z7V0ocyyOQF4Q2fZbSlHj58F/ovl4W9LSj/+xn3Uuprbc2fK0ePNhpZvxfLTjDyaG451e2p9Q/om5aSeY3FEz4pdN5tTWl6fVa+/gjIr7dGUAczLKGFvbEJ45w1/Q0or5DmUSRXnUc799UZqCzKlNeJqSovl7fuquY/nW9/bMcVz7WhKgF1GOUny2+vyO1CO8K+mnHbh0PoBfjGd1ti+L9wwXK/XeS6uLOztM6bvATux/BQpH+4s734ob005cNqWMQnblO7a91DC846d5Z+o27JXZ9nLKK12bx7ct3Pb2H7eTNqFMozkapa3yt+h/t8/w/Ix3WMxhrj3Alq4dF88LJ9YcU/K0dW1lK6l0+sH8kspY3TGdmzU0LZ1ZxHdv7N8S8rR7kX1CT/84bs3dRZu35f6Jtn94N2/fvg8kvItJQ+iDPI/sLPOifUDdwFj0K3O8i7N2ZSw83+19sGJag+jTBx5Pctb9h5RPwTe0nf969LzbaiuT1Ba+PenHPm/idJi/IV6++aUsbxnU7p2fsyYdqtRTu565/r7XG4Y9j5Jp8uQcs7M3vbJyt5jKQffJ9TnUvfchXNnoq6bsD1fBz7Tuf40yqk8XlGvdz+HnsGKLcpjEbQn/TL0P/43StDbob4/DZ8z9kn186P3SXu9/+NauFBaVd5Tf38iZfr04HQDr6vL51LOo3d43/Wu5jZ1jwK3B35O+Vqg3TvLt6zb+rupPnz7vlBaHjYaWvZpyukgut1lL6aMwbtVZ9n/Us5Wfk53ec/bM4/Syvg94CvcMMB+jDIkYNfOsnszhmPxWny+TbEdd6J07Tyd5YH8XpQTWS+gc2Jhyqy9TYFN+96GlWzXLjUYnU+drMSKYe8Q6rkA6Xk8W62n20L3QMrB0Sad/XBHysHSLyhfOzVYdyyeUyw/sJtFOUPDevVz5iN1+TPr//vV9fo8ypi9xw49zlhsTysXOp/19foPKL0rV9b35k3q8q0pLXvH0lN37Qp1913ApF8oTfxnU75bkPqi3J4yhX1w9BuUI/eTKOfPusGsqXG6TPXmQGmW/jrlfFN7dJYPPnzPpzOzq+8LpRv9R8BzOstuRTlNwiNZcfD7G4FFneu3oByZ3YuhbsQ+9wXlyPHnlEkhx3eWdwPDhcBHB8+7zvKxDXuT/nyrAeI9UyzflTK5Z896fae67z7L8q6dh/Vd/0q26Qb/V0oL8U/rB9sd67L168/bUA4yllEmNs146xhlKMybh5YtqP/zZbXu+Sz/erAdKWHvl8AL+v6fD//vKb0Op7D8a9j+i3KQ8ErKMJPu+RYfCJwKPKXv+lu9sPyz/gCWH+A8jtJbt5AVT2t1JGUcaO+9QZkGvVHs/NdRvh/xjjeyzl0oA+b/emPrjcNlKFgcQJkd+ErKZIRdKefMGv7wvQWl5esXjMF32FKOfr9Hmfk3OML6aH1BnkZtoeu8od65fgj8knKaha9QBjmv1a+lWY3tGJxOYUPgOZQxQ7ehjHtazNBXZlGO6k8Djux7H6xjz7f9KJOqNhhavlP9YH4EZSD9cNfOoymz9np9nq1in+xCOTAanCbqgZSeiQtYcazYvSiTg54xWLeHup9CnQlcrz+HMpHnybW+r1K60V8NbFHX2bE+x/4IHDQG//vBa349ShfzNZSDuwcDt6cEv2Ws2Kp0Z0rL0knYgrc2980NPuvrfnp63UdX1NfGGZTW7136rvlfdfZdwCRf6hv5X4CX1es3OAEiZUzeqXXHj+XYm5Vs2xcpY4Z+Tekq+wvwPMog/xMpRzDdbrWbMyZjpCinRvgt8Mh6fT7l/HiX1RfjbkPrzwGeQBkY/3tKWOp7pt1gdu0m9c3jW8AL67Kd6xv+r6jfeEEZu3e7ut3/3fc+WBeeb5Tz+Q2+s3LQJdhtQZ5DmYjxl/qBfRTLQ96WlK6d/2WMumtZcSjAMZQhDddSWvIOr8sfUD/M/kQJIPelHMh+p4+gQWnJGwyIP6j+rz9KOXB45dC6x9bn2WtYHvbuTJmYdbuZrHtl//v6mv9ufU2cRTlYuLD+rx8HfIPyjTdvrvvop5TzY87pPo6Xke6bqT7rB+/RQTkYP5hyoPp0xuxbe3ovYBIvnR28X/1gGrzJDF6om7L8CHh/yvn0JmLmY635NZQj3PtQB5JSBi9fUT94d61vNpczhmeLp3TbnkU5yv1S/aDape6Lq+ub4wZ13W735izKBIdeu2s79WxICXnfoHQbdE8DsTOlZW8ZpfvpxHr9V4xxN20rz7f65n44Zbbzw+uyB9f9cXRnvUfUD+3LgcfXZbtQunb+xhTnOxuHC2UW7e8orXk3p7R6LaOeTLjuk8Hz71JKt+2MH8jW/XAs9RQj9bX/XMqBwTLgP+vy7pi9Qdh7JXDzumwsJmJQxj2eSpl4dC/KWK+HU4YrXMTylr3XUQ72vkgZejIIeRP12h/3C6v+rN+cnlqwp7UdfRcwqZcaCs5hxVlQm1C6Y75W32ReVN+IxuJNZBrbdnT9IBq0Uty6vpl/luXnC7oHZYbgH+gEkL4vLB87sSflXFhXU8/lRenafDal2/MjQ2/+YzdmkhJMf82Kp624O2Vs2j0ppx35JuWE21+lM95rkt7wJ/X5Rpls8Q1Kl/9eddlzKQcWn+qs90TKN2FcX98zflMvY9nCT5nB/LPONr2kvpY+QRl+8oXOuo+lhMHeWjAoQxoGp60ajB2cX2v9eme97ljWT9X3hsFZEMbi9U8ZmnERK04Qifqa+BGlZW9w/r+5Q/e123bt7JNVfdYvAV7T3V9913yDbei7gEm7dILEsyldfLvW66+ltKwspRwxPnf4hTgJl/qk/i7w5Xr9dpQZRZ9j+QDyAylHlXcGtu+75pVsx9sprQyXUc5dNjgC24AVw956fda5im14eQ0RsymzMl9KCXWXsvxAYivKkf0PqROCBvux7/rXhecbZdLIqZSW1MEH8HPr8+vTnfW2pLTMvLAGo7GYyT3F9gTl6+ZeXq8/h9J1+LT64Tb49ojjxu0DjXIuzL8N7YdFrNjC2g17n2DMxkxTvnLtIuCtU+yXR1EC91nUlu1JeZ1P4mWan/VjHbJ7L2BSL5TzfJ1DmdL+U8rR42HAQ6d6skzSpW7TacBjWD6AfNN62041VDyj7zpXsQ07AHeljL37MyXsDbo3BmFv0I07lmGP0oq1uO6L02u9L6Kcff1QyulHNqJ0o51Sn4cv7Lvude35toqw96k+a1vD7dm4BtON6355K8tD9+0p55dcRg3n43Kpz5VTKQdHw/uhG/bG4qTaK9mGDSjd5D9iaMYmpeXybMo44osYOnWUl7W2Tyb+s773AibxQhkftaxevlKfCFuxvJspuj8n7VID0tWDN/NOQLoF5Sj4l4zZYNMb2Za5lK6zqcLeCyitfr1+Wfkq6r8fZbzUK4B7dpa/mNK9NhhQftcaBr/NGA3uX1eeb6sIe0f0Xd8abtOtKBMuXt5Z9mTK2NcD6Xnywkpqvv2k74f6eriGcmqY7gmo70MZi3t/Sth+R9+1tn5p5bN+UKSmISI2pLzR/R04ITOvrMtnZeayXosbkYj4N8oT+8eUNxwoXQcPoZwX7Jd91TZdEbE+pfaPUGalPjgzl0TEPMrR/VW9FjgNETGX8mH2ccqb/b6D51xE3AW4JjN/32OJa6SF51tE3IEyQWMr4EWZ+a2IOIhyIuvDMvMFvRY4TREx+DaSn1MG/P+TMqtwI+B5mXlNj+WtVAv7ISIeSZlo8SvKbOZLgP+gzPx8EmW/nJ6ZB/VW5Dqglc96g94aiojZmbm0cz2ysX9mROxOGeu2HWU8wrnA6zPz7F4LWwMRsR5l8Oz/UGbk3aO7/yZBRNycMmbqcZRutd1rYJ0NLJv0518Lz7ehkPHCzPxORBwAnJaZv+m3uumLiAdSTlx9DSXobUCZpDHWwbuF/RAROwNvoQzhCMrErCez/DyhJ1FmrDPpr/1ViYgHZOb3e/rbE/9Zb9DTjYqIDShvLEuB6zNzcc8lrbEa9p5IGd/2qMz8Xb8VTU9EPIzSsnIRcEANeXMyc0nPpY1MC8+3GjI+Qun22aevD6hRiYi7Uca6LgK+kpkX9FzSamlhPwx6HSgzzS+NiI2BD1IO9u6bmef3WuAMiIi9gJMp3+n7nr7rmUQGPa1TathbLzOv7ruW6YqIwWkW/piZOXykqfEREXcC3g0ckpm/7buedVVL+6Ee6B1KeQ94XGb+vOeSZkREbAb8J/DZzDyn73omkUFPmkCTNkZkXRQR62XmdX3Xsa5rZT/U1rwDgJPWhZa8Lt/vbhqDniRJUqNm9V2AJEmS1g6DniRJUqMMepIkSY0y6EmSJDXKoNejiJjfdw2j4raMn1a2A9yWcdTKdoDbMq5a2Za+t8Og168mnsSV2zJ+WtkOcFvGUSvbAW7LuGplWwx6kiRJGj3Po7cSEdHMP2bbHW631v/GNVcvZKONb7ZW/8aia/65Vh//X39n0bXMm7fhWv0bV1x+6Vp9fIDMpHyZxtr+O57HVJLGwGWZueXwwjl9VDIpItpo8HzxG9/edwkj8ZsfTcR3ka+WBZ/6775LGJlFi67puwTdQDPHqZJW3++nWthGkpEkSdINGPQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEY1HfQi4r4RcXxE/DkiromIn0fEfn3XJUmSNBPm9F3AWrYD8APgMGARcH/gyIhYlpnH9lqZJEnSWtZ00MvMBYPfIyKA7wLbAQcBNwh6ETEfmD9jBUqSJK1FTQe9iNgcOBR4ArAtMLvedPFU62fm4cDh9b45EzVKkiStLU0HPeAo4D7AW4BfAwuB51OCnyRJUtOaDXoRMQ94DHBwZh7WWd70BBRJkqSBlkPP+pSu2sWDBRGxCfD43iqSJEmaQc226GXmVRHxU+ANEbEQWAa8GrgKuFmvxUmSJM2Allv0APYFLgKOAT4AfLH+LkmS1LxmW/QAMvMC4KFT3PSmGS5FkiRpxrXeoidJkrTOMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjYrM7LuGsRQRzfxjNt/8ln2XMBJf+P43+y5hZF70lPl9lzAy5573k75LGJmlS5f2XcKINPP2JWn1nZGZuw0vtEVPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1Jkl5HgtsAAB7KSURBVKRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGrTLoRcRDIyIj4ladZT+KiKURsVln2VkR8bb6+y4RcUpEXBsRV0bEZyJi6866t6mP+fSIODIiFkbEnyLimfX2V0bEnyPibxHxzoiY1bnvThGxICL+WB//7Ig4ZGidB9fHf3BEfD4iro6I30bEC276v0ySJGkyrE6L3o+A64EHAkTEhsC9gOuA+9dlWwA7A9+LiC2BbwMbAvsCLwL2BE6OiPWGHvudwF+AJwPfA46OiPcAewDPBt4PvBJ4Wuc+2wLnAi8AHg18HDgUeNUUtX8c+AWwd63pwxGxx2pssyRJ0sSbs6oVMvOfEXEGJeh9DrgPsBD4Zl32VeABQAI/BF5b7/qIzFwIEBHnAT+mBLpjOw9/ama+tq7zY+ApwOOBnTJzKXBSRDyBEtQW1HpOAU6p9wng+5RQeRDw9qHyj83Mt9Z1vw08DngS8JNV/2skSZIm2+qO0fsetUUPeFC9/p2hZb+owW4P4BuDkAeQmT8BfkcJhF2ndNZZCPwN+E4NeQMXUFrxAIiIeRFxaERcACymtDa+DbhtRAwH1290Hv964Hxgu5VtZETMj4jTI+L0la0jSZI0KVY36H0XuGsdk/dAStD7HrBbRMzrLAPYBrh0ise4FNhiaNnfh65ft5Jl8zrX3wm8HDic0nW7O/DWetu8Fe+6ysdaQWYenpm7ZeZuK1tHkiRpUqxu0PsBEMCDKV233wXOBq4G9gLuyfKg9xdgqykeY2vgiptQ68BTgQ9m5rsy85uZeTqwZASPK0mS1JTVCnqZeSXwK+ClwFLgzMxMyvi4V1LG+n2/rv5j4BERscng/hGxO3Cbzjo3xQaULtvBY88Gnj6Cx5UkSWrKdM6j913KWLwfdsbQfa8uOz8zL6nL3lt/fj0inhAR+wFfAs4CvjiCmk8GXhgR/x4RjwFOANYfweNKkiQ1ZTpBb9A1+90plv2rpS4z/wY8BFhEmWH74brewzLzujUv9V9eVB/vw8ARlJbG4dm2kiRJ67woPbAaFhHN/GM23/yWfZcwEl/4/jf7LmFkXvSU+X2XMDLnntfO2YqWLl266pUmQjNvX5JW3xlTTSb1K9AkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJalRkZt81jKWIaOYfE9FGnt9k4837LmFkrlp4Wd8ljMzcuev3XcLILFlyfd8ljEgzb1+SVt8Zmbnb8MI2EoAkSZJuwKAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqNGFvQiIiPi4FE9Xudxj4qI00f9uJIkSa2b03cBq+EtwAZ9FyFJkjRpxj7oZeaFfdcgSZI0iVar63bQfRoRT4yIcyJiUUR8PyLuciP3eUxEnBwRf42IhRFxWkQ8vHP7zrW7d8+h+20cEVdHxIu7f7tz+/71fnerj39NrelJQ48TEfGWzt8/IiKeXu97m9X790iSJE2u6YzR2wF4L6UrdV9gU+DrETFvJevfFjgB+HfgycAPga9FxP0BMvNs4DTggKH7PRWYC3x2FfV8Fjge2Bs4H1gQEdt1bj8EeC1wGPAU4J/Au1a5lZIkSY2YTtftLYAnZOYPASLiDOBCYH9KmFpBZn5o8HtEzAK+BewMPAf4Qb3pk8D7I+LgzLy6LjsAOCEzL1tFPe/LzCM6tVwKPBY4LCJmA68EDsvMN9T1vxERtwVuPY1tliRJmljTadH76yDkAWTm74EzgD2mWjkitouIoyPiYmAJcD3wcGDHzmoL6s+n1vvcHngAcORq1PONTi2XA38FBi16twZuSWnx6xq+Plzz/NpF7SxfSZI08aYV9FaybJvhhbUF73jgfsAbgIcAuwNfA/7V1Vtb8Y5jefft/sAlwEmrUc/fh65f13nsW9affxtaZ/j6CjLz8MzcLTN3W42/L0mSNNam03W71UqWnT3F8jsAuwKPysx/hbaImOo0KZ8AfhARdwT+AzgmM5dOo66pXFJ/bjm0fPi6JElSs6bTordVRNxvcCUitgfuCfxkinUHgW5xZ/0dgPsPr1i7g88BjgC2B46aRk0r80dK2HvC0PLHj+CxJUmSJsJ0WvQuAz4VEa+nzGB9M6Xr9qgp1j0H+BPwnrr+JsChwMUreexPAu8GfpSZ50yjpill5tKIeDfw7oj4G2Xyx+OBu9VVlt3UvyFJkjTuptOi93vgFcCbKJMoFgKPyMxFwytm5mLgSZRJGF+gnJLl7cB3VvLYX6k/j5hGPavyPuD/AS8AvghsXq9DqV2SJKlpkZmrXiniKOCua2uSQkS8gHKOu1tl5loLYRHxCeBhmbnDaqy76n/MhChzYybfJhtv3ncJI3PVwlWdPWhyzJ27ft8ljMySJdf3XcKINPP2JWn1nTFVTuv1K9DqN1TsSDmx8VGjDHkRcVdgH8qJmpcBj6LM7n3VqP6GJEnSOOv7u27fRPmWje8Arx/xY19DOSffwcBGlK7nVwHvGfHfkSRJGkur1XW7LrLrdvzYdTue7LodR828fUlafVN23baRACRJknQDBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGzem7gHE2a9bsvksYiczsu4SRuO76xX2XMDI/OO+8vksYmW23vWPfJYzMFVf8pe8SRuIf/7ii7xIkjQlb9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVFNB72IuG9EHB8Rf46IayLi5xGxX991SZIkzYQ5fRewlu0A/AA4DFgE3B84MiKWZeaxvVYmSZK0ljUd9DJzweD3iAjgu8B2wEGAQU+SJDWt6aAXEZsDhwJPALYFZtebLl7J+vOB+TNTnSRJ0trVdNADjgLuA7wF+DWwEHg+JfjdQGYeDhwOEBE5MyVKkiStHc0GvYiYBzwGODgzD+ssb3oCiiRJ0kDLoWd9Slft4sGCiNgEeHxvFUmSJM2gZlv0MvOqiPgp8IaIWAgsA14NXAXcrNfiJEmSZkDLLXoA+wIXAccAHwC+WH+XJElqXrMtegCZeQHw0CluetMMlyJJkjTjWm/RkyRJWmcZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGRWb2XcNYioiE6LuMEWllH7eyP2DvvQ/pu4SRWbp0Sd8ljEwuW9Z3CSNxwokf6bsETWHWrDbaVpYtW9p3CZraGZm52/DCNp51kiRJugGDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjZrxoBcRO0fESRFxRURcExG/iYgXdm4/OCLOj4jFEXFBRLx06P5viojLImLXiDgtIq6NiDMj4oFD660fER+NiL9HxOUR8e6IOCQicqa2VZIkqU99tOgdDywFngk8HvggsAlARBxUrx8PPA74PPCeiHj10GNsCBwNfAx4MrAY+HJEbNhZ513A/sChwH7A9sDL1soWSZIkjaE5M/nHIuIWwO2AJ2bmWXXxKfW2WcCbgKMycxDIvhERmwKviYj3Z+aiunwD4JDMPLXe9y/AmcCDgJMi4ubAfOANmfm+us7XgV+tor759X6SJEkTb6Zb9K4A/ggcFhH7RMRWndu2A25FacXr+hxwM+BunWXXA9/uXP915zGo686jtAwCkJkJnHBjxWXm4Zm5W2butlpbI0mSNMZmNOhl5jLg4cAlwBHAJRHxvYjYFdimrnbp0N0G17foLFtYH2vwuNfVX+fVn7esP/829FjD1yVJkpo142P0MvOczHwysBnwb5Rw9lXgL3WVrYbusnX9ecU0/swl9eeWQ8uHr0uSJDWrt9OrZOb1dYzdeymteVcDfwaeOrTq04CFwFmsvrOARcATBgsiIigTPCRJktYJMz0Z4+7Af1PG3f0W2Bx4FfCLzLwiIt4EfCwiLgdOBvYEng+8tjMRY5Uy8/KI+DhwaERcD/wGOIAy1s/Tq0iSpHXCjAY9SpfqpcB/USZe/B34FiXskZkfj4j1gUOAlwB/Al42mDk7Ta8E5lJm8i4DPgV8sj62JElS86JMRl03RMQ3gbmZuedqrJsQM1DVTGhlH7eyP2Dvvds53li6dEnfJYxMLlu26pUmwAknfqTvEjSFWbPa+DKqZcuW9l2CpnbGVGcNmekWvRkTEQ8B7g38jNKytw+wFzccAyhJktSkZoMeZXLHE4HXUGb2ng/sn5lf6LUqSZKkGdJs0MvMnwL36bsOSZKkvrQxYECSJEk3YNCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElq1Jy+C5DWReeff0bfJYzMrFntHC8+9bnP6buEkfjmKcf0XcLILFp0Td8ljMycOXP7LmEkrrtuWd8laEo55dJ23qElSZK0AoOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjJjroRcTTIuKsiFgcEX+MiLdFxJx62/4RkRFxt4g4OSKuiYhzIuJJfdctSZI0EyY26EXEw4HPAT8DngB8EHg58KGhVT8LHA/sDZwPLIiI7WawVEmSpF7M6buAm+DNwLcz81n1+kkRAfD2iHhrZ733ZeYRABFxBnAp8FjgsJksVpIkaaZNZIteRMwG7gl8fuimz1G26b6dZd8Y/JKZlwN/BaZs0YuI+RFxekScPtqKJUmSZt5EBj3gFsBcSutc1+D6Fp1lfx9a5zpg3lQPmpmHZ+ZumbnbSKqUJEnq0aQGvcuA64GthpZvXX9eMbPlSJIkjZ+JDHqZuRQ4A3jq0E1PA5YBP5rxoiRJksbMJE/GeCPw9Yg4ElgA3A14C/DxzPxTnZghSZK0zprIFj2AzPwG8HRgN+AE4BDgPcDBfdYlSZI0Lia5RY/M/Bxlpu1Utx0FHDXF8tus1aIkSZLGxMS26EmSJOnGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGjWn7wKk1Zd9FzAy5533k75LGJmNN9687xJG5uLz/9R3CSOx++6P6ruEkfn5z0/tu4SR2XLLW/ddwkhceOGZfZegabBFT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaNdZBLyKOiojT+65DkiRpEo110JMkSdKaWyeDXkTMjYjZfdchSZK0Nk1E0IuIh0XELyPimoj4fkTs3LltVkS8OiIuiIjFEXFeRDxr6P7fjogvRMT8iLgQWATcaqa3Q5IkaSbN6buA1bA98G7gbcA/gf8GjouIu2ZmAh8EngW8GfgZ8DDgiIi4PDNP7DzO/YHbA68CrgWumrlNkCRJmnmTEPS2AO6fmedDacEDvgzcKSKWAM8HDsjMo+v634yIbYA3At2gtxmwa2ZeMnOlS5Ik9WcSgt7vBiGv+nX9uR2lhW4Z8OWI6G7LKcAzImJ2Zi6ty85YVciLiPnA/BHVLUmS1KtJCHp/H7p+Xf05D7gFMJuVd8NuA/yp/n7pqv5QZh4OHA4QETntSiVJksbIJAS9G3MFsIQy/m7ZFLf/tfO7wU2SJK1TJj3onUpp0ds0M0/uuxhJkqRxMtFBLzPPjYjDgAUR8S7gdEqX7s7Ajpl5YK8FSpIk9Wiig171QuA84CDKKVYWUiZsfLLPoiRJkvo21kEvM/efYtnvgOhcT+D99bKyx3nw6KuTJEkabxPxzRiSJEmaPoOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqPm9F3AeMu+C1Cjrrtucd8ljMzVV1/Zdwkj848r/tF3CSNx7rk/6buEkdloo037LmFkHrTXE/ouYSQuvPDMvkvQNNiiJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmN6j3oRcS3I+IL07zPgyMiI+KunWUZEQePvkJJkqTJNKfvAoAXANeP4HHuC1w0gseRJElqQu9BLzN/PaLHOW0UjyNJktSKtdp1GxEHRMTiiNhsaPnOtat1r6m6biPioRHx44hYFBGXRsRHImLjVfytFbpuB48bEftGxAURsTAivhYR2412KyVJksbT2h6j96X6c++h5fsAfwW+PXyHiLgLcBJwGfBk4I3AvsC0xvFV9wYOBl4GzAfuCRy+Bo8jSZI0cdZq121mXhURJ1GC3ZGdm/YBPp+ZSyNi+G5vAH4PPD4zlwJExBXA5yLivpn5o2mUcDPgMZl5ZX2cWwLvi4gNMvOfwytHxHxKIJQkSZp4MzHr9nPAXhFxC4CI2AXYsS6fyh7Alwchr/oisAR4wDT/9k8HIa8ajAfcdqqVM/PwzNwtM3eb5t+RJEkaOzMR9I6nzKp9Ur2+D3Ax8P2VrL8NcGl3QQ19lwNbTPNv/33o+nX157xpPo4kSdLEWetBLzOvBr5KCXgATwOOy8xcyV3+AmzVXRARs4GbA1esrTolSZJaM1MnTF4A7BkRjwNuV6+vzI+BvWu4G3gSZTzhyloBJUmSNGSmzqP3VeBa4GPARZn5kxtZ963AmcBXIuKjwHbAO4GvT3MihiRJ0jptRlr0MnMRZazeNqx8EsZg3bOBR1G6b79ECX7HAk9Zy2VKkiQ1Zca+GSMznwk8c4rlD55i2SmUc+Ct7LG+DcTQsuHrUz3uDe4nSZLUqpkaoydJkqQZZtCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGjWn7wKkddF6663fdwkjM3v23L5LGJkzT/9O3yWMxCab3LzvEkbm2muv6ruEkdl8my36LmEkZs82OoyjpUuXTLncFj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGGfQkSZIaZdCTJElqlEFPkiSpUQY9SZKkRhn0JEmSGmXQkyRJapRBT5IkqVEGPUmSpEYZ9CRJkhpl0JMkSWqUQU+SJKlRBj1JkqRGzem7gHESEfOB+X3XIUmSNAoGvY7MPBw4HCAisudyJEmSbhK7biVJkhpl0JMkSWrUOhf0IuI/ImJJROzQdy2SJElr0zoX9CjbPBuIvguRJElam9a5oJeZR2VmZObv+q5FkiRpbVrngp4kSdK6wqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1Kg5fRcw3qLvArSC7LuAkYlo5xgrop3XyfXXL+67hJFYb+76fZcwMrHRZn2XMDKXXnRJ3yWMxOzZ7USHaOhzfunSJVMub+fTRpIkSSsw6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjTLoSZIkNcqgJ0mS1CiDniRJUqMMepIkSY0y6EmSJDXKoCdJktQog54kSVKjDHqSJEmNMuhJkiQ1yqAnSZLUKIOeJElSowx6kiRJjVprQS8ibr+2HvtG/uYtI2LDmf67kiRJ42ikQS8i5kXEfhFxKnB+Z/msiHh1RFwQEYsj4ryIeNYU9z84Is6v61wQES8dun27iDguIv4aEf+MiAsj4i2dVR4J/CUiPhYRu49y2yRJkibNnFE8SETsAhwI7AdsCBwPPKazygeBZwFvBn4GPAw4IiIuz8wT62McVNd7L/B14CHAeyJi/cx8R32cY4ANgPnA34HbATt1/s6XgZsBBwDzI+Is4JPApzLzilFsqyRJ0qSIzFyzO0ZsSgl2zwHuCfwcOBL4dDdURcQdgPOAAzLz6M7yY4A7Z+buETEL+CPwjcw8oLPOR+rf2DozF0XE1cAzMvOE1ahvV+DZwL7ARsBXgE8Ap+RqbHREJMSqVtOMWrPn6jhaf/12RhjMnj2S48WxsO22O/ZdwkjMnbNe3yWMzPVLruu7hJHZ474P77uEkfj8gvf2XcLIREOf84uv++cZmbnb8PI16rqNiEcCfwHeAvwA2DUzd83M/5mi5WwvYBnw5YiYM7gApwC7RMRsYDvgVsDnh+77OUoL3d3q9Z8Db4+I/SNi+xurMTPPzMwX1cd9FrAZpaXwtzeyXfMj4vSIOH1V/wNJkqRxt6Zj9BYD1wLzgE2BzSJiZbH4FsBs4Crg+s7lKErX8Tb1AnDp0H0H17eoP/cBTgfeB/w+In4eEXutotZBjZtStvfKla2YmYdn5m5TJWJJkqRJs0Z9Lpn5rYjYFtib0nV7KvC7iDgKODozf99Z/QpgCXB/SsvesL+yPHBuNXTb1p3HIDMvBvavXb17AG8Cjo+I7TPz8sGdauh8KGWs3pOA64DPAi/IzDPXZJslSZImzRrPus3MxZm5IDMfRpkU8RngIOCiiPhmROxXVz2V0qK3aWaePsXlOuBPwJ+Bpw79macBC4Gzhv72ssw8DTiUMvljB4CI2Doi3gRcBHwT2B54HrBNZhryJEnSOmUko6gz83fA62vIeiRlBu5RwGcy89yIOAxYEBHvonS9zgN2BnbMzAMzc1m978ci4nLgZGBP4PnAa+tEjE0pY+yOoUzuWB94GXAJ8JtayqMowe5o4BOZ+a9TvEiSJK1rRjpdLjOXAl8FvhoRW3dueiElnB1EOcXKQuDXlFOfDO778YhYHzgEeAmlle9lmfm+usoiSsveS4BbU8YIngY8PDP/Wdc5njLrd8kot0uSJGkSrbXzImTmpZ3fE3h/vdzYfT4EfGglty2mBMUbu7/nypMkSar8rltJkqRGGfQkSZIaZdCTJEn/v107NlIghoIoOMoBbJK4aImJJLDvchD+2YB+TXWb60jjvdpdSgk9AIBSQg8AoJTQAwAoJfQAAEoJPQCAUkIPAKCU0AMAKCX0AABKCT0AgFJCDwCglNADACgl9AAASgk9AIBSQg8AoJTQAwAotfbep+8w0lrrN8nzw8dckvx9+IxvsWWelh2JLRO17Ehsmaply7d23Pbe1/8Phd5Ba63H3vvn9D3ewZZ5WnYktkzUsiOxZaqWLad3+HQLAFBK6AEAlBJ6Z91PX+CNbJmnZUdiy0QtOxJbpmrZcnSHf/QAAEp5owcAUEroAQCUEnoAAKWEHgBAKaEHAFDqBRdWDPu5ezKgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "translation, attention = translate_sentence(src, SRC, TRG, model, device)\n",
    "\n",
    "print(f'predicted trg = {translation}')\n",
    "\n",
    "display_attention(src, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's get an example from the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "src = ['die', 'person', 'im', 'gestreiften', 'shirt', 'klettert', 'auf', 'einen', 'berg', '.']\n",
      "trg = ['the', 'person', 'in', 'the', 'striped', 'shirt', 'is', 'mountain', 'climbing', '.']\n"
     ]
    }
   ],
   "source": [
    "example_idx = 18\n",
    "\n",
    "src = vars(test_data.examples[example_idx])['src']\n",
    "trg = vars(test_data.examples[example_idx])['trg']\n",
    "\n",
    "print(f'src = {src}')\n",
    "print(f'trg = {trg}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, it produces a slightly different translation than target, a more literal version of the source sentence. It swaps *mountain climbing* for *climbing on a mountain*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted trg = ['the', 'person', 'in', 'the', 'striped', 'shirt', 'is', 'climbing', 'on', 'a', 'mountain', '.', '<eos>']\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlUAAAJ9CAYAAAAc+YfpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdebgcZZn38e8dAgRBFgVklx1cUBAEtxlxUBFR2ZRN3xEdZRQdl0FGQdzAFUVcAaMiOi6gKAw6sisOojiAgghGEVnFABKQESFAcr9/PE+TSnOSc5LUOdV9zvdzXX0lXV3d/XSd7qpfPVtFZiJJkqRlM63rAkiSJE0GhipJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiQNpIhYbhHLY6LLIkljEV77T9KgiYjlMnNeRKwMHAUsD9yamR/tuGiStEiGKkkDJSIiMzMiHgX8qi6eB2wG/AR4VWbe3lkBJWkRbP6TNDBqDVXWpr9nA7OA5wE7A3sCWwOnRcS63ZVSkkZmTZWkgRIRM4DPAY8HbszM19XlywHPAr4JXA/sm5mzOyuoJPWxpkrSoNmKUku1LfBQb2FmzgN+ChxICVzfioj1OymhJI3AUCWpU/2j+TLzSuAg4Frg5RHxqsZjSQlWr6QEryMmrqSStHg2/0nqTERMz8yHatPeKsD9wEN15N+zgY8BKwDHZuapjecF8BTgN7UGS5I6Z6iS1InGtAmPBk4CNqWEqlnA2zPznsUFq/7XmciyS9JIDFWSOlOnTbgU+CvwA2BtYCdgE+CfMvOaiHgmcAywHDAzM0/uqLiStFj2qZI04Rr9qP4VuA84KDM/nJlvY0G4ehpAZv4ceBfwGEo/KkkaSNO7LoCkqSMipmXm/FxQRb45EMBN9fF9KTOovzMzvx4RqwFzM/PiiHgFcE0nBZekMbCmStK4iohH1XBEZs6PiBUjYvXew8C8zLw/IvYHTgHenZkfj4jlKTVUb65h7KraB2vEawJKUtcMVZLGTURMA94EHBoRq0fEdOBq4IV1la8DO0TEKfX/725c3++plMk+52bm/N5r2ild0qCy+U/SuKgdzG8Dbgc+DKwFvAi4Bbio9qu6EjgOeD1wfmZ+pM6oviXwGWAucHwHxZekJWaoktS6iNgRuBj4WGYeHhGbAEcCtwKHZeaf66r3RsTM+v+3R8S5wKMpUyjMA57ba/KzhkrSoHNKBUmtioiVKSP47szMl9e+Ud+h1D5tCHwFOC4zr+97zrbAfpTRgL8DvloD1fTMfKj/fSRp0FhTJalt0ygd0P8SERsDXwbOycw9I+JdlEvLREQcl5l/rM/5e2ZeTKndelitoTJQSRoKdlSX1JqIiMz8P+B84LXAL4DVgC8C1E7oHwFeTWnue3x96roRcWBvVGBvHiub/Mauf1Rk/zUVJY0/m/8kLbOIeAawUmb+uLFsPvAQ5TIzx2XmnMZj76JMl/BfwI8pk4A+Gtgm3SmNWQ1STwauz8x76rK9MvP0bksmTU02/0laJvXafUcCvwF+XKdR2A04FXgQOBz4a0ScnJl/gVJjFREPAG8FngdcC/xjZmat7TJYjc0WwBeAc4H3RsR/AztHxKaZeVu3RZOmHmuqJC2zehD/Y72W39aZ+cteB/OI+DylJupw4Mt9NVZPoJzcXV0nBrVT+hKoAfbNwKconfsfBeydmZd3WjBpijJUSUvJGpWFt0Htw/NZ4BDKxZAvbKy3yGDVWGdac5JPLV5zmomI+CPweOBk4C2ZeW+XZZOmKpv/pDGoE1LuRBnccVFmPmRTFTQ/e90ep1MO7t+OiP16fawy80213/SHgfkR8dVeU2Dj+QaqMarfu16gOoAyieoZlObUWyPimDpgQNIEMlRJo6h9hi4C1qXMCv6jiPhUZv5gKgerkSbkzMwLal+pI4BTRwhW84GPA38GvjnhhZ4EmjV6EXEqpV/VwcAvgZuBT5SH4phG5/XlM/PBrsosDbq29uM2/01S/V+Q3o54qgaAZRER/wk8DjiWMprtBOAe4NjM/FZdZ0pt10Z/qZWAfYDlgXsy87v18Z2ADwBPAx4OVvWxQ4FP23dq2dQJU79AmVj1+42g9XZKsPogpa/VfcCJwBWZ+amOiisNrL4TlcdQ9ml/z8xvLPFrTaHjwJTR189lDWB3YHXgi5k5t9PCDZF60PonYEfgksz877p8E+A0ygSXH59qwaoR0B8NXEIJVKtQOkn/L+UyNFfWa/+9B9ieEqwu7HsdO6UvpYj4DPBKynUVX56ZV/c93gtWP6FcP/F5wD9k5qUTXVZpUPX22fWqD6sAR1NOoPeh/G62AP60JPt1J/+cRBqT/a0QEWtHxInAScDXKH0tNuiscMPpTZR5lP4d+CtARKxQL6+yD5DAOyJif1i4f9Fk1hulRwmWfwH2pATPJwL/CBwbEetk5s+BDwGXUppMn9b3OgaqpVC3/S+Amyi/6UfX5Q9P/pmZxwGvogTee4HtDVTSwmqg2pkywOZ3wDOAu4H/Az6Rmbcs6X7dUDWJ1C/ILsCngWuA7ShnsvcCp2TmdV2WbwidRulYvQKwK0BmPlD7p9wA7E256O8nIuL5nZVyAjWC+7rAepRmpd9m5i3A0ynb6uzMnA1QLz3zKeCTlM7UWkJ12oSH1TD6X5Sz6ruA42qt37y+YPUt4EXAK/trsrSAM89PTRHxxog4mXL1h42Az1FODs8Ffg2cV9dbou+HoWqSiIhDIuIUyhdhQ+BTmbkT5cK2VwH/U9dzBzJGWa5L90XKj+3dEfHGuvzBGqxuBA6g/Ah/vOhXGn6Ng3VvcMsqwObA3Brm9wdOB47IzE9ExBoR8XqAzDw/M9/Rf9DX6OpggF5fjyfU27qZ+Tfg+5Qa6M2A82uz7LxmCMvMe23yX7S6fXtdJR7r/nHyi4hVIuJIygnzepRWhwMz86j6W/sXSteOi2DJWyAc/TfkalvwJ4GXArOAlwAXZ+Zf6ypvpPSdOw+mThPV0qidrvcD1qB07j0pM2+MiA8DywGfjwgy84RGsLqOco27EUfDTQa94ftRrst3ZkS8FbiFUgu6WUS8itLEfATlkjRQ+qK9PiIuycyreq81GbfPeOmFpPr/k4F/AFalTEnxYUrt8+k1B5xAaWL9p9o865xfo+ibluKTlFqKeXVbn5mZd3ZZPo2PzPxbRHwP+C4wOzPv6oXpiNgLeApwQD1ZXOLfkaFqyNWD+2cotSm31y/INICIeBnwJCb5Qb8NtdP1z4AVgcdQOl2/OSLeCfyQ0tSSlGCVmXli/xD1ybhte9+Z2o/n+Lp4XmbeERFfZEGIOiLLxZKJiC2Bt1AuPfObCS/0JNGoofoipaP5YZS+fVtSmvi3ryMp/5sy4epngMsjYnsD1eL1jfaaSTkpPR3YgfJb3z4ijk4v9TOpRMT6mfmnzLymsSworXbzKH1C/wT8AZZu7jxD1RCLiHWBOzPz2sayoFRdQvmCzAF+D5PzoN+GRqfreyizft8FrE05+/8y8NrM/GFEHEv54R0fEbflFLhobQ1UKwHPodSSfBT4bX34k5RO0ocBy0XErpSRM/9Gqdl79dKe7U20QR25GRHrU/qqHQWcUfv0XUQJVXcCf6vLfkDpz/YBSvP/jV2VeRg0AtUalO/wazLz7Lrs48DLgBUj4kiD1eRQ/64bR5lj8OLe8vq7nxcRT6a07Lw5M/+0tO9jn6ohFRGfpsxD8+zm8izmRcQ2lNFrJ2TmzV2UcYisA2wM/Gdm/qaeyfyKsm1vBD5WOwLfRKkNeDulP8uk1df36avAOZRaz6tr7ei0zPw75WD/Xkpt6KmU79z1wI5Z5rFablADVRTLwYJm8YhYteMyLd+3aF1Kc8Qfanh6AqXp9TTg3Zl5X0Rsm5kPUGtaal8/jSIiPgdcQRk2//Agnsw8DPge8FzggxHxuG5KWPQPVLDf15KLiO9Q+k79CLh1hMdXoPSPnUXph7zUDFVDKCK+Tek79RvKMND+x5cD9qiPnTexpRtKD1FmSl+lt6A3morSrLIZcBBAZt6QmZ+ugWFS1fTWDpxrw4IaqnqQfyVlvqPHA6+OiJV6Qal2hP4gpSZre8qcaPv1ts+g1o7WA9NLgHc3lp1E+XwT2pk+IparYYlek3Kt9YMSoK4HNomIJwIXU37Tr83Mv0fEKygH/g0zc256aZolcTZlZPTWlBOrh08mMvNwSp+bZwGf6v0uulD7yM2IiM3r/YGrUR1kEfFeYFtKf9mvZOb1EbF8RKzYWG0+pS/tpb2Ry0vLUDVkIuIIylwaB1BqoW7tfTlq2gaYAWwD/DTLnErqUw9kvVB0P2UI7S4RsREsNIfSHfXxFfpfIyfRPEv1u/M64L0RsXG9fx3whnqgfwFlvqk3Arv2alQaB6E/Z+Z1mfmX2uQXA759glI7+eqIOLE2n72Ucl3HiQ6C21MO3IcCRMSZwH9Emdn5Nkr/jqOAn1KGfx8A3BsRawK71c/ytwku81Dpr+2pzqc0Vf8V+EBErNkcPVmD1QWUUa4TPmq1L9xfAMyMiG0nuhyTwKaU3/WlmXl/PYE5CfhhRBwTEavUfdUJlP3bstUGZqa3IblRfthfAT7fWLY15TIVZwPfAB5Xl28GrFz/H12XfZBulBqpEykHqhl12YspZyvHAhs11t2WclA7sOtyT8B22Qt4gNK0dANwIeUMfoX6+HTgV5RJJ/cElh/m7xcljBxFCc1/BZ7TxeehhLsvUEZTXkm5ft+2je27BnBZ/X6+AliZcvmfkymTrz6x6205yDdgucb/t6rbdtXGshdQTp7OBtasy6Y1Hl+rgzJPr/+uDOwMnFL//qcC23S9TYfhVo+XKwBnUppzdwcOB/5OmTz3VMoo7w+2+b5epmbIRMSXKEOr30Bp838X5VIhd1H6BvwUeEuWPhbqU0f5/YJyAPs2JaTen5kZZR6qz1OaV35CmVX3IMrO7Bk5oE1ZbYqI11IC553AP2ediiMiVszMubWG6n8pIyT/DTgrh+xCvc1RsBHx78CbKTOPn0vppHrfRIyU7Y06ylI7shGlaW994DOZ+ba6zgpZ+lKtSzk4rEYZRHEDZaTq/pnppKqL0DfK72uUwTtrUE4ejgG+l5nXRcQLgK9TThpemZl3dj3AIiJWofzWbqUMNnoM5cTnPODwbExVokWrtXs/oIzevpvSd/aY+vv7BuU39dK2/taGqiETEVtTqil3BK4GvpOZH69fkNMpTe57dVnGQVWbtP6bEpIOAW7M0vdneco0AfMj4sWUjujbAbMptVSvyNI5e1JOSdF34HkvpQ/VupRt9dHeQbsRrKZTgvxTgF0y86KOir7EmqP8IuIcSlP56ynheW9KsHlLZt47XgfV2qyzUpYJPHvL9gV2oQSmZ1GuKfmJ+lgvWC1XH9uCMgLzhsz8c9vlW1L1ROWfKfNmDeTcThHxBcpVEd5JGXzybOA/gJ9TmnxmA8+n1P7dBOyemXM6KWxVR6vtDeyWmb+vzZIvpPT1+jEGqxFFxMspM6Q/APw4M6+ugw0eTck819b1HkupAfwt8NZsKwx1XUXnbdQqzP2AQykhYKfG8qcAGzbur0YJVSdSmmmGsklmnLflppTO/Xs3lu1KmRrg28Az67JVKR3X12XBicf0rss/Ttuk18ywAgs3e+5POas7Bdi2sbzXJLU8pZZvuYkqawuftdkM9HLKlQZeWu+vCHyEMvrnyyxoFl6JcqmXNVssx4sotQ3r1fvfpTRPrEYJTCdSalIPazxnha633yI+y8qU5sqzgEfVZdO6LNMIZVyX0mfyDcCKddkalJOrT/X+1nX57pRaoY06KGfz+7lc3Sdd1P84pavC3Pr4thNZxkG/1W1yMyUkz67b6c2972Zjva0o/apuA7ZqtQxdbwRvi/2CfKfuXG+gpO6bKZef6V9vO2AmpX/F1l2Xe9BudQe6Q/0h/YnSL+V5lOr/eZSz1d9T5vR6/AjPH6iDRIvbpbeTXoUyZcLXgOc1Hj+gEayeXJetS7nA9PT+1xmWG6Um7kuUSxCt0Fg+oxGsvl5/V1+gTGL6uBbf/2nA5ZQm1vPqd3LHxuNb1ve9DTi0LluRMlLxtV1vv77P8gHKiUqv392E9z8aQxmfWgPUs+v9J9Zt/20WBMEd6zaO/gPwBJVxWv13Dcq0GFCmKrmxuU+ihK3lKEF8PvAtYJ2ut/Eg3ID319/SLsBjgSdT5nN7CHhbXScoExb/qv6un9p6ObreEN4W+QV5D6Ua+h8pZ8sbAcdRQtbxjfXeSGmK+d14fEFa+iyd1ZrVHeVtwPfr/eMpk3xeTwlSe1FraSj90l7T9faayL8JJVBdTWlO2JPGWXt9/FV1u/wAeAeleew6hiBoUqr7P98MRPX3NJ8SoI9pLO/VwM2oQeEGyknKDcD241C27SkdZh8A9qjLmjUVvWB1Vz1w/ielQ/0Tut6ufZ/jbXVbPpFSW3U5A1Kr1viOrw78kdIHcOta3lOBVerj+1AC1uM7LueKlNqVH1JqAF9KOZF+H301pfW78VVKTcwHut7WXd/qPvwHwNf6ty1lwuIHWRBWt6cEsE3GpSxdbwxvi/ySfA84o7ezr8vWBj5Embtmv/qFeTGlenPjrsvcKOeINRdMcLiiNIO+oG7HJzWWv6D+sDZpLNuJErJ27Xr7TeD2CUoV+M+ADVhwtvwYyhlzr2lwf0rw+h1lhFQvgAx0sKIM5DiP0n+pufxASg3cLdQRf73vS/13ecos5nvQcjNQYxvvQpmi4hpKeNus996NdTetO/+rKCMxn9L1Nh3h+7Mj5WLtt1NqCbbssDyL2u+sROkacS0lUH2n97eg1Gh8hRIIH9NBmXvfh+a+6gmNxz9GGaH2gd4+njIJ78/q+kdROrJvMNH710G61e/iOZSrDvQvX59Sm3rCROy77Kg+YGpnxOmU0Wc3ZOYBtWPw/Cwdqden/KDOyMy39p6TAzJrdd/IqrdQpnZ4EDgn60iyCSrHipS+MZsDd2XmbiN1NI9yCZatgM9SmgJ36V9nsqod98+k9Nv4UF12AKWj/qMow8xflKVz+kaUZocb6/dweg72PFTAQtcufCNwQWb+vi4/kNKf5seUzvi/qsvH5XP1f/ciYgalNmJrSp++zSgjTP/QX4aIWJkS+P7a/7qDICJ+Tfmd/ZEyYvSXE71P6htscTDlUj2/pkzmeENErEcJphtRrkv5RUpwfgPlkjTPzcyrJ6q8fWVv7qvuzswX9e1HP0o5EZhGCa7rUy4EvENEHAm8htJKMSXnKusNPomIz1Nq23fNzN/0rfNz4JbMfMV4l8fJPwdIHUKbWaZD+C6wd0Q8u+5gp9Uvz58oqXuLxsSLAxGoYMH1BSPiNEpz0faUs9lzIuKjvVmBJ8CqlJ3P5tSZ0uvB9eEJ9eqopa9TqtKXA17Qv85kVr9nDwD7RcRro8zU/w3KpTvOosyf9Mm67k2ZeX0NVNOGIVDBw3/zJ1OaAT8YEZvW5d+kjP56LvDOiNiuLn9omSb+G0HfAfJ5dfj+0zPzzizXIHsLpTn6kojYpJZhRkQcFhE7Zpm1fmACVT3J613mZzVKE+lHKBOQnhD1gs5tb8fFaQSqb1Mu3/VaSs3ECRHx9My8lTLf0+WUgT9/pdTS7kQ5keokUFXNfdXK8PD3doX6/3dRrkn6Ocpo5JnAM+tzn0H57ky5S9dExLqNUX1Qfs/3Uf7mmzbWW4eyn7shIqaN+/ey62o7bw9XUx5Tb0+s97ekVGdeycKj/tai9KF6RIf1QblRRireTLl0SW8C0vdQ+rLszQQ1G1E6VfcmeHt/Y3lzYr/DKTvhXqftgRnlR191fv/9JXytXtNWNF8HWI8yt9m1lCkU/qkun0G5TtZnu94OLW3LXSl96b4DbNpYfhClH8vXaXQWH4+/IaXD/02U5r776v3t6mPPqL/rOym1J1+jzJPWWXPaGD7b62l0TKc0E19CmVtp+/7PP05laPZD26W+/86UJr/X1fu/oNQCQulf9bT6d386HXTyHmn/t5h91Yj902rZT6KEwyd3/V3oYBt+kTIh7hxKE//r6vIdKTWmv6c0mb6t/ubvouVRfossW9cbx9vDw0D/QBntsW5j+V6UKuu7gSOAIymdVu9mAEb5UTpWrj/C8s9SOlv22q+3rAeuU+jrCN1yeabXkPAY4NF12QaUWr9rgP9orjvC8wdmFFvfwWLVNl6LUmP3ScrFoD9DYyQZJayv1tiOm9ad1pFdb4sl+JyxuL8hZSqDe3lksPpnShP1l6hD7tsoS9/9T1EC1S6UGb2fQAl5P+/9lim1JmcCfwZ+yQAPl6eEwPmU/khbNZYfwDgHKxb0Q2oG1g9Q+hd9hYVHpu7HgmDVemheirL3Tm6WZF+1XN9rrEOpSf41A9bPboK24dcoJ+3/CrwV+ET9Lr67Pr4W8F91O97EBPdH7HwDTfUb8GHKsNmdWFCrs2Lj8S3rDnk2ZdTVjwfhh0TpzHszdahq32NfBv6n/n8zytnEKY3P937qnFAtlufRlBmxr6E0R5zV24lSqta/R+ls3Zz7Z2BCVN9naR4sPk257tf5lOu8rda/zlheixKoZlGC0ml1+9xBGUHUPAitRpkE8aeUppKBqblbzGd8xN+x7mzfS52SoLF8UcHqQFqoFWLhWtDetl+57tgPZ8H8V6vUcsykMYS//q62psV5scZxu7+yHsxOYuRgdTGNWvaW3vNRlO4PWzeWrUIZRDGfcsIwre85+41XeZbme7qU+6r+kL4hjRPwqXKjdCe5htIPrnfSvmP92y90UkQZ2LUONbROWBm73khT+UZpS/8+8N7Gsk0ofXy+SZmXphdE1qo751UGoNy9M8U9gNXr/x/TePxIytn23pSmjlOptS2UfjrnAYf17/yWoTwrUc7aLqJU6x9KqdGbz4LJHdejhIlfA0d1vQ3H+Lk+TemY+gVKM/AdlBrL3vXJxhysKNc0vJhGcKg7ofmUTrpQhiV/iBK+zmnstAYyfNayzah/639rLPseZTTajZTmlJ/TqFFlQbD6FrBFi2V5FCUkvaxv+XqU/kZvrve3oDRHNOdJ2o1xrMVdxs/VH1JWYEFgXFSw2q9+j86nzv/UUlmeQKltXbVv+dqUcPI3yojo/tqdV4xHeZai/Mu8r+qq7INwozTt/o0FEzVvTjlp/0bjt7Rdp2XseiNNxRuNYETpx/JDSjv/v7PgYo+XUdqF304ZUDAQB7Z6ELuAMiqst+xkyjw6G9f7y7HgArA/aHzZH0epxZpFi1NAAK+mhI7mtAlvqe//hsYBYANKjcHXB3HHNMKB4AvAyxv3T6KErA+w5MHqHOAbjfv7Upq8Duv9Xeu/T60HoIHrY7aIz7UFpWP9pZTOyU+gjJzdgXI2//z6O/oVC0+hsSsLmq9a+Yx12/2NctKwa2P5o+tB9FjKmXZvnqTeicbOlJrBHbrenqN8vo9Rh/tTatT6g9WXWXg6gH3a/J03Xrf3vp+nTuhZ769FGRl9fd2m/b+nvcajPEtY5kmxr5rgbdc8Xj6//sYeX//e/XOO7U1pFWltst4lLm/XG2wq3ijzuhxb/78n5Uz675Rq7SPr8uUp/Spmdl3evrKvUQ9iN1Fn36aM8ptPqVnZtC57av1cN1FqpY6itHPfScvNl5QaveuBNer9/Wp5eoFhdRYEvrUYoU9G1zcW7kO1DeXaZF+nr18NpSZkscGq8fl603OsUL9zx9flr6rb5131/gxKH6vnL6pMg3hrHIC2odRAXEI5sJ/NghreaZSANYtHBqtdaKlvYqMsz6D0j/wRC594fJAys/PfgW83lj+WUmP4P8DaXW/TxXy+betv7Fpg87qsGazexoLLvmwznn/v+v+N6n7or5SRlL3la9X9zg2MEKy6vk2GfVUH2+zh42W9f3H9jd1FOaHv9Ul7HKXG6ltMcJPfQuXteoNNtRtl1MbV1E7C9aC3EeVCqb2zwKCEl7MpzTExSD+q+mO/gNLE0gtW/1p3Dp+jXpOwfoavU864fkUZsdHajNCNHc7hwJ/q//dm4cAwve7IPsbCfVcGcuJKSrPvHBZct+qNNCaErOvMrAeNjwOPXcQ2WbH+jXat999NaQr7D8rB/cjGc/6BEgJePl6fa5y2VbMv2BPr5/0z8MMR1u0Fq0upE22OY1mey4L+jy9uLD+xfjePpEzg+Lz6+5jDgI3gGun3Qandu7Qe0Lbofc/qvxvX72xvH7B8y+UZqd/c5pQa2P9j4cv89ILVtZQJMjv5rbNw/7pJt6+aoG3YO16+hgUB/qWUlpB7qCdJlL7HX6m//04HcXW+0abare5Qb2UxfTnqAWImJbS01uejhbI3DxwbUmZ6voYFweqNjZ3qxo11V64H+baaWXqXlemF0PXrNr2UMoHnoY11n0o50xnIUWwsPIP2IZSmqtdRmgKupHTAfQaPrI36Vt32azaWNS+OvAml39AVlDP2zSihYz4Ln/U9gXLmd/ZIB65BvbFwzd7ulCHpG1NqrO4DDhnhOU+rv6mftPVdrK/bP23CCyknSddRwmozWH2WcrC/t/79/pcBGHiymG27LaUPWm+ql3+g1KD/gYX7521Pqe08oLfuOJXnNcA7KScHG1Kuz3gmjwxWa1JC9JV0cy2/3rUQJ82+qqPv4iOOl3Wb7l/3bXPq9/FySg1g5yNmO99oU+lGGdXzZxZcJHXaCAfLt9cd8fUM0LX8+g4cMymXffhtPUjfyCOD1WcZh2srUfqn/JwFZ8UXAP9St9vVwNWN9Xao6/60zYNoC59hZWrn8MayV1Lm/XlbY9kmLAiuIwWr5vQbvTPhR9cd83frc++jHNx3ppzhnUu5htxRlKHJl1JqEac3X2eQb30H2f+sO9X31ftPqt+JXzHCxYcpIWGZa6ooHY6f1SwTJdjNBp5Wl/WC1Y+B3RrrbkmZvHFTajPQoNxYuHbla/U79Pf6PZlZlz+nHsRuqd+rZ9Z9wk8Yx2Bev9M3sWDU3J8pJx+7Ufpu3sPCTYGPZQL7UNXv3gsa91cb9n1Vx9/FkY6Xvb6eQRkY8mZKyN6f2kLS9a3zAkyFW+OL8Mq6Q+iNXOgdCFdjwZngQbQ1FXwAACAASURBVJSh4K03UbT0WT5Xv+h7Uvqy7EvpyzKbhYPVPEp17AYtvvc0Sk3EhZTanD0oZ/q/oDQtvp3S3+j3LOhD8zMGaBRb3Rl8kzKVQe97sXPd6c4H3tr3ndmABcFqJxYTeih9XH5UP/P2lD4GL6w76+tZUGN1ZN3Bf5dysdaH587pevss4bb8OiW0vIyFL5q8Tf18VzJCsGrpb/gFyoW6X9hYtnZd1jywj1hjNeg3yqCIGyi1VI+l1AbNB75XH9+u/hbn1888m3E8CaQ0m91MObl4XF32fUpNxW61POdS+mw+a7zKsZjyrUgZfXgzNVgN+76qw+/eaMfLNWi5NrTV8nddgKlyowSCWSw8AuvRlOG/Z9Wd07/VnXOr/RFa/AyPoQSoTzSWBaUa/qK6Q+kFq7dTzhxbmbGYUjOwOyWQPLOx/HH1APALStPA+pQzl7dT+iwM3Ci2WsZeR+rexI8H1wPT2Y31msHqV5Qwu8gRYpTmr+upQ/f7/j4/pxzce3+f/n5aQ7UTp/SVuYE6A3zj+/nUuh3WoQSryxihKbCF99+qHsSvYkG/tVUpB/nnUPrH9P5+z67b/nz6plsYxBtllNwvKZdvgTLn1wOUDvW3A6c11n0JJXiNay0B5STkKyxoVtuQEua+yYIg8tS6H7iJDqanYMFVMK6iNIOeMuz7qg6/g6MdLx8CDm88Njh9jrsuwGS/saBz3WspgaR3WYojKFMpzKP0j/nX/gPdoN0oZ2NXAieP8NjOlH4NV7Dg7H31trYhpZPvHMqZ6MZ1eW8Huw6leeLcRTx/IAMDZVTkHSzc2f9+4Kv9ZacMIb6YxdRgUkLF9cAHR9h+u9UD41XUM3mGoKlvMZ91H8rlKFajNKc+j3LW/ycWnKCsSwlVP6FOmtpyGTanBLer6s5+OcqJxSNGv1FqUu4CzqAG6kG81e/KLsA76v1/oTQh71sPaifU7fvtiTqQ1QPs/wCn1/ub1m15Kguma3kdpRb2CcBGHW6/zSnheVbdX21clw/1vmoiv3/137EcLwdyW3VegKlyo8ypMosy7cCllDO+E2mcadf1BiZxj/AZplOGrF/Rf+CoB7Yr6g73Wkp7d2ufhdK+fnZ9/X9uLO/trF5eH9uKIRmGXD/TjyiT/DWD1VxGDlaLPYOl1OadSePSJ43H1qf047iRErwG9sA+xm23Vd1OZ9ad7b2Umow9KM2bD9YD2JOAx49jOXrB6mrgTZQaqXfW7+MelI7dz6TM+vwK6nQEg3yjzFC+Vv33Esp0EL3wshkLguvpE1imT9ey7M6CuYl6VxfYmtJ0dkDX266WZwvKXGWTZl/VwTYc2uNl5wWYCre6Y+/1mTmjfmHWZkG1dTT/HeQb5Uzw/yhV283RP5vWz7YDI1wPsKX33qTuPG8Fdu977JC6vLU+XBO0PTejBKvf8MhgddJSvN6Ta8A4hYUnYnwGpTPvs+tB8aNdf/YWtt3Oddt9BnhlY/nelNFpm05QOTavB9HepVLOpcyfdA+lM/df621cfhfj+LnWq+V/R2PZPpQa09dN1Pat7/tkyqSP8ymDZHr9ANekNEv+mgHpqFzLNen2VRO47Yb6eNkrnMZRRDyKshO6G/h+Zt5Vl0/LzPmdFm4pRMSulB3b5ZQv/c2UfhhPB56TmbPH8b03o+xEt6Z0Xv055Qf3CcoopV2GbZtGxOaU0VNrUy638uOIeD2lM/SJmXnIEr7eiyid0H9DafqaTblocO/SQT8DLsvM17f3KboREdOBednby0asTZnbbRvKwezOCSrHFpRBHFtQLoZ7WkQ8jtJ0tSJwb2beMRFlaUtE9GYov4IyoOE+Si3cysAbMvPeCS7P8yn7m19QThqgNGs/jzKa9tcTWZ7RTMZ91UQY9uOloWqCRMRymTmvcT9yiDd+RDyFUkOwJWXU2Rxg38y8cgLee1PKyK9nUGoAfkgZEbJXZs4dlh9fU1+welNm/iQiXgNckpm/XYrXexJwNKXzblBG0exDmePlIkpT6uEAw/w9bIqIAykd2Pekg4Ns/RueQOnL9bbMPH8i3388RMQ/UDpf30sJVStRwkAnASYing58hDJ4Yx6ldvA9mXl1F+UZTd++6m7KZckeC+yRmQ8O476qJyKek5k/HafXHtrjpaFKSy0iHk3pKLwGcOtE1QrU996EEkI2AN6fmafW5Stk5gMTVY421YPy8ZTq7/2WdYcVETMotSQzMvO2iFiFMn/YSymjkq5d1jIPioh4BqX/xQPAGzPzNx2Vo/c3fCLw6sy8oItytCkitqH0D7sfOCMz/9BxeVainBzMAx7MzLldlmc0dV91AqUm/8DMPKcun56ZD3VauKUUEbtQmrwPy8xjuy7PIDFUaWjVJpcTKEOV/z0zz+u4SMssIraiXH7mbZn5xxZf9wWU6wVuCLw0M69o67UHQURMo9Sa3tl1M1v9Gx4DvL3Nv6GGV0RsTZmo9NDMnDdMNS8jiYjVgX8HvpmZs7ouzyAxVGmo1ZqBz1NqdyZLzUDrtW21luo1lHmwJk0N1aAa5hpTja/+pq1hNcxNl+PJUKWhZ82AJGkQGKo0KVgzIEnqmqFKkiSpBdO6LoAkSdJkYKiSJElqgaFqiETEwV2XYSwsZ7uGoZzDUEawnG2znO0ZhjKC5RyNoWq4DMWXGcvZtmEo5zCUESxn2yxne4ahjGA5F8tQJUmS1AJH/42TiBiKDbv99tu3/pp33HEHa621Vquv+ctf/rLV1wPIhIg2X28o/uSSpKXzl8xc7MHNUDVOSqhq8Yg9ToZlQtwZKz6q6yKMau4D93VdBEnS+Lk8M3dY3Ao2/0mSJLXAUCVJktQCQ5UkSVILDFWSJEktMFRJkiS1wFAlSZLUAkOVJElSCwxVkiRJLTBUSZIktcBQJUmS1AJDlSRJUgsMVZIkSS0wVEmSJLVg0oeqiNg3Ig7qW3ZhRJzWUZEkSdIkNOlDFbAvcFDXhZAkSZPbVAhVkiRJ425Sh6qIOBnYB3huRGS9vb/x+IER8YeIuCcizoqIDfqePyMijomImyNibkRcGREvnthPIUmShsH0rgswzo4GNgJWBw6py24BdgZ2AtYDDgVWAj4NzASaoek0YEfgfcB1lKbEMyNih8y8YgLKL0mShsSkDlWZeV1EzAGmZeYlveURAbAqsHtm3lWXrQMcFxErZeZ9EbELsDuwc2b+pD713IjYEng38IqJ/CySJGmwTermv1Fc2gtU1TX13/Xrv88HZgMXR8T03g24ANhhpBeMiIMj4rKIuGzcSi1JkgbSpK6pGsXdffcfqP/OqP+uCawDPDjCc+eN9IKZOZPShEhEZAtllCRJQ2Iqh6rRzAH+BOzZdUEkSdLgmwqh6gEW1D4tiQsondj/lpmz2i2SJEmabKZCqJoF7BERe1JG/t06xuedB5wDnBcRHwOupnRu3xaYkZmHj0dhJUnScJoKoep4YDvgJGAN4ANjeVJmZkTsDRwBvI0yNcMc4Args+NTVEmSNKwi0/7U46F0VI+uizGqzPldF2FMZqz4qK6LMKq5D9zXdREkSePn8swccfR/z1SeUkGSJKk1hipJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiRJklpgqJIkSWqBoUqSJKkF07suwOSWXRdgVLvu+i9dF2FMPvTFr3VdhFF98K1v7roIY3LPPX/pughjMn/+vK6LIElLxJoqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiRJklpgqJIkSWqBoUqSJKkFhipJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBZMqlAVEctHxHJdl0OSJE09nYSqiDg5Ii6LiD0jYlZE3B8RP42IJzbWmRYR74qIP0TE3Ij4fUS8uu91LoyI0yLi4Ii4DrgfWC8iNoiIb0fE7RFxX0RcFxFH9z1334i4qr72zRHxoYiY3nj8oIjIiNgmIs6LiHtrWfce7+0jSZKGz/TRVxk3jwc+CbwHuA/4AHBORGyRmfcDnwVeDRwF/BJ4AXBSRNyZmT9ovM6zgc2AdwJ/B/4KnAGsBBwM3A1sCmzde0JEvBA4FfgacBjwFOBo4LHAG/rK+U1gJvBx4N+AUyJi08y8pZ3NIEmSJoMuQ9WawB6Z+TOAiLgcuA44KCLOB94IvCYzv1rXPz8i1gXeBzRD1erAdpk5u7cgInYEDsjM79dFF/a991HAhZnZq/k6OyIAPhIRH+wLTMdl5kmNMt4GvAQ4sf8DRcTBlCAnSZKmmC77VN3eC1QAmXkjcDmwI7ALMB84PSKm927ABcC2ff2mLm8GquoKSkA6KCI2aj5Qn/s04Dt9zzmVsj2e2bf83EYZ7wRuBzYY6QNl5szM3CEzd1jcB5ckSZNPp6FqEcvWpdRiLUdpynuwcTuZUru2buM5t43wOvsBlwHHATdGxBURsUt9bE1g+RGe17v/mL7ld/fdfwCYMeInkiRJU1aXzX9rL2LZ1cAc4CFKf6n5I6zXDGTZ/2Bm/onSjDiNUvP1fuDMWmv1F0pA63//x9V/54z9I0iSJBVd1lStHRHP6t2pgedpwP8CP6LUVK2WmZeNcHtgLG+QmfMz8xJKJ/hHAY/PzHmUZsZX9K2+LyXA/XyZP5kkSZpyuqyp+gvwnxHRG/13FKUG6uTMvD8iTqSMtDuG0pQ3A3gSsGVmvm5RLxoRqwHnUEb2/R5YETgUmA38tq72PspIw68ApwDbUEb/fdFRfZIkaWl0GapuBD4MfJQyvcJllBF799fH30QJRa+nBK57gGuAL4/yuvcDVwFvBTakTLNwCfDCzLwPIDPPjYj9gSOBV1LC3LGUsCVJkrTEugxVZOb3gO8t4rEEPlVvi3r+ziMsm0sJYqO996mUEX+LevxkSsf4/uUbj/bakiRp6plUl6mRJEnqiqFKkiSpBZ00/2XmQV28ryRJ0nixpkqSJKkFhipJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBZEZnZdhkkpIoZiw66wwoyuizAmW221U9dFGNW/HPH2roswJp8+4r1dF2FMrr/+110XQZKaLs/MHRa3gjVVkiRJLTBUSZIktcBQJUmS1AJDlSRJUgsMVZIkSS0wVEmSJLXAUCVJktQCQ5UkSVILDFWSJEktMFRJkiS1wFAlSZLUAkOVJElSCwxVkiRJLTBUSZIktcBQJUmS1AJDVUNEnBwRl3VdDkmSNHymd12AAXM0sFLXhZAkScPHUNWQmdd1XQZJkjScbP5raDb/RcRBEZERsU1EnBcR90bErIjYu+tySpKkwWOoGt03gTOBvYBrgVMiYoNuiyRJkgaNzX+jOy4zTwKIiMuB24CXACf2rxgRBwMHT2zxJEnSIDBUje7c3n8y886IuB0YsaYqM2cCMwEiIiemeJIkaRDY/De6u/vuPwDM6KIgkiRpcBmqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBY4+q8hMw9q/P9k4OQR1tl4wgokSZKGhjVVkiRJLTBUSZIktcBQJUmS1AJDlSRJUgsMVZIkSS0wVEmSJLXAUCVJktQCQ5UkSVILDFWSJEktMFRJkiS1wFAlSZLUAkOVJElSC7yg8riKrgswqhiCMgLcccfNXRdhVKussUrXRZhUll9+xa6LMCYPPfRg10UYk8z5XRdBmvSsqZIkSWqBoUqSJKkFhipJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiRJklpgqJIkSWqBoUqSJKkFhipJkqQWTPpQFRH7RsRBfcsujIjTOiqSJEmahCZ9qAL2BQ7quhCSJGlymwqhSpIkadxN6lAVEScD+wDPjYist/c3Hj8wIv4QEfdExFkRsUHf82dExDERcXNEzI2IKyPixRP7KSRJ0jCY3nUBxtnRwEbA6sAhddktwM7ATsB6wKHASsCngZlAMzSdBuwIvA+4jtKUeGZE7JCZV0xA+SVJ0pCY1KEqM6+LiDnAtMy8pLc8IgBWBXbPzLvqsnWA4yJipcy8LyJ2AXYHds7Mn9SnnhsRWwLvBl4xkZ9FkiQNtknd/DeKS3uBqrqm/rt+/ff5wGzg4oiY3rsBFwA7jPSCEXFwRFwWEZeNW6klSdJAmtQ1VaO4u+/+A/XfGfXfNYF1gAdHeO68kV4wM2dSmhCJiGyhjJIkaUhM5VA1mjnAn4A9uy6IJEkafFMhVD3AgtqnJXEBpRP73zJzVrtFkiRJk81UCFWzgD0iYk/KyL9bx/i884BzgPMi4mPA1ZTO7dsCMzLz8PEorCRJGk5TIVQdD2wHnASsAXxgLE/KzIyIvYEjgLdRpmaYA1wBfHZ8iipJkobVpA9VmfkXYK8xrHchEH3L5lLmqHrfuBROkiRNGlN5SgVJkqTWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiRJklpgqJIkSWqBoUqSJKkFhipJkqQWRGZ2XYZJKSISoutiTBorrjCj6yKMarunvbDrIozJ/3vnwV0XYUwOPWC/roswJg8+eH/XRRiTefMe6roIYzQM+02Pm1PU5Zm5w+JWsKZKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiRJklpgqJIkSWqBoUqSJKkFhipJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWjBQoSoiDo6IPZdg/ZMj4rLxLFPf+60SERkRB03Ue0qSpOEwvesC9DkY+A1wxhjXPxpYafyKI0mSNDaDFqrGJCJWysz7MvO6rssiSZIEHTT/RcSTIuLsiJgTEfdGxG8j4k0RcSGwPfDq2sT2cDNbRNwQEcdGxHsi4hbgnrp8oea/iDioPu/pEXFRRNwXEb+PiL1GKMceEXFZRNwfEbMj4piIWL5vnX3q8++LiP8Bth6/LSNJkoZZFzVVZwKzgFcBc4GtgFWBQ4DvAn+kNOsBNGuiDgSuruuNVu5TgeOBDwOvA74TEdtn5pUAEbEv8C3gC8ARwGbARygh8x11nafV1zkdeCvwJODbS/mZJUnSJDehoSoi1gQ2BfbMzKvq4gsaj98L3JGZlyziJV6SmfeP4a2+lJmfqK95DnANcDiwf0QE8HHga5l5SOO95wKfj4iPZOadwLuA3wP7ZmYCZ0XEisAHl+AjS5KkKWKim//mADcDJ0bEfhGx9hI894IxBiootUsAZOZ84L+AHeuiLYGNgG9HxPTeDfgRMAN4cl1vR+DMGqh6vre4N62jFy+byBGJkiRpMExoqKoB54XAbOAkYHbt+7TdGJ5+2xK81e0j3F+3/n/N+u8PgQcbt+vr8g3rv+ss4nUWKTNnZuYOmbnDEpRVkiRNAhPepyozZwH71E7h/wB8DPjviNhgtKcuwdusDdzZd//P9f9z6r8HA78a4bm9cDW7Pq//dSVJkh6hs8k/M/PBzPwR8ElKLdLqwAOUJrhl9fBov4iYBuwB/G9d9DvgT8DGmXnZCLdeGLsUeFntg9WzdwtlkyRJk9BEd1R/CvAJyqi6PwJrAO8ErszMORExC9g1Inal1DRd3wg5S+J1EfEAZSLR1wObAwdAaYKMiEOB/4yIVYGzKGFuU2BP4OWZ+XdKDdovKH2vvkzpa/UvS/nRJUnSJDfRNVWzKX2j3k0JM8cDvwVeVh//YL3/bUpN0UuX8n32p9RWnQE8FdgvMx9u6svMUym1V9sC36F0QD8E+CUlYJGZl9XX2a6+zp7AfktZHkmSNMlNaE1VZt4O/L/FPP5H4PkjLN94EesftIiXuiYznz1KWc6iBLvFrfMdSuhqipHWlSRJU9tAXVBZkiRpWBmqJEmSWjCpQlVmnpyZkZl/67oskiRpaplUoUqSJKkrhipJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBZEZnZdhkkpYlpOn75818UY1UMPPdh1EcZkGLblk5707K6LMCZz597XdRHG5Nn/tHvXRRiTc878RtdFGJNbbvld10UYk2nTBv9cf/78eV0XQd24PDN3WNwKg//tlSRJGgKGKkmSpBYYqiRJklpgqJIkSWqBoUqSJKkFhipJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiRJklowaUNVRFwYEaeNss7GEZER8ZIxvN6+EXFQawWUJEmTyqQNVWP0Z+CZwE/HsO6+wEHjWhpJkjS0pnddgC5l5lzgksWtExErZeZ9E1QkSZI0pIa6pioinhQRZ0fEnIi4NyJ+GxFv6lvnwIj4Q0TcExFnRcQGjcce0fwXETdExLER8Z6IuAW4JyJOBvYBnlvXz4h4/wR9TEmSNASGvabqTGAW8CpgLrAVsGrj8Z2A9YBDgZWATwMzgReP8roHAlcDh1C20ZXARsDqdRnALa18AkmSNCkMbaiKiDWBTYE9M/OquviCvtVWBXbPzLvqc9YBjhtjk95LMvP+xvvNAaZl5mKbCyVJ0tQ0zM1/c4CbgRMjYr+IWHuEdS7tBarqmvrv+qO89gXNQDVWEXFwRFwWEZdBLunTJUnSEBvaUJWZ84EXArOBk4DZEXFRRGzXWO3uvqc9UP+dMcrL37aUZZqZmTtk5g4QS/MSkiRpSA1tqALIzFmZuQ+lr9PzKWHpvyNiWT+X1UySJGmJDHWo6snMBzPzR8AngXUpIattDzB6DZckSZqihrmj+lOATwCnAn8E1gDeCVyZmXMiWm9+mwXsERF7Ukb+3ZqZt7b9JpIkaTgNc03VbErfp3cDZwHHA78FXjZO73c8cC6l/9alwMHj9D6SJGkIDW1NVWbeDvy/xTy+8wjLLqTRgzwzb6CvR3lmbryI1/sLsNfSlFWSJE1+w1xTJUmSNDAMVZIkSS0wVEmSJLXAUCVJktQCQ5UkSVILDFWSJEktMFRJkiS1wFAlSZLUAkOVJElSCwxVkiRJLTBUSZIktcBQJUmS1AJDlSRJUgumd12AySoCIqLrYoxBdl2AMXnooQe6LsKorrnmZ10XYUyWX37FroswJg+d+2DXRRiTo08+oesijMm/vni3roswJvPmPdR1ESaRYTgGDZPRj5fWVEmSJLXAUCVJktQCQ5UkSVILDFWSJEktMFRJkiS1wFAlSZLUAkOVJElSCwxVkiRJLTBUSZIktcBQJUmS1AJDlSRJUgsMVZIkSS0wVEmSJLXAUCVJktQCQ5UkSVILDFV9IuLkiLis63JIkqThMr3rAgygo4GVui6EJEkaLoaqPpl5XddlkCRJw8fmvz7N5r+IWD0ivhQRt0bE/RFxU0R8sesySpKkwWNN1eJ9EngW8HZgNrAh8I+dlkiSJA0kQ9Xi7Qh8PjNPbSz7+qJWjoiDgYPHvVSSJGngGKoW7wrgsIiYB5yfmb9f3MqZOROYCTBt2rScgPJJkqQBYZ+qxXszcAbwXuB3EXFtROzfcZkkSdIAMlQtRmbenZlvycx1gKcCvwC+ERFP7LhokiRpwBiqxigzfw0cRtlmW3dcHEmSNGDsU7UYEfFT4HTgN0ACrwfuBf63y3JJkqTBY6havJ8DBwEbA/OAXwG7ZeYtHZZJkiQNIENVn8w8qPH/wyhNfpIkSYtlnypJkqQWGKokSZJaYKiSJElqgaFKkiSpBYYqSZKkFhiqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiRJklpgqJIkSWqBoUqSJKkFhipJkqQWTO+6AJNVZvLggw90XQxNoGH5e8+b91DXRRiTW275XddFGJMnb7JR10UYk9VWW7vrIozJnDm3dl2ESSS7LsCUY02VJElSCwxVkiRJLTBUSZIktcBQJUmS1AJDlSRJUgsMVZIkSS0wVEmSJLXAUCVJktQCQ5UkSVILDFWSJEktMFRJkiS1wFAlSZLUAkOVJElSCwxVkiRJLTBUSZIktWDcQ1VEHBQRGRGr1Psb1/svaeG1d66v9eRR1js5Ii5b1veTJElalOkdvOefgWcCsybwPY8GVprA95MkSVPMhIeqzJwLXDLB73ndRL6fJEmaelpr/ouIf4yIH0fE3yLirxFxYURsN8J6j2j+i4gbIuITEfGuiPhzff6xUbw4Iq6OiP+LiDMiYo0R3n69iPhBRNwbETdFxBv63nOh5r9Gk+Q2EXFefd6siNi773kREUdHxO0RcU9EnBQR+9fnbrzMG02SJE0arYSqiNgZuAB4EHg1sB9wEbD+ErzM/sCOwGuAY4B/Bz5Jabp7D/AG4LnAR0Z47peBXwN7A2cBJ4yxz9Y3gTOBvYBrgVMiYoPG428DjgBOBF4O3FfLJkmStJC2mv8+AlwJ7JqZWZedDaVWaIyvcT/wisycB5wdEXsA/wZskZnX19d6KiW0vaHvuWdl5hH1/+dExKbAkcAPRnnP4zLzpPralwO3AS8BToyI5YD/AE7MzPfW9c+NiE2ADUd6sYg4GDh4jJ9XkiRNIstcUxURKwM7AV9tBKqlcWENVD1/AG7oBarGsrUiYoW+557ed/97wPY1GC3Oub3/ZOadwO1Ar6ZqQ2AdSk1WU//9h2XmzMzcITN3GOV9JUnSJNNG898aQFBG9S2Lu/vuP7CIZQH0h6rbR7g/HVhzKd5zRv3/OvXfO/rW6b8vSZLUSqi6C5gPrNvCay2ttUe4/xDwl2V4zdn137X6lvfflyRJWvZQlZn3Ar8A/jkiYtmLtFT2GuH+5X3NiUvqZkqw2qNv+cuW4TUlSdIk1VZH9XcB5wNnRcRM4F7KBJ8TNYv5dtocwQAAEEpJREFUbhHxIeAnlBGAL+CRYWiJZOa8iPg48PGIuAO4mBKotqmrzF+W15ckSZNLK1MqZOb/UILMo4CvA6dSpj+4pY3XH4PXAU8DzqCM3ntTZi6yQ/kSOA74MHAI8F1K/7EP18fuaeH1JUnSJBHLNmBv6omILwEvyMzHj7Jelj71g86/f3uG4e8N06YNx3XUV1hhOK4sddHVV3ZdhDF58TOe13URxmTOnFu7LsKo5s17qOsiqBuXjza6v4tr/w2NeqHm/YCfUZr7dqNMTvrOLsslSZIGj6Fq8e4FngO8GVgZuJESqI7tslCSJGnwGKoWo048Ohx15pIkqVPD0blCkiRpwBmqJEmSWmCokiRJaoGhSpIkqQWGKkmSpBYYqiRJklpgqJIkSWqBoUqSJKkFhipJkqQWGKokSZJaEJnZdRkmpYhww0450XUB1IHdd//XroswJlvt8MSuizAmp3/tS10XYVTXX39V10VQJ/LyzNxhcWtYUyVJktQCQ5UkSVILDFWSJEktMFRJkiS1wFAlSZLUAkOVJElSCwxVkiRJLTBUSZIktcBQJUmS1AJDlSRJUgsMVZIkSS0wVEmSJLXAUCVJktQCQ5UkSVILDFWSJEktMFRJkiS1wFAlSZLUAkOVJElSC6Z0qIqIfSPiqoiYGxE3R8SHImJ6feygiMiI2CYizouIeyNiVkTs3XW5JUnS4JmyoSoiXgicCvwS2AP4LPAO4HN9q34TOBPYC7gWOCUiNpjAokqSpCEwvesCdOgo4MLMfHW9f3ZEAHwkIj7YWO+4zDwJICIuB24DXgL/v737D7a0rusA/v7srrCSijEG8SPoBzk2jo04qxPajOavNCqVlFImgZSdNEsdxxlzUlF0rCbLkBpcSBfNRKmxQKZARWuymMSknNFSFJwWUZJNUBFY3E9/nHPpel323t2+u+eee1+vmTP3nuf5Ps/zvpd7d998n+85mwuWnrCqtibZekBTAwCr0rqcqaqqjUkeleTSJbvel8n35ORF265a+KS7b01yS5I9zlR197bu3tLdW8YmBgBWu3VZqpI8JMn9Mpl1Wmzh+RGLtn19yZi7k2w+QLkAgDm1XkvV15LsSnLkku1HTT/uPLhxAIB5ty5LVXd/J8knkzxnya7TkuxO8s8HPRQAMNfW80L11yW5sqremeSSJI9Icm6SC7t7x3TROgDAiqzLmaok6e6rkvxKki1JLk/ysiRvSfKSWeYCAObTep6pSne/L5NX/O1p3/Yk2/ew/YcPaCgAYC6t25kqAICRlCoAgAGUKgCAAZQqAIABlCoAgAGUKgCAAZQqAIABlCoAgAGUKgCAAZQqAIABlCoAgAGUKgCAAZQqAIABqrtnnWFNqirfWFgHDjlk86wjrMjpZ75q1hFW5JgfP3bWEZb1pldunXWENWZu/rr8ZHdv2dsAM1UAAAMoVQAAAyhVAAADKFUAAAMoVQAAAyhVAAADKFUAAAMoVQAAAyhVAAADKFUAAAMoVQAAAyhVAAADKFUAAAMoVQAAAyhVAAADKFUAAAMoVQAAAyhVy6iqk6vqsqr6clV9q6quq6rTZ50LAFhdNs06wBw4IcnHk1yQ5M4kj0vyzqra3d3vnWkyAGDVUKqW0d2XLHxeVZXkH5Icl+TsJEoVAJBEqVpWVX1/ktcneUaSY5NsnO66aQ9jtybZevDSAQCrhVK1vO1JfirJuUk+k+T2JC/KpGR9l+7elmRbklRVH7yIAMCsKVV7UVWbk5yS5CXdfcGi7Rb4AwDfRTnYu0Mzud1318KGqnpgkl+cWSIAYFUyU7UX3X1bVX0iyWur6vYku5O8KsltSR4003AAwKpipmp5z0tyQ5J3JfnjJH81/RwA4F5mqpbR3dcneeIedp1zkKMAAKuYmSoAgAGUKgCAAZQqAIABlCoAgAGUKgCAAZQqAIABlCoAgAGUKgCAAZQqAIABlCoAgAGUKgCAAZQqAIABlCoAgAE2zToArB016wBrTM86wIrs2nX3rCOsyI4bbpx1hBU5+ZmPnXWENWQ+fofWEjNVAAADKFUAAAMoVQAAAyhVAAADKFUAAAMoVQAAAyhVAAADKFUAAAMoVQAAAyhVAAADKFUAAAMoVQAAAyhVAAADKFUAAAMoVQAAA6ypUlVVp1XVmft57DlV9bXBkQCAdWJNlaokpyU5cz+PvSjJz46LAgCsJ5tmHWC16O4dSXbMOgcAMJ9WNFNVVdur6tqqOqWqPlNVd1TVFVV1RFWdWFUfrapvTcf85KLjDquq86rqK1V1Z1V9oqqeuuTcN1bVHyzZdmZVdVU9YPr8CdPnT6iqS6vqm1X1xap68eKMSX4pyeOnY7uqzpnuO6WqPlRVt1TV7VV1zR5yfNftv5VcEwBgwb7c/js+yRuS/E6SrUkem2Rbkkumj2dnMvN1SVXV9JgLk5yV5E1JnpXkv5JcUVU/vZ95L0zyb9NzfSzJn1TVY6b7zk3y0SSfSnLy9HHRdN+PJLk8ya9mUrz+KcnfVtXj/p/XBABIsm+3/45IcnJ3fyFJpjNSr0xyRne/a7qtklyR5GHTXvXcJGd198XT/Vcm+fckr8n+rV96b3e/cXqujyX5hSSnJvmX7v5CVe1MsqG7r1l8UHefv/B5VW3IpHw9PMkLknx8f6+5H/kBgDVqX2aqblwoVFPXTz9evYdtxyZ5dJJKcunCzu7ePX2+vzNVVy06164kn09y3HIHVdVxVXVxVd2U5J4ku5I8NclDR16zqrZOb4Feu4LzAgBryL7MVH19yfO797B9YdvmJEcn+WZ337HkuK8mOayqDu3uu/bh+veVYfPeDpjOTF2W5IFJXptJ8ftWJrcyjxx5ze7elskt0VRVr+DcAMAacSBf/XdzkgdU1WFLitVRSe5YVKjuTHLIkmOPGJjjxCQnJXl6d//dwsaquv/AawAA69yBfJ+qTyTpTBawJ7l3zdWzk/zjonE7kvzEkmOfsp/X3NMs0kJ5undWrKpOSLKSReoAACtywGaquvuzVfXeJOdX1YMyue12dpKHJXnRoqEfSPK2qnp1JkXs1EwWke+P/0jyjKp6ZiZl7cvTbTuSvKWqXpPJbcDXJ7lpP68BAPA9DvQ7qp+d5OJMXu33N0lOSPLz3b14pmpbkrcm+a0k789ktumN+3m9P81kYfk7MiloW6e3GU/NZIH6X2by1gtvTvL3+3kNAIDvUd3WUx8IFqqvR7X8EPbBfPwKTV4Ls/o9+cnPn3WEFXnOy5836wjL2vpz8/Ivms3H79Ac+WR3b9nbgPn40wAAYJVTqgAABlCqAAAGUKoAAAZQqgAABlCqAAAGUKoAAAZQqgAABlCqAAAGUKoAAAZQqgAABlCqAAAGUKoAAAbYNOsAa1vNOsAK+FfM15uqefi5THpOfjQ3bJiP/ze94YZPzzrCihx3zJGzjrCseflvPi96Tn7Zu3cvO8ZPBgDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwACbZh1gLamqrUm2zjoHAHDwKVUDdfe2JNuSpKp6xnEAgIPI7T8AgAGUKgCAAZSqfVRVz6+qe6rqhFlnAQBWD6Vq321IsjFJzToIALB6KFX7qLu3d3d1942zzgIArB5KFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAJtmHWDtqmzadL9Zh1jWPffsmnWEFepZB1jWxo0bZx1hTdm9e/esI6zI5s3fN+sIK3LI/Q6ddYQVufzdV846wrI2bpyPvzqPOebEWUdYkW98Y+esI6zIzp03LzvGTBUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAKu2VFXVj83gmj9YVYcd7OsCAPNvVZWqqtpcVadX1dVJPr9o+4aqelVVXV9Vd1XV56rqjD0c/5Kq+vx0zPVV9fIl+4+rqvdX1S1V9e2q+kJVnbtoyNOS3FxVb6+qRx+wLxQAWHM2zTpAklTVI5O8MMnpSQ5LclmSUxYNeVuSM5K8Icm/JnlKkndU1a3d/cHpOc6ejvvDJFcm+Zkkb6mqQ7v7d6fneVeS+yfZmuTrSX40ycMWXecDSR6U5KwkW6vq00n+LMm7u3vn6K8bAFg7ZlaqqurwTErUC5I8Ksl1SV6X5M8XF5iqOjHJi5Kc1d0XTzd/uKqOno7/YFVtSHJOku3d/YrpmKum1/jtqnprd9+Z5DFJntvdl0/HfGxxpu6+Lcl5Sc6rqpOS/FqS1yb5var66yQXJflId/d9fE1bMylsAMA6M5Pbf1X1tCQ3Jzk3yceTnNTdJ3X3eXuYEXpSkt1JPlBVmxYeST6S5JFVtTHJcUmOSXLpkmPfl8nM0yOmz69L8uaqOrOqjt9bxu7+VHf/5vS8ZyR5cCYzYF/cyzHbuntLd29JarlvAwCwhsxqTdVdSe5IsjnJ4UkeXFX31UIekmRjktuS7Fr02J7JTNvR00eSfHXJsQvPj5h+/OUk1yb5oyRfqqrrqupJy2RdyHh4Jt+v/1lmPACwDs3k9l93f7Sqjk3yrExu/12d5Maq2p7k4u7+0qLhO5Pck+RxmcxYLXVL/q8cHrlk31GLzpHuvinJmdPbhY/J5JbhZVV1fHffunDQtOA9MZO1VacmuTvJXyR5cXd/an++ZgBgbZvZq/+6+67uvqS7n5LJgvH3JDk7yQ1V9eGqOn069OpMZqoO7+5r9/C4O8mOJF9O8pwllzktye1JPr3k2ru7+5okr89kYfwJSVJVR1XVOUluSPLhJMcn+fUkR3e3QgUA3KdV8eq/7r4xyWumheZpmbwScHuS93T3f1bVBUkuqarfz+T23eYkD0/y0O5+YXfvnh779qq6NcmHkjw+kwXur+7uO6eL1q/M5BWAn0tyaJJXJPlKks9Oozw9kxJ1cZKLuvvet3UAANibVVGqFnT3d5JckeSKqjpq0a7fyKQInZ3J2yrcnuQzmbzdwcKxF1bVoUleluSlmcxevaK7/2g65M5MZqxemuSHMlnTdU2Sp3b3t6djLsvk1Yf3HJivEABYq1ZVqVqsu7+66PNO8tbpY2/HnJ/k/PvYd1cmpWxvx3svKgBgv6yqd1QHAJhXShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAAShUAwABKFQDAAEoVAMAA1d2zzrAmVdV/J/nS4NM+JMnXBp/zQJBzrHnIOQ8ZEzlHk3OceciYrO+cJ3T3D+xtgFI1R6rq2u7eMuscy5FzrHnIOQ8ZEzlHk3OceciYyLkct/8AAAZQqgAABlCq5su2WQdYITnHmoec85AxkXM0OceZh4yJnHtlTRUAwABmqgAABlCqAAAGUKoAAAZQqgAABlCqAAAG+F+DvgLGyrw1QQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "translation, attention = translate_sentence(src, SRC, TRG, model, device)\n",
    "\n",
    "print(f'predicted trg = {translation}')\n",
    "\n",
    "display_attention(src, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BLEU\n",
    "\n",
    "Previously we have only cared about the loss/perplexity of the model. However there metrics that are specifically designed for measuring the quality of a translation - the most popular is *BLEU*. Without going into too much detail, BLEU looks at the overlap in the predicted and actual target sequences in terms of their n-grams. It will give us a number between 0 and 1 for each sequence, where 1 means there is perfect overlap, i.e. a perfect translation, although is usually shown between 0 and 100. BLEU was designed for multiple candidate translations per source sequence, however in this dataset we only have one candidate per source.\n",
    "\n",
    "We define a `calculate_bleu` function which calculates the BLEU score over a provided TorchText dataset. This function creates a corpus of the actual and predicted translation for each source sentence and then calculates the BLEU score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data.metrics import bleu_score\n",
    "\n",
    "def calculate_bleu(data, src_field, trg_field, model, device, max_len = 50):\n",
    "    \n",
    "    trgs = []\n",
    "    pred_trgs = []\n",
    "    \n",
    "    for datum in data:\n",
    "        \n",
    "        src = vars(datum)['src']\n",
    "        trg = vars(datum)['trg']\n",
    "        \n",
    "        pred_trg, _ = translate_sentence(src, src_field, trg_field, model, device, max_len)\n",
    "        \n",
    "        #cut off <eos> token\n",
    "        pred_trg = pred_trg[:-1]\n",
    "        \n",
    "        pred_trgs.append(pred_trg)\n",
    "        trgs.append([trg])\n",
    "        \n",
    "    return bleu_score(pred_trgs, trgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a BLEU of around 29. If we compare it to the paper that the attention model is attempting to replicate, they achieve a BLEU score of 26.75. This is similar to our score, however they are using a completely different dataset and their model size is much larger - 1000 hidden dimensions which takes 4 days to train! - so we cannot really compare against that either.\n",
    "\n",
    "This number isn't really interpretable, we can't really say much about it. The most useful part of a BLEU score is that it can be used to compare different models on the same dataset, where the one with the **higher** BLEU score is \"better\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BLEU score = 29.04\n"
     ]
    }
   ],
   "source": [
    "bleu_score = calculate_bleu(test_data, SRC, TRG, model, device)\n",
    "\n",
    "print(f'BLEU score = {bleu_score*100:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next tutorials we will be moving away from using recurrent neural networks and start looking at other ways to construct sequence-to-sequence models. Specifically, in the next tutorial we will be using convolutional neural networks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}