yelp_humor_detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Yelp Review Humor Detection\n",
    "## Running the notebook\n",
    "To run this Jupyter notebook, you can clone the [Conda](https://conda.io/docs/) environment that I used by installing the dependencies listed in the spec-file.txt in this repository. Simply run\n",
    "\n",
    "`conda create --name myenv --file spec-file.txt\n",
    "source activate myenv`\n",
    "\n",
    "where \"myenv\" is the name of your environment. Then, to get the word vectors, download the spaCy model with \n",
    "\n",
    "`python -m spacy download en_vectors_web_lg`\n",
    "\n",
    "My Python version is 3.6.2.\n",
    "\n",
    "## The premise\n",
    "Plenty of machine learning research has investigated sentiment analysis in the context of positive versus negative online reviews. By contrast, very few works have focused on computational humor... in this case, the detection of _funny_ reviews.\n",
    "\n",
    "[Oliveira & Rodrigo](https://cs224d.stanford.edu/reports/OliveiraLuke.pdf) determined that making such predictions is quite possible with a variety of machine learning architectures. This notebook attempts to reproduce a subset of their results - specifically, the accuracy they obtained from the use of a recurrent neural network (RNN).\n",
    "\n",
    "## The problem\n",
    "Using the [Yelp Open Dataset](https://www.yelp.com/dataset) of over 4 million business reviews, I want to see if it's possible to train a classifier that will correctly determine whether a Yelp review is funny or not. Humor, of course, is a largely subjective measure. Yelp allows users to mark a particular review as being funny. So, to more rigorously define the binary classification problem and the source of truth, I use the same assumption made by Oliveira & Rodrigo: that a review is \"funny\" if it has 3 or more \"funny\" votes.\n",
    "\n",
    "## The approach\n",
    "A review can be seen as a sequence of word vectors, where each vector represents a similarity measure for a particular \"token\" in the review. A token can be a word, punctuation mark, etc. (The exact tokenization of a review is dependent on the language in which it is written.) We can view these sequences as many-to-one mappings via an RNN that outputs a single label, denoting \"funny\" or \"not funny,\" after feeding it an entire sequence. With an RNN that includes memory units in its architecture, we can attempt to capture long-term dependencies within the text for improved classification.\n",
    "\n",
    "### Overview of the data\n",
    "Let's begin by first looking at what a single Yelp review looks like from our dataset. I can't include the JSON file due to its size (and possibly some restrictions on how it should be shared), so **don't run the code block below!** Just observe the output. If you want to replicate this work, you'll want to get the dataset yourself from the link above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'review_id': 'VfBHSwC5Vz_pbFluy07i9Q', 'user_id': 'cjpdDjZyprfyDG3RlkVG3w', 'business_id': 'uYHaNptLzDLoV_JZ_MuzUA', 'stars': 5, 'date': '2016-07-12', 'text': 'My girlfriend and I stayed here for 3 nights and loved it. The location of this hotel and very decent price makes this an amazing deal. When you walk out the front door Scott Monument and Princes street are right in front of you, Edinburgh Castle and the Royal Mile is a 2 minute walk via a close right around the corner, and there are so many hidden gems nearby including Calton Hill and the newly opened Arches that made this location incredible.\\n\\nThe hotel itself was also very nice with a reasonably priced bar, very considerate staff, and small but comfortable rooms with excellent bathrooms and showers. Only two minor complaints are no telephones in room for room service (not a huge deal for us) and no AC in the room, but they have huge windows which can be fully opened. The staff were incredible though, letting us borrow umbrellas for the rain, giving us maps and directions, and also when we had lost our only UK adapter for charging our phones gave us a very fancy one for free.\\n\\nI would highly recommend this hotel to friends, and when I return to Edinburgh (which I most definitely will) I will be staying here without any hesitation.', 'useful': 0, 'funny': 0, 'cool': 0}\n",
      "\n",
      "Total reviews: 4736897\n",
      "Funny reviews: 176788\n",
      "Normal reviews: 4560109\n"
     ]
    }
   ],
   "source": [
    "### DON'T RE-RUN THIS BLOCK\n",
    "\n",
    "import json\n",
    "\n",
    "total = 0\n",
    "num_funny = 0\n",
    "funny_threshold = 3\n",
    "with open('yelp_dataset/review.json') as reviews_file:\n",
    "    for line in reviews_file:\n",
    "        total += 1\n",
    "        review = json.loads(line)\n",
    "        if total == 1:\n",
    "            # Let's print the very first review, just to see what it looks like.\n",
    "            print(review)\n",
    "        if review['funny'] >= funny_threshold:\n",
    "            num_funny += 1\n",
    "            \n",
    "print('\\nTotal reviews: %d\\nFunny reviews: %d\\nNormal reviews: %d' % (total, num_funny, total - num_funny))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can probably guess, for our purposes, the most important fields in a Yelp review \"data point\" are the 'text' and 'funny' keys. The 'text' key maps to the text of the review, which will ultimately be transformed into a sequence of word vectors. The 'funny' key maps to the number of \"funny\" votes that the review received from other Yelp users. If this value is greater than or equal to the funny threshold (set at 3), then the review is labeled as funny. Otherwise, it's not funny.\n",
    "\n",
    "To create my own dataset, I simply read through the JSON file and store the reviews as a list of dictionaries, with each dictionary representing a single sample containing the text of a particular review along with its label **(0 for not funny, 1 for funny)**. I stop reading the JSON file when I've created a balanced dataset of the desired size, with equal numbers of funny and not funny reviews. **The first half of the dataset consists entirely of funny reviews while the second half contains only of not funny reviews.** This observation will allow me to easily split the dataset for training/testing while still maintaining balanced sets.\n",
    "\n",
    "I wrap this dataset inside a PyTorch Dataset object to take advantage of easier training/testing splitting afforded by the Sampler and DataLoader classes. To save time, I loaded the dataset through a pickle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading from pickle...\n",
      "Loaded 100000 reviews from dataset.p!\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "class ReviewsDataset(Dataset):\n",
    "    def __init__(self, reviews=[], fields=['text']):\n",
    "        '''\n",
    "        Args:\n",
    "            reviews (List[dict]): List of dictionaries, each one a sample from the dataset\n",
    "            fields (List[string]): List of fields to extract from each review\n",
    "        '''\n",
    "        self.reviews = reviews\n",
    "        self.fields = fields\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.reviews)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        return self.reviews[idx]\n",
    "    \n",
    "    def read_from_file(self, raw_data_path, count=2000, pickle_filename=None, funny_threshold=3):\n",
    "        '''\n",
    "        Reads a JSON file containing the Yelp reviews and creates a dataset from the raw data.\n",
    "        Args:\n",
    "            raw_data_path (string): Path to the Yelp JSON file\n",
    "            count (int): Number of reviews to extract from each label (dataset size will be twice this number)\n",
    "            pickle_filename (string): Name used for the pickled dataset; if None, pickle is not created\n",
    "            funny_threshold (int): Number of \"funny\" votes needed for a review to be considered funny\n",
    "        '''\n",
    "        funny_reviews = []\n",
    "        not_funny_reviews = []\n",
    "        with open(raw_data_path) as reviews_file:\n",
    "            for line in reviews_file:\n",
    "                # Break when we've created a balanced dataset with count reviews in each category.\n",
    "                if len(funny_reviews) == count and len(not_funny_reviews) == count:\n",
    "                    break\n",
    "                    \n",
    "                sample = {}\n",
    "                review = json.loads(line)\n",
    "                for f in self.fields:\n",
    "                    sample[f] = review[f]\n",
    "                    \n",
    "                if review['funny'] >= funny_threshold and len(funny_reviews) < count:\n",
    "                    sample['label'] = 1\n",
    "                    funny_reviews.append(sample)\n",
    "                elif review['funny'] < funny_threshold and len(not_funny_reviews) < count:\n",
    "                    sample['label'] = 0\n",
    "                    not_funny_reviews.append(sample)\n",
    "        \n",
    "        # Notice that the first half of self.reviews is all funny reviews \n",
    "        # and the second half is all not funny reviews.\n",
    "        self.reviews = funny_reviews + not_funny_reviews[:len(funny_reviews)]\n",
    "        \n",
    "        if pickle_filename is not None:\n",
    "            print('Pickling %d reviews...' % (len(self)))\n",
    "            pickle.dump(self.reviews, open(pickle_filename, 'wb'))\n",
    "            print('Pickled as %s' % (pickle_filename))\n",
    "            \n",
    "        print('Loaded a balanced dataset of %d reviews!' % (len(self.reviews)))\n",
    "\n",
    "    def read_from_pickle(self, pickle_path):\n",
    "        '''\n",
    "        Loads a dataset from a pickle.\n",
    "        Args:\n",
    "            pickle_path: Path to the pickle file to be used\n",
    "        '''\n",
    "        print('Loading from pickle...')\n",
    "        self.reviews = pickle.load(open(pickle_path, \"rb\"))\n",
    "        print('Loaded %d reviews from %s!' % (len(self), pickle_path))\n",
    "    \n",
    "data = ReviewsDataset()\n",
    "# data.read_from_file('yelp_dataset/review.json', 50000, 'dataset.p')\n",
    "data.read_from_pickle('dataset.p')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a single sample from my dataset (the review is kind of funny, I guess, but maybe I should have bumped up the humor threshold)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': 'Slots-A-Fun has very strange odor. A mixture of cigarettes, hot dogs, pizza, B.O., and industial strength cleaner (the kind they use in the \"adult\" bookstores). I took a couple friends through here on the obligatory ten minute tour. Enough time to buy a couple beers, lose $3 in a slot machine (at least I got a quarter back!!! A real quarter-not a coupon-the only reason this place gets 2 stars!) and cruise by the $1 roulette table. I was hoping to teach my table game-wary friend how to play a simple game of roulette, but the proliferation of Lil\\' John and Ludacris wannabes hovering nearby scared her away. YEEEEAAAAH! Oh well- I never do well at those low limit tables anyway. I played craps here once and dealers were extemely rude. I wonder why this place, and circus circus,still exist when everything else nearby has been imploded.', 'label': 1}\n"
     ]
    }
   ],
   "source": [
    "print(data[200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizing the data\n",
    "Each review sample must be converted into a sequence of word vectors before it can be fed through the RNN. To make this conversion, I use the [spaCy](https://spacy.io/) library, which allows one to easily tokenize a text and generate word vectors for each token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple\n",
      "is\n",
      "looking\n",
      "at\n",
      "buying\n",
      "U.K.\n",
      "startup\n",
      "for\n",
      "$\n",
      "1\n",
      "billion\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_vectors_web_lg')\n",
    "\n",
    "# Example of using spaCy to generate tokens\n",
    "doc = nlp(u'Apple is looking at buying U.K. startup for $1 billion')\n",
    "for token in doc:\n",
    "    print(token.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These vectors (of length 300) will then be converted into PyTorch tensors and fed, one at a time and in order, into the RNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "def token2tensor(token):\n",
    "    return torch.from_numpy(token.vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One unresolved issue here is the fact that some tokens do not have word vectors. A common reason for this problem is that the token represents a word that is out-of-vocabulary or a lot less common than others (or, in the case of Yelp reviews, the word is misspelled).\n",
    "\n",
    "In those cases, `token.vector` simply returns a zero-vector. There's a good chance that these words are negatively impacting the learning process, since zero-vectors are treated no differently by the RNN. Skipping over the word doesn't seem like the best idea (since we want to preserve the sequence of words as much as possible), so for now, I'm betting on the zero-vector not showing up enough during training to matter in the grand scheme of things.\n",
    "\n",
    "### The RNN\n",
    "Now that we've defined our input to the RNN, let's make the network itself. The RNN uses Gated Recurrent Units (GRU) with 128 hidden units and a dropout ratio of 0.5. The input and hidden sequences are also sent through a linear transformation and LogSoftmax layer to generate the output (label)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "\n",
    "# create the RNN\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size):\n",
    "        '''\n",
    "        Initialize the RNN layers.\n",
    "        Args:\n",
    "            input_size: Length of the input word vector\n",
    "            hidden_size: Number of hidden units in the RNN\n",
    "        '''\n",
    "        super(RNN, self).__init__()\n",
    "        \n",
    "        self.hidden_size = hidden_size\n",
    "        \n",
    "        self.input2hidden = nn.GRU(input_size, hidden_size, dropout=0.5)\n",
    "        self.input2output = nn.Linear(input_size + hidden_size, out_features=2)\n",
    "        self.softmax = nn.LogSoftmax()\n",
    "        \n",
    "    def initHidden(self):\n",
    "        '''\n",
    "        Initialize all hidden sequences to zero.\n",
    "        '''\n",
    "        return Variable(torch.zeros(1, 1, self.hidden_size))\n",
    "    \n",
    "    def forward(self, input, hidden):\n",
    "        '''\n",
    "        Generate a new hidden state and output given an input vector and current hidden state.\n",
    "        Args:\n",
    "            input (Variable): The input word tensor\n",
    "            hidden (Variable): The last hidden state tensor\n",
    "            \n",
    "        Returns:\n",
    "            output (Tensor): 2x1 tensor representing likelihoods that a review is funny based on the input so far\n",
    "            hidden (Tensor): The next hidden state tensor\n",
    "        '''\n",
    "        # Combine the input and hidden tensors for use in generating the output.\n",
    "        # We use index 0 for the first two dimensions because the tensor dimensions \n",
    "        # are structured as (Number of Batches x Number of Instances Per Batch x Length of Each Instance's Tensor)\n",
    "        combined = torch.cat((input[0,0,:], hidden[0,0,:]))\n",
    "        \n",
    "        # Generate next hidden state.\n",
    "        hidden, _ = self.input2hidden(input, hidden)\n",
    "        \n",
    "        # Generate next output label by passing the previous input and hidden state through \n",
    "        # a linear transformation and LogSoftmax layer.\n",
    "        output = self.input2output(combined)\n",
    "        output = self.softmax(output)\n",
    "        return output, hidden\n",
    "    \n",
    "# data specifications\n",
    "input_size = 300 # length of a single word vector\n",
    "hidden_units = 128\n",
    "\n",
    "# initialize the RNN\n",
    "model = RNN(input_size, hidden_units)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned earlier, I train the RNN by taking each sample's text and feeding the word vectors for each token one at a time as input. I optimize on the negative log-likelihood loss function using the _adam_ optimizer.\n",
    "\n",
    "To make a prediction given a particular review, I first do the same thing as I do when training except I don't use the output to optimize the network parmeters. Then, I simply return the index (1 if funny, 0 if not funny) of the maximum value of the 2-length vector output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "# training specifications\n",
    "criterion = nn.NLLLoss() # negative log-likelihood loss function\n",
    "learning_rate = 0.00001\n",
    "optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n",
    "\n",
    "def train_one(review):\n",
    "    '''\n",
    "    Trains the RNN on one review.\n",
    "    Args:\n",
    "        review (Dictionary): A batch containing a single review\n",
    "        \n",
    "    Returns:\n",
    "        output (Tensor): 2x1 tensor representing likelihoods that a review is funny based on the input so far\n",
    "        loss (float): The value of the loss function at the last layer of the RNN\n",
    "    '''\n",
    "    # Zero out the model gradients and hidden state so they aren't influenced by previous reviews.\n",
    "    model.zero_grad()\n",
    "    hidden = model.initHidden()\n",
    "    \n",
    "    # review is technically a batch, so we take the text of the first sample of the batch...\n",
    "    # (hence the 0 index)\n",
    "    doc = nlp(review['text'][0])\n",
    "    \n",
    "    # Feed the input one token at a time through our model.\n",
    "    for token in doc:\n",
    "        input_tensor = torch.zeros(1, 1, input_size)\n",
    "        input_tensor[0,0,:] = token2tensor(token)\n",
    "        output, hidden = model(Variable(input_tensor), hidden)\n",
    "        \n",
    "    # Calculate the loss functions at each layer; backpropagate and optimize.\n",
    "    loss = criterion(torch.unsqueeze(output, 0), Variable(torch.Tensor([review['label'][0]]).long()))\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    return output, loss.data[0]\n",
    "\n",
    "def get_label(output):\n",
    "    '''\n",
    "    Determine whether an output tensor says a review is funny or not funny.\n",
    "    Args:\n",
    "        output: 2x1 tensor\n",
    "        \n",
    "    Returns: \n",
    "        1 if the review corresponding to the tensor is more likely to be funny than not funny.\n",
    "        0 otherwise.\n",
    "    '''\n",
    "    _, indices = output.data.topk(1)\n",
    "    return indices[0]\n",
    "\n",
    "def predict(review):\n",
    "    '''\n",
    "    Predict whether a review is funny or not funny.\n",
    "    Args:\n",
    "        review (Dictionary): A batch containing a single review\n",
    "        \n",
    "    Returns:\n",
    "        1 if the review corresponding to the tensor is more likely to be funny than not funny.\n",
    "        0 otherwise.\n",
    "    '''\n",
    "    hidden = model.initHidden()\n",
    "    doc = nlp(review['text'][0])\n",
    "    for token in doc:\n",
    "        input_tensor = torch.zeros(1, 1, input_size)\n",
    "        input_tensor[0,0,:] = token2tensor(token)\n",
    "        output, hidden = model(Variable(input_tensor), hidden)\n",
    "\n",
    "    return get_label(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and testing\n",
    "Now, it's time to train the model on the dataset. First, I split the dataset into a training set and a testing set such that both sets remain balanced. Since the first half of the full dataset consists of funny reviews and the second half consists of not funny reviews, I can simply take half of the testing set from the front of the dataset and half from the back. The rest of the reviews become my training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set has 95000 samples, validation set has 5000 samples\n"
     ]
    }
   ],
   "source": [
    "# 5% of the dataset (5,000 reviews for a 100,000-review dataset) is used for testing/validation.\n",
    "valid_size = 0.05\n",
    "total = len(data)\n",
    "indices = list(range(total))\n",
    "num_validation_samples = int(total / 2 * (valid_size)) # Each label makes up half of the testing set.\n",
    "\n",
    "testing_indices = indices[:num_validation_samples] + indices[-num_validation_samples:]\n",
    "training_indices = indices[num_validation_samples:-num_validation_samples]\n",
    "num_testing = len(testing_indices)\n",
    "num_training = len(training_indices)\n",
    "print('Training set has %d samples, validation set has %d samples' % (num_training, num_testing))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can train. On my laptop, training on just one epoch of a set of 95,000 reviews takes over 10 hours. However, in the event that I ever train for multiple epochs, I've implemented a stop condition for whenever the testing accuracy drops from one epoch to the next. That way, we might avoid overfitting to the training set (though the dropout should help as well)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.sampler import SubsetRandomSampler\n",
    "import math, time\n",
    "\n",
    "def timeSince(since):\n",
    "    '''\n",
    "    Get the amount of time that has elapsed since some reference time.\n",
    "    Args:\n",
    "        since (float): The reference time\n",
    "        \n",
    "    Returns:\n",
    "        (string): Amount of time in minutes and seconds that has elapsed since the reference time\n",
    "    '''\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)\n",
    "\n",
    "def train(training_set, testing_set, epochs=1):\n",
    "    '''\n",
    "    Trains the RNN.\n",
    "    Args:\n",
    "        training_set (DataLoader): The training set\n",
    "        testing_set (DataLoader): The testing set\n",
    "        epochs (int): Number of epochs\n",
    "        \n",
    "    Returns:\n",
    "        loss_function (List[float]): List of loss function values aggregated over len(training_set)/200 sample intervals.\n",
    "    '''\n",
    "    start = time.time()\n",
    "    print_every = int(len(training_set) / 20)\n",
    "    plot_every = int(len(training_set) / 200)\n",
    "    loss_function = []\n",
    "    total_loss = 0 # Keep track of loss function values over the course of training plot_every samples.\n",
    "    prev_accuracy = 0 # Testing accuracy after training on the previous epoch.\n",
    "    for epoch in range(epochs):\n",
    "        print('Epoch %d' % (epoch))\n",
    "        for i, batch in enumerate(training_set):\n",
    "            # Each batch has just one sample/review.\n",
    "            output, loss = train_one(batch)\n",
    "            total_loss += loss\n",
    "            guess = get_label(output)\n",
    "\n",
    "            # Print sample number, progress percentage, and the loss and guess for that particular sample at regular intervals.\n",
    "            if i % print_every == 0:\n",
    "                correct = '✓' if get_label(output) == batch['label'][0] else '✗ (%s)' % batch['label'][0]\n",
    "                print('Sample %d Progress: %d%% (%s) Loss: %.4f Guess: %d %s' % (i, i / (len(training_set) * epochs) * 100, timeSince(start), loss, guess, correct))\n",
    "\n",
    "            if i % plot_every == 0 and i > 0:\n",
    "                loss_function.append(total_loss / plot_every)\n",
    "                total_loss = 0\n",
    "\n",
    "        print('Testing now...')\n",
    "        accuracy = test(testing_set)\n",
    "        print('Testing accuracy after epoch %d: %.4f%%' % (epoch, accuracy * 100))\n",
    "        if accuracy < prev_accuracy:\n",
    "            print('Testing accuracy dropped... model appears to be overfitting. Stopping training at epoch %d.' % (epoch))\n",
    "            break\n",
    "        else:\n",
    "            prev_accuracy = accuracy\n",
    "            \n",
    "    return loss_function\n",
    "\n",
    "def test(testing_set):\n",
    "    '''\n",
    "    Tests the RNN.\n",
    "    Args:\n",
    "        testing_set (DataLoader): The testing set\n",
    "        \n",
    "    Returns:\n",
    "        accuracy (float): Testing accuracy\n",
    "    '''\n",
    "    correct = 0\n",
    "    print_every = int(len(testing_set) / 20)\n",
    "    start = time.time()\n",
    "    for i, batch in enumerate(testing_set):\n",
    "        # Again, each batch has just one review.\n",
    "        correct += 1 if predict(batch) == batch['label'][0] else 0\n",
    "        \n",
    "        # Print statements at regular intervals to indicate testing progress.\n",
    "        if i % print_every == 0:\n",
    "            print('Sample %d Progress: %d%% (%s) Accuracy: %.4f' % (i, i / len(testing_set) * 100, timeSince(start), correct / (i + 1)))\n",
    "    \n",
    "    accuracy = correct / len(testing_set)\n",
    "#     print('Testing accuracy: %.4f' % (accuracy))\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output below the next cell shows the progress of the overnight training and testing process. As you can see, the model achieves a testing accuracy of around 75%. Pretty good and only a little less than the accuracy achieved by Oliviera & Rodrigo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0\n",
      "Sample 0 Progress: 0% (0m 0s) Loss: 0.6776 Guess: 0 ✓\n",
      "Sample 4750 Progress: 5% (35m 57s) Loss: 0.7309 Guess: 0 ✗ (1)\n",
      "Sample 9500 Progress: 10% (83m 31s) Loss: 0.7166 Guess: 1 ✗ (0)\n",
      "Sample 14250 Progress: 15% (117m 11s) Loss: 0.5417 Guess: 0 ✓\n",
      "Sample 19000 Progress: 20% (150m 11s) Loss: 0.6048 Guess: 1 ✓\n",
      "Sample 23750 Progress: 25% (182m 17s) Loss: 0.8632 Guess: 1 ✗ (0)\n",
      "Sample 28500 Progress: 30% (212m 56s) Loss: 0.6235 Guess: 0 ✓\n",
      "Sample 33250 Progress: 35% (242m 50s) Loss: 0.9318 Guess: 1 ✗ (0)\n",
      "Sample 38000 Progress: 40% (273m 3s) Loss: 1.2925 Guess: 0 ✗ (1)\n",
      "Sample 42750 Progress: 45% (302m 42s) Loss: 0.1228 Guess: 0 ✓\n",
      "Sample 47500 Progress: 50% (332m 27s) Loss: 0.1150 Guess: 0 ✓\n",
      "Sample 52250 Progress: 55% (362m 31s) Loss: 0.2325 Guess: 1 ✓\n",
      "Sample 57000 Progress: 60% (393m 1s) Loss: 0.2295 Guess: 1 ✓\n",
      "Sample 61750 Progress: 65% (423m 41s) Loss: 0.2383 Guess: 1 ✓\n",
      "Sample 66500 Progress: 70% (453m 52s) Loss: 0.3823 Guess: 1 ✓\n",
      "Sample 71250 Progress: 75% (483m 37s) Loss: 0.2018 Guess: 0 ✓\n",
      "Sample 76000 Progress: 80% (513m 27s) Loss: 0.2113 Guess: 0 ✓\n",
      "Sample 80750 Progress: 85% (544m 6s) Loss: 0.2104 Guess: 1 ✓\n",
      "Sample 85500 Progress: 90% (574m 26s) Loss: 0.2550 Guess: 0 ✓\n",
      "Sample 90250 Progress: 95% (603m 59s) Loss: 0.3770 Guess: 1 ✓\n",
      "Testing now...\n",
      "Sample 0 Progress: 0% (0m 0s) Accuracy: 1.0000\n",
      "Sample 250 Progress: 5% (0m 40s) Accuracy: 0.7251\n",
      "Sample 500 Progress: 10% (1m 17s) Accuracy: 0.7385\n",
      "Sample 750 Progress: 15% (1m 52s) Accuracy: 0.7590\n",
      "Sample 1000 Progress: 20% (2m 30s) Accuracy: 0.7542\n",
      "Sample 1250 Progress: 25% (3m 8s) Accuracy: 0.7562\n",
      "Sample 1500 Progress: 30% (3m 44s) Accuracy: 0.7615\n",
      "Sample 1750 Progress: 35% (4m 21s) Accuracy: 0.7487\n",
      "Sample 2000 Progress: 40% (4m 58s) Accuracy: 0.7506\n",
      "Sample 2250 Progress: 45% (5m 38s) Accuracy: 0.7463\n",
      "Sample 2500 Progress: 50% (6m 16s) Accuracy: 0.7461\n",
      "Sample 2750 Progress: 55% (6m 53s) Accuracy: 0.7437\n",
      "Sample 3000 Progress: 60% (7m 35s) Accuracy: 0.7434\n",
      "Sample 3250 Progress: 65% (8m 12s) Accuracy: 0.7450\n",
      "Sample 3500 Progress: 70% (8m 51s) Accuracy: 0.7438\n",
      "Sample 3750 Progress: 75% (9m 30s) Accuracy: 0.7451\n",
      "Sample 4000 Progress: 80% (10m 9s) Accuracy: 0.7438\n",
      "Sample 4250 Progress: 85% (10m 48s) Accuracy: 0.7466\n",
      "Sample 4500 Progress: 90% (11m 28s) Accuracy: 0.7472\n",
      "Sample 4750 Progress: 95% (12m 6s) Accuracy: 0.7459\n",
      "Testing accuracy after epoch 0: 74.5400%\n"
     ]
    }
   ],
   "source": [
    "### DON'T RUN THIS BLOCK UNLESS YOU WANT TO TRAIN AGAIN\n",
    "training_set = DataLoader(dataset=data, sampler=SubsetRandomSampler(training_indices))\n",
    "testing_set = DataLoader(dataset=data, sampler=SubsetRandomSampler(testing_indices))\n",
    "loss_function = train(training_set, testing_set, epochs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the run above, we see that the classifier was able to achieve an accuracy of 74.5% on the 5,000-review testing set. Oliveiria & Rodrigo managed to get a test accuracy of around 78%, so we aren't too far off! We can plot the loss function to see it slowly converging (though it's clearly jumping around a lot)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x1209b65f8>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXmUJFWZ6H9f7llZ+9L7vrEpizQIiAqCI+6MC4rPbWbU\nh6MzjuubeTOjOMcZHUH0qaiDuKID4iiKiCAgO7J0szb0Su/d1d1V1bXmHpn3/RFxIyOzMquyqiur\nqrPu75w8VRkZEXkjMvN+99tFKYXBYDAYDOPhm+kBGAwGg+H4wAgMg8FgMFSFERgGg8FgqAojMAwG\ng8FQFUZgGAwGg6EqjMAwGAwGQ1UYgXGcICLLRGRERPyTPH5ERFbNpjEdw/vOF5EHRGRYRL42ne9d\nCRE5QUSedsb099P4vtP2GYjIfSLyoWM4/g8i8oGpHNNUMpHfSC1+T8cDRmDUCBH5oIg8JyIJETkk\nIt8VkdYJHL9bRC7Wz5VSe5VSjUqp3GTG4xy7czLH1mpMx8BHgF6gWSn16Vq/mYhcKSI/G2e3zwH3\nKqWalFLfrOFYZstnMGGUUq9XSv1kqs8rIheIyP5jPc9EfiNT8Xs6HjECowaIyKeB/wQ+C7QA5wDL\ngbtEJDSTY6sTlgMvqNmVdboceH6mBzEbEZsZnWtEJDCT7183KKXMYwofQDMwAlxWsr0R6AH+2nl+\nJfA/wC+AYeBJ4DTntRuAPJB0zvU5YAWggICzz33Al4BHnH1+B3QAPweGgCeAFZ73V8AaYJGzv34k\n7K+BAlgN/Anow17B/xxoncCYFgG3AkeBHcCHPe9/JXAz8FPnep8H1o9xH89zrmHQ+Xues/3HQBbI\nOOO4uMyxUeBrwB7n+IeAqPPaW5z3HnDu4Ume4/4PcMAZ31bgIuAS572yzvs9U+b9/gTkgJSzzzrn\n3B/y7PNB4KGSz+MKYLszlmsB8bz+YWCzM5YXgJfNwGfwWmCLcw+/Ddyvr8k51888+5aO4z7g34GH\nnfGu8d4TfT+Aq4F+YBfwes/5VgIPOOO827k/Pyszxphz/jyF7/QiCr+vn2H/Hj4EnA382bnf3c41\nhUp/I57v2bXA750xPAasnuS+f4H9fRoEvuO9j8fbY8YHUG8P7AnG0j+cktd+Atzo/H8l9iT0DiAI\nfMb50QSd13fjmQwr/CB3YE/yLdiTyjbgYiDgTAo/8hzvfsFLxvRzz5jWYE8SYaDL+cF+w7PveGN6\nwPlBRIDTsQXkazzXmwLeAPiBLwOPVriH7diTyPuca7nced7hvP5j4EtjfAbXOvdnsfNe5znXtA6I\nO9cYxJ5wdwAh4ARgH7DIc22rPWMfNVmVvOd9FAuI0ucfZLTAuA1oBZY59+oS57V3YguuswBxPpfl\n0/wZdGJPfvr7+Uns7/VEBMZe4BTnMwwyWmBksQWjH/gocBBHaGJP7Fc7n8352JN+2c8AuADYX7Lt\nSuf8l2JbUqLAmdjafsAZ72bgH8r9RrC/Y33YQiaA/Tu5aaL7OvdxCHib89onnHEdlwLDmKSmnk6g\nVylllXmt23lds1Ep9T9KqSxwDfaP/JwJvNePlFIvKqUGgT8ALyql7nbe+5fAGWMdLCL/BzgR+GsA\npdQOpdRdSqm0UqrHGdOrqxmIiCwFXgH8H6VUSin1NHA98H7Pbg8ppW5Xtr39BuC0Cqd7I7BdKXWD\nUspSSt2IvdJ9cxXj8DnX8wml1AGlVE4p9YhSKg28C/i9c41Z7Akpii1QcthC5WQRCSqldiulXqzm\n2o+BryilBpRSe4F7sSd4sFfDX1VKPaFsdiil9ox3sin+DN4APO/5fn4DODTB6/uxUup55zPMlnl9\nj1Lq+85YfgIsBOaLyDJsYfl5pVRGKfUQttY0Uf6slPqNUiqvlEoqpTYqpR51xrMb+C/G/n7fopR6\n3Pk9/ZzC5zORffV9/LXz2jeZ+H2cNRiBMfX0Ap0VbKYLndc1+/Q/Sqk8sB9bna6Ww57/k2WeN1Y6\nUERej73auVQplXS2zReRm0TkgIgMYavznZXOUcIi4KhSatizbQ/2Kl/j/aEkgEiF+7TIOdZL6bkq\n0YkteMtN9kXnde75PmCxUmoH8A/YK9Mjzn2YyGcxGUrvh/68llJ+/OMx1Z+B9/upvM+rZLz93bEo\npRLOv40UriPh2Xei7z3qGBFZJyK3OUEoQ8B/MPb3u9LnM5F9y93HY3bQzxRGYEw9fwbS2Cqoi4g0\nAq8H7vFsXup53QcswVbLwVZ5a4KInIC9ortMKeX9Uf2H874vVUo1A+/FNoloxhrTQaBdRJo825Zh\nm1YmykFsJ7KXas/Vi212WT3eeUVEsD+DAwBKqf9WSp3v7KOwAxdgcp9FHGjwPF8wgWP3UX78441l\nKj+Dboq/n+J9TnXXN9nvcDf2dXjPv7TSzmO8T+n272Jrqmud7/f/pfj7XQu6sX/XgHsfl1TefXZj\nBMYU45iHvgh8S0QuEZGgiKzAdjbuxzYDaM4Ukbc5K7x/wBY0jzqvHQamPM5bRJqB3wL/7Kj6Xpqw\nnYaDIrIYO8rLS8UxOYLnEeDLIhIRkVOBv8HWUibK7cA6EXmPiARE5F3Aydg2/zFxtIYfAteIyCIR\n8YvIuSISxv4M3igiF4lIEPg09j1/xMmjeI2zX4qCI1Vf94oJRvo8DbxNRBpEZA32vaiW64HPiMiZ\nToTRGhHRgm66PoPfA6d4vp9/T7FQeBp4lZMH0gL80yTeoyyO+W0DcKWIhETkXMY2Rx4GOpxxjEUT\ntj9hREROxPab1JrfAy8VkUud+/gxJrZ4mFUYgVEDlFJfxV69XI39BX0Me9V4kWNL1/wW266uHbxv\n89h6vwz8i4gMiMhnpnB4L8N28H7dST4aEZER57UvOq8PYn/Rf11y7HhjuhzbmXgQuAX4glLq7okO\nUCnVB7wJe0Lvw3ZOv0kp1TvmgQU+AzyHHV11FFtT8CmltmJrTd/C1kTeDLxZKZXB9l98xdl+CJhH\nYRL8pfO3T0SerHIMX8eOrjqMrc39vMrjUEr9EjvC6L+xHc+/wQ4EgOn7DHqxne9fwf4M1mJHPOnX\n78KO8HsW2EgVwnyC/C/gXOe9v+S8V7rcjkqpLcCNwE7nvlQyJX4GeA/2Pf2+c86a4rmPX8W+lpOx\nhWHZa5nt6IgEwzQjIldiR1m8d6bHYjDMdkTkF8AWpdQXZnosx4Kjpe4H/pdS6t6ZHs9EMRqGwWCY\ndYjIWSKyWkR8InIJ8FZsTeu4Q0ReJyKtjrlT+00eHeewWYnJfjQYDLORBdgm0Q7sFflHlVJPzeyQ\nJs252ObFEHa+lBuZeLxhTFIGg8FgqApjkjIYDAZDVdSVSaqzs1OtWLFipodhMBgMxw0bN27sVUp1\nVbNvXQmMFStWsGHDhpkehsFgMBw3iMi4ZWc0xiRlMBgMhqowAsNgMBgMVWEEhsFgMBiqwggMg8Fg\nMFRFTQWGU3xvq4jsEJF/LPP6Z0XkaeexSURyItJezbEGg8FgmF5qJjBExI/d+ez12AW3LheRk737\nKKWuUkqdrpQ6HbvQ2/1KqaPVHGswGAyG6aWWGsbZwA6l1E6nGuhN2PVgKnE5dsXJyRxrMBgMhhpT\nS4GxmOKOV/up0DHNaZRyCfCriR471dy/rYfth4fH39FgMBjmGLPF6f1m4GGl1NGJHigiHxGRDSKy\noaen55gGkbZyXHHDRr70+83HdB6DwWCoR2opMA5Q3FZxCZVbRb6bgjlqQscqpa5TSq1XSq3v6qoq\nu70iG/f0k8zmeHzXUdJWzt2ezeXHOMpgMBjmBrUUGE8Aa0VkpYiEsIXCraU7OW0VX43dfW5Cx041\nD223G7olszme3DNAPq/4zzu2cMoX7mTHEWOmMhgMc5ua1ZJSSlki8nHgTsAP/FAp9byIXOG8/j1n\n178E/qiUio93bK3Gqnlwey8nLWxm2+FhHtzew42P7+XWZw4C8Niuo6yZ11TrIRgMBsOspabFB5VS\ntwO3l2z7XsnzHwM/rubYWnI0nmHTwUE+efE6GkJ+fvjwLlLZPJ9+7Tque3AnLxwcmq6hGAwGw6xk\ntji9Z5ShVJav37UNpeD8tZ2cv6aTVDbPa0+ez8dfs4aTFzbzQrcRGAaDYW5TV+XNJ8NgMssFV91L\nfyLL285YzGlLWulqDNM9mOT/vuEkRISTFzVz0+P7yOUVfp/M9JANBoNhRpjzAqMlGuRjF67hnFUd\nvGRxCwBL2xv46jtOc/c5ZVELyexudvXGWTOvkd29cYZSWU5d0jpTwzYYDIZpx5ikgA+9cpUrLMpx\n8sJmAF7oHqJ7MMk7vvcIl1/3KP3xTNF+Nz+xj7/4+v3k8qZPusFgqD+MwKiCNfMaCfqFe7cc4aM/\ne5J4Okcim+P6h3YW7Xf35sNsOzzCjiMj7OmL8617tpM3wsNgMNQJRmBUQSjgY+28Jm556gCbDgxy\nzWWn8YaXLuTHD+/mqEfLeO7AIADP7BvgRw/v5mt3bWNHz8hMDdtgMBimFCMwquTzbz6Zf3vrKTzy\nT6/h9S9dyCcuWkvaynPZf/2Z7YeH6R1J0z2YAuCpfQM88qKdBLhxT/+UvP8D23p43w8eI5XNjb+z\nwWAw1AAjMKrknFUdvP/cFcxrigCwbn4TP/3rsxlIZHjP9Y+5gqE5EuC+rUfYdtjWLLwCo3ckzVAq\n6z63cnlueWp/URmSSnzznu08uL2XXz25fyovy2AwGKrGCIxj4Lw1nVxz2en0DKf5+l3bEIG3vWyJ\nq2ksaYvy5F5bYOTzind89xE+ffMz7vE3PbGPT/7iGW58bG/ReZVS/PTPu92qudsOD7NhTz8Bn3D9\ng7uMU91gMMwIRmAcI+ev6WRZewNbDg2zsjPG+Ws6AWhrCPLus5aysydOfzzDozv72N2X4P6tPYyk\nLTJWnu/e9yJgCw6lFFsPDZPPK57aN8Dnf/s8b//uI9y/rYcfPbybkN/HF958Mrt649z1wuGZvGSD\nwTBHMQLjGPH5hPe8fBkApy5u4bSldm7Geas7Wb+iHYCn9vVz84Z9+AQyuTz3b+3hV0/u58BAkktO\nWcCWQ8N89n+e5XXfeIDrHtzJbc90E/L7aI+F+MAPH+fGx/dyyUsWcPnZy1jcGuWGR3fP1OUaDIY5\nzJxP3JsK3nnmEq69dwfnre6kqynMp167jleu7eTEBc34fcL3H9jFk3v7effZy7hj0yH++/E9bOke\n5vSlrVz1zlO5f1sP/7NxPyG/j+sf3IXfB68+oYur33Ea9207wmAyy+tOWUDA7+NdZy3lmru2sbcv\nwbKOhpm+dIPBMIcwGsYU0NEY5ol/vph3rl8CwN9ftJYzlrURDfn5h4vW8sz+AdJWnsvPWsbFJ83j\n4R19pLI5rn7naTRFgnz0gtW87pT5/Nf7zqR3JM3hoTRvOnUhLQ1B3nr6Yt5/7grmN9vO9necuQSf\nwC837htrSAaDwTDlGA1jiogE/WW3/91Fa/nr81eyrz/BiQuaedOpi7h5w36+/PZTWTOvEbAFDNjO\n7tOWtLDl0DAXnTS/7PkWtUZ59boubt6wj09ctJaAf2yZ/8fnD7G5e5hPXLz2GK7OYDAYjIYxLcTC\nAU5cYJcXedW6Ljb+y8W85bRFo/YTEb7+rtO5/gPraQxXluXvPnsZh4fS3L35CMCoEiVebnpiH9++\nd7vJ3zAYDMeMERgzQEdjuOJrq7oaeeXasVvNXnzSfBa3RvnRw7v40cO7OPNLd3Hf1iNl993dGyeb\nU2w25dkNBsMxYgTGcYjfJ3zwvBU8tuso/3H7ZvIKvvi7F0YlAFq5PHuPJgB4dv/gqPNkrDzvvf4x\nrvnj1rLvM5K2+NJtLzCYyJZ93WAwzC2MwDhOueyspTSE/LREg3ztnaexqzfO+37wOB/6yRPsc4TE\ngYEklpPk98z+gVHn+Plje3hoRy/f/NMOrr13BxkrX/T6A9t6uP6hXfygpMiiwWCYmxiBcZzSEg3y\n4786mxs/fA5vP3MJl61fQvdgkod29PKZXz5DPq/Y1Wu3SW+PhXhmX7HAGExk+X/3bOcVazp446kL\nuerOrZz6xTv56Z93u/tsPWRnmt/w6B6SGeMDMRjmOkZgHMecvbKdtfObAPjqO07jwc+9hi++5RQe\n23WUGx7dw25HYLzp1IXsdJo+gR2N9YVbNzGYzPLPbziZ//eu0/mv953JvKYId2w65J5/2+FhwgEf\n/YmsqWFlMBiMwKg3Llu/lFes6eBbf9rOjp4RYiE/rzlxHkrBJseP8YOHdvGbpw/yyYvXcfKiZgJ+\nH687ZQHrl7e5WgnA1sPDvHpdFy9d3MKNj++t9JYGg2GOYARGnSEivO+cFfSOZLj16YOs6IxxulOu\nZMOefgaTWf7zji289uT5fPzCNUXHruyM0T2YIpnJkcrm2N0b58QFTVzykgU8f3CIvpH0TFySwWCY\nJZjEvTrkwhO7aI4EGEpZrOiM0doQ4pRFzTy8o5d185vI5hQffuUqfD4pOm5lVwyA3X1xlIK8gnUL\nmljcGgXgkRf7eHOZ/BGDwTA3MBpGHRIO+HnDSxcCsLLDFgLnr+nkyb39/PGFQ8RCfs5Y1jrquJWd\n9r67euNsc0qrnzC/iZcubqEpEuDhHb3TdAUGg2E2YgRGnXLpGYsB3PIjr1jTSTan+M1TBzh3dQfB\nMiVFVnQUBMbWw8ME/cKKzhgBv49zV3XwkBEYBsOcxgiMOuWcVR38/EMv542n2prGWSvaCfl95BUV\nM8lj4QDzm8O2wDg0zKrORlewvGJNJ/v7k+ztS0zbNRgMhtmFERh1zCvWdLoTfjTkZ/2KNgDOX9tZ\n8ZiVnTE27D7KQ9t7OWdVu7v9LKe3x9NlEgANBsPcwDi95xDvefky2hpCrHJ8FeVY2dnIozuP4vcJ\nH3rlKnf7wha7vLqJlDIY5i5GYMwh3nTqIt506thRTlqYvOnUhSxtLzRoaokG8fuE3goCYyCRYXP3\nMOeu7pi6ARsMhlmFMUkZinjZ8jaawgH+9oLiHA2fT2iPhegbGV1KPZdXfOSnG3nvDx4zJUQMhjrG\naBiGIs5c3sazV/4FIjLqtc7GML1lBMZ37t3B47uPAnBgIMGaeU01H6fBYJh+jIZhGEU5YQHQ2Rgq\nMkkppbj23h1cc/c2TnBqWu3rT07LGA0Gw/RjBIahajpiIfriBYHxg4d2cdWdW3nLaYu47v1nArDf\nCAyDoW4xJilD1XQ2hukdLpikHt91lFVdMb7xrtNRCkJ+H/v7TZ6GwVCvGA3DUDUdjWGS2RyJjAXA\nzt44a7oaERF8PmFxW9RoGAZDHWMEhqFqOhpDAPSNZLByefb0xVnV1ei+vsQIDIOhrjECw1A1XY1h\nAHpG0hwYSJLNKVZ1FZIAl7RFOWBMUgZD3WJ8GIaq8WoYgwm7e583a3xJWwO9IxlS2RyRoH9Gxmgw\nGGqH0TAMVdPpaBh9I2l2Op35Sk1SYCKlDIZ6paYCQ0QuEZGtIrJDRP6xwj4XiMjTIvK8iNzv2b5b\nRJ5zXttQy3EaqqM9ZmsYvSNpdvaM0BIN0tYQdF/XjZZMpJTBUJ/UzCQlIn7gWuC1wH7gCRG5VSn1\ngmefVuA7wCVKqb0iMq/kNBcqpUwThllCJOinKRygdyTDzp44q7piRUl+S9rs2lNGwzAY6pNaahhn\nAzuUUjuVUhngJuCtJfu8B/i1UmovgFLqSA3HY5gCOpvC9I6k2dUbdzv0aeY1hQn5fex2zFUGg6G+\nqKXAWAzs8zzf72zzsg5oE5H7RGSjiLzf85oC7na2f6SG4zRMgI5YiE0HBjk0lGK1x38BdoHCM5a1\n8vCLfTM0OoPBUEtm2ukdAM4E3gi8DvhXEVnnvHa+Uup04PXAx0TkVeVOICIfEZENIrKhp6dnWgY9\nl+loDLG7L8GilghvdPqGe7nghHls7h7i8FCKT/7iaW58fO8MjNJgMNSCWgqMA8BSz/MlzjYv+4E7\nlVJxx1fxAHAagFLqgPP3CHALtolrFEqp65RS65VS67u6yrceNUwdHzhvBZ+8eB1//NSrWVGmEdMF\nJ9ifwb//fjO3PHWAe7cYK6PBUC/UUmA8AawVkZUiEgLeDdxass9vgfNFJCAiDcDLgc0iEhORJgAR\niQF/AWyq4VgNVXLe6k4+cfFaGsPl4yVOXNDEguYItz5zEID+xOhy6AaD4fikZgJDKWUBHwfuBDYD\nNyulnheRK0TkCmefzcAdwLPA48D1SqlNwHzgIRF5xtn+e6XUHbUaq2HqEBFXywj5ffTFjcAwGOqF\nmmZ6K6VuB24v2fa9kudXAVeVbNuJY5oyHH9cfvYyjsYzNEWC3LPl8EwPx2AwTBEz7fQ21CGnLW3l\nuvevZ0lblIFEFiuXn+khGQyGKcAIDEPN0LWn+p26UwaD4fjGCAxDzdClRI4aP4bBUBcYgWGoGVpg\neNu6GgyG4xcjMAw1oyNmV7c1GobBUB8YgWGoGcYkZTDUF0ZgGGqGLn3eN2IEhmFusO9ogiNDqZke\nRs0wAsNQMwJ+H60NQVfD2HJoiBP+5Q+82DMywyMzGGrDp25+mi/+7oXxdzxOMQLDUFPaG0KuwNi4\np5+0leeZfQMzPCqDoTYMJS0GkvWrURuBYagp7bGQGyW1s8fuk2H6ZRjqlWw+Typbv4mqRmAYakp7\nrKBhaFPUrj7TwtVQn1g5RdrKzfQwaoYRGIaa0tFYEBhGwzDUO7m8qmsNo6bFBw2G9liI/kSWVDbH\nvv4EIrbAUEoV9QM3GOqBbC6P1HElHKNhGGpKeyxMLq94et8ASsFpS1oZTlum7LmhLrHyirQ1xzUM\nETkPWOHdXyn10xqNyVBHdDXZ2d53vWCXOb/oxHk8vW+APX1xOhvDMzk0g2HKyebyZOvXhTG+hiEi\nNwBXA+cDZzmP9TUel6FOuOCELlqiQX78yG4ALjxxHgC7eo3j21B/5IyGwXrgZKWUqvVgDPVHcyTI\nR161iqvu3MrClgjr5jfh94lxfBvqEiunyOTy5PMKn6/+fHTV+DA2AQtqPRBD/fKB81bQ1hBk7fwm\nQgEfi1uj7OozAsNQf2TztnaRqdOmYdVoGJ3ACyLyOODWqVZKvaVmozLUFY3hADd+5ByiQT8A6+Y3\n8vTeARMpZagrcnmFtsOksjkizve9nqhGYFxZ60EY6p8TFzS7/1/ykoXcvfkIT+7t58zl7ZM636YD\ng8TCAVZ2xqZqiAbDMWHlC1pFvfoxxjVJKaXuB7YATc5js7PNYJgUrztlPpGgj1ueOjDpc3z65me4\n+s6tUzgqg+HYsHIFN2+qTkOlqomSugx4HHgncBnwmIi8o9YDM9QvTZEgF580n98/2012krbeg4NJ\nEhlrikdmMEwer8CYsxoG8M/AWUqpDyil3g+cDfxrbYdlqHcuPX0x/Yksj+08OuFjU9kcwykLK28C\n9wyzB69Jas5qGIBPKXXE87yvyuMMhoqcsKAJgIMDyQkf2zNsx15MVjsxGGqBdwFTr/WkqnF63yEi\ndwI3Os/fBdxeuyEZ5gIdjXb71smUCDkybHc085oADIaZxruAqdeKteMKDKXUZ0Xk7cArnE3XKaVu\nqe2wDPVOQyhAJOjjaDw9/s4lHBlyNAxjkjLMIoqd3nNXw0Ap9SvgVzUei2GO0RELT1LDsAWGZUxS\nhlmE1yQ15zQMEXlIKXW+iAwD3qWcAEop1VzhUIOhKjoaQ/SNTFxg9LgCw2gYhtlDsdO7PhczFQWG\nUup852/T9A3HMJdoj40vMO7beoQTFjSxsCXqbtM+jGy+Pn+UhuOT4rDa+tQwqq1WO+42g2GieNu3\nlkMpxf++YSM/enh30fYjJkrKMAvxfh/rVcOoJjz2FO8TEQkAZ9ZmOIa5REcsRN8YTu+0lSdt5ekv\nESrGJGWYjeTmgA+josAQkX9y/BenisiQ8xgGDgO/nbYRGuqW9liYVDZfMWM7nra3D6WKe14WNIzq\nBMYfnuseU5MxGKaC7ByIkqooMJRSX3b8F1cppZqdR5NSqkMp9U/TOEZDneLmYjh+DKUUGU9JhXja\nXqUNJQsCJZdX9I04GkYVPoy+kTQf/fmT/PrJ/VM27nrgj88fYnP30EwPo64oLj44xzQMD4+LSIt+\nIiKtInJpDcdkmCN0xIqT9755zw4uvPo+RhzNYqSMhtEXT5NXEAr4qjJJHRqyHeRa+BhsPv/b5/n+\nAztnehh1RZHTe65pGB6+oJQa1E+UUgPAF2o3JMNcod0RGDp57+l9/RwYSPK9+14EIO6YqgaTBYGh\nk/YWtkSqcnpr81WyTmv7TJZ4xmIoZYo3TiVzIQ+jqlpSZbZVlfBnMIxFRywMFExSe/rsPt/ff3An\nBwaSBR+GR2Boh/eilmhVxQePOBpGvRaDmyzpbJ6RdHb8HQ1VY5koKQA2iMg1IrLaeVwDbKz1wAz1\nT3uj1jAyWLk8+/oTvOW0RaStPLc9c9A1Iw2nLfKOcOh1/BcLWiJOh7OxhcZhRyNJZozA0Fi5PJlc\nnmGjYXDrMwe5+4XDU3IuXapGpH4XKNUIjL8DMsAvnEca+FgtB2WYG8RCfkIBH33xDN2DKbI5xXmr\nOwj5fRxNZFwNQykYKTFPdTrCZrxIqcOOhmFMUgVSTmCB9hHNZa574EV++PCuKTmX1jAaQ4G67YdR\nTfHBOPCP0zAWwxxDROh0sr1398UBWNEZozkaZChpFU1oQ8kszZEgg8ksItDm+D+sfJ7QGOseV8Pw\nCIyBRIYNu/u5+OT5tbisWY9e/Y4YDYNUNk/ANzWLCW0ijYUDc1fDEJF1InKdiPxRRP6kH9MxOEP9\n094Y4mg8zW7Hf7GiI0ZLNMBQMutqGFDQLAYdwRHy21/d8TSMnuHRPoxfPXmAD9+wgeHU3LTha/Oc\nMUnZ92KqzJU6SqoxUr8aRjUmqV8CTwH/AnzW8xgXEblERLaKyA4RKauliMgFIvK0iDwvIvdP5FjD\n8U97LMzReIa9fXEiQR/zmsK0RG1NYiTj1TAKJqmWaJCgIzDGq1hbzoeRSFsoNXfNVFp4ZnL5uo3m\nqZa0lSORnRrBqfMw6lnDqCbayVJKfXeiJxYRP3At8FpgP/CEiNyqlHrBs08r8B3gEqXUXhGZV+2x\nhvpgZUeyZ0efAAAgAElEQVQDN+3sI+D3sbw9hs8nNEeDHI1nijQMnYuhBUbALwBjRkrl8oqekdEm\nKb36q9dY+fHwRvCMpCzCjf4ZHM3MksrmGSduompcDSPs5/BQfWpv1WgYvxORvxWRhSLSrh9VHHc2\nsEMptVMplQFuAt5ass97gF8rpfYCeFrBVnOsoQ5437nLSVt5Nu7pZ3lHA4CrYSTSOXy2XHBDa10N\nw6dNUpUn/b542q3v4xUYGeeY6V5dz5b+Hd57MdfNUqlsjsRUmaS0hhGqXw2jGoHxAWwT1CPY4bQb\ngQ1VHLcY2Od5vt/Z5mUd0CYi94nIRhF5/wSOBUBEPiIiG0RkQ09PTxXDMswm1sxr4rWO83lFZwwo\nCIyRtMX85gjg8WEkSjSMMXwYOsmvKRIg5ZkU0s6PeTpj5Xf1xjnp83ew/fDwtL1nJbyT2XRGSvXH\nM27U2mwgm8tj5RXJbM4N2z628xkfBkqplWUeq6bo/XXl2zcCrwP+VUTWTeQESqnrlFLrlVLru7q6\npmhYhunkilevBmDNvEYAmiNBhhyBsaDFFhg6K3kwmaWlIUhA+zDGqCel+2as6IjNuIZxoD9JNqfY\n2RuftvesxExpGF/6/Wau+NnsSeHyCs6p8GcVTFL1q2GM68PwrPqLUEr9dJxDDwBLPc+XONu87Af6\nnNDduIg8AJzmbB/vWEOdcObyNm77u/NZN9/u1dUSDZJXdg7F4rYGmiJ21JRSymOSsjWMsaKktMN7\neUcD248UVvYz4cPQwmk2hLKmigTG9EWK9Scyk+qwWCu8GmY8YxELH1sBi1w+jwhEQ/65q2EAZ3ke\nrwSuBN5SxXFPAGtFZKWIhIB3A7eW7PNb4HwRCYhIA/ByYHOVxxrqiJcsbiEUsL+OLdEgAN2DKRrD\nflvjSGVJZHJYeeWYpMb3YWjzx9L2BlLZvGt2cAXGNP6oM7MoWW6mTFLZXH5WrbyLNIwp8GNk84qA\nT4gE/GSs/JSYuWYb1STu/Z33uRPZdFMVx1ki8nHgTsAP/FAp9byIXOG8/j2l1GYRuQN4FsgD1yul\nNjnvM+rYiV2a4Xil2REYiUyOhlDASeTLun4Mrw9DaxjxtMW3793BFa9e7Qqcw0MpOmIhmiP287SV\nJxryu5P3dE5e2gw2G3I/vJPjdAqMjJWfVaHMXpPkVDi+rVyegM9HJOh3zm9/3+qJyehgcWBlNTsq\npW4Hbi/Z9r2S51cBV1VzrGFu0BwtfC0bwwGaIwGGkhYDiYLA0FFSOvLo+gd38d37XuTkhc28+bRF\ngF3McGl7A9GgvW8ymysyFxyLhvHc/kEefrHX9b+U8o27t7GyM8ZbT7djNbT5a3gWaBhJjylmOn0Y\n2Vx+VoUye01SlZp4TYRsThHwC2FHU05buboTGNVkev9ORG51HrcBW4Fbaj80w1xFawgAsbCflqht\nkiqnYVh526/xg4fs3g7dg0n32D19CZZ3NLg/Wr26zVg6Smryq8rfPn2Ar/xhC3udDPVSbn5iH3c+\nf8h9nnYE22zyYQT9Ms0CQ5HJ5Ytamc4kXm1nKjSMnDZJORpGPVasrUbDuNrzvwXsUUqZ9mWGmlEs\nMMqbpLR2kM3l+fHDuxlKWQR8Qveg7bfIWHm6B5Msb1/s/oC1KWYqNAx97J3PH+LDrxodNDiStopW\n0zqUdzbkPaSyOcIBH43hwLSWONf+plQ2d8wO5qnAu2CYigZbVj5PwO8r0jDqjbF6ep8DoJS63/N4\n2AgLQ61p9ggM2yRl52UMeQRG0JOH8fCLvZy+tJUVnTG6B2yBsb8/QV7Bso4YUXfFpzWMYw+r1ef6\n4wuHRr2mlLIFhkcgaR/GbHB6a9NcYyQwrQJM34PZ4sfwagDJKSgPks0pgnWuYYxlkvqO/kdE/jwN\nYzEYALs8tM7wbggFaIkGiWdybivXloYgAV8hDyOdzdESDbKwJUK3Exm156htKipnkkq7Tu9j1zA2\n7Ol3mzppUtk8eVW8gnWjpFIWVi7PNXdtYyAxMyGmqWyOSMBPUyQwrSYyr4YxG0hNsUnKys1hDQMQ\nz/+RWg/EYNDoelJg1+Vpj9n/bzs8jE9sgRL0REmlrTzhgM8WGAO2D0P7Fpa3N7gahjZJTYWGkbZy\nRII+lIJ7Nhc34NFahFfD0P8PpbJs7h7mm/ds5+7NR5gJklk7eqcxHJhWJ3zWsn0Xs1JgTIlJqtiH\nUY9Nu8YSGD4RaRORDs//E6klZTBMGu3HiIUDrF9hf93u3nyY5mgQn0+KMr1T2RyRoJ8FLVF6RtJk\nc3n29CWIBv10NYU9JgKtYRRKgyQyFnds6p7w+NJWnnXzmwj5fW5pdk1BYJTRMNIWvU4P8/74zGgY\nyYx9vxrDwWmPkoLZY6qZeg3DjpKKhf1Tds7ZxlgCo4VC3ahm4EkmVkvKYJg0OnciFg5w4oImOhvD\nDKcsV5AEfKM1jEUtEZSTIb73aJzlHQ2ISJkoqYKGccemQ1zxsyfZ2TMyofF5zTqluRW6yq53YvQK\nDJ3tfHSGTFJpK0c06KM5Mr1O72PxYWw7PMwrvvKnUea/Y2Gqw2qtvJ2H0RSxHfqzIYR6qqkoMJRS\nK5RSq2pcS8pgKEuLa5IKICKcv6YDgFZne6EfhiMwgj637tShwRR7+hIsa7er35Y6vb2lQXRux4GB\nQjhuNej3bCrjOC6nYej/h1OW25d8xjWM48iHsbl7iAMDSfb0TV0tLj2OpkhgSrSBbE4R9IsbATYb\nQqinmmpKgxgM047XJAXwyrV2YUnt2yjkYeTd1f6i1ihgT/57jybccumVfBgpK+9qAzq6qlrSWVur\naSyjYeiJoihKyvk/l1cc6LeF09GZEhjZHNGg48NIWfzNj5/gqju3TOl7WLk8dz5/COVpNqGz8idj\n29fFJ6cyyixl5Qj4hOZIkPgUaBi5vMLvExqd72x8LmkYBsNMorO9Y4456fy1nUBBkBT6YYzWMB7d\neZS0lWdZh10u3XVCZu2kMd10KZ3NEXcmr4ODE9UwcoSDfprCwVGTmJ58iqKkPDWvdP/ymRIYqWyO\nSMhPUySIlVfcs+UIzx0YmtL3eOTFPv73DRvZsKcfsCdTnbCXmkT+iw6pnkq/QDKTJxL00xDyT00t\nKSdKKhaagyYpg2EmaS7RMOY3R/jLMxbzSkdwaA0jlc2RyysiAbtIYWM4wM0b9hH0CxedOA/ADXNM\nZnPuSh9sDWCyGkbK0TDGNknl3RW2N4lvj+MknykfRiqbJxKwTVKFbVProNUdEl88YvuGvEUiJ/Ne\n+h5P5ao95US6NYSnxiRl5W2TlM8nxEL+utQwqilvvhrYr5RKi8gFwKnAT5VSA7UenGHucunpi2mN\nhlxfBcDX33W6+78WGHpyDjv1oha0RNhxZIR3nrXUNVH5fEIk6COVzRX5FVLZnPujnriGkScc8ENk\ndPa2PqdStgYUCkiRhrG/3xYYM+XDSGVzREM+uhpDgG3DT0+xwNAr9l1O/49jFxhTr2HYGe9+GoJ+\nEhmLnzyym5G0xccuXDOp81m5PAFngTPd/qHpohoN41dATkTWANdh96n475qOyjDnOWlhMx+9oHxh\nPyiYpPTkHA7YZqeFLRF8wqiigNGgbXYo1TC0wNElRUr54UO7+PLtm0dtT1t2eQ1det2Ld6JIOQIq\nnc3jdyK7dCmlgWR2RuoqaR/GRSfN55a/PY/zVndMefa1FgoFgaFGvTYRhkt8GFsODRX5RyZDOpu3\nNYyQn0Qmx80b9nH7cxMPsdboPAywNeORKfCLzDaqERh5pZQF/CXwLaXUZ4GFtR2WwTA2Pp/gk8IE\nEnE0jL96xQq++JZT3HavmmjQTzKbK3JEp7I519/QPZAsOwHd9uxBfrFh36jXvFFSI2mr6PURTxKY\nNkWlc3naY6Gi8SjFtGd7K2W3JI0E/QT9Ps5Y1kYk6J/y3IjkKIHhKcORmYQPw9UwLJ4/OMgl33jQ\n9Y9MlpRTIkWbpHb3xsfsrzIeOg8DoCk8dzWMrIhcjt3b+zZnW3CM/Q2GaSHg97k/Sq1hvObE+bzv\n3BWj9o2EigWGiB25pIvOxTM5NxLHy/7+JAOJrFuWBOxJN2Pl3TwMpXCd51BsZ9cmsIyVp8MjMNbO\nt9vR9k+zwMjk8ihVCAQAiAT8U+7D0EJhz9EEubwq0uxSk8iwL/gwchxytMF9R8tXCq56jE50XUPQ\nz4H+JPESDXSiZJ3ig+CYpOrQh1GNwPgr4Fzg35VSu0RkJXBDbYdlMIxPyO8bpWFUIhr0k/JMCI3h\ngOv01qai7hI/Riqb44iTKLb9cCGxTwudcNBHY9heO3lDa70ThV65p60cHY0FgaH7lx+NT29DpZQz\nkUc9AiMaqoHA8CRJHhxIlmgYx+LDsFzhcaxRZrpCQEPY7/qYxmr5Ox65vHJbB8dCgbp0eo8rMJRS\nLyil/l4pdaOItAFNSqn/nIaxGQxjEvBLwekdGLtRTcEkZU9WzZGg6/TW+RqlkVIHPcl8OzyZ4NrM\nFHY0DCj2W4xU0DDaY2F3e0FgTK+GoSdyb2OfcNA3qVDXsfAKoF298aKJeDI1vIaSBQ1Dm6d6j7E/\neMrjw9BkjtEk5fcVNIzZUMp+qqmmgdJ9ItLs1I96Evi+iFxT+6EZDGMT8PlGRUlVIuqYpLSG0ez0\n1BhJW6x1Ju/SSKn9/YXnOjwUChOeDqsFisxZxSYp+/0yVp5YyO9qQmvnNQHTLzD0RO7VyGrRgzqZ\nyRFyzDO7++JTpmHEizSMYysTknJyaRpChWDRsUxStz17kCNDlcOvs7m8WxSzMRyYkmTA2UY1JqkW\npdQQ8DbscNqXAxfXdlgGw/gE/TIqSqoSESdKSk/gTZEAqWyORCbHis4Yfp+4tnHNPif8dV5TmO1H\nht3t+hyRYEHDKDVJ6QRDbzmSUKBgwtJCarp9GK6G4fVh6NIpU1iOO5nNsag1QizkZ2dPvGjlPlEH\ney6vXB9RIp1zk/iOVdimnXwUr4ZRyemdzOT4+H8/xS83Vm4HZOULTu9Gx+l9rJFcs41qBEZARBYC\nl1FwehsMM07ALx6ndxU+DK+G4WQ5W3lFcyTIvKYwB0tMUvv7kwT9wivWdLLDo2FoIWBrGNqHUWyS\n0g5ur4Zhh+HaAmahM5nOlEmqyOntaBtTGSllN2kKsLwjZmsYnpV7MptjJG1xeIzVuhevuS+esVxt\nzmuSuu6BF3nv9Y9NYowlJqkKGsawU6RxLO3IyuXdPi2NkQBWXh1TV8fZSDUC49+AO4EXlVJPiMgq\nYHtth2UwjE/Q53NXnt4JsBylYbW69AjY5UdaosFRNaH29ydZ1Bpl7fxGDg+lXdu56/T2mKS8fot4\n2nId3Nrfkck5GkYkQFMkQDjgpy0WmvbkvVQZgVFanHGyHBxI8q+/2UQ2lyeZsSvitsdCDCazrg/D\nJ/b7fO2PW7n8ukerOq83zyWettznXmH7zP5BHtrRO6Gqw7oGmdckZeVVWdOc7pcxlv/Fm4eh60nV\nW6RUNU7vXyqlTlVKfdR5vlMp9fbaD81gGBut/kMVGoZTLyiTKzi9NbFwgFDAN8rhub8/wZK2KGu6\nbPOR1jJcH0bQ79Ewik1SHY6DO23lsHJ2DatwwC7419lov9YeC017eZDUWCapKgRG30i6oq/jgW09\n3PDoHnb3xt02sDrHQ5t6mqN2sMGevoQbgTYeWkC0x0J2+LNjkuobKRyvTZP3VNmUSilViJJyNIzF\nTmWAco5vPfGP5eOw8zAcDWOMirWprP2dOB6pxum9RERuEZEjzuNXIrJkOgZnMIyFVv9hfA0jojWM\nbGHi0jSGA4T8vlGTwf7+JEtaG1g733ZQbz9s+zH0OSIBHw1BPyIFk1TaypHNKdo9GoaegEIBH5e8\nZAGXnr4YcATGdJukdFhtaGImKaUUP3xoF2f/xz38YsO+svvoSXU4bTkaht8tyaLvge07ytM7kiaR\nqc7Gr+/t/OYIiXTB6R3P5FwhpwXGXSXdDyuRzSnyyr72+c120crTl7Y6r42+D/r8Y5mYsvmC0ztW\nRsOIpy3O+Y97OPFf7+Ct1z5c1ThnG9WYpH4E3Aosch6/c7YZDDNK0KthjBMl1RINks0pBpzVabOn\n8F6D1jBKssB7htMsaYuyrL2BhpCfzd2OwHDzMPz4nHLW3sQywOPDKAipkN/H+89dwScuXgtAW0No\n2p3erkkq4A2rLW4wVY6f/nkP/3bbC+Tyit4KmoG+B8Mpy129a9+Rq2FEgqQs+97m1dgTcOl5F7ZE\nSGQLGgbgJlTq7PoNu49WZebTDv5I0M9LFrdw/2cv4OWr7M6O5bQIXcOq0njzeYVSuDk9TWUERvdg\nikNDKZrCATcD/nijGoHRpZT6kVLKch4/BrpqPC6DYVwCnsKE45mk2hpsjUJHQnlNUo1h/yiTlG6o\ntKQ9it8nnLCgiRe67RLgXqe3Ppdb68j5qwVGyqNhlAq1iZbVTmZy/PyxPccUeZOsEFYLjFmA8Nn9\ngyxotut0VZo09eQ4krLcelVas8t6NIxEJuc2kaqmmKAWEAs8HRW1We+o4/iOpy1WdcXIK7h/W8+4\n50yV+L6Wd8TcQpflkvfKNcXyks3b1xf0ZHpDicPeOcfitiipbO64jKCqRmD0ich7RcTvPN4L9NV6\nYAbDeGgHowhuzH8l2pwJXEfmFDm9y5ik9Cq6q9E2V5y0sJnN3XbBO6/TG3A0DHtS0xNLR2PBh6HP\nWzrGidZwumfLYf75lk1sOTQ8/s4V8GpHhXE4JqkxHLoj6SytDUHCAX/FSbNwD7IkszkaQn43izxr\n2ZNjcyRI30janZSraY2qz7vAMR3FMzlWdtrJlro/ejxtcfaKdvw+KYpoq4S+715Tpv58ymkY8XF8\nGJZzPd7ig/ZYRyd0djWFyatjyyqfKaoRGH+NHVJ7COgG3gF8sIZjMhiqQq/mwgEfIjLmvm0NtsA4\n5AgMnQ8BdhmHUpNUwlltN4TtCeWkhc0MpywODCTdCVNPNt6eGHqCaG0IuqtxvX+oRAvSkVvVrjSn\noieEN+nQHUdIO70rC6/hlEVjOEA46BtXwxhO2T6MiMfpnXY1jCBen3k1Goa+bi0wAFY4zbG0hqFz\nXxa2RNz8mbEomKQK9yHo3JOxnN6Vrt0VGM53UpukSsOtwaN9TmHey3RRTZTUHqXUW5RSXUqpeUqp\nSwETJWWYcXSU1HgOb4D2mC0gDg+mCAV8REOFr76OkvJOBtpUpKOJTl7YDMALB4dGaRhNnkJz2gQR\nC9uhsylPKG9pcmE05CeXV1WvNLWgiB9DT4hy2o42SY0VJTWStmiKBMYsVKgnx6FklrSVd53eULgv\nXs0OqhQYaYtI0EdLQ0HI62rEffE0Vi5P2soTCwdY2tZQVVHCcr6ckGuSGsOHUUGoWo5Jyg2rjYxu\n06rvgTanTXX9rulgsh33PjWlozAYJoGOkhrPfwHQ6mgYR4bThAO+osk7FvYTLvFh6AlCh1yeuKAJ\nEdjcPVzwYbgaRnCUSaopHCDirMYzJQJGE6nC2exFO9QTx6BhZKy82xWudBxjaRgjKYvGSLAqDaPH\nWfVHHac3FEJjmyLFha6ruZahZJamSNBtfQqwqDVCyO+jL55x70ssHGBpe5R9/eM3w0qWyd8JBex7\nMpZJqpI5Trf91YuYaNBfVH4fCtpnZ5NjrpyAOXJ/FVrTdDBZgTG2/m8wTAP6Bz5eWRCAVieM1sor\nR2DYX/2gXwgH/ISdekqapPPj1uaaWDjA8vYGNncPeYoPji40p80hLdren82P0kg0E02Y0/b+Y9Ew\n3E6BHrQWMJbgGtImqYCv4kSnV9A9w7bZT+dhQMEP4Y1Og+pNUs2RgGsetM8TpKMxRN9Ixm1U1Bj2\ns7StgZ7h9Lj3VBdbLDJJjaFhjGeS0sfoxl4iYjdRSo82SU1Uw3hqbz/n/+e9bDs8ed/VVDFZgXH8\neWsMdYfWMMYrbQ62bVlPViG/z53ItHPSNkkVfsB68vRmAZ+0sJkth2yTlE8K5gftw1BKccuTBzhj\nWSvzmiJOFViP07tUYDhmsWojpfQKtRpHcSUyTk0rL9Uk7o2ks26GemWntxYYafe8robhVJttKhUY\nTk/2sTKih1K2hqGT4ezzBN08Fr36bwgFWNpuO8PHW5GXy3gfy+mtBdu4Tm9PqHdjSROlkZRdSr9Q\nZ6w6DePwkH0/Z4OWUfGXJiLDIjJU5jGMnY9hMMwo+sdZjYYBuB3vwkG/u9rXZo7SKKlEiQ8D7Ezg\nQ0MpN8dAO9pboyEyuTy/3LCf7UdGuGz9UsC2j6ez+YoCQ9vPJ2qSiqePzYdRGq2l70WlsNpsLk8q\nm3fNbJUmOtck5QgMrw9jKJUl6Beizv3WOTSJtMUPHtrJBVfdW1FgDaVs/4m35lNzNEBHY5i+kbT7\nvo2OSQpg39GxzVKuj8pzTq/Te3P3EPduLWSNj+v0dkxSfl+JwChJ3Gt07iFU7/TWArp/mnunlKOi\nwFBKNSmlmss8mpRSgUrHGQzTRXACGgYU/Bghv8/1P8QcM0co4COvcEs2JDN2z27vBNDZFCaVzXM0\nkSkyL/3lGYvpagrzuV89SyTo402n2h2MbXt/Zad3JDQxk5ReSU9Uw9hyaIhv/8ku/5a2cqPyQUTE\nFgQVJkP9vo1jaBh5j5bQM+IVGNokZRH0+4g4921pW4NzLTl29yXoHclw75byZT2Gk1mao8U+jKZI\nkI5YyPFhFAINlrRVp2HoBYH3nF4N47/uf5F/uWXTqHtQ2YdRnIcBo7vuDbsCY2Kfu95vupM8yzFZ\nk5TBMONMXsMo+DC8JikohFQmMrmiFS0UbM8H+pNF77mgJcL1719POODjzacuch274YDPSdyrHFYL\nY2sYj+zo5TVX30cyk3MnuYlqGL9+8gBX/3Eb2ZydRFguZyUSHD/6yfVhlHMKe4SYjvoq9WEE/T53\nRb+sQwsMi0EnMe+2Z7vLvn/PcJquxnCJDyNgC4wRr8Dw09UYJhTwjev4TpT4qKDw+WRzdjl1b791\nV2BUipIqycMAiioA6HM0hgOeqLTCuT5189P8aUv5siZ6v8HkzGsYRlMwHLd48zCqodUJywz5fWVN\nUmCvLhtCWmAU/zw6nfpQ+/uTo1bppy1t5f7PXui+B9iTcDxtFUqDTMLp/ciLfezsjdM9mHRXqxPV\nMLSJKOXU0iodB9jmsUq+FD3pNUUq52HosfkEN88iEvS7E6irYTjXvLAlStAvJDzFBO/Zcph42nKF\nuL7W4bTFvOYw4YCfoF+w8opYKEB7Y4hkNudGZTWGA/h8wpK26LihtaVRcOD5DuTsGlXxjF0kMOAv\nVEUud+3/9rsXXM3Dq2E0R4NuxQCwBX0s7B8VZJDLK3795AFaokFec+L8Uec3GobBMAXoyaiaPAwo\nJO+Fg3aiXzjgKzJJQcGpmcxaRatPKGgYh4ZSRfH7mgUtkaKx6NW4WxpklNPb0TAylZ2fe52JbzCZ\nnXSUlBYYeizlBOxYJqmCjyBYMQ9DO3fnNRWS66JBv3uNQ8ksIb+4QrKrKUw06CeRyTGYtLPIU9k8\n95SYpY44Dl993oZQgCZHMHQ6FYG1cNCCZmlbw7jJewmnI6B3gtc+jKyl3GscKkmWzOTyoxItb33m\nAL94wi7I6DVhNkeCrrMfbJNUrIxJSmtnlQS21jD6EzOvYRiBYThuCUxQw9AmqZDnuFKTlF5BljNJ\ndTnx87m8GrfYoX3+4qZN1Tq942mLFw7adav2eATGeHkYw6msKxy8FGkYZaKkYGyT1Eha51BU1jD0\nxLqw1SMwQn73GuOZHMGAz11ddzWFaQgFSGQshpJZzl/TSUPIz9N7B4rOq0ugz2+2730sVCgprz/P\nPX12IT8dRbW0PcrevvEEhlVk4gKvhpF3PxOt/RT3aS++/uGUNSoPA+yCl0PJrCtg4k7yo/7u6CCD\ngm+qgsBwtJdBIzAMhsmjI23CVWoY2lyk/Q8XnDCPl69sd7aN9mGUai56gvLuPxZ6cq2UhxEJFZsm\nND/5827eeu1DDCaz7HUmw8Fk1l2JVuoV/ZU/bOGDP3ocsIWaFgDaCZ1yckJCZbSjqnwY2unt7Pf7\nZ7vd1b2eUBe2FGsYEU9GfdDvY3lHjHeftZQLT+iiIVzQMNoaQixsidDt9FXXuRRHnJwOrWHEwgG3\nNL1uUrX3aBK/T9z7u7KzkaGUNWbp+EQmR0OwgsCw8u6qfihlT/gJp1w7FAuMjOfz1deoaY4GyOQK\nr4+kLGIhr4ZR2A7F34Nn9g3wmqvvYyiVdU2axiRlMBwDE8n0BmjXUVLO/t+8/AzeddYye1tJDH6y\njIYR9PtGCZ2xsCOKChNGqbPZnYBKJurdvXGyOcWfX+xzzRBDyazb9c27En10Z59bnfXgQJL9jrP3\nu/ft4JJvPEA2l3cnTl0IsZJJqpJD1/VheGpJKaX4xE1P8Z37XgQKk96C5mjR9XmFbtAxAX3l7aey\npM0uGZ/I5BhKWTRHAyxsiXLQqSZ86bUP87U/bnVzEOY52l0sHHDzaTo8JqlYqBDmvLrLLhvy4hjd\n92wNo9hHFXQSQXXHQLAFddqyG2C1e0rWa0rzR7wmKZ1voZ3V8bRFYyQwqhTLcHq0SWrjnn529sY5\nOJB0NYyBetcwROQSEdkqIjtE5B/LvH6BiAyKyNPO4/Oe13aLyHPO9g21HKfh+MSNkppoWG2ZCXO0\nD2O0wICCH6OaUF47Sirn5j6UFkh0S4OUmCK0o/T25wpRQ70jGVf78dYn+vafdvDVO7YAtlloKJUl\nn1fsODLC7r5E0aSZyubJWLnKJikrx7X37uCau7YVvTYyKqzWNtlYecWmA4POPvZktshjkoqEfEV5\nLCF/8fU3hAL0DKfJ5ZVbOLB7IMlgMsuBgSTP7h/kyHCKkEdQf+51J/Cp164DChrGiBN9pFntdEh8\ncYyqteVMjsUahjZJWe71a4Hhzdcpbesb9DT10iX0tVlqJGOPM+gX/D5xBYEWtt5ghsOOZpXwNIka\nmFsbs+oAACAASURBVAUaRs2ipETED1wLvBbYDzwhIrcqpV4o2fVBpdSbKpzmQqVUb63GaDi+0Sap\ncg7ocrhhtWMJDE8eRjQ4+ufR2Rhix5HqNIxIUGsYubLvaa+4ZZRJ6uCAPVnc4+kep001UKxhDKcL\nYan2xGRrBLpR1Ibd/e6+aceHES4XVutESd369EH64hk+efFaV8DpDOWoJ+FRaz5bDw2TsfKFirKO\nScrnlJwXsSfHXF4VmWvAjlDSPbhbokEWtkbpGUm7Qm7HkREWtUbpagq7YzlvTWfR8TqwwBtZtbg1\nSjjgG1vDSI8WGH6fIKITFQsahhbQba6G4RUYlvueBwaSo3wY+hyJTA6lbD+LiBAJFBIgR8r4MLSz\nP5kpNOCKZ3JlM/Wnk1q+89nADqcHeAa4CXhrDd/PMMdwTVJVahi6iVJZDaPEJJXIWGNqGFX5MJyS\n6ZUczVBoHatRSrkaho6GagoH6HZMNVEnVFeT8AgMPXkNJDPuhL5h91F3X10Isdz9ijhlTPb1J+gd\nSbumIPu8WXei09ety4pncnm2HR4eVYI86smE11pGOYHR65ynJRpkkdMg6Yld9pj74hm2Hhp2Hd6l\niIj7eXgFhs8nrOyM8WJP5a52iaw1KmxaRAj5faS9Tu9UIdjA7aLoMd3pyf7SMxYRCvjc8UChDbB9\njkJyIRT7jMr5MHTfFq+GATOvZdRSYCwGvM1/9zvbSjlPRJ4VkT+IyCme7Qq4W0Q2ishHKr2JiHxE\nRDaIyIaenvE7bRnqh4lqGNWYpLR9upzJAjwCoxqTlLPPcMoaU2B4J4S+eIaMlXcnyfZYiIWtEQ46\nQmRec9hZreoGRDmGUxa5vHIrwg4ksu7E8oRHw3CjpMpoGNGQn8ODaXeV++z+QrTSsMfko81oRz0T\n13MHBhlJW8RCftd0VNwz3BEYgVKBUZiwmx0NA2y/jGbzoaGiUN1StNbYWOKPWD2vcRwfRm5U2DTY\nC4d42nJzSbzBBuV8GFpQXnLKQjZd+To3kg4KhRYHk1nXT9FYJDBswVPOh6GjwxIZq6iEyMAMJ+/N\ntNP7SWCZUupU4FvAbzyvna+UOh14PfAxEXlVuRMopa5TSq1XSq3v6jKdY+cSbuJelRpGKODjH19/\nIm85bXQpNK8PI5e3u+qVm1D0hFCVSSpQyEGopJFEg8UJcwccp/XrTlkAwLL2BlqiQddM1dUYxsor\n13SmV7j9iYw72Q8ks25fa2/imC6EWE54hQP+ovLu2jcB9gpYFw10NYx4QQN57sCgU/484Dam8jq7\ntb9ntA+jsI/WMMAWcjrHRilbSFZC+zFiJSGyq7sa2Xc0UTHyK5HOESsnMAK+otyJoWR2lA/Da5LS\nvpvGSGDUfdUmqaGkVSiv4ggMXZgSPBpGZrSGkczkSGXzrjO9mn7ltaSWAuMAsNTzfImzzUUpNaSU\nGnH+vx0Iikin8/yA8/cIcAu2ictgcCnkYVSnYQBc8erVnLKoZdR2twCflfdUqi2nYRSS/8ajoGFk\nK2oY0RKTlNYktMBY3tFAcyTo7qMnTz25aEfpAU8pjP54xs2L8JLO6sS98mG1moaQn2c9AmPYKzCc\n/focU1Is5GeTo2E0hgNu4yCvs7uyScqjYUSCrv9jJG3x0iUt7nHzm8fXMGKlGobT33tPhXwM2+Q4\n2kcV9PtcTQ3sQAIdnVbe6V2+Ci8UTFKDHqHjmqQ84cla6CSc7ouJjOWeV5ukuhzNdqaT92opMJ4A\n1orIShEJAe8GbvXuICILxDF0isjZznj6RCQmIk3O9hjwF8AmDAYPrkmqSg1jLPQkmrHynjpD5Zze\n1WsYep+hsUxSoeK+3lojOGVRM+89ZxlvPnWRu1IF3IkjnsmRtnJu3SZvZrOeJHWEp57o4hm7lHj5\nKKnCtgtPmMemA4Ou2csbhVRwetsCY/2KdrZ0D9MznKYxEqQh6EekgkmqjA9D09IQpCkSdFubruyI\nscoJj/WaeUrRn8cok5SOlKpglqpkcgwGxE3WgxKnd0Nlp3fp+4N9vQ0hv62llAgWb9XfkXShREgm\nl3cd3mD7NVLZnCtMB5N1qmEopSzg48CdwGbgZqXU8yJyhYhc4ez2DmCTiDwDfBN4t7K/pfOBh5zt\njwO/V0rdUauxGo5PCnkY1WsYlfBGSZW2Z/XSMQGnt56E+0bSFccYDfpKNIwUDSE/LdEgX7r0pVx8\n8nx3pQqFyTORLqx8ATf/AmC3k+y3bn4TAEvabN+AnggrhdWCrUGdvbKd3pGM62gfSdvd9rz76dyO\nvzxjMZlcnsd3H3VLdngrstrXOLbA8Ak0OsJZZ4ov62hg7Tx70p+MhqGFza7e0Y7vjJXHyquyAiPk\n9xVpZ16TlDZ/leZhePurlNIcCRb5QbxO76SrYRTeL5nJuf4LcHwY2bybEDnTGkZNiw86ZqbbS7Z9\nz/P/t4FvlzluJ3BaLcdmOP4JTKGG4Y2SKleYTuOapKoQGC9Z1EJDyE9/IlvW0Qz2ZNrnsUsfGEiw\nqDValLPh1TC0AzieyRH1xO17y3nrSfKMZW1sOTTM0rYGnt0/6Jpayo1dT+pL2xs4eZHdv3zroWEW\ntUYZTo3WMLRJ6lXrurjoxHncs+WIu3puChf3rtCmOd0hUaNNQs3RoNsydmFLlG2HR1je0eD6MeaN\noWF0VHB6N4TsMZSz+WsNspJJSgvDmKMd6P1dDSOb5+3ffYS3vWyxHUFWxhylaYkGGUoVNAyv01tH\nt414TGDJbM71X9hjtTXJtliIkN8349neM+30NhgmTXCC5c3Hwuv01iu/ck7vRS1R/vaC1Vx80uiq\noqWs6Ixx32cu4IPnreAdZy4pu0+kxOl9cCDFotZo0T4tFTQMb5lzb8MgXVvpjKWtgL1CD/kLztyx\nTFJL2xrc99Or4uFU1o34KTVJxcJ+PnfJiYgUzC1L2hpGlQiByhqG9/p04t+y9hiXvGQBr3/JAldb\nKIfr9C7zWTVHgkX+CM1YC4JwwOdO5PObI/Zkn7aTHfVkn8jm2Linnw27+21nfxlzlDuGaMDxYdjv\nWRAYvkJYrTdMOlMQGJGgz3V6RwJ2BNpM15My5c0Nxy2nLGrhlWs7WTe/8ZjPVSQw9IRSxszg8wmf\nu+TEqs87rznClW85peLr5ZzeL1lc7JTXE6pIYYKMZ3JEymgYzZGAa7Y4fVkrTeEAq7pihAMFZ26l\nfhhgR2UVOvDl3TySUWG18QwBn523cMKCJq59z8tY45iQrv/g+qKM5/F8GF6BsdgRlss7GuhsDPPd\n955Z8d5BoTxIqUkK7MnaG/GkcTWMCn4H7aOY3xxhf3+Sff0J5jWF3fvS40zohwZt82E5h7dGR7jF\n0xY+KQhmu+qvE1abstyy8NokFQ74WNAccZ3ekaCd7T7TGoYRGIbjlvnNEW74m5dPybkCTpZvJuc1\nSdX+52E7ve3323FkhL54hhVOcyGNnlAbgv7CKjdjEU0XBJr2YSxtb+B5p9LtgpYI9332AlqiQb5x\n93bXh1GuWKPW0pa2R93/01a+qNuevZ8Oq83Q4Knf9IaXLnTPpUtiaMaLkvIKjMvPXsbqrsaiBLix\nWDe/ideePJ+znSKSXsbVMMrcB6/2Nb85TCaX57GdfZyzqsM1remkxsNDKTqbwmNrGJEgm5PDbuCA\nvl/hYKFz4UjaoqMxTM+wnQdzZCjFvOYw0VCA4VQWK6+IBP20NYTcRMeZwpikDAYKWb7FUVLHbuoa\nD28exrf/tJ1o0D/KfNXiJMM1hAPuSjqezpUtua1X6H6f0BS2+14H/HZZce3MLadhNEft867oiHlC\njHOjwkG1sBlIZsuu6steo3MfR+VhOLkTXgHT0Rjm9R7hU825v//+9SzvGG22ao6OIzDCoz9fr1DT\nzvbekQynL21179shrWEMpZyQ4+Co85SOobTeVXGUlOX6aRIZi8NDaeY3RVz/F9iCevW8RrYdHh7V\nj2M6MQLDYHAIOXWJkmPYuKeaaNBPysqzs2eEW585yPvOXe5GYmn0CrzR40xOZCxXsGnnsC0gbJNV\nazRY5DgPB3wFDaOMD+OclR386INncfbK9kK/Bk8RPm1W0scqVd4MVA59vkomKW8U2FTSHBnHJFXB\n6a2Z54nOOmNZGwG/3ePdW7bj0GByTJNUczTIcMpic/cQi9uKK/mmnLyLkVRBYCQzOQ4Pp5jfbAsM\nnbEfCfo5eWEzwymrKBlzujECw2BwCAd8JSapaRAYIT+5vOKnf95DwOfjw69cNWof1yQV8ruTXDyd\nc2tN6Rj95mjQnXy9rWLBnnDGEhg+n3DhifNcTQtsH4ZeBUecY7zho+UczWWvsVJpkOBok9RUMp6G\nUW784RKTFNjBFac4kWPhgK8o7LU/kR3TJKWv7fmDQ1xwwjx3eyTox8rbfTasvHKj35LZnN3DvClM\nJOh3I7YiQR8nLbTHsLl7uIqrrw1GYBgMDiG/3RNirCipqUZPUM/sH2Dt/MaySWp60omFA/h9QiTo\n4/+3d+4xclXnAf99M3deu5592OvH+m1sYzDGtRdjbB7CQEJJRENEaeNEIbhVFIUAoiR9gKgSqrZC\nqao0bV1CadMmaijQEJpQNwgIKRKlvByHGBtjwBD8wIa1Aa9f632d/nHPuXtndmb3znpnZr33+0kr\n3zn3seecHZ/vfo/zfcd7BtNNzGz231zzWY+WnK9huBBQRzaVDHIWjZTt1Esm8BLCyb7+IH1FsYYB\n0X08ZZ3emaFO77HEL5HaO8SE4/avlPr7pkJmM2eSWtreVDD+4sJMw2oYoXNXnB0WGP5cHLTFrdwO\n/o+O93Kku4+2SWl/0581I2ZTSc6akUcEduzvYvu7h/m/XbVP5K0CQ1Es6UDD8NN5l9s7MZa4RWvH\n/i7OmFo62ivjJUgnE8EbcUPa41hPH8dP9iEyuNg0ZVOBZlGsYYQX+ijpsYPU7E7DsAuml5BgB3lk\nH0aqtA9jSmOaWy5fxCeWzYj0nEppynkMmKE10J1JqrGEwHNzk0xIsClwhQ1PDp8PM9I+DPB9S0vs\nRkoYnE8nMNyLgtux3zYpU7iXxUvSmPGYN7mB7e8e5tYHX+ZPfrS17O+tFholpSiWtJegp68/KMdZ\nXPCoGrjFtLt3gDPaSu83EBGacqkgDLQhneT4yX7SyX4a016wKDXlPFoCk9RQDcMRLa2Jv09g0IeR\nCPqS8fxQ4OKEf+XIlvFhiAhfu3JJpGeMBudMP3zCNxsdPdnHM693BgKktIbhwl4TzGzOcXZ7E1ct\nG3TCu7nLZ7xAY8sPuw/D78MVZ08r+D65xJSdR3xtxaV8cSVv2yZlCuqxuDlcOrOJn736Pj39fkLC\nvv6BIKdaLVANQ1EsaVu/oru3dOrrahBOP7JwWvn9JL930fwgy26j0zB6+mjMJAOBkc+mguPWIT6M\nyjQMV5io2CQVftapmqSqTVCPwvpu7n/+HW68fwvb9h0mIcMX0sqlk+TSSR679RLWLpwSnHf3TGvK\nBFrccFFSC9oaac6luGZFYYbkTJFJqrkhRTqZYLcTGPlCDcPN4dkzmoKswv0DhveOnOSJ7Qe45+k3\naxI9pRqGoljSyUGndy0c3uDvw3CU0zAAbrpsUXDc3JDig2M9pL1koYaR9YIQ3GINI6xVRBIY1iQ1\n6PQu1lB6mRRZwyjt9K424RKp4NfJBnjx7Q9oTHslNUhnhiyXG8ot9C0NaVLJBB+N4PSe3pTlV9+4\ncki7e36ndaDnMyly6WSwY39KY7pkAkfn+F42q4lt+7rY+8FxNm3dz5bdH/KVdYuoNqphKIrFaRjO\nJFULwr9nuBQYYWa35ILdww0hDaMpl2JqPkNCBivfOcIaRtRqgSdDJqlwOvdMhRpGOR9GtXF7S7q6\n+zDGsGW3XxTq0LGeshpkukQ0WBgneFsbUoFTfDindzmKfRiTsn7ItAu4mJov9mH4/bpw0RQ2XDif\nb/yWnz1g74cneOP9o0GixmqjGoaiWNJeksPHezhRSw3DLhwzm7ORF+CZLTkOdHXT3pylMe0Fppd8\n1mNaPstPbrqYJTPyBfdUrGE4k5TzYXhDF6+oPgy3ONfaJNUcMknt+eBEsDhDeYe962O5FwangbQ0\npElaDWU4p3c5XJhyIDAyXjBPLttvKZNUQ9rjrk+dE+wSf+eD4+zqPMoli9uoBaphKIrFLZJd3b1B\nOu9q4xamchFSpZjVmqN/wPD2wWM0ZsImKf/fc2c3DxEKYQ0hSvRXxvNTV7gd5OH73eIVNUpqyYw8\nV5w1jXNnDS1cVU0Ck1R3L1t2++aolXP9iKdyAiFwepfJgByYpHKDBZ/ymcq/K24OXdbffHZwU6bL\niByux1Lcn4yXZHpThud3HaKnbyDI41VtVGAoisWF1R443M2MYcqCjiVu4VoY0RwFBNlsD9l8Ti4k\ns3iHeJhSGsJwZFIJG1bbjxQ5iAMNI6JG1JRN8d0N5xfsnK4FzlTUdaKPLbs/pDGd5NoOP+1KOe1o\nZJOUf761MU3HvFZmteSGLfBUDvf8Hfu7aG1IkfESwUZGl0crnOuqVN36WS05fmEFYa1MUiowFMWS\nSfrppDuPnmRGc27kG8aA5oaU3Ukc/e17Vij9eWPa48zpee7/4gVcfta0svcENSmSiUjhwhkv6e/0\n7hsg4yWK0ow480htzHajxbN7V5yG8RtzWlhutZxS1RRh0M9STgNxY29pSHHpmVN59vbLRxVR5zSG\nYz39/HbHbEQkCIBw6V1KmaTCzG5toH/Aj4yqlYahPgxFsaS9BO91dWPMUKdxtWjOpXjytkuZM7lh\n5IstrmYEDJqFLlo0vA3bvaFG8V+A0zBcau3CxcotdsNFB40XmnIpDnR1s2P/EW68dCFLZuRJJqRs\nWpORNAx33u2oHy1hgbR+9VxgUKNwGkauhNM7jKukOLM5O2xo71iiGoaiWNJeAvvCVlAAqNrMb2sk\nmYgeQdSQ9kKlSSsLbY1ijnLXOad3sTkk0DBOB4GRTfHcrkP0Dxg65rWQTSW5enk7581rLXn9SE7v\nwCTVcGoLtMv6e8GCyYF2MOjDsCYpqwWlvURQkTDM7Fb/JWPR9PyQc9Vi/P/FFaVGpEukth6vzGzJ\n2poUETPGeq5MalSB4Rf46e4dKHB4h58VNflgPWnKeex8z0/Wt3KOLyT+dv3KsteHN+6VwgnL5lMU\nGPmMx9XL27l+zbygzf3OtrwTGP7nbJm/mdMwauW/ABUYihIQXkxrqWGMhlktObbt66p481x0gREy\nSRVrGG4fxmmiYYC/KbK1cWQzktMwioWkw7UXJ3eslERC2Pi5joI2JyCmBlFSVmCU0XYW2I2eS+1m\nvlow/v/iilIj0qG38OLkfeMNFykVPT2HXQgr8mH4Tu9SIZ0Ak2pQkfBUcXtUVs4tbYIqJj2CSSob\ncnqPNc4RP2VSoYZRTnjNmdzAozdfpAJDUeqBExjtzdmaJB48FVykVFQfRqZSp7eXDPJqFZd0dQtY\nrfJtnQouvXjHvJYRrvQJTFJlBManV86kLZ+uSvneXJHT2wmnUiG1juWzo41rrFCBoSgW93Y53v0X\nEBYYlWkYUVO2u+u7TvQO2T+x5owp7P3wRGThU0+chtERUcNIjZBLat6UxpLlYMeCpTObWDi1MTCH\nJmztk3J9qQcqMBTFkglpGOOdtQuncO3KWZHfMAejpCrTSLpO9DK3KOT3siXTuGxJ+T0f44lLFk/l\nrYPHODNiJNFIGkY1ufTMqTz1tXUFbQ1pr+yu83qgAkNRLG6xmHEaCIyWhjTf+syKyNdXHiXlX3f4\nRO+4esOtlNULJrN6weTI17uKe+X8BrUml0qOq/kfH7OiKOOAQGCcBiapShlNlBT4O5GjOsonAoun\n5bl+zbwRN0LWiqZcKnIKllowfnqiKHUmnfQX1dPBJFUpbtGPHiU1fFqKiUraS/Dnn15W724E3H3t\nuZFDp2uBCgxFsbjdu3MnV8epWU8yo9QwoHzmVqX6hOuJjwf0m6AolrULp/BfN1/M0pm1i2uvFRXv\nwygQGOPnDVepLyowFMUiIpw7u7Y1G2qFn6W28igpUIGhDKICQ1FigIgwLZ+JXLuh0pKuSjxQH4ai\nxIRNt1wSuf60ahhKKVRgKEpMqKQyXEY1DKUE+k1QFGUI6vRWSqECQ1GUIahJSimFCgxFUYYQNknp\nPgzFod8ERVGGoCYppRQqMBRFGYLbtwHD12NQ4oUKDEVRhiAig/mn1CSlWKr6TRCRq0Rkp4i8KSK3\nlzi/TkQOi8jL9ufrUe9VFKW6ZCJUfFPiRdX2YYhIEvgH4OPAXuAlEXnUGPNq0aXPGGOuHuW9iqJU\nCadhqNNbcVTzm7AaeNMY85Yxpgd4ELimBvcqijIGOFNUcU1vJb5UU2DMAvaEPu+1bcVcKCJbReQx\nETmnwnsRkS+JyGYR2dzZ2TkW/VYUhZBJSjUMxVLvb8IWYK4xZjnw98CPK32AMeY+Y8wqY8yqqVOn\njnkHFSWuZDw/UiqdrPcyoYwXqvlN2AfMCX2ebdsCjDFdxpij9vinQEpE2qLcqyhKdcl4CbJeEnHx\ntUrsqabAeAlYLCILRCQNrAceDV8gIjPEfhtFZLXtz6Eo9yqKUl2yqaSG1CoFVC1KyhjTJyI3A48D\nSeBfjDHbReTL9vy9wHXAjSLSB5wA1htjDFDy3mr1VVGUoTgNQ1EcVU1vbs1MPy1quzd0vBHYGPVe\nRVFqR8ZLqsNbKUDrYSiKUpIvrJ3He0e6690NZRyhAkNRlJJcuKit3l1QxhmqbyqKoiiRUIGhKIqi\nREIFhqIoihIJFRiKoihKJFRgKIqiKJFQgaEoiqJEQgWGoiiKEgkVGIqiKEokxE/dNDEQkU7gnVHe\n3gYcHMPunI7EfQ7iPn7QOYD4zcE8Y0yk2hATSmCcCiKy2Rizqt79qCdxn4O4jx90DkDnYDjUJKUo\niqJEQgWGoiiKEgkVGIPcV+8OjAPiPgdxHz/oHIDOQVnUh6EoiqJEQjUMRVEUJRIqMBRFUZRIxF5g\niMhVIrJTRN4Ukdvr3Z9TQUTmiMj/iMirIrJdRG617ZNF5EkRecP+2xq65w479p0i8puh9vNE5BV7\n7u9ERGx7RkQesu0viMj8Wo8zCiKSFJFfisgm+zlWcyAiLSLysIi8JiI7RGRtnOZARG6z/we2icgD\nIpKN0/irhjEmtj9AEtgFnAGkgV8BS+vdr1MYTzvQYY/zwOvAUuCvgNtt++3AN+3xUjvmDLDAzkXS\nnnsRWAMI8BjwCdv+FeBee7weeKje4y4zF18F/h3YZD/Hag6A7wNftMdpoCUucwDMAt4GcvbzfwAb\n4jL+qs5tvTtQ5y/WWuDx0Oc7gDvq3a8xHN9PgI8DO4F229YO7Cw1XuBxOyftwGuh9s8C/xi+xh57\n+Dtipd5jLRr3bOAp4PKQwIjNHADNdsGUovZYzIEVGHuAybZvm4Ar4zL+av7E3STlvliOvbbttMeq\nyCuBF4Dpxpj99tQBYLo9Ljf+Wfa4uL3gHmNMH3AYmDLmAzg1vg38MTAQaovTHCwAOoF/tWa5fxaR\nRmIyB8aYfcBfA7uB/cBhY8wTxGT81STuAmNCIiKTgB8Bf2CM6QqfM/4r0YSNpRaRq4H3jTG/KHfN\nRJ8D/DfeDuA7xpiVwDF8E0zARJ4D65u4Bl9wzgQaReTz4Wsm8virSdwFxj5gTujzbNt22iIiKXxh\ncb8x5hHb/J6ItNvz7cD7tr3c+PfZ4+L2gntExMM3fxwa+5GMmouAT4nIr4EHgctF5AfEaw72AnuN\nMS/Yzw/jC5C4zMHHgLeNMZ3GmF7gEeBC4jP+qhF3gfESsFhEFohIGt959Wid+zRqbATHd4Edxphv\nhU49Ctxgj2/A92249vU24mMBsBh40artXSKyxj7zC0X3uGddB/zcvq2NC4wxdxhjZhtj5uP/PX9u\njPk88ZqDA8AeEVlim64AXiU+c7AbWCMiDbbfVwA7iM/4q0e9nSj1/gE+iR9NtAu4s979OcWxXIyv\nZm8FXrY/n8S3rT4FvAH8DJgcuudOO/ad2AgQ274K2GbPbWQwK0AW+CHwJn4EyRn1Hvcw87GOQad3\nrOYAWAFstt+FHwOtcZoD4M+A12zf/w0/Aio246/Wj6YGURRFUSIRd5OUoiiKEhEVGIqiKEokVGAo\niqIokVCBoSiKokRCBYaiKIoSCRUYyoRBRO60GUq3isjLInJBlX/f0yKyqoLrvyci+0QkYz+32Q2G\nY9GXdWIz8ypKtVCBoUwIRGQtcDV+tt7l+Lt99wx/V13oB36/3p0oRkSS9e6DMv5RgaFMFNqBg8aY\nkwDGmIPGmHcBROTrIvKSrY1wX6imwdMi8jcistnWjDhfRB6x9RL+wl4zX/yaEvfbax4WkYbiXy4i\nV4rIcyKyRUR+aPN5leLbwG02nUT4/gINQUQ2isgGe/xrEbnbak2bRaRDRB4XkV0i8uXQY5pE5L9t\nTYd7RSQxXN/sc78pIluA3xnNpCvxQgWGMlF4ApgjIq+LyD0icmno3EZjzPnGmGVADl8TcfQYY1YB\n9+KnfbgJWAZsEBGXfXQJcI8x5mygC78WQoCItAF/CnzMGNOBv8P6q2X6uRv4X+D6Cse32xizAngG\n+B5+Ooo1+DuaHauBW/DrOywEro3Qt0PGmA5jzIMV9keJISowlAmBMeYocB7wJfzU3g+5N3TgMlsV\n7RX8GhnnhG51ucNeAbYbY/ZbLeUtBhPS7THGPGuPf4CfgiXMGvxF+lkReRk/x9C8Ybp7N/BHVPb/\nL9zPF4wxR4wxncBJEWmx5140xrxljOkHHrD9HKlvD1XQByXmeCNfoiinB3ahfBp42gqHG0TkQeAe\nYJUxZo+I3IWfB8hx0v47EDp2n93/j+L8OcWfBXjSGPPZiP18wy7evxtq7qNQgGQL7xp1P0fq27Eo\nfVYUUA1DmSCIyBIRWRxqWgG8w+DCe9Da7q8bxePnWqc6wOfwTUphngcuEpFFti+NInLmCM/841qi\nbQAAAMRJREFUS+APQ5/fAZbajKkt+BlWK2W1zbycAD5j+zmavilKSVRgKBOFScD3ReRVEdmKb4a5\nyxjzEfBP+BlHH8dPaV8pO4GbRGQHftbX74RPWtPQBuAB+7ufA84a7oHGmO3AltDnPfi1p7fZf385\nin6+hJ9RdQd+idb/HE3fFKUcmq1WUYZB/FK3m6zDXFFijWoYiqIoSiRUw1AURVEioRqGoiiKEgkV\nGIqiKEokVGAoiqIokVCBoSiKokRCBYaiKIoSif8H/HItfW+WNKgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1968800f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "### DON'T RUN THIS BLOCK IF YOU DIDN'T RUN THE LAST ONE\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "%matplotlib inline\n",
    "\n",
    "fig = plt.figure()\n",
    "plt.plot([i * int(num_training / 200) for i in range(len(loss_function))], loss_function)\n",
    "plt.title('Optimization of cost function during training')\n",
    "plt.xlabel('Sample Number')\n",
    "plt.ylabel('Loss Function')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The results\n",
    "Since it takes so long to train the classifier, I've saved the parameters of my best model, trained on 95,000 reviews. In the next code block, I load the parameters into a new RNN and test the accuracy on a small 600-sample testing set. The testing here runs fairly quickly, so feel free to run the block to see the model in action! Make sure you've downloaded the dataset and successfully ran all of the code blocks above **except** for the very first block and the two immediately before this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample 0 Progress: 0% (0m 0s) Accuracy: 1.0000\n",
      "Sample 30 Progress: 5% (0m 4s) Accuracy: 0.9032\n",
      "Sample 60 Progress: 10% (0m 9s) Accuracy: 0.8525\n",
      "Sample 90 Progress: 15% (0m 15s) Accuracy: 0.8791\n",
      "Sample 120 Progress: 20% (0m 19s) Accuracy: 0.8760\n",
      "Sample 150 Progress: 25% (0m 24s) Accuracy: 0.8742\n",
      "Sample 180 Progress: 30% (0m 31s) Accuracy: 0.8343\n",
      "Sample 210 Progress: 35% (0m 39s) Accuracy: 0.8199\n",
      "Sample 240 Progress: 40% (0m 46s) Accuracy: 0.8257\n",
      "Sample 270 Progress: 45% (0m 53s) Accuracy: 0.8303\n",
      "Sample 300 Progress: 50% (0m 58s) Accuracy: 0.8272\n",
      "Sample 330 Progress: 55% (1m 1s) Accuracy: 0.8218\n",
      "Sample 360 Progress: 60% (1m 7s) Accuracy: 0.8283\n",
      "Sample 390 Progress: 65% (1m 13s) Accuracy: 0.8184\n",
      "Sample 420 Progress: 70% (1m 17s) Accuracy: 0.8100\n",
      "Sample 450 Progress: 75% (1m 24s) Accuracy: 0.8049\n",
      "Sample 480 Progress: 80% (1m 32s) Accuracy: 0.8046\n",
      "Sample 510 Progress: 85% (1m 36s) Accuracy: 0.8063\n",
      "Sample 540 Progress: 90% (1m 43s) Accuracy: 0.8133\n",
      "Sample 570 Progress: 95% (1m 48s) Accuracy: 0.8126\n",
      "Testing accuracy on 600 samples: 81.1667%\n"
     ]
    }
   ],
   "source": [
    "# Create new RNN\n",
    "model = RNN(input_size, hidden_units)\n",
    "\n",
    "# Load the learned parameters into the RNN from a file\n",
    "model.load_state_dict(torch.load('model_100000'))\n",
    "\n",
    "# Create the testing set of 600 samples\n",
    "small_testing_indices = indices[:300] + indices[-300:]\n",
    "small_testing_set = DataLoader(dataset=data, sampler=SubsetRandomSampler(small_testing_indices))\n",
    "\n",
    "accuracy = test(small_testing_set)\n",
    "print('Testing accuracy on %d samples: %.4f%%' % (len(small_testing_set), accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "81.1667%... Not bad, and definitely better than random guessing!\n",
    "\n",
    "## Future improvement\n",
    "To speed up the training process in the future, I'd like to investigate the use of padding to group word vectors into equal-length batches so that training them can be parallelized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}