maxout_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Maxout Networks in Pytorch "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is an implementation of maxout units (https://arxiv.org/abs/1302.4389) in Pytorch. My primary motivation of this was to ensure that I actually understood what maxout units were, as I found most explanations available quite opaque or incorrect! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import torch \n",
    "import torch.nn as nn \n",
    "from torchvision import datasets, transforms \n",
    "from torch.autograd import Variable \n",
    "from torch.nn import init\n",
    "import torch.nn.functional as F\n",
    "import matplotlib.pyplot as plt \n",
    "import seaborn as sns\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we define a custom object so we can have models which contain objects which behave like lists while holding multiple modules. This is needed as we're going to store lists of layers per maxout unit. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class ListModule(object):\n",
    "    def __init__(self, module, prefix, *args):\n",
    "        self.module = module\n",
    "        self.prefix = prefix\n",
    "        self.num_module = 0\n",
    "        for new_module in args:\n",
    "            self.append(new_module)\n",
    "\n",
    "    def append(self, new_module):\n",
    "        if not isinstance(new_module, nn.Module):\n",
    "            raise ValueError('Not a Module')\n",
    "        else:\n",
    "            self.module.add_module(self.prefix + str(self.num_module), new_module)\n",
    "            self.num_module += 1\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.num_module\n",
    "\n",
    "    def __getitem__(self, i):\n",
    "        if i < 0 or i >= self.num_module:\n",
    "            raise IndexError('Out of bound')\n",
    "        return getattr(self.module, self.prefix + str(i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.0001\n",
    "batch_size = 50\n",
    "n_epochs = 3\n",
    "cuda = torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The magic normalization numbers below are actually the mean and the std of the training set of MNIST, hardcoded to save compute each run. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_loader = torch.utils.data.DataLoader(\n",
    "    datasets.MNIST('../data', train=True, download=True,\n",
    "                   transform=transforms.Compose([\n",
    "                       transforms.ToTensor(),\n",
    "                       transforms.Normalize((0.1307,), (0.3081,))\n",
    "                       ])),\n",
    "                       batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_loader = torch.utils.data.DataLoader(\n",
    "    datasets.MNIST('../data', train=False, transform=transforms.Compose([\n",
    "                       transforms.ToTensor(),\n",
    "                       transforms.Normalize((0.1307,), (0.3081,))\n",
    "                       ])),\n",
    "                       batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Maxout? \n",
    "Maxout is a piece of a deep learning model that sits somewhere between an activation function and a layer. It computes a nonlinearity (namely the max function) but also has learnable parameters. For reference, it is generally considered an activation but I find that terminology troublesome as it forces there to be multiple layers to max over. `¯\\_(ツ)_/¯`\n",
    "\n",
    "In their paper, Goodfellow et. al. describe the maxout activation as: \n",
    "\n",
    "$$h_i = max_{j \\in [1,k]} z_{i,j}$$\n",
    "\n",
    "where $h_i$ is the hidden layer output and $z = x^TW_{\\cdots i,j} + b_{i,j}$ , where $x$ could be the input or the output of the previous layer. \n",
    "\n",
    "\n",
    "I find this description incredibly confusing and for a while I thought that all that you did for maxout was to compute a linear layer, group into k groups, and take the max over those groups. (the use of $k$ in the indices above should give you a hint that this interpretation is wrong) Alas, Karpathy was to the rescue here: http://cs231n.github.io/neural-networks-1/\n",
    "\n",
    "The tldr; is that a maxout unit is an element-wise max _over_ multiple linear (or convolutional/other) layers, so for a maxout layer with two units it is computed, like so: \n",
    "$max\\{W_1^Tx +b_1, W_2^Tx + b_2\\}$\n",
    "\n",
    "so in the case where $W_2^Tx+b_2 = 0$ we see that the maxout layer is a ReLU. For other properties about maxout nets, we refer back to Goodfellow's paper. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Models \n",
    "Here we define the two classes of models to test the maxout network against. \n",
    "\n",
    "We have an MLP and ConvNet using just ReLU's and the maxout equivalent of those networks. \n",
    "\n",
    "Note that for each layer which uses a maxout we have $lu$ total parameters, where $l$ is the number of parameters per unit and $u$ is the number of units. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class rectifier_mlp(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(rectifier_mlp, self).__init__()\n",
    "        self.fc1 = nn.Linear(784, 1024)\n",
    "        self.fc2 = nn.Linear(1024, 10)\n",
    "\n",
    "\n",
    "    def forward(self, x): \n",
    "        x = x.view(-1, 784)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.dropout(x, training=self.training)\n",
    "        x = self.fc2(x)\n",
    "        return F.log_softmax(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "class maxout_mlp(nn.Module):\n",
    "    def __init__(self, num_units=2):\n",
    "        super(maxout_mlp, self).__init__()\n",
    "        self.fc1_list = ListModule(self, \"fc1_\")\n",
    "        self.fc2_list = ListModule(self, \"fc2_\")\n",
    "        for _ in range(num_units):\n",
    "            self.fc1_list.append(nn.Linear(784, 1024))\n",
    "            self.fc2_list.append(nn.Linear(1024, 10))\n",
    "\n",
    "    def forward(self, x): \n",
    "        x = x.view(-1, 784)\n",
    "        x = self.maxout(x, self.fc1_list)\n",
    "        x = F.dropout(x, training=self.training)\n",
    "        x = self.maxout(x, self.fc2_list)\n",
    "        return F.log_softmax(x)\n",
    "\n",
    "    def maxout(self, x, layer_list):\n",
    "        max_output = layer_list[0](x)\n",
    "        for _, layer in enumerate(layer_list, start=1):\n",
    "            max_output = torch.max(max_output, layer(x))\n",
    "        return max_output\n",
    "                            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class rectifier_conv_net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(rectifier_conv_net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 32, 5, padding=2)\n",
    "        self.conv2 = nn.Conv2d(32, 64, 5, padding=2)\n",
    "        self.fc1 = nn.Linear(64*7*7, 1024)\n",
    "        self.fc2 = nn.Linear(1024, 10)\n",
    "\n",
    "\n",
    "    def forward(self, x): \n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), 2)\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n",
    "        x = x.view(-1, 64*7*7)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.dropout(x, training=self.training)\n",
    "        x = self.fc2(x)\n",
    "        return F.log_softmax(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class maxout_conv_net(nn.Module):\n",
    "    def __init__(self, num_units=2):\n",
    "        super(maxout_conv_net, self).__init__()\n",
    "        self.conv1_list = ListModule(self, \"conv1_\")\n",
    "        self.conv2_list = ListModule(self, \"conv2_\")\n",
    "        self.fc1_list = ListModule(self, \"fc1_\")\n",
    "        self.fc2_list = ListModule(self, \"fc2_\")\n",
    "        for _ in range(num_units):\n",
    "            self.conv1_list.append(nn.Conv2d(1, 32, 5, padding=2))\n",
    "            self.conv2_list.append(nn.Conv2d(32, 64, 5, padding=2))\n",
    "            self.fc1_list.append(nn.Linear(64*7*7, 1024))\n",
    "            self.fc2_list.append(nn.Linear(1024, 10))\n",
    "\n",
    "    def forward(self, x): \n",
    "        x = F.max_pool2d(self.maxout(x, self.conv1_list), 2)\n",
    "        x = F.max_pool2d(self.maxout(x, self.conv2_list), 2)\n",
    "        x = x.view(-1, 64*7*7)\n",
    "        x = self.maxout(x, self.fc1_list)\n",
    "        x = F.dropout(x, training=self.training)\n",
    "        x = self.maxout(x, self.fc2_list)\n",
    "        return F.log_softmax(x)\n",
    "\n",
    "    def maxout(self, x, layer_list):\n",
    "        max_output = layer_list[0](x)\n",
    "        for _, layer in enumerate(layer_list, start=1):\n",
    "            max_output = torch.max(max_output, layer(x))\n",
    "        return max_output\n",
    "                            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(epoch,  net, train_loss, train_acc): \n",
    "    optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)\n",
    "    net.train()\n",
    "    for batch_idx, (data, target) in enumerate(train_loader): \n",
    "        if cuda:\n",
    "            data, target = Variable(data).cuda(0), Variable(target).cuda(0)\n",
    "        else:\n",
    "            data, target = Variable(data), Variable(target)\n",
    "            \n",
    "        optimizer.zero_grad()\n",
    "        output = net(data)\n",
    "        loss = F.cross_entropy(output, target)\n",
    "        train_loss.append(loss.data[0])\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        prediction = output.data.max(1)[1]\n",
    "        accuracy = prediction.eq(target.data).sum()*1.0/batch_size*100.0 \n",
    "        train_acc.append(accuracy)\n",
    "        if batch_idx % 100 == 0:\n",
    "            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
    "                epoch, batch_idx * len(data), len(train_loader.dataset),\n",
    "                100. * batch_idx / len(train_loader), loss.data[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test(epoch, net):\n",
    "    net.eval()\n",
    "    test_loss = 0\n",
    "    correct = 0\n",
    "    for data, target in test_loader:\n",
    "        if cuda:\n",
    "            data, target = data.cuda(), target.cuda()\n",
    "        data, target = Variable(data, volatile=True), Variable(target)\n",
    "        output = net(data)\n",
    "        test_loss += F.cross_entropy(output, target).data[0]\n",
    "        pred = output.data.max(1)[1] # get the index of the max log-probability\n",
    "        correct += pred.eq(target.data).cpu().sum()\n",
    "\n",
    "    test_loss = test_loss\n",
    "    test_loss /= len(test_loader) # loss function already averages over batch size\n",
    "    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
    "        test_loss, correct, len(test_loader.dataset),\n",
    "        100. * correct / len(test_loader.dataset)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 0 [0/60000 (0%)]\tLoss: 2.290519\n",
      "Train Epoch: 0 [5000/60000 (8%)]\tLoss: 0.285147\n",
      "Train Epoch: 0 [10000/60000 (17%)]\tLoss: 0.146821\n",
      "Train Epoch: 0 [15000/60000 (25%)]\tLoss: 0.272205\n",
      "Train Epoch: 0 [20000/60000 (33%)]\tLoss: 0.221471\n",
      "Train Epoch: 0 [25000/60000 (42%)]\tLoss: 0.224038\n",
      "Train Epoch: 0 [30000/60000 (50%)]\tLoss: 0.274867\n",
      "Train Epoch: 0 [35000/60000 (58%)]\tLoss: 0.038073\n",
      "Train Epoch: 0 [40000/60000 (67%)]\tLoss: 0.135521\n",
      "Train Epoch: 0 [45000/60000 (75%)]\tLoss: 0.104026\n",
      "Train Epoch: 0 [50000/60000 (83%)]\tLoss: 0.293587\n",
      "Train Epoch: 0 [55000/60000 (92%)]\tLoss: 0.146798\n",
      "\n",
      "Test set: Average loss: 0.0731, Accuracy: 9766/10000 (98%)\n",
      "\n",
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 0.049651\n",
      "Train Epoch: 1 [5000/60000 (8%)]\tLoss: 0.025212\n",
      "Train Epoch: 1 [10000/60000 (17%)]\tLoss: 0.130879\n",
      "Train Epoch: 1 [15000/60000 (25%)]\tLoss: 0.078905\n",
      "Train Epoch: 1 [20000/60000 (33%)]\tLoss: 0.039131\n",
      "Train Epoch: 1 [25000/60000 (42%)]\tLoss: 0.010621\n",
      "Train Epoch: 1 [30000/60000 (50%)]\tLoss: 0.037744\n",
      "Train Epoch: 1 [35000/60000 (58%)]\tLoss: 0.075127\n",
      "Train Epoch: 1 [40000/60000 (67%)]\tLoss: 0.070165\n",
      "Train Epoch: 1 [45000/60000 (75%)]\tLoss: 0.045732\n",
      "Train Epoch: 1 [50000/60000 (83%)]\tLoss: 0.037628\n",
      "Train Epoch: 1 [55000/60000 (92%)]\tLoss: 0.069472\n",
      "\n",
      "Test set: Average loss: 0.0409, Accuracy: 9864/10000 (99%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 0.112145\n",
      "Train Epoch: 2 [5000/60000 (8%)]\tLoss: 0.144255\n",
      "Train Epoch: 2 [10000/60000 (17%)]\tLoss: 0.005676\n",
      "Train Epoch: 2 [15000/60000 (25%)]\tLoss: 0.044572\n",
      "Train Epoch: 2 [20000/60000 (33%)]\tLoss: 0.009544\n",
      "Train Epoch: 2 [25000/60000 (42%)]\tLoss: 0.069822\n",
      "Train Epoch: 2 [30000/60000 (50%)]\tLoss: 0.035140\n",
      "Train Epoch: 2 [35000/60000 (58%)]\tLoss: 0.005060\n",
      "Train Epoch: 2 [40000/60000 (67%)]\tLoss: 0.043649\n",
      "Train Epoch: 2 [45000/60000 (75%)]\tLoss: 0.015031\n",
      "Train Epoch: 2 [50000/60000 (83%)]\tLoss: 0.055224\n",
      "Train Epoch: 2 [55000/60000 (92%)]\tLoss: 0.048069\n",
      "\n",
      "Test set: Average loss: 0.0301, Accuracy: 9898/10000 (99%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if cuda:\n",
    "    net_relu = rectifier_conv_net().cuda()\n",
    "else:\n",
    "    net_relu = rectifier_conv_net()\n",
    "relu_loss = [] \n",
    "relu_acc = []\n",
    "for epoch in range(n_epochs): \n",
    "    train(epoch, net_relu, relu_loss, relu_acc)\n",
    "    test(epoch, net_relu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 0 [0/60000 (0%)]\tLoss: 2.363012\n",
      "Train Epoch: 0 [5000/60000 (8%)]\tLoss: 0.478533\n",
      "Train Epoch: 0 [10000/60000 (17%)]\tLoss: 0.089672\n",
      "Train Epoch: 0 [15000/60000 (25%)]\tLoss: 0.102352\n",
      "Train Epoch: 0 [20000/60000 (33%)]\tLoss: 0.224029\n",
      "Train Epoch: 0 [25000/60000 (42%)]\tLoss: 0.155579\n",
      "Train Epoch: 0 [30000/60000 (50%)]\tLoss: 0.085204\n",
      "Train Epoch: 0 [35000/60000 (58%)]\tLoss: 0.110757\n",
      "Train Epoch: 0 [40000/60000 (67%)]\tLoss: 0.073194\n",
      "Train Epoch: 0 [45000/60000 (75%)]\tLoss: 0.145176\n",
      "Train Epoch: 0 [50000/60000 (83%)]\tLoss: 0.027566\n",
      "Train Epoch: 0 [55000/60000 (92%)]\tLoss: 0.060530\n",
      "\n",
      "Test set: Average loss: 0.0493, Accuracy: 9836/10000 (98%)\n",
      "\n",
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 0.097051\n",
      "Train Epoch: 1 [5000/60000 (8%)]\tLoss: 0.011630\n",
      "Train Epoch: 1 [10000/60000 (17%)]\tLoss: 0.023141\n",
      "Train Epoch: 1 [15000/60000 (25%)]\tLoss: 0.032429\n",
      "Train Epoch: 1 [20000/60000 (33%)]\tLoss: 0.120462\n",
      "Train Epoch: 1 [25000/60000 (42%)]\tLoss: 0.072779\n",
      "Train Epoch: 1 [30000/60000 (50%)]\tLoss: 0.118338\n",
      "Train Epoch: 1 [35000/60000 (58%)]\tLoss: 0.020701\n",
      "Train Epoch: 1 [40000/60000 (67%)]\tLoss: 0.038554\n",
      "Train Epoch: 1 [45000/60000 (75%)]\tLoss: 0.041294\n",
      "Train Epoch: 1 [50000/60000 (83%)]\tLoss: 0.005239\n",
      "Train Epoch: 1 [55000/60000 (92%)]\tLoss: 0.012394\n",
      "\n",
      "Test set: Average loss: 0.0345, Accuracy: 9881/10000 (99%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 0.072008\n",
      "Train Epoch: 2 [5000/60000 (8%)]\tLoss: 0.011492\n",
      "Train Epoch: 2 [10000/60000 (17%)]\tLoss: 0.054487\n",
      "Train Epoch: 2 [15000/60000 (25%)]\tLoss: 0.006063\n",
      "Train Epoch: 2 [20000/60000 (33%)]\tLoss: 0.118288\n",
      "Train Epoch: 2 [25000/60000 (42%)]\tLoss: 0.039214\n",
      "Train Epoch: 2 [30000/60000 (50%)]\tLoss: 0.037069\n",
      "Train Epoch: 2 [35000/60000 (58%)]\tLoss: 0.013439\n",
      "Train Epoch: 2 [40000/60000 (67%)]\tLoss: 0.010580\n",
      "Train Epoch: 2 [45000/60000 (75%)]\tLoss: 0.012139\n",
      "Train Epoch: 2 [50000/60000 (83%)]\tLoss: 0.062259\n",
      "Train Epoch: 2 [55000/60000 (92%)]\tLoss: 0.127273\n",
      "\n",
      "Test set: Average loss: 0.0381, Accuracy: 9870/10000 (99%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if cuda:\n",
    "    net_maxout = maxout_conv_net(num_units=5).cuda() #this uses 5 \"maxout units\" per \"layer\" \n",
    "else:\n",
    "    net_maxout = maxout_conv_net(num_units=5)\n",
    "maxout_loss = [] \n",
    "maxout_acc = []\n",
    "for epoch in range(n_epochs): \n",
    "    train(epoch, net_maxout, maxout_loss, maxout_acc)\n",
    "    test(epoch, net_maxout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 0 [0/60000 (0%)]\tLoss: 2.348691\n",
      "Train Epoch: 0 [5000/60000 (8%)]\tLoss: 0.682529\n",
      "Train Epoch: 0 [10000/60000 (17%)]\tLoss: 0.646795\n",
      "Train Epoch: 0 [15000/60000 (25%)]\tLoss: 0.302137\n",
      "Train Epoch: 0 [20000/60000 (33%)]\tLoss: 0.265129\n",
      "Train Epoch: 0 [25000/60000 (42%)]\tLoss: 0.374851\n",
      "Train Epoch: 0 [30000/60000 (50%)]\tLoss: 0.476626\n",
      "Train Epoch: 0 [35000/60000 (58%)]\tLoss: 0.351321\n",
      "Train Epoch: 0 [40000/60000 (67%)]\tLoss: 0.422480\n",
      "Train Epoch: 0 [45000/60000 (75%)]\tLoss: 0.324298\n",
      "Train Epoch: 0 [50000/60000 (83%)]\tLoss: 0.330992\n",
      "Train Epoch: 0 [55000/60000 (92%)]\tLoss: 0.216023\n",
      "\n",
      "Test set: Average loss: 0.2178, Accuracy: 9364/10000 (94%)\n",
      "\n",
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 0.397623\n",
      "Train Epoch: 1 [5000/60000 (8%)]\tLoss: 0.346569\n",
      "Train Epoch: 1 [10000/60000 (17%)]\tLoss: 0.124273\n",
      "Train Epoch: 1 [15000/60000 (25%)]\tLoss: 0.188086\n",
      "Train Epoch: 1 [20000/60000 (33%)]\tLoss: 0.086380\n",
      "Train Epoch: 1 [25000/60000 (42%)]\tLoss: 0.271180\n",
      "Train Epoch: 1 [30000/60000 (50%)]\tLoss: 0.160728\n",
      "Train Epoch: 1 [35000/60000 (58%)]\tLoss: 0.398956\n",
      "Train Epoch: 1 [40000/60000 (67%)]\tLoss: 0.136644\n",
      "Train Epoch: 1 [45000/60000 (75%)]\tLoss: 0.100698\n",
      "Train Epoch: 1 [50000/60000 (83%)]\tLoss: 0.144666\n",
      "Train Epoch: 1 [55000/60000 (92%)]\tLoss: 0.105870\n",
      "\n",
      "Test set: Average loss: 0.1531, Accuracy: 9554/10000 (96%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 0.146667\n",
      "Train Epoch: 2 [5000/60000 (8%)]\tLoss: 0.111268\n",
      "Train Epoch: 2 [10000/60000 (17%)]\tLoss: 0.150802\n",
      "Train Epoch: 2 [15000/60000 (25%)]\tLoss: 0.102013\n",
      "Train Epoch: 2 [20000/60000 (33%)]\tLoss: 0.167289\n",
      "Train Epoch: 2 [25000/60000 (42%)]\tLoss: 0.114355\n",
      "Train Epoch: 2 [30000/60000 (50%)]\tLoss: 0.123657\n",
      "Train Epoch: 2 [35000/60000 (58%)]\tLoss: 0.154903\n",
      "Train Epoch: 2 [40000/60000 (67%)]\tLoss: 0.087987\n",
      "Train Epoch: 2 [45000/60000 (75%)]\tLoss: 0.081518\n",
      "Train Epoch: 2 [50000/60000 (83%)]\tLoss: 0.083032\n",
      "Train Epoch: 2 [55000/60000 (92%)]\tLoss: 0.119544\n",
      "\n",
      "Test set: Average loss: 0.1170, Accuracy: 9666/10000 (97%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if cuda:\n",
    "    mlp_relu = rectifier_mlp().cuda() \n",
    "else:\n",
    "    mlp_relu = rectifier_mlp()\n",
    "relu_mlp_loss = [] \n",
    "relu_mlp_acc = []\n",
    "for epoch in range(n_epochs): \n",
    "    train(epoch, mlp_relu, relu_mlp_loss, relu_mlp_acc)\n",
    "    test(epoch, mlp_relu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 0 [0/60000 (0%)]\tLoss: 3.156196\n",
      "Train Epoch: 0 [5000/60000 (8%)]\tLoss: 0.565568\n",
      "Train Epoch: 0 [10000/60000 (17%)]\tLoss: 0.644320\n",
      "Train Epoch: 0 [15000/60000 (25%)]\tLoss: 0.355569\n",
      "Train Epoch: 0 [20000/60000 (33%)]\tLoss: 0.463959\n",
      "Train Epoch: 0 [25000/60000 (42%)]\tLoss: 0.441238\n",
      "Train Epoch: 0 [30000/60000 (50%)]\tLoss: 0.456167\n",
      "Train Epoch: 0 [35000/60000 (58%)]\tLoss: 0.370821\n",
      "Train Epoch: 0 [40000/60000 (67%)]\tLoss: 0.294294\n",
      "Train Epoch: 0 [45000/60000 (75%)]\tLoss: 0.256920\n",
      "Train Epoch: 0 [50000/60000 (83%)]\tLoss: 0.238175\n",
      "Train Epoch: 0 [55000/60000 (92%)]\tLoss: 0.527092\n",
      "\n",
      "Test set: Average loss: 0.5914, Accuracy: 9490/10000 (95%)\n",
      "\n",
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 0.281222\n",
      "Train Epoch: 1 [5000/60000 (8%)]\tLoss: 0.229687\n",
      "Train Epoch: 1 [10000/60000 (17%)]\tLoss: 0.449179\n",
      "Train Epoch: 1 [15000/60000 (25%)]\tLoss: 0.501339\n",
      "Train Epoch: 1 [20000/60000 (33%)]\tLoss: 0.267238\n",
      "Train Epoch: 1 [25000/60000 (42%)]\tLoss: 0.406091\n",
      "Train Epoch: 1 [30000/60000 (50%)]\tLoss: 0.285968\n",
      "Train Epoch: 1 [35000/60000 (58%)]\tLoss: 0.470900\n",
      "Train Epoch: 1 [40000/60000 (67%)]\tLoss: 0.226877\n",
      "Train Epoch: 1 [45000/60000 (75%)]\tLoss: 0.414314\n",
      "Train Epoch: 1 [50000/60000 (83%)]\tLoss: 0.300500\n",
      "Train Epoch: 1 [55000/60000 (92%)]\tLoss: 0.258165\n",
      "\n",
      "Test set: Average loss: 0.4473, Accuracy: 9633/10000 (96%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 0.435306\n",
      "Train Epoch: 2 [5000/60000 (8%)]\tLoss: 0.141389\n",
      "Train Epoch: 2 [10000/60000 (17%)]\tLoss: 0.276851\n",
      "Train Epoch: 2 [15000/60000 (25%)]\tLoss: 0.212474\n",
      "Train Epoch: 2 [20000/60000 (33%)]\tLoss: 0.321834\n",
      "Train Epoch: 2 [25000/60000 (42%)]\tLoss: 0.121022\n",
      "Train Epoch: 2 [30000/60000 (50%)]\tLoss: 0.233975\n",
      "Train Epoch: 2 [35000/60000 (58%)]\tLoss: 0.367659\n",
      "Train Epoch: 2 [40000/60000 (67%)]\tLoss: 0.370711\n",
      "Train Epoch: 2 [45000/60000 (75%)]\tLoss: 0.255677\n",
      "Train Epoch: 2 [50000/60000 (83%)]\tLoss: 0.080479\n",
      "Train Epoch: 2 [55000/60000 (92%)]\tLoss: 0.393202\n",
      "\n",
      "Test set: Average loss: 0.3117, Accuracy: 9689/10000 (97%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if cuda:\n",
    "    mlp_maxout = maxout_mlp(num_units=5).cuda() \n",
    "else:\n",
    "    mlp_maxout = maxout_mlp(num_units=5)\n",
    "maxout_mlp_loss = [] \n",
    "maxout_mlp_acc = []\n",
    "for epoch in range(n_epochs): \n",
    "    train(epoch, mlp_maxout, maxout_mlp_loss, maxout_mlp_acc)\n",
    "    test(epoch, mlp_maxout)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plots! \n",
    "Here I show the accuracy and loss of both the MLP and the ConvNet versions of the ReLU and maxout nets. \n",
    "\n",
    "These plots aren't incredibly informative as MNIST is so easy that both nets do very well quickly. That said, it is clear from the plots and numbers above that the maxout network is doing better than the ReLU net. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fabc4065a10>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/cse/home/duncanw/summer/env/local/lib/python2.7/site-packages/matplotlib/font_manager.py:1297: UserWarning: findfont: Font family [u'sans-serif'] not found. Falling back to DejaVu Sans\n",
      "  (prop.get_family(), self.defaultFamily[fontext]))\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEMCAYAAAArnKpYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecHHX9+PHXbL+aTXKXnpCEhAkhjRAiJAQQsKCifpGv\n/fdF8auColQBBQT0C0KQIhBAVESkikqVGiCNhPRe5tIuyV3a5Xq/2935/bFtdne2XNm9m/B+Ph55\nZHdmdvZ9e3vznk9XdF1HCCGEALD1dQBCCCH6D0kKQgghIiQpCCGEiJCkIIQQIkKSghBCiAhJCkII\nISIcuXqjqqrGHvV9LSx009TU3lvh5ITEnBsSc25IzLmTi7hLS4sUs+2WKSk4HPa+DqHLJObckJhz\nQ2LOnb6M2zJJQQghRPZJUhBCCBEhSUEIIUSEJAUhhBARkhSEEEJESFIQQog+tG7dGiorK7r12jff\nfJ3XX3+lV+ORpCCEEH1o/fq1HDxY2ddhRORs8JoQQljFunVreO65p7Hb7dTV1fGVr1zMO++8icvl\n4oYbbuY3v7kVn8/HiSdO5Prrb+Kjj5ayYcM6rrjiZ1x//VXccMPNANx11+10dnYwd+7ZfPe73+PO\nO2/n0kt/wKhRo7nyyh/xwAMLeOutN1iyZBGzZs3mZz+7JiaON998neXLl9Le3o7f7+ess87h/fff\nZfToMdx0062R4w4dOshvfnMLxcUDqK2t5fbb72TEiJHd+tktUVJ44f2dbN1T3ddhCCH6kaO1LZQf\nbkj5b3dFXdJ9R2tbUp7f4XBwzz0PMHfuPMrKNP7wh8coKRlCVdVRHnhgAY899hdaWpo5cGA/c+fO\no6ammvnz72Tu3HkMGzaMZ5/9Gz/4wY957LEnWbduLceOVSW8h9Pp5MILv8SVV16dkBDCSkqGcO+9\nf2Do0OF0dnawYMGfOHLkMA0N9THHNTQ0cOed93LVVdfz7LN/6/bnaomSwvIthxnkzWPkoFF9HYoQ\noh9oaOngl098TE8WjlQUeOBnZ1Gc7zLdP27ciQCUlJTi9Q4MPS6houIAf//7X2lsbOTw4UMcO1bF\n6NFj+MpXLub666/i2mtvBKCysoKTTpoEwMSJJ3HwYCWKYjqzRErjx4fjKDE8LqWxsTHuuAk4HA4m\nTjyJiorutVGARZJCNz5HIcRxrDjfxe9+dAYt7b6UxxUVemhsajPdl+92JE0IQMwF3Pj46NEjzJt3\nLl/4wkXcccct6LpOIBDgqaf+wve//0OeeeYpLrvsR4wcOQpN286MGTPZuVPjkku+QUFBIdXVxxgy\nZCgHDuwHgiWSQCCQURwQfRy/lPLevbvx+/3s2lXGyJHdqzoCiyQFAGQpaSGEwZCB+WmP8XrzqatL\nXU3UVTabjRdeeIalSxdFtv3zny9w9tnn8tWvfo1bbrmBPXt2853v/A//93+34fP5mDt3HqWlQ/j8\n57/IXXfdwcSJJzF48GAATj31NP74xwVs27aF73//h92Oy+sdyC9/eT11dbXcdtv/dfs8Sny2yZae\nzJJ69UNLuXDOOD43y1rVR9n4QmabxJwbEnNuWDFm6Hrchw4d5E9/eoxf//q3Gb8m2Syp1igpKEpC\nUUkIIY4n+/eXM3/+XUBwllS73cl99z2U8zgskRSkSUEIcbwbM2YsjzzyBND1ksLw4SO6VEpIxRJd\nUlHoUS8DIYQQmUlbUlBVdQrwBOAHdgGXaZqmG/Y9TvBm/gpN0zZlI0gFaWcWQohcyKSkoGmaNkfT\ntHmh57MM+34LfAv4euhxViiKFBWEECIX0iYFTdM6DU/bgQOG5wM1TTugaVol4O3t4IwkJQgh+osr\nrvhBr53rzjtvp6LiQPoDcySjhmZVVb8M3AXsBIzzTRiTStbag6WgIIToK4FAAJvNGs2vvSGjpKBp\n2mvAa6qqPgx8CXg5tMt4qU4+JA8oLHR3ezFqm82GogRb5K3EbrdJzDkgMefGJynmVatW8fTTwfmD\nLrnkEv761yfx+fxcccUVnHXWPBwOO15vPt/73qX8+c9/weFw8L3vXcpTTyXOOXTzzb/C7XZTVlbG\n7NmzaWxsZOPGDXzzm9/i4ou/hsvloLjYw6JF7/D++wtpb+/A43Fz333343QmH3GdLZk0NLs1TWsP\nPW0AWg27a1RVHUUwITSkOk9TU3uq3SnpAZ1AQLfcIBQrDpyRmHNDYu65qpZqWn2tKY8pKvLQ2Gg+\nzUWeI4/S/MGm+5qa2mhpaeP3v/8DP//55Tz44KPous711/+cKVNOw+fzU1fXEvnf4XBEHsfr6PAx\ne/Zcfv7zX/Ctb13MHXfcxQ9/eCVXXXU55513IR0dPhoa2mhp6aCwcAC//e0t/Otfz/Hqq//hggs+\n1/UPJkOlpUWm2zMpKXxeVdVrQ493AptUVb1Z07Q7gduAF0P7ftrjKJNQFGlTEEJENXY0ccfH89F7\ncGVQUPjdWbdS5Co03a+qk6ivr2PfvnKuvvonANTV1cYMpM10grvwRHaDB5dEJq4zq3GfOFEFYNKk\nSaxeva4rP06vSZsUNE17FXg1bvOdoX2bgLlZiEsIIZIqchVy2xk39LikkCwhQPCCP2CAl/HjJ3D/\n/Q9jt9vx+XwxiaCwMDjBndvtobr6WMpzZWLXrjIANE1j5Mi+mdbHEiOaAWlpFkLESFb1Y+T15lOn\ndL/Ky2az8c1vfpurrroCRVEYO3Y81113Y2T/l7/8X9x447VMmzY9Mr12TzQ01HPNNT+loCCfX//6\nzh6frzssMSHejY8vZ860EXxlzthejCj7+lsdbCYk5tyQmHPDSjG/+ebr+P1+LrroqzmJ29IT4iko\nUlAQQvR777//Li+//M/I8ylTpnH55Vf2YURdZ4mkIHMfCSGs4PzzP8v553+2W6/9whcu6uVouscS\nIzKCZRzJCkIIkW2WSArB9RT6OgghhDj+WSIpyCypQgiRG9ZICgrSqCCEEDlgiaRQ3dBGdYP5ABQh\nhBC9xxJJoaMzwMdbDvd1GEIIcdyzRJfUPLedUUPMJ28SQgjReyyRFMYNL6a4wN3XYQghxHHPEtVH\n0vtICCFywxpJQVHI1RxNQgjxSWaJpCDTXAghRG5YIikoKFJ9JIQQOWCNpCCD14QQIieskRSAgOQE\nIYTIOmskBUX6HwkhRC5YJClI7ZEQQuSCJZICSFIQQohcsERSUBQFXaqPhBAi6yySFKSkIIQQuWCN\npAAyolkIIXLAGklBluMUQoicsEhS6OsIhBDik8ESSQEgIEUFIYTIurTrKaiq+ingASAArNY07RrD\nvtuB/wJqgdc0Tbs/G0HaFEXGrgkhRA5kUlLYB5ynadpZwBBVVafG7b9O07Rzs5UQgOAsqVk7uRBC\niLC0JQVN04yLI3cC/rhD7lFVtRa4XtO0Db0ZXFhw7iNJC0IIkW0ZL8epquo0oFTTtG2GzQ9pmna7\nqqoTgSeBecleX1joxuGwdytIt9sJTR14vfnden1fsdttEnMOSMy5ITHnTl/GnVFSUFV1EPAI8HXj\ndk3TakL/71RVNeU5mprauxkidHb40HWdurqWbp+jL3i9+RJzDkjMuSEx504u4i4tLTLdnrZNQVVV\nB/AMweqhw3H7ikP/l9CFUkdXtfsClO2vo63Dl623EEIIQWYNzf8NnA7MV1V1kaqqZ6qq+nBo372q\nqn4EvA7clK0g1+w4CsCKrUey9RZCCCHIrKH5eeD5uM0rQvt+nI2gkpLGZiGEyCpLDF6TAc1CCJEb\n1kgKoXkupJwghBDZZYmkYAtFKbVHQgiRXZZIClKBJIQQuWGJpGCTnCCEEDlhjaQwZgu24mOy0I4Q\nQmRZ1gac9aqiY9h9UlwQQohss0RJQQnYQQlI7yMhhMgySyQFdBsouvRJFUKILLNGUsAGtkBfByGE\nEMc9ayQFXQFFl4KCEEJkmSWSgqLbUBQpKQghRLZZIikEB6/pMqRZCCGyzBJJQdFtYJPqIyGEyDZL\nJAVQQKqPhBAi6ywxeC1QdBg7UNvY/SU9hRBCpGeNkoLfCcC7qw/0cSBCCHF8s0RScNWNI9Du6esw\nhBDiuGeJpAA2FEWamYUQItsskRQUbNLQLIQQOWCJpNDU7JdpLoQQIgcskRT00DQXQgghsssSSUHR\npfpICCFywRLjFJx2e2iZZiktCCFENlmipBAIhMKU0oIQQmSVJZICgdBSnNKuIIQQWWWJpDBx1MDg\nA5u/bwMRQojjXNo2BVVVPwU8AASA1ZqmXWPYNwJ4BvAAv9Y0bWE2ghw4qhaOgKPkYDZOL4QQIiST\nksI+4DxN084ChqiqOtWw7ybgVuCzwC1ZiA+A80efDYDe6crWWwghhCCDpKBp2mFN09pCTzsBYx3O\nVGC5pmlNQKOqqsVZiJGRhcODD2QAmxBCZFXGXVJVVZ0GlGqats2w2a5pWrj1tx7wAg1mry8sdONw\n2LsVpB5ZcU3H683v1jn6gt1us1S8IDHnisScG1aMGfo27oySgqqqg4BHgK/H7TLeuhcDdcnO0dTU\ns7UQ9EBwVHNdXUuPzpNLXm++peIFiTlXJObcsGLMkJu4S0uLTLdn0tDsINiYfL2maYfjdm9SVfVM\nYBNQrGmaaSmhd8hUF0IIkW2ZlBT+GzgdmK+qKsAvgW9rmvYzYD7wNJAH3JatIAEUWwBbYdKCiBBC\niF6QNilomvY88Hzc5hWhfRXAeVmIy5RjcHxBRQghRG+yxOA1IYQQuSFJQQghRIQkBSGEEBGSFIQQ\nQkRYLikcrm7u6xCEEOK4Zbmk0NTW2dchCCHEcctyScGn+/o6BCGEOG5ZLikEAjIpnhBCZItlkoK/\nthSAAJIUhBAiWyyTFHxVowDw67L6mhBCZItlkgJ6cJ1mqT4SQojssU5SIJQUpPpICCGyxjpJIRAM\nVaqPhBAie6yTFEIlBX9AkoIQQmSLdZJCuE0BWWhHCCGyxTJJQdeDoQak+kgIIbLGMkkhXFI43FbZ\nx4EIIcTxy3JJYVHVu30ciBBCHL8slxSEEEJkj4WSgnVCFUIIq7LOlVZKCkIIkXWWSQq6JAUhhMg6\nyyQFqT4SQojss86V1lBS6OiUsQpCCJENlkwKu44c6cNAhBDi+GWhpBANdX/L3j4MRAghjl+OdAeo\nqjoCeAOYDBRqmuYz7HsKOBloBZ7QNO25LMXJ/J+ezW2rggPXFEUanYUQIhvSJgWgBjgfeDnJ/u9o\nmrar90Iy5y1yRx7bbZIUhBAiG9JWH2ma1qZpWm2S3TrwtKqqr6uqekLvhhbLJqUDIYTIup62KVyn\nadoc4B7gvl6IJyljTpDV14QQIjsyqT5KStO0mtD/y1RVvTvVsYWFbhwOe7ffq7G1M/LY5bbj9eZ3\n+1y5YrfbLBGnkcScGxJzblgxZujbuHuUFFRVLdY0rUFVVRWoS3VsU1N7T94KmzMa6sLyDzl31Bk9\nOl8ueL351NW19HUYXSIx54bEnBtWjBlyE3dpaZHp9kx6HzmBt4DpwDuqqv4GOEvTtDuBZ1VVHUiw\nbeGK3gs3kbH6qLYjWROHEEKInkibFDRN6wQuiNu8OLTvomwEZUYamoUQIvssM3hNcoIQQmSfhZKC\nZAUhhMg2yyQFqT4SQojss0xSUBToPDi+r8MQQojjmoWSgoK/ZmjkuT8gA9iEEKK3WSYp2BTQ26OD\nOR765+Y+jEYIIY5PlkkKiqKA3xl5vnlPdR9GI4QQxycLJYXg/50HJqL7nKkPFkII0S0WSgqhrKDb\nQJH2BCGEyAbLJIUwPSBJQQghssVySQHdhmLTUfIa+zoSIYQ47lgvKQSCIXumfgRAY0sHb63ch67r\nfRmVEEIcF6yXFPRoyLqu89RbO3jpw93UNvZsam4hhBAWTAp6IBry0k2HWL/zWHC7FBSEEKLHLJcU\njCWFLTJWQQghepWlk4KZgK5z2d0fsGr7kRwFJIQQxw9LJYVHrp4HAfPZUnWC9Ud+f7C76nurD+Qs\nLiGEOF5YKinkuR1g90eet9sakh8sM20LIUSXWSopKIoCevRqv6vo1ehOaWgWQoges1RSCEpdBJBe\nSEII0X0WTArm4nOBIvVHQgjRZZZLCoGmARkdp0t9khBCdJnlkgIBh+lmPe5/IYQQXWe9pJBOKCtI\n9ZEQQnSdJZOCv7Y0cWOohVmqjYQQovssmRR8R06IPnG24Ri5kz2Halm+5VC095EUFIQQosssmRSM\nYxXyTl2Ec+Runlz9Fn9+Y3vMYau2H2Hr3ppcRyeEEJZl3mproKrqCOANYDJQqGmaz7BvCvA4wfvy\nKzRN25StQGOZFANCq7Hd+PiKyKbHX90KwJM3nZeTqIQQwuoyKSnUAOcDH5vs+y3wLeDroce5oSev\nG2pq7QSk9kgIIbojbUlB07Q2oE1VVbPdAzVNOwCgqqq3l2MTQgiRY2mTQhrGkkbKm/PCQjcOh73b\nb2S32/B68wHQzUoKSmyvI+N7hV+Xa8aYrUJizg2JOTesGDP0bdw9TQrGK3Eg1YFNTT1bLtPrzaeu\nriX0rukrh/y+6GyqkdflWEzMFiEx54bEnBtWjBlyE3dpaZHp9p72PqpRVXVUqDE6xTzWfau+uYPL\n7v6AfYcb+zoUIYTo1zLpfeQE3gKmA++oqvob4CxN0+4EbgNeDB3606xFGc+0oJB80Nq+w8F8tUY7\nygnDzLOjEEKIzBqaO4EL4jYvDu3bBMzNQlxdF5coyirqI4/DA9oU6ZIkhBApWXLwmqKYNV9kMr1F\nbFaoqmulsqqpV2ISQojjgSWTAqZJIb03lpdz+5OrIs9vfHwFt/5lVYpXWMf6sioqjzX3dRhCCIuz\nZFIINA7s0vFt/rbI4/1Hj8+SwcP/3sytf17Z12EIISzOkkmhK2Hbiqp5uvIhlPxoG8Otf1nJwjUH\nshGYEEJYmkWTAnSUnxzzXHGZj4OwFQR7HtnyolUrlVXNPLdwZ/aCC/H5AzS0dGT9fYQQordYLilc\n/d/TuO4bM9BbimO2O0oOdvucZQfquOOp1dQ3tXPZ3R+wy9BzqSceeWkjVz+0DIDFGyp58s3taV4h\nhBB9y3JJYdqJJZwybhC9ufDmS4t2se9wI/uOBNsb1pYd7ZXzrtOi5/nb2xrLNh3qlfP2F8++V8YP\n7v6gr8MQQvQiyyWFCKX3kkJ4HIPNFvu8t87bU52+AG+v3E8g0L9WlXt/bYWscyfEcca6ScHWvW6p\nZvYcDLY7GNd1fm/NAS4L3QX/cP6HPPjSRtPXvrJ0D5v3VJvuy2Rp0Ja2TpZuTF319eH6Sv7x4S42\n7jqW9nxCCNETlk0KgYbBWT3/4g3RC7U/oLNpdzXb99Vy3YKP8AeiCem1j8p54B8baWnrTDxJBrfR\nz7xbxl/f2kF9c/IG6fD7dfh6LxEKIYQZyyYF9MTQHaPKsA2owjF6B91pc/jPivLgqfXolBi6oQ7o\njeXl1Da2097hT3jt5j01HKltibm4p4pg0fpKjtW10tIeXMguVdWQPVSvZUxGQgiRDT2dOrtfcY7Y\nE3nsOzQefK4uvX7H/jogWHUUFjAkhUiiMHmtruv88o8f47ArPPGLT6d9r6ff0RhZUkDJAE/aY+22\n4Bv7/VKDL0R/5w8EWKtVcfqkISgWnHDNuiUFwN8wKOk+19itvfIexptzJbJN5/Xl5bS2R5arjiQK\nn1+npqGNy+7+IGa/kc8fPGl7pz/ypdEzaJW24hest3X6Ahl9VkL0lYVrKnj81a3s7KWu7blm6aTg\nqZyddJ990JGk+xRXC9jML9jxjCWFcFFhz8EGXl6yh1eX7Y3sMl6otAN1pueyDTxMWe0uHntlS8K+\nVNe58Lk/6Tmh0xfgx79fxHtrKvo6FCGSag61L7aZVDNbgWWTwq2XzuL2S7s3a7dnxhJc6hrTfbai\nalCiv0xjXX/4mvyHf24CghepsExuXt0TN/CH9U+wfme0F1EmF/pc3he/umR3pNdVfxP+vDfsrOrj\nSLqnroerD/algK5z99OrqZBZhTNg7bs3yyaFccOL8Ra6Ux7jHLsF++BQLyKbH8fInYRXDbUXmdzN\n23y4T16N84ToyGNjSWHL3pqYw5NVY3TnK1FWUcdld39AvdmFI4frQby6ZE/6g5JYX1bFgdCEg2u1\no70+a2tv//yvLtubswS4rbyGax/5iJ0V5qXI/q6ptZNV247wjw929XUo/V70a2rNak7LJoVMOIZU\nYCsILsHpGLoP58jd2AamGK0cmpJbcbdGNn20+XDSw40dhgIpigrJkkewl1PwK7ROC979HqpOXJc1\nUn2UhTuQNTuO8tpHe9MfmIGH/72Z20JTky94eUuvz9oa7RHWO+f7YF1iNdTuynq0/bXdPueh6mYO\n1yT+DsMJ8khNa8I+Kwh/86x5mcut3v6e5tpxnRSMbPndK/a+8H7yifNqGqNTcqdsE0hxfiXuQTgB\n1IXmYdL218a8fvu+7l+wzDz6yhZeWbo3MoAvVw4cbUqZSI0Wra+MqXpJ9apD1c3c+PjypI38RmZv\nf+ff13LPc+szisvMzX9aya+e+Dhh+5Y9wVKmVduFlF660nV0+vn7O1pGvx/RNz4xSSFMsSdp/HF0\nYCvu2gU3/IcO8NRbO5IfGPd3pBTUR3eE/tbi78TCJYaNu6ojf4dLNh7k3ufX8+bH+7js7g+obUys\naupuzxyzc2XL0bpWbntyFQtXp5++PBDQefodjSdeM/QmS/EzLt5wkKq6NvYfaUx77lz2YgqPeu9K\nUmho7sg4ceZKT6NZox3lw/WVaUfxHw96+zdX09CW/qBe8IlLCq7xm023u9U1uCeG7xCDv05bYS2K\np+f14vHTXdg8zYBOnaOccBtH5Fg9/kFUeGDcR5uDE+tt31eTcMyH6yu7FWO27mBfWZrYRhEe/X2o\npoWfPbiEO582b/Q3au8MRD6STP7YjB/fpt3VpgMO+0K6KsB/L9nD+2sr6Oj0c/XDy/jP8vLcBJZG\nb1eJ9K9UlyW98EMeqm7m7mfXsfdQA9c/upw1O3pnss5UPnFJwYzibo6suxDcEPzPPXklnmlLu3y+\n9buqwBG98z5am1iPbCuqwTVhIzWuYPXUrspg6SFhviQlui18MQ3/Yf75jcSpuJdtOsShavNEVn64\nAW1/LW0dPjo6Yy+S0YF5Xf8mv782eRfR1z4qT7pP16G5zcfuFFVXZvGkrI6LG2DY0tbJgy9t5Nn3\nykzfP5lkd2Urth5O+Oy6JE3yfWN5Oc++VxYZ6Z6se3Ou9cY9Q0DXIwNEAdo6fDz55nbaOo6vqqTI\n2KNeyAr/WbGPsgN1bCsP3gDmYsldyycFf83QHp/DfXJig6hxpbbINlcr9kHBu3R76X7c0xbH7vc0\noXiaWF+3kryZH0bGQtz8p7jzKzrYQ9NbKMG7/7qm0PQYcd+jt1fu56UPdwPQ0NJpdkiM8sONie8X\n8pun1nDPc+v5yf1L+GVcvXcgoHPv8+upD8VRVdea8AXsDPho7Ehsm3n2vTJ+8ejyFFHFUkyaLX9y\n/2K2lieWfEwv3Ck+AIXYW1pfaBT4x9sSx62k+hyvf3R5QvVS5bFm/vT6Nl4xjE/p6PQnrXqrb+5I\nOMf6sqqMpivJZEbcjk4/zy0so70nSSpjmQ+yTGbR+srI9PG6Diu2HGbZpkOsNPndWFkkgfZCSSHy\nl5LDopXlk0LHrlN7fhKTdgbPlBUJ21yTVuOaEJwt1TVuGzZPbAnAM20ZnmnLsBWFLm42Pzg6ot1i\nw1L8gjOaHbsXviHxF7KahvaYRuwbH1+R0Hvoz5uf5qZlv2HT7sRZYat7WN/Z1uHnnVX7Ux4TrT5K\n/vMnm4okPIo87owp3+9IXAnPFxonYZz88JGXN3Pdgo8SX1vTwjUPL4uZWBFgjVbFO6u6vxSs8ef4\neNsRFq6pYNH6SvYdTt+G0tcybbdatL6SY/XW7KUFRDuNJNmt63q//vksnxR6wjYw2R1KkvEHdpOZ\nUM2PjPznGrcF14mbggnC5JD493rz431pz27MCZ2+AA+FBtN1lXGpUJsteQVBU2snHZ1+tlQHG9Mf\nfGljykavcLfUdBJym+F52YE61uw4GndMtFHh6oeW8tHmQ9Q0tMUMIiRF3Xd8YkiXW32+AHc9s5aK\n0NiLxtDnday+LXJ3vnVvYukG4O7n1gGw+2BiibOppZNj9a0pe5LFT2my52AD97+4gR/duyjSphT2\n4ge7uOOp1VmeMDH4YcV/Zq8s3RO5099WXsNarecDC59+R+OeZ9exeEPq9rE1O45yMAfVKV2Vrqrt\nvdUHuOGxFdQ1tXO0tiX2+9sPfKKTgiP+Dj7Elp/krqvLFas6ODqjjw1c4zabbA1eDLfsrU55D2vs\nkVJ+uIENSdZZaGnz8cL7O5PcJRNZKhRS113//A9Lmf98bDfNziTnBCID2MIybRwz/sx3P7uOR1/Z\nEtm691AD/whVo0GwKu1fi3dz/aPLeeKVzWzafYxOnz9SfXTfixuob2qPOWd8ScS4r6KqiYVrYu/g\n65ra2VVRz6uhcRz3/yNYStxWXhuZqiRZw3G4Gi6ZmxZ8xL2Gz1TXdV4zVEvF+7+n10QGT0Z6MsUd\nky7Jbd1bk1FX0NZ2Hz+45wN2V0YTmh73f9hrH5Xzx1DPsN+/sIEFL5t35IDYzgz/+HAXf383sZ0n\nrLqhnb+9rcV0RS4/3BDz3X/0lS3ckmYszMFjTfxr8e6s9TRbsvEgR2sTx6VA7O/jwNEmrlvwEW0d\nPvaGSnUNzR3c9MeP+dvbKXoupil1ZMMnOinYBh4x7V2kOOL+cJTkF0DFnfiFUFyGu+gkv83wexxu\nPUTe7LfBGX3N/S9uTNmgZPyyJfuub9p9jLdX7efd1QdMq3vipbtw92Qcw6OGuZ5qGtqobw7+oS81\nWZ7UHwgk3AmHLQl1YwxXsYXvpj9cV8GDL23iX4v3xFx4jtbFFtET5qIJFzx0nd89s47nFsaOSQkn\nAePdr3vyCjwzF1IWSqLpem3tPFCfMGq6qa0zoaRVUdUc01ZhlGkJwPhdeH15eeR9dV3n/bUV3Pfi\nhozWCa+qa0XXYbFZt9EeXVy73lyt6/DWyn2s2n6E3zy1JqNuzEYPPL+e/6zYl9FFdeverpd0nnpr\nB/c+vyH3I8DdAAAgAElEQVR2o5LYZvbhugpqG9t57JWtkZLV7X9dDZC0o8WaHUejg2dz2Khg+aTw\nq++exhmD53XrtYoCnmlLk49dCHGdtBaSVB25TXonJS1pRN44+gu2h0ZYx78mVRdKY536+p1V2Ipq\nsA2oAvRI4/aDL20i/KV85N+bTbuG9hrFj2vSShR38kR22d0fsHlPNdc/ujwUW6KDx5p5f20lf/lP\n9MJl9rcQf5EMN8rWNbXHXKTjXxtfogoX2wO6nrZHUXjOH1thPYrDl1EbCCQmJgj2EIuvrYu/kzW2\nU9Q1Jil1JDnHko0HeTk0XYmu6xysbon0vqppyHw8ivH06boDv2Wo9lyy8aBp43cmKaHBZLGplz7c\nzeOvBksjq3cc7V4VVQbX1PteTF3SSSb+ZzVrHA7fwCRbpdHMH19LnOk5F2MfM0oKqqo+oKrqUlVV\n/xC3/SlVVVeqqrpIVdVvZyfE1CaMGsB3pn2Ri4Z9PWvvYR9QTd5p76M4EhODkula0fZo6cM1PnGW\n1Hhvtz2OfWg52PwJDdXGL9s7qw7gPnkVbnUt7lNWkDdroTG6yKNUXUN7Sslrwl5ci2N46ukylpmU\nDIxqG9s5ElcUD/+o9sEHUQpiq7jM79ITrpSRh5VVsUkrXBXx8L8240/Twn9H6K7O6MUPdqV4nY57\n+qJQsk5ks/XsfmytdjTSsyrs8vsWo+t6zEDK/33wP/z5vcROEwBLNx7ksrs/iCS8g8ea2VVRH/nI\nzD5f4zuWH47e4b60KFq199RbO/jd39cmvDaTsTBXP7ws5f7dBxtY8PJmmloza98Lx2usdtJ1nZeX\n7IlpU+uJptbO2IkzTX5OWwY/fKfPH9O+YPbdykV5Ie03U1XVmUChpmnzAJeqqqfHHfIdTdPO1TTt\nuaxEmAGbYmN0/vi+evuMuJPMyhr9ruqAjuvkj7EV1qITwDlqJ85RGq4TN8VUUyW7EEXHWoTuGNM0\n1HWXrbCW97YZ16wON6yHSipKd7tI6izaviN+EwCuEzfhOeXjmG3xd72KosTd3erRrr4ha/ft4da/\nrIz548ukei1d0gi/n5HN3YZzlPk0Kfa4okKy2oGOzgC/eCy2u+/R2lYWvLzFtPQXfxrP9CUcHfpu\n5Lmx59RfQ8kj3OX5lj+v5K5nEi/msXEG32FXRT2/eSr5oMP9ce1Kx+pak96YHK5pYbtJd+RUfv6H\n5OOH1mpHI3fk4Xh1PVgKqW/u4EhtK68vL+eFhcmnsOmqdWWJyX/l9mhHllQ5ITwJ5uX3LeaG0O96\nfVlsdW4u2xQyWXntDOC90OOFwJlA+LZJB55WVbUauFLTtPRdZ7JE1/Xgojt+R6RKJqdMLoR5py5C\n9wU/4qRzL+lK8GKqAI7O4Oyto7XQOQM4hu2PPA7zxzTyJq9vDo9r6G3uyStZ0Q7w+YR9ebMWEmj3\n0L7x3IR9qyPtFgEUVxt6R37wZxy+B9+h8dgHHsE1YSNtW89Ab/YGjzS5WsZfcIyMf3z/XLyb3ZXR\nu1mloJ4ndz9Oh386tY1TM/lRM/bgSxsZNijfEEjqP2O7XYHO4Pc21eJJZg3D4Ub+RpPfb7oGVV0P\nXqBXbI1O9BjfxTf6PBpXOImGP8+uTgO+51DyNql3Vh3gnVUHePKm8xL23ft85vNQ7ayoo/xwI8+H\nLvax59MjpZAvnnkCkHoSy64ydrwI/z7XalV0dPpxOe0pf8fhti5dj85a8Nv4Hnw5zAqZJAUvEL4l\nqQdOMey7TtO0GlVVzwLuAy5JdpLCQjcOh73bgdrtNrze/KT7Cwqa6NgRXHQnb/bb3X6fTCScXwmQ\nd/p7pscmNFonHBD9befN/CDpPiPf4F24BxyifduZeGYsTjxA0YPJJlOODvA5CV8EFHcLiruFQEMJ\nr3xUDujYh6bI93Fh2typxyw4Ru3EOWIvrWs+g31wJc5Ru1Cc0Tt6xdWGHqrpeb8L03as3HaEi84a\nF3luTAgASqgxX/E0s+tQinYfRzv2AdX4q0dk/N6bdldnVOIIC6+77XA5qahqYtmW0EVa8WMvOYi/\nahSgYLMnFubtKaqeBgxI/jcCwTaOP76xjd2GVcEqjjbHNIYXFQaXiHW7HZG/uR/d+2Fkv9ebT0FB\n6mnrAdx5LvYfbkQ9YSD5+emXxh0wIC9hm9mMs/HCMf4urkH/pcW7I20UxcXRz+U/K4LfZZfLYXpN\nSXWdSeZPr29j7EgvJ48dhMfjjGz//YsbmH/lvJhtqX6GZO8ffv2ry/Yyc9JQpk4o6XKMmcokKdQD\nxaHHxUCkYlfTtJrQ/8tUVb071UmaerjAiNebT11d8i9IYw4ndIuXd/q76Q9KojtzDtlGhqtYdBRX\nip/b5sM5bgud5ZPBb/yj1HGfsoLOfScTaC0kb+YHdOybhP/IWADcpyxHcfhoXfV5Xlu6B1thHa4T\nUnSbSzKxsmP4HvQOT8LF1VYYuiApAbAFX+MYGttoay+pRHG1smnngBTvm+j1FN06w5yjdvHn1zaT\nrPbUNWED9uJaWruQFBKlbpkNj3m49Lex3x3HyN04R+yhraUYvXkAAZOeR6km+6uuMW/sV9zN6O35\ngBKTECDxrr8hNPtvR4cv8jdnbL+oq2uhKsn7GN3/7FrWllXx5E3n0dSc/u/zUFUtzhO20bl/EuiZ\nt7lcfNMbPHbdOQnb31oRvZGprUuMd9OuY1x80xv8/idzGFQcXSs91XUmlWXrKxju9VBuGJtytLaV\n/ZV1dKaZysP4nh+tT5w2ptVQ7ffGsj2MLul64opXWlpkuj2TT34FcH7o8QVAZH4EVVWLQ/+rGJJF\nX7D6ur3uk9YZnoXu2A0Jw+49CgRilhHNm/1OkrPpoddU4Rh8GMcQw5fM5gNHJ7aCBhwjd6GEGsDt\nRdE63YTSjc28iio85Udc2EGOdpyjy4ID97pMwTV+M85RuyJzQsXs9TSh5PVsBG+qZGrWoQBHB/Yh\n6XsbRU+SWAWT0ctCvw8lVF2YyXQXRnc89bFp47Zn+tLUpb0uem5h8jEGYWtD9ey/fOLjlOuShL1e\ntgjH0P3Ykw4qTUbn3b2Lk/YQBPM2m3Ap4lB1S9qG67+/q0XaKTbsPGY6eDPc7mTsUNHQ3MHVDy/D\nl6Zb8cuGxa1eN5kE0Xh9y/b062mTgqZp64A2VVWXAn5gv6qqN4d2P6uq6jLgz8BN2QszvfBnFq4v\nPN44x5SRN/vduN5FSYQuSNGLQPQL5Tn1Q5wnbEt8ias90ssp/Fk6x24hb/bbuCcl9rwBIlN+dGWf\nUlBnCCfJBS/NddAzbRmeqYlTS4TZio+Fxn5kVnp0jCrDPXVp8KISs3Z3sOE8b/bbeGZ8iGts4udm\nxjl+I+6pqXvRZCqTBm6jY3mbcKtrUVwmkzAauj0rBfXYClNPFZ/q2tPRmfko3CM1LUlHfht1dyZb\nxd3K2xVv4xyTfAxGujXQ/xoav6EU1LF452YCus41jyyL9OT6cF0lD4TGrTz0r0387pl1Cefx+/Wk\nyUVP83EZE4HTrgSrpGe/nXjjRWY9mXoik+ojNE27Km7TnaHtF/V6RN0UzqR2mxJMXRZmL+rZQjr2\nQUfwHxsRWXLUNuAYeaN30rr+XBS7H8fg4F2b4uiI9GqyFdbjKtyEr7A2cicSU8LIiOFuxmRQn62o\nGvfJ0QSTd9oHdOw7uYvvYWDvxD74EP6jY3AM24O/cRB6szfS0UBxtaJ3pq/7do7YE4rnffROF3pn\ntKpNCSUWxZbhxdnmw1ES+4esFNRFGs67ymyG3VQiAyfjp1WJ4zkl2E21dVViZ4HIBVRRaOvwJQzq\ny9a8PTaTQV+ZCR6farxRqvmyAjo0hi7mnlM+5h8HPmZn5UXUN7exZONB0yVUqxvauOfZ2MSgo3P/\nixsSjgWo6sJn5rDbIr8/e+kB/DXDY/b3eUnBKsI3VMYsOq3klCRHH99c4zfHjG2whxYPssWN3rYV\nNMZcpAEcQ7s2YlQpqMMxrDz4WsPF0NjYrOQ1kjf7bRwjencAnXPMDlxjt+E57T2cY8qiXVZDFwDH\n4OTjImwDqlDyG1A8sT2ZjA3e3RE/VsNWWI/nlI+xeYOJyj15Bc7xsVVqNu9RFFdL3Ou6URvr6EAx\nVvU5etbOtnLbYZ57b2eoOkTHMWwvONrTTi3RXW5X6B61yxe99C9IVVLp6PQnnGF94HWcY4K9AM2W\nyMXZzh77Moy9/xQUypNMTNiVTggdJnN5GXtKbd1bk3TUf284bpJC5G7B8NvNdyb2ZvikcJ2YODIz\nPgH0Bs8pH+MoTd1DKFzVYx+Q6R9GhleF0N1Uwh1iqPrMMWxf0rYH17hteKYsx2HWThAZaKhnHIuS\n14i99AD2AebzUIXv4G2F9ThKjIMRddwnrcM9JXYsQviC1BV5Mz/APihYH2/LbwxO394Dre1+loUv\nPo5OnGM0XOO2dqnqqCscXRzQp7hbko+JUfzYB0e/l+ER0WYefWWL6W/ZuFZ7WLiXlnPkLhxDKlAK\not+v903W/O6O7ftqE3oPvrFqZ6RKsKGlM2bUf287bpJCdBRm9MPMxkL3IvtiG92JTkUOkTtuyGw0\necygQdO//MRNkVJOF74+nqkf4Rq3NdqzKkPhnydt1+UuUjzd60ED5iv6RU/cmwkhWG/uGBYsXTU0\nm08emYxn+hKcSVZSdIzYjevEzSj5qefscp64gbzZb1NRlavZVgNp23LMeKYtMe9+ngXHTVIIF68U\nBeYMn43Tlrpf8PFO70hfl24FtsJa3CdHB/LEJAxH+qoexdWOY8QuHKO6fuftOe29lHXziqs1WEWT\n0QjuZDMjGrbHvVd8u4ziak1sfFT8ST6H7vfGix/fYcasXjscn1kjt6lQVZdzjIZn1rvRUkkX2Itq\nTX/USJJNk8TC7WstGcweG8vsTQMp5/8CcIzYg3vyyoTqwnR6+6YhleMmKYQpisJ3Tr6EB8+9s69D\n6VMpxy/0I/aBqbsquicnr78Ot5WERRtIY/9gnaN24RxhPn4hfnyEkWLTcY7clXS/Z8biYBWNWRfW\nbsibFZuEHEPLY/aHezQZ1wFxn7wycdAjpB1RHTnMWL1m70TxNJk2rMaL781jH1yJvSRYXWNLUoVm\neDXOE7ahGJJZTFuIhQr49uJolahz3BY801Mv3xtJ9Gkm4QyzeZpjSspGZpMH9oaMeh9ZgbGkEJbt\nrlui5+Iv7L3D/ILoHN71hu5wHX2CXq1GMZzW2I027usbbTuJ7rAVZj6luaO0EsXdGhn5D8FBgoHG\nQeidLlzjtmDLb6I51CPJOWY7jmH78B0+gc6DJ5qcUcc+5AD+2qGmbVjJKPkNwWQcP65A727vo777\nO3eOLkP3OfEfG4GtoCudA8x+Rh3n2K34jpyA3h5sD1Vc7TElZaMerRWewnGTFEYMLgBg3LDiyLYv\nn3ghBc4C3t3XswY3YQ2KEhzY5ihNsnhSF+v7U3FNzHxOHiDj65bda7zLTnVxTHPhTFJSsBcn3nXG\nt+EEBXAMC45zcQzbZ5oUbN6q4NiNDMdvGIIzjdGeordYUnafoa493LCYWcI2lric4zcRqCuNjdLT\nHBzP4W5J6BZqLA25xm3Fl99I7Mj+ZL/w6Hb3KcvxHRsRmUkAdBxDKrDlN9JZMTF9/ClWS+yJ46b6\naMzQIh679hwmnTAwsq3QWcBXTryQn0y/zPQ144qPz4Fun2Seab0zaCwdu7fny06mE+kerARwTzM0\nMurB0ezhBlpzGVYfJUkeebMzmLqli6Ul+5B92Ev3Y08ynXh4XI1pQrP5Ql1sY/fF9DxTAEc7eae/\nazroK+GUhgZfR8nBhAGXNk8LnlNWBLc72wAdz6kfYBt4OGF+L2NPJaWgnrzZb6duUFaCsxrHTB9j\nuMa7JyWbgTb680tSyIDbZT7h3imDJ7HgvPnMG3lmzPZvTbo48rhj7+SsxiY+2ZyjyhLWg8iU4mzH\n5jE03oauBXbjIDlb/GqBqc4YvZiHSwNGntMTJ5Q0bbfoItfY7bjGbcM5OjgYzu5NMpuxzY9jtIZj\nVFnw4jrwCHmzFpI380PsJealwLBwW5rijFZNeU57F3tpYttRxmuhGI5VnB3m7UyGU9kHBdvJUraH\nmS4FHDxJ6hKtjnP8xmBiylL1+HGVFNK5eMIX+eGU/xd5bpxP5EtzpNQgusd0rqT4Y+x+w+C64Oju\nYC+d1H/YtqJqnAnVM3rc/2Avje8jn/yCl24xpFTXGrv3GPbS/SgFdbgnmo/ezXSGXsVuXtJwDCvH\nOXxvZKS5cfoKW0Hogmkyz5GtsDaxx5aio9gDOEdGFwHC3olz7Jakc3qZ03GeGCxJmE+Dr0R+bufw\n8pjXmfU0ij0meoq0lOAgUffEDVkrKRw3bQqZcNldzBgSnUffbouWLGaMnMC7xp5+KCmHxgsRlmoe\npmTCAwk7K80acBOPM2W4042fxVZJccFT0kxtno5rXAZtCI52CDggYMc5Zhv+hsyneo6vmjGbit0z\n8/2EbYqzMyFROcKLHNn8OIbvxjF0H4or2Gsn0O6JP0VK4e6ryZjdHNhLKnGN34Luc9C27gLz847e\nga/ipMy68hp+51nKCZ+skkLYOaPmMNE7nuEFQyPbTigezUPn/o7W1Z+hs2IC31T/qw8jFJ8UMXew\nGYpenFKUBlJ0tbWlGdDVU4qnhbyZH+KeFKw+cQzbn6Qxu+scQ/fjHLc54/l/wg3risOHc/TOSEKA\n6Iy0vUHJbzTtBh6ehDA8ziBS0jFwDi/H5j2adOJJI2OJUNoUetHXT/oqV8+8HIABriLOGnkGECw5\nKLod38EJzBkxO+Y1atFkLpnw1ZzHKkQytrzujVruzV5YZsLVPl3pLtsV6aZVyViaSQO7dCqTKTFM\nJUlEztFlkMG8W64TolVpHld2Kno+UdVHZu4669aY5/OvmENTayc2xcZg92Cq24ODU/7f1K+yrTrz\nUbH+2lLsA7PfQ0UI0T0Zz3xL9weDhhudIThdS7KZXG09mJakt33ik0K8wQM8DB4QrGs0Lq4+0OPt\n0hgZ3Z/5Rxtoze/2XZ8Q/ZqzZ+0X/UWqnkSpGJNJskFo/c0nsvooU+GG5tZVn0vY15VpuS8aHxoh\n2jIsYZ+/YRDtW+d0M0Ih+re8Uxf1dQiiiyQpZCS0PGbo/xEFwyh0BkdQXzvzJzFHzhwyDYAzTok2\nYn9+7HncO+8Oio/OJdBSGNnudQ8ITjkQcDC6sCfrAQshRO+QpJDCp4adBsDdl8cOeps38gw+N/bT\nTBo4kTHFo7h08je5cvr/cvPsa/lGqNfSqaHkEJbvzGNkSSHtW+Zyy+xrAbAp0Y//x9O+xxfHfSZp\nLHOGnx7z3N0YHVcxxJ1YAhFCiO6QpJDCheMuYMF58xniDS/WEywp6EBJ3mB+duoPcdoczB42k5MH\nn8SIwmAJYsF585lROiXhfN+7cBK3/M/pDC8MXsQ/M+bcyL6BHi9fCCWFkYXDOW3I9JjXflO9GKct\n2k4xoi2aqE4dNCvhvb5ccml3fmQALp/2vbTHXHbKt3HYpElKiOONJIUumDV0OnNHfCpSgugqt9PO\n+BHBCfsWnDefs0edmXDM/Hm3c+Osn1OaHzvYx26zc/6YcyLPL//yVC4YfS4Ag4sT10743LRTuHjU\nNxlbODay7bYzbjCN6+zS8wm05UeeTxmcft3k04bOoDRvcNL9nQfST+glhOh/JCl0gcvu4tuTvobH\nkb0FbAqc+dhtdj57wqcB+NqEL/HDqf8Tc8ylk79JYZ6T8d5gFVKhq8C0TeL8k2bys5nRyQCH5JuP\nKp0wZCjXTr8q8lxRFH5/9h1pYw3owVGzc0fM5uZQlVhkX1tB2tdnSiYuFCJ3pPyfRQ6bA1+ge6Mm\n3XYXC86bH3s+JfjrKnIGG6unlUzmZzN+iDpwApMHqfh0P09ve5GpJdE7fY/Dw+1n3EizL3FFKIdi\nx6cH+01PHDmQR0bcwwBvHg31beQ5outb/2r2Ndy16oHI86H5QwDwB4Kv/fakSzjQGJzga0bpVOaM\nmM19qyrxHT4hMuHaOcPPZvGhJcl/3o5SWpoxHdsxyONl1tAZvLTz1aSvNypSBtOoZ75QevvOGcnn\n8RGin9J1PWb54d4iJYUsWvCFO7lr7q3pD8zQ+WPmcdH4z6MOmgAE7+gnDZqIoig47U7yHB5+PO3S\nhNHYpfmDGVs8BiCmHeDCcReEzmOLnM/Y+A3gsjkZWTg8pp3hutOCPa78enR+nYGeAQCcNfJTnDJY\nBSCvehp5juCYj3PGxMYEEGgawKl553HNzCsYVnM+HbunJxwTjvnc0XNTfTQxzs77OneceWPCdt/R\nUXxl/IWJcdSmbqjvPDg+4/eOl67dRfebz+ybbYHm4vQHiX4toGdnoScpKWRRoasAn7v3MrnL7uLz\nY8/r0TluP+MGtlbvYGrJZPIcefgCfqYnGXNx4dgLmFIyCYCpJZM5Y9gszhxxOgXOYPvDT6Zfxoaq\n4Ipb4Qb2sAKPg4vPPpFpk6ZxrLWGYnfsRegzxd/mtbVHGD/qFCZ4R9HeeQwCDu6ZezsVzZU8vOFP\nkWMvGh87TuQbJ/0XJ3rH8tbehayv2szo4hEcaIhORVzgcVGSNxiX3UWHPzp1wOdGfJHhhcHBRKOL\nRnKgMThdwnXfmMGjexOniw4LX0B/MOW7/GXLM0CwTei+1Y+zpzH1am73zruDlWX7eeHQHwFo2zKH\nQY4hNJSswlFyEH/NsJhpG9q103Cra1OeszcEmouxFWR3DqSecNtdtPt7Z7nJ4qozaShd0Svn6k9a\nfK0UuQrTH9hFkhT62O3fPz2ny4YO9Hgjcz0BfGn8Z5MeG7/v/03+eszzEYXDGFFofpf98NVnRx57\n3cFSxOXTvse/d75BsbsIr1IK/ppI8fdb55/Ea8v3UujOZ6DfmxCz0fCCIYwsHM7pw05lfdVmfnza\nd9lUWcY/yl4BYN60UPtKaGr0e+fdQbu/nYEeb6Sa68Kx5/PE5qcBOGXcIAjNJj2jdAobqrbEvF+g\ndhit689l5nnTqBx7Pm+XB2fonD1iOnu0Pfx2zi851HwEl83FxIHj2VNfzn1rH+W6036Ky+5k3skn\nMu/k+fgDfg5Pa6M438lVCxrQW4r45ozPUOPawYeHFwbfqz529a/hBUM51JxkSdAe8Dgd5G4p+K47\nY/jpLK5IP/vsnOGfYvmh5KONfUdGU+gbzdimk9hU+LfI9ny8zByhsuxg90Yq9wfhm7PeJkmhj40Z\nWtTXIeTM1JLJTC0JLmZU39TOWyv3cerEYOP3hFEDuPbrMwAY6PbisbuZMWQqE73RqpsF582nqqWa\n0vxgr6fppVO448ybGDdwFLuPHIgcF549MjwiPd+ZR74z2EYyumgE/zfnVwz0eLlh1s+obArOTXPb\nGTfgtDliEtCCjX9hZMFwxo6eFjnnReM/Fym5nDXiDKYMPpmBHi+DPNEV/8YPGMv8ebcn/NHabXZG\nloQa4P0uxjtP5bwZY4AxzD1hOoeajlJd7KVgRBHP7ngJgJtnX8ux1hrWH93Eq3veSvrZdlaeGDPj\n6sUTvsS/d70ReX7igLHsri+nyFnI8MJhfOfMS7htxd3BNQAM0zF/adzneGPvOzHnNk4jb1Ns3HT6\nVTFtTKmM5BSaGhXqi2IT7VkjPpX0gvz9yd9i0qCTOHvkGfx25X0pz/+dk78WSQpm090HWosYN6aY\nxpZOWtefS96piwg0F3HWwG8xamBDnySFjt1TM17T2l9XErdEa9DUkskJVb29RTEuNJOMqqoPALOA\ndZqmXWXYPgV4nGAH/is0TduU7BxVVY09WpzA682nrs5a8wNJzLnh9eZTXdPI+weW8OnR8yLjOdYf\n3cxHB1dy5Yz/7eMITTjsdLR24HKatylsqtpKm7+d2cNmRrZtrymjuaMZr8fLA+sew73/TCZOb+Rb\nk7/CtQ+sxTlmOyPHt/KLWT8lz5FHWe0u/rD+CZw2Jw+eeyf17Y247U48oXae5s4W7IoN7WAVrx14\nmcPtFfx8xo8ozR9Mh7+D2954Afvggyw4/26u/DDYRmOsIlx1eB1+PcCGo5sZnDeQxRXL+dqEL3HS\nwAn8bvWDAHx30n8zwF3Mgo1/ibzuB1O+y9GWY7y+J7bKbqJ3PL6Aj+tnXRmzfeH+xVS31vAN9b+Y\nv+Zh9jVEbwAWnDefFYfWsL1awxfwsfHYVn41+xoa2lp44J13GNA0jd/97xya23y8s3I/Z04bxG1/\nXs+vvjuL8SOLeHb922yo2kKbM/HCG3btzJ9w/7pHI89/Mv0H/PXjd6k/NICSSfto6GjkC2Mv4M3y\nYGlv/ICx7Kkv59TSqayvir34T/SOZ5rtC7xa8yd8pJ9kb+6wOYwqGsaLO/9NiWcQx9pqIjGd6B2b\n9vWplJYWmVZRpE0KqqrOJHjB/6Gqqo8BT2qatjq072Xg5wTX93tU07SvJDuPJAVrkJhzo6cxt/na\nY7pGX3Z3cLnMJ2+Ktjk1d7Zww9Lb+Z+Tv8GnhqcfW9Pp78Rpd0aef7Cugu3ltfz04qkcaKykxFtM\nnj+zku1PP7iBYQVDuWX2tSiKwr6GAwwvGEZV6zFGFAwjoAfYUbuTccUnsKRyBcPyS2MWwEpmX8MB\n7l/3GM3bpzPIM4j534/OAtDY0cTKw2s5f/TZKIrCh+sr+fSsMeBPPUX2lmPb+dPmv9OyR2XmpEFc\ncuocntz6LPsbK/hS/k+58IwTuHv1HzjQWMl1p/2E8QPG8vsX1rOtvJbHrzubDr2dQlcB96x+iP2N\nFdx39m9p7mzB6y6m2deCTbezoWoLz5f9kwfOuROX3Ymu68xf8xCjCkey/FBworyLxnyZc8bO4lhr\nNaNKSrnyrVv4xawrKXYVcevy33HNzCuY4B2HP+CPWSCsu3qSFH4CHNM07R+qqn4NGKlp2kOhfYs0\nTTs39HixpmnnJDuPJAVrkJhzo7dj/vHvFzF3yjD+5/OTYrb31gUEuhbz0ZZjeN0DcBmSTG9avuUQ\nJ3WYMFwAAATuSURBVI32UjIgL+VxXYk5ENBRlGAvPH/Aj18PROJv87XR4muNVBPWNbWzZsdRLpg1\nOvJ6f8BPQA/EJNZM+AN+1h3dxKyhMyJtbLn4TidLCpm0KXiBcBeLesDYVcVYqZW71lIhRIw/Xn+u\n6fbeSghdlWygZG+ZM2V4r5/TuJKZ3WbHTvSz8zg8kWo3AG+hOyYhmL0mU3abndOHndqNiLMjk6RQ\nD4T7ExYDdYZ9xrv/lJ1mCwvdOBzd/4La7Ta83uy0tmeLxJwbEnNuSMy505dxZ5IUVgA/Bv4BXAA8\nZdhXo6rqKIIJIWWn56am7q1cFCZVBLkhMeeGxJwbVowZclZ9ZLo9bZ8mTdPWAW2qqi4F/MB+VVVv\nDu2+DXgReAn4de+EKoQQoq9kNE7B2A015M7Q9k1A5vMPCCGE6Ndk7iMhhBARkhSEEEJESFIQQggR\nIUlBCCFEhCQFIYQQERlNiCeEEOKTQUoKQgghIiQpCCGEiJCkIIQQIkKSghBCiAhJCkIIISIssUZz\nsuVA+wNVVccCK4HtQIemaZ9VVfUXwFeAfcD3NE3rNNvWB7GOAN4AJgOFmqb5zD7bTLf1RczAKOI+\n79Bx/eYzV1X1U8ADBGcPXq1p2jWZxtfPYq4H1ocOuVjTtBpVVb8D/BSoAb6taVqD2bYcxTwFeILg\nRJ27gMuA++nf3+f4mH8DfEw/+j73+5JCaDnQQk3T5gEuVVVP7+uYTLynadq5oYQwBPi0pmlnAZuA\nr5pt66M4a4DzCX4JTT/bTLf1Vcwhkc879HP0t898H3Be6L2HqKp6Tibx9bOYpwKbQ5/zuaGE4AQu\nB84G/g782GxbDmPWNE2bE/peAsym/3+f42MuoZ99n/t9UgDOAN4LPV4InNmHsSTzaVVVl6qqeg3B\nu49Foe3heM225ZymaW2aptUaNpl9tpluywmTmCH284Z+9plrmnZY07S20NNOgqsVZhJff4rZD5wc\n+pzvVlVVASYSTBQ+Q3xm23IVs/FOuZ3gzUN//z7Hx2ynn32frZAUvEQX8KkPPe9PDgEnAZ8muAjR\nLBLj7a8/g1lcmW7rKzGft6qq0+inMYdiKyW4WqElPudwzJqmbSN4wT8bGAhclCS+Po1ZVdUvq6q6\nBRgKODOMrz/FvJ5+9n22QlJItRxon9M0rV3TtObQndIbwG4S4+2vP4NZXJlu6xMmn/eUJPH1acyq\nqg4CHgF+0IX4+lPMaJpWo2maDrxCP/2cNU17TdO0KUAF4Mswvv4U8xf62/fZCklhBcFiIQTvxD9O\ncWzOqapqXNNuLsHGo3NCz8PxrjbZ1h+YfbaZbusTJp/3bsw/3z77zFVVdQDPANdrmna4C/H1m5hV\nVS1QVTW8qHr4cy4DpoS2h+Mz25armN2Gpw0E14zv199nk5h9huf94vvc75NC/HKgmqat6uuY4sxT\nVXWtqqrLgUpN01YCS1RVXQbMAF7RNO1o/La+CFRVVaeqqguB6cA7BIvbMZ+t2efdl78Dk5ivjf+8\nzT7fPv7M/xs4HZivquoi4MRM4utnMU8DVququgQYDfwzVB/+J2ApcCnwR7NtOYz586qqLlZVdTHB\nqpi76effZ5OY/f3t+ywT4gkhhIjo9yUFIYQQuSNJQQghRIQkBSGEEBGSFIQQQkRIUhBCCBEhSUEI\nIUSEJAUhhBARkhSEEEJE/H9biMoOFH71XQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fabc40f2490>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set_context(\"paper\")\n",
    "plot1, = plt.plot(maxout_mlp_loss, label='maxout_mlp')\n",
    "plot2, = plt.plot(relu_mlp_loss, label='relu_mlp')\n",
    "plt.legend(handles=[plot1, plot2])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fabad939f50>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEMCAYAAADeYiHoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3WmAHFW58PF/9TJ7Zkkyk32yTir7SjYgZGOJIiKgIKIS\nAQOIgmzC9VURRQRUFgW9gHK9guK9FxUVEQxgIOxLCISEVPZksq+zz/RW9X7ovbt6ne6eqcnzy4f0\nVFedeqqqu56uc+qcUgzDQAghhIhl6+kAhBBC9E6SIIQQQpiSBCGEEMKUJAghhBCmJEEIIYQwJQlC\nCCGEKUehV3j4cGu37qutqCimrc2Vq3AKQmIuDIm5cKwYt8Rsrra2n5LoPctdQTgc9p4OIWMSc2FI\nzIVjxbgl5sxZLkEIIYQoDEkQQgghTEmCEEIIYUoShBBCCFOSIIQQQpiSBCGEEMJUyn4QqqoOBZ4B\nJgEVmqZ5VVW9DzgJWKtp2nWB+eKmCSGEsK50riCOAcuANwFUVZ2FP1EsBIpUVZ1jNi1vEQshhCiI\nlFcQmqZ1AV2qqgYnzQdWBV6/ACwAvCbT3slppL3Mo+sf55Shc5k0QE09cxo+2n6Ulz/YxzXnTY17\n789bnqG2bAALhy3o1jrcPjc/f/9RLp30eWrLBgDw+Mf/y+QBE5hVNy3tcp7b+SKv7n2LWYOmcf64\nT6W1zNHO4zy24fd8Y8ZXOdhxiEfXP05VcSVTBkzkjY37uGjS2Uwe3T9uudvfuIdDnUeYWTuVK6Z+\nKek6DrUf4YF3H8Ou2NnWvINfLLkLm2Kj2dXCw+v/m/HVYylxlLB81NLo5TqO8LuNf+TamVdSZHey\natdqvLoXm2LDgND8H20/ysvr9nHN+VO5b+2v2Nq0g7sX3kapvYT/ePWHtHs7+NToM1k0/GQeXPcb\nrpy2gj9t+RtzB89iysCJADy99VlW7V5NXdlAyhxlzC/7DBsObMVWu5uVU78cF8sbOzfwxPb/BuC6\nmSvZ1rSTD45soLF1L/0ZzjH2AHD+kC+zbOIUXvlgH5sPN9JSs5bq4kom9B/Pk9qf0A0dgGEVQxhV\nWc9r+97i7oW3UeEsB6DD08nNa24L7ZPThp3MWwfeZdKACdTY63hp/wuUtY+mfkgZziKddreL7S3b\nGVExjM9POI8nN/2Zo13H6Oes4Jyxy/nNR08wlvls8/+u5Jwxy/nwyAZ2tTQyrGIIe9v2A3Dj7Gv4\n2XsPRR2PU4bO5bV9b3PSoBmM6DcMwzB4etuzDHM0UFzu4RszvsrhziPc+fZ9oWV+vvjH6IbOz9c9\nwkXjz+OP2l/oX1LNe4c+wIYDz9aZXLpwEXPGD+QnT75P9aSPOXnEDKbXTgHg1x89way6aby4+xWO\nt7hp5kDSz1qkstYGblp0MU9sfZztzTtB8Q8WcXr9Ipw2By82rmFQ6UAa2/ZFLWfDTrleS6vNfF3V\nxVU0uZpDf48y5rJT8e+Xdw+uC00Pfs7zIZuhNqqB7YHXzcBk/AkidpqpioribvUOtNttVFeXZb18\nrqw7vJ6tzdt5+FN3p5w3nZgf+fsa2jo9pvO92PgKAOdMXpZdsAEfH9nHjpZdvHH4Lb48/bMAvLn/\nXd7c/y5Lz38w7Zj/vv15f1y7X+Gyky5Ma93/3PMvdrbs5oh+iL/t/CfHXU0cdzWxs2U3lMJv/jGG\nx75zRtxyhzqPAPD+4fUp9+Hv1z/nLy+gorKIInsRa7TX2NXSyK6WRgA+PyM6qf1px2vsaNlNi3Kc\ncdWjePqlZ6PeD87/6DNraO3wH6OtTTsA2NaxjSm1Ku3eDgCe2fEv6gcOYVdrIxtaPuK9Qx/w0dGP\neezcewFYtXu1f7s6/NulvbuGolEbUA67qa4ui4vlqa1/DV3nP7n5zxxqPxKKK5gcAP685R9csGAu\nv/3nJpxj1+EInHTeO/RB1LbsbdsfOjnv6NjOwpHz/HHs2RQ13yt7Xwfg/UMfhqZ1lO9gU0v0Pm9s\n28uTm//EnkCZnd4ufvPRE/59E0gOAH/f/lxUDEFPbv4TsV7b9zYA7x5cF3Ui3OvdAs3QbmvmL9v/\nHrWMUeKh3d3B9uZd/H7z/7G7ZS87WnYBoOPFVv8Bj/51ADNuWsKWPc2UDv2Qj5o+5PeBz/37hz6M\n2tZMdPTbwssbdrDdtQMiBq14YffLodexycEfly9hcgCikgPATiW8XyKVVjgodZZkE3pK2SSIZqAy\n8LoSaAJ8JtNMdXdckerqMpqaOrpVRq7oup5WLOnEHHz0a7L5urvdLS3+5b3u+Lhj/053P6cbU0en\n/7h3trvxefW499PZl6ne1/Xoco83dVBs99LZ5U5ajsvlBaCtrYsmW/w6ku2r9nYXLcWdUe93tvvX\n19Hp/19BSRy7Quik0tTUERdL5BOBPV6veRkARsQ6jIRD60Rp73CFluno8KS1jOmq4w9n2ny+zBdu\nbe3C54uZ1tJFmydwHMyKVILLRh+rXJ1LulxJjk2eNTd34HJkfxBqa/slfC+b65I38LdJAJyOv23C\nbFqfZ5C753kX4tHgPsP/rcrX5WgywSoOm2JHIf4ElovN12NK0dM+c/mXM4srkqKYxG2yjuB84ee9\nJyvXCFVJRJURWia8bLLtifr8pJkgIhdJbwlzqfZb+lGkzxazzshjY1fMaiiM4IxZrS+VXJ4LepOU\nZwpVVZ2qqr4ATAeeB5z42yTWAD5N097WNG1t7LS8Rt1LGIU4q+dQ8ARjtxV+ALBwglCy/pKm2t+x\nJ9DQ3wkW8+heDMNImZw9uv/XoVnUOnrcyUEJJOBgvIqihMqIpShGTIDh1/749Zi/E1AM/zoU3TTh\nmIrc8G6cOLtzcsx2WbcefcWjYAuVZfoDSIn6L+d68lSQz1UrhT7JdXe473xVMV3z0reYXTedy6Zc\nYvpe0P2LfoTT7uSal75Fkb2I+xbdEXrv2R2r+McOf1v95AET+Nr0ywB488hbPP7hn/jFkru44eXv\nMrJyONfPuhqA777+Y4psTg6+MZcOl5fHbl3Kza/cxqCyWkb0G8Yre9+Ii+dr0y/nlx/8hmUjTuP8\nhuj69HcPvM9/bXwSgO/Pv4V/7FjFOwfXordVYatojisrmbmDZ4XrrD0l3DDvMu5d+8uoeW6cfQ1j\nqkbS2LqPu965H4AiexHDK4Zw4+xrovZdJmZXL6C2qpTndr0U995dp36PW1/9AXpXGbaSxJ+FBXyJ\nLy6dmnYM84ecxN62/TS27jWfwQDD58BwlWErbzGfR4gC+/LEi5g3ZHbWy/ep4b7zKbZBz0zUL5eY\n5Prm/ndDrzccDTf6vd4Ynu7RPaEGToBjXcc50HEoqpwObyc7WnabJgeAj49qQLjxOtKHRzaGXh/o\nOMg7B9cCZJwcAN4+sDbcoOjsYltE3EHBabtbG0PT3D4325t3Zby+SGub3uLVfW+ZvtfibgVImhwA\n3tgQ3zCYdJ0HP0icHAAUUBxeSQ6iVzGr+swVSRDdkOnlcbKrtXxcx3Wvbti0wDjhy/rcVlsZgX9m\nzOuYTaRb1SKEhTls+XvumySIFJKe1Huo4rGnGsRME04glNhGw+4zEmbNdBNExvspj7/EhMiX3H/3\nwgr+yNFC0g2dTce2UF1cxdCKwWkts6/tQNS8sSeZZlcLnd4uALyGj7UHP+TgsS7GVTZwtOt41Lxv\n7X+PhqqxuNz+u4eCVSMAPt3HG3si7mcubUJxeHl0/eMpYzzSeTT0+p2PD7K/6H0OtbTQUNUQVU22\napt5FVW2nt72bNy0V/e9yeSBE/jLtn/EvffX99/r1vqCfQtiPfRe6n0E4KvdzA9e09Jen9vnTj2T\nEL1NHn/Y9OlG6ud2vhjq1PXQ0nuSzhvZkBk5r27ofOPft6Zcl2f/KJxDdpq+Z9ZAfEb94lCnqe5w\nbZlJccP73S5HCGFNK6deyvTahH2TUzphG6kPdxxNPVMK6SbQVA2msWIbprOlFHXlpBwhhDXls2K0\nTyeI3ix3V27SECvEiUzuYspSLhpz89cgLCd2IUT35fxuxQh9upHajGEYPL/r37R72jl56Fw2HN3E\n6fWLouY50tzJz9f+mqNGY4JS4tlrElcZmfVB+OjoJpM5M1c0MjflCCGsKZ8/NU+4BHGw41BoZMmX\nGtcAcOrQeVHz/OJP6zk6Kv3kIIQQPeXA0Q6mDsxP2X26ismMz3Q8m+hLNLfHZzKPEKIv8B4ckfWy\nhp7/U6bv2KCM5u/OaLqpnHAJwqxxOLaRJ3ZUUCFEH2L0rdOekcc2iL61p9Kgmw4WH81sCGchRN/Q\nvauA3vfjMZ9d2fpUG8THxzbzXx/9gR+f+l3sNjtvHQj35L3mpW8xo3YK6w5/FLfcR/u3R/3dPv5v\neY9VCNFDvEU9HUFShi/Dcc3ymCH61BXEv3b+m3ZvB27dfMgEs+QAsGrHa/kMS4hu8+wdG3rtPTAy\nJ2UaeuHGnvIeDQ9f49k7FvfOSbi3TwnF4GutDr2fqA5ed4Ufq2m4i/HsVvHsH5VRHJ79o/AeHImn\nsSG8vpaajMpwbTop4XveI0NNp7t3TMK9LfFz3z37R4fjOToU76HhSWOILEuqmDKW2Q7L533EQuSC\nd2/4hOZpVAHQ2yvj5nNvn5J2me5Nc5K+r3dUpF2W6fIRJ3TfMX+C8DQ24N3bgO9QPb4jw+l69yw6\n316Od++4cFyBk5/hcUYlAO8e/z4wvE661i3Be2A03sYJcet175iUMCZvowqGDe/+saH959mZeJ91\nbZgfv10t5rcM+Vpq8GyPTwJ6eyW+w/X4jg5JuB7f4WHhPwwlaUz+dfWPmD/prN3SRxNEpiRBCAtJ\nekLI5LOc38+9YjrcejrrTDBPToZvjyg7+GjWpFdSOdifRor3oXsN52k+YjYbfaINwjAMOr2dtHra\nAGhpd3HI25piqbAWvftjNglROMlONIWLIhPmySJyBvNkErVc2gkizWdy68FHwyY5OWeyPxPOm0Y8\n3Thueh7bIPpEgnipcQ1/3vpM6O/bVv0ae/+DaS/f5MvNwHlCFJKvZUDo6XaGpwjF6UbvKo+bT++o\nwFbWFjfd0JM3huodlabLpRKMxdc8EFu/49hKOtBdZYEy+5kv4yo1mQh6e1X4z66ylOvOpFpMbxmA\nvfI4JNsPMQ3aelNtkhLNE0FUdVAChs+ZspxESm3dqwpMpk8kiM3Ht0b9nUlyENbRuXYppbPin1Ft\nOu/7i1Ecbkqmvg6Ae+ckikZtTLGUOb29MuFjRl2b5lA84Z2Ey7q3T6FoTPTNEV3rFlEy4+VQnKUz\nV2N4nSgOj1kRJhQ61y4BbxHe/aNQirowOvuBww2eEgx3MUqRKxzjxgUUT3gLW0XMNvjsofWb8R0e\nhqdxfNT7ne8v9p9QFR10O59eOJy/vbbLvz6vw3+iMxSw+cDn9P/qV3TwOf3LekpM12V0VWB4HSgO\nb1SVjO/oEDqDJ9gEy4Zie28ZGDbsA+IfNau3V+LSop/b7N03Fu/h4YHYloDP7t8O3UaR+g620g4M\nTzFd6xYFrjYU8DlC61Ls3tBxjOTeNpWisevxHqjHs2+s6V1T/uU9YPdheJ3gc+I9MgTHwP3hcrZO\np2hc+Pkusceq871l2EpbqRqWWSN7JvpIG4S0IfQGelt8o2lOZXJ7oqcEI/KE4nVieLK7vdH0121A\nql+shqc4fpo7Iq7A+4Y7fr6kvMWAAt5ijI4qfx12YHvjrgx0e+KrhaQnXcW/HyPr6D0l/hO/txh0\nB3MbRgZiqMRwl/nf0x3+9w2bP5kEfx2nOMGb7atgDFHLJqpR8TkTXg0YnqLAPjMpG/zHQXdgdJVj\nuKOPt+Eu9S/rLQq3FficcfMF2wKC+9rwlISPU1R5RYHly/yJPRhDzOfb8Dqj/o7bfz4nelt/fHr+\n+m31iQQhT4rsJXr7ceil9fM9JgfHKz8PHOtFH6SMNs8fd6jdJNtdk+Fyui79IBJaf2Qj64983NNh\nCKB3n4ENsj3xGEnvEunuySwPJ8O0D0MOhsPP4yFPvt8LJYMY4vZFomVTlZnZdvskQST2nx/+tqdD\nEAFmDaSm83Waz+fZrUb/3Ti+ewH5wk1selv29bRRVULJ5ktwQvM1DcQbeZ97bJWDAd59Y6KmmXWG\n8x6sT6tTl3ffWAxdQW+ryqhTnffoYPS2qtA69M7o6jPvofhB7gZUpbdv0lp/IO5kDK8Tz56GpPPo\nLQPC8wfuVKq3p98/JBOe/aNC1YzeQ/X+9bf5O/35TBq0Da8DT0Sfj0jeQD+J4PdD70hSZRtRHTWo\nJnXjfbb6RCP1icC9cyJFo6KvlLyHhuOo29NDEUVz75qIrST+jhfX5lkUj18L+L8w9urDuDbNpXTm\nv6Pm69owH6O9Gme9BkDn28sBsFUdwV55LK7czreXUzr3uYTx2G0KPt0WKid2WaWshZIpr6e3cT6H\nf5nSVkqm+nvdmzVcuz5YSMmMV/AdGxS+UcJQcG/297x11O41bWvoemd5IK7oXri6q5SikZtwb58K\ngGdX4g5gUeEeHYbv6LDUM8bwbJuR/P2d8c89Li3O3SnEd3QovqNDSXZl07V2WcpyZtTX837Mcf/2\nrUupvrSM7z/6Bh9uS3xb+11XzufWh98knMSTJyxv4wS8MU8GMNyldL69nEvOGM/vV22Oif/0hGUZ\n7dWhz+tjty7lsrteSvw5j7g1d9zwqvj3c8TyVxAnjJx0EuoJRvxr003JbXVCwdqlCnFYcrIOkx3S\naz9T6Z2cE7ElOfjJ3kvn/Uz0hbZRSRAid9L+QhTimxO/jvzUaStxL3O1nnAnsT5wpslGlvmrO2kv\nl8937gtHTRKEyJ2Ib6Z546X1egADEUMZ9OYgU7Bw6JlKdmeV3Zb8tB3KDzlI8kqKdVmBJIheLHKE\nS9N78bsxrr1Z42XsL9/IRuPgoGmRMcXy7G3A8DhxbZyHa8PJuHdORG/2N9T5jtfi3jHZPyqp14l7\n+xRcG+fFDW3s+niO6aiX7q3T/e9vnBc3IJ3vWF14vh2TcG+dTibfTfe2qaHX9sPjqTo6D09jg79d\nxGfHezCwrxKU6dk/mjHlKoarlFHKTMboCyL6XJgv1PXRAtw7k7cpeA+NwLN/VGigOzMrPjGBC5eY\nN3o6HeHPh2fHVDx7wvN59o3GCDRCu3dOouujBXHLz5+c/MlmXz0nvTaR0UOy7R+T3kF0bZ6Ja/Ms\nAIYOLKe6InGfki+emfjGh9OmD6WmXzELJg9idtEnTG+SOOOkEXz5LJWrPxP+DH5uydi4+QDmTczs\nyXAAN1w4PVT2p04eBfgHL3R97B9YsWvDgqjBCK86N75dKJckQfQyhtcRuvPC/XH0SJKRJ0Ig9P1x\nbZyX9O4Ps4Zaz+6JcdO61i2KOll6D4wOxxS4k8cXuFPDlLeIrveXobfVYHRU4js0Egx/Q7F7y2zw\nlARGJVXwHRmO3lYTd/LTWwcEGiqj+Zr9I2jqbTX4jkQPhezZE/4i+w7X4zs2JMG5JX7irIGzQsnX\ne2AkP7/oCu783AV494/FaK+m670z/B2/EjEULp32GW6cdzmP3bqMm5dczC0XLcDeHLjjJ8GPWaOj\nynRfPnbr0vAfusM/WmmSsYJOmz6U5fPMj8kn54d/BBjuUrz7wgnCu0cFFGqrSzi74TR/Z7sYFy1N\nfrfQgsmJE1ekRPHF6lfmTD2TCb1pEHpTHXMn1nHHFfOS5pWqimIGmtx5tXjmMFZ8YgKKovDVcyZz\nxZlzmFI+N/T+0IHlPHbrUi4+vYHFM4cxZ0L4u/iJeeZ3ikUm6FTJNmjKmAGhsseP8B8T39Gh6K3+\nO7OmDxmL73B4f87NIgllQhKEVSjkvVIz5YBq+V17jkszKc9k86KrnFNtfy72T+H2cTqd2PLZj6Hb\nMqzmSbf9wHSbTR9FnNHqU6w0N8UUuuFbEkSvZPoJTjA9UDWU1w+O0sPJIwsZ7Q8li2WyZKFq6VyF\nmm45vW3XdLfBOnJxi317QiRB9EJGp8mIl97osV8M3RbuwKXbMRKMCZSoU5rpenUbhsl4R2Zl6GmM\nrJnWOoPbkOAxi6ERPJP9mjQbf8fkG2l0xe8jm6JgBDrUpewQF1nVFDkKtVlo3sC4SIFxiMoC/QUy\nOR75YnjD25H0CiKYN13mI7BmI2kjccyONNLseBk0oNL8+FXGVF0NHhD/2a3pF99uEVkVNagm8Xhc\nZvwX/OHtyXR5gPISf9wj6vI3Wmsq0lEuTe4dk7GVteAYFO4V4z04IvS3r7Uao70Sx+Dd3VuRYuDa\ndBJKSScA+oaleCv2obcOQG+rxndsMIbXieF1MqKmP3s3VWF09sOlzUEp7qRk8huhovwNw/42ha51\ni7D1Ow4ON+fOOIk/cwTXxnnYKo/iOzoUpagTvMXozQNxb5uKHkhSXRvmY3SVs+wMndeP+TuxXT3x\nKu79/WYUu4eTF/pY22Le4Wz5vHqeeyv5/vDuG4ve0h/DFX0yuPzsifzmHx/j2TUR35Ghpu0AXR+c\nBhgY7lLcW6eju8q48fMz+Nkf15muy711OrYy/3NCLjmjgT/tfx27TWHsgOFs3zSHC0+aa7pckNFV\njnvHJK5cuojfrP99YKr5Cc92dAydTaXcct5i7DaFgVUlHHCNwNVSwv2b4oeGmTdpEOefNsakpLAb\nLpzO3iPtHG7q5KW1e5nZYP5ks1C8Jif/rvWnxPwIiJ7p1ktmcdfv14b+vv2yuTicM3Hb2/BOqOR4\ni4tfPm3+6F6AGy+awYhBFWi7m9ixr4WTpwzmwLEOwF8P/7nF47jxocSP+P3hFfMoLbLzrd8djxoa\nvGF4FWp9DSdPGcy3H3kztC6HXWHfkXZ0AxbNiG67+ubnptHU5mb6uOj9dPW5U3jwzx+yaXcTFy4Z\nR1V5EXMnxbTvARcsGsuMcQNx2G0Mq41PVndftSA0zMVdV87HALpcPrrcXupqyrDZFG69ZBZtnR5m\nNgykYUQ1q95pTNpRL9LoIZXcdvk8hg8o5bX1B/jtPzehKAo/Xjmf/wjsg3yTBBHBe2AkjsG7TN8z\nvE48uyZHJQj/WPWBBHGoHt/RoXEJItlQ0VHlB8bQ9xdWhNHu/xJPGjqU97cEvtCGHb01PLb8Seog\nGtd0BIIvivv1r7dXhbrkG+5SfEdLsdsUPjVzGn9+/iX0tprQEBSGK/irSonqhWu0++9aGtTfDscA\nA6YMGQPenRjeIiYPq2Rtgs07ZcrghAnC39PZAMMWaoCL1BDoHTp8YBV7Die4unCFfwn6jvmHKZg8\nKsnY+7ojtL0Nw2sgMLLyzHED2bq6maqK6F+MB493xhXhO1zPSaNH85v1wSDM2zsUw/9oyvEjwnd9\nVVWMxuhvAPEJYu6EOmqrk//KnDJmAFPGDGDH/hZeWruXqvLko9OaVlTGXJ3qRnRVyvgR1UwaVcPG\nncexKUrEr9caqATMH7kcsXwVToedORPqQo2twQQxbGC56S/1ICUwD8QPjTJkQHlcAp082n+s1Xrz\n4UfKS5xMGxufRMtKHIwZWsWm3U2MqKsIlRPL6bAxYWTioU0ij1ddguEuIo//5FH9WfVOo+l8iUxv\nqKWpqYNJo/xxKMCg/vkbWiOWVDGlLUV9ZLcrGTMvIJs60u53BMpNTbEt5f3oBaqRjmj8NwpQU5xw\nuzLY3GARqcdoy257utdw3Y3jlnTR9INKJ4JCHGszWY+82kONGJZOEG2e9sKtrEBD98YtnmT51OfQ\n+BlsWR7xXA/rnCr20GALWZ5vUp0AQsVGXgFktYk56jWdQTnheXtf02d+hv/Ogx7qpJ7t/knnsdb5\nYOkEccua23Nanq+1P4Yn0b3YwYeBhHeZ2eMTfcei70v2HU/vPuXgvf9mfQAA5k6MryMdZdIBKXIw\nON2kU9vJU4akFU/cuioDI1XGjPA5rMJfntGeujPU8EA97sSRNZySIo6yQAPdbDXZIx7DSosdUY2K\nJ0fcox/beDl/8iCqiv3xTugfvtc/ne+uWcIyq59Oh9kxTUd1oJomWfUHQMNw//FPFt/ciXWhwd7K\nS/w1zjMCdfbFzsxPD6muDM3iKQqsZ/6k+O+Kw+5/b+LI6GqgVO0vkDx9Bqt+6lJU6+XabDX+mI8c\nnPomgGCD9dQx8dWx+XTCtUHoHf1CDZWROt85Aww7nqJOikZuSrh817unA4r/MYpGZN24/4vh3joD\nFIPSOf8CwLt/DM4RWzB8drrWLgVDoXTuv+LKnVR0Kh+82xC6I6eqvIjmdjeGYfDIzYux2RQuXtbA\n9Q++Rl1NKbetmGP6ZexatygcjmHjGxdM5Rd/8leYB8uJdefK+Tzw1IccPNZBZZmTlo7woy+XzR7O\nRUvH4bDb6HznzFCnrUe/tRivz6DYaWdm55d4ve1wwn0G8PBNi7HbFTxeHafdBgpcfHoDK3+yOma+\nRegGFDvtPHKzv4E32OFL141Q/Ct/spqq8iJ+8rWT0XUDu12huqqM1tau0HauXud/9OSg/qXcudLf\n6VBR/Ccxm6Jw/6If4bQ7eX534ob0h64/jWvueyW0/2ITxC++uZDy4syeBvfIzYsBfxxD6/rx9Mvb\non4ZPnLz4tB+OWvuCJ5/O7reuqq8iIdvWhzVEevhmxah7W7i3v/1P6LyE/PrmTpmAA/ftAi73cZv\n/7mJVz/cz//70mx+9Ph7ofXYbQqKovCrGxaFyjv9pOEsnjksqvxU7DaFX96wKHRCT+b2r8zlv5/b\nxJoP9/PtL81m9JB+GIZ5cvnVjaeh6wZOR/i7luhznInp4wbG7cNCWDxzGPMmDeL7//U2h5u6ePim\nxTjsCpff/e+ky5WVOELzFlJWCUJV1TLg/4ByoBm4ELgLOAlYq2nadTmLMNcS3S5pJH+Ae/jniC3F\n/ErMOoLVF0rSdZQVO6Pu1Al+0YyI1/bA/wr+X8wut8+kpGB8/v/sgTolm6Ik/PL6T5j+17FfGEUJ\nrz+yR6/dZiM42WlP3QM2WG6xM7wPbCYf9sgTQXC9oS9FzO6z2QLbFJhuD80fvQ2GEb9dkXGHK2zi\nf3NGDmdttv9KnJl/hSLLcQR2vJLg/URtFrHb43TYo24hDY5KGtyfzmCZEcVFrqe4KLxzFUXB6ci0\nk5r5PjaMuHLdAAAddklEQVRjsymhY2VTlNBn1Ezk5ywonSSUjkInh6DSYkeomjCTGHoi3mzXuBx4\nS9O0xcDbwK1AhaZpC4EiVVXn5Ci+XiTfmTv65BT8zhSiStcwjNCJKO6ElE61Sx5iSkfO2rEDBUXt\n6wI1koeqwvPeSz5mhT2st3WKKziL7IBsE8Q2/FcPANX4P3arAn+/AMSP/CUyEvwFGPmrNvglz/l3\nPEmBaa2rlw98n26DeHZld2/bg0kpYTkZHOxk+U0xS4K9QD7uJsqkwb+n9P4I/bJtg9gCLFBVdQNw\nCH9yCN4N3wwkHGKwoqIYhyNFdU4SdruN6uru3Acc/4GMHOUy0ZASwccIxpXmLkYpciVc29CB5Rxp\nqQmPCBrBtWUGxQ3r8B4ZirMy+lBUVhTjdNr5wlkTQtvr7PS3DdhtCtXVZei6QUWpk7ZOT1zZAHU1\npcyaOIhp4way7KQRUfvts0vG8dS/twIwekRN6JLfHnP5XlzsCC234uxJ7NjXHLf/i4vMP0bjRoUb\n1FIds+sumsHLa/emfWzH11dzweKGqPljPxsj6ipoPNSGw2FPWm5Zmb8/QWlpUWi+r547hT/8S4ta\nLvL1OMcCjjneo6a63PTkftX50/jbmu0pt+fMefW8+sFeZk0cTHlpuKru7JNH8Y/Xd3L2wjG4fQZD\na8tTllVeHr6rr6TYGTX/uYvG8tGOY0xuqOWseSOpqSzO+nu0dPYIRgzuF1p+2riBnD5nRMLy5k0d\nyv/+eytnzBsVmuczi8excddxJo2tTfpUumxiPHfxWD7aeYzJ46LL7v65I3e++pmpPLkq/Pk6+5TR\nlBTFf05TxZzv7ck2QVwK/F3TtJ+oqnoT/quJ4G0slUBTogXb2hKfTNNRXV1GU1NHt8qIFTnKZSJn\nzx3HP96I70TnPTASZ/1mkyX87rhiHs+/O4z/2RQ/j358cGikVdckb/SbhsEPLvP37A1ub0eXfx6f\nzwhN+/l1C6MWu+yulwA499TRnHvqaLxuL9/87LSocgA+Oa+eTwZG2exod6H79EDZelR5ri5vaLnT\npg7mtKmD4/a/2x0Te0Bne/hYpzpm00f3Z/ro/mkf21u/MCuu3NjPxs0Xz+TaB9bg9fqSltvZ6e+g\n2N7uCs03ZlAF3/nS7KjlIl/f+MllwDKam+M70wFMGF7FhItnptyeupoy7rpyAR6XhyZXONFfcNoY\nLgh0DPviGQ1x6zcT+d3qcnmj5q8osvPjlfNxdbq5KDA8dbbfoy+e0RC1r80+X7HuufrkqHn6FYfj\ncQX2v5lsYozc1siy83HuyNaYQRX8vy+GP18XLPSPnBwbX6qYc7E9tbWJ76LKtopJwd+vFuBI4P/g\nw2JPBwrTD9wisrmcNFumEDU5sY9c7KkORYUUqn7p4TiE6G2yTRB/AC5UVXU1cAnwC6BLVdU1gE/T\ntLdzFF8PMD9NZNMDNlQXmsaJPa4DTZJskI+Tdrc64lilQrWPi2qv6sE4RN+RVRWTpmlNwFkxkwt6\na2untyu7BVONmGHycBbDZ0dJsKeCHecMk6e7lTlKA6uM+VXus6HYo6tyIm//BCgpim+nyeQKItv7\npWPXm86tdb31ZBTcX7H7NlZRYBtTPY6yt4u8XbQoi05uQsSybEe5bU070prPvWsCit0LhoJzxBYA\nXBvn4hi2Dc+eBv97EXyH6vFVHgs8klPB3v8A3oMjUWabl+87VI9HMdCbontIurdO47Yv+tsXgieq\n0mIHTrtC64aTufCTdfz9AzcdLv/6v3SWyrDaCoYNLOfldXv54lkqsTK5O+PMOSPSnhfCVzDXnDeV\nPYfbcOvwxof7OPfU0RmVc815U3noL+tDf39vxUm0tCeuY77mvKkM6p+f3qzlJU6+dOZ4TppQl3S+\nBVMG097lTfjUrx9eMY/9Rwo4rEuW1PpqLlwyDo/Xl/Hx722+dfHMbneG68uuvWAa/Ssz66CZDcsm\nCLuS+k6org3zQ6ORYvMGEoSB3tYft+bvuh9XWWPYcG+ZFfozOHpqZBWTw27DG2zMNWyhR3NG8h0b\nSl2ZfziA4Ad9xScmsH7bUV5d72Fi1WTaZx/mmdd3hsoMfqkTjS4Z6nOXRg2TM8s7xWr6FVNbXUp1\ndRnz0hzmIjJxxQ6NMWpw8iE40h1KI1tLZg1POY/Dbkv6WMxhA8tDo4z2ZjZFSfvxnr1dqmFETnQz\n0hhqJBcsex1qUwobend+ywQThE/XQ/XEsY3B+Y4hlfD9+HlciRDCUk6gBNHNM1/E4pmeRO3Bu2T0\niF//CmR630w+T97hsLJYiSQVIfokyyYIpz3D2rFu3vijKErosYHjhlVltOygwCMOqyuKwr/UgbqM\nO7mk7g0bHJEza1mc7IfEPMKxqiL5g2yEENZg2TaIIlvuTkKhp5sloQDfvXQOHV0e+pUV0dTuwu3R\nKSmy09bpobqiGMMwsNsUrn8w+pGKcycN5vtfmUP9oH68uj7wGDNF4ZSpg3ns2finiyWMQYn+38yP\nr1yA22M2iF8KgayTzcXAkpnDqKsupT4wbPEdV8yjy5VFDEKIXsWyCSJz4VNfTb9ijreGe50Or61g\n18H4IcCjllb8Q+6WBX6hDyoK/2pO9ahIgPpB/pNnZA1TtuP4JFusotQJpalHV40VHjQui7YRRWFK\nxDj15SXO0Pj1QgjrsmwVU3fE3e9eyDr0bjQGh68g8hCwdCMWQsSwZILw6T5e2LUmu4UVIy5BpHO6\nzeauIzN6sConq7uY4p8dkCsnwpAaQojMWDJB/HPni7x18J3MFoo4/6Vzb3ysTO/L/kSC+9HnTfR3\nxqoqz6INJY9XEL1tGGghRM+zZBvE4c4jqWcCon9r+1+XFNk5c84IzpwzIjTyaeT59rFblwLhUVGD\nf2ci2TIzx9dmVSaEt0b6KgghCsGSVxBxA9t1mzXOuAmf+iaEEHlgyQSRa1Y73+aljVqqmIQQMU6g\nBJH4rGqx/JCneCVDCCGiWTJBJLrjxnvQ3zDs2jgP37FBGB2Jn5QU6cvLJ8RNWz6vnusvnJ59kHly\n6tQhXH72pJyXe9W5Uzht+tCclyuEsC5LNlKbpYfO9xeDpwTPLv/J0701/buORtRV4LAreH3hki9c\nkvoxpD3hsrMn5qXcscOqGJvhECJCiL7NklcQ3SNVKUIIkQ5rJghpURVCiLyzZIIwTQ9Gek23xYr5\nCKr9+5VkH5AQQvRBlmyDME0RXv/j94YMKOOcU0bxyN82xs2yrP95LGqYbFriLZfMYu+RtpxGmY7b\nVsyRRysKIXolS15BJDNqcD/mTxps+tD282csYEC5+SMwa/oVM2X0ANP38mnk4H6MqKso+HqFECIV\nSyYIaYEQQoj8s2SCcHldqWeSLCKEEN1iyQSx6fiWJO+mX59fWSYPtRFCiEQs2kgdzb1tavzENPLE\n3VednPJRo0IIcaLqEwnC8BSbTEy9XHGRPffBCCFEH2HJKqZ44csFq43MKoQQvVXfSBAmVwtScSSE\nEN1juQTR5m7PeJkLl4xjQKX0lBZCiExYrg3Cp/viJyrxLyNrmpbPq2d5gmdECyGEMGe5Kwhz8RVK\nUsUkhBDd00cShBBCiFyzYIJIcZtS4G0ZAE8IIbrHggkiPf/vi7N7OgQhhLC0vpEglHCLgxK4hBgu\nI6QKIUS39I0EIYQQIuckQQghhDBluQTx8q43kr7fv9JkXCYhhBAZs1yCeH7b6tBrQ/e3N3xuyVgA\nqiuKOOeUUT0QlRBC9D1Z96RWVfXLwKWAHbgEuAk4CViradp1uQkvBa8TityUBEZlnTiyBrvNcjlP\nCCF6pazOpqqqDgMWaZq2TNO0xcAgoELTtIVAkaqqc3IYYxQlqh+E/7X0mhZCiNzL9griLMCuquqL\nwEZgE7Aq8N4LwALgne6HZ8akA5yhJ35PCCFEVrJNEIOAIk3TlqmqejdQBWwLvNcMTE60YEVFMQ5H\n9g/qiXzeg6+1P44B+5k0YjSwjrlTBlNdXRa3jNm0QrLbbT0eQ6Yk5sKwYsxgzbgl5sxlmyCagZcD\nr1/C3/ZQGfi7EmhKtGBbmyvLVcbTmwdQfHg2dUv78/BNi3E6bDQ1dcTNZzatkKqry3o8hkxJzIVh\nxZjBmnFLzOZqa/slfC/bFt3XgWmB1zPwNwMsC/x9OvBmluWmpMRUIxXZnQA4HdI4LYQQuZTVWVXT\ntHVAp6qqq4E5wE+BLlVV1wA+TdPezl2ISRjS5iCEEPmS9W2umqbdFDOpILe2yh1LQghRGJarl2nv\n9ET8pfCF0xt6LBYhhOjLLJcgYi8hpo0d2DNxCCFEH2e9BBHZSC1tEEIIkTcWTBBCCCEKwXIJwqW0\nhl4buuXCF0IIy8j6LqbeQG+qS/r+DRdNp6zYWaBohBCib7F0gkg19tKU0QMKFIcQQvQ9UkcjhBDC\nlCQIIYQQpiRBCCGEMCUJQgghhClJEEIIIUxJghBCCGFKEoQQQghTkiCEEEKYsnSCmNkgI7kKIUS+\nWDpBCCGEyB9JEEIIIUxJghBCCGFKEoQQQghTkiCEEEKYkgQhhBDClCQIIYQQpiyVIHy6r6dDEEKI\nE4alEkSXzxV67dkzDsPowWCEEKKPs1SCiHzAqOEt6rE4hBDiRGCpBBH7DGol+SOphRBCdIOlEkR0\nQjCkikkIIfLIUgki9gqiYURVD8UhhBB9n8USRPiSoX5QBWfNre/BWIQQom+zWIIIqywvxiaNEEII\nkTeWShCRbQ42yQ1CCJFXlkoQkVVMcvEghBD5ZakEEXnTks1SkQshhPVY6jRrRF5BIJcQQgiRT5ZK\nEJFs0gghhBB5Za0EEVHHJG0QQgiRX5ZKEJFVTHbFUqELIYTlWPYse+6UU3o6BCGE6NO6lSBUVb1e\nVdVXA6/vU1V1jaqqD+QmtHiRVxBVJf3ytRohhBB0I0GoqloMzAi8ngVUaJq2EChSVXVOjuITQgjR\nQ7pzBXE58N+B1/OBVYHXLwALuhNUIoY0UgshRMFklSBUVXUCizVNeykwqRpoCbxuDvydBzK+txBC\nFIojy+W+BPwh4u9moDLwuhJoSrRgRUUxDoc9q5Uane7Q6+rqMhx2a7Sx2+02qqvLejqMjEjMhWHF\nmMGacUvMmcs2QajADFVVrwImAwOBacD/AqcDv020YFubK9FbKTW7OkOvm5o6LJMgqqvLaGrq6Okw\nMiIxF4YVYwZrxi0xm6utTXzDT1ZnWE3TbtE07SxN05YDGzRNux3oUlV1DeDTNO3t7EJNzpBHyAkh\nRMFkewURomnaqYH/r+t+OOmTRmohhMgva9TRxDB0S4YthBCWYqkzbbCjnHvLjB6ORAgh+j5rJYhQ\nE4Qiw30LIUSeWSpBSD8IIYQoHEslCJ+u+18YCnIBIYQQ+WWpBPGudrinQxBCiBOGpRKE16f3dAhC\nCHHCsFSCiOwoJzVMQgiRX5ZKEOHHUEt6EEKIfLNUglCC3aflZiYhhMg7SyUIlIgqJhlrQwgh8spS\nCUKSghBCFI6lEoTULQkhROFYKkGEryDkSkIIIfLNYgmipyMQQogThyQIIYQQpiyVIEI1S9IUIYQQ\neWepBKGYvBJCCJEf1koQkheEEKJgLJUghBBCFI6lEoR0lBNCiMKxVoKQ1mkhhCgYSyWIYNu0YciV\nhBBC5JulEoTUMAkhROFIghBCCGHKWgkCGYtJCCEKxVIJIvJ5EEIIIfLLYglCrhyEEKJQLJUgdrZv\n9r+QCwkhhMg7SyWId5te97+QqiYhhMg7SyWIYCO1IglCCCHyzmIJIhCuJAghhMg7SyUImyIPhBBC\niEKxVIJQuqoAMLxFPRyJEEL0fZZKEO0HagEwOvv1cCRCCNH3WSpBABi65UIWQghLstTZVlEMaX4Q\nQogCsVSC8JPe1EIIUQgWTBBCCCEKwWIJQuqXhBCiUBzZLKSq6jzgPkAH3tE07XpVVW8GzgV2ASs0\nTfPkLswABZCnyQkhREFkewWxC1iqadqpQJ2qqouAJYG/PwQ+k6sAhRBC9IysriA0TTsQ8acHmAys\nDvz9AnAJ8H/disyUgTRSCyFEYWSVIIJUVZ0G1AJN+KubAJqB6kTLVFQU43DYs1+pAddeOIPq6rLs\nyygwu91mqXhBYi4UK8YM1oxbYs5c1glCVdX+wIPAhcBsYHjgrUr8CcNUW5sr21WGBumrqyymqakj\n+3IKrLq6zFLxgsRcKFaMGawZt8RsrrY28cgUWbVBqKrqAJ4AbgpUN70DLAq8fTrwZjblpkeRe5mE\nEKIAsm2k/hwwB7hHVdXVwFjgFVVVXwVmAE/nJjwTBmBIihBCiHzLtpH6SeDJmMlvAHd3O6JkFH8j\ntaQHIYTIP4t1lBNCCFEoFksQRtR/Qggh8sdaCUIBw5AqJiGEKARrJYgAQxqphRAi7yyVIKorigL/\nF/dwJEII0fdZKkEMHlCGgkJluTyTWghhbWvXvsvevXt6OoykLJUgZCwmIURf8f7777Fv396eDiMp\nSyUIaXsQQuTT2rXvctNN13LLLddz5ZVf4dln/851113NzTdfx+HDh/jGN67k6qsv56c/vQuA115b\nw0MPPYCu69xwwzc4cOAABw4c4Nprr+Lqqy/jiSd+C8CPfvR99uxpBODrX1+Jx+Phn/98hgcfvJ9f\n/OK+uDg6Ozv5zne+xYoVX+bOO28H4Pnnn2XlyhVcffVlbNmyGYCVK1dw9913sGLFF3jzzddZv/4D\nfvnLnwPQ0tLMrbfe0K390a3B+gqtrdMjnaiFOMEdOt5Bh8ub8XL92ty0tnVRVuygribxAHgOh4O7\n7rqX3/3uMTZv1njggV9x990/4vDhQ9x330M4HA5+8IPv0ti4m1NOWchLL63innt+xCmnLGTw4MH8\n7Gd3c/nlVzJ9+kxuuOEbLF9+dtw6nE4nn/jEp5g2bQZz5syLe/9vf/szc+bM59JLv8ixY234fD6e\neuqP/OpXjwXiuId77rmflpYWVq78Gl6vl3vvvYc77/wJDz/8EACvvvoKp566KK7sTFgqQew72oat\nqqejEEL0lJYON//xyJvd+qGoKHDfN06lssy8LXP06LEADBxYS3V1TeD1QPbsaeTxx/+L1tZWDhzY\nz5Ejhxkxop5zzz2fm266jhtuuAWAvXv3MH78BAAaGsazb99eFCWzqvHGxt2cf/7nALDZbBw9eoRB\ng4bgcDgYMmQobW1tANTUVFNT0x+AtrZWFEVh3LgGNm/exKuvvsKtt34nw70TzVIJwk/aIIQ4UVWW\nFfHjlfOzu4KoKAldQSRKDkDUyTzy9aFDB1m4cDGf/OQ53H77dzAMA13X+e1vf8NXvvJVnnjit1x2\n2UqGDRuOpn3MjBmz2LJF47OfvYjy8gqOHj1CXd0gGht3A/4rFV3X49YPUF8/kg0bPmLWrGnouk51\ndQ0HD+7H6/Vy+PAhKioqghGGlglWwS9evIxnn/07Pp+Xysru/aK2WIKQ+iUhTnTJqoeS6e7Q2Tab\njT/+8QnWrFkdmvbUU3/ktNMW85nPXMB3vvMttm/fxiWXfJk77rgNr9fLKacspLa2juXLz+bOO2+n\noWE8AwYMAGDmzNk8/PBDbNz4EV/5ylej1nXOOedxxx3f48UXn6Oubgjf/vZtnH/+hXzta1dgs9m4\n4YZvJYxz2rQZ3H77d1ix4oqstzVIKXTD7+HDrVmv8MonH8TW7xi/+tT3chlS3sk49IUhMReOFeOW\nmM3V1vZLWC1jsSsIIYToW3bv3sk999wZ+ru4uISf/eznPRhRmLUShGKAIW0QQoi+o75+FA8++EhP\nh2HKUv0gUHRJEEIIUSAWSxAGhmGtkIUQwqqsdbaVKwghhCgYayUI3Q4+azWbCCGEVVkqQXgaVdw7\np/R0GEKIE9TVV1/e0yEUlKV+jiveIukqJ4QoCF3Xsdks9Rs65yyVIFCQztRCnOAOdxyl09uZ8XLH\njRJaW7sodZRSWzbAdJ61a9/lf/7n94C/N/OTTz6Oz+djxYormD//5NB8X//6Su6//5c4HA6+/vWV\nprep7tq1k5/8xN+/4bTTlnDhhRdz//0/YcuWzZSXl/O9791Ba2sLP/zh96ipqWH//v3cddfPeO65\nf9DQoLJgwSmsXv1vNmzYxCWXXJrx9uaCpRKEgoIhGUKIE1aru43b37ynW+cBBYUfn/pd+hVVmL7v\n8Xj56U8f4Nprr+KBB36FYRjcdNO1UQkiHf/5nw9y883fZuTIUei6zscfb6Czs5OHHnqU559/lqef\nfoply86ks7ODBx98hBdeeJ7Vq19i8eJl/OEPv2PBglNYtWoVX/jCiqy3tbsslSA+uWAkz7y+s6fD\nEEL0kH5FFdw2/1tZXUH06xe+gkiUHABUdQLNzU3s2rWTb37zawA0NR2Peh5NOqOzNjc3MXLkKMA/\njlPkKK8TJkxi3bq1AIwaNQabzcbAgbXs2dNIff1I9u3bi8vVxcGDBxg2bHjG25orlkoQ5582hss+\nPcVy46kIIXInUfVQKtXVZTQpqc8diqJQVVXNmDHjuPfeX2C32/F6vVFJoaLCPzprcXEJR48eSbC+\nGnbv3kV9/Uh0XWfYsOG8/fabAGzatJGhQ4eH1hcUTEIzZ87m179+mLlz458VUUiWShBCCFEINpuN\nz3/+C1x33dUoisKoUWO48cZbQu9/+tPnccstNzBt2vTQMyNiXXnlNdxzz4+AcBvEs88+w9e+dgVl\nZWXcdtuPaGtrNV128eJlrFhxMX/9699zv3EZsNRoriAjMhaKxFwYVowZrBm3xGxORnMVQog8evHF\nf/GXvzwV+nvKlGlcddXXezCi3JAEIYQQ3bRs2ZksW3ZmT4eRcyd2LxAhhBAJSYIQQghhShKEEEII\nU5IghBBCmJIEIYQQwlTB+0EIIYSwBrmCEEIIYUoShBBCCFOSIIQQQpiSBCGEEMKUJAghhBCmLDUW\nk6qq9wEnAWs1Tbuup+OJpKrqKOAt4GPArWnamaqq3gycC+wCVmia5jGb1gOxDgWeASYBFZqmec32\nbbrTeiJmYDgx+zswX6/Z56qqzgPuA3TgHU3Trk83vl4WczPwfmCW8zVNO6aq6iXANcAx4AuaprWY\nTStQzFOARwAfsBW4DLiX3v15jo35B8Cb9LLPs2WuIFRVnYX/ZLYQKFJVdU5Px2RilaZpiwPJoQ5Y\nomnaqcCHwGfMpvVQnMeAZfg/kKb7Nt1pPRVzQGh/B7ajt+3zXcDSwLrrVFVdlE58vSzmqcD6wH5e\nHEgOTuAq4DTgceBKs2kFjFnTNO3kwOcSYC69//McG/NAeuHn2TIJApgPrAq8fgFY0IOxJLJEVdU1\nqqpej/9XyerA9GC8ZtMKTtO0Lk3TjkdMMtu36U4rCJOYIXp/Qy/b55qmHdA0rSvwpweYnGZ8vSlm\nHzAxsJ/vUlVVARrwJw1vRHxm0woVc+QvaBf+HxK9/fMcG7OdXvh5tlKCqAaCl6zNgb97k/3AeGAJ\ncDr+Axkbb2/dBrO40p3WU6L2t6qq0+ilMQdiqwWa0oyv18SsadpG/Cf/04Aa4JwE8fVozKqqflpV\n1Y+AQYAzzfh6U8zv0ws/z1ZKEM1AZeB1Jf4vW6+haZpL07T2wC+oZ4BtxMfbW7fBLK50p/UIk/09\nJUF8PRqzqqr9gQeByzOIrzfFjKZpxzRNM4Cn6aX7WdO0v2maNgXYA3jTjK83xfzJ3vh5tlKCeAP/\npSP4f6G/mWTeglNVtV/En6fgb3haFPg7GO87JtN6A7N9m+60HmGyv7dhvn97bJ+rquoAngBu0jTt\nQAbx9ZqYVVUtV1XVHng7uJ83A1MC04PxmU0rVMzFEX+2AAa9/PNsErM34u9e83m2TILQNG0t0KWq\n6hrAp2na2z0dU4yFqqq+p6rq68BeTdPeAl5RVfVVYAbwtKZph2Kn9USgqqo6VVV9AZgOPI//kjxq\n35rt7548BiYx3xC7v832bw/v888Bc4B7VFVdDYxNJ75eFvM04B1VVV8BRgBPBerPHwXWAJcCD5tN\nK2DMy1VVfVlV1ZfxV9fcRS//PJvE7OuNn2cZrE8IIYQpy1xBCCGEKCxJEEIIIUxJghBCCGFKEoQQ\nQghTkiCEEEKYkgQhhBDClCQIIYQQpiRBCCGEMPX/Aco4w18fMfEjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fabc415db90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot1, = plt.plot(np.arange(len(maxout_mlp_acc)), maxout_mlp_acc, label='maxout_conv')\n",
    "plot2, = plt.plot(np.arange(len(relu_mlp_acc)), relu_mlp_acc, label='relu_conv')\n",
    "plt.legend(handles=[plot1, plot2])\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion \n",
    "\n",
    "Maxout units are cool! I'm interested to see how much of an improvement these make on lower dimensional linear layers at the end of large convolutional nets and will be testing that in the coming weeks. The lack of saturation might help gradients back prop, while not requiring that you have many times more parameters for you entire model. \n",
    "\n",
    "This work also made me realize viscerally how important clear explanations are! There are *so many* incorrect implementations that you can find online of maxout units, so hopefully this will serve as a guide for people who were in my position a couple of days ago. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References \n",
    "- https://arxiv.org/pdf/1302.4389.pdf (original paper) \n",
    "- http://cs231n.github.io/neural-networks-1/ (Karpathy to the rescue)\n",
    "- http://www.simon-hohberg.de/2015/07/19/maxout.html (really helpful visualizations) \n",
    "- tons of Pytorch forum posts, thanks y'all "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}