Project3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You may need to install [OpenCV](https://pypi.python.org/pypi/opencv-python) and [scikit-video](http://www.scikit-video.org/stable/).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "import numpy as np\n",
    "import io\n",
    "import base64\n",
    "from IPython.display import HTML\n",
    "import skvideo.io\n",
    "import cv2\n",
    "import json\n",
    "\n",
    "from keras.models import Sequential,model_from_json\n",
    "from keras.layers.core import Dense\n",
    "from keras.optimizers import sgd\n",
    "from keras.layers import Conv2D, MaxPooling2D, Activation, AveragePooling2D,Reshape,BatchNormalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MiniProject #3: Deep Reinforcement Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Notations__: $E_p$ is the expectation under probability $p$. Please justify each of your answer and widely comment your code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a reinforcement learning algorithm, we modelize each step $t$ as an action $a_t$ obtained from a state $s_t$, i.e. $\\{(a_{t},s_{t})_{t\\leq T}\\}$ having the Markov property. We consider a discount factor $\\gamma \\in [0,1]$ that ensures convergence. The goal is to find among all the policies $\\pi$, one that maximizes the expected reward:\n",
    "\n",
    "\\begin{equation*}\n",
    "R(\\pi)=\\sum_{t\\leq T}E_{p^{\\pi}}[\\gamma^t r(s_{t},a_{t})] \\> ,\n",
    "\\end{equation*}\n",
    "\n",
    "where: \n",
    "\\begin{equation*}p^{\\pi}(a_{0},a_{1},s_{1},...,a_{T},s_{T})=p(a_{0})\\prod_{t=1}^{T}\\pi(a_{t}|s_{t})p(s_{t+1}|s_{t},a_{t}) \\> .\n",
    "\\end{equation*}\n",
    "\n",
    "We note the $Q$-function:\n",
    "\n",
    "\\begin{equation*}Q^\\pi(s,a)=E_{p^{\\pi}}[\\sum_{t\\leq T}\\gamma^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a] \\> .\n",
    "\\end{equation*}\n",
    "\n",
    "Thus, the optimal Q function is:\n",
    "\\begin{equation*}\n",
    "Q^*(s,a)=\\max_{\\pi}Q^\\pi(s,a) \\> .\n",
    "\\end{equation*}\n",
    "\n",
    "In this project, we will apply the deep reinforcement learning techniques to a simple game: an agent will have to learn from scratch a policy that will permit it maximizing a reward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The environment, the agent and the game"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```Environment``` is an abstract class that represents the states, rewards, and actions to obtain the new state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Environment(object):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "\n",
    "    def act(self, act):\n",
    "        \"\"\"\n",
    "        One can act on the environment and obtain its reaction:\n",
    "        - the new state\n",
    "        - the reward of the new state\n",
    "        - should we continue the game?\n",
    "\n",
    "        :return: state, reward, game_over\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"\n",
    "        Reinitialize the environment to a random state and returns\n",
    "        the original state\n",
    "\n",
    "        :return: state\n",
    "        \"\"\"\n",
    "        pass\n",
    "    \n",
    "    def draw(self):\n",
    "        \"\"\"\n",
    "        Visualize in the console or graphically the current state\n",
    "        \"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method ```act``` allows to act on the environment at a given state $s_t$ (stored internally), via action $a_t$. The method will return the new state $s_{t+1}$, the reward $r(s_{t},a_{t})$ and determines if $t\\leq T$ (*game_over*).\n",
    "\n",
    "The method ```reset``` simply reinitializes the environment to a random state $s_0$.\n",
    "\n",
    "The method ```draw``` displays the current state $s_t$ (this is useful to check the behavior of the Agent).\n",
    "\n",
    "We modelize $s_t$ as a tensor, while $a_t$ is an integer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of the ```Agent``` is to interact with the ```Environment``` by proposing actions $a_t$ obtained from a given state $s_t$ to attempt to maximize its __reward__ $r(s_t,a_t)$. We propose the following abstract class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Agent(object):\n",
    "    def __init__(self, epsilon=0.1, n_action=4):\n",
    "        self.epsilon = epsilon\n",
    "        self.n_action = n_action\n",
    "    \n",
    "    def set_epsilon(self,e):\n",
    "        self.epsilon = e\n",
    "\n",
    "    def act(self,s,train=True):\n",
    "        \"\"\" This function should return the next action to do:\n",
    "        an integer between 0 and 4 (not included) with a random exploration of epsilon\"\"\"\n",
    "        if train:\n",
    "            if np.random.rand() <= self.epsilon:\n",
    "                a = np.random.randint(0, self.n_action, size=1)[0]\n",
    "            else:\n",
    "                a = self.learned_act(s)\n",
    "        else: # in some cases, this can improve the performance.. remove it if poor performances\n",
    "            a = self.learned_act(s)\n",
    "\n",
    "        return a\n",
    "\n",
    "    def learned_act(self,s):\n",
    "        \"\"\" Act via the policy of the agent, from a given state s\n",
    "        it proposes an action a\"\"\"\n",
    "        pass\n",
    "\n",
    "    def reinforce(self, s, n_s, a, r, game_over_):\n",
    "        \"\"\" This function is the core of the learning algorithm. \n",
    "        It takes as an input the current state s_, the next state n_s_\n",
    "        the action a_ used to move from s_ to n_s_ and the reward r_.\n",
    "        \n",
    "        Its goal is to learn a policy.\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "    def save(self):\n",
    "        \"\"\" This function returns basic stats if applicable: the\n",
    "        loss and/or the model\"\"\"\n",
    "        pass\n",
    "\n",
    "    def load(self):\n",
    "        \"\"\" This function allows to restore a model\"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "__Question 1__:\n",
    "Explain the function act. Why is ```epsilon``` essential?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function act return the action to do when the agent is in the state s. We can choose two behaviors. If we choose train == False the agent will just choose a learned action to do in this state. If we choose train == True, the agent has a probability $p= 1- \\epsilon$ to do the learned action in this state, and a probability $1-p = \\epsilon$ to do a random action. This possibility with the $\\epsilon$ is essential if we want that the agent doesn't just choose an actual learned action but try to explore sometimes other actions which could have better rewards. It's is essential to continue to learn and improve the decisions of the agent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### The Game"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```Agent``` and the ```Environment``` work in an interlaced way as in the following (take some time to understand this code as it is the core of the project)\n",
    "\n",
    "```python\n",
    "\n",
    "epoch = 300\n",
    "env = Environment()\n",
    "agent = Agent()\n",
    "\n",
    "\n",
    "# Number of won games\n",
    "score = 0\n",
    "loss = 0\n",
    "\n",
    "\n",
    "for e in range(epoch):\n",
    "    # At each epoch, we restart to a fresh game and get the initial state\n",
    "    state = env.reset()\n",
    "    # This assumes that the games will end\n",
    "    game_over = False\n",
    "\n",
    "    win = 0\n",
    "    lose = 0\n",
    "    \n",
    "    while not game_over:\n",
    "        # The agent performs an action\n",
    "        action = agent.act(state)\n",
    "\n",
    "        # Apply an action to the environment, get the next state, the reward\n",
    "        # and if the games end\n",
    "        prev_state = state\n",
    "        state, reward, game_over = env.act(action)\n",
    "\n",
    "        # Update the counters\n",
    "        if reward > 0:\n",
    "            win = win + reward\n",
    "        if reward < 0:\n",
    "            lose = lose -reward\n",
    "\n",
    "        # Apply the reinforcement strategy\n",
    "        loss = agent.reinforce(prev_state, state,  action, reward, game_over)\n",
    "\n",
    "    # Save as a mp4\n",
    "    if e % 10 == 0:\n",
    "        env.draw(e)\n",
    "\n",
    "    # Update stats\n",
    "    score += win-lose\n",
    "\n",
    "    print(\"Epoch {:03d}/{:03d} | Loss {:.4f} | Win/lose count {}/{} ({})\"\n",
    "          .format(e, epoch, loss, win, lose, win-lose))\n",
    "    agent.save()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The game, *eat cheese*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A rat runs on an island and tries to eat as much as possible. The island is subdivided into $N\\times N$ cells, in which there are cheese (+0.5) and poisonous cells (-1). The rat has a visibility of 2 cells (thus it can see $5^2$ cells). The rat is given a time $T$ to accumulate as much food as possible. It can perform 4 actions: going up, down, left, right. \n",
    "\n",
    "The goal is to code an agent to solve this task that will learn by trial and error. We propose the following environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Environment(object):\n",
    "    def __init__(self, grid_size=10, max_time=500, temperature=0.1):\n",
    "        grid_size = grid_size+4\n",
    "        self.grid_size = grid_size\n",
    "        self.max_time = max_time\n",
    "        self.temperature = temperature\n",
    "\n",
    "        #board on which one plays\n",
    "        self.board = np.zeros((grid_size,grid_size))\n",
    "        self.position = np.zeros((grid_size,grid_size))\n",
    "\n",
    "        # coordinate of the cat\n",
    "        self.x = 0\n",
    "        self.y = 1\n",
    "\n",
    "        # self time\n",
    "        self.t = 0\n",
    "\n",
    "        self.scale=16\n",
    "\n",
    "        self.to_draw = np.zeros((max_time+2, grid_size*self.scale, grid_size*self.scale, 3))\n",
    "\n",
    "\n",
    "    def draw(self,e):\n",
    "        skvideo.io.vwrite(str(e) + '.mp4', self.to_draw)\n",
    "\n",
    "    def get_frame(self,t):\n",
    "        b = np.zeros((self.grid_size,self.grid_size,3))+128\n",
    "        b[self.board>0,0] = 256\n",
    "        b[self.board < 0, 2] = 256\n",
    "        b[self.x,self.y,:]=256\n",
    "        b[-2:,:,:]=0\n",
    "        b[:,-2:,:]=0\n",
    "        b[:2,:,:]=0\n",
    "        b[:,:2,:]=0\n",
    "        \n",
    "        b =  cv2.resize(b, None, fx=self.scale, fy=self.scale, interpolation=cv2.INTER_NEAREST)\n",
    "\n",
    "        self.to_draw[t,:,:,:]=b\n",
    "\n",
    "\n",
    "    def act(self, action):\n",
    "        \"\"\"This function returns the new state, reward and decides if the\n",
    "        game ends.\"\"\"\n",
    "\n",
    "        self.get_frame(int(self.t))\n",
    "\n",
    "        self.position = np.zeros((self.grid_size, self.grid_size))\n",
    "\n",
    "        self.position[0:2,:]= -1\n",
    "        self.position[:,0:2] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "\n",
    "        self.position[self.x, self.y] = 1\n",
    "        if action == 0:\n",
    "            if self.x == self.grid_size-3:\n",
    "                self.x = self.x-1\n",
    "            else:\n",
    "                self.x = self.x + 1\n",
    "        elif action == 1:\n",
    "            if self.x == 2:\n",
    "                self.x = self.x+1\n",
    "            else:\n",
    "                self.x = self.x-1\n",
    "        elif action == 2:\n",
    "            if self.y == self.grid_size - 3:\n",
    "                self.y = self.y - 1\n",
    "            else:\n",
    "                self.y = self.y + 1\n",
    "        elif action == 3:\n",
    "            if self.y == 2:\n",
    "                self.y = self.y + 1\n",
    "            else:\n",
    "                self.y = self.y - 1\n",
    "        else:\n",
    "            RuntimeError('Error: action not recognized')\n",
    "\n",
    "        self.t = self.t + 1\n",
    "        reward = self.board[self.x, self.y]\n",
    "        self.board[self.x, self.y] = 0\n",
    "        game_over = self.t > self.max_time\n",
    "        state = np.concatenate((self.board.reshape(self.grid_size, self.grid_size,1),\n",
    "                        self.position.reshape(self.grid_size, self.grid_size,1)),axis=2)\n",
    "        state = state[self.x-2:self.x+3,self.y-2:self.y+3,:]\n",
    "\n",
    "        return state, reward, game_over\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"This function resets the game and returns the initial state\"\"\"\n",
    "\n",
    "        self.x = np.random.randint(3, self.grid_size-3, size=1)[0]\n",
    "        self.y = np.random.randint(3, self.grid_size-3, size=1)[0]\n",
    "\n",
    "\n",
    "        bonus = 0.5*np.random.binomial(1,self.temperature,size=self.grid_size**2)\n",
    "        bonus = bonus.reshape(self.grid_size,self.grid_size)\n",
    "\n",
    "        malus = -1.0*np.random.binomial(1,self.temperature,size=self.grid_size**2)\n",
    "        malus = malus.reshape(self.grid_size, self.grid_size)\n",
    "\n",
    "        self.to_draw = np.zeros((self.max_time+2, self.grid_size*self.scale, self.grid_size*self.scale, 3))\n",
    "\n",
    "\n",
    "        malus[bonus>0]=0\n",
    "\n",
    "        self.board = bonus + malus\n",
    "\n",
    "        self.position = np.zeros((self.grid_size, self.grid_size))\n",
    "        self.position[0:2,:]= -1\n",
    "        self.position[:,0:2] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.position[:, -2:] = -1 #correction of self.position[-2:, :] = -1\n",
    "        self.board[self.x,self.y] = 0\n",
    "        self.t = 0\n",
    "\n",
    "        state = np.concatenate((\n",
    "                               self.board.reshape(self.grid_size, self.grid_size,1),\n",
    "                        self.position.reshape(self.grid_size, self.grid_size,1)),axis=2)\n",
    "\n",
    "        state = state[self.x - 2:self.x + 3, self.y - 2:self.y + 3, :]\n",
    "        return state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following elements are important because they correspond to the hyper parameters for this project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters\n",
    "size = 13\n",
    "T=200\n",
    "temperature=0.3\n",
    "epochs_train=10 # set small when debugging\n",
    "epochs_test=3 # set small when debugging\n",
    "\n",
    "# display videos\n",
    "def display_videos(name):\n",
    "    video = io.open(name, 'r+b').read()\n",
    "    encoded = base64.b64encode(video)\n",
    "    return '''<video alt=\"test\" controls>\n",
    "                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
    "             </video>'''.format(encoded.decode('ascii'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Question 2__ Explain the use of the arrays ```position``` and ```board```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- the array **position** contains the information of the position of the rat on the board and the limits of the game. It's a matrix with 0 in the cells where the rat is not, with 1 in the unique cell where the rat is, and -1 if the cells are out of the island (the rat can not go here).\n",
    "- the array **board** contains the information of the positions of the cheese and the poisonous cells. It's a matrix with +0.5 in the cells where there is cheese, -1 in the cells where there is poison, 0 in the cells with nothing.\n",
    "\n",
    "The parameter temperature allows to influence the number of cheeses and poisons on the board. Board is first fill with cheese then with poison in the remaining empty cells. The temperature is the probability for a cell to contain a cheese. And it's also the probability for  a cell to contain a poison (if we do not put cheese in before). **The number of cheese and poison on the board increase with the temperature.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "__Question 3__ Implement a random Agent (only ```learned_act``` needs to be implemented):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomAgent(Agent):\n",
    "    def __init__(self):\n",
    "        super(RandomAgent, self).__init__()\n",
    "        pass\n",
    "\n",
    "    def learned_act(self, s):\n",
    "        # choose a random action in [0, 1, 2, 3]\n",
    "        a = np.random.randint(0, 4)\n",
    "        return a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "__Question 4__ Visualize the game moves. You need to fill in the following function for the evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(agent,env,epochs,prefix=''):\n",
    "    \n",
    "    # Number of won games\n",
    "    score = 0\n",
    "        \n",
    "    for e in range(epochs):\n",
    "        \n",
    "        state = env.reset()\n",
    "        game_over = False\n",
    "        \n",
    "        win = 0\n",
    "        lose = 0\n",
    "        \n",
    "        while not game_over:\n",
    "            \n",
    "            # the agent chooses an action\n",
    "            action = agent.act(state)\n",
    "            # act on the environment\n",
    "            prev_state = state\n",
    "            state, reward, game_over = env.act(action)\n",
    "            # Update the counters\n",
    "            if reward > 0:\n",
    "                win = win + reward\n",
    "            if reward < 0:\n",
    "                lose = lose -reward\n",
    "            \n",
    "        # Save as a mp4\n",
    "        env.draw(prefix+str(e))\n",
    "\n",
    "        # Update stats\n",
    "        score = score + win-lose\n",
    "\n",
    "        print(\"Win/lose count {}/{}. Average score ({})\"\n",
    "              .format(win, lose, score/(1+e)))\n",
    "    print('Final score: '+str(score/epochs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Win/lose count 8.0/11.0. Average score (-3.0)\n",
      "Win/lose count 11.0/14.0. Average score (-3.0)\n",
      "Win/lose count 7.0/10.0. Average score (-3.0)\n",
      "Final score: -3.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAGRFtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAAL9ZYiEADf//vaH+BTZWBO6f/0/4LrOnzl/nbABNNgBSG5gp/rE//5O6fMPwgPgUuAa/gU0mcEGbSaVRHhtNP6y/aD/4M/4DQagOsUmCYh32/d8Iz+VVgIUR69ieBDDmKiUzlGrIWoKROapAX6RYhNbdprVT8BoVy7Ct55uXTHf2narRsO5X8BqdDssF35+OpcHQu/qu7aHTZHcguhE+rmB1HjCpo1Lr9PWMtn+3fDmtzpg12sKR5/PJbf4/Mfs032dobSaiAl4BH+7EYEPScIhqk286Pd6sjxDLkS7Dmsnz733ScQkIQiH1Ym8wxQfKJUQHy62e2U7TmXhSm9TnP7q/FJVSUuERMCnwwJT/RLLhducbC5CTWBFR4RQ0I2H4CU81P7ls8KjmCUkuujMquVRwEOrFbmCxGenF7hs26rERpY4W6iWiSCVIBULVgQi0qHl0rEhkK3yTw/rNK4rLA/bIBKkQtlp8zOTg+CGHpRGabvLKDOALvVUx3QVBFj5CPVUt3LT7A/XOeijcorjmtACZ1nQqZnWAyNbFVclShf7Ic4zxpZl+VNRZPDQKLPvq/DyxY4PzYJOTO2Ux5XKkW8SoLVhccWVB7QKUnSvZYcTrnosWs+sTfIW2gJOTSQvxKhPnycETFyyk+abarJ/hoTokMz46ohRupTzOcjvZsE/sbRYp5R2lBubzIYvpnxHMLetnli0WkvfffMw9vQp+vL4m146X1bwTkDf3oyp42cNS6L8LU1MYWPsTH/mxOMZK4MjhC2zfVWAr2JBCHqWRIgM5VJfKdMSuNjArdGTEgo6CQA87ws23MAI00JDA1kVyJmQOTKRd8MxqdsUz8Td1vVgzjPpD7vIchC0kQ1GFLM3SzXaIuf07JQxTuspU3/Vd07RG9ausS4AWO9n6VSkkP9QeaFflhF+aAsiC8MmA9q1VUwDEb8Id/YWvvyZuHuoYP4sExHjiRRKwBxHUf9/4Hh0lREMC4GecRRz/6z0pUbmRDBJLhqs9/VLAAHIiARNAAAAFEGaIWxDv/6plgAJj8efwEg3FTfAAAAAGkGaRTwhkymEN//+p4QADE+wfzaQS2QZbWyxAAAAD0GeY2pTwv8AB0E1Z3Q7zAAAAA8BnoJ0Qr8ACfRhAZJdM4EAAAAPAZ6EakK/AAnzWUzbMjevAAAAGkGahkmoQWiZTAh3//6plgAF499X12INxViRAAAAH0GaqknhClJlMCG//qeEAAdz2D+bS7yA8Dwbnk9vaq0AAAARQZ7IRTRML/8ABHc8ZuzrrswAAAAPAZ7ndEK/AAYhJRCmCVmAAAAAEAGe6WpCvwAGIJkmm+kg9pEAAAAqQZruSahBaJlMCG///qeEAAfL3b+ZZVtlPwKS0D+BTLaL9l7FF17BG6+IAAAAEUGfDEURLC//AAS3POzG1uIsAAAADwGfK3RCvwAD+F+LgPz9wQAAABABny1qQr8ABnHVPJgevnqBAAAAGkGbL0moQWyZTAhv//6nhAAHy9g/wnBboXrBAAAAGUGbUEnhClJlMCHf/qmWAAKlpZXGaX9sU0AAAABOQZt0SeEOiZTAh3/+qZYAAs0R9N73PKoFPICOdEKxelI5y4//4hI8HyM2FQ/iBg3lVx0UOARzpCBDaUepXv/+IQMrzQDfAyhwf0yvMOnKAAAAFUGfkkURPC//AANMqZ+9x2HK5EiMOQAAABABn7F0Qr8ABDXak8r8lPDwAAAAEAGfs2pCvwAEd2Q8sbNYZ4AAAAAcQZu4SahBaJlMCHf//qmWAALf76vf5u+mljZXwQAAABFBn9ZFESwv/wADYB9Y+3pMWAAAAA4Bn/V0Qr8ABJdx3nnGrwAAAA8Bn/dqQr8ABHZW6UaQ80cAAAAnQZv8SahBbJlMCHf//qmWAAQn4nnMsrVNV4FKJAvApmuWORX/QnjAAAAAEkGeGkUVLC//AAT6gRY9zjY7vQAAABABnjl0Qr8ABHfJLutBIDvAAAAAEAGeO2pCvwAG6BY17zStIcEAAAAjQZogSahBbJlMCHf//qmWAAnP0c+Vj8yyz59tbB+4qGur5WEAAAAWQZ5eRRUsL/8AC6T69Z4vosZz6uChzAAAABABnn10Qr8ACj5okT4sxS4QAAAAEAGef2pCvwAPhzhr3mlZ78EAAAATQZpkSahBbJlMCHf//qmWAACVgAAAAAxBnoJFFSwv/wAAsoEAAAAQAZ6hdEK/AA8KhvZdV/BxwAAAABABnqNqQr8AF+Sti9XYcqLBAAAAEkGaqEmoQWyZTAhv//6nhAABJwAAABxBnsZFFSwv/wAawhgd8Qe5//iEF8T/9MqhDogNAAAAEAGe5XRCvwAkvpTwOmU4AIEAAAAQAZ7nakK/ACSyuRV4AoAAgAAAABpBmulJqEFsmUwId//+qZYADpJkJNw4KPm4sAAAAEdBmw1J4QpSZTAh3/6plgAO/xehr0tB0uP/+ISPFTV3mh2CVX64iPD///xAuWaumWwQW/CuL//+IQMqdXckbgkldQgM2sWbQQAAABRBnytFNEwv/wAR2etiCigUcG/54AAAABABn0p0Qr8AGIeTeVsoeqLAAAAADwGfTGpCvwAQXYjyXM+TUwAAABpBm1FJqEFomUwId//+qZYADpNUCuFhvCzoYQAAABVBn29FESwv/wASXHj6LFtdINTa5XEAAAAQAZ+OdEK/ABkpNCJ8WYpD8AAAAA8Bn5BqQr8AGSIvmbZkbL0AAAAsQZuVSahBbJlMCHf//qmWAA6nF6O8QnwKabL//CU972L//CStDi//7/zyEEEAAAAVQZ+zRRUsL/8AEVoRBPRXJ07WKwbgAAAADwGf0nRCvwAX4A+KTbJWFwAAABABn9RqQr8AF0UaJkTSs7lBAAAAHEGb2UmoQWyZTAh3//6plgAV331ff8XeSUubbBAAAAAWQZ/3RRUsL/8AGl9csZtf62kWEPSEgQAAABABnhZ0Qr8AI75qgdO1DhSBAAAAEAGeGGpCvwAkuaN5pirafMAAAAAcQZodSahBbJlMCHf//qmWABdtLMWmaA7vox65UwAAABBBnjtFFSwv/wAbpV3f5vu4AAAADwGeWnRCvwAkwgDoTkv6wQAAAA8BnlxqQr8AJbs8tw2bVAMAAAAcQZpBSahBbJlMCHf//qmWABePlJA4f6/rtGRAgAAAABBBnn9FFSwv/wAboRu9wGtAAAAADwGennRCvwAlrsoUm2SrZQAAAA8BnoBqQr8AJLa13fd7+sAAAAAcQZqFSahBbJlMCHf//qmWABZNLOUGaBT6MfplewAAABBBnqNFFSwv/wAaZV43sEz4AAAADwGewnRCvwAYh5N55xdPgQAAABABnsRqQr8AI7tEJuM+vT9pAAAAE0GayUmoQWyZTAh3//6plgAAlYEAAAAQQZ7nRRUsL/8AGmiW5+uJnwAAABABnwZ0Qr8AI67Unlfkpt7QAAAAEAGfCGpCvwAkrzRMiaVnL0AAAAAjQZsNSahBbJlMCG///qeEACx8yuNqWILLLID//hKljz//ZqEAAAAVQZ8rRRUsL/8AGmEbv6p+uQ33Vd2wAAAADwGfSnRCvwAjrsoUm2SrdwAAABABn0xqQr8AFra+c60ML2rBAAAAGkGbTkmoQWyZTAh3//6plgANlUgzQB6S+xLRAAAASkGbcknhClJlMCHf/qmWACAHt///4hI8VNXeaHYJVfriI8P///EC5Zq6ZbBBb8K4v//4hAyp1dyRuCSV1CAzXnCFB2/uXRYYPO1LAAAAFkGfkEU0TC//ACa590s5P3awGW9Pm4AAAAAQAZ+vdEK/ACC+okT4sxSAsAAAABABn7FqQr8ANM7Utw2bU7yBAAAAE0GbtkmoQWiZTAh3//6plgAAlYAAAAAMQZ/URREsL/8AALKAAAAAEAGf83RCvwA2DybzBLG0X2kAAAAQAZ/1akK/ADQ6ieRHX8DIQAAAACVBm/pJqEFsmUwId//+qZYAID9HPyr+SHL4hISD/+EqILP/9l+xAAAAFUGeGEUVLC//ACa0Byp/TOMJCr2xuQAAABABnjd0Qr8ANMApnlfkptdIAAAAEAGeOWpCvwAisnznWhhejMEAAAAfQZo+SahBbJlMCHf//qmWAA2nF6MK/57vMss+faoOQAAAABVBnlxFFSwv/wAP3/D1T/osW4ABK+kAAAAPAZ57dEK/ABYk5QpNslYvAAAAEAGefWpCvwAOKrg1x4q2vaAAAAAZQZpiSahBbJlMCHf//qmWAAiCpxH99X3dJgAAABBBnoBFFSwv/wAKPQIrSi/5AAAADwGev3RCvwAOLYrGEKuhQAAAABABnqFqQr8ADic4a95pWffBAAAAHEGapkmoQWyZTAh3//6plgANlBZi0zQHd9GPXVUAAAAQQZ7ERRUsL/8AD9/w9dZVwQAAAA8BnuN0Qr8AFizJ3Bsl5L8AAAAPAZ7lakK/ACG2td33fADBAAAAE0Ga6kmoQWyZTAh3//6plgAAlYEAAAAMQZ8IRRUsL/8AALKAAAAADwGfJ3RCvwAV7/4o0SV9jgAAAA8BnylqQr8AFe/+HMdabh0AAAAcQZsuSahBbJlMCHf//qmWAA2nF6MTQqBaKczPaAAAABVBn0xFFSwv/wAP3/FcNGA59FldqygAAAAQAZ9rdEK/ABYk1oyS3+vUwQAAABABn21qQr8ADoMweS5nyb2BAAAAJEGbckmoQWyZTAh3//6plgAJD1bIh7cXWWWU2X/+EqILP/9pLQAAABVBn5BFFSwv/wAKyyxSydLjcNz+vUgAAAAQAZ+vdEK/AA7bYGtplD12QAAAABABn7FqQr8ACayfOdaGF/XBAAAAI0GbtkmoQWyZTAh3//6plgADv8Xo7zZWWWQ///CVUM3//F1BAAAAFUGf1EUVLC//AAa/1x7e3Ez7sSbn0AAAABABn/N0Qr8ACS+okT4sxTBRAAAAEAGf9WpCvwAJLJ851oYX+0AAAAATQZv6SahBbJlMCHf//qmWAACVgQAAAAxBnhhFFSwv/wAAsoEAAAAQAZ43dEK/AAX55N0dt8OUgAAAAA8BnjlqQr8ABfgWNErnmOkAAAAkQZo+SahBbJlMCHf//qmWAAkPGTl8Qg2//CVUM3//pQF7tuWwAAAAFUGeXEUVLC//AArNAgm5v1yG+2aoYQAAAA8Bnnt0Qr8ACS+k7g2S85cAAAAQAZ59akK/AA6DMHkwPXwSgAAAABJBmmJJqEFsmUwIb//+p4QAAScAAAAQQZ6ARRUsL/8ACs5LZv0hWwAAABABnr90Qr8ADn8MBklv9glAAAAAEAGeoWpCvwAOgzB5MD18EoEAAAAaQZqjSahBbJlMCHf//qmWAA3lSDNAHpL7ElAAAAASQZrHSeEKUmUwId/+qZYAAJWBAAAAE0Ge5UU0TC//ABkfXLGbcTpj6vsAAAAQAZ8EdEK/ACLCAOdscaapYQAAABABnwZqQr8AIrmjeaYq2oDBAAAAGUGbC0moQWiZTAh3//6plgAVMJ0j++r7tpIAAAAQQZ8pRREsL/8AGSEcZ3KvoAAAABABn0h0Qr8AIb6iRPizFIARAAAADwGfSmpCvwAhwaB5MEZugAAAAB1Bm09JqEFsmUwIb//+p4QALFsIyam2GNd6J+9kNwAAABVBn21FFSwv/wAnybPzNuJwFUjwtBEAAAAQAZ+MdEK/ADYSLKvAiu5AgQAAABABn45qQr8ANgTJNN9JByJxAAAAEkGbk0moQWyZTAhv//6nhAABJwAAABNBn7FFFSwv/wAaaJbNTMsuQ0UaAAAAEAGf0HRCvwAjrq0ZJb/XT4EAAAAPAZ/SakK/ACO7PLcNm1QTAAAAGkGb1EmoQWyZTAh3//6plgAiCLDdGIRz7BPgAAAAJ0Gb+EnhClJlMCG//qeEAGldWkEHH8CmwOnfgUy2dnwKFJc6l+F0kQAAABBBnhZFNEwv/wA+H3ndXhDSAAAADwGeNXRCvwBWegHQnJee4QAAABABnjdqQr8AVCyITcZ9enG5AAAAGkGaOUmoQWiZTAh3//6plgA0Hwo+uxBuKggQAAAAJkGaXUnhClJlMCG//qeEAEG+On97XvF8CmvqDfgUqWj8CmdgRx4vAAAAEEGee0U0TC//ACfUGzzxeCAAAAAPAZ6adEK/ADYJNT1Z34rBAAAAEAGenGpCvwA2BHbnVMxvFcEAAAASQZqBSahBaJlMCGf//p4QAAR8AAAAFUGev0URLC//ACW7ZYrt1GPayek5wAAAABABnt50Qr8AM4ApnlfkpteZAAAAEAGewGpCvwA0wLGveaVnDcAAAAAaQZrCSahBbJlMCG///qeEAD5e+z6jjQkONSEAAAAXQZrjSeEKUmUwIb/+p4QAPgnhrPs+atoAAAAbQZsESeEOiZTAh3/+qZYAMVUgzQB6kgcP8RDxAAAAG0GbKEnhDyZTAh3//qmWADLfCj4Pj1r9LddQ+QAAABFBn0ZFETwv/wA7X3nm14n8zQAAAA8Bn2V0Qr8AUe0d55xa9IEAAAAQAZ9nakK/AE+bcirwBP7/gAAAABNBm2xJqEFomUwId//+qZYAAJWAAAAAE0GfikURLC//ACax86ZxXU9iZ+0AAAAQAZ+pdEK/ADTSLKvAiu5FgAAAAA8Bn6tqQr8ANMS0qRQJVfMAAAASQZuwSahBbJlMCG///qeEAAEnAAAAE0GfzkUVLC//ABiIluUzHzERV/8AAAAQAZ/tdEK/ACCu1J5X5KbgsQAAABABn+9qQr8AIa80TImlZzdAAAAAHkGb9EmoQWyZTAhv//6nhAApHup+4LHvlmI5w8n7WAAAABBBnhJFFSwv/wAYhVz0UPIpAAAADgGeMXRCvwAhu47zzi4nAAAADwGeM2pCvwAgsrkVeAKAGwAAABlBmjVJqEFsmUwIb//+p4QAEVHzHkYn+W7nAAAAIkGaWUnhClJlMCG//qeEACx7rmvxCAH/8JUsef/+al0Ij2EAAAAVQZ53RTRML/8AGmVd3m9+uQ32zQfhAAAADwGelnRCvwAWvMncGyXktQAAAA8BnphqQr8AI7sR5MD17p8AAAAZQZqaSahBaJlMCG///qeEACx8yuKZFCfBPQAAACNBmr5J4QpSZTAhn/6eEABu+Euo9NGOYKaQi//4hGSo//jekAAAABVBntxFNEwv/wAQ2gOVP6ZxhIVe3ykAAAAQAZ77dEK/ABdE6k8r8lN1cQAAABABnv1qQr8ADzAvOdaGF8vAAAAAGUGa/0moQWiZTAhv//6nhAAL/wmZXILenOYAAAAYQZsASeEKUmUwIb/+p4QAB5+EzK5Bb08jAAAAEUGbJEnhDomUwIZ//p4QAAR8AAAADEGfQkURPC//AACygQAAAA8Bn2F0Qr8AA++DwLYD8/cAAAAPAZ9jakK/AAPvg8CN1/l9AAAAGUGbZUmoQWiZTAhn//6eEAATbpt0shj80pkAAAAYQZuGSeEKUmUwIZ/+nhAADJ8JXluhmY/hAAAAGkGbqUvhCEOiRGCCgH8gH9h4AhX//jhAABFxAAAAJUGfyEIV/wKvY+1BxN2qw0km5aqGByy1u80qIJosmuadSkrOFDwAAAAQAZ/naRCvAAP4y/iR83A24AAADFltb292AAAAbG12aGQAAAAAAAAAAAAAAAAAAAPoAAAfkAABAAABAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAALg3RyYWsAAABcdGtoZAAAAAMAAAAAAAAAAAAAAAEAAAAAAAAfkAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAABEAAAARAAAAAAACRlZHRzAAAAHGVsc3QAAAAAAAAAAQAAH5AAAAQAAAEAAAAACvttZGlhAAAAIG1kaGQAAAAAAAAAAAAAAAAAADIAAAGUAFXEAAAAAAAtaGRscgAAAAAAAAAAdmlkZQAAAAAAAAAAAAAAAFZpZGVvSGFuZGxlcgAAAAqmbWluZgAAABR2bWhkAAAAAQAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAKZnN0YmwAAACWc3RzZAAAAAAAAAABAAAAhmF2YzEAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAABEAEQAEgAAABIAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAY//8AAAAwYXZjQwH0AA3/4QAXZ/QADZGbKCIR0IAAAAMAgAAAGQeKFMsBAAZo6+PESEQAAAAYc3R0cwAAAAAAAAABAAAAygAAAgAAAAAUc3RzcwAAAAAAAAABAAAAAQAABjBjdHRzAAAAAAAAAMQAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAgAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAMAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAIAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAgAAAAAAQAABAAAAAABAAAAAAAAABxzdHNjAAAAAAAAAAEAAAABAAAAygAAAAEAAAM8c3RzegAAAAAAAAAAAAAAygAABbIAAAAYAAAAHgAAABMAAAATAAAAEwAAAB4AAAAjAAAAFQAAABMAAAAUAAAALgAAABUAAAATAAAAFAAAAB4AAAAdAAAAUgAAABkAAAAUAAAAFAAAACAAAAAVAAAAEgAAABMAAAArAAAAFgAAABQAAAAUAAAAJwAAABoAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAWAAAAIAAAABQAAAAUAAAAHgAAAEsAAAAYAAAAFAAAABMAAAAeAAAAGQAAABQAAAATAAAAMAAAABkAAAATAAAAFAAAACAAAAAaAAAAFAAAABQAAAAgAAAAFAAAABMAAAATAAAAIAAAABQAAAATAAAAEwAAACAAAAAUAAAAEwAAABQAAAAXAAAAFAAAABQAAAAUAAAAJwAAABkAAAATAAAAFAAAAB4AAABOAAAAGgAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACkAAAAZAAAAFAAAABQAAAAjAAAAGQAAABMAAAAUAAAAHQAAABQAAAATAAAAFAAAACAAAAAUAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAIAAAABkAAAAUAAAAFAAAACgAAAAZAAAAFAAAABQAAAAnAAAAGQAAABQAAAAUAAAAFwAAABAAAAAUAAAAEwAAACgAAAAZAAAAEwAAABQAAAAWAAAAFAAAABQAAAAUAAAAHgAAABYAAAAXAAAAFAAAABQAAAAdAAAAFAAAABQAAAATAAAAIQAAABkAAAAUAAAAFAAAABYAAAAXAAAAFAAAABMAAAAeAAAAKwAAABQAAAATAAAAFAAAAB4AAAAqAAAAFAAAABMAAAAUAAAAFgAAABkAAAAUAAAAFAAAAB4AAAAbAAAAHwAAAB8AAAAVAAAAEwAAABQAAAAXAAAAFwAAABQAAAATAAAAFgAAABcAAAAUAAAAFAAAACIAAAAUAAAAEgAAABMAAAAdAAAAJgAAABkAAAATAAAAEwAAAB0AAAAnAAAAGQAAABQAAAAUAAAAHQAAABwAAAAVAAAAEAAAABMAAAATAAAAHQAAABwAAAAeAAAAKQAAABQAAAAUc3RjbwAAAAAAAAABAAAAMAAAAGJ1ZHRhAAAAWm1ldGEAAAAAAAAAIWhkbHIAAAAAAAAAAG1kaXJhcHBsAAAAAAAAAAAAAAAALWlsc3QAAAAlqXRvbwAAAB1kYXRhAAAAAQAAAABMYXZmNTYuMzYuMTAw\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize the game\n",
    "env = Environment(grid_size=size, max_time=T,temperature=temperature)\n",
    "\n",
    "# Initialize the agent!\n",
    "agent = RandomAgent()\n",
    "\n",
    "test(agent,env,epochs_test,prefix='random')\n",
    "HTML(display_videos('random0.mp4'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## DQN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us assume here that $T=\\infty$.\n",
    "\n",
    "***\n",
    "__Question 5__ Let $\\pi$ be a policy, show that:\n",
    "\n",
    "\\begin{equation*}\n",
    "Q^{\\pi}(s,a)=E_{(s',a')\\sim p(.|s,a)}[r(s,a)+\\gamma Q^{\\pi}(s',a')]\n",
    "\\end{equation*}\n",
    "\n",
    "Then, show that for the optimal policy $\\pi^*$ (we assume its existence), the following holds: \n",
    "\n",
    "\\begin{equation*}\n",
    "Q^{*}(s,a)=E_{s'\\sim \\pi^*(.|s,a)}[r(s,a)+\\gamma\\max_{a'}Q^{*}(s',a')].\n",
    "\\end{equation*}\n",
    "Finally, deduce that a plausible objective is:\n",
    "\n",
    "\\begin{equation*}\n",
    "\\mathcal{L}(\\theta)=E_{s' \\sim \\pi^*(.|s,a)}\\Vert r+\\gamma\\max_{a'}Q(s',a',\\theta)-Q(s,a,\\theta)\\Vert^{2}.\n",
    "\\end{equation*}\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1)** We have:\n",
    "\n",
    "\\begin{align}\n",
    "Q^\\pi(s,a) &= E_{p^{\\pi}}[\\sum_{t\\leq T}\\gamma^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a] \\\\\n",
    "           &= E_{p^{\\pi}}[r(s_{0},a_{0}) + \\sum_{1\\leq t\\leq T}\\gamma^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a] \\\\\n",
    "           &= E_{p^{\\pi}}[r(s_{0},a_{0})|s_{0}=s,a_{0}=a] + \\gamma E_{p^{\\pi}}[\\sum_{1\\leq t\\leq T}\\gamma^{t-1}r(s_{t},a_{t})|s_{0}=s,a_{0}=a] \\\\\n",
    "           &= r(s,a) + \\gamma E_{p^{\\pi}(a_{1},s_{1},...,a_{T},s_{T}|s_{0}=s,a_{0}=a)}[\\sum_{1 \\leq t\\leq T}\\gamma^{t-1}r(s_{t},a_{t})]\n",
    "\\end{align}\n",
    "\n",
    "But:\n",
    "\n",
    "\\begin{align}\n",
    "p^{\\pi}(a_{1},s_{1},...,a_{T},s_{T}|s_{0}=s,a_{0}=a) &= p^{\\pi}(a_{1}, s_{1}|s_0 = s, a_0 = a) \\times p^{\\pi}(a_{2},s_{2},...,a_{T},s_{T}|s_{0},a_{0}, a_{1}, s_{1}) \\\\\n",
    "           &= p^{\\pi}(a_{1}, s_{1}|s_0 = s, a_0 = a) \\times p^{\\pi}(a_{2},s_{2},...,a_{T},s_{T}|a_{1}, s_{1})\n",
    "\\end{align}\n",
    "\n",
    "because of the Markov property of $\\{(a_{t},s_{t})_{t\\leq T}\\}$\n",
    "\n",
    "So:\n",
    "\n",
    "\\begin{align}\n",
    "Q^\\pi(s,a) &= r(s,a) + \\gamma E_{p^{\\pi}(a_{1},s_{1},...,a_{T},s_{T}|s_{0}=s,a_{0}=a)}[\\sum_{1 \\leq t\\leq T}\\gamma^{t-1}r(s_{t},a_{t})] \\\\\n",
    "           &= r(s,a) + \\gamma E_{p^{\\pi}(a_{1}, s_{1}|s_0 = s, a_0 = a)}[E_{p^{\\pi}(a_{2},s_{2},...,a_{T},s_{T}|a_{1}, s_{1})}[\\sum_{1 \\leq t\\leq T}\\gamma^{t-1}r(s_{t},a_{t})]]\\\\\n",
    "           &= r(s,a) + \\gamma E_{p^{\\pi}(a_{1}, s_{1}|s_0 = s, a_0 = a)}[E_{p^{\\pi}(a_{2},s_{2},...,a_{T},s_{T})}[\\sum_{1 \\leq t\\leq T}\\gamma^{t-1}r(s_{t},a_{t})|a_{1}, s_{1}]] \\\\\n",
    "           &= r(s,a) + \\gamma E_{p^{\\pi}(a_{1}, s_{1}|s_0 = s, a_0 = a)}[Q^\\pi(s_1,a_1)] \\\\\n",
    "           &= E_{p^{\\pi}(a_{1}, s_{1}|s_0 = s, a_0 = a)}[r(s,a) + \\gamma Q^\\pi(s_1,a_1)] \\\\\n",
    "           &= E_{(s',a')\\sim p(.|s,a)}[r(s,a)+\\gamma Q^{\\pi}(s',a')]\n",
    "\\end{align}\n",
    "\n",
    "I.e : \\begin{equation*}\\boxed{\n",
    "Q^{\\pi}(s,a)=E_{(s',a')\\sim p(.|s,a)}[r(s,a)+\\gamma Q^{\\pi}(s',a')]}\n",
    "\\end{equation*}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2)** Then, we have:\n",
    "\n",
    "\\begin{align}\n",
    "Q^*(s,a) &= \\max_{\\pi} Q^\\pi(s,a)\\\\\n",
    "           &= \\max_{\\pi} E_{(s',a')\\sim p(.|s,a)}[r(s,a)+\\gamma Q^{\\pi}(s',a')]\\\\\n",
    "           &= r(s,a) +  \\gamma \\max_{\\pi}E_{(s',a')\\sim p(.|s,a)}[Q^{\\pi}(s',a')]\n",
    "\\end{align}\n",
    "\n",
    "But:\n",
    "\n",
    "\\begin{align}\n",
    "\\max_{\\pi}E_{(s',a')\\sim p(.|s,a)}[Q^{\\pi}(s',a')] &= \\max_{\\pi} \\sum\\limits_{s', a'} p(s'|s,a)\\pi(a'|s) Q^\\pi(s',a')\\\\\n",
    "           &= \\max_{\\pi} \\sum\\limits_{s'} p(s'|s,a) \\sum\\limits_{a'} \\pi(a'|s) Q^\\pi(s',a')\\\\\n",
    "           &= \\sum\\limits_{s'} p(s'|s,a) \\max_{\\pi}\\sum\\limits_{a'} \\pi(a'|s) Q^\\pi(s',a')\n",
    "\\end{align}\n",
    "\n",
    "So, we want to maximize $\\sum\\limits_{a'} \\pi(a'|s) Q^\\pi(s',a')$ by choosing the policy $\\pi$. \n",
    "\n",
    "But we know that $\\forall s,a \\:Q^\\pi(s,a) \\leq Q^*(s,a)$, so: by choosing $\\pi_0 = \\pi^* \\: \\forall s \\neq s'$ and $ \\: \\pi_0(a'|s') =1 $ for $a' = \\underset{a}{\\mathrm{argmax}} Q^*(s',a)$, we have:\n",
    "\n",
    "$\\sum\\limits_{a'} \\pi(a'|s) Q^\\pi(s',a') \\leq \\sum\\limits_{a'} \\pi(a'|s) Q^*(s',a') \\leq  Q^*(s',\\underset{a}{\\mathrm{argmax}} Q^*(s',a)) = \\sum\\limits_{a'} \\pi_0(a'|s) Q^{\\pi_0}(s',a')$. We found the policy that maximizes.\n",
    "\n",
    "So:\n",
    "\n",
    "\\begin{align}\n",
    "\\max_{\\pi}E_{(s',a')\\sim p(.|s,a)}[Q^{\\pi}(s',a')] &= \\sum\\limits_{s'} p(s'|s,a) \\max_{\\pi}\\sum\\limits_{a'} \\pi(a'|s) Q^\\pi(s',a')\\\\\n",
    "           &= \\sum\\limits_{s'} p(s'|s,a) \\max_{a'} Q^*(s',a') \\\\\n",
    "           &= E_{s' \\sim p(.|s,a)}[\\max_{a'} Q^*(s',a')]\n",
    "\\end{align}\n",
    "\n",
    "Then, we conclude that:\n",
    "\n",
    "\\begin{align}\n",
    "Q^*(s,a) &= \\max_{\\pi} Q^\\pi(s,a)\\\\\n",
    "           &= \\max_{\\pi} E_{(s',a')\\sim p(.|s,a)}[r(s,a)+\\gamma Q^{\\pi}(s',a')]\\\\\n",
    "           &= r(s,a) +  \\gamma \\max_{\\pi}E_{(s',a')\\sim p(.|s,a)}[Q^{\\pi}(s',a')] \\\\\n",
    "           &= r(s,a) +  \\gamma E_{s' \\sim p(.|s,a)}[\\max_{a'} Q^*(s',a')] \\\\\n",
    "           &= \\boxed{E_{s' \\sim p(.|s,a)}[r(s,a) + \\gamma \\max_{a'} Q^*(s',a')]}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3)**\n",
    "\n",
    "Finally:\n",
    "\n",
    "\n",
    "We want to find a policy (a Q function) such that : $Q^\\pi(s,a) = E_{s' \\sim p(.|s,a)}[r(s,a) + \\gamma \\max_{a'} Q^\\pi(s',a')]$ \n",
    "\n",
    "i.e $Q^\\pi(s,a) = r(s,a) + \\gamma \\max_{a'} Q^\\pi(s',a')$ in expectancy w.r.t $s'$. \n",
    "\n",
    "So, we want $Q^\\pi(s,a) - r(s,a) + \\gamma \\max_{a'} Q^\\pi(s',a')$ close to 0 in expectancy w.r.t $s'$.\n",
    "\n",
    "So, we deduce a plausible objective:\n",
    "\n",
    "\\begin{equation*}\n",
    "\\mathcal{L}(\\theta)=E_{s' \\sim \\pi^*(.|s,a)}\\Vert r+\\gamma\\max_{a'}Q(s',a',\\theta)-Q(s,a,\\theta)\\Vert^{2}.\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The DQN-learning algorithm relies on these derivations to train the parameters $\\theta$ of a Deep Neural Network:\n",
    "\n",
    "1. At the state $s_t$, select the action $a_t$ with best reward using $Q_t$ and store the results;\n",
    "\n",
    "2. Obtain the new state $s_{t+1}$ from the environment $p$;\n",
    "\n",
    "3. Store $(s_t,a_t,s_{t+1})$;\n",
    "\n",
    "4. Obtain $Q_{t+1}$ by minimizing  $\\mathcal{L}$ from a recovered batch from the previously stored results.\n",
    "\n",
    "***\n",
    "__Question 6__ Implement the class ```Memory``` that stores moves (in a replay buffer) via ```remember``` and provides a ```random_access``` to these. Specify a maximum memory size to avoid side effects. You can for example use a ```list()``` and set by default ```max_memory=100```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Memory(object):\n",
    "    def __init__(self, max_memory=100):\n",
    "        self.max_memory = max_memory\n",
    "        self.memory = list()\n",
    "\n",
    "    def remember(self, m):\n",
    "        if len(self.memory) > self.max_memory:\n",
    "            del self.memory[0]\n",
    "        self.memory.append(m)\n",
    "\n",
    "    def random_access(self):\n",
    "        random_index = np.random.randint(0,len(self.memory))\n",
    "        return self.memory[random_index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The pipeline we will use for training is given below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(agent,env,epoch,prefix=''):\n",
    "    # Number of won games\n",
    "    score = 0\n",
    "    loss = 0\n",
    "\n",
    "    for e in range(epoch):\n",
    "        # At each epoch, we restart to a fresh game and get the initial state\n",
    "        state = env.reset()\n",
    "        # This assumes that the games will terminate\n",
    "        game_over = False\n",
    "\n",
    "        win = 0\n",
    "        lose = 0\n",
    "\n",
    "        while not game_over:\n",
    "            # The agent performs an action\n",
    "            action = agent.act(state)\n",
    "\n",
    "            # Apply an action to the environment, get the next state, the reward\n",
    "            # and if the games end\n",
    "            prev_state = state\n",
    "            state, reward, game_over = env.act(action)\n",
    "\n",
    "            # Update the counters\n",
    "            if reward > 0:\n",
    "                win = win + reward\n",
    "            if reward < 0:\n",
    "                lose = lose -reward\n",
    "\n",
    "            # Apply the reinforcement strategy\n",
    "            loss = agent.reinforce(prev_state, state,  action, reward, game_over)\n",
    "\n",
    "        # Save as a mp4\n",
    "        if e % 10 == 0:\n",
    "            env.draw(prefix+str(e))\n",
    "\n",
    "        # Update stats\n",
    "        score += win-lose\n",
    "        \n",
    "\n",
    "        print(\"Epoch {:03d}/{:03d} | Loss {:.4f} | Win/lose count {}/{} ({})\"\n",
    "              .format(e, epoch, loss, win, lose, win-lose))\n",
    "        agent.save(name_weights=prefix+'model.h5',name_model=prefix+'model.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "__Question 7__ Implement the DQN training algorithm using a cascade of fully connected layers. You can use different learning rate, batch size or memory size parameters. In particular, the loss might oscillate while the player will start to win the games. You have to find a good criterium."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model, Sequential\n",
    "from keras.layers import Dropout, Input, Conv1D, MaxPooling1D, Flatten, Dense\n",
    "\n",
    "class DQN(Agent):\n",
    "    def __init__(self, grid_size,  epsilon = 0.1, memory_size=100, batch_size = 16,n_state=2):\n",
    "        super(DQN, self).__init__(epsilon = epsilon)\n",
    "\n",
    "        # Discount for Q learning\n",
    "        self.discount = 0.99\n",
    "        \n",
    "        self.grid_size = grid_size\n",
    "        \n",
    "        # number of state\n",
    "        self.n_state = n_state\n",
    "\n",
    "        # Memory\n",
    "        self.memory = Memory(memory_size)\n",
    "        \n",
    "        # Batch size when learning\n",
    "        self.batch_size = batch_size\n",
    "        \n",
    "        # Q learned by the agent (init with zero matrix)\n",
    "        self.model = Sequential()\n",
    "\n",
    "    def learned_act(self, s):\n",
    "        return np.argmax(self.model.predict(np.array([s])))\n",
    "        \n",
    "        \n",
    "    def reinforce(self, s_, n_s_, a_, r_, game_over_):\n",
    "        # Two steps: first memorize the states, second learn from the pool\n",
    "\n",
    "        self.memory.remember([s_, n_s_, a_, r_, game_over_])\n",
    "        \n",
    "        input_states = np.zeros((self.batch_size, 5,5,self.n_state))\n",
    "        target_q = np.zeros((self.batch_size, 4))\n",
    "        \n",
    "        for i in range(self.batch_size):\n",
    "            \n",
    "            prev_state, state,  action, reward, game_over_ = self.memory.random_access()\n",
    "            \n",
    "            target_q[i, :] = self.model.predict(np.array([state]))\n",
    "\n",
    "            if game_over_:\n",
    "                y = reward\n",
    "            else:\n",
    "                y = reward + self.discount*np.max(self.model.predict(np.array([state])))\n",
    "                \n",
    "            target_q[i, action]= y # change target for this action\n",
    "            input_states[i] = prev_state\n",
    "            \n",
    "        # HINT: Clip the target to avoid exploiding gradients.. -- clipping is a bit tighter\n",
    "        target_q = np.clip(target_q, -3, 3)\n",
    "\n",
    "        l = self.model.train_on_batch(np.array(input_states), np.array(target_q))\n",
    "\n",
    "\n",
    "        return l\n",
    "\n",
    "    def save(self,name_weights='model.h5',name_model='model.json'):\n",
    "        self.model.save_weights(name_weights, overwrite=True)\n",
    "        with open(name_model, \"w\") as outfile:\n",
    "            json.dump(self.model.to_json(), outfile)\n",
    "            \n",
    "    def load(self,name_weights='model.h5',name_model='model.json'):\n",
    "        with open(name_model, \"r\") as jfile:\n",
    "            model = model_from_json(json.load(jfile))\n",
    "        model.load_weights(name_weights)\n",
    "        model.compile(\"sgd\", \"mse\")\n",
    "        self.model = model\n",
    "\n",
    "        \n",
    "        \n",
    "            \n",
    "class DQN_FC(DQN):\n",
    "    def __init__(self, *args, lr=0.1,**kwargs):\n",
    "        super(DQN_FC, self).__init__( *args,**kwargs)\n",
    "        \n",
    "        # NN Model\n",
    "        model = Sequential()\n",
    "        model.add(Dense(100, input_shape=(5,5,self.n_state), activation='relu'))\n",
    "        model.add(Dense(100, activation='relu'))\n",
    "        model.add(Flatten())\n",
    "        model.add(Dense(4, activation='relu'))\n",
    "\n",
    "\n",
    "        model.compile(sgd(lr=lr, decay=1e-4, momentum=0.0), \"mse\")\n",
    "        self.model = model        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Test of FC**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 000/010 | Loss 0.0030 | Win/lose count 4.0/4.0 (0.0)\n",
      "Epoch 001/010 | Loss 0.0046 | Win/lose count 2.0/3.0 (-1.0)\n",
      "Epoch 002/010 | Loss 0.0051 | Win/lose count 5.5/6.0 (-0.5)\n",
      "Epoch 003/010 | Loss 0.0122 | Win/lose count 0.5/0 (0.5)\n",
      "Epoch 004/010 | Loss 0.0023 | Win/lose count 4.0/1.0 (3.0)\n",
      "Epoch 005/010 | Loss 0.0118 | Win/lose count 2.5/1.0 (1.5)\n",
      "Epoch 006/010 | Loss 0.0073 | Win/lose count 2.5/6.0 (-3.5)\n",
      "Epoch 007/010 | Loss 0.0115 | Win/lose count 1.0/3.0 (-2.0)\n",
      "Epoch 008/010 | Loss 0.0268 | Win/lose count 5.0/5.0 (0.0)\n",
      "Epoch 009/010 | Loss 0.0061 | Win/lose count 3.5/3.0 (0.5)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFsttZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAALzZYiEADv//vb8/AptUwndP/9P+VxN2UIE1zAALTEAIgTYlI6+z4vwKZrZf8CiWgOtBO4OjGRz32riKnQn3+g3zFLlzLMKpJEIRm4FKFhNViiJdj0Ddby3aX/3z/+FiY9b0jBbqAjpeS35fgKsHxZZ2VBNgHZLV744E5X7s/9qnGmBFqp9PgnCdwU0/lUJRf2ehcw5+yBc+s220VkcWVkn4558unm2L1cqqNExGI3XwybpUYQDO7aNrjkkvogjMedFqOoJADICgGzZFVDQTi6qXz9zJEZrxxwSjY6lLYiTBHGaKTETJMOeXsk+a1ctcvdqfk/mCkeCqe8GPAc52RxhWX0OBcNgWGnf6Od/Gw6jx8ETtcl8YrD+XtnxzWFD1KQOEfQEMOHxZRBSErfkHHD9l8gRxhZahCXjk/TacJW4ICz/CMK1K/WX1s8RQe1zPdRM+pt8CNpOV5lisSttB0N8Ge9wxKTr8ymdP8TyWkQV/591QKOpFYKvmK6QkFQi0fJZ6jFiTJ4WPiDb95uN7b5oUYoT/ACjkcYOvKyZ2MFbWkJpxW9Y5HQigOqHMDhHRyZ8EGKQVSlJKH/mZFqTvb4UCuMh7CGFzcJtbd5QkyYPxuut3ziI9Pxlv8VrX2nJOL2VXs56FLcAdPfzuTazeeYWc9GwQ2gYNGNeFcaFFSIqSh+J5pByVP4kaIqD7foiQJAYttVpPNB1UQZIztFKOlEADSx4QMYKWlnfKwh3xLaVYqVodqlZO4tOMZMdgdfDNigu7EqRHCrAfne0K5EWTgdAlyefM863u5QVjWeaGi+4lHD6SW+LyRMspIHIEkXkYauZZ1H8tC/xe7UHO0YA4ga4RFB7p/wkOgXC2kCsP+dZnUVb9jiKpvTxLOE7nLaQZv4sx3PXjSf/9GuH2aqFU4dLxtxuSzgXVTIYBoqw5IUt14o/qkcVJcAQk26LDJv0Aap0fkwOMd2/bHLnhiIZX/mRjho6FnkDfCQAEESo3mpWQb5zhg0AAAAfQZokbEO//qmWAGWP56g+Cwl1sTHB2gBhzO5+d/MuDgAAAA5BnkJ4hf8Adr+KuzJFiwAAABABnmF0Qr8Ao6dSeV+SmzQQAAAAEAGeY2pCvwBsErYsjnWgrZkAAAATQZpoSahBaJlMCHf//qmWAACVgQAAAAxBnoZFESwv/wAAsoEAAAAQAZ6ldEK/AGwsq7YmI6Vy8QAAABABnqdqQr8AbBK2LI51oK2YAAAAE0GarEmoQWyZTAh3//6plgAAlYAAAAAMQZ7KRRUsL/8AALKBAAAAEAGe6XRCvwBsLKu2JiOlcvAAAAAQAZ7rakK/AGwStiyOdaCtmAAAABNBmvBJqEFsmUwId//+qZYAAJWBAAAADEGfDkUVLC//AACygQAAABABny10Qr8AbCyrtiYjpXLxAAAAEAGfL2pCvwBsErYsjnWgrZgAAAATQZs0SahBbJlMCHf//qmWAACVgAAAAAxBn1JFFSwv/wAAsoEAAAAQAZ9xdEK/AGwsq7YmI6Vy8AAAABABn3NqQr8AbBK2LI51oK2YAAAAEkGbeEmoQWyZTAhv//6nhAABJwAAAAxBn5ZFFSwv/wAAsoAAAAAQAZ+1dEK/AGwsq7YmI6Vy8QAAABABn7dqQr8AbBK2LI51oK2ZAAAAJkGbvEmoQWyZTAhv//6nhACCoAs3TvDnzLK8YZ+BTLZ2fAoUex0XAAAAEEGf2kUVLC//AE+oNnniyCEAAAAPAZ/5dEK/AGwSUQpgi5eAAAAAEAGf+2pCvwBsCO3OqZjd1cEAAAAaQZv9SahBbJlMCG///qeEAHy99n1HGhIcNSEAAAAZQZoeSeEKUmUwId/+qZYAKX76vrsQbioN0AAAABZBmiJJ4Q6JlMCHf/6plgARBFhujE90AAAADkGeQEURPC//ABR2VDZhAAAAEAGef3RCvwAqtlHfgA+3nsAAAAAQAZ5hakK/ACq2Ud7PH289gQAAABNBmmZJqEFomUwId//+qZYAAJWAAAAADEGehEURLC//AACygQAAABABnqN0Qr8AKrZR34APt57BAAAAEAGepWpCvwAqtlHezx9vPYEAAAAcQZqqSahBbJlMCG///qeEADT+wf55BWqZCRb52QAAABBBnshFFSwv/wAfD+KvIqSgAAAAEAGe53RCvwArKdSeV+Sm2vAAAAAQAZ7pakK/AB0FcGuxLwK0qQAAABtBmutJqEFsmUwId//+qZYAEZ+l2UfndIUwjSQAAAAYQZsPSeEKUmUwId/+qZYAC8fAQB/f1dygAAAADkGfLUU0TC//AA3Qi5AhAAAADwGfTHRCvwAS3cd0dt8LtwAAAA8Bn05qQr8AEuCUGvu7eHEAAAASQZtTSahBaJlMCG///qeEAAEnAAAADEGfcUURLC//AACygAAAABkBn5B0Qr8AEqfWN8JSp//xCA5J/33vcLtxAAAADwGfkmpCvwAS4JQa+7t4cAAAABxBm5dJqEFsmUwIZ//+nhAAiohyrcF52vr77c6QAAAAEEGftUUVLC//ABWaBBShkJkAAAAPAZ/UdEK/ABLboB80yF24AAAAEAGf1mpCvwAdBmDyYHr3coEAAAAZQZvYSahBbJlMCGf//p4QAIt8Q863QMkRfQAAABhBm/lJ4QpSZTAhv/6nhAAi30c0FazKbDcAAAAYQZoaSeEOiZTAhv/+p4QAId8dMf4fVtxxAAAAGEGaO0nhDyZTAh3//qmWABCerYTLM+fJrwAAACpBml9J4Q8mUwIb//6nhAAf32D+fB41/vTSeBTWZrtwKSzzuBTLaLThKYEAAAAUQZ59RRE8L/8AE1nrLFQspYsCOMEAAAAQAZ6cdEK/ABpgAAyQ8m9vgAAAABABnp5qQr8AGSI7c6pmN7xAAAAAGkGagEmoQWiZTAh3//6plgAOp8KMqszbMDrhAAAAHEGapEnhClJlMCHf/qmWAA5PtL+xbKRuJtLnEzgAAAAUQZ7CRTRML/8AENoDnBwT0zphakEAAAAQAZ7hdEK/ABdE5jgPygAOYAAAABABnuNqQr8ADzMweTA9fAmBAAAAE0Ga6EmoQWiZTAh3//6plgAAlYEAAAAMQZ8GRREsL/8AALKBAAAAIQGfJXRCvwAX/S8//8f4vSg1Pzh2rWrrlUvVn44lac5rwQAAABABnydqQr8AF+twHQnK5CUWAAAAE0GbLEmoQWyZTAh3//6plgAAlYAAAAAMQZ9KRRUsL/8AALKBAAAAEAGfaXRCvwAX5QgdC0sOL+4AAAAQAZ9rakK/ABfrcB0JyuQlFgAAABNBm3BJqEFsmUwId//+qZYAAJWBAAAADEGfjkUVLC//AACygQAAABABn610Qr8AF+UIHQtLDi/vAAAAEAGfr2pCvwAX63AdCcrkJRYAAAATQZu0SahBbJlMCHf//qmWAACVgAAAAAxBn9JFFSwv/wAAsoEAAAAQAZ/xdEK/ABflCB0LSw4v7gAAABABn/NqQr8AF+twHQnK5CUWAAAAE0Gb+EmoQWyZTAh3//6plgAAlYEAAAAMQZ4WRRUsL/8AALKAAAAAEAGeNXRCvwAX5QgdC0sOL+8AAAAQAZ43akK/ABfrcB0JyuQlFwAAABNBmjxJqEFsmUwId//+qZYAAJWAAAAADEGeWkUVLC//AACygQAAABABnnl0Qr8AF+UIHQtLDi/uAAAAEAGee2pCvwAX63AdCcrkJRcAAAAYQZpgSahBbJlMCHf//qmWABb/lOf+5/zBAAAAJEGenkUVLC//ABrA78fCU3P/8QgIDLP/8QMdqz/8/2uuIsoeAwAAABABnr10Qr8AJL6Rvr+01/0gAAAAEAGev2pCvwAksrkVeAKAAIEAAAATQZqkSahBbJlMCHf//qmWAACVgAAAAAxBnsJFFSwv/wAAsoEAAAAQAZ7hdEK/ABfrKu76hyEosAAAABABnuNqQr8AF+StjCBqTH7hAAAAF0Ga6EmoQWyZTAhv//6nhAAMO6tIMglhAAAADkGfBkUVLC//AAdBOe7RAAAAEAGfJXRCvwAX6yru+ochKLEAAAAQAZ8nakK/AA8KhvYrR9whgAAAABJBmyxJqEFsmUwIb//+p4QAAScAAAAMQZ9KRRUsL/8AALKBAAAAEAGfaXRCvwAPCob2XVfwccAAAAAQAZ9rakK/AA8KhvYrR9whgAAAABpBm21JqEFsmUwIb//+p4QAEu+On1HGhIdZQQAAABlBm45J4QpSZTAh3/6plgAGUgsrjNL+2ETBAAAAJ0GbsknhDomUwId//qmWACQ/PI5llapqvApRIF4FM1yxp1YcdKvdZwAAABdBn9BFETwv/wArNAabxRd0uPn2UPNvwAAAABABn+90Qr8AGIkteB0ynECAAAAAEAGf8WpCvwA6DPAuv7cP0cEAAAATQZv2SahBaJlMCHf//qmWAACVgAAAAAxBnhRFESwv/wAAsoAAAAAQAZ4zdEK/ADlqG9l1X8C+QQAAABABnjVqQr8AOWob2K0fbwmAAAAAEkGaOkmoQWyZTAhv//6nhAABJwAAAAxBnlhFFSwv/wAAsoEAAAAQAZ53dEK/ADlqG9l1X8C+QAAAABABnnlqQr8AOWob2K0fbwmBAAAAHEGae0moQWyZTAh3//6plgAkPyOghn79kG4qEeAAAAAbQZqfSeEKUmUwId/+qZYAF499X3xhUC0UxDZPAAAAEEGevUU0TC//ABuhHGdyqqEAAAAQAZ7cdEK/ACa7jvK2UPTxgAAAAA8Bnt5qQr8AGSsWBdf4J8AAAAATQZrDSahBaJlMCHf//qmWAACVgQAAAAxBnuFFESwv/wAAsoAAAAAQAZ8AdEK/ABjc5O/AB9vxQQAAABABnwJqQr8AGNzk72ePt+KAAAAAE0GbB0moQWyZTAh3//6plgAAlYEAAAAMQZ8lRRUsL/8AALKBAAAAEAGfRHRCvwAY3OTvwAfb8UEAAAAQAZ9GakK/ABjc5O9nj7figQAAABNBm0tJqEFsmUwId//+qZYAAJWAAAAADEGfaUUVLC//AACygAAAABABn4h0Qr8AGNzk78AH2/FBAAAAEAGfimpCvwAY3OTvZ4+34oAAAAATQZuPSahBbJlMCHf//qmWAACVgAAAAAxBn61FFSwv/wAAsoEAAAAQAZ/MdEK/ABjc5O/AB9vxQQAAAA8Bn85qQr8AGcBY0SueXosAAAAcQZvTSahBbJlMCHf//qmWABdtLOUGaBT6MfplUgAAABBBn/FFFSwv/wAboRxncqqgAAAAEAGeEHRCvwAlvqJE+LMUfbEAAAAPAZ4SakK/ADoGodC0bYNAAAAAE0GaF0moQWyZTAh3//6plgAAlYAAAAAMQZ41RRUsL/8AALKBAAAAEAGeVHRCvwA6FisXn8DkvkAAAAAQAZ5WakK/ADoGoc/zLeAPQQAAAB9BmltJqEFsmUwId//+qZYAF499X3ZP07uZZZ8+zO/BAAAAFUGeeUUVLC//ABuhG7+qfrkNlt53iAAAABABnph0Qr8AJa6tGSW/10eBAAAAEAGemmpCvwAX4ltOvAFARIAAAAATQZqfSahBbJlMCHf//qmWAACVgQAAAAxBnr1FFSwv/wAAsoEAAAAQAZ7cdEK/AA81isXn8DlxwAAAABABnt5qQr8ADzGoc/zLeFJAAAAAHEGaw0moQWyZTAh3//6plgAJwUdECzQHd9GPXe8AAAAQQZ7hRRUsL/8AC6UCClDOmAAAAA8BnwB0Qr8ADzWKxhCrmsEAAAAQAZ8CakK/AA+LMHkwPXwEgAAAAB5BmwdJqEFsmUwIb//+p4QAE26bcztYbmWWJkdey5UAAAAVQZ8lRRUsL/8AEVj50ziu3sNiTbXJAAAAEAGfRHRCvwAYiyruQ2VKXuEAAAAQAZ9GakK/ABfiO3OtDC9lwQAAABpBm0hJqEFsmUwId//+qZYACU/Hn79kG4qc4AAAABJBm2xJ4QpSZTAh3/6plgAAlYAAAAAMQZ+KRTRML/8AALKBAAAAEAGfqXRCvwAJkqR34APuOMAAAAAQAZ+rakK/AAmSpHezx9xxgAAAABNBm7BJqEFomUwId//+qZYAAJWBAAAADEGfzkURLC//AACygQAAABABn+10Qr8ACZKkd+AD7jjBAAAAEAGf72pCvwAJkqR3s8fccYAAAAATQZv0SahBbJlMCHf//qmWAACVgAAAAAxBnhJFFSwv/wAAsoEAAAAQAZ4xdEK/AAmSpHfgA+44wAAAABABnjNqQr8ACZKkd7PH3HGAAAAAHEGaOEmoQWyZTAh3//6plgAJgUdECzQHd9GPXf8AAAAQQZ5WRRUsL/8AC10CClDO6AAAAA8BnnV0Qr8ACa2jFwH5o2EAAAAQAZ53akK/AA8zPAuv1ilSwQAAABNBmnxJqEFsmUwId//+qZYAAJWAAAAADEGemkUVLC//AACygQAAAA8Bnrl0Qr8ADw4PAtgPzGsAAAAPAZ67akK/AA8ODwI3X+FJAAAAEkGaoEmoQWyZTAhv//6nhAABJwAAAAxBnt5FFSwv/wAAsoAAAAAPAZ79dEK/AA8ODwLYD8xrAAAADwGe/2pCvwAPDg8CN1/hSQAAABJBmuRJqEFsmUwIb//+p4QAAScAAAAMQZ8CRRUsL/8AALKBAAAADwGfIXRCvwAPDg8C2A/MawAAABABnyNqQr8AF+Sti9XYcqLBAAAAEkGbKEmoQWyZTAhf//6MsAAEjQAAAAxBn0ZFFSwv/wAAsoEAAAAPAZ9ldEK/AA8ODwLYD8xrAAAADwGfZ2pCvwAPDg8CN1/hSQAAABpBm2lLqEIQWyRGCCgH8gH9h4AhX/44QAARcAAADGFtb292AAAAbG12aGQAAAAAAAAAAAAAAAAAAAPoAAAfkAABAAABAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAALi3RyYWsAAABcdGtoZAAAAAMAAAAAAAAAAAAAAAEAAAAAAAAfkAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAABEAAAARAAAAAAACRlZHRzAAAAHGVsc3QAAAAAAAAAAQAAH5AAAAQAAAEAAAAACwNtZGlhAAAAIG1kaGQAAAAAAAAAAAAAAAAAADIAAAGUAFXEAAAAAAAtaGRscgAAAAAAAAAAdmlkZQAAAAAAAAAAAAAAAFZpZGVvSGFuZGxlcgAAAAqubWluZgAAABR2bWhkAAAAAQAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAKbnN0YmwAAACWc3RzZAAAAAAAAAABAAAAhmF2YzEAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAABEAEQAEgAAABIAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAY//8AAAAwYXZjQwH0AA3/4QAXZ/QADZGbKCIR0IAAAAMAgAAAGQeKFMsBAAZo6+PESEQAAAAYc3R0cwAAAAAAAAABAAAAygAAAgAAAAAUc3RzcwAAAAAAAAABAAAAAQAABjhjdHRzAAAAAAAAAMUAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAgAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAQAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAFqAAAACMAAAASAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAACoAAAAUAAAAEwAAABQAAAAeAAAAHQAAABoAAAASAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAIAAAABQAAAAUAAAAFAAAAB8AAAAcAAAAEgAAABMAAAATAAAAFgAAABAAAAAdAAAAEwAAACAAAAAUAAAAEwAAABQAAAAdAAAAHAAAABwAAAAcAAAALgAAABgAAAAUAAAAFAAAAB4AAAAgAAAAGAAAABQAAAAUAAAAFwAAABAAAAAlAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAHAAAACgAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAbAAAAEgAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAAB4AAAAdAAAAKwAAABsAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAIAAAAB8AAAAUAAAAFAAAABMAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAATAAAAIAAAABQAAAAUAAAAEwAAABcAAAAQAAAAFAAAABQAAAAjAAAAGQAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACAAAAAUAAAAEwAAABQAAAAiAAAAGQAAABQAAAAUAAAAHgAAABYAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACAAAAAUAAAAEwAAABQAAAAXAAAAEAAAABMAAAATAAAAFgAAABAAAAATAAAAEwAAABYAAAAQAAAAEwAAABQAAAAWAAAAEAAAABMAAAATAAAAHgAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1Ni4zNi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras import backend as K \n",
    "K.clear_session()\n",
    "epochs_train = 10\n",
    "env = Environment(grid_size=size, max_time=T, temperature=0.3)\n",
    "agent = DQN_FC(size, lr=.001, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "train(agent, env, epochs_train, prefix='fc_train')\n",
    "HTML(display_videos('fc_train10.mp4'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Stopping criterium:**\n",
    "\n",
    "If we want a stopping criterion for the learning. A criterion could be to **check if the policy is updated or not**. However, it's quite expensive because the number of state is big. It's the number of combination of cheese/poison/empty cells in a 5x5 grid (malus 1 : the agent cell): it's $3^{25-1} = 3^{24}$. However, we can decide to reduce the number of state (just for the stopping criterion) by doing something similar to max-pooling for instance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "__Question 8__ Implement the DQN training algorithm using a CNN (for example, 2 convolutional layers and one final fully connected layer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DQN_CNN(DQN):\n",
    "    def __init__(self, *args,lr=0.1,**kwargs):\n",
    "        super(DQN_CNN, self).__init__(*args,**kwargs)\n",
    "        \n",
    "        model = Sequential()\n",
    "        model.add(Conv2D(16, (2, 2), activation='relu', input_shape=(5,5,self.n_state), padding='same'))\n",
    "        model.add(Conv2D(32, (2, 2), activation='relu', padding='same'))\n",
    "        model.add(Flatten())\n",
    "        model.add(Dense(4))\n",
    "        #model.add(Dropout(0.5))\n",
    "        \n",
    "        model.compile(sgd(lr=lr, decay=1e-4, momentum=0.0), \"mse\")\n",
    "        self.model = model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Test of CNN**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 000/010 | Loss 0.0185 | Win/lose count 3.0/9.0 (-6.0)\n",
      "Epoch 001/010 | Loss 0.0180 | Win/lose count 1.5/6.0 (-4.5)\n",
      "Epoch 002/010 | Loss 0.0078 | Win/lose count 2.0/2.0 (0.0)\n",
      "Epoch 003/010 | Loss 0.0480 | Win/lose count 3.0/4.0 (-1.0)\n",
      "Epoch 004/010 | Loss 0.0034 | Win/lose count 3.0/7.0 (-4.0)\n",
      "Epoch 005/010 | Loss 0.0314 | Win/lose count 2.5/5.0 (-2.5)\n",
      "Epoch 006/010 | Loss 0.0145 | Win/lose count 2.0/7.0 (-5.0)\n",
      "Epoch 007/010 | Loss 0.0057 | Win/lose count 5.0/3.0 (2.0)\n",
      "Epoch 008/010 | Loss 0.0034 | Win/lose count 3.0/5.0 (-2.0)\n",
      "Epoch 009/010 | Loss 0.0138 | Win/lose count 6.0/5.0 (1.0)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFXdtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAALFZYiEADf//vaH+BTZWBO6f/0/4LrOnzl/nbABNNgBSG5gp/w1NF8Cktv/wKZbHk60DzAM9KJLpsWn+In8Ef/rz0aYC0tOWWNfW79rQ4N9t3GXgYeSDhpEpypk51RWUyALVa4WvnPGzvJG+N0GeL4h8WgaOeOYpzS9DARwRAUIyA8goSYalQ1oWkZ9Los27vPJvO8knVGPkUGAUpZEoFDmnTrfvkf8AZ6UXCZurZ3aQWknHecCKqCtDjZf8WW4MBhcyj2/MeOv5rVk8rZ8NozD5T7i/6oiXlG+CqNpd+YGtFArFDFlR6/4LwyGtLhgVaiAnSpO0SnsSHCZ38ywqJeYrAIeLpTWE5xbSNpe1/A5RLLxxwrKhChFjJmTXCfUYDBOFgLOgNyZvfZlgQ/SZwkWJFUECo2pdzNMz7OdY+GS8HDnduOlphZjnOcW8QTMn70oBd4VwgFfdPBLV2xS1GDCqKsmJJ/uolPnY5RD1OcdgHDdaglfiViKyiqIlljGt4Px4HuKvlisJ7qpgufkLDo9hkSJSGGqDm3oP3cwyLnLBua0HaG7d8IDbpDuHVizj+L4XAZenp+x/2D1kFGz7WaWgpbla5IAkzggwCRIyzbaYovk4Qc0mMYKISwj/6yw0rFSFvb9G1/lil3N04J3Af9dMzSdEgvcBrSMgllIPSWAV6+MsX/dQhOKZN4ZqgCuNEVMGEnzFdAV7s/6J7bKx5viOJH7SBRvw2Iv3Ka3mfaTVX4BsrijJxQzwTAqy32OgpqjlAhf8DK4/210RBZYl7I9kmS9nm2ZoxxEGRf0xUOwfpkOZbweJ0Xhl1CYWzr3V493zLwrgPBhbUcqshvVbUkzpVCyHcmPQwuShvEnAD83zMErgcb5kSREvc33FovA6E70heFUjaZY4phdYRfxAl/yiaItQ+dQmya4nes5sspxMIBQ/WDOgQAAABVBmiFsQ3/+p4QABPeBZifzq1yawegAAAAYQZpCPCGTKYQ7//6plgACc/RzSzo6noPBAAAAPkGaZknhDyZTAh3//qmWAAJQ7+Rgf/+IJ0fV1s+CI3gZzL//xCJgXq7rs8EnoYEJft//4flO1cfmBwWgl8sXAAAADkGehEURPC//AALEyrJhAAAAEAGeo3RCvwADwqG7p2XaEIEAAAAPAZ6lakK/AAPCobsM9WjpAAAAE0GaqkmoQWiZTAh3//6plgAAlYEAAAAMQZ7IRREsL/8AALKAAAAAEAGe53RCvwADwqG7p2XaEIAAAAAPAZ7pakK/AAPCobsM9WjpAAAAE0Ga7kmoQWyZTAh3//6plgAAlYAAAAAMQZ8MRRUsL/8AALKAAAAAEAGfK3RCvwADwqG7p2XaEIEAAAAPAZ8takK/AAPCobsM9WjpAAAAE0GbMkmoQWyZTAh3//6plgAAlYEAAAAMQZ9QRRUsL/8AALKAAAAAEAGfb3RCvwADwqG7p2XaEIAAAAAPAZ9xakK/AAPCobsM9WjpAAAAE0GbdkmoQWyZTAh3//6plgAAlYAAAAAMQZ+URRUsL/8AALKAAAAAEAGfs3RCvwADwqG7p2XaEIEAAAAPAZ+1akK/AAPCobsM9WjpAAAAE0GbukmoQWyZTAh3//6plgAAlYEAAAAMQZ/YRRUsL/8AALKBAAAAEAGf93RCvwADwqG7p2XaEIAAAAAPAZ/5akK/AAPCobsM9WjpAAAAE0Gb/kmoQWyZTAh3//6plgAAlYAAAAAMQZ4cRRUsL/8AALKBAAAAEAGeO3RCvwADwqG7p2XaEIEAAAAPAZ49akK/AAPCobsM9WjpAAAAE0GaIkmoQWyZTAh3//6plgAAlYAAAAAMQZ5ARRUsL/8AALKBAAAAEAGef3RCvwADwqG7p2XaEIAAAAAPAZ5hakK/AAPCobsM9WjpAAAAE0GaZkmoQWyZTAh3//6plgAAlYAAAAAMQZ6ERRUsL/8AALKBAAAAEAGeo3RCvwADwqG7p2XaEIEAAAAPAZ6lakK/AAPCobsM9WjpAAAAE0GaqkmoQWyZTAh3//6plgAAlYEAAAAMQZ7IRRUsL/8AALKAAAAAEAGe53RCvwADwqG7p2XaEIAAAAAPAZ7pakK/AAPCobsM9WjpAAAAE0Ga7kmoQWyZTAh3//6plgAAlYAAAAAMQZ8MRRUsL/8AALKAAAAAEAGfK3RCvwADwqG7p2XaEIEAAAAPAZ8takK/AAPCobsM9WjpAAAAE0GbMkmoQWyZTAh3//6plgAAlYEAAAAMQZ9QRRUsL/8AALKAAAAAEAGfb3RCvwADwqG7p2XaEIAAAAAPAZ9xakK/AAPCobsM9WjpAAAAE0GbdkmoQWyZTAh3//6plgAAlYAAAAAMQZ+URRUsL/8AALKAAAAAEAGfs3RCvwADwqG7p2XaEIEAAAAPAZ+1akK/AAPCobsM9WjpAAAAE0GbukmoQWyZTAh3//6plgAAlYEAAAAMQZ/YRRUsL/8AALKBAAAAEAGf93RCvwADwqG7p2XaEIAAAAAQAZ/5akK/AAX5K2L1dhz+wQAAABNBm/5JqEFsmUwId//+qZYAAJWAAAAADEGeHEUVLC//AACygQAAABABnjt0Qr8AA8Khu6dl2hCBAAAADwGePWpCvwADwqG7DPVo6QAAABNBmiJJqEFsmUwId//+qZYAAJWAAAAADEGeQEUVLC//AACygQAAABABnn90Qr8AA8Khu6dl2hCAAAAADwGeYWpCvwADwqG7DPVo6QAAABNBmmZJqEFsmUwId//+qZYAAJWAAAAADEGehEUVLC//AACygQAAABABnqN0Qr8AA8Khu6dl2hCBAAAADwGepWpCvwADwqG7DPVo6QAAABNBmqpJqEFsmUwId//+qZYAAJWBAAAADEGeyEUVLC//AACygAAAABABnud0Qr8AA8Khu6dl2hCAAAAADwGe6WpCvwADwqG7DPVo6QAAABNBmu5JqEFsmUwId//+qZYAAJWAAAAADEGfDEUVLC//AACygAAAABABnyt0Qr8AA8Khu6dl2hCBAAAADwGfLWpCvwADwqG7DPVo6QAAACRBmzJJqEFsmUwIb//+p4QABxtRVn4hAD/+EqWPP//PfRV8k8kAAAAQQZ9QRRUsL/8ABDc/c4WeCAAAAA8Bn290Qr8AA8xegMkvEIAAAAAQAZ9xakK/AAXSyITcZ9evOQAAABpBm3NJqEFsmUwId//+qZYAA5PtL+d0hTChEAAAABJBm5dJ4QpSZTAh3/6plgAAlYAAAAAMQZ+1RTRML/8AALKBAAAADwGf1HRCvwADzNgaHnPM6QAAAA8Bn9ZqQr8AA8vOGiVzzOkAAAATQZvbSahBaJlMCHf//qmWAACVgQAAAAxBn/lFESwv/wAAsoAAAAAPAZ4YdEK/AAPM2Boec8zpAAAAEAGeGmpCvwAF0ja7rIYdAUAAAAATQZofSahBbJlMCHf//qmWAACVgQAAAAxBnj1FFSwv/wAAsoEAAAAPAZ5cdEK/AAPM2Boec8zpAAAADwGeXmpCvwADy84aJXPM6QAAACdBmkNJqEFsmUwId//+qZYAAnPxPOZZYwqrwKZrip4FEnz7Wl6xwYEAAAAQQZ5hRRUsL/8AAulBs883IAAAAA8BnoB0Qr8AA8zYGuvjdIEAAAAQAZ6CakK/AAPizB5LmfLZgAAAABdBmodJqEFsmUwIb//+p4QABRvRP9WLMQAAACRBnqVFFSwv/wADCEMDviD3P/8QdBks//w0/LP/8QMdv4QjyZEAAAAPAZ7EdEK/AAQXzBg2Y4tHAAAADwGexmpCvwAEFkymbZkdZwAAABpBmshJqEFsmUwId//+qZYAAnP0c0s6Op6DwAAAABJBmuxJ4QpSZTAh3/6plgAAlYAAAAAMQZ8KRTRML/8AALKBAAAAEAGfKXRCvwADwqG7p2XaEIAAAAAPAZ8rakK/AAPCobsM9WjpAAAAE0GbMEmoQWiZTAh3//6plgAAlYEAAAAMQZ9ORREsL/8AALKBAAAAEAGfbXRCvwADwqG7p2XaEIEAAAAPAZ9vakK/AAPCobsM9WjpAAAAE0GbdEmoQWyZTAh3//6plgAAlYAAAAAMQZ+SRRUsL/8AALKBAAAAEAGfsXRCvwADwqG7p2XaEIAAAAAPAZ+zakK/AAPCobsM9WjpAAAAE0GbuEmoQWyZTAh3//6plgAAlYEAAAAMQZ/WRRUsL/8AALKAAAAAEAGf9XRCvwADwqG7p2XaEIEAAAAPAZ/3akK/AAPCobsM9WjpAAAAE0Gb/EmoQWyZTAh3//6plgAAlYAAAAAMQZ4aRRUsL/8AALKBAAAAEAGeOXRCvwADwqG7p2XaEIAAAAAPAZ47akK/AAPCobsM9WjpAAAAE0GaIEmoQWyZTAh3//6plgAAlYEAAAAMQZ5eRRUsL/8AALKAAAAAEAGefXRCvwADwqG7p2XaEIAAAAAPAZ5/akK/AAPCobsM9WjpAAAAE0GaZEmoQWyZTAh3//6plgAAlYAAAAAMQZ6CRRUsL/8AALKBAAAAEAGeoXRCvwADwqG7p2XaEIAAAAAPAZ6jakK/AAPCobsM9WjpAAAAE0GaqEmoQWyZTAh3//6plgAAlYEAAAAMQZ7GRRUsL/8AALKBAAAAEAGe5XRCvwADwqG7p2XaEIEAAAAPAZ7nakK/AAPCobsM9WjpAAAAE0Ga7EmoQWyZTAh3//6plgAAlYAAAAAMQZ8KRRUsL/8AALKBAAAAEAGfKXRCvwADwqG7p2XaEIAAAAAPAZ8rakK/AAPCobsM9WjpAAAAE0GbMEmoQWyZTAh3//6plgAAlYEAAAAMQZ9ORRUsL/8AALKBAAAAEAGfbXRCvwADwqG7p2XaEIEAAAAPAZ9vakK/AAPCobsM9WjpAAAAE0GbdEmoQWyZTAh3//6plgAAlYAAAAAMQZ+SRRUsL/8AALKBAAAAEAGfsXRCvwADwqG7p2XaEIAAAAAPAZ+zakK/AAPCobsM9WjpAAAAE0GbuEmoQWyZTAh3//6plgAAlYEAAAAMQZ/WRRUsL/8AALKAAAAAEAGf9XRCvwADwqG7p2XaEIEAAAAPAZ/3akK/AAPCobsM9WjpAAAAE0Gb/EmoQWyZTAh3//6plgAAlYAAAAAMQZ4aRRUsL/8AALKBAAAAEAGeOXRCvwADwqG7p2XaEIAAAAAPAZ47akK/AAPCobsM9WjpAAAAE0GaIEmoQWyZTAh3//6plgAAlYEAAAAMQZ5eRRUsL/8AALKAAAAAEAGefXRCvwADwqG7p2XaEIAAAAAPAZ5/akK/AAPCobsM9WjpAAAAE0GaZEmoQWyZTAh3//6plgAAlYAAAAAMQZ6CRRUsL/8AALKBAAAAEAGeoXRCvwADwqG7p2XaEIAAAAAPAZ6jakK/AAPCobsM9WjpAAAAE0GaqEmoQWyZTAh3//6plgAAlYEAAAAMQZ7GRRUsL/8AALKBAAAAEAGe5XRCvwADwqG7p2XaEIEAAAAPAZ7nakK/AAPCobsM9WjpAAAAE0Ga7EmoQWyZTAh3//6plgAAlYAAAAAMQZ8KRRUsL/8AALKBAAAAEAGfKXRCvwADwqG7p2XaEIAAAAAPAZ8rakK/AAPCobsM9WjpAAAAE0GbMEmoQWyZTAh3//6plgAAlYEAAAAMQZ9ORRUsL/8AALKBAAAAEAGfbXRCvwADwqG7p2XaEIEAAAAPAZ9vakK/AAPCobsM9WjpAAAAE0GbdEmoQWyZTAh3//6plgAAlYAAAAAMQZ+SRRUsL/8AALKBAAAAEAGfsXRCvwADwqG7p2XaEIAAAAAPAZ+zakK/AAPCobsM9WjpAAAAE0GbuEmoQWyZTAh3//6plgAAlYEAAAAMQZ/WRRUsL/8AALKAAAAAEAGf9XRCvwADwqG7p2XaEIEAAAAPAZ/3akK/AAPCobsM9WjpAAAAE0Gb/EmoQWyZTAh3//6plgAAlYAAAAAMQZ4aRRUsL/8AALKBAAAAEAGeOXRCvwAF+sq7q/HePkAAAAAPAZ47akK/AAPCobsM9WjpAAAAEkGaIEmoQWyZTAhv//6nhAABJwAAAAxBnl5FFSwv/wAAsoAAAAAQAZ59dEK/AAPCobunZdoQgAAAAA8Bnn9qQr8AA8Khuwz1aOkAAAASQZpkSahBbJlMCG///qeEAAEnAAAADEGegkUVLC//AACygQAAABABnqF0Qr8AA8Khu6dl2hCAAAAADwGeo2pCvwADwqG7DPVo6QAAABJBmqhJqEFsmUwIX//+jLAABI0AAAAUQZ7GRRUsL/8ABDY+fRYrsItfgN0AAAAQAZ7ldEK/AAXTNEifFmKecQAAABABnudqQr8ABdG5DD6AkHvYAAAAGkGa6UuoQhBbJEYIKAfyAf2HgCFf/jhAABFwAAAMeW1vb3YAAABsbXZoZAAAAAAAAAAAAAAAAAAAA+gAAB+QAAEAAAEAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAujdHJhawAAAFx0a2hkAAAAAwAAAAAAAAAAAAAAAQAAAAAAAB+QAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAEQAAABEAAAAAAAJGVkdHMAAAAcZWxzdAAAAAAAAAABAAAfkAAABAAAAQAAAAALG21kaWEAAAAgbWRoZAAAAAAAAAAAAAAAAAAAMgAAAZQAVcQAAAAAAC1oZGxyAAAAAAAAAAB2aWRlAAAAAAAAAAAAAAAAVmlkZW9IYW5kbGVyAAAACsZtaW5mAAAAFHZtaGQAAAABAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEAAAAMdXJsIAAAAAEAAAqGc3RibAAAAJZzdHNkAAAAAAAAAAEAAACGYXZjMQAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAEQARAASAAAAEgAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABj//wAAADBhdmNDAfQADf/hABdn9AANkZsoIhHQgAAAAwCAAAAZB4oUywEABmjr48RIRAAAABhzdHRzAAAAAAAAAAEAAADKAAACAAAAABRzdHNzAAAAAAAAAAEAAAABAAAGUGN0dHMAAAAAAAAAyAAAAAMAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAAcc3RzYwAAAAAAAAABAAAAAQAAAMoAAAABAAADPHN0c3oAAAAAAAAAAAAAAMoAAAV6AAAAGQAAABwAAABCAAAAEgAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAKAAAABQAAAATAAAAFAAAAB4AAAAWAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAFAAAABcAAAAQAAAAEwAAABMAAAArAAAAFAAAABMAAAAUAAAAGwAAACgAAAATAAAAEwAAAB4AAAAWAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAWAAAAEAAAABQAAAATAAAAFgAAABAAAAAUAAAAEwAAABYAAAAYAAAAFAAAABQAAAAeAAAAFHN0Y28AAAAAAAAAAQAAADAAAABidWR0YQAAAFptZXRhAAAAAAAAACFoZGxyAAAAAAAAAABtZGlyYXBwbAAAAAAAAAAAAAAAAC1pbHN0AAAAJal0b28AAAAdZGF0YQAAAAEAAAAATGF2ZjU2LjM2LjEwMA==\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "K.clear_session()\n",
    "epochs_train = 10\n",
    "env = Environment(grid_size=size, max_time=T, temperature=0.3)\n",
    "agent = DQN_CNN(size, lr=.001, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "train(agent,env,epochs_train,prefix='cnn_train')\n",
    "HTML(display_videos('cnn_train10.mp4'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "__Question 9__ Test both algorithms and compare their performances. Which issue(s) do you observe? Observe also different behaviors by changing the temperature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observe the behaviour of the two agents (same temperature = 0.3)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Test of the CNN \n",
      "\n",
      "Win/lose count 1.5/4.0. Average score (-2.5)\n",
      "Win/lose count 2.5/7.0. Average score (-3.5)\n",
      "Win/lose count 3.0/1.0. Average score (-1.6666666666666667)\n",
      "Win/lose count 2.5/4.0. Average score (-1.625)\n",
      "Win/lose count 1.5/4.0. Average score (-1.8)\n",
      "Win/lose count 3.5/0. Average score (-0.9166666666666666)\n",
      "Win/lose count 2.0/10.0. Average score (-1.9285714285714286)\n",
      "Win/lose count 9.0/4.0. Average score (-1.0625)\n",
      "Win/lose count 0.5/2.0. Average score (-1.1111111111111112)\n",
      "Win/lose count 1.5/3.0. Average score (-1.15)\n",
      "Win/lose count 2.0/6.0. Average score (-1.4090909090909092)\n",
      "Final score: -1.4090909090909092\n",
      "\n",
      " Test of the FC \n",
      "\n",
      "Win/lose count 2.0/3.0. Average score (-1.0)\n",
      "Win/lose count 3.5/4.0. Average score (-0.75)\n",
      "Win/lose count 4.0/2.0. Average score (0.16666666666666666)\n",
      "Win/lose count 3.5/7.0. Average score (-0.75)\n",
      "Win/lose count 4.5/10.0. Average score (-1.7)\n",
      "Win/lose count 2.0/4.0. Average score (-1.75)\n",
      "Win/lose count 0.5/1.0. Average score (-1.5714285714285714)\n",
      "Win/lose count 0.5/2.0. Average score (-1.5625)\n",
      "Win/lose count 1.5/7.0. Average score (-2.0)\n",
      "Win/lose count 2.5/1.0. Average score (-1.65)\n",
      "Win/lose count 3.5/1.0. Average score (-1.2727272727272727)\n",
      "Final score: -1.2727272727272727\n"
     ]
    }
   ],
   "source": [
    "epochs_test = 11\n",
    "\n",
    "\n",
    "env = Environment(grid_size=size, max_time=T,temperature=0.3)\n",
    "agent_cnn = DQN_CNN(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "agent_cnn.load(name_weights='cnn_trainmodel.h5',name_model='cnn_trainmodel.json')\n",
    "\n",
    "agent_fc = DQN_FC(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "agent_cnn.load(name_weights='fc_trainmodel.h5',name_model='fc_trainmodel.json')\n",
    "\n",
    "print('\\n Test of the CNN \\n')\n",
    "test(agent_cnn,env,epochs_test,prefix='cnn_test')\n",
    "print('\\n Test of the FC \\n')\n",
    "test(agent_fc,env,epochs_test,prefix='fc_test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The agents get mediocre rewards. Let's look at the agents' behaviours with the videos.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFk5tZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAANFZYiEADv//vb8/AptUwndP/9P+VxN2UIE1zAALTEAIgTYlI6+z4vwKZrZf8CiWgOtBO4OjGRz32riKnQs1pGGaMM3jyjI0LNOYeKjDOV5P2qj737oL79Qh386ML2MGcNPoCigX91pfGOcT8fvFb6+S2a+OPzsrTV6e+HlCQyVUy0+CYCnEM50PiwX3n4/gI4vw8/j00vxeTG/6fAirHIv/goTKr/mrWJb6eyZNy2fy6Z+XWFyvKLqR32bP2XxZ3/OS78uhEOAxcDtg+AgHgWZ+cbBUlRfLvPsvxCA1N1rddBgvs1DCRdd9sDJVcJFjSQpL72M/85lTgRIhpp3QnFhVH12CL7GaRX1EAMOUcaay3PdleHyh/2ImFLhdUSySpo37ZeQQAJUmg9fyR9OWqfd4sUL/narzps7PB1GtoN7bWUwc4X9rQ+sRbbpOTxTNd7w7ZGTRCTLNFBMGMPl8LiDcTnRZicvSMOSwBpHwPOGtFWXy6X00DybPw9hk68pOI2/NiQJ9HIw4hdI6jyPS7HO8qxbpx8S/3o1pORJa0wkEbkUIbGyn7Ys0qBTsjSCWGd3gi8kCsjUSGVWaj/zGlAr1FMLiWMV4gDpSfMgaeMS+ODI0hNmDppf1sLEOlU5EdxxGHbJKzIOoEoyABdQ7dLJHmpmcCBDwMc+8klyTHeggMVNE8nAWjWy9E1ZXioRX2cIFUj54T/zK6+19ewiV5ysuQ/bQvAjLpRPnDRYLP+TryDIuSdpqPXA7HqKbAvzUYCFgv0vLoJAiGQ9ZfqWRQjDJrcojZUviqQN9jiTzFEZ24OcjLVsuBBibDCrICYKg5VC636iywo4ax54/CBiJq5G9MHFw0OfKVL0ZQTuIc9PeBteaFkAzTUK73Xd/Mp7xFot2LSuSjebJWobbStc8151xqTtVRudA5Wf6p25jLNp5AiMwOvJgpxvC+s6cHfH4FOd/4Cnw5IoxgRMY4ZFKiHDc7H03HPImeeFo2S9F7eJTbMriv483C4jYVONE9AbhyLFOyGZJ0fQaWFOsSGd56LFP2wqenVLdO804VVGGoH2VG0rNqwTSFgJrqbYiWJa2mfT+SRgXBrdei/nFiki/ykZHBAZiiPhAAAAJEGaJGxDf/6nhAj27fzLK8YZ+BTLZ2fAoUjMjW57TtgTX8tPmAAAAA5BnkJ4hf8Bw+Aot5ElYQAAAA8BnmF0Qr8BdegHQnJduqAAAAAQAZ5jakK/Al7O8Cv7RPlswQAAABlBmmVJqEFomUwId//+qZYEe5zWkhTfpOuBAAAAHkGaiUnhClJlMCHf/qmWBRtUC0SbH19e090v28J/gQAAABBBnqdFNEwv/wHV+t67RDghAAAADwGexnRCvwJ2iE2DZLtSFgAAABABnshqQr8CapANyPX795WAAAAAE0GazUmoQWiZTAh3//6plgAAlYEAAAAMQZ7rRREsL/8AALKAAAAAEAGfCnRCvwJqkA3SAPt3WUAAAAAQAZ8MakK/AmqQDcj1+/eVgQAAABNBmxFJqEFsmUwId//+qZYAAJWBAAAADEGfL0UVLC//AACygQAAABABn050Qr8CapAN0gD7d1lAAAAAEAGfUGpCvwJqkA3I9fv3lYAAAAATQZtVSahBbJlMCHf//qmWAACVgQAAAAxBn3NFFSwv/wAAsoAAAAAQAZ+SdEK/AmqQDdIA+3dZQAAAABABn5RqQr8CapANyPX795WBAAAAE0GbmUmoQWyZTAh3//6plgAAlYAAAAAMQZ+3RRUsL/8AALKBAAAAEAGf1nRCvwJqkA3SAPt3WUEAAAAQAZ/YakK/AmqQDcj1+/eVgAAAABNBm91JqEFsmUwId//+qZYAAJWBAAAADEGf+0UVLC//AACygAAAABABnhp0Qr8CapAN0gD7d1lBAAAAEAGeHGpCvwJqkA3I9fv3lYEAAAATQZoBSahBbJlMCHf//qmWAACVgAAAAAxBnj9FFSwv/wAAsoAAAAAQAZ5edEK/AmqQDdIA+3dZQQAAABABnkBqQr8CapANyPX795WAAAAAK0GaRUmoQWyZTAh3//6plgcb4OfxCQkH/8JWNAn//7tIb/hSu/ugF7Z2VsEAAAAVQZ5jRRUsL/8CAeEbLNqcfTOTpspIAAAAEAGegnRCvwKQI7Fbjv45LaEAAAAPAZ6EakK/Aq9nR+GzaPH3AAAAE0GaiUmoQWyZTAh3//6plgAAlYEAAAAMQZ6nRRUsL/8AALKBAAAAEAGexnRCvwKwQB0IcOQbI2AAAAAQAZ7IakK/Aq7Wu7fhyDZGwAAAABNBms1JqEFsmUwId//+qZYAAJWBAAAADEGe60UVLC//AACygAAAABABnwp0Qr8CsEAdCHDkGyNgAAAAEAGfDGpCvwKu1ru34cg2RsEAAAATQZsRSahBbJlMCHf//qmWAACVgQAAAAxBny9FFSwv/wAAsoEAAAAQAZ9OdEK/ArBAHQhw5BsjYAAAABABn1BqQr8Crta7t+HINkbAAAAAE0GbVUmoQWyZTAh3//6plgAAlYEAAAAMQZ9zRRUsL/8AALKAAAAAEAGfknRCvwKwQB0IcOQbI2AAAAAQAZ+UakK/Aq7Wu7fhyDZGwQAAABNBm5lJqEFsmUwId//+qZYAAJWAAAAADEGft0UVLC//AACygQAAABABn9Z0Qr8CsEAdCHDkGyNhAAAAEAGf2GpCvwKu1ru34cg2RsAAAAATQZvdSahBbJlMCHf//qmWAACVgQAAAAxBn/tFFSwv/wAAsoAAAAAQAZ4adEK/ArBAHQhw5BsjYQAAABABnhxqQr8Crta7t+HINkbBAAAAE0GaAUmoQWyZTAh3//6plgAAlYAAAAAMQZ4/RRUsL/8AALKAAAAAEAGeXnRCvwKwQB0IcOQbI2EAAAAQAZ5AakK/Aq7Wu7fhyDZGwAAAABNBmkVJqEFsmUwId//+qZYAAJWBAAAADEGeY0UVLC//AACygAAAABABnoJ0Qr8CsEAdCHDkGyNhAAAAEAGehGpCvwKu1ru34cg2RsEAAAATQZqJSahBbJlMCHf//qmWAACVgQAAAAxBnqdFFSwv/wAAsoEAAAAQAZ7GdEK/ArBAHQhw5BsjYAAAABABnshqQr8Crta7t+HINkbAAAAAE0GazUmoQWyZTAh3//6plgAAlYEAAAAMQZ7rRRUsL/8AALKAAAAAEAGfCnRCvwKwQB0IcOQbI2AAAAAQAZ8MakK/Aq7Wu7fhyDZGwQAAABNBmxFJqEFsmUwId//+qZYAAJWBAAAADEGfL0UVLC//AACygQAAABABn050Qr8CsEAdCHDkGyNgAAAAEAGfUGpCvwKu1ru34cg2RsAAAAATQZtVSahBbJlMCHf//qmWAACVgQAAAAxBn3NFFSwv/wAAsoAAAAAQAZ+SdEK/ArBAHQhw5BsjYAAAABABn5RqQr8Crta7t+HINkbBAAAAHEGbmUmoQWyZTAh3//6plgbXIMz3UH4UHEWiZswAAAAQQZ+3RRUsL/8CAd8dznti4QAAAA8Bn9Z0Qr8CrtF7VZ3wVMEAAAAPAZ/YakK/Aq5WXIaQ70NaAAAAHEGb3UmoQWyZTAh3//6plgUbVCyEmzn7S/pFFbEAAAAQQZ/7RRUsL/8B1fvX5DRTQAAAABABnhp0Qr8CkkAc7YxNqkrBAAAADwGeHGpCvwJ20DjA/LBtwQAAABNBmgFJqEFsmUwId//+qZYAAJWAAAAADEGeP0UVLC//AACygAAAABABnl50Qr8CapAN0gD7d1lBAAAAEAGeQGpCvwJqkA3I9fv3lYAAAAATQZpFSahBbJlMCHf//qmWAACVgQAAABRBnmNFFSwv/wHNRF6LN4+iykqvmAAAAA8BnoJ0Qr8CdQEq0aUP8cEAAAAPAZ6EakK/AnY+9HDZtKkLAAAAE0GaiUmoQWyZTAh3//6plgAAlYEAAAAMQZ6nRRUsL/8AALKBAAAAEAGexnRCvwJqkA3SAPt3WUAAAAAQAZ7IakK/AmqQDcj1+/eVgAAAABxBms1JqEFsmUwIb//+p4QKTsx+J04WzFCNozAhAAAAEEGe60UVLC//AdX7v1Ftk4AAAAAQAZ8KdEK/AnT+AySuEZzugAAAAA8BnwxqQr8CXmodCabQcEEAAAAaQZsOSahBbJlMCHf//qmWARfx50s6Oo7gbMEAAAAZQZsySeEKUmUwId/+qZYEF5Jfb7RuaYtP8QAAABBBn1BFNEwv/wGyGp7wdHtwAAAAEAGfb3RCvwJJYrF6NA43nYAAAAAQAZ9xakK/AWiyjvZ4+3S0gQAAABNBm3ZJqEFomUwId//+qZYAAJWAAAAADEGflEURLC//AACygAAAABABn7N0Qr8CSWKxejQON52BAAAAEAGftWpCvwJIahz+rw43nYAAAAATQZu6SahBbJlMCHf//qmWAACVgQAAAAxBn9hFFSwv/wAAsoEAAAAQAZ/3dEK/AklisXo0DjedgAAAABABn/lqQr8CSGoc/q8ON52BAAAAE0Gb/kmoQWyZTAh3//6plgAAlYAAAAAMQZ4cRRUsL/8AALKBAAAAEAGeO3RCvwJJYrF6NA43nYEAAAAQAZ49akK/AkhqHP6vDjedgAAAABNBmiJJqEFsmUwId//+qZYAAJWAAAAADEGeQEUVLC//AACygQAAABABnn90Qr8CSWKxejQON52AAAAAEAGeYWpCvwJIahz+rw43nYEAAAATQZpmSahBbJlMCHf//qmWAACVgAAAAAxBnoRFFSwv/wAAsoEAAAAQAZ6jdEK/AklisXo0DjedgQAAABABnqVqQr8CSGoc/q8ON52BAAAAE0GaqkmoQWyZTAh3//6plgAAlYEAAAAMQZ7IRRUsL/8AALKAAAAAEAGe53RCvwJJYrF6NA43nYAAAAAQAZ7pakK/AkhqHP6vDjedgQAAABNBmu5JqEFsmUwId//+qZYAAJWAAAAADEGfDEUVLC//AACygAAAABABnyt0Qr8CSWKxejQON52BAAAAEAGfLWpCvwJIahz+rw43nYEAAAATQZsySahBbJlMCHf//qmWAACVgQAAAAxBn1BFFSwv/wAAsoAAAAAQAZ9vdEK/AklisXo0DjedgAAAABABn3FqQr8CSGoc/q8ON52BAAAAE0GbdkmoQWyZTAh3//6plgAAlYAAAAAMQZ+URRUsL/8AALKAAAAAEAGfs3RCvwJJYrF6NA43nYEAAAAQAZ+1akK/AkhqHP6vDjedgAAAABxBm7pJqEFsmUwId//+qZYEeCzFpme+e7S+uMxZAAAAEEGf2EUVLC//AcMjJbgIl4EAAAAPAZ/3dEK/Al8YTYNku1ImAAAADwGf+WpCvwJfYDjA/LBwQQAAABNBm/5JqEFsmUwId//+qZYAAJWAAAAADEGeHEUVLC//AACygQAAABABnjt0Qr8CVLAN0gD7d1tBAAAAEAGePWpCvwJUsA3I9fv3mYAAAAATQZoiSahBbJlMCHf//qmWAACVgAAAAAxBnkBFFSwv/wAAsoEAAAAQAZ5/dEK/AlSwDdIA+3dbQAAAABABnmFqQr8CVLANyPX795mBAAAAE0GaZkmoQWyZTAh3//6plgAAlYAAAAAMQZ6ERRUsL/8AALKBAAAAEAGeo3RCvwJUsA3SAPt3W0EAAAAQAZ6lakK/AlSwDcj1+/eZgQAAABNBmqpJqEFsmUwId//+qZYAAJWBAAAADEGeyEUVLC//AACygAAAABABnud0Qr8CVLAN0gD7d1tAAAAAEAGe6WpCvwJUsA3I9fv3mYEAAAATQZruSahBbJlMCHf//qmWAACVgAAAAAxBnwxFFSwv/wAAsoAAAAAQAZ8rdEK/AlSwDdIA+3dbQQAAABABny1qQr8CVLANyPX795mBAAAAHEGbMkmoQWyZTAh3//6plgSfhR9KfbULIUt8igkAAAAQQZ9QRRUsL/8Bw6uS/N0ccAAAAA8Bn290Qr8CXl+AVZ33W0AAAAAPAZ9xakK/Al4PxqNId/HNAAAAE0GbdkmoQWyZTAh3//6plgAAlYAAAAAMQZ+URRUsL/8AALKAAAAAEAGfs3RCvwJJYrF6NA43nYEAAAAQAZ+1akK/AkhqHP6vDjedgAAAAB5Bm7pJqEFsmUwId//+qZYDm55HMss+fbdWmy9kAocAAAAQQZ/YRRUsL/8Bo5+i+wL3hQAAAA8Bn/d0Qr8CSWKxgv7Qc0AAAAAQAZ/5akK/AjLtqG4z688LuQAAABNBm/5JqEFsmUwId//+qZYAAJWAAAAADEGeHEUVLC//AACygQAAABABnjt0Qr8CKaieQRWctWBBAAAAEAGePWpCvwIpqJ45wPt9YEAAAAASQZoiSahBbJlMCG///qeEAAEnAAAADEGeQEUVLC//AACygQAAABABnn90Qr8CKaieQRWctWBAAAAAEAGeYWpCvwIpqJ45wPt9YEEAAAASQZpmSahBbJlMCGf//p4QAAR8AAAADEGehEUVLC//AACygQAAABABnqN0Qr8CKaieQRWctWBBAAAAEAGepWpCvwIpqJ45wPt9YEEAAAAaQZqpS6hCEFskRggoB/IB/YeAIV/+OEAAEXEAAAAnQZ7IQhX/Aq9j7UHEHXOdBTEx1dzMeemDZm07SRGvvI8wG9V4d2vgAAAAEAGe52kQrwJIrg1th8DIqGAAAAyJbW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAAH5AAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAC7N0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAAH5AAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAARAAAAEQAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAB+QAAAEAAABAAAAAAsrbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAAAAyAAABlABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAAK1m1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAACpZzdGJsAAAAlnN0c2QAAAAAAAAAAQAAAIZhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAARABEABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAMGF2Y0MB9AAN/+EAF2f0AA2RmygiEdCAAAADAIAAABkHihTLAQAGaOvjxEhEAAAAGHN0dHMAAAAAAAAAAQAAAMoAAAIAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAZgY3R0cwAAAAAAAADKAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAEAAAQAAAAAAQAAAAAAAAAcc3RzYwAAAAAAAAABAAAAAQAAAMoAAAABAAADPHN0c3oAAAAAAAAAAAAAAMoAAAX6AAAAKAAAABIAAAATAAAAFAAAAB0AAAAiAAAAFAAAABMAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAALwAAABkAAAAUAAAAEwAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACAAAAAUAAAAEwAAABMAAAAgAAAAFAAAABQAAAATAAAAFwAAABAAAAAUAAAAFAAAABcAAAAYAAAAEwAAABMAAAAXAAAAEAAAABQAAAAUAAAAIAAAABQAAAAUAAAAEwAAAB4AAAAdAAAAFAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAIAAAABQAAAATAAAAEwAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAIAAAABQAAAATAAAAEwAAABcAAAAQAAAAFAAAABQAAAAiAAAAFAAAABMAAAAUAAAAFwAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAHgAAACsAAAAUAAAAFHN0Y28AAAAAAAAAAQAAADAAAABidWR0YQAAAFptZXRhAAAAAAAAACFoZGxyAAAAAAAAAABtZGlyYXBwbAAAAAAAAAAAAAAAAC1pbHN0AAAAJal0b28AAAAdZGF0YQAAAAEAAAAATGF2ZjU2LjM2LjEwMA==\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(display_videos('cnn_test10.mp4'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFYhtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAALHZYiEADv//vb8/AptUwndP/9P+VxN2UIE1zAALTEAIgTYlI6+z4vwKZrZf8CiWgOunhgGOLEl07YT/ER+EMCMy6XvNeYnszJI+GqdHWy27kbFJE31qmjkCZTL/75//CyALP64U5LyIEv1nhKuJLVRoSoFM0VlYM4HnEu2wMR0na18qoHcPCQnVmghdLtW/13t8L9hjQwE+KjpF/V0pjzRTrC0rfMGVsDKcb+xznkX05jQhLUuz7npcmYAcnARQ0mk6FLj8gJmDO8UF1NGVBalI/HqSZROaFJirVz2FrzYR3Cl03CpbSJ+OQp2Q8PByEJjcLMkxd8lV4B+rf9nBftQsJ9QxgIIwPmxg8eEUQO9uUNB+4hdBBJaEZJbdtR9sLLcDBMqAHn84M2uQh55myICmaV8XGQ7QDxwpFj6SI7VJlPAlw5G4g0iLGD3bg2C+FnicILEWsoAMnib5JGn9ERLLNJbIJu1tjp9F04yNDo7uCwJg5fJNFrtW/e+AcTt0n7a2nsauZSjh22TXZNVp32JxbD/TOuUKpNSqr3piZgIewV4+QpY808LwQ2G0OeNxgJ78s7525WozVCKCK3IWhzIz2UTF8Op6iUi9PRhf+FRXbCCo8t/pW/j6fknNYqJRP9t+biZgfyykluSC81JmMnSeXxxOc8q5RvjYVko9O7sYBZaENtG8GSGb2ZJ6Pvmbi8tfpP3jHObbQowu/Gy2UBsTCaDBnP1hB45M7Tpn0RUnMoEbGAsyrYROGO9OtyL2oRLbBGCSgYjvY7KiC6o6JIijt1tyMnf+ievGaxrN9T/837oFvHMkeExyOh++v4HfGh7uSqFA/xPfOBKMwZ0WHuyR44aLK7IckgqGNGRRaMxOh2q6AUj7AIF2tonm55HVTcPIJTfvLRxncUwhevRsS55AiEHXieCZ0qNNBOpUX9dmbFvwAFVQARNAAAADUGaJGxDv/6plgAAlYAAAAAKQZ5CeIX/AACygQAAAA8BnmF0Qr8AEioeS1AfmGYAAAAPAZ5jakK/ABIqHkpQH5hnAAAAHEGaaEmoQWiZTAh3//6plgALfzzIjHAcC0U5mjMAAAAQQZ6GRREsL/8ADYKvG9gquQAAAA8BnqV0Qr8AEltGLgPzDMEAAAAQAZ6nakK/ABJZZDD6AkHRmAAAABlBmqxJqEFsmUwId//+qZYAC3BOkf31fdxeAAAAEEGeykUVLC//AA2AjjO5b+EAAAAQAZ7pdEK/ABJfUSJ8WYpKUAAAAA8BnutqQr8AEmDWBdf4QUAAAAATQZrwSahBbJlMCHf//qmWAACVgQAAAAxBnw5FFSwv/wAAsoEAAAAPAZ8tdEK/ABIqHktQH5hnAAAADwGfL2pCvwASKh5KUB+YZgAAABNBmzRJqEFsmUwId//+qZYAAJWAAAAADEGfUkUVLC//AACygQAAAA8Bn3F0Qr8AEioeS1AfmGYAAAAPAZ9zakK/ABIqHkpQH5hmAAAAEkGbeEmoQWyZTAhv//6nhAABJwAAAAxBn5ZFFSwv/wAAsoAAAAAPAZ+1dEK/ABIqHktQH5hnAAAADwGft2pCvwASKh5KUB+YZwAAABpBm7lJqEFsmUwId//+qZYAEYRYboxCOfYa4AAAACxBm91J4QpSZTAh3/6plgAoWRIv4hBt/+EqYlMX/+EndXxf/4wnu/vko6c4QQAAABVBn/tFNEwv/wAvyrxje5XIi954FbAAAAAQAZ4adEK/ACs5okT4sxR68QAAABABnhxqQr8AP4z5jdDkg4+lAAAAF0GaAUmoQWiZTAh3//6plgAnPx5/JH0gAAAADkGeP0URLC//AC6MqDAgAAAAEAGeXnRCvwA+6wDEdl2VtoEAAAAPAZ5AakK/AD7rALrPVns3AAAAE0GaRUmoQWyZTAh3//6plgAAlYEAAAAMQZ5jRRUsL/8AALKAAAAAEAGegnRCvwA+6wDEdl2VtoEAAAAPAZ6EakK/AD7rALrPVns3AAAAE0GaiUmoQWyZTAh3//6plgAAlYEAAAAMQZ6nRRUsL/8AALKBAAAAEAGexnRCvwA+6wDEdl2VtoAAAAAPAZ7IakK/AD7rALrPVns3AAAAR0GazUmoQWyZTAh3//6plgA+Z+ff/8QkeKmrvNDsEqv1xEeH///iBcs1dMtggt+FcX//8QgZU6u5I3BJK6hAZqfse+OqjbC/AAAAEEGe60UVLC//AEtzzsxtvyoAAAAPAZ8KdEK/AD+F6AyS5baAAAAAEAGfDGpCvwBnHVPJgevbyoEAAAAgQZsRSahBbJlMCHf//qmWAGUgsxaZoA9SQOH7k3+TPakAAAAQQZ8vRRUsL/8Adr9lUV0pIQAAAA8Bn050Qr8Ao+ZO4NkvGeEAAAAPAZ9QakK/AKPysC6/v5BAAAAAEkGbVUmoQWyZTAhv//6nhAABJwAAAAxBn3NFFSwv/wAAsoAAAAAQAZ+SdEK/AKHZR34APt1jwAAAABABn5RqQr8AodlHezx9useBAAAAGkGblkmoQWyZTAh3//6plgCcIsN0Umd92lnwAAAAFkGbuknhClJlMCHf/qmWATfyS/BUZUEAAAAOQZ/YRTRML/8BHqACraEAAAAQAZ/3dEK/AYmyrur8d37BwAAAAA8Bn/lqQr8BibcB0K2fnjUAAAATQZv+SahBaJlMCHf//qmWAACVgAAAAAxBnhxFESwv/wAAsoEAAAAPAZ47dEK/AYlQgdCVX9g5AAAADwGePWpCvwGJtwHQrZ+eNAAAABNBmiJJqEFsmUwId//+qZYAAJWAAAAADEGeQEUVLC//AACygQAAAA8Bnn90Qr8BiVCB0JVf2DgAAAAPAZ5hakK/AYm3AdCtn541AAAAE0GaZkmoQWyZTAh3//6plgAAlYAAAAAMQZ6ERRUsL/8AALKBAAAADwGeo3RCvwGJUIHQlV/YOQAAAA8BnqVqQr8BibcB0K2fnjUAAAATQZqqSahBbJlMCHf//qmWAACVgQAAAAxBnshFFSwv/wAAsoAAAAAPAZ7ndEK/AYlQgdCVX9g4AAAADwGe6WpCvwGJtwHQrZ+eNQAAABNBmu5JqEFsmUwId//+qZYAAJWAAAAADEGfDEUVLC//AACygAAAAA8Bnyt0Qr8BiVCB0JVf2DkAAAAPAZ8takK/AYm3AdCtn541AAAAE0GbMkmoQWyZTAh3//6plgAAlYEAAAAMQZ9QRRUsL/8AALKAAAAADwGfb3RCvwGJUIHQlV/YOAAAAA8Bn3FqQr8BibcB0K2fnjUAAAATQZt2SahBbJlMCHf//qmWAACVgAAAAAxBn5RFFSwv/wAAsoAAAAAPAZ+zdEK/AYlQgdCVX9g5AAAADwGftWpCvwGJtwHQrZ+eNAAAABNBm7pJqEFsmUwId//+qZYAAJWBAAAADEGf2EUVLC//AACygQAAAA8Bn/d0Qr8BiVCB0JVf2DgAAAAPAZ/5akK/AYm3AdCtn541AAAAE0Gb/kmoQWyZTAh3//6plgAAlYAAAAAMQZ4cRRUsL/8AALKBAAAADwGeO3RCvwGJUIHQlV/YOQAAAA8Bnj1qQr8BibcB0K2fnjQAAAATQZoiSahBbJlMCHf//qmWAACVgAAAAAxBnkBFFSwv/wAAsoEAAAAPAZ5/dEK/AYlQgdCVX9g4AAAADwGeYWpCvwGJtwHQrZ+eNQAAABNBmmZJqEFsmUwId//+qZYAAJWAAAAADEGehEUVLC//AACygQAAAA8BnqN0Qr8BiVCB0JVf2DkAAAAPAZ6lakK/AYm3AdCtn541AAAAE0GaqkmoQWyZTAh3//6plgAAlYEAAAAMQZ7IRRUsL/8AALKAAAAADwGe53RCvwGJUIHQlV/YOAAAAA8BnulqQr8BibcB0K2fnjUAAAAXQZruSahBbJlMCHf//qmWBeEw3Q/kIuAAAAAjQZ8MRRUsL/8B4USYeEpuf/4hAQGWf/4gY7Vn/5/rbrXmEbAAAAAQAZ8rdEK/ApHRogPxwhpZQQAAAA8Bny1qQr8CkFaGB01I5Z8AAAATQZsySahBbJlMCHf//qmWAACVgQAAAAxBn1BFFSwv/wAAsoAAAAAQAZ9vdEK/AYmyrur8d37BwAAAABABn3FqQr8BiUrYvV2HI2/BAAAAEkGbdkmoQWyZTAhv//6nhAABJwAAAAxBn5RFFSwv/wAAsoAAAAAQAZ+zdEK/AYmyrur8d37BwQAAABABn7VqQr8BiUrYvV2HI2/AAAAAGkGbt0moQWyZTAh3//6plgEz8gzPyeY+5TehAAAAEkGb20nhClJlMCHf/qmWAACVgQAAAAxBn/lFNEwv/wAAsoAAAAAQAZ4YdEK/ApJAHP10DiyRgQAAABABnhpqQr8BhM5O9nj7dKaAAAAAE0GaH0moQWiZTAh3//6plgAAlYEAAAAMQZ49RREsL/8AALKBAAAAEAGeXHRCvwGEzk78AH26U0AAAAAQAZ5eakK/AYTOTvZ4+3SmgAAAABNBmkNJqEFsmUwId//+qZYAAJWBAAAADEGeYUUVLC//AACygAAAABABnoB0Qr8BhM5O/AB9ulNBAAAAEAGegmpCvwGEzk72ePt0poAAAAATQZqHSahBbJlMCHf//qmWAACVgQAAAAxBnqVFFSwv/wAAsoEAAAAQAZ7EdEK/AYTOTvwAfbpTQQAAABABnsZqQr8BhM5O9nj7dKaBAAAAEkGay0moQWyZTAhv//6nhAABJwAAAAxBnulFFSwv/wAAsoAAAAAQAZ8IdEK/AYTOTvwAfbpTQQAAABABnwpqQr8BhM5O9nj7dKaAAAAAGkGbDEmoQWyZTAh3//6plgE376vp8YdIATegAAAAEkGbMEnhClJlMCHf/qmWAACVgQAAAAxBn05FNEwv/wAAsoEAAAAPAZ9tdEK/AP7cd0dt8KltAAAADwGfb2pCvwD+eaILUeXSbgAAABNBm3RJqEFomUwId//+qZYAAJWAAAAADEGfkkURLC//AACygQAAAA8Bn7F0Qr8A/tx3R23wqW0AAAAPAZ+zakK/AP55ogtR5dJuAAAAE0GbuEmoQWyZTAh3//6plgAAlYEAAAAMQZ/WRRUsL/8AALKAAAAADwGf9XRCvwD+3HdHbfCpbQAAAA8Bn/dqQr8A/nmiC1Hl0m8AAAATQZv8SahBbJlMCHf//qmWAACVgAAAAAxBnhpFFSwv/wAAsoEAAAAPAZ45dEK/AP7cd0dt8KltAAAADwGeO2pCvwD+eaILUeXSbwAAABNBmiBJqEFsmUwId//+qZYAAJWBAAAADEGeXkUVLC//AACygAAAAA8Bnn10Qr8A/tx3R23wqW0AAAAPAZ5/akK/AP55ogtR5dJvAAAAE0GaZEmoQWyZTAh3//6plgAAlYAAAAAMQZ6CRRUsL/8AALKBAAAADwGeoXRCvwD+3HdHbfCpbQAAAA8BnqNqQr8A/nmiC1Hl0m8AAAATQZqoSahBbJlMCHf//qmWAACVgQAAAAxBnsZFFSwv/wAAsoEAAAAPAZ7ldEK/AP7cd0dt8KltAAAADwGe52pCvwD+eaILUeXSbgAAABNBmuxJqEFsmUwId//+qZYAAJWAAAAADEGfCkUVLC//AACygQAAAA8Bnyl0Qr8A/tx3R23wqW0AAAAPAZ8rakK/AP55ogtR5dJuAAAAE0GbMEmoQWyZTAh3//6plgAAlYEAAAAMQZ9ORRUsL/8AALKBAAAADwGfbXRCvwD+3HdHbfCpbQAAAA8Bn29qQr8A/nmiC1Hl0m4AAAATQZt0SahBbJlMCHf//qmWAACVgAAAAAxBn5JFFSwv/wAAsoEAAAAPAZ+xdEK/AP7cd0dt8KltAAAADwGfs2pCvwD+eaILUeXSbgAAABNBm7hJqEFsmUwId//+qZYAAJWBAAAADEGf1kUVLC//AACygAAAAA8Bn/V0Qr8A/tx3R23wqW0AAAAPAZ/3akK/AP55ogtR5dJvAAAAE0Gb/EmoQWyZTAh3//6plgAAlYAAAAAMQZ4aRRUsL/8AALKBAAAADwGeOXRCvwD+3HdHbfCpbQAAAA8BnjtqQr8A/nmiC1Hl0m8AAAASQZogSahBbJlMCG///qeEAAEnAAAADEGeXkUVLC//AACygAAAAA8Bnn10Qr8A/tx3R23wqW0AAAAPAZ5/akK/AP55ogtR5dJvAAAAEkGaZEmoQWyZTAhv//6nhAABJwAAAAxBnoJFFSwv/wAAsoEAAAAPAZ6hdEK/AP7cd0dt8KltAAAADwGeo2pCvwD+eaILUeXSbwAAABJBmqhJqEFsmUwIX//+jLAABI0AAAAMQZ7GRRUsL/8AALKBAAAADwGe5XRCvwD+3HdHbfCpbQAAAA8BnudqQr8A/nmiC1Hl0m4AAAAaQZrpS6hCEFskRggoB/IB/YeAIV/+OEAAEXAAAAyJbW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAAH5AAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAC7N0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAAH5AAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAARAAAAEQAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAB+QAAAEAAABAAAAAAsrbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAAAAyAAABlABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAAK1m1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAACpZzdGJsAAAAlnN0c2QAAAAAAAAAAQAAAIZhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAARABEABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAMGF2Y0MB9AAN/+EAF2f0AA2RmygiEdCAAAADAIAAABkHihTLAQAGaOvjxEhEAAAAGHN0dHMAAAAAAAAAAQAAAMoAAAIAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAZgY3R0cwAAAAAAAADKAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAAcc3RzYwAAAAAAAAABAAAAAQAAAMoAAAABAAADPHN0c3oAAAAAAAAAAAAAAMoAAAV8AAAAEQAAAA4AAAATAAAAEwAAACAAAAAUAAAAEwAAABQAAAAdAAAAFAAAABQAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAWAAAAEAAAABMAAAATAAAAHgAAADAAAAAZAAAAFAAAABQAAAAbAAAAEgAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAABLAAAAFAAAABMAAAAUAAAAJAAAABQAAAATAAAAEwAAABYAAAAQAAAAFAAAABQAAAAeAAAAGgAAABIAAAAUAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABsAAAAnAAAAFAAAABMAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAAB4AAAAWAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAAB4AAAAWAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAWAAAAEAAAABMAAAATAAAAFgAAABAAAAATAAAAEwAAABYAAAAQAAAAEwAAABMAAAAeAAAAFHN0Y28AAAAAAAAAAQAAADAAAABidWR0YQAAAFptZXRhAAAAAAAAACFoZGxyAAAAAAAAAABtZGlyYXBwbAAAAAAAAAAAAAAAAC1pbHN0AAAAJal0b28AAAAdZGF0YQAAAAEAAAAATGF2ZjU2LjM2LjEwMA==\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(display_videos('fc_test10.mp4'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:** We observe the following issue. The agent tends to oscillates between two empty cells or to stay in small areas of empty cells. In this way the agent avoids poisons. It's a \"locally optimal\" policy. But the agent also avoids lots of cheese on the board... In a way, the agent is too cautious and do not try to learn a better policy (globally optimal) because to do that the agent need to do more exploration and so to take more risk. The agents gets mediocre total rewards."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Change the temperature**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Test of the CNN, temp = 0.1 \n",
      "\n",
      "Win/lose count 1.5/1.0. Average score (0.5)\n",
      "Win/lose count 0.5/1.0. Average score (0.0)\n",
      "Win/lose count 0/1.0. Average score (-0.3333333333333333)\n",
      "Win/lose count 1.5/1.0. Average score (-0.125)\n",
      "Win/lose count 0.5/2.0. Average score (-0.4)\n",
      "Win/lose count 1.5/1.0. Average score (-0.25)\n",
      "Win/lose count 0/0. Average score (-0.21428571428571427)\n",
      "Win/lose count 1.5/1.0. Average score (-0.125)\n",
      "Win/lose count 1.5/2.0. Average score (-0.16666666666666666)\n",
      "Win/lose count 0.5/0. Average score (-0.1)\n",
      "Final score: -0.1\n",
      "\n",
      " Test of the FC, temp = 0.1 \n",
      "\n",
      "Win/lose count 0.5/1.0. Average score (-0.5)\n",
      "Win/lose count 1.0/5.0. Average score (-2.25)\n",
      "Win/lose count 0.5/1.0. Average score (-1.6666666666666667)\n",
      "Win/lose count 0.5/0. Average score (-1.125)\n",
      "Win/lose count 1.0/6.0. Average score (-1.9)\n",
      "Win/lose count 1.5/0. Average score (-1.3333333333333333)\n",
      "Win/lose count 0/0. Average score (-1.1428571428571428)\n",
      "Win/lose count 1.0/3.0. Average score (-1.25)\n",
      "Win/lose count 0/0. Average score (-1.1111111111111112)\n",
      "Win/lose count 1.5/1.0. Average score (-0.95)\n",
      "Final score: -0.95\n",
      "\n",
      " Test of the CNN, temp = 0.4 \n",
      "\n",
      "Win/lose count 1.0/10.0. Average score (-9.0)\n",
      "Win/lose count 2.5/5.0. Average score (-5.75)\n",
      "Win/lose count 5.0/6.0. Average score (-4.166666666666667)\n",
      "Win/lose count 2.0/2.0. Average score (-3.125)\n",
      "Win/lose count 2.0/3.0. Average score (-2.7)\n",
      "Win/lose count 4.0/6.0. Average score (-2.5833333333333335)\n",
      "Win/lose count 3.0/5.0. Average score (-2.5)\n",
      "Win/lose count 3.0/6.0. Average score (-2.5625)\n",
      "Win/lose count 3.5/6.0. Average score (-2.5555555555555554)\n",
      "Win/lose count 3.0/4.0. Average score (-2.4)\n",
      "Final score: -2.4\n",
      "\n",
      " Test of the FC, temp = 0.4 \n",
      "\n",
      "Win/lose count 5.5/2.0. Average score (3.5)\n",
      "Win/lose count 4.5/6.0. Average score (1.0)\n",
      "Win/lose count 6.0/6.0. Average score (0.6666666666666666)\n",
      "Win/lose count 7.0/5.0. Average score (1.0)\n",
      "Win/lose count 1.0/2.0. Average score (0.6)\n",
      "Win/lose count 2.0/2.0. Average score (0.5)\n",
      "Win/lose count 4.0/6.0. Average score (0.14285714285714285)\n",
      "Win/lose count 5.0/8.0. Average score (-0.25)\n",
      "Win/lose count 5.5/3.0. Average score (0.05555555555555555)\n",
      "Win/lose count 1.5/8.0. Average score (-0.6)\n",
      "Final score: -0.6\n",
      "\n",
      " Test of the CNN, temp = 0.6 \n",
      "\n",
      "Win/lose count 10.5/5.0. Average score (5.5)\n",
      "Win/lose count 9.0/3.0. Average score (5.75)\n",
      "Win/lose count 4.5/5.0. Average score (3.6666666666666665)\n",
      "Win/lose count 8.5/4.0. Average score (3.875)\n",
      "Win/lose count 8.0/2.0. Average score (4.3)\n",
      "Win/lose count 11.5/3.0. Average score (5.0)\n",
      "Win/lose count 6.5/3.0. Average score (4.785714285714286)\n",
      "Win/lose count 7.0/0. Average score (5.0625)\n",
      "Win/lose count 7.0/4.0. Average score (4.833333333333333)\n",
      "Win/lose count 1.5/6.0. Average score (3.9)\n",
      "Final score: 3.9\n",
      "\n",
      " Test of the FC, temp = 0.6 \n",
      "\n",
      "Win/lose count 8.0/5.0. Average score (3.0)\n",
      "Win/lose count 9.0/11.0. Average score (0.5)\n",
      "Win/lose count 5.0/1.0. Average score (1.6666666666666667)\n",
      "Win/lose count 7.5/9.0. Average score (0.875)\n",
      "Win/lose count 11.5/5.0. Average score (2.0)\n",
      "Win/lose count 4.5/2.0. Average score (2.0833333333333335)\n",
      "Win/lose count 5.0/3.0. Average score (2.0714285714285716)\n",
      "Win/lose count 8.0/8.0. Average score (1.8125)\n",
      "Win/lose count 11.5/9.0. Average score (1.8888888888888888)\n",
      "Win/lose count 9.5/8.0. Average score (1.85)\n",
      "Final score: 1.85\n",
      "\n",
      " Test of the CNN, temp = 0.9 \n",
      "\n",
      "Win/lose count 12.0/3.0. Average score (9.0)\n",
      "Win/lose count 9.5/1.0. Average score (8.75)\n",
      "Win/lose count 21.0/0. Average score (12.833333333333334)\n",
      "Win/lose count 6.5/2.0. Average score (10.75)\n",
      "Win/lose count 15.0/5.0. Average score (10.6)\n",
      "Win/lose count 10.0/2.0. Average score (10.166666666666666)\n",
      "Win/lose count 20.0/7.0. Average score (10.571428571428571)\n",
      "Win/lose count 11.5/2.0. Average score (10.4375)\n",
      "Win/lose count 16.0/1.0. Average score (10.944444444444445)\n",
      "Win/lose count 7.0/4.0. Average score (10.15)\n",
      "Final score: 10.15\n",
      "\n",
      " Test of the FC, temp = 0.9 \n",
      "\n",
      "Win/lose count 17.0/3.0. Average score (14.0)\n",
      "Win/lose count 8.5/1.0. Average score (10.75)\n",
      "Win/lose count 4.5/2.0. Average score (8.0)\n",
      "Win/lose count 11.0/6.0. Average score (7.25)\n",
      "Win/lose count 9.0/2.0. Average score (7.2)\n",
      "Win/lose count 5.0/3.0. Average score (6.333333333333333)\n",
      "Win/lose count 5.0/2.0. Average score (5.857142857142857)\n",
      "Win/lose count 5.5/3.0. Average score (5.4375)\n",
      "Win/lose count 16.5/5.0. Average score (6.111111111111111)\n",
      "Win/lose count 6.0/1.0. Average score (6.0)\n",
      "Final score: 6.0\n"
     ]
    }
   ],
   "source": [
    "epochs_test = 10\n",
    "\n",
    "for temp in [0.1, 0.4, 0.6, 0.9]:\n",
    "    \n",
    "    env = Environment(grid_size=size, max_time=T,temperature=temp)\n",
    "    agent_cnn = DQN_CNN(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "    agent_cnn.load(name_weights='cnn_trainmodel.h5',name_model='cnn_trainmodel.json')\n",
    "\n",
    "    agent_fc = DQN_FC(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "    agent_cnn.load(name_weights='fc_trainmodel.h5',name_model='fc_trainmodel.json')\n",
    "\n",
    "    print(f'\\n Test of the CNN, temp = {temp} \\n')\n",
    "    test(agent_cnn,env,epochs_test,prefix=f'{temp}cnn_test')\n",
    "    print(f'\\n Test of the FC, temp = {temp} \\n')\n",
    "    test(agent_fc,env,epochs_test,prefix=f'{temp}fc_test')\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observe behaviours**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFWRtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAAHDZYiEADf//vaH+BTZWBO6f/0/4LrOnzl/nbABNNgBSG5gp/uE0XwKZbFr8ChdA6xxpfXkif4M81EVOhL//lN/8bv6d+9Pe2acYOl4gGKXI6rQ3jHMN9Rhm3QXVgtuVIJMNSEUHSDbdycBzftHxS1SSYWe88KyIfJg3f6DvuXZ4xvxGl+kGOxXwFiIJPIRJGbfJZhVxm2UejKMZFRSgpLN+4/v10+NpT2MBYQQ1qmyBUtv4XZtp0JFzIWooJIDkEgNO2LpAX+eTjBWjBs0Ju9SM5+bww0MY1ochGGmXjmObDrb99+S+qCEXxo1HzA/4zzgfEdDKciqU93faXZxwhqBT4SuLzxIUsZ+wDmFHUcSc4y4VuS6GaKTEnXpzMTj2Vpel8hLWhUBIApb/lKvt0jSOmXpyZ8n5PZ39En56Ikl8o/DpryCMtrbj1iNKxzgGLi5V0fApK+XkbsX2y4MaTEBwGhol91SqqMtCn/9DiePMCZyDMFkC+/ZAlvu8a4zHmzvDhEYAA6QAAs8ALA6wCB+HRCL5e+EzHzWGGDzOkgF9FzQVMgYwQnD/mojARibBmbhsIulIdZ1bZ1+kYWhlVjiVngKWQAAABRBmiFsQ7/+qZYAER6tkN6g3dTFgAAAABpBmkU8IZMphDv//qmWAAs3PMiBm53sm9JuYQAAAA9BnmNqU8L/AA0wikW47WAAAAAQAZ6CdEK/ABHXZjgPygAaYQAAAA8BnoRqQr8AC6Rtd33fJqEAAAAjQZqJSahBaJlMCHf//qmWAATnQY38Qg2//CVUM3//7qXbaJUAAAAQQZ6nRREsL/8ABdKDZ55MIQAAAA8BnsZ0Qr8AC6dAOhOTJqAAAAAQAZ7IakK/AAfFmDyXM+UhgAAAABNBms1JqEFsmUwId//+qZYAAJWBAAAAE0Ge60UVLC//AAisfPosV3Fo/GIAAAAQAZ8KdEK/AAxFlXchsqVA4AAAABABnwxqQr8AC/EyTTfSQdkxAAAAE0GbEUmoQWyZTAh3//6plgAAlYEAAAAQQZ8vRRUsL/8ACK+gixwGMQAAABABn050Qr8ADEWVdyGypUDgAAAAEAGfUGpCvwAL8TJNN9JB2TAAAAAZQZtVSahBbJlMCHf//qmWAAdJdMj++r7u0QAAABBBn3NFFSwv/wAIrQHLyMYgAAAAEAGfknRCvwAMRZV3IbKlQOAAAAAPAZ+UakK/AAv1iwLr/DfBAAAAGUGbmUmoQWyZTAh3//6plgAHU4vRibSrGyYAAAAQQZ+3RRUsL/8ACK0By8jGIQAAABABn9Z0Qr8AC/AKZ5X5KcLxAAAADwGf2GpCvwAMQlbGFZuvQAAAABxBm91JqEFsmUwId//+qZYAB0mqBH+AgD+/sa2BAAAAEEGf+0UVLC//AAiufucLMYgAAAAPAZ4adEK/AAfEvxcB+bLBAAAAEAGeHGpCvwAMQCxr3mlaBMEAAAATQZoBSahBbJlMCHf//qmWAACVgAAAABBBnj9FFSwv/wAIr6CLHAYxAAAAEAGeXnRCvwAL8ApnlfkpwvEAAAAQAZ5AakK/AAxALGveaVoEwAAAABNBmkVJqEFsmUwId//+qZYAAJWBAAAAEEGeY0UVLC//AAivoIscBjEAAAAQAZ6CdEK/AAvwCmeV+SnC8QAAABABnoRqQr8ADEAsa95pWgTBAAAAE0GaiUmoQWyZTAh3//6plgAAlYEAAAAQQZ6nRRUsL/8ACK+gixwGMQAAABABnsZ0Qr8AC/AKZ5X5KcLwAAAAEAGeyGpCvwAMQCxr3mlaBMAAAAATQZrNSahBbJlMCHf//qmWAACVgQAAABBBnutFFSwv/wAIr6CLHAYxAAAAEAGfCnRCvwAL8ApnlfkpwvAAAAAQAZ8MakK/AAxALGveaVoEwQAAABNBmxFJqEFsmUwId//+qZYAAJWBAAAAEEGfL0UVLC//AAivoIscBjEAAAAQAZ9OdEK/AAvwCmeV+SnC8AAAABABn1BqQr8ADEAsa95pWgTAAAAAE0GbVUmoQWyZTAh3//6plgAAlYEAAAAQQZ9zRRUsL/8ACK+gixwGMQAAABABn5J0Qr8AC/AKZ5X5KcLwAAAAEAGflGpCvwAMQCxr3mlaBMEAAAATQZuZSahBbJlMCHf//qmWAACVgAAAABBBn7dFFSwv/wAIr6CLHAYxAAAAEAGf1nRCvwAL8ApnlfkpwvEAAAAQAZ/YakK/AAxALGveaVoEwAAAABlBm91JqEFsmUwId//+qZYAB1OL0YmxcjtFAAAAEEGf+0UVLC//AAitAcvIxiAAAAAQAZ4adEK/AAvwCmeV+SnC8QAAAA8BnhxqQr8AB8a+aHWjdoEAAAATQZoBSahBbJlMCHf//qmWAACVgAAAABFBnj9FFSwv/wAIrxn53kMYgAAAABABnl50Qr8ADEWVdyGypUDhAAAAEAGeQGpCvwAL8TJNN9JB2TAAAAATQZpFSahBbJlMCHf//qmWAACVgQAAABBBnmNFFSwv/wAIr6CLHAYxAAAAEAGegnRCvwAMRZV3IbKlQOEAAAAQAZ6EakK/AAvxMk030kHZMQAAABNBmolJqEFsmUwId//+qZYAAJWBAAAAEEGep0UVLC//AAivoIscBjEAAAAQAZ7GdEK/AAxFlXchsqVA4AAAABABnshqQr8AC/EyTTfSQdkwAAAAE0GazUmoQWyZTAh3//6plgAAlYEAAAAQQZ7rRRUsL/8ACK+gixwGMQAAABABnwp0Qr8ADEWVdyGypUDgAAAAEAGfDGpCvwAL8TJNN9JB2TEAAAATQZsRSahBbJlMCHf//qmWAACVgQAAABBBny9FFSwv/wAIr6CLHAYxAAAAEAGfTnRCvwAMRZV3IbKlQOAAAAAQAZ9QakK/AAvxMk030kHZMAAAABNBm1VJqEFsmUwId//+qZYAAJWBAAAAEEGfc0UVLC//AAivoIscBjEAAAAQAZ+SdEK/AAxFlXchsqVA4AAAABABn5RqQr8AC/EyTTfSQdkxAAAAE0GbmUmoQWyZTAh3//6plgAAlYAAAAAQQZ+3RRUsL/8ACK+gixwGMQAAABABn9Z0Qr8ADEWVdyGypUDhAAAAEAGf2GpCvwAL8TJNN9JB2TAAAAATQZvdSahBbJlMCHf//qmWAACVgQAAABBBn/tFFSwv/wAIr6CLHAYxAAAAEAGeGnRCvwAMRZV3IbKlQOEAAAAQAZ4cakK/AAvxMk030kHZMQAAABNBmgFJqEFsmUwId//+qZYAAJWAAAAAEEGeP0UVLC//AAivoIscBjEAAAAQAZ5edEK/AAxFlXchsqVA4QAAABABnkBqQr8AC/EyTTfSQdkwAAAAGUGaRUmoQWyZTAh3//6plgAHSXTI/vq+7tEAAAAQQZ5jRRUsL/8ACK0By8jGIAAAABABnoJ0Qr8ADEWVdyGypUDhAAAADwGehGpCvwASW1ru+74iwQAAABlBmolJqEFsmUwId//+qZYAB1OL0Ym0qxsnAAAAEEGep0UVLC//AAitAcvIxiEAAAAQAZ7GdEK/AAvwCmeV+SnC8AAAAA8BnshqQr8ADEJWxhWbr0AAAAAaQZrNSahBbJlMCHf//qmWAAdJqgR/fV93aIEAAAAQQZ7rRRUsL/8ACK5+5wsxiAAAAA8Bnwp0Qr8AB8S/FwH5ssAAAAAQAZ8MakK/AAxALGveaVoEwQAAABlBmxFJqEFsmUwId//+qZYAB1OL0YmxcjtFAAAAEEGfL0UVLC//AAitAcvIxiEAAAAQAZ9OdEK/AAvwCmeV+SnC8AAAAA8Bn1BqQr8AB8a+aHWjdoAAAAATQZtVSahBbJlMCHf//qmWAACVgQAAABFBn3NFFSwv/wAIrxn53kMYgAAAABABn5J0Qr8ADEWVdyGypUDgAAAAEAGflGpCvwAL8TJNN9JB2TEAAAATQZuZSahBbJlMCHf//qmWAACVgAAAABBBn7dFFSwv/wAIr6CLHAYxAAAAEAGf1nRCvwAMRZV3IbKlQOEAAAAQAZ/YakK/AAvxMk030kHZMAAAABNBm91JqEFsmUwId//+qZYAAJWBAAAAEEGf+0UVLC//AAivoIscBjEAAAAQAZ4adEK/AAxFlXchsqVA4QAAABABnhxqQr8AC/EyTTfSQdkxAAAAG0GaAUmoQWyZTAh3//6plgAHSXTI/wEAf39jWwAAABBBnj9FFSwv/wAIrn7nCzGIAAAADwGeXnRCvwAHxL8XAfmywQAAABABnkBqQr8ADEAsa95pWgTAAAAAE0GaRUmoQWyZTAh3//6plgAAlYEAAAAQQZ5jRRUsL/8ACK+gixwGMQAAABABnoJ0Qr8AC/AKZ5X5KcLxAAAAEAGehGpCvwAMQCxr3mlaBMEAAAATQZqJSahBbJlMCHf//qmWAACVgQAAABBBnqdFFSwv/wAIr6CLHAYxAAAAEAGexnRCvwAL8ApnlfkpwvAAAAAQAZ7IakK/AAxALGveaVoEwAAAABNBms1JqEFsmUwId//+qZYAAJWBAAAAEEGe60UVLC//AAivoIscBjEAAAAQAZ8KdEK/AAvwCmeV+SnC8AAAABABnwxqQr8ADEAsa95pWgTBAAAAE0GbEUmoQWyZTAh3//6plgAAlYEAAAAQQZ8vRRUsL/8ACK+gixwGMQAAABABn050Qr8AC/AKZ5X5KcLwAAAAEAGfUGpCvwAMQCxr3mlaBMAAAAATQZtVSahBbJlMCHf//qmWAACVgQAAABBBn3NFFSwv/wAIr6CLHAYxAAAAEAGfknRCvwAL8ApnlfkpwvAAAAAQAZ+UakK/AAxALGveaVoEwQAAABNBm5lJqEFsmUwId//+qZYAAJWAAAAAEEGft0UVLC//AAivoIscBjEAAAAQAZ/WdEK/AAvwCmeV+SnC8QAAABABn9hqQr8ADEAsa95pWgTAAAAAE0Gb3UmoQWyZTAh3//6plgAAlYEAAAAQQZ/7RRUsL/8ACK+gixwGMQAAABABnhp0Qr8AC/AKZ5X5KcLxAAAAEAGeHGpCvwAMQCxr3mlaBMEAAAATQZoBSahBbJlMCHf//qmWAACVgAAAABBBnj9FFSwv/wAIr6CLHAYxAAAAEAGeXnRCvwAL8ApnlfkpwvEAAAAQAZ5AakK/AAxALGveaVoEwAAAABxBmkVJqEFsmUwIb//+p4QAFhxWzE/1dvdT9rfJAAAAEEGeY0UVLC//AA0wjjO5cSAAAAAQAZ6CdEK/ABHfUSJ8WYpK0QAAAA8BnoRqQr8AG6StjCs278EAAAAaQZqGSahBbJlMCHf//qmWAAs3PMhtiDd1S8EAAAAiQZqqSeEKUmUwId/+qZYAB37iX/xCDb/8JVQzf//updtmjwAAABBBnshFNEwv/wAI7oJAj+hwAAAADwGe53RCvwALpGMXAfmaIAAAABABnulqQr8ADEOqeS5nyd2BAAAAE0Ga7kmoQWiZTAh3//6plgAAlYAAAAATQZ8MRREsL/8ACO+ggm5v1yGmHAAAABABnyt0Qr8ADEAABklv9hpBAAAAEAGfLWpCvwAMQ7Utw2bVi4EAAAASQZsySahBbJlMCG///qeEAAEnAAAAEEGfUEUVLC//AAjvoIKmyk4AAAAQAZ9vdEK/AAxAAAZJb/YaQAAAABABn3FqQr8ADEO1LcNm1YuBAAAAGkGbc0moQWyZTAh3//6plgAHf9peFqCf2DmgAAAAG0Gbl0nhClJlMCHf/qmWAAsmlnKDNAp9GP00egAAABBBn7VFNEwv/wANMI4zuXEhAAAAEAGf1HRCvwAR31EifFmKStAAAAAPAZ/WakK/ABukrYwrNu/BAAAAHEGb20moQWiZTAh3//6plgALNzzIiianUIN2FFUAAAAQQZ/5RREsL/8ADTCN3uBoQAAAABABnhh0Qr8AEddWjJLf6++BAAAADwGeGmpCvwARW1ru+74mwAAAABJBmh9JqEFsmUwIb//+p4QAAScAAAAMQZ49RRUsL/8AALKBAAAAEAGeXHRCvwARYQBz+tA5ZsAAAAAQAZ5eakK/ABFbWu6yGHLNgAAAABJBmkNJqEFsmUwIb//+p4QAAScAAAAMQZ5hRRUsL/8AALKAAAAAEAGegHRCvwARYQBz+tA5ZsEAAAAQAZ6CakK/ABFbWu6yGHLNgAAAABJBmodJqEFsmUwIZ//+nhAABH0AAAAMQZ6lRRUsL/8AALKBAAAAEAGexHRCvwARYQBz+tA5ZsEAAAAQAZ7GakK/ABFbWu6yGHLNgQAAABtBmslLqEIQWyRGCCgH8gH9h4BRMK/+OEAAEXAAAAAmAZ7oakK/Aq9j7UHE3arDSSblqoYHLLW7zSogox6Zjx5WlNOAvWAAAAyBbW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAAH5AAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAC6t0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAAH5AAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAARAAAAEQAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAB+QAAAEAAABAAAAAAsjbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAAAAyAAABlABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAAKzm1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAACo5zdGJsAAAAlnN0c2QAAAAAAAAAAQAAAIZhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAARABEABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAMGF2Y0MB9AAN/+EAF2f0AA2RmygiEdCAAAADAIAAABkHihTLAQAGaOvjxEhEAAAAGHN0dHMAAAAAAAAAAQAAAMoAAAIAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAZYY3R0cwAAAAAAAADJAAAAAgAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAEeAAAABgAAAAeAAAAEwAAABQAAAATAAAAJwAAABQAAAATAAAAFAAAABcAAAAXAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAHQAAABQAAAAUAAAAEwAAAB0AAAAUAAAAFAAAABMAAAAgAAAAFAAAABMAAAAUAAAAFwAAABQAAAAUAAAAFAAAABcAAAAUAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAFwAAABQAAAAUAAAAFAAAABcAAAAUAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAFwAAABQAAAAUAAAAFAAAAB0AAAAUAAAAFAAAABMAAAAXAAAAFQAAABQAAAAUAAAAFwAAABQAAAAUAAAAFAAAABcAAAAUAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAFwAAABQAAAAUAAAAFAAAABcAAAAUAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAFwAAABQAAAAUAAAAFAAAABcAAAAUAAAAFAAAABQAAAAdAAAAFAAAABQAAAATAAAAHQAAABQAAAAUAAAAEwAAAB4AAAAUAAAAEwAAABQAAAAdAAAAFAAAABQAAAATAAAAFwAAABUAAAAUAAAAFAAAABcAAAAUAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAHwAAABQAAAATAAAAFAAAABcAAAAUAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAFwAAABQAAAAUAAAAFAAAABcAAAAUAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAFwAAABQAAAAUAAAAFAAAABcAAAAUAAAAFAAAABQAAAAXAAAAFAAAABQAAAAUAAAAIAAAABQAAAAUAAAAEwAAAB4AAAAmAAAAFAAAABMAAAAUAAAAFwAAABcAAAAUAAAAFAAAABYAAAAUAAAAFAAAABQAAAAeAAAAHwAAABQAAAAUAAAAEwAAACAAAAAUAAAAFAAAABMAAAAWAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAfAAAAKgAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1Ni4zNi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(display_videos('0.1cnn_test9.mp4'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFeZtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAAMPZYiEADv//vb8/AptUwndP/9P+VxN2UIE1zAALTEAIgTYlI6+z4vwKZrZf8CiWgOunhgGOLEl07YT/ER+EMBOCqYsf15ipu3JI0XxdyTbLcNY7+/9EPGtEDmYkNBOaoAs7rhTEvMrvfrPSVcWWqnc9lmVAtGitzBnA8iAxHUEBHlIEo5uSxZ6wzmCK5N52G0ENpdrH+ktvhfsMaHEjoE0iLGfnNQzrmk/FLiEcGWi0hvpdCdPRRTFPyJbI930itD0jzyeLlLVjmnE2slpB4bMvP9TuqN+WFsuL5KClSyTefxbkyXXqhFut1uqBineGqtfoS/nNm2qucqH18TMZlBUIknj5uEEdLXzm31rraTVLgb3pRHgrsJp9R0G6QOOK/cjlBrnXDrasaHmMkiNoC3xal1FIEwfykj4gD161qdPzquFaqGCoL+xeXWKh23gnE5reyhHt3KanABME2Fa3nOxH1uF36C47PR2O13AQGjH/4xKcJJSSSc2aNyNk8kReHZLxNxz8LMaDg2c/dah6ZAY1I0ES3UyWAyTGPek/rX7ScLL/gVy94ljbAYEB0S+MyxYfnmLbyH77ENHJS+g4IwSzeD3KzoGAU3C8Slr80TyU2el823r/9UWQwGhgXcnpii6NAttYqzQ8uKNMyGNPXzCvrAP538aFFMQ3WhUKLYY3HjCZ2BWFjzbUBg9eQQtpfBlNxDGkcsj3vMCxPQUMDeRcA2kLNHbXNc//R+vvLOXzuKTpf3dxZHHrxfXvRAdP1wBAc2qrnA8K/IqVOEft6rw9Meo7KMMUu9kdZNtOvFj1F8x6xYARc1PUcMoBi6v7EUGFb7mZauzQkeXBzwVdaVBO48zCesSWy/epR3NPH3Qo7GiSBUiXeVwf/BBJEbPrE3VdV82mLPzfQPYBMLFZEF2g1TXTInGbvrft1iW8xqWuumRiYvL9hI/FSXAEJNuxXSff7LUpq5XtaXia0f3bCUgbVNPwL72jVMgVRx9dsBxOpUIzlOCfU9JkICoR+vHmGMRdWYCgMIZ4DUQAACisAETAAAADUGaJGxDv/6plgAAlYAAAAAKQZ5CeIX/AACygQAAABABnmF0Qr8AfGxg2SKRyERfAAAAEAGeY2pCvwB8TU/oy2OTFWcAAAATQZpoSahBaJlMCHf//qmWAACVgQAAAAxBnoZFESwv/wAAsoEAAAAQAZ6ldEK/AHxsYNkikchEXwAAABABnqdqQr8AfE1P6MtjkxVmAAAAE0GarEmoQWyZTAh3//6plgAAlYAAAAAMQZ7KRRUsL/8AALKBAAAAEAGe6XRCvwB8bGDZIpHIRF8AAAAQAZ7rakK/AHxNT+jLY5MVZgAAABNBmvBJqEFsmUwId//+qZYAAJWBAAAADEGfDkUVLC//AACygQAAABABny10Qr8AfGxg2SKRyERfAAAAEAGfL2pCvwB8TU/oy2OTFWYAAAATQZs0SahBbJlMCHf//qmWAACVgAAAAAxBn1JFFSwv/wAAsoEAAAAQAZ9xdEK/AHxsYNkikchEXwAAABABn3NqQr8AfE1P6MtjkxVmAAAAE0GbeEmoQWyZTAh3//6plgAAlYEAAAAMQZ+WRRUsL/8AALKAAAAAEAGftXRCvwB8bGDZIpHIRF8AAAAQAZ+3akK/AHxNT+jLY5MVZwAAABNBm7xJqEFsmUwId//+qZYAAJWAAAAAFkGf2kUVLC//AF0GmFkVQT+BYvJBVmEAAAAQAZ/5dEK/AHw4nik2yVTUgAAAABABn/tqQr8AfFnhDxoaxp2BAAAAE0Gb4EmoQWyZTAh3//6plgAAlYEAAAAMQZ4eRRUsL/8AALKAAAAAEAGePXRCvwB8bGDZIpHIRF8AAAAQAZ4/akK/AHxNT+jLY5MVZwAAABNBmiRJqEFsmUwId//+qZYAAJWAAAAADEGeQkUVLC//AACygQAAABABnmF0Qr8AfGxg2SKRyERfAAAAEAGeY2pCvwB8TU/oy2OTFWcAAAATQZpoSahBbJlMCHf//qmWAACVgQAAAAxBnoZFFSwv/wAAsoEAAAAQAZ6ldEK/AHxsYNkikchEXwAAABABnqdqQr8AfE1P6MtjkxVmAAAAE0GarEmoQWyZTAh3//6plgAAlYAAAAAMQZ7KRRUsL/8AALKBAAAAEAGe6XRCvwB8bGDZIpHIRF8AAAAQAZ7rakK/AHxNT+jLY5MVZgAAABNBmvBJqEFsmUwId//+qZYAAJWBAAAADEGfDkUVLC//AACygQAAABABny10Qr8AfGxg2SKRyERfAAAAEAGfL2pCvwB8TU/oy2OTFWYAAAATQZs0SahBbJlMCHf//qmWAACVgAAAAAxBn1JFFSwv/wAAsoEAAAAQAZ9xdEK/AHxsYNkikchEXwAAABABn3NqQr8AfE1P6MtjkxVmAAAAE0GbeEmoQWyZTAh3//6plgAAlYEAAAAMQZ+WRRUsL/8AALKAAAAAEAGftXRCvwB8bGDZIpHIRF8AAAAQAZ+3akK/AHxNT+jLY5MVZwAAABNBm7xJqEFsmUwId//+qZYAAJWAAAAADEGf2kUVLC//AACygQAAABABn/l0Qr8AfGxg2SKRyERfAAAAEAGf+2pCvwB8TU/oy2OTFWcAAAATQZvgSahBbJlMCHf//qmWAACVgQAAAAxBnh5FFSwv/wAAsoAAAAAQAZ49dEK/AHxsYNkikchEXwAAABABnj9qQr8AfE1P6MtjkxVnAAAAE0GaJEmoQWyZTAh3//6plgAAlYAAAAAMQZ5CRRUsL/8AALKBAAAAEAGeYXRCvwB8bGDZIpHIRF8AAAAQAZ5jakK/AHxNT+jLY5MVZwAAABNBmmhJqEFsmUwId//+qZYAAJWBAAAADEGehkUVLC//AACygQAAABABnqV0Qr8AfGxg2SKRyERfAAAAEAGep2pCvwB8TU/oy2OTFWYAAAATQZqsSahBbJlMCHf//qmWAACVgAAAAAxBnspFFSwv/wAAsoEAAAAQAZ7pdEK/AHxsYNkikchEXwAAABABnutqQr8AfE1P6MtjkxVmAAAAE0Ga8EmoQWyZTAh3//6plgAAlYEAAAAMQZ8ORRUsL/8AALKBAAAAEAGfLXRCvwB8bGDZIpHIRF8AAAAQAZ8vakK/AHxNT+jLY5MVZgAAABNBmzRJqEFsmUwId//+qZYAAJWAAAAADEGfUkUVLC//AACygQAAABABn3F0Qr8AfGxg2SKRyERfAAAAEAGfc2pCvwB8TU/oy2OTFWYAAAAyQZt4SahBbJlMCHf//qmWAjHXn5llapqvApRIF4FM1yuHoiQNLMOCYQ1xvc7LH57yu6EAAAAVQZ+WRRUsL/8BZVWjmf86dpoo4jT+AAAADwGftXRCvwDIyWbg2S8ZjQAAABABn7dqQr8B3x98tf22fL/BAAAAE0GbvEmoQWyZTAh3//6plgAAlYAAAAAMQZ/aRRUsL/8AALKBAAAAEAGf+XRCvwHfaVjCWHIQdxAAAAAQAZ/7akK/Ad7tDoRY5MMoYQAAABNBm+BJqEFsmUwId//+qZYAAJWBAAAADEGeHkUVLC//AACygAAAABABnj10Qr8B32lYwlhyEHcQAAAAEAGeP2pCvwHe7Q6EWOTDKGEAAAATQZokSahBbJlMCHf//qmWAACVgAAAAAxBnkJFFSwv/wAAsoEAAAAQAZ5hdEK/Ad9pWMJYchB3EAAAABABnmNqQr8B3u0OhFjkwyhhAAAAE0GaaEmoQWyZTAh3//6plgAAlYEAAAAMQZ6GRRUsL/8AALKBAAAAEAGepXRCvwHfaVjCWHIQdxEAAAAQAZ6nakK/Ad7tDoRY5MMoYAAAABNBmqxJqEFsmUwId//+qZYAAJWAAAAADEGeykUVLC//AACygQAAABABnul0Qr8B32lYwlhyEHcQAAAAEAGe62pCvwHe7Q6EWOTDKGAAAAATQZrwSahBbJlMCHf//qmWAACVgQAAAAxBnw5FFSwv/wAAsoEAAAAQAZ8tdEK/Ad9pWMJYchB3EQAAABABny9qQr8B3u0OhFjkwyhgAAAAE0GbNEmoQWyZTAh3//6plgAAlYAAAAAMQZ9SRRUsL/8AALKBAAAAEAGfcXRCvwHfaVjCWHIQdxAAAAAQAZ9zakK/Ad7tDoRY5MMoYAAAABNBm3hJqEFsmUwId//+qZYAAJWBAAAADEGflkUVLC//AACygAAAABABn7V0Qr8B32lYwlhyEHcRAAAAEAGft2pCvwHe7Q6EWOTDKGEAAAATQZu8SahBbJlMCHf//qmWAACVgAAAAAxBn9pFFSwv/wAAsoEAAAAQAZ/5dEK/Ad9pWMJYchB3EAAAABABn/tqQr8B3u0OhFjkwyhhAAAAE0Gb4EmoQWyZTAh3//6plgAAlYEAAAAMQZ4eRRUsL/8AALKAAAAAEAGePXRCvwHfaVjCWHIQdxAAAAAQAZ4/akK/Ad7tDoRY5MMoYQAAACBBmiRJqEFsmUwId//+qZYCY9mPtjIHD+jSoFopiURqQAAAABBBnkJFFSwv/wFv+0KNsd7RAAAAEAGeYXRCvwHsJA1tMaBeTcAAAAAQAZ5jakK/AevtDoRY32MnYQAAABNBmmhJqEFsmUwId//+qZYAAJWBAAAADEGehkUVLC//AACygQAAABABnqV0Qr8B7GlYwlhu4HZxAAAAEAGep2pCvwHr7Q6EWN9jJ2AAAAAcQZqsSahBbJlMCHf//qmWAiNnUIMz7P2l/UoRcAAAABBBnspFFSwv/wFlVdx7AkqZAAAADwGe6XRCvwHsaVjByWZOwAAAABABnutqQr8B64XvSJjWZYOAAAAAE0Ga8EmoQWyZTAh3//6plgAAlYEAAAAMQZ8ORRUsL/8AALKBAAAAEAGfLXRCvwHYrW7G6v4B3EEAAAAQAZ8vakK/AditbsCo+3CFgAAAABNBmzRJqEFsmUwId//+qZYAAJWAAAAADEGfUkUVLC//AACygQAAABABn3F0Qr8B2K1uxur+AdxAAAAAEAGfc2pCvwHYrW7AqPtwhYAAAAATQZt4SahBbJlMCHf//qmWAACVgQAAAAxBn5ZFFSwv/wAAsoAAAAAQAZ+1dEK/Aditbsbq/gHcQQAAABABn7dqQr8B2K1uwKj7cIWBAAAAE0GbvEmoQWyZTAh3//6plgAAlYAAAAAMQZ/aRRUsL/8AALKBAAAAEAGf+XRCvwHYrW7G6v4B3EAAAAAQAZ/7akK/AditbsCo+3CFgQAAABJBm+BJqEFsmUwIb//+p4QAAScAAAAMQZ4eRRUsL/8AALKAAAAAEAGePXRCvwHYrW7G6v4B3EAAAAAQAZ4/akK/AditbsCo+3CFgQAAABpBmiFJqEFsmUwId//+qZYCMdmPx+MOj75aQAAAABxBmkVJ4QpSZTAh3/6plgH0z5m0uuPP04PXYmfBAAAAEEGeY0U0TC//AVsRuHHIkg4AAAAQAZ6CdEK/AdJEyI7FmKNN6QAAAA8BnoRqQr8B0mglyv79b0EAAAATQZqJSahBaJlMCHf//qmWAACVgQAAAAxBnqdFESwv/wAAsoEAAAAQAZ7GdEK/AcxIBvwAfbpEwAAAABABnshqQr8BzEgG9nj7dImAAAAAJEGazUmoQWyZTAhv//6nhAPvvs+lPir2XxCAf/8JUsef/7JumQAAABBBnutFFSwv/wFbVcfFDouIAAAADwGfCnRCvwHR58XAflnTQAAAAA8BnwxqQr8B0bYxV4An8jMAAAAaQZsOSahBbJlMCHf//qmWAMLFhui3cx+Ao+EAAAASQZsySeEKUmUwId/+qZYAAJWBAAAADEGfUEU0TC//AACygAAAABABn290Qr8B0mlYvP4HI15AAAAAEAGfcWpCvwHR7Q5/mW79b0EAAAATQZt2SahBaJlMCHf//qmWAACVgAAAAAxBn5RFESwv/wAAsoAAAAAQAZ+zdEK/AdJpWLz+ByNeQQAAABABn7VqQr8B0e0Of5lu/W9AAAAAE0GbukmoQWyZTAh3//6plgAAlYEAAAAMQZ/YRRUsL/8AALKBAAAAEAGf93RCvwHSaVi8/gcjXkAAAAAQAZ/5akK/AdHtDn+Zbv1vQQAAABNBm/5JqEFsmUwId//+qZYAAJWAAAAADEGeHEUVLC//AACygQAAABABnjt0Qr8B0mlYvP4HI15BAAAAEAGePWpCvwHR7Q5/mW79b0AAAAASQZoiSahBbJlMCG///qeEAAEnAAAADEGeQEUVLC//AACygQAAABABnn90Qr8B0mlYvP4HI15AAAAAEAGeYWpCvwHR7Q5/mW79b0EAAAASQZpmSahBbJlMCGf//p4QAAR8AAAADEGehEUVLC//AACygQAAABABnqN0Qr8B0mlYvP4HI15BAAAAEAGepWpCvwHR7Q5/mW79b0EAAAAaQZqpS6hCEFskRggoB/IB/YeAIV/+OEAAEXEAAAAoQZ7IQhX/Aq9j7UHE3arDSm/zVeyfuqXxhGfDiwZBxkSOU1z6eKln+wAAABABnudpEK8B0WSiZIyO0bBgAAAMiW1vb3YAAABsbXZoZAAAAAAAAAAAAAAAAAAAA+gAAB+QAAEAAAEAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAuzdHJhawAAAFx0a2hkAAAAAwAAAAAAAAAAAAAAAQAAAAAAAB+QAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAEQAAABEAAAAAAAJGVkdHMAAAAcZWxzdAAAAAAAAAABAAAfkAAABAAAAQAAAAALK21kaWEAAAAgbWRoZAAAAAAAAAAAAAAAAAAAMgAAAZQAVcQAAAAAAC1oZGxyAAAAAAAAAAB2aWRlAAAAAAAAAAAAAAAAVmlkZW9IYW5kbGVyAAAACtZtaW5mAAAAFHZtaGQAAAABAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEAAAAMdXJsIAAAAAEAAAqWc3RibAAAAJZzdHNkAAAAAAAAAAEAAACGYXZjMQAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAEQARAASAAAAEgAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABj//wAAADBhdmNDAfQADf/hABdn9AANkZsoIhHQgAAAAwCAAAAZB4oUywEABmjr48RIRAAAABhzdHRzAAAAAAAAAAEAAADKAAACAAAAABRzdHNzAAAAAAAAAAEAAAABAAAGYGN0dHMAAAAAAAAAygAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACAAAAAABAAAEAAAAAAEAAAAAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAFxAAAABEAAAAOAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAaAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAADYAAAAZAAAAEwAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAJAAAABQAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAgAAAAFAAAABMAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAeAAAAIAAAABQAAAAUAAAAEwAAABcAAAAQAAAAFAAAABQAAAAoAAAAFAAAABMAAAATAAAAHgAAABYAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAAB4AAAAsAAAAFAAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1Ni4zNi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(display_videos('0.4cnn_test9.mp4'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFShtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAAIyZYiEADv//vb8/AptUwndP/9P+VxN2UIE1zAALTEAIgTYlI64FO/xNiswi78+BS4Bt+BTSZwV//6QrVChkdBLAYkq+Kvv4iHNi90xS62iy8wpMdx9xbrfRyrzvTuZ7JfU9R3KjP3e9osW8y/xiG6chl+shHLCQnjm50NmPCqn4BM+zu2r7L3lxv9hqiyKrVEdBXzGqwr/U/WAYATlLuEPPHh0kxwAl0qbu4VPRfI3uiO8OC4SRyD/DwG4G7asvvKT8v+USh/lU8tBp3LLH4NWJZOp5giGNMTPy4dLpUW47JcWh6NyQ1zjKbTnXjftc7u2skuMBLKUVPaKcZbzrVfmKq9vr+GjD7zURDz2ELrq9p8cZYEPpk4E0kfk5JdCWHJsbx8hWip2UEPIGNiD9scCIR/ovM1YdbFI/ggDcaCGkK6jZaYDAQ8mJYhBOgSc0sLe20+kDwsd4KP3g7c0BkXQ/oYoCxX2KzuDTmy4QnuBjyqIFFlHWH6idJi962RRXMG++dTaL6f4XANxEiX97TzJxXUXzj1ltMqI+VoPqzYnZlO3ZNOSpLHkshIWX8wWSdhlpy9Gqc37o1W2wo3P3tzJXwaC3WEhhwljJhv4Y/xcD2EyhnbABIFj+KUe1yij9/Ox/S9NDQihyDhuqL41iTH2YRMUyoMPjUZgd1WEqwvjjFxeznaklcY9tsAc8SAAFkTwSUj0iN1bZRLhbrkcaSoeXWIWsum2re418GR1i284RVAoYQAAACZBmiRsQ7/+qZYBCPE85lljCqvApmuKngUSejP6uM9pfLbqeqBFgAAAACRBnkJ4hf8BBuITCL8n/EIs//8QdBks//w0/LP/8QMduglE0HEAAAAQAZ5hdEK/AJb6Rvr9KEi24AAAABABnmNqQr8BbLHjlf24fNbBAAAAF0GaaEmoQWiZTAh3//6plgPxLK5AaFVBAAAADkGehkURLC//AbIaWkfBAAAAEAGepXRCvwJJYrGD4N2952EAAAAQAZ6nakK/AkhqHQhNPsg5oAAAABNBmqxJqEFsmUwId//+qZYAAJWAAAAADEGeykUVLC//AACygQAAABABnul0Qr8CSWKxg+DdvedgAAAAEAGe62pCvwJIah0ITT7IOaAAAAATQZrwSahBbJlMCHf//qmWAACVgQAAAAxBnw5FFSwv/wAAsoEAAAAQAZ8tdEK/AklisYPg3b3nYQAAABABny9qQr8CSGodCE0+yDmgAAAAE0GbNEmoQWyZTAh3//6plgAAlYAAAAAMQZ9SRRUsL/8AALKBAAAAEAGfcXRCvwJJYrGD4N2952AAAAAQAZ9zakK/AkhqHQhNPsg5oAAAABNBm3hJqEFsmUwId//+qZYAAJWBAAAADEGflkUVLC//AACygAAAABABn7V0Qr8CSWKxg+DdvedhAAAAEAGft2pCvwJIah0ITT7IOaEAAAATQZu8SahBbJlMCHf//qmWAACVgAAAAAxBn9pFFSwv/wAAsoEAAAAQAZ/5dEK/AklisYPg3b3nYAAAABABn/tqQr8CSGodCE0+yDmhAAAAE0Gb4EmoQWyZTAh3//6plgAAlYEAAAAMQZ4eRRUsL/8AALKAAAAAEAGePXRCvwJJYrGD4N2952AAAAAQAZ4/akK/AkhqHQhNPsg5oQAAABNBmiRJqEFsmUwId//+qZYAAJWAAAAADEGeQkUVLC//AACygQAAABABnmF0Qr8CSWKxg+DdvedgAAAAEAGeY2pCvwJIah0ITT7IOaEAAAAeQZpoSahBbJlMCG///qeEBuPmqaza0vRP58HiSSkhAAAAEUGehkUVLC//AaNARXPgmG9BAAAAEAGepXRCvwJJYrFsbBJeMWEAAAAPAZ6nakK/AjNiusr+9WBAAAAAGUGaqUmoQWyZTAh3//6plgObo5pZ0dPfkTMAAAAaQZrNSeEKUmUwId/+qZYC5dYD/xZ3IuzsbMEAAAAQQZ7rRTRML/8Bh5BYhgSTcAAAABABnwp0Qr8CFaic9wkUKO6AAAAAEAGfDGpCvwIVqJ45wPt9YsEAAAATQZsRSahBaJlMCHf//qmWAACVgQAAAAxBny9FESwv/wAAsoEAAAAQAZ9OdEK/AhWonkEVnLViwAAAABABn1BqQr8CFaieOcD7fWLAAAAAE0GbVUmoQWyZTAh3//6plgAAlYEAAAAMQZ9zRRUsL/8AALKAAAAAEAGfknRCvwIVqJ5BFZy1YsAAAAAQAZ+UakK/AhWonjnA+31iwQAAABNBm5lJqEFsmUwId//+qZYAAJWAAAAADEGft0UVLC//AACygQAAABABn9Z0Qr8CFaieQRWctWLBAAAAEAGf2GpCvwIVqJ45wPt9YsAAAAATQZvdSahBbJlMCHf//qmWAACVgQAAAAxBn/tFFSwv/wAAsoAAAAAQAZ4adEK/AhWonkEVnLViwQAAABABnhxqQr8CFaieOcD7fWLBAAAAHkGaAUmoQWyZTAh3//6plgM3xCAf38rKnRAtxfFIeAAAABBBnj9FFSwv/wGVn6xCyYwIAAAADwGeXnRCvwIek0dVnf1iwQAAABABnkBqQr8CHkxXTfSQaR0wAAAAE0GaRUmoQWyZTAh3//6plgAAlYEAAAAMQZ5jRRUsL/8AALKAAAAAEAGegnRCvwIzZV3VGO71YEEAAAAQAZ6EakK/AjKVsXqDDjyhgQAAABNBmolJqEFsmUwId//+qZYAAJWBAAAADEGep0UVLC//AACygQAAABABnsZ0Qr8CM2Vd1Rju9WBAAAAAEAGeyGpCvwIylbF6gw48oYAAAAAcQZrNSahBbJlMCG///qeEBib4Uaza0vRP8CBTQQAAABBBnutFFSwv/wGVQFfyJjAgAAAAEAGfCnRCvwIzZV3IbBpgMqAAAAAPAZ8MakK/Ah9iusr+9WLBAAAAEkGbEUmoQWyZTAhv//6nhAABJwAAAAxBny9FFSwv/wAAsoEAAAAQAZ9OdEK/AhWonkEVnLViwAAAABABn1BqQr8CFaieOcD7fWLAAAAAGkGbUkmoQWyZTAhv//6nhAZXfZ80AUJA2OOBAAAAGEGbc0nhClJlMCHf/qmWAPDi8xJ0f1sJWwAAABJBm5dJ4Q6JlMCHf/6plgAAlYAAAAAMQZ+1RRE8L/8AALKBAAAAEAGf1HRCvwIL0A5/FgceVUAAAAAQAZ/WakK/AgsbXdWrnoaVgQAAABNBm9tJqEFomUwId//+qZYAAJWBAAAADEGf+UURLC//AACygAAAABABnhh0Qr8CC9AOfxYHHlVBAAAAEAGeGmpCvwILG13Vq56GlYAAAAATQZofSahBbJlMCHf//qmWAACVgQAAAAxBnj1FFSwv/wAAsoEAAAAQAZ5cdEK/AgvQDn8WBx5VQAAAABABnl5qQr8CCxtd1auehpWAAAAAE0GaQ0moQWyZTAh3//6plgAAlYEAAAAMQZ5hRRUsL/8AALKAAAAAEAGegHRCvwIL0A5/FgceVUEAAAAQAZ6CakK/AgsbXdWrnoaVgAAAABNBmodJqEFsmUwId//+qZYAAJWBAAAADEGepUUVLC//AACygQAAABABnsR0Qr8CC9AOfxYHHlVBAAAAEAGexmpCvwILG13Vq56GlYEAAAATQZrLSahBbJlMCHf//qmWAACVgAAAAAxBnulFFSwv/wAAsoAAAAAQAZ8IdEK/AgvQDn8WBx5VQQAAABABnwpqQr8CCxtd1auehpWAAAAAE0GbD0moQWyZTAh3//6plgAAlYAAAAAMQZ8tRRUsL/8AALKBAAAAEAGfTHRCvwIL0A5/FgceVUEAAAAQAZ9OakK/AgsbXdWrnoaVgQAAABNBm1NJqEFsmUwId//+qZYAAJWAAAAADEGfcUUVLC//AACygAAAABABn5B0Qr8CC9AOfxYHHlVBAAAAEAGfkmpCvwILG13Vq56GlYAAAAATQZuXSahBbJlMCHf//qmWAACVgAAAAAxBn7VFFSwv/wAAsoEAAAAQAZ/UdEK/AgvQDn8WBx5VQAAAABABn9ZqQr8CCxtd1auehpWBAAAAE0Gb20moQWyZTAh3//6plgAAlYEAAAAMQZ/5RRUsL/8AALKAAAAAEAGeGHRCvwIL0A5/FgceVUEAAAAQAZ4aakK/AgsbXdWrnoaVgAAAABxBmh9JqEFsmUwId//+qZYDICoFok2e+9GPSQm5AAAAEEGePUUVLC//AZWfpXBBFJEAAAAPAZ5cdEK/AgvQDoR8tlVAAAAADwGeXmpCvwIe7TJw2bTZKwAAABNBmkNJqEFsmUwId//+qZYAAJWBAAAADEGeYUUVLC//AACygAAAABABnoB0Qr8CFaieQRWctWLBAAAAEAGegmpCvwIVqJ45wPt9YsAAAAATQZqHSahBbJlMCHf//qmWAACVgQAAAAxBnqVFFSwv/wAAsoEAAAAQAZ7EdEK/AhWonkEVnLViwQAAABABnsZqQr8CFaieOcD7fWLBAAAAJEGay0moQWyZTAhv//6nhAZXfZ9AI3BNfiEA//4SpY8//2EZkAAAABBBnulFFSwv/wGVn5XlDoooAAAADwGfCHRCvwIek0dVnf1iwQAAAA8BnwpqQr8CHks5V3/38gsAAAAaQZsMSahBbJlMCHf//qmWAO40hJtro59RJeAAAAASQZswSeEKUmUwId/+qZYAAJWBAAAADEGfTkU0TC//AACygQAAABABn210Qr8CH2Vd1Rju9WLBAAAAEAGfb2pCvwIelbF6gw48poAAAAATQZt0SahBaJlMCHf//qmWAACVgAAAAAxBn5JFESwv/wAAsoEAAAAQAZ+xdEK/Ah9lXdUY7vViwAAAABABn7NqQr8CHpWxeoMOPKaAAAAAE0GbuEmoQWyZTAh3//6plgAAlYEAAAAMQZ/WRRUsL/8AALKAAAAAEAGf9XRCvwIfZV3VGO71YsEAAAAQAZ/3akK/Ah6VsXqDDjymgQAAABNBm/xJqEFsmUwId//+qZYAAJWAAAAADEGeGkUVLC//AACygQAAABABnjl0Qr8CH2Vd1Rju9WLAAAAAEAGeO2pCvwIelbF6gw48poEAAAATQZogSahBbJlMCHf//qmWAACVgQAAAAxBnl5FFSwv/wAAsoAAAAAQAZ59dEK/Ah9lXdUY7vViwAAAABABnn9qQr8CHpWxeoMOPKaBAAAAE0GaZEmoQWyZTAh3//6plgAAlYAAAAAMQZ6CRRUsL/8AALKBAAAAEAGeoXRCvwIfZV3VGO71YsAAAAAQAZ6jakK/Ah6VsXqDDjymgQAAABNBmqhJqEFsmUwId//+qZYAAJWBAAAADEGexkUVLC//AACygQAAABABnuV0Qr8CH2Vd1Rju9WLBAAAAEAGe52pCvwIelbF6gw48poAAAAATQZrsSahBbJlMCHf//qmWAACVgAAAAAxBnwpFFSwv/wAAsoEAAAAPAZ8pdEK/AVb/5LUB+We/AAAAEAGfK2pCvwIelbF6gw48poAAAAATQZswSahBbJlMCHf//qmWAACVgQAAAAxBn05FFSwv/wAAsoEAAAAQAZ9tdEK/Ah9lXdUY7vViwQAAABABn29qQr8CHpWxeoMOPKaAAAAAHEGbdEmoQWyZTAh3//6plgN+jogWZ8Z3tL7FD/AAAAAQQZ+SRRUsL/8Bo0A5usRQQQAAAA8Bn7F0Qr8CMyVpg2S7skYAAAAPAZ+zakK/AjNiusr+9WBAAAAAE0GbuEmoQWyZTAh3//6plgAAlYEAAAAMQZ/WRRUsL/8AALKAAAAAEAGf9XRCvwIpqJ5BFZy1YEEAAAAQAZ/3akK/AimonjnA+31gQQAAABNBm/xJqEFsmUwId//+qZYAAJWAAAAADEGeGkUVLC//AACygQAAABABnjl0Qr8CKaieQRWctWBAAAAAEAGeO2pCvwIpqJ45wPt9YEEAAAASQZogSahBbJlMCG///qeEAAEnAAAADEGeXkUVLC//AACygAAAABABnn10Qr8CKaieQRWctWBAAAAAEAGef2pCvwIpqJ45wPt9YEEAAAASQZpkSahBbJlMCG///qeEAAEnAAAADEGegkUVLC//AACygQAAABABnqF0Qr8CKaieQRWctWBAAAAAEAGeo2pCvwIpqJ45wPt9YEEAAAASQZqoSahBbJlMCF///oywAASNAAAADEGexkUVLC//AACygQAAABABnuV0Qr8CKaieQRWctWBBAAAAEAGe52pCvwIpqJ45wPt9YEAAAAAaQZrpS6hCEFskRggoB/IB/YeAIV/+OEAAEXAAAAyBbW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAAH5AAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAC6t0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAAH5AAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAARAAAAEQAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAB+QAAAEAAABAAAAAAsjbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAAAAyAAABlABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAAKzm1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAACo5zdGJsAAAAlnN0c2QAAAAAAAAAAQAAAIZhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAARABEABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAMGF2Y0MB9AAN/+EAF2f0AA2RmygiEdCAAAADAIAAABkHihTLAQAGaOvjxEhEAAAAGHN0dHMAAAAAAAAAAQAAAMoAAAIAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAZYY3R0cwAAAAAAAADJAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAE5wAAACoAAAAoAAAAFAAAABQAAAAbAAAAEgAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACIAAAAVAAAAFAAAABMAAAAdAAAAHgAAABQAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAiAAAAFAAAABMAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAgAAAAFAAAABQAAAATAAAAFgAAABAAAAAUAAAAFAAAAB4AAAAcAAAAFgAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACAAAAAUAAAAEwAAABMAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACgAAAAUAAAAEwAAABMAAAAeAAAAFgAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAEwAAABQAAAAXAAAAEAAAABQAAAAUAAAAIAAAABQAAAATAAAAEwAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAHgAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1Ni4zNi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(display_videos('0.6cnn_test9.mp4'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFT9tZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAAErZYiEADv//vb8/AptUwndP/9P+VxN2UIE1zAALTEAIgTYlI7FKYvwKUU8/ApmttTF/93H/46/3o/WPwJzUHo7OxkoIR705cr2BBSJzjOJLMGOvoe40n/zR6yebxa6RSslptDUX0CA4GTXkP7z4IVoBHUDaGX7+rNyS+ZuGtmViiuF/M+TVNxPMfFwaE3dpfsLswnDTjfmfrr3AnzSnZ/hNqPCIYZ9xcNA5JGHTsAewWeiOsY0j5FoodwNL/wajX+3CRKcVKqaEcqiT7YwAARlUg4TCSrcHYh/ZE1wBP3iACtxag5oDhdj1tjlFjWUrYtq8nA8EApYL+7q7VgwS5abmkTPrITETCOw+cwxOqc+staj0kaXlvokHfAasVn2pVjNpuxLaiE4FlWABa0AAABNQZokbEO//qmWADQew3zLK1TVeBSiQLwKZrlfroH//EC9jq6aXBBiIK85//4hAyp1dyRuCSV1CAyU//8PG1q2gwKAPoNFqe6zu/Y8rzAAAAAkQZ5CeIX/ADzNSuIUI3hKrP//EHQZLP/8NPyz//EDHbRub2rhAAAAEAGeYXRCvwAivpG+v7TX/6AAAAAQAZ5jakK/AFQseOV/bh99wQAAAEFBmmhJqEFomUwId//+qZYAUEo856Uo//iCEL1dVnghowGD2//8QhkI6u6MXBJmwhBj+f/+H1JtXEJgaioFVJ6PgQAAAA5BnoZFESwv/wBfhFs3oQAAABABnqV0Qr8AfyxWMKkbuEVRAAAADwGep2pCvwB/LcB0K2fqrAAAABNBmqxJqEFsmUwId//+qZYAAJWAAAAADEGeykUVLC//AACygQAAABABnul0Qr8AftrmD0EvDVVgAAAADwGe62pCvwB/LcB0K2fqrAAAABNBmvBJqEFsmUwId//+qZYAAJWBAAAADEGfDkUVLC//AACygQAAAA8Bny10Qr8Afw7A6Eqv9KkAAAAPAZ8vakK/AH8twHQrZ+qsAAAAaUGbNEmoQWyZTAh3//6plgIkDf//+ISPFTV3mh2CVX64iPD///xAuWaumWwQW/CuL//+IQMqdXckbgkldQgM1l71MtLqfstPgMcrKN4EhVkkSzHniPWM8Smcdw/20sxP4c7V/bN5y8GG4AAAACZBn1JFFSwv/wFlVaOP75P+IRZ//4g6DJZ//hp+Wf/4gY7cUy00vQAAABABn3F0Qr8AyMle+v7TX0bgAAAAEAGfc2pCvwHfH3y1/bZ8v8AAAAATQZt4SahBbJlMCHf//qmWAACVgQAAAAxBn5ZFFSwv/wAAsoAAAAAQAZ+1dEK/Ad9pWMJYchB3EQAAABABn7dqQr8B3u0OhFjkwyhhAAAAE0GbvEmoQWyZTAh3//6plgAAlYAAAAAMQZ/aRRUsL/8AALKBAAAAEAGf+XRCvwHfaVjCWHIQdxAAAAAQAZ/7akK/Ad7tDoRY5MMoYQAAABNBm+BJqEFsmUwId//+qZYAAJWBAAAADEGeHkUVLC//AACygAAAABABnj10Qr8B32lYwlhyEHcQAAAAEAGeP2pCvwHe7Q6EWOTDKGEAAAAoQZokSahBbJlMCHf//qmWAiT0jM7qVT/W3Y3P4hImj//CVUM3/++KqgAAABBBnkJFFSwv/wFlVNDRLpsbAAAADwGeYXRCvwHe58NgflZQwAAAABABnmNqQr8B3rL9UfMfi2NBAAAAG0GaaEmoQWyZTAh3//6plgHV4hAP7+ZCp6pMCQAAABVBnoZFFSwv/wFRTZ+ZtxC3P3rnk48AAAAQAZ6ldEK/AcaMyI7FmKNOOQAAABABnqdqQr8BxgiZpvpIOJFwAAAAE0GarEmoQWyZTAh3//6plgAAlYAAAAAMQZ7KRRUsL/8AALKBAAAAEAGe6XRCvwHSaVi8/gcjXkAAAAAQAZ7rakK/AdHtDn+Zbv1vQAAAABNBmvBJqEFsmUwId//+qZYAAJWBAAAADEGfDkUVLC//AACygQAAABABny10Qr8B0mlYvP4HI15BAAAAEAGfL2pCvwHR7Q5/mW79b0AAAAATQZs0SahBbJlMCHf//qmWAACVgAAAAAxBn1JFFSwv/wAAsoEAAAAQAZ9xdEK/AdJpWLz+ByNeQAAAABABn3NqQr8B0e0Of5lu/W9AAAAAE0GbeEmoQWyZTAh3//6plgAAlYEAAAAMQZ+WRRUsL/8AALKAAAAAEAGftXRCvwHSaVi8/gcjXkEAAAAQAZ+3akK/AdHtDn+Zbv1vQQAAABJBm7xJqEFsmUwIb//+p4QAAScAAAAMQZ/aRRUsL/8AALKBAAAAEAGf+XRCvwHSaVi8/gcjXkAAAAAQAZ/7akK/AdHtDn+Zbv1vQQAAAChBm+BJqEFsmUwIZ//+nhAPvwgcvMsrnvHzLEsF8yybBu37buU/3mn/AAAAEEGeHkUVLC//AWVV1vBAXHAAAAAPAZ49dEK/AdJpWMIVadNAAAAAEAGeP2pCvwHfH398aGr8poEAAAAaQZohSahBbJlMCGf//p4QELU4/oHp/6dtIuAAAAAYQZpCSeEKUmUwIZ/+nhAQtTj+u5EflhoHAAAAGUGaY0nhDomUwIb//qeEAaHup+l8UJDCh4AAAAAZQZqESeEPJlMCG//+p4QA+HsH+E4LdCRbQQAAABlBmqVJ4Q8mUwId//6plgBS/fV9diDcU/3RAAAAFkGayUnhDyZTAh3//qmWACI/Rz8khcEAAAAOQZ7nRRE8L/8AKOyoNmEAAAAQAZ8GdEK/AFVso78AH27ewAAAABABnwhqQr8AVWyjvZ4+3b2AAAAAE0GbDUmoQWiZTAh3//6plgAAlYEAAAAMQZ8rRREsL/8AALKAAAAAEAGfSnRCvwBVbKO/AB9u3sAAAAAQAZ9MakK/AFVso72ePt29gQAAABNBm1FJqEFsmUwId//+qZYAAJWBAAAADEGfb0UVLC//AACygQAAABABn450Qr8AVWyjvwAfbt7AAAAAEAGfkGpCvwBVbKO9nj7dvYAAAAATQZuVSahBbJlMCHf//qmWAACVgQAAAAxBn7NFFSwv/wAAsoAAAAAQAZ/SdEK/AFVso78AH27ewAAAABABn9RqQr8AVWyjvZ4+3b2BAAAAE0Gb2UmoQWyZTAh3//6plgAAlYAAAAAMQZ/3RRUsL/8AALKBAAAAEAGeFnRCvwBVbKO/AB9u3sEAAAAQAZ4YakK/AFVso72ePt29gAAAABNBmh1JqEFsmUwId//+qZYAAJWBAAAADEGeO0UVLC//AACygAAAABABnlp0Qr8AVWyjvwAfbt7BAAAAEAGeXGpCvwBVbKO9nj7dvYEAAAATQZpBSahBbJlMCHf//qmWAACVgAAAAAxBnn9FFSwv/wAAsoAAAAAQAZ6edEK/AFVso78AH27ewQAAABABnoBqQr8AVWyjvZ4+3b2AAAAAE0GahUmoQWyZTAh3//6plgAAlYEAAAAMQZ6jRRUsL/8AALKAAAAAEAGewnRCvwBVbKO/AB9u3sEAAAAQAZ7EakK/AFVso72ePt29gQAAABJBmslJqEFsmUwIb//+p4QAAScAAAAMQZ7nRRUsL/8AALKBAAAAEAGfBnRCvwBVbKO/AB9u3sAAAAAQAZ8IakK/AFVso72ePt29gAAAABJBmw1JqEFsmUwIZ//+nhAABH0AAAAMQZ8rRRUsL/8AALKAAAAAEAGfSnRCvwBVbKO/AB9u3sAAAAAQAZ9MakK/AILa13eST7KowQAAABpBm05JqEFsmUwIb//+p4QAaf2D/CcFuhJswQAAABhBm29J4QpSZTAhv/6nhABDvjpj/D6tt3EAAAAZQZuQSeEOiZTAh3/+qZYAM9UgzQB6S+wIEAAAABJBm7RJ4Q8mUwId//6plgAAlYAAAAAMQZ/SRRE8L/8AALKBAAAAEAGf8XRCvwB/LFYvP4HJFUAAAAAQAZ/zakK/AH8NQ5/mW7+lQAAAABJBm/hJqEFomUwIb//+p4QAAScAAAAMQZ4WRREsL/8AALKAAAAAEAGeNXRCvwB/LFYvP4HJFUEAAAAQAZ43akK/AH8NQ5/mW7+lQQAAABpBmjlJqEFsmUwId//+qZYAUD5BmgD0l9f+kAAAABJBml1J4QpSZTAh3/6plgAAlYEAAAAMQZ57RTRML/8AALKAAAAAEAGemnRCvwDI2Vd1fju/f8EAAAAQAZ6cakK/AH3UN7FaPt17QQAAABNBmoFJqEFomUwId//+qZYAAJWAAAAADEGev0URLC//AACygAAAABABnt50Qr8AfdQ3suq/gRVBAAAAEAGewGpCvwB91DexWj7de0AAAAATQZrFSahBbJlMCHf//qmWAACVgQAAAAxBnuNFFSwv/wAAsoAAAAAQAZ8CdEK/AH3UN7Lqv4EVQQAAABABnwRqQr8AfdQ3sVo+3XtBAAAAE0GbCUmoQWyZTAh3//6plgAAlYEAAAAMQZ8nRRUsL/8AALKBAAAAEAGfRnRCvwB91Dey6r+BFUAAAAAQAZ9IakK/AH3UN7FaPt17QAAAABNBm01JqEFsmUwId//+qZYAAJWBAAAADEGfa0UVLC//AACygAAAABABn4p0Qr8AfdQ3suq/gRVAAAAAEAGfjGpCvwB91DexWj7de0EAAAAfQZuRSahBbJlMCHf//qmWAFC99X3e3y4eBTaW7TKe4QAAABBBn69FFSwv/wBflXPRQ6ppAAAADwGfznRCvwB/C/FwH5aqwAAAABABn9BqQr8AfwIBOvAE/qmAAAAAJ0Gb1UmoQWyZTAh3//6plgA1Xv1cyytU1XgUokC8Cma5Y/3my8XC4QAAABBBn/NFFSwv/wA+KdO/zeX4AAAADwGeEnRCvwA2CTU9Wd+KwAAAABABnhRqQr8AVmwjyYHr2+6BAAAAE0GaGUmoQWyZTAh3//6plgAAlYAAAAAMQZ43RRUsL/8AALKBAAAAEAGeVnRCvwBVbKO/AB9u3sEAAAAQAZ5YakK/AFVso72ePt29gAAAABJBml1JqEFsmUwIb//+p4QAAScAAAAMQZ57RRUsL/8AALKAAAAAEAGemnRCvwBVbKO/AB9u3sEAAAAQAZ6cakK/AFVso72ePt29gQAAAB5BmoFJqEFsmUwIZ//+nhACaiHOmwXojsfAUz+MVtAAAAAQQZ6/RRUsL/8AX4RxncoaYAAAABABnt50Qr8AfyMyI7FmKNypAAAADwGewGpCvwB/LAlyv7+lQAAAABlBmsJJqEFsmUwIb//+p4QAouK0ghE/y20zAAAAGEGa40nhClJlMCG//qeEAPccZ/qUgFS+YAAAABhBmwRJ4Q6JlMCHf/6plgDR+QZn24QW1JEAAAASQZsoSeEPJlMCHf/+qZYAAJWBAAAADEGfRkURPC//AACygQAAABABn2V0Qr8B7GlYwlhu4HZxAAAAEAGfZ2pCvwHr7Q6EWN9jJ2AAAAATQZtsSahBaJlMCHf//qmWAACVgAAAAAxBn4pFESwv/wAAsoEAAAAQAZ+pdEK/AexpWMJYbuB2cAAAABABn6tqQr8B6+0OhFjfYydgAAAAE0GbsEmoQWyZTAh3//6plgAAlYEAAAAMQZ/ORRUsL/8AALKBAAAAEAGf7XRCvwHsaVjCWG7gdnEAAAAQAZ/vakK/AevtDoRY32MnYAAAABNBm/RJqEFsmUwId//+qZYAAJWAAAAADEGeEkUVLC//AACygQAAABABnjF0Qr8B7GlYwlhu4HZwAAAAEAGeM2pCvwHr7Q6EWN9jJ2AAAAATQZo4SahBbJlMCHf//qmWAACVgQAAAAxBnlZFFSwv/wAAsoAAAAAQAZ51dEK/AexpWMJYbuB2cQAAABABnndqQr8B6+0OhFjfYydhAAAAE0GafEmoQWyZTAh3//6plgAAlYAAAAAMQZ6aRRUsL/8AALKBAAAAEAGeuXRCvwHsaVjCWG7gdnAAAAAQAZ67akK/AevtDoRY32MnYQAAABJBmqBJqEFsmUwIb//+p4QAAScAAAAMQZ7eRRUsL/8AALKAAAAAEAGe/XRCvwHsaVjCWG7gdnAAAAAQAZ7/akK/AevtDoRY32MnYQAAABJBmuRJqEFsmUwIb//+p4QAAScAAAAMQZ8CRRUsL/8AALKBAAAAEAGfIXRCvwHsaVjCWG7gdnAAAAAQAZ8jakK/AevtDoRY32MnYQAAABJBmyhJqEFsmUwIX//+jLAABI0AAAAMQZ9GRRUsL/8AALKBAAAAEAGfZXRCvwHsaVjCWG7gdnEAAAAQAZ9nakK/AevtDoRY32MnYAAAABpBm2lLqEIQWyRGCCgH8gH9h4AhX/44QAARcAAADEltb292AAAAbG12aGQAAAAAAAAAAAAAAAAAAAPoAAAfkAABAAABAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAALc3RyYWsAAABcdGtoZAAAAAMAAAAAAAAAAAAAAAEAAAAAAAAfkAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAABEAAAARAAAAAAACRlZHRzAAAAHGVsc3QAAAAAAAAAAQAAH5AAAAQAAAEAAAAACuttZGlhAAAAIG1kaGQAAAAAAAAAAAAAAAAAADIAAAGUAFXEAAAAAAAtaGRscgAAAAAAAAAAdmlkZQAAAAAAAAAAAAAAAFZpZGVvSGFuZGxlcgAAAAqWbWluZgAAABR2bWhkAAAAAQAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAKVnN0YmwAAACWc3RzZAAAAAAAAAABAAAAhmF2YzEAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAABEAEQAEgAAABIAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAY//8AAAAwYXZjQwH0AA3/4QAXZ/QADZGbKCIR0IAAAAMAgAAAGQeKFMsBAAZo6+PESEQAAAAYc3R0cwAAAAAAAAABAAAAygAAAgAAAAAUc3RzcwAAAAAAAAABAAAAAQAABiBjdHRzAAAAAAAAAMIAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAABQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAMAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAADAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAD4AAAAFEAAAAoAAAAFAAAABQAAABFAAAAEgAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAEwAAABMAAABtAAAAKgAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAALAAAABQAAAATAAAAFAAAAB8AAAAZAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAACwAAAAUAAAAEwAAABQAAAAeAAAAHAAAAB0AAAAdAAAAHQAAABoAAAASAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAeAAAAHAAAAB0AAAAWAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAAB4AAAAWAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACMAAAAUAAAAEwAAABQAAAArAAAAFAAAABMAAAAUAAAAFwAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAiAAAAFAAAABQAAAATAAAAHQAAABwAAAAcAAAAFgAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAHgAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1Ni4zNi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(display_videos('0.9cnn_test9.mp4'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:**\n",
    "\n",
    "When the temperature increases, the number of cheeses and poisons increases. Moreover, when the temperature starts to be high (above 0.5), the number of cheese increases and there is just few remaining empty cells to put poison so the number of poison decrease even if the probability in an empty cell to have a poison increases.\n",
    "\n",
    "What we can say about the **agent's behaviour**. The agent always tries to find an area with empty cells (to stay in). When the temperature is low (there are few cheeses and poisons), the agent will quickly find this kind of area. When the temperature increases there are more cheeses and poisons. The agents should accept to take some risk and go in cheese and maybe poison cells. But when the agent did some \"clean up\" of the board (remove some cheese/poison), then the agent starts to oscillate in a small group of empty cells that he created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "The algorithm tends to not explore the map which can be an issue. We propose two ideas in order to encourage exploration:\n",
    "1. Incorporating a decreasing $\\epsilon$-greedy exploration. You can use the method ```set_epsilon```\n",
    "2. Append via the environment a new state that describes if a cell has been visited or not\n",
    "\n",
    "***\n",
    "__Question 10__ Design a new ```train_explore``` function and environment class ```EnvironmentExploring``` to tackle the issue of exploration.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_explore(agent,env,epoch,prefix=''):\n",
    "    # Number of won games\n",
    "    score = 0\n",
    "    loss = 0\n",
    "    \n",
    "    epsilon0 = agent.epsilon\n",
    "    gamma = 0.9 # parameter to decrease epsilon\n",
    "\n",
    "    for e in range(epoch):\n",
    "        \n",
    "        # At each epoch we decrease epsilon\n",
    "        agent.set_epsilon(agent.epsilon*gamma)\n",
    "        \n",
    "        # At each epoch, we restart to a fresh game and get the initial state\n",
    "        state = env.reset()\n",
    "        # This assumes that the games will terminate\n",
    "        game_over = False\n",
    "\n",
    "        win = 0\n",
    "        lose = 0\n",
    "\n",
    "        while not game_over:\n",
    "            # The agent performs an action\n",
    "            action = agent.act(state)\n",
    "\n",
    "            # Apply an action to the environment, get the next state, the reward\n",
    "            # and if the games end\n",
    "            prev_state = state\n",
    "            state, reward, game_over = env.act(action, train=True)\n",
    "\n",
    "            # Update the counters\n",
    "            if reward > 0:\n",
    "                win = win + reward\n",
    "            if reward < 0:\n",
    "                lose = lose -reward\n",
    "\n",
    "            # Apply the reinforcement strategy\n",
    "            loss = agent.reinforce(prev_state, state,  action, reward, game_over)\n",
    "\n",
    "        # Save as a mp4\n",
    "        if e % 10 == 0:\n",
    "            env.draw(prefix+str(e))\n",
    "\n",
    "        # Update stats\n",
    "        score += win-lose\n",
    "        \n",
    "\n",
    "        print(\"Epoch {:03d}/{:03d} | Loss {:.4f} | Win/lose count {}/{} ({})\"\n",
    "              .format(e, epoch, loss, win, lose, win-lose))\n",
    "        agent.save(name_weights=prefix+'model.h5',name_model=prefix+'model.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EnvironmentExploring(object):\n",
    "    def __init__(self, grid_size=10, max_time=500, temperature=0.1):\n",
    "        grid_size = grid_size+4\n",
    "        self.grid_size = grid_size\n",
    "        self.max_time = max_time\n",
    "        self.temperature = temperature\n",
    "\n",
    "        #board on which one plays\n",
    "        self.board = np.zeros((grid_size,grid_size))\n",
    "        self.position = np.zeros((grid_size,grid_size))\n",
    "        # new state to foster exploration\n",
    "        self.malus_position = np.zeros((grid_size,grid_size))\n",
    "    \n",
    "        # coordinate of the cat\n",
    "        self.x = 0\n",
    "        self.y = 1\n",
    "\n",
    "        # self time\n",
    "        self.t = 0\n",
    "\n",
    "        self.scale=16\n",
    "\n",
    "        self.to_draw = np.zeros((max_time+2, grid_size*self.scale, grid_size*self.scale, 3))\n",
    "\n",
    "\n",
    "    def draw(self,e):\n",
    "        skvideo.io.vwrite(str(e) + '.mp4', self.to_draw)\n",
    "\n",
    "    def get_frame(self,t):\n",
    "        b = np.zeros((self.grid_size,self.grid_size,3))+128\n",
    "        b[self.board>0,0] = 256\n",
    "        b[self.board < 0, 2] = 256\n",
    "        b[self.x,self.y,:]=256\n",
    "        b[-2:,:,:]=0\n",
    "        b[:,-2:,:]=0\n",
    "        b[:2,:,:]=0\n",
    "        b[:,:2,:]=0\n",
    "        \n",
    "        b =  cv2.resize(b, None, fx=self.scale, fy=self.scale, interpolation=cv2.INTER_NEAREST)\n",
    "\n",
    "        self.to_draw[t,:,:,:]=b\n",
    "\n",
    "\n",
    "    def act(self, action, train= False):\n",
    "        \"\"\"This function returns the new state, reward and decides if the\n",
    "        game ends.\"\"\"\n",
    "\n",
    "        self.get_frame(int(self.t))\n",
    "\n",
    "        self.position = np.zeros((self.grid_size, self.grid_size))\n",
    "\n",
    "        self.position[0:2,:]= -1\n",
    "        self.position[:,0:2] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "\n",
    "        self.position[self.x, self.y] = 1\n",
    "        if action == 0:\n",
    "            if self.x == self.grid_size-3:\n",
    "                self.x = self.x-1\n",
    "            else:\n",
    "                self.x = self.x + 1\n",
    "        elif action == 1:\n",
    "            if self.x == 2:\n",
    "                self.x = self.x+1\n",
    "            else:\n",
    "                self.x = self.x-1\n",
    "        elif action == 2:\n",
    "            if self.y == self.grid_size - 3:\n",
    "                self.y = self.y - 1\n",
    "            else:\n",
    "                self.y = self.y + 1\n",
    "        elif action == 3:\n",
    "            if self.y == 2:\n",
    "                self.y = self.y + 1\n",
    "            else:\n",
    "                self.y = self.y - 1\n",
    "        else:\n",
    "            RuntimeError('Error: action not recognized')\n",
    "            \n",
    "            \n",
    "        reward = 0\n",
    "        if train:\n",
    "            # We add a malus for already visited positions\n",
    "            reward = -self.malus_position[self.x, self.y]\n",
    "        self.malus_position[self.x, self.y] += 0.05\n",
    "\n",
    "        \n",
    "        self.t = self.t + 1\n",
    "        reward += self.board[self.x, self.y]\n",
    "        self.board[self.x, self.y] = 0\n",
    "        game_over = self.t > self.max_time\n",
    "        # 3 \"feature\" states instead of 2\n",
    "        state = np.concatenate((self.malus_position.reshape(self.grid_size, self.grid_size,1),\n",
    "                                self.board.reshape(self.grid_size, self.grid_size,1),\n",
    "                        self.position.reshape(self.grid_size, self.grid_size,1)),axis=2)\n",
    "        state = state[self.x-2:self.x+3,self.y-2:self.y+3,:]\n",
    "\n",
    "        return state, reward, game_over\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"This function resets the game and returns the initial state\"\"\"\n",
    "\n",
    "        self.x = np.random.randint(3, self.grid_size-3, size=1)[0]\n",
    "        self.y = np.random.randint(3, self.grid_size-3, size=1)[0]\n",
    "\n",
    "\n",
    "        bonus = 0.5*np.random.binomial(1,self.temperature,size=self.grid_size**2)\n",
    "        bonus = bonus.reshape(self.grid_size,self.grid_size)\n",
    "\n",
    "        malus = -1.0*np.random.binomial(1,self.temperature,size=self.grid_size**2)\n",
    "        malus = malus.reshape(self.grid_size, self.grid_size)\n",
    "\n",
    "        self.to_draw = np.zeros((self.max_time+2, self.grid_size*self.scale, self.grid_size*self.scale, 3))\n",
    "\n",
    "\n",
    "        malus[bonus>0]=0\n",
    "\n",
    "        self.board = bonus + malus\n",
    "\n",
    "        self.position = np.zeros((self.grid_size, self.grid_size))\n",
    "        self.position[0:2,:]= -1\n",
    "        self.position[:,0:2] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.board[self.x,self.y] = 0\n",
    "        self.t = 0\n",
    "        \n",
    "        # reset malus_position\n",
    "        self.malus_position = np.zeros((self.grid_size, self.grid_size))\n",
    "\n",
    "        state = np.concatenate((self.malus_position.reshape(self.grid_size, self.grid_size,1),\n",
    "                                self.board.reshape(self.grid_size, self.grid_size,1),\n",
    "                        self.position.reshape(self.grid_size, self.grid_size,1)),axis=2)\n",
    "\n",
    "        state = state[self.x - 2:self.x + 3, self.y - 2:self.y + 3, :]\n",
    "        return state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 000/020 | Loss 0.0227 | Win/lose count 11.5/28.25000000000004 (-16.75000000000004)\n",
      "Epoch 001/020 | Loss 0.0118 | Win/lose count 16.0/36.75 (-20.75)\n",
      "Epoch 002/020 | Loss 0.0284 | Win/lose count 12.0/38.94999999999998 (-26.94999999999998)\n",
      "Epoch 003/020 | Loss 0.0315 | Win/lose count 9.5/56.95000000000001 (-47.45000000000001)\n",
      "Epoch 004/020 | Loss 0.0350 | Win/lose count 10.5/34.20000000000001 (-23.70000000000001)\n",
      "Epoch 005/020 | Loss 0.0317 | Win/lose count 11.0/28.950000000000014 (-17.950000000000014)\n",
      "Epoch 006/020 | Loss 0.0330 | Win/lose count 14.5/40.299999999999976 (-25.799999999999976)\n",
      "Epoch 007/020 | Loss 0.0505 | Win/lose count 17.5/28.400000000000016 (-10.900000000000016)\n",
      "Epoch 008/020 | Loss 0.0309 | Win/lose count 10.0/22.150000000000016 (-12.150000000000016)\n",
      "Epoch 009/020 | Loss 0.0317 | Win/lose count 15.5/19.600000000000012 (-4.100000000000012)\n",
      "Epoch 010/020 | Loss 0.0473 | Win/lose count 21.0/20.200000000000014 (0.7999999999999865)\n",
      "Epoch 011/020 | Loss 0.0494 | Win/lose count 22.5/20.200000000000024 (2.299999999999976)\n",
      "Epoch 012/020 | Loss 0.0368 | Win/lose count 21.5/17.150000000000013 (4.349999999999987)\n",
      "Epoch 013/020 | Loss 0.0503 | Win/lose count 16.0/18.150000000000023 (-2.1500000000000234)\n",
      "Epoch 014/020 | Loss 0.0290 | Win/lose count 19.0/25.850000000000026 (-6.850000000000026)\n",
      "Epoch 015/020 | Loss 0.0529 | Win/lose count 22.0/19.55000000000002 (2.4499999999999815)\n",
      "Epoch 016/020 | Loss 0.0243 | Win/lose count 25.5/20.000000000000014 (5.499999999999986)\n",
      "Epoch 017/020 | Loss 0.0211 | Win/lose count 19.0/16.75000000000001 (2.2499999999999893)\n",
      "Epoch 018/020 | Loss 0.0692 | Win/lose count 19.0/13.800000000000022 (5.199999999999978)\n",
      "Epoch 019/020 | Loss 0.0304 | Win/lose count 17.0/20.45 (-3.4499999999999993)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAGittZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAALqZYiEADf//vaH+BTZWBO6f/0/4LrOnzl/nbABNNgBSG5gp/w1NF8Cktv/wKZbHk60DzAM9KJLpsWn+In7/HRperS05BY1+Bv/ebCfHSWTIgnIMQEibcexiCBMoJ/gxKAogXzileDrPfxTqttwhBAl8msFby0vlu+N+pzobj7DaGLMe1/PxzgKDKlQ2q0X/OjfmCNE/NS+y+qozqPE6QlnuOYFlmwQS0cXJPwwqaskUH6R/FM6lDujwh4dHHyfsn21mRnRi/CYYESnRW5fwL/rtfmRLvyFz2t4jlZTrQGvSLBgw+YEdxxuwaD5nr43ucLCiUHil23yDlyTHWMxXImnVnzwRNaKAug+VBmM71C6SGVTQqH/6Vj4n/EacF1brZyNtFpYg/YmFcmx6n199MW92gugQuWR4Rw/VRkYJ9UA5rii7ntDu/I4PtUx8IQI3Xy4UXwFmDfKPXKXJOaAOhwPwWSl46mmvjDVpP1dsHIAFF7iM8UL8U2ZrwP0Z3BQgzuWXJ0bdLmNAD/O/VJwogqd4AjfDeVlQi91bcSnaSajSx24iMNF1Y+6OpxHXeCO0zQ+MsGKHaPZS7aLNZ2ZF+dgaqPus/qZigtJf/d4pXlDwjis6FqI7wznaZqquoQgqqevqnmB+tBk4pKN/Jqptb7ofBCzuhc2wNFRixFPJFnmA5FRnivq2gZQ1QVNy8eA1cgrjnQ725M/Edx7wdevx8EGPfdPtpILD5zvTHkQaT5QFBDvIKVj08LbZl/FTlgdt03mjymBVGWM1jrrb7he96WLrQqXJwGjB6vRAfC9u/VkiV+QqX2Xsv8Glgb8wDe8ln4vwgxidSLD7Zf4S1Z9eSaWoapM8DoQbM7TK/ZKkU5BILnECxJ/pjeXebRjYUTYUC9huw7XqbrnQMCq/+9edwxB0VuZcQnK7KVGknXt8T2shrLnRba7DDqUjG37b2N7rE6eypFxgTttxu2BfimeY3hnquqCcLRYAUrPwwsAAAATQZohbEN//qeEAKz7qcng+rbaRgAAACdBmkU8IZMphDf//qeEAPh6w3Msq2yn4FJaB/ApltHNutLxyvnCwcEAAAAQQZ5jalPC/wCW5+5wV1a+YAAAAA8BnoJ0Qr8AhtoQGSXKrYEAAAAQAZ6EakK/AM27cJuM+vTXzQAAABpBmoZJqEFomUwIb//+p4QBneif6obMfgKHgQAAACBBmqpJ4QpSZTAhn/6eEBC1OP7OOm+GrwQcyyvE6B+wIQAAABFBnshFNEwv/wFv/ZMYit9x8AAAAA8Bnud0Qr8B7GlYwclmTsAAAAAPAZ7pakK/AT9tulGkPEnrAAAAGUGa60moQWiZTAhv//6nhADy+wevZnwRXRcAAAAbQZsMSeEKUmUwIb/+p4QA7XwGAf3efBbdbXdAAAAAGUGbLUnhDomUwIb//qeEAOf7B/hOC3QkYsEAAAAgQZtRSeEPJlMCG//+p4QBj8CMmpthaCXuJ/i9j7Q6aykAAAASQZ9vRRE8L/8A4ibIab7dzjuVAAAADwGfjnRCvwDDyWbg2S8ZlwAAAA8Bn5BqQr8BNtiPJgevbQcAAAAaQZuSSahBaJlMCHf//qmWAiRMN0SzjnzObMEAAAAeQZu2SeEKUmUwIb/+p4QES7VQIT+dR+J7+Y9CHxVwAAAAEUGf1EU0TC//AWVUnBjXdHmYAAAADwGf83RCvwHe50BklqjCgQAAABABn/VqQr8B7A8GuPCmzq4gAAAAHEGb90moQWiZTAh3//6plgJWTDdEjJdA4f2NR8EAAAAcQZobSeEKUmUwIb/+p4QFC0T/R5j8euwPwWidgQAAABBBnjlFNEwv/wF6+87k7/vGAAAADgGeWHRCvwH5uO884fE3AAAAEAGeWmpCvwHrtwYfQEg0j5gAAAAnQZpfSahBaJlMCG///qeEAbLvDDZ/gSlV7L4hEf6f/wlVDN//soZhAAAAFUGefUURLC//AOz/FbSuVyItPyrzHwAAABABnpx0Qr8BSE6k8r8lNlQQAAAAEAGenmpCvwDXktp14An85oAAAAAZQZqASahBbJlMCG///qeEAG74TFHwYQa2LQAAABpBmqRJ4QpSZTAhv/6nhACsBUrNuM3up8W1sAAAABVBnsJFNEwv/wBnFaYwP0WLYLlIJbEAAAAQAZ7hdEK/AIL6iRPizFG4sAAAAA8BnuNqQr8AiuxHkuZ8kwMAAAAZQZrlSahBaJlMCG///qeEALBitIIRP8ttGwAAABhBmwZJ4QpSZTAh3/6plgBZQnQAYdPqP8EAAABIQZsqSeEOiZTAhv/+p4QAtfup+62NNf/8QkH03DbMr2EqOlJEHyP//iA7c4aqHwf8dE7H//8QfRfw2jIISME4/S0al6yv5GDhAAAAE0GfSEURPC//AGwhUbv8GlyWTBcAAAAPAZ9ndEK/AJL6TuDZLxoWAAAAEAGfaWpCvwCSyyGH0BIOLZkAAAAaQZtrSahBaJlMCHf//qmWAD0Dp+U0Y/WkwcAAAAAoQZuPSeEKUmUwId/+qZYAnPxPOZZYwqrwKZrip4FEnr+XYZ4o89hD4AAAABVBn61FNEwv/wC6UCCbm/XIR2oyGKkAAAAQAZ/MdEK/AJ9lqgdO1DVegQAAABABn85qQr8A+DPCHjQ1jJ2BAAAAF0Gb00moQWiZTAhv//6nhAEsQBZttxbQAAAADkGf8UURLC//ALWyoDFgAAAAEAGeEHRCvwD1rAMR2XZUu4EAAAAQAZ4SakK/AP55omXQdPLPmAAAACxBmhdJqEFsmUwIb//+p4QBNfo5+Ndvhq/EIB//wlRBZi//wkxH4v/9f97N6AAAABBBnjVFFSwv/wC6UBp06Uq5AAAADgGeVHRCvwD+3HeecWk3AAAAEAGeVmpCvwD4BAJ14An8zYEAAAAaQZpYSahBbJlMCG///qeEAH99g/wnBboSWUEAAABIQZp8SeEKUmUwIZ/+nhABSPdN9gLXnJceklf//H+aE0BLQ+FVUvL7/On1//+PnpoCTaMqYtf0ML//8fyXNASv9L1TTA35Zgb7AAAAFEGemkU0TC//ADJCKRd8bcpGDWIdAAAAEAGeuXRCvwBDXZjgPyf/yOAAAAAQAZ67akK/ABxQiZpvpIOZcQAAABtBmr1JqEFomUwIZ//+nhAAipwjn8Oy2UfhYsEAAAAYQZreSeEKUmUwIZ/+nhAAipwjmR90pG6AAAAAGEGa/0nhDomUwIZ//p4QANLIY5/DnN9adwAAABhBmwBJ4Q8mUwIb//6nhAA2PsHr2Z8EWEEAAAAZQZshSeEPJlMCG//+p4QANP77Mf4fVtvqgAAAAB1Bm0VJ4Q8mUwIb//6nhABNR81TWbc16OfPNNYG3QAAABFBn2NFETwv/wAulAitEdXT4AAAAA8Bn4J0Qr8AKhGEBklzC4EAAAAQAZ+EakK/AD4s+Y3Q5IOPzQAAABlBm4ZJqEFomUwIb//+p4QAT3FaQQif5bc7AAAAHEGbqknhClJlMCGf/p4QAc9No5/SOx8BTP4xd0EAAAAQQZ/IRTRML/8AR2gOXkUW4AAAABABn+d0Qr8AZKyruQ2VKRHgAAAADwGf6WpCvwBkgWNgcpukgQAAABlBm+tJqEFomUwIZ//+nhAB2fXG3vTfdbemAAAAGEGaDEnhClJlMCG//qeEAL36J/qUgFTUwAAAABlBmi1J4Q6JlMCG//6nhAEkQBZttn2fNE3BAAAAGUGaTknhDyZTAh3//qmWARQTDdFUhR8hK2EAAAAVQZpySeEPJlMCG//+p4QCSd1P1lkfAAAAFUGekEURPC//AdW7dM4q5ZntfOBwQAAAAA8Bnq90Qr8CdohNg2S7UhYAAAAQAZ6xakK/AnVl+qPiRc53QQAAABpBmrNJqEFomUwId//+qZYBF/HnSzo6juBswAAAABxBmtdJ4QpSZTAhv/6nhAJCAakI96Oe3nrbNKggAAAAEUGe9UU0TC//ARbQSBHtH/nxAAAADwGfFHRCvwFsjGLgPyz1YAAAABABnxZqQr8Bf3VPJcz5JMKBAAAAIUGbG0moQWiZTAhv//6nhAJJ3hj5/XFnWOnPgU2RgkydfQAAABVBnzlFESwv/wEWoDNU/6LFuAAR7PgAAAAPAZ9YdEK/AX8A+KTbJVEXAAAAEAGfWmpCvwDyq4NceKto7iAAAAAZQZtcSahBbJlMCG///qeEAMPwmZXILemmkwAAAB5Bm2BJ4QpSZTAhn/6eEATX4t23Mss+fbh2vPdjC7kAAAAVQZ+eRTRML/8AvtAgm5v1yG+2ZtjgAAAADwGfvXRCvwCj5k7g2S8Z4QAAAA8Bn79qQr8A/tnluGzamTMAAAAcQZuhSahBaJlMCG///qeEAT35HAJr17M+B/5rQAAAABtBm8JJ4QpSZTAhv/6nhAJh0T/UigMAmv7BN6EAAAAZQZvjSeEOiZTAh3/+qZYBRwp+JeQlH5BIWAAAACBBmgdJ4Q8mUwId//6plgbYkmrquKRhvAOO1AP7+xgnYQAAABBBniVFETwv/wIBGtRQfCphAAAAEAGeRHRCvwKwQBztjE2qSMEAAAAPAZ5GakK/Aq3miDHToKKLAAAAG0GaS0moQWiZTAh3//6plgYfS6GJ/BJHqC4UkAAAABBBnmlFESwv/wHp+9fkNFFAAAAAEAGeiHRCvwKvcd5WvEFK9IEAAAAPAZ6KakK/AZOxA8mCLKSAAAAAHkGaj0moQWyZTAh3//6plgFM7agH9/P6CWcoNwYCmgAAABBBnq1FFSwv/wEmz9m4IDDxAAAADwGezHRCvwGTSUQpgiykgQAAAA8Bns5qQr8BkyWlSKBKog8AAAAdQZrTSahBbJlMCHf//qmWBg9huUGwxDifsOsHI+AAAAASQZ7xRRUsL/8B6p0STuWbGdf4AAAAEAGfEHRCvwGTkWVeBFdsz4EAAAAQAZ8SakK/Aq3miZEr5OSygAAAABNBmxdJqEFsmUwId//+qZYAAJWAAAAADEGfNUUVLC//AACygQAAABABn1R0Qr8Cg2LddwD7cFbAAAAADwGfVmpCvwKDYtsM9WGhlQAAABJBm1tJqEFsmUwIb//+p4QAAScAAAAVQZ95RRUsL/8B6r1jVhnN0sOX6KKAAAAAEAGfmHRCvwKP2iPK/JTBnTEAAAAQAZ+aakK/Aq3miZEr5OSygAAAAB5Bm59JqEFsmUwIb//+p4QLtsx+GF1HyzEc4eTJZ8EAAAASQZ+9RRUsL/8B6fvZ+BK1bFu5AAAAEAGf3HRCvwKvcd5WvEFK9IAAAAAPAZ/eakK/AZMlpUigSqIOAAAAGUGbwEmoQWyZTAhv//6nhAE16bcUyKE9/fMAAAAZQZvhSeEKUmUwId/+qZYAZbi9FuyDd1CDgAAAABFBmgVJ4Q6JlMCG//6nhAABJwAAABNBniNFETwv/wBNfQRSkdM5YtE+AAAAEAGeQnRCvwBpgFM8r8lNn0kAAAAQAZ5EakK/AGmduE3GfXpvpQAAABpBmkZJqEFomUwId//+qZYAQH486WdHU8jNwQAAAB5BmmpJ4QpSZTAhv/6nhAB8uEzMssRDj+BTZGCWRA0AAAAUQZ6IRTRML/8AS2f7pmCy43T1UmkAAAAPAZ6ndEK/AGcAPik2yVT/AAAAEAGeqWpCvwA/gRM030kHH0kAAAArQZquSahBaJlMCG///qeEADT8Jma7XAfQpgP/+EqILMX/+EmI/F//r/uhTAAAABBBnsxFESwv/wAfFOnf5vn4AAAADwGe63RCvwAsVo7zzi3jgQAAABABnu1qQr8AKy1851oYXnfBAAAAGkGa70moQWyZTAhv//6nhAAzdIn+pHRyLW3BAAAAGUGbE0nhClJlMCGf/p4QANLaxxp3abhDzWwAAAAUQZ8xRTRML/8AMQ7YbcTfH4fJXiAAAAAPAZ9QdEK/AEF81QOnahtHAAAAEAGfUmpCvwBBZZDD6AkHHugAAAAZQZtUSahBaJlMCG///qeEADY+wevZnwRYQQAAABhBm3VJ4QpSZTAhv/6nhAA0/CYo+DCDXn0AAAAZQZuWSeEOiZTAh3/+qZYAGg4vMSdH95FFgAAAAEVBm7pJ4Q8mUwId//6plgAZbi9GBnVi9qKf/8Qj2KWrvECMEp0SRD2Hv//iAJTVznYHsuEH///xB35ertvPBIRYD3/GynEAAAAQQZ/YRRE8L/8AHbTmNpPNJQAAAA8Bn/d0Qr8AKPGMXAfl0qAAAAAPAZ/5akK/ACjtt0o0h4rhAAAASEGb/kmoQWiZTAh3//6plgAPr8KP7WoUl//4hHsUtXeIEYJTokiHsPf//EASmrnOwPZcIP///iDvy9XbeeCQiwHv8xt6/9DqQAAAABVBnhxFESwv/wAS2hEE4Qezp2stN7kAAAAPAZ47dEK/ABnAD4pNslX/AAAAEAGePWpCvwAZIFjXvNKzskAAAAAfQZoiSahBbJlMCG///qeEAB3PfZ9wW6YXMssTI5nagAAAABJBnkBFFSwv/wAR2gOf4GFVu50AAAAQAZ5/dEK/ABknk3lbKHqfwAAAABABnmFqQr8AD+AvOdaGF8fBAAAAGkGaY0moQWyZTAh3//6plgAGegsrjNL+2ELAAAAAG0Gah0nhClJlMCG//qeEAAzvCZmaVPucE0aEeQAAABBBnqVFNEwv/wAHl/ZXlDuRAAAAEAGexHRCvwAKgnMcB+UAIOEAAAAQAZ7GakK/AAbBK2L1dhzxwQAAABpBmshJqEFomUwId//+qZYAArvvq+uxBuK7UAAAAElBmuxJ4QpSZTAh3/6plgAC7dwaf/4hI8VNXeaHYJVfriI8P///EC5Zq6ZbBBb8K4v//4hAyp1dyRuCSV1CAzU/Y99NutSxOwy6AAAAFEGfCkU0TC//AAN0qaBaYXKoVcoxAAAADwGfKXRCvwAEd9J3Bsl65gAAABABnytqQr8ABLdiPJcz5YGAAAAASEGbMEmoQWiZTAhv//6nhAAGHl6jj//xCSwxcNtQ0YSsS4kSwm//+IHWjhrYnCFMUXVP//xCDqTw2oj6EldJkDro2L6Lb2gIGQAAABVBn05FESwv/wAFilY6XGrq0g1TSWEAAAAPAZ9tdEK/AAduMPKGgZw7AAAAEAGfb2pCvwAHbCJmm+kg8nAAAAAdQZt0SahBbJlMCG///qeEAAZP2Vgx/6aSGnva6XAAAAAWQZ+SRRUsL/8AA7aV1P7gm6XIcdkwQQAAABABn7F0Qr8ABPk6k8r8lO2QAAAAEAGfs2pCvwAFHUaJkTStPUAAAAAaQZu1SahBbJlMCG///qeEAAlqALNts+z560EAAAAZQZvWSeEKUmUwId/+qZYAB0kyEm4cFHzokAAAAB5Bm/pJ4Q6JlMCHf/6plgAHf9tQHN+lI99NK3zNLWcAAAAVQZ4YRRE8L/8ACOz1qp0crkW+wosbAAAAEAGeN3RCvwAMQ8m8rZQ9isAAAAAPAZ45akK/AAfEFKZtmRwlAAAAHEGaPkmoQWiZTAh3//6plgAEp6tj20QwzitT5oAAAAAQQZ5cRREsL/8ABYp8vobfTQAAAA8Bnnt0Qr8AB2y9AZJdZYEAAAAPAZ59akK/AAdsFKZtmRw3AAAAHUGaYkmoQWyZTAhv//6nhAAOilQqmQYhXE9/2RB4AAAAFUGegEUVLC//AAiugj950zi1891A4QAAAA8Bnr90Qr8AC15k7g2S8zUAAAAQAZ6hakK/AAvzqnkuZ8nigQAAAB5BmqZJqEFsmUwIZ//+nhAAh3wYOBTYGwnD09dGwgwAAAAVQZ7ERRUsL/8AFQoEUpHTOWNCZ7nxAAAADwGe43RCvwAS30ncGyXlBwAAABABnuVqQr8AHP5w17zSs6XBAAAAGkGa6UuoQhBbJEYIKAfyAf2HgCFf/jhAABFxAAAAIkGfCEIV/wKvY+1BxN2qw0km5aqGByy1u80ozu7zDQi0wOwAAAAQAZ8naRCvABxOLM0vESm5HAAADAFtb292AAAAbG12aGQAAAAAAAAAAAAAAAAAAAPoAAAfkAABAAABAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAALK3RyYWsAAABcdGtoZAAAAAMAAAAAAAAAAAAAAAEAAAAAAAAfkAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAABEAAAARAAAAAAACRlZHRzAAAAHGVsc3QAAAAAAAAAAQAAH5AAAAQAAAEAAAAACqNtZGlhAAAAIG1kaGQAAAAAAAAAAAAAAAAAADIAAAGUAFXEAAAAAAAtaGRscgAAAAAAAAAAdmlkZQAAAAAAAAAAAAAAAFZpZGVvSGFuZGxlcgAAAApObWluZgAAABR2bWhkAAAAAQAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAKDnN0YmwAAACWc3RzZAAAAAAAAAABAAAAhmF2YzEAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAABEAEQAEgAAABIAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAY//8AAAAwYXZjQwH0AA3/4QAXZ/QADZGbKCIR0IAAAAMAgAAAGQeKFMsBAAZo6+PESEQAAAAYc3R0cwAAAAAAAAABAAAAygAAAgAAAAAUc3RzcwAAAAAAAAABAAAAAQAABdhjdHRzAAAAAAAAALkAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAMAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAIAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAAFAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAQAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAwAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAIAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAwAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACAAAAAABAAAEAAAAAAEAAAAAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAFnwAAABcAAAArAAAAFAAAABMAAAAUAAAAHgAAACQAAAAVAAAAEwAAABMAAAAdAAAAHwAAAB0AAAAkAAAAFgAAABMAAAATAAAAHgAAACIAAAAVAAAAEwAAABQAAAAgAAAAIAAAABQAAAASAAAAFAAAACsAAAAZAAAAFAAAABQAAAAdAAAAHgAAABkAAAAUAAAAEwAAAB0AAAAcAAAATAAAABcAAAATAAAAFAAAAB4AAAAsAAAAGQAAABQAAAAUAAAAGwAAABIAAAAUAAAAFAAAADAAAAAUAAAAEgAAABQAAAAeAAAATAAAABgAAAAUAAAAFAAAAB8AAAAcAAAAHAAAABwAAAAdAAAAIQAAABUAAAATAAAAFAAAAB0AAAAgAAAAFAAAABQAAAATAAAAHQAAABwAAAAdAAAAHQAAABkAAAAZAAAAEwAAABQAAAAeAAAAIAAAABUAAAATAAAAFAAAACUAAAAZAAAAEwAAABQAAAAdAAAAIgAAABkAAAATAAAAEwAAACAAAAAfAAAAHQAAACQAAAAUAAAAFAAAABMAAAAfAAAAFAAAABQAAAATAAAAIgAAABQAAAATAAAAEwAAACEAAAAWAAAAFAAAABQAAAAXAAAAEAAAABQAAAATAAAAFgAAABkAAAAUAAAAFAAAACIAAAAWAAAAFAAAABMAAAAdAAAAHQAAABUAAAAXAAAAFAAAABQAAAAeAAAAIgAAABgAAAATAAAAFAAAAC8AAAAUAAAAEwAAABQAAAAeAAAAHQAAABgAAAATAAAAFAAAAB0AAAAcAAAAHQAAAEkAAAAUAAAAEwAAABMAAABMAAAAGQAAABMAAAAUAAAAIwAAABYAAAAUAAAAFAAAAB4AAAAfAAAAFAAAABQAAAAUAAAAHgAAAE0AAAAYAAAAEwAAABQAAABMAAAAGQAAABMAAAAUAAAAIQAAABoAAAAUAAAAFAAAAB4AAAAdAAAAIgAAABkAAAAUAAAAEwAAACAAAAAUAAAAEwAAABMAAAAhAAAAGQAAABMAAAAUAAAAIgAAABkAAAATAAAAFAAAAB4AAAAmAAAAFAAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1Ni4zNi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "K.clear_session()\n",
    "epochs_train = 20\n",
    "# Training\n",
    "env = EnvironmentExploring(grid_size=size, max_time=T, temperature=0.3)\n",
    "agent = DQN_CNN(size, lr=.5, epsilon = 0.5, memory_size=2000, batch_size = 32, n_state=3)\n",
    "train_explore(agent, env, epochs_train, prefix='cnn_train_explore')\n",
    "HTML(display_videos('cnn_train_explore10.mp4'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Win/lose count 16.5/4.0. Average score (12.5)\n",
      "Win/lose count 19.5/4.0. Average score (14.0)\n",
      "Win/lose count 18.5/10.0. Average score (12.166666666666666)\n",
      "Win/lose count 22.5/2.0. Average score (14.25)\n",
      "Win/lose count 23.5/3.0. Average score (15.5)\n",
      "Win/lose count 25.5/5.0. Average score (16.333333333333332)\n",
      "Win/lose count 21.0/6.0. Average score (16.142857142857142)\n",
      "Win/lose count 22.0/3.0. Average score (16.5)\n",
      "Win/lose count 24.5/5.0. Average score (16.833333333333332)\n",
      "Win/lose count 19.0/8.0. Average score (16.25)\n",
      "Win/lose count 24.0/4.0. Average score (16.59090909090909)\n",
      "Final score: 16.59090909090909\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAGiltZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE0NiByMjUzOCAxMjEzOTZjIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAAMKZYiEADv//vb8/AptUwndP/9P+VxN2UIE1zAALTEAIgTYlI7FKYvwKUU8/ApmttTF/97dKYMnoxHKB0/oD9V4Dac0u1UzDq2rY7P9f698A5TsY/sfObSDpBtR9Og0GTHg+kYCdQEEXeaISLL8UbbjaK3bNy6ZYuDuTgKwucj9k9bAV4JTeZGj5ZLQGqYWZ9oGivPhITZccX8efxaaZSV1CH+990X3fv10Gpi1HqgHpa86E/l2sBa/mv2uRwRP1wQuonhBaANj3f9FCRLnx8oZeI2mMqQuK7dvesYBd97FQaZexOEWKSDGHf9Az8xUyKEWNvcSxecQ0V+ma4aGcOuolMrqKP1ewa/2we60HE6GEket/HWapdyeTBxZffpIIw86AxSOl9yTBRCCNmZ8EkyiWFP4kRsToC64cowulxckHeRx2knF0YEBdFrV9xoEHUTIKbdhUQPorD4mZCbwC8qn4ovQGhOvzkGohOEaULMQC/BGn+UsdADV4JqgVsoRBmtSTQX/3bFJwd0nkwdEMgCapiUSLE+FA+EaN1f0Nu+NlXZCBTEbqA6f5Geg3ZLIQsXt8kpB+vnZwYo/Fh2sUWBXz4HgvmG3uX8fTs4TBj0PdOK2RV4GUaIfRBxq6WDNEu2tEBpfwvHOelkiWtrpqVYBkhVUvEL/q1y7XAh0dD8FMhlwbYS1DhSzcnvphx1nArRuH4CGUshWOYC/AX6ABO+epZzaUO4U7oNvvBqxLKPNC5RWNNvlkqtI84iif6aOyDGp3w9woRNOIDVw/a/6syUBZLIJMMmKZrlQDdtfThZ8m6LTjfDZa7+fFYsxzXP3FLDETeQnN+KUVzmfgaxCP3H+rtagtVcTSIXnpJ8WFZHjFNVmhhisZw+TuBvb6/K5/dCBcb5FCCCdO9ytSFUZA4F/pDXC6N6vQO2tmkyuUuRatpwGLCViPogi4GnukHPO+eB7hjsNaJF/jKUkA57j34tG2QXx67Sr7SS4U5Lww8l6YvEaU3zHceQ2hGWmt0m6MQSmxL5MguRCIlWDGwAAAKRBmiRsQ3/+p4QBJGQX//4hIPpuG2ZXsJUdKSIPkf//EB25w1UPg/46J2P//4g+i/htGQQkYJx+jtdzoxh9H6uBWYAtyUfFxh7cQ/QP2Sdwr/Arnl3HWfzRAHp1/38nB6hds7JENugv2QoL/9RYGGiXaNsvjdseAq5so1A19rgLc/539NoplJDhEbWEOsWDup9YLfZULw0fmjzEj9Bv/3VjlD1r0wAAABNBnkJ4hf8AsVpGyx0ziwkW22LZAAAAEAGeYXRCvwDiRVqvAiu2toAAAAAQAZ5jakK/AO0zB5LmfJKZgQAAAC1BmmhJqEFomUwIb//+p4QLtrx8yyrbKfgUloH8CmW0aD4LN025pxIiuDm8ZQ8AAAAYQZ6GRREsL/8B6pyw8Sfu1a+UOLMnYnehAAAAEAGepXRCvwF/kWVeBFds2YEAAAAPAZ6nakK/ApFnR+GzaPIGAAAAGkGaqUmoQWyZTAh3//6plgXiifiuKB25CHHAAAAAJEGazUnhClJlMCG//qeECk7VZR9Yb77+rzqPorH5lliZGurpgQAAABRBnutFNEwv/wHV9oZIcEdi5sXrUgAAABABnwp0Qr8CdP4DJK4RnO6AAAAAEAGfDGpCvwF1a+c60MLw4EEAAAAaQZsOSahBaJlMCHf//qmWAJQUc60PV98hQ8EAAABKQZsySeEKUmUwIb/+p4QBLem27K/4E/3mUGlU//4hJYYuG2oaMJWJcSJYTf//EDrRw1sThCmKLqn//4hB1J4bUR9CSukyB1zvZ8EAAAAVQZ9QRTRML/8AtbG23LfRYzn6kYM+AAAAEAGfb3RCvwDytga2mUPR3EAAAAAQAZ9xakK/AJ83IYfQEg4sqQAAABpBm3NJqEFomUwId//+qZYAQgo50miG8HkMvAAAAC5Bm5dJ4QpSZTAh3/6plgCh3Ev/iEhIP/4Snvexf/4SVocX//f/VKe2n6zl+c1MAAAAEEGftUU0TC//AL6yktOMqaEAAAAPAZ/UdEK/AP70p4HTKbmTAAAADwGf1mpCvwD/DQPJgiz5gQAAACNBm9tJqEFomUwId//+qZYFSDTb8Qg2//CVUM3//guEzB7GLQAAABVBn/lFESwv/wHWnRE3uVyIuPWAXtAAAAAQAZ4YdEK/AYmTQifFmKNQcQAAABABnhpqQr8Cj+aJkSvk5LaAAAAAJEGaH0moQWyZTAh3//6plgVLZj8hJcdy+IQdf/hKqGb//SQk4QAAABVBnj1FFSwv/wHV+7+Qr9ciQ6xnOOEAAAAQAZ5cdEK/AnT+AySuEZzugAAAAA8Bnl5qQr8BdWspm2ZGsd0AAAAnQZpDSahBbJlMCG///qeEAfHon+A13hRfApr6g34FKlo/ApnYC+qnAAAAFUGeYUUVLC//AP6gM20r9ciQ6yY8sAAAABABnoB0Qr8BY05jgPs5kS9hAAAADwGegmpCvwDckXzNsyNZlQAAABpBmoRJqEFsmUwId//+qZYA2jkGZ/Kjn1CxYQAAAB5BmqhJ4QpSZTAhv/6nhAUvj9B2/vZ1PWzEGrKRZmEAAAARQZ7GRTRML/8Bev2ZF8rfcXEAAAAPAZ7ldEK/AfogDoRkudfBAAAAEAGe52pCvwFIa+c60MLw7MAAAAAeQZrsSahBaJlMCG///qeEAPhwmZtusNzLLEyO5DriAAAAEEGfCkURLC//AJbn7nCygTkAAAAPAZ8pdEK/AT/oB0JyXcTAAAAAEAGfK2pCvwDNkyTTfSQcU3AAAAAaQZstSahBbJlMCHf//qmWAFU55kNsQbuoUEEAAAAgQZtRSeEKUmUwIb/+p4QAbHhMzaO5mpt6N09MX5VsfBEAAAASQZ9vRTRML/8AP3/FXZhx4o/BAAAAEAGfjnRCvwBYk6k8r8lNonAAAAAPAZ+QakK/ADoWBLlf4A9AAAAAGkGbkkmoQWiZTAh3//6plgA3lSDNAHpL7AZRAAAAJUGbtknhClJlMCHf/qmWADk+0v2ebwjYH4FNpbrUIMziE3UaIIAAAAAWQZ/URTRML/8AQ2etiCctJofMR16NfAAAABABn/N0Qr8AXS0d5Wyh6ThBAAAAEAGf9WpCvwA+LMHkuZ8lEYAAAAAgQZv6SahBaJlMCG///qeEAE2+On3a06OZZYmR0wjWBt0AAAARQZ4YRREsL/8ALpQIrRoy6fEAAAAPAZ43dEK/AD4l6AyS5buAAAAAEAGeOWpCvwA+IRM030kHH5kAAAAaQZo7SahBbJlMCHf//qmWABoPaX87pCmEW9AAAAAqQZpfSeEKUmUwId/+qZYAEJ6tkQM3dvdi5llapqvApRIF4FM1yFpfYgjBAAAAFUGefUU0TC//ABPmUl19N6fZ+kYxIQAAABABnpx0Qr8AGwAQuA/KAAfgAAAAEAGenmpCvwAL86p5LmfJ4oAAAAApQZqDSahBaJlMCHf//qmWABvvYb5llapqvApRIF4FM1yxxm/6EP5uQ5kAAAAVQZ6hRREsL/8AILn6Dj+dM4pRVD3iAAAAEAGewHRCvwAS3zVA6dqHeYEAAAAQAZ7CakK/AC12PHK/tw/mwAAAAClBmsdJqEFsmUwId//+qZYAofsN8yyxhVXgUzXFTwKJPXC3nm5BiMcDgQAAABRBnuVFFSwv/wC+0BpzbdLLPwq+mQAAABABnwR0Qr8AbCS14HTKbsaBAAAAEAGfBmpCvwD+2eOV/bh890EAAAAbQZsLSahBbJlMCHf//qmWATOlnKDM/iPBVRNwAAAAEEGfKUUVLC//AR6gOXkT5uAAAAAQAZ9IdEK/AYmTQifFmKNQcQAAAA8Bn0pqQr8CkNa7uf9ckYAAAAATQZtPSahBbJlMCHf//qmWAACVgAAAAAxBn21FFSwv/wAAsoEAAAAQAZ+MdEK/ApJAHQhw3bskYQAAABABn45qQr8CkNa7t+G7dkjBAAAAE0Gbk0moQWyZTAh3//6plgAAlYAAAAAMQZ+xRRUsL/8AALKAAAAAEAGf0HRCvwKSQB0IcN27JGEAAAAQAZ/SakK/ApDWu7fhu3ZIwAAAACRBm9dJqEFsmUwIb//+p4QCad1P94hmvxCAf/8JUsef/8E3NmAAAAAVQZ/1RRUsL/8B1bt0zit0sNhOPSbhAAAAEAGeFHRCvwKSQBztjE2qSsAAAAAQAZ4WakK/AnVl+qPiRc53QQAAABpBmhhJqEFsmUwId//+qZYBF9WyGKMOnwStgQAAABpBmjxJ4QpSZTAhv/6nhAEl6bcZ0AxwTRzhvwAAABBBnlpFNEwv/wCxMpLTjK0hAAAAEAGeeXRCvwDs8UXAfk//P+AAAAAPAZ57akK/AGSsQPJgi6SBAAAAHEGaYEmoQWiZTAhn//6eEAHc9ff02ULl1s1bHdEAAAAQQZ6eRREsL/8ASXP2bggY8AAAAA8Bnr10Qr8AZJJqerO+0UAAAAAQAZ6/akK/AGSI7c60MLx5QQAAABpBmqFJqEFsmUwIb//+p4QATb46fUcaEhxWwAAAABhBmsJJ4QpSZTAhv/6nhAAyfCZlcgt6a4MAAAAZQZrjSeEOiZTAh3/+qZYAEB6tkNsQbupnwAAAABZBmwdJ4Q8mUwIb//6nhAANj77PtjzBAAAADkGfJUURPC//AAfv9yahAAAADwGfRHRCvwAQqh5LUB+YlwAAAA8Bn0ZqQr8AEKoeSlAfmJcAAAAaQZtISahBaJlMCHf//qmWAAqnPMJlmfPlLoAAAAASQZtsSeEKUmUwId/+qZYAAJWAAAAADEGfikU0TC//AACygQAAAA8Bn6l0Qr8AECoeKNElfrwAAAAPAZ+rakK/ABAqHhzHWm+UAAAAJ0GbsEmoQWiZTAh3//6plgAPV7DfMssYVV4FM1xU8CiT2H60vu3egQAAABBBn85FESwv/wASWgOXkWHhAAAAEAGf7XRCvwAZKTQifFmKQ/EAAAAPAZ/vakK/ABnAWNgcp2CAAAAAJkGb9EmoQWyZTAh3//6plgA32smD4hISD/+EqILP//Sm7ThOv7dAAAAAFUGeEkUVLC//AEFz9DHENH7tM6h3owAAABABnjF0Qr8AJsIA52xxpqSgAAAAEAGeM2pCvwBa7Hjlf24fdsAAAAAfQZo4SahBbJlMCHf//qmWAIT8TzmWWfPtyr/nm2UMwQAAABVBnlZFFSwv/wCfUCKUjpnLGhM9UvAAAAAQAZ51dEK/AI75qgdO1DV0gQAAABABnndqQr8A17twm4z69Nd1AAAAHEGafEmoQWyZTAh3//6plgCE/Hn8uz2oWQpc9b0AAAAQQZ6aRRUsL/8An1AgpQwfmQAAAA8Bnrl0Qr8BUegHQnJdwMAAAAAQAZ67akK/ANeR251oYXiSQQAAAB9BmqBJqEFsmUwId//+qZYAd/2G+ZZZ8+3QW535mX5BAAAAE0Ge3kUVLC//AJbH8Gnu7qbwai8AAAAQAZ79dEK/AM3JoRPizFGvmAAAAA8Bnv9qQr8AzZF8zbMjWbMAAAAeQZrkSahBbJlMCG///qeEAXHfhRrNt39E/nweZpg4AAAAEUGfAkUVLC//ANeq8bzkwTZhAAAADwGfIXRCvwDDpKIUwRapgAAAABABnyNqQr8BLxMBzuK5tH21AAAAHEGbJUmoQWyZTAh3//6plgHMkhJtOxCAf37XKCEAAAAdQZtJSeEKUmUwIb/+p4QD2cZ/o8x+bgZd0cIfk4EAAAAUQZ9nRTRML/8BWw9DX1WM5eE+GqkAAAAQAZ+GdEK/AdIkDW0yh6NgwAAAAA8Bn4hqQr8BLpW6UaQ8SgYAAAAaQZuKSahBaJlMCHf//qmWAHdHT8pox+tJQ8EAAAAaQZuuSeEKUmUwId/+qZYAyiIg62jr48+fregAAAAQQZ/MRTRML/8A4f8PXWDZQAAAAA8Bn+t0Qr8BNvSdwbJeMfcAAAAPAZ/takK/ATcNYF1/ftTBAAAAGUGb8kmoQWiZTAh3//6plgDL+jn5HIXYWP8AAAAQQZ4QRREsL/8A4f8PXWDZQAAAABABni90Qr8BNnVoyS3+toOAAAAADwGeMWpCvwDD2IHkwRapgQAAABNBmjZJqEFsmUwId//+qZYAAJWAAAAADEGeVEUVLC//AACygAAAAA8BnnN0Qr8AwZXu5HrTU7sAAAAPAZ51akK/AMGV7nKR3LVMAAAAIUGaekmoQWyZTAhv//6nhADtcJmZpU/SE2kOZZYmRyA7QQAAABRBnphFFSwv/wCOz/d6x/6ISgFZUQAAABABnrd0Qr8Aw4CFwH5P/0ggAAAADwGeuWpCvwBR226UaQ8T4QAAABpBmrtJqEFsmUwId//+qZYAH19pfzukKYRW0AAAABFBmt9J4QpSZTAhv/6nhAABJwAAAAxBnv1FNEwv/wAAsoEAAAAPAZ8cdEK/ACG7jujtvhZbAAAADwGfHmpCvwAhrzRBajy8TgAAAB5BmwNJqEFomUwIb//+p4QAKR7qfu1p0cyyxMjtCd0AAAAQQZ8hRREsL/8AGIVeN7BNuAAAAA4Bn0B0Qr8AIbuO884uJwAAABABn0JqQr8AIbmjeaYq2oNAAAAAGUGbREmoQWyZTAhv//6nhAAbHhMyuQW9Ns8AAAAZQZtlSeEKUmUwId/+qZYACM/Hn79kG4qe4QAAAB9Bm4lJ4Q6JlMCG//6nhAALr7qfd5ucngeDdIQ8NQaBAAAAEUGfp0URPC//AAboPQ0g2VV1AAAAEAGfxnRCvwAJbuO8rZQ9rYAAAAAQAZ/IakK/AAYPOTvZ4+52gAAAAElBm81JqEFomUwIZ//+nhAALnp6o5//4/yamgJZ/zCqlOq/5a0v//x8lNASXvFSyp+V5f//j+K5oCV67eqYym/HUxRvh62e5lvvAAAAEUGf60URLC//AAcVOnf4WIz4AAAADwGeCnRCvwAGISUQpglZgAAAAA8BngxqQr8ACa7EeTA9fGcAAAAaQZoOSahBbJlMCG///qeEAAw7q0ghE/y3u4EAAAAZQZovSeEKUmUwIb/+p4QADIurR0fcbMFxsQAAACBBmlNJ4Q6JlMCG//6nhAANP6w3MssTI77taW8+KEj2VgAAACRBnnFFETwv/wAHxT1esajo4f/+P8npoan8ZfD5DkKaN9j+2OYAAAAQAZ6QdEK/AAo6dSeV+SnGEQAAABABnpJqQr8ACs2PLcNm1aWAAAAAGUGalEmoQWiZTAhv//6nhAANPwmZXILenJ4AAAAYQZq1SeEKUmUwId/+qZYABEfjzrHdsqOBAAAAIUGa2UnhDomUwIb//qeEAAg30c/EXbBzZxmqa3RKZ9rugAAAABRBnvdFETwv/wAE+ZSVHd9nTD+1eQAAABABnxZ0Qr8ABsAELgPygBzhAAAAEAGfGGpCvwAEN2iE3GfXsngAAAAcQZsdSahBaJlMCG///qeEAAgq2lzN7qeo0s4+QQAAABFBnztFESwv/wAE+oEFJ4QxeAAAAA8Bn1p0Qr8ABDbQgMku/YEAAAAQAZ9cakK/AAbB1TyYHr5wgQAAABpBm15JqEFsmUwId//+qZYABCfo59+yDcVi4AAAABpBm2JJ4QpSZTAhv/6nhAAH7R5nwNvdT9stwAAAABRBn4BFNEwv/wAE1z9y3WFjg1e3cQAAAA8Bn790Qr8ABFXZQpNslsEAAAAQAZ+hakK/AAaZ2o5X9uI7wQAAACVBm6ZJqEFomUwIZ//+nhAAR74MHAprnmnzLI6nPMq9PU30AcdtAAAAFUGfxEURLC//AAsVAilI6ZyxqOrRCQAAABABn+N0Qr8ACfZokT4sxS6RAAAAEAGf5WpCvwAO2z5jdDkg6jkAAAAaQZvpS6hCEFskRggoB/IB/YeAIV/+OEAAEXEAAAAnQZ4IQhX/Aq9j7UHE3arDSSblqoYHLLW7zSojkSYHstoar9mfNNC0AAAAEAGeJ2kQrwAO1wus60MZ5uAAAAxhbW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAAH5AAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAC4t0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAAH5AAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAARAAAAEQAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAB+QAAAEAAABAAAAAAsDbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAAAAyAAABlABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAAKrm1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAACm5zdGJsAAAAlnN0c2QAAAAAAAAAAQAAAIZhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAARABEABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAMGF2Y0MB9AAN/+EAF2f0AA2RmygiEdCAAAADAIAAABkHihTLAQAGaOvjxEhEAAAAGHN0dHMAAAAAAAAAAQAAAMoAAAIAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAY4Y3R0cwAAAAAAAADFAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAADAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAgAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAgAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAIAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAgAAAAAAQAABAAAAAABAAAAAAAAABxzdHNjAAAAAAAAAAEAAAABAAAAygAAAAEAAAM8c3RzegAAAAAAAAAAAAAAygAABb8AAACoAAAAFwAAABQAAAAUAAAAMQAAABwAAAAUAAAAEwAAAB4AAAAoAAAAGAAAABQAAAAUAAAAHgAAAE4AAAAZAAAAFAAAABQAAAAeAAAAMgAAABQAAAATAAAAEwAAACcAAAAZAAAAFAAAABQAAAAoAAAAGQAAABQAAAATAAAAKwAAABkAAAAUAAAAEwAAAB4AAAAiAAAAFQAAABMAAAAUAAAAIgAAABQAAAATAAAAFAAAAB4AAAAkAAAAFgAAABQAAAATAAAAHgAAACkAAAAaAAAAFAAAABQAAAAkAAAAFQAAABMAAAAUAAAAHgAAAC4AAAAZAAAAFAAAABQAAAAtAAAAGQAAABQAAAAUAAAALQAAABgAAAAUAAAAFAAAAB8AAAAUAAAAFAAAABMAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACgAAAAZAAAAFAAAABQAAAAeAAAAHgAAABQAAAAUAAAAEwAAACAAAAAUAAAAEwAAABQAAAAeAAAAHAAAAB0AAAAaAAAAEgAAABMAAAATAAAAHgAAABYAAAAQAAAAEwAAABMAAAArAAAAFAAAABQAAAATAAAAKgAAABkAAAAUAAAAFAAAACMAAAAZAAAAFAAAABQAAAAgAAAAFAAAABMAAAAUAAAAIwAAABcAAAAUAAAAEwAAACIAAAAVAAAAEwAAABQAAAAgAAAAIQAAABgAAAAUAAAAEwAAAB4AAAAeAAAAFAAAABMAAAATAAAAHQAAABQAAAAUAAAAEwAAABcAAAAQAAAAEwAAABMAAAAlAAAAGAAAABQAAAATAAAAHgAAABUAAAAQAAAAEwAAABMAAAAiAAAAFAAAABIAAAAUAAAAHQAAAB0AAAAjAAAAFQAAABQAAAAUAAAATQAAABUAAAATAAAAEwAAAB4AAAAdAAAAJAAAACgAAAAUAAAAFAAAAB0AAAAcAAAAJQAAABgAAAAUAAAAFAAAACAAAAAVAAAAEwAAABQAAAAeAAAAHgAAABgAAAATAAAAFAAAACkAAAAZAAAAFAAAABQAAAAeAAAAKwAAABQAAAAUc3RjbwAAAAAAAAABAAAAMAAAAGJ1ZHRhAAAAWm1ldGEAAAAAAAAAIWhkbHIAAAAAAAAAAG1kaXJhcHBsAAAAAAAAAAAAAAAALWlsc3QAAAAlqXRvbwAAAB1kYXRhAAAAAQAAAABMYXZmNTYuMzYuMTAw\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluation\n",
    "epochs_test = 11\n",
    "test(agent,env,epochs_test,prefix='cnn_test_explore')\n",
    "HTML(display_videos('cnn_test_explore10.mp4'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:**\n",
    "\n",
    "We encourage exploration thanks to the two previous ideas. I added an increasing position malus (+0.05 of malus each time the  agent comes back in a cell)  to avoid oscilations of the agent. I added too a geometrically decreasing epsilon exploration parameter ( with $\\gamma = 0.9$) to do less and less exploration.\n",
    "\n",
    "We can see that the total reward is increased a lot (**total reward about +15**). And the behaviour of the agent seems to be closer to the desired behaviour: the agents explores the board and tries to find the cheeses and to avoid the poisons. The agent is not blocked in a local optimal policy anymore.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "__BONUS question__ Use the expert DQN from the previous question to generate some winning games. Train a model that mimicks its behavior. Compare the performances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}