SARSA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "deepSARSA-final_version.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "X0uiTOAeVs1n"
      },
      "source": [
        "## Deep SARSA in Cartpole using Keras-RL.\n",
        "#### This is a step-by-step guide to using a neural network-based SARSA in Cartpole, a simple OpenAI gym environment.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LGKv_fE3V6d0"
      },
      "source": [
        "### Table of Contents\n",
        "\n",
        "#### [Setup and Keras-RL](#Setup_and_Environment)\n",
        "- in this section, we import all the necessary libraries and do a minor modification to the Keras-RL Sarsa agent;\n",
        "\n",
        "#### [Deep SARSA](#Sarsa)\n",
        "- in this section, we use Deep SARSA to solve the Cartpole problem.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aLLcBxdHXnqy"
      },
      "source": [
        "## Setup and Keras-RL <a name='Setup_and_Environment'></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qE00evye7xo0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        },
        "outputId": "b407615e-6a77-482b-9f80-d5429639aa10"
      },
      "source": [
        "# pip install keras-rl"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting keras-rl\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ab/87/4b57eff8e4bd834cea0a75cd6c58198c9e42be29b600db9c14fafa72ec07/keras-rl-0.4.2.tar.gz (40kB)\n",
            "\r\u001b[K     |████████                        | 10kB 18.5MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 20kB 3.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 30kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 40kB 3.3MB/s \n",
            "\u001b[?25hRequirement already satisfied: keras>=2.0.7 in /usr/local/lib/python3.6/dist-packages (from keras-rl) (2.3.1)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from keras>=2.0.7->keras-rl) (1.0.8)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from keras>=2.0.7->keras-rl) (1.1.0)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras>=2.0.7->keras-rl) (2.10.0)\n",
            "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.6/dist-packages (from keras>=2.0.7->keras-rl) (1.18.2)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras>=2.0.7->keras-rl) (1.4.1)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras>=2.0.7->keras-rl) (3.13)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from keras>=2.0.7->keras-rl) (1.12.0)\n",
            "Building wheels for collected packages: keras-rl\n",
            "  Building wheel for keras-rl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-rl: filename=keras_rl-0.4.2-cp36-none-any.whl size=48380 sha256=0d9d989884b80afae77ab2ea7436a232141a7d4a8d5632fc54aad46383d882ad\n",
            "  Stored in directory: /root/.cache/pip/wheels/7d/4d/84/9254c9f2e8f51865cb0dac8e79da85330c735551d31f73c894\n",
            "Successfully built keras-rl\n",
            "Installing collected packages: keras-rl\n",
            "Successfully installed keras-rl-0.4.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "onV50lj71FpI",
        "colab_type": "text"
      },
      "source": [
        "### Importing the necessary libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "klTUy1NnyF-s",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import gym\n",
        "# %tensorflow_version 1.14\n",
        "from keras.layers import Dense, Flatten, Activation\n",
        "from keras.models import Sequential\n",
        "from keras.optimizers import Adam\n",
        "\n",
        "from rl.agents import SARSAAgent\n",
        "from rl.policy import EpsGreedyQPolicy\n",
        "from rl.callbacks import TrainEpisodeLogger"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xTOA7WUd0Ev8"
      },
      "source": [
        "The code below is from [Keras-RL](https://github.com/keras-rl/keras-rl/blob/master/rl/agents/sarsa.py#L17) with a minor modification to get the probabilities vector (which we need for the Ensemble learning part as explained at the end of the associated Medium post)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IAHUBNb3YbCq"
      },
      "source": [
        "*Note that we are importing some libraries again; some versions of tensorflow are not compatible with parts of this implementation, and importing twice seemed to fix it.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gqygzl17EcbW",
        "colab": {}
      },
      "source": [
        "import collections\n",
        "\n",
        "from keras.callbacks import History\n",
        "from keras.models import Model\n",
        "from keras.layers import Input, Lambda\n",
        "import keras.backend as K\n",
        "\n",
        "from rl.core import Agent\n",
        "from rl.agents.dqn import mean_q\n",
        "from rl.util import huber_loss\n",
        "from rl.policy import EpsGreedyQPolicy, GreedyQPolicy\n",
        "from rl.util import get_object_config\n",
        "\n",
        "\n",
        "class SARSAAgent(Agent):\n",
        "    \"\"\"This class defines the SARSA agent\n",
        "    \"\"\"\n",
        "    def __init__(self, model, nb_actions, policy, test_policy=EpsGreedyQPolicy(), gamma=.9, nb_steps_warmup=10,\n",
        "                 train_interval=1, delta_clip=np.inf, *args, **kwargs):\n",
        "        super(SarsaAgent, self).__init__(*args, **kwargs)\n",
        "\n",
        "        if policy is None:\n",
        "            policy = EpsGreedyQPolicy()\n",
        "        if test_policy is None:\n",
        "            test_policy = GreedyQPolicy()\n",
        "\n",
        "        self.model = model\n",
        "        self.nb_actions = nb_actions\n",
        "        self.policy = policy\n",
        "        self.test_policy = test_policy\n",
        "        self.gamma = gamma\n",
        "        self.nb_steps_warmup = nb_steps_warmup\n",
        "        self.train_interval = train_interval\n",
        "\n",
        "        self.delta_clip = delta_clip\n",
        "        self.compiled = False\n",
        "        self.actions = None\n",
        "        self.observations = None\n",
        "        self.rewards = None\n",
        "        self.q_values=[]\n",
        "\n",
        "    def compute_batch_q_values(self, state_batch):\n",
        "        batch = self.process_state_batch(state_batch)\n",
        "        q_values = self.model.predict_on_batch(batch)\n",
        "        assert q_values.shape == (len(state_batch), self.nb_actions)\n",
        "        return q_values\n",
        "\n",
        "    def compute_q_values(self, state):\n",
        "        q_values = self.compute_batch_q_values([state]).flatten()\n",
        "        assert q_values.shape == (self.nb_actions,)\n",
        "        return q_values\n",
        "\n",
        "    def process_state_batch(self, batch):\n",
        "        batch = np.array(batch)\n",
        "        if self.processor is None:\n",
        "            return batch\n",
        "        return self.processor.process_state_batch(batch)\n",
        "\n",
        "    def get_config(self):\n",
        "        config = super(SarsaAgent, self).get_config()\n",
        "        config['nb_actions'] = self.nb_actions\n",
        "        config['gamma'] = self.gamma\n",
        "        config['nb_steps_warmup'] = self.nb_steps_warmup\n",
        "        config['train_interval'] = self.train_interval\n",
        "        config['delta_clip'] = self.delta_clip\n",
        "        config['model'] = get_object_config(self.model)\n",
        "        config['policy'] = get_object_config(self.policy)\n",
        "        config['test_policy'] = get_object_config(self.test_policy)\n",
        "        return config\n",
        "\n",
        "    def compile(self, optimizer, metrics=[]):\n",
        "        metrics += [mean_q]  \n",
        "\n",
        "        def clipped_masked_error(args):\n",
        "            y_true, y_pred, mask = args\n",
        "            loss = huber_loss(y_true, y_pred, self.delta_clip)\n",
        "            loss *= mask  \n",
        "            return K.sum(loss, axis=-1)\n",
        "\n",
        "        ### Creating trainable model. \n",
        "\n",
        "        # The problem is that we need to mask the output since we only\n",
        "        # ever want to update the Q values for a certain action. The way we achieve this is by\n",
        "        # using a custom Lambda layer that computes the loss. This gives us the necessary flexibility\n",
        "        # to mask out certain parameters by passing in multiple inputs to the Lambda layer.\n",
        "\n",
        "        y_pred = self.model.output\n",
        "        y_true = Input(name='y_true', shape=(self.nb_actions,))\n",
        "        mask = Input(name='mask', shape=(self.nb_actions,))\n",
        "        loss_out = Lambda(clipped_masked_error, output_shape=(1,), name='loss')([y_pred, y_true, mask])\n",
        "        ins = [self.model.input] if type(self.model.input) is not list else self.model.input\n",
        "        trainable_model = Model(inputs=ins + [y_true, mask], outputs=[loss_out, y_pred])\n",
        "        assert len(trainable_model.output_names) == 2\n",
        "        combined_metrics = {trainable_model.output_names[1]: metrics}\n",
        "        losses = [\n",
        "            lambda y_true, y_pred: y_pred,  # loss is computed in Lambda layer\n",
        "            lambda y_true, y_pred: K.zeros_like(y_pred),  # we only include this for the metrics\n",
        "        ]\n",
        "        trainable_model.compile(optimizer=optimizer, loss=losses, metrics=combined_metrics)\n",
        "        self.trainable_model = trainable_model\n",
        "\n",
        "        self.compiled = True\n",
        "\n",
        "    def load_weights(self, filepath):\n",
        "        self.model.load_weights(filepath)\n",
        "\n",
        "    def save_weights(self, filepath, overwrite=False):\n",
        "        self.model.save_weights(filepath, overwrite=overwrite)\n",
        "\n",
        "    def reset_states(self):\n",
        "        self.actions = collections.deque(maxlen=2)\n",
        "        self.observations = collections.deque(maxlen=2)\n",
        "        self.rewards = collections.deque(maxlen=2)\n",
        "        if self.compiled:\n",
        "            self.model.reset_states()\n",
        "\n",
        "    def forward(self, observation):\n",
        "        # Select an action.\n",
        "        q_values = self.compute_q_values([observation])\n",
        "        if self.training:\n",
        "            action = self.policy.select_action(q_values=q_values)\n",
        "        else:\n",
        "            action = self.test_policy.select_action(q_values=q_values)\n",
        "\n",
        "        # Book-keeping.\n",
        "        self.observations.append(observation)\n",
        "        self.actions.append(action)\n",
        "        # self.q_values=q_values\n",
        "\n",
        "        return action\n",
        "\n",
        "    def backward(self, reward, terminal):\n",
        "        metrics = [np.nan for _ in self.metrics_names]\n",
        "        if not self.training:\n",
        "            # We're done here. No need to update the experience memory since we only use the working\n",
        "            # memory to obtain the state over the most recent observations.\n",
        "            return metrics\n",
        "\n",
        "        # Train the network on a single stochastic batch.\n",
        "        \n",
        "        if self.step > self.nb_steps_warmup and self.step % self.train_interval == 0:\n",
        "            # Start by extracting the necessary parameters (we use a vectorized implementation).\n",
        "            self.rewards.append(reward)\n",
        "            if len(self.observations) < 2:\n",
        "                return metrics  # not enough data yet\n",
        "\n",
        "            state0_batch = [self.observations[0]]\n",
        "            reward_batch = [self.rewards[0]]\n",
        "            action_batch = [self.actions[0]]\n",
        "            terminal1_batch = [0.] if terminal else [1.]\n",
        "            state1_batch = [self.observations[1]]\n",
        "            action1_batch = [self.actions[1]]\n",
        "\n",
        "            # Prepare and validate parameters.\n",
        "            state0_batch = self.process_state_batch(state0_batch)\n",
        "            state1_batch = self.process_state_batch(state1_batch)\n",
        "            terminal1_batch = np.array(terminal1_batch)\n",
        "            reward_batch = np.array(reward_batch)\n",
        "            assert reward_batch.shape == (1,)\n",
        "            assert terminal1_batch.shape == reward_batch.shape\n",
        "            assert len(action_batch) == len(reward_batch)\n",
        "\n",
        "            batch = self.process_state_batch(state1_batch)\n",
        "            q_values = self.compute_q_values(batch)\n",
        "            q_values = q_values.reshape((1, self.nb_actions))\n",
        "            probs=q_values[0]\n",
        "            probs/=np.sum(probs)\n",
        "            self.q_values.append(probs)\n",
        "            # self.q_values/=np.sum(q_values)\n",
        "\n",
        "            q_batch = q_values[0, action1_batch]\n",
        "\n",
        "            assert q_batch.shape == (1,)\n",
        "            targets = np.zeros((1, self.nb_actions))\n",
        "            dummy_targets = np.zeros((1,))\n",
        "            masks = np.zeros((1, self.nb_actions))\n",
        "\n",
        "            # Compute r_t + gamma * Q(s_t+1, a_t+1)\n",
        "            discounted_reward_batch = self.gamma * q_batch\n",
        "            # Set discounted reward to zero for all states that were terminal.\n",
        "            discounted_reward_batch *= terminal1_batch\n",
        "            assert discounted_reward_batch.shape == reward_batch.shape\n",
        "            Rs = reward_batch + discounted_reward_batch\n",
        "            for idx, (target, mask, R, action) in enumerate(zip(targets, masks, Rs, action_batch)):\n",
        "                target[action] = R  # update action with estimated accumulated reward\n",
        "                dummy_targets[idx] = R\n",
        "                mask[action] = 1.  # enable loss for this specific action\n",
        "            targets = np.array(targets).astype('float32')\n",
        "            masks = np.array(masks).astype('float32')\n",
        "\n",
        "            # Finally, perform a single update on the entire batch. We use a dummy target since\n",
        "            # the actual loss is computed in a Lambda layer that needs more complex input. However,\n",
        "            # it is still useful to know the actual target to compute metrics properly.\n",
        "            state0_batch = state0_batch.reshape((1,) + state0_batch.shape)\n",
        "            ins = [state0_batch] if type(self.model.input) is not list else state0_batch\n",
        "            metrics = self.trainable_model.train_on_batch(ins + [targets, masks], [dummy_targets, targets])\n",
        "            metrics = [metric for idx, metric in enumerate(metrics) if idx not in (1, 2)]  # throw away individual losses\n",
        "            metrics += self.policy.metrics\n",
        "            if self.processor is not None:\n",
        "                metrics += self.processor.metrics\n",
        "        return metrics\n",
        "\n",
        "    @property\n",
        "    def layers(self):\n",
        "        return self.model.layers[:]\n",
        "\n",
        "    @property\n",
        "    def metrics_names(self):\n",
        "        # Throw away individual losses and replace output name since this is hidden from the user.\n",
        "        assert len(self.trainable_model.output_names) == 2\n",
        "        dummy_output_name = self.trainable_model.output_names[1]\n",
        "        model_metrics = [name for idx, name in enumerate(self.trainable_model.metrics_names) if idx not in (1, 2)]\n",
        "        model_metrics = [name.replace(dummy_output_name + '_', '') for name in model_metrics]\n",
        "\n",
        "        names = model_metrics + self.policy.metrics_names[:]\n",
        "        if self.processor is not None:\n",
        "            names += self.processor.metrics_names[:]\n",
        "        return names\n",
        "\n",
        "    @property\n",
        "    def policy(self):\n",
        "        return self.__policy\n",
        "\n",
        "    @policy.setter\n",
        "    def policy(self, policy):\n",
        "        self.__policy = policy\n",
        "        self.__policy._set_agent(self)\n",
        "\n",
        "    @property\n",
        "    def test_policy(self):\n",
        "        return self.__test_policy\n",
        "\n",
        "    @test_policy.setter\n",
        "    def test_policy(self, policy):\n",
        "        self.__test_policy = policy\n",
        "        self.__test_policy._set_agent(self)\n",
        "\n",
        "# Aliases\n",
        "SarsaAgent = SARSAAgent"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xT_nhQ0aXw5D"
      },
      "source": [
        "## Deep SARSA in Cartpole <a name='Sarsa'></a>\n",
        "In this section, we:\n",
        "\n",
        "1.   set up the Cartpole environment and determine its space;\n",
        "2.   create a neural network model using the Keras-RL Sarsa agent; \n",
        "3.   train and test our agent in the environment.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "E6WpQd6gEjoT",
        "colab": {}
      },
      "source": [
        "#Setting up the environment\n",
        "env = gym.make('CartPole-v1')\n",
        "seed_val = 456\n",
        "env.seed(seed_val)\n",
        "np.random.seed(seed_val)\n",
        "\n",
        "#Getting the state and action space\n",
        "states = env.observation_space.shape[0]\n",
        "actions = env.action_space.n\n",
        "\n",
        "#Defining a Neural Network function for our Cartpole agent \n",
        "def agent(states, actions):\n",
        "    \"\"\"Creating a simple Deep Neural Network.\"\"\"\n",
        "    model = Sequential()\n",
        "    model.add(Flatten(input_shape = (1, states)))\n",
        "    model.add(Dense(24, activation='relu'))\n",
        "    model.add(Dense(24, activation='relu'))\n",
        "    model.add(Dense(24, activation='relu'))\n",
        "    model.add(Dense(actions, activation='linear'))\n",
        "    return model\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2M-A8mMOFVq5",
        "outputId": "08b82211-c1e3-4367-85fb-1f79fe37ab53",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "source": [
        "#Getting our neural network\n",
        "model = agent(states, actions)\n",
        "#Defining SARSA Keras-RL agent: inputing the policy and the model\n",
        "sarsa = SARSAAgent(model=model, nb_actions=actions, policy=EpsGreedyQPolicy())\n",
        "#Compiling SARSA with mean squared error loss\n",
        "sarsa.compile('adam', metrics=[\"mse\"])\n",
        "\n",
        "#Training the agent for 50000 steps\n",
        "sarsa.fit(env, nb_steps=50000, visualize=False, verbose=1)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training for 50000 steps ...\n",
            "Interval 1 (0 steps performed)\n",
            "10000/10000 [==============================] - 32s 3ms/step - reward: 1.0000\n",
            "547 episodes - episode_reward: 18.221 [8.000, 116.000] - loss: 0.015 - mse: 0.938 - mean_q: 1.436\n",
            "\n",
            "Interval 2 (10000 steps performed)\n",
            "10000/10000 [==============================] - 31s 3ms/step - reward: 1.0000\n",
            "310 episodes - episode_reward: 32.365 [8.000, 312.000] - loss: 0.003 - mse: 0.997 - mean_q: 1.445\n",
            "\n",
            "Interval 3 (20000 steps performed)\n",
            "10000/10000 [==============================] - 31s 3ms/step - reward: 1.0000\n",
            "307 episodes - episode_reward: 32.433 [8.000, 296.000] - loss: 0.003 - mse: 1.006 - mean_q: 1.444\n",
            "\n",
            "Interval 4 (30000 steps performed)\n",
            "10000/10000 [==============================] - 32s 3ms/step - reward: 1.0000\n",
            "296 episodes - episode_reward: 33.639 [8.000, 500.000] - loss: 0.002 - mse: 1.004 - mean_q: 1.446\n",
            "\n",
            "Interval 5 (40000 steps performed)\n",
            "10000/10000 [==============================] - 31s 3ms/step - reward: 1.0000\n",
            "done, took 157.973 seconds\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.callbacks.History at 0x7f2dcaf44e80>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ikHuz4zZFh9s",
        "outputId": "1f516314-83b1-4203-81a2-c957ffde9032",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "#Fitting and testing our agent model for 500 episodes\n",
        "scores = sarsa.test(env, nb_episodes = 500, visualize= False)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Testing for 500 episodes ...\n",
            "Episode 1: reward: 165.000, steps: 165\n",
            "Episode 2: reward: 141.000, steps: 141\n",
            "Episode 3: reward: 144.000, steps: 144\n",
            "Episode 4: reward: 149.000, steps: 149\n",
            "Episode 5: reward: 155.000, steps: 155\n",
            "Episode 6: reward: 144.000, steps: 144\n",
            "Episode 7: reward: 142.000, steps: 142\n",
            "Episode 8: reward: 151.000, steps: 151\n",
            "Episode 9: reward: 162.000, steps: 162\n",
            "Episode 10: reward: 152.000, steps: 152\n",
            "Episode 11: reward: 135.000, steps: 135\n",
            "Episode 12: reward: 151.000, steps: 151\n",
            "Episode 13: reward: 149.000, steps: 149\n",
            "Episode 14: reward: 149.000, steps: 149\n",
            "Episode 15: reward: 161.000, steps: 161\n",
            "Episode 16: reward: 158.000, steps: 158\n",
            "Episode 17: reward: 168.000, steps: 168\n",
            "Episode 18: reward: 148.000, steps: 148\n",
            "Episode 19: reward: 161.000, steps: 161\n",
            "Episode 20: reward: 161.000, steps: 161\n",
            "Episode 21: reward: 149.000, steps: 149\n",
            "Episode 22: reward: 150.000, steps: 150\n",
            "Episode 23: reward: 154.000, steps: 154\n",
            "Episode 24: reward: 161.000, steps: 161\n",
            "Episode 25: reward: 161.000, steps: 161\n",
            "Episode 26: reward: 165.000, steps: 165\n",
            "Episode 27: reward: 151.000, steps: 151\n",
            "Episode 28: reward: 151.000, steps: 151\n",
            "Episode 29: reward: 152.000, steps: 152\n",
            "Episode 30: reward: 144.000, steps: 144\n",
            "Episode 31: reward: 158.000, steps: 158\n",
            "Episode 32: reward: 143.000, steps: 143\n",
            "Episode 33: reward: 157.000, steps: 157\n",
            "Episode 34: reward: 148.000, steps: 148\n",
            "Episode 35: reward: 146.000, steps: 146\n",
            "Episode 36: reward: 148.000, steps: 148\n",
            "Episode 37: reward: 143.000, steps: 143\n",
            "Episode 38: reward: 159.000, steps: 159\n",
            "Episode 39: reward: 156.000, steps: 156\n",
            "Episode 40: reward: 162.000, steps: 162\n",
            "Episode 41: reward: 165.000, steps: 165\n",
            "Episode 42: reward: 145.000, steps: 145\n",
            "Episode 43: reward: 147.000, steps: 147\n",
            "Episode 44: reward: 164.000, steps: 164\n",
            "Episode 45: reward: 160.000, steps: 160\n",
            "Episode 46: reward: 145.000, steps: 145\n",
            "Episode 47: reward: 148.000, steps: 148\n",
            "Episode 48: reward: 153.000, steps: 153\n",
            "Episode 49: reward: 153.000, steps: 153\n",
            "Episode 50: reward: 138.000, steps: 138\n",
            "Episode 51: reward: 146.000, steps: 146\n",
            "Episode 52: reward: 157.000, steps: 157\n",
            "Episode 53: reward: 144.000, steps: 144\n",
            "Episode 54: reward: 150.000, steps: 150\n",
            "Episode 55: reward: 146.000, steps: 146\n",
            "Episode 56: reward: 142.000, steps: 142\n",
            "Episode 57: reward: 156.000, steps: 156\n",
            "Episode 58: reward: 157.000, steps: 157\n",
            "Episode 59: reward: 158.000, steps: 158\n",
            "Episode 60: reward: 147.000, steps: 147\n",
            "Episode 61: reward: 150.000, steps: 150\n",
            "Episode 62: reward: 146.000, steps: 146\n",
            "Episode 63: reward: 139.000, steps: 139\n",
            "Episode 64: reward: 154.000, steps: 154\n",
            "Episode 65: reward: 146.000, steps: 146\n",
            "Episode 66: reward: 153.000, steps: 153\n",
            "Episode 67: reward: 141.000, steps: 141\n",
            "Episode 68: reward: 143.000, steps: 143\n",
            "Episode 69: reward: 150.000, steps: 150\n",
            "Episode 70: reward: 145.000, steps: 145\n",
            "Episode 71: reward: 146.000, steps: 146\n",
            "Episode 72: reward: 154.000, steps: 154\n",
            "Episode 73: reward: 155.000, steps: 155\n",
            "Episode 74: reward: 145.000, steps: 145\n",
            "Episode 75: reward: 151.000, steps: 151\n",
            "Episode 76: reward: 153.000, steps: 153\n",
            "Episode 77: reward: 140.000, steps: 140\n",
            "Episode 78: reward: 148.000, steps: 148\n",
            "Episode 79: reward: 139.000, steps: 139\n",
            "Episode 80: reward: 150.000, steps: 150\n",
            "Episode 81: reward: 169.000, steps: 169\n",
            "Episode 82: reward: 145.000, steps: 145\n",
            "Episode 83: reward: 140.000, steps: 140\n",
            "Episode 84: reward: 143.000, steps: 143\n",
            "Episode 85: reward: 145.000, steps: 145\n",
            "Episode 86: reward: 157.000, steps: 157\n",
            "Episode 87: reward: 155.000, steps: 155\n",
            "Episode 88: reward: 156.000, steps: 156\n",
            "Episode 89: reward: 153.000, steps: 153\n",
            "Episode 90: reward: 145.000, steps: 145\n",
            "Episode 91: reward: 151.000, steps: 151\n",
            "Episode 92: reward: 162.000, steps: 162\n",
            "Episode 93: reward: 153.000, steps: 153\n",
            "Episode 94: reward: 149.000, steps: 149\n",
            "Episode 95: reward: 137.000, steps: 137\n",
            "Episode 96: reward: 156.000, steps: 156\n",
            "Episode 97: reward: 156.000, steps: 156\n",
            "Episode 98: reward: 165.000, steps: 165\n",
            "Episode 99: reward: 161.000, steps: 161\n",
            "Episode 100: reward: 150.000, steps: 150\n",
            "Episode 101: reward: 160.000, steps: 160\n",
            "Episode 102: reward: 148.000, steps: 148\n",
            "Episode 103: reward: 150.000, steps: 150\n",
            "Episode 104: reward: 163.000, steps: 163\n",
            "Episode 105: reward: 155.000, steps: 155\n",
            "Episode 106: reward: 149.000, steps: 149\n",
            "Episode 107: reward: 143.000, steps: 143\n",
            "Episode 108: reward: 154.000, steps: 154\n",
            "Episode 109: reward: 158.000, steps: 158\n",
            "Episode 110: reward: 144.000, steps: 144\n",
            "Episode 111: reward: 146.000, steps: 146\n",
            "Episode 112: reward: 153.000, steps: 153\n",
            "Episode 113: reward: 142.000, steps: 142\n",
            "Episode 114: reward: 138.000, steps: 138\n",
            "Episode 115: reward: 154.000, steps: 154\n",
            "Episode 116: reward: 155.000, steps: 155\n",
            "Episode 117: reward: 149.000, steps: 149\n",
            "Episode 118: reward: 153.000, steps: 153\n",
            "Episode 119: reward: 160.000, steps: 160\n",
            "Episode 120: reward: 137.000, steps: 137\n",
            "Episode 121: reward: 149.000, steps: 149\n",
            "Episode 122: reward: 153.000, steps: 153\n",
            "Episode 123: reward: 155.000, steps: 155\n",
            "Episode 124: reward: 151.000, steps: 151\n",
            "Episode 125: reward: 157.000, steps: 157\n",
            "Episode 126: reward: 136.000, steps: 136\n",
            "Episode 127: reward: 146.000, steps: 146\n",
            "Episode 128: reward: 173.000, steps: 173\n",
            "Episode 129: reward: 148.000, steps: 148\n",
            "Episode 130: reward: 181.000, steps: 181\n",
            "Episode 131: reward: 157.000, steps: 157\n",
            "Episode 132: reward: 152.000, steps: 152\n",
            "Episode 133: reward: 162.000, steps: 162\n",
            "Episode 134: reward: 144.000, steps: 144\n",
            "Episode 135: reward: 137.000, steps: 137\n",
            "Episode 136: reward: 146.000, steps: 146\n",
            "Episode 137: reward: 150.000, steps: 150\n",
            "Episode 138: reward: 151.000, steps: 151\n",
            "Episode 139: reward: 153.000, steps: 153\n",
            "Episode 140: reward: 156.000, steps: 156\n",
            "Episode 141: reward: 171.000, steps: 171\n",
            "Episode 142: reward: 146.000, steps: 146\n",
            "Episode 143: reward: 161.000, steps: 161\n",
            "Episode 144: reward: 141.000, steps: 141\n",
            "Episode 145: reward: 142.000, steps: 142\n",
            "Episode 146: reward: 165.000, steps: 165\n",
            "Episode 147: reward: 149.000, steps: 149\n",
            "Episode 148: reward: 137.000, steps: 137\n",
            "Episode 149: reward: 139.000, steps: 139\n",
            "Episode 150: reward: 156.000, steps: 156\n",
            "Episode 151: reward: 161.000, steps: 161\n",
            "Episode 152: reward: 160.000, steps: 160\n",
            "Episode 153: reward: 168.000, steps: 168\n",
            "Episode 154: reward: 154.000, steps: 154\n",
            "Episode 155: reward: 158.000, steps: 158\n",
            "Episode 156: reward: 146.000, steps: 146\n",
            "Episode 157: reward: 156.000, steps: 156\n",
            "Episode 158: reward: 146.000, steps: 146\n",
            "Episode 159: reward: 152.000, steps: 152\n",
            "Episode 160: reward: 148.000, steps: 148\n",
            "Episode 161: reward: 146.000, steps: 146\n",
            "Episode 162: reward: 162.000, steps: 162\n",
            "Episode 163: reward: 145.000, steps: 145\n",
            "Episode 164: reward: 152.000, steps: 152\n",
            "Episode 165: reward: 145.000, steps: 145\n",
            "Episode 166: reward: 149.000, steps: 149\n",
            "Episode 167: reward: 144.000, steps: 144\n",
            "Episode 168: reward: 144.000, steps: 144\n",
            "Episode 169: reward: 152.000, steps: 152\n",
            "Episode 170: reward: 156.000, steps: 156\n",
            "Episode 171: reward: 138.000, steps: 138\n",
            "Episode 172: reward: 137.000, steps: 137\n",
            "Episode 173: reward: 134.000, steps: 134\n",
            "Episode 174: reward: 152.000, steps: 152\n",
            "Episode 175: reward: 152.000, steps: 152\n",
            "Episode 176: reward: 147.000, steps: 147\n",
            "Episode 177: reward: 147.000, steps: 147\n",
            "Episode 178: reward: 143.000, steps: 143\n",
            "Episode 179: reward: 146.000, steps: 146\n",
            "Episode 180: reward: 138.000, steps: 138\n",
            "Episode 181: reward: 144.000, steps: 144\n",
            "Episode 182: reward: 145.000, steps: 145\n",
            "Episode 183: reward: 133.000, steps: 133\n",
            "Episode 184: reward: 147.000, steps: 147\n",
            "Episode 185: reward: 154.000, steps: 154\n",
            "Episode 186: reward: 142.000, steps: 142\n",
            "Episode 187: reward: 165.000, steps: 165\n",
            "Episode 188: reward: 147.000, steps: 147\n",
            "Episode 189: reward: 156.000, steps: 156\n",
            "Episode 190: reward: 148.000, steps: 148\n",
            "Episode 191: reward: 134.000, steps: 134\n",
            "Episode 192: reward: 160.000, steps: 160\n",
            "Episode 193: reward: 157.000, steps: 157\n",
            "Episode 194: reward: 145.000, steps: 145\n",
            "Episode 195: reward: 159.000, steps: 159\n",
            "Episode 196: reward: 145.000, steps: 145\n",
            "Episode 197: reward: 156.000, steps: 156\n",
            "Episode 198: reward: 160.000, steps: 160\n",
            "Episode 199: reward: 156.000, steps: 156\n",
            "Episode 200: reward: 167.000, steps: 167\n",
            "Episode 201: reward: 155.000, steps: 155\n",
            "Episode 202: reward: 144.000, steps: 144\n",
            "Episode 203: reward: 151.000, steps: 151\n",
            "Episode 204: reward: 149.000, steps: 149\n",
            "Episode 205: reward: 137.000, steps: 137\n",
            "Episode 206: reward: 146.000, steps: 146\n",
            "Episode 207: reward: 145.000, steps: 145\n",
            "Episode 208: reward: 161.000, steps: 161\n",
            "Episode 209: reward: 137.000, steps: 137\n",
            "Episode 210: reward: 150.000, steps: 150\n",
            "Episode 211: reward: 172.000, steps: 172\n",
            "Episode 212: reward: 155.000, steps: 155\n",
            "Episode 213: reward: 155.000, steps: 155\n",
            "Episode 214: reward: 164.000, steps: 164\n",
            "Episode 215: reward: 157.000, steps: 157\n",
            "Episode 216: reward: 146.000, steps: 146\n",
            "Episode 217: reward: 167.000, steps: 167\n",
            "Episode 218: reward: 157.000, steps: 157\n",
            "Episode 219: reward: 160.000, steps: 160\n",
            "Episode 220: reward: 149.000, steps: 149\n",
            "Episode 221: reward: 160.000, steps: 160\n",
            "Episode 222: reward: 154.000, steps: 154\n",
            "Episode 223: reward: 144.000, steps: 144\n",
            "Episode 224: reward: 157.000, steps: 157\n",
            "Episode 225: reward: 162.000, steps: 162\n",
            "Episode 226: reward: 152.000, steps: 152\n",
            "Episode 227: reward: 149.000, steps: 149\n",
            "Episode 228: reward: 142.000, steps: 142\n",
            "Episode 229: reward: 147.000, steps: 147\n",
            "Episode 230: reward: 154.000, steps: 154\n",
            "Episode 231: reward: 162.000, steps: 162\n",
            "Episode 232: reward: 162.000, steps: 162\n",
            "Episode 233: reward: 141.000, steps: 141\n",
            "Episode 234: reward: 160.000, steps: 160\n",
            "Episode 235: reward: 142.000, steps: 142\n",
            "Episode 236: reward: 154.000, steps: 154\n",
            "Episode 237: reward: 166.000, steps: 166\n",
            "Episode 238: reward: 147.000, steps: 147\n",
            "Episode 239: reward: 141.000, steps: 141\n",
            "Episode 240: reward: 142.000, steps: 142\n",
            "Episode 241: reward: 131.000, steps: 131\n",
            "Episode 242: reward: 157.000, steps: 157\n",
            "Episode 243: reward: 145.000, steps: 145\n",
            "Episode 244: reward: 139.000, steps: 139\n",
            "Episode 245: reward: 152.000, steps: 152\n",
            "Episode 246: reward: 161.000, steps: 161\n",
            "Episode 247: reward: 151.000, steps: 151\n",
            "Episode 248: reward: 140.000, steps: 140\n",
            "Episode 249: reward: 161.000, steps: 161\n",
            "Episode 250: reward: 157.000, steps: 157\n",
            "Episode 251: reward: 149.000, steps: 149\n",
            "Episode 252: reward: 166.000, steps: 166\n",
            "Episode 253: reward: 135.000, steps: 135\n",
            "Episode 254: reward: 148.000, steps: 148\n",
            "Episode 255: reward: 148.000, steps: 148\n",
            "Episode 256: reward: 160.000, steps: 160\n",
            "Episode 257: reward: 165.000, steps: 165\n",
            "Episode 258: reward: 157.000, steps: 157\n",
            "Episode 259: reward: 162.000, steps: 162\n",
            "Episode 260: reward: 134.000, steps: 134\n",
            "Episode 261: reward: 152.000, steps: 152\n",
            "Episode 262: reward: 154.000, steps: 154\n",
            "Episode 263: reward: 152.000, steps: 152\n",
            "Episode 264: reward: 150.000, steps: 150\n",
            "Episode 265: reward: 157.000, steps: 157\n",
            "Episode 266: reward: 144.000, steps: 144\n",
            "Episode 267: reward: 158.000, steps: 158\n",
            "Episode 268: reward: 153.000, steps: 153\n",
            "Episode 269: reward: 164.000, steps: 164\n",
            "Episode 270: reward: 38.000, steps: 38\n",
            "Episode 271: reward: 151.000, steps: 151\n",
            "Episode 272: reward: 148.000, steps: 148\n",
            "Episode 273: reward: 141.000, steps: 141\n",
            "Episode 274: reward: 135.000, steps: 135\n",
            "Episode 275: reward: 151.000, steps: 151\n",
            "Episode 276: reward: 160.000, steps: 160\n",
            "Episode 277: reward: 154.000, steps: 154\n",
            "Episode 278: reward: 148.000, steps: 148\n",
            "Episode 279: reward: 156.000, steps: 156\n",
            "Episode 280: reward: 150.000, steps: 150\n",
            "Episode 281: reward: 139.000, steps: 139\n",
            "Episode 282: reward: 148.000, steps: 148\n",
            "Episode 283: reward: 148.000, steps: 148\n",
            "Episode 284: reward: 145.000, steps: 145\n",
            "Episode 285: reward: 142.000, steps: 142\n",
            "Episode 286: reward: 149.000, steps: 149\n",
            "Episode 287: reward: 146.000, steps: 146\n",
            "Episode 288: reward: 149.000, steps: 149\n",
            "Episode 289: reward: 155.000, steps: 155\n",
            "Episode 290: reward: 153.000, steps: 153\n",
            "Episode 291: reward: 150.000, steps: 150\n",
            "Episode 292: reward: 151.000, steps: 151\n",
            "Episode 293: reward: 156.000, steps: 156\n",
            "Episode 294: reward: 153.000, steps: 153\n",
            "Episode 295: reward: 141.000, steps: 141\n",
            "Episode 296: reward: 159.000, steps: 159\n",
            "Episode 297: reward: 169.000, steps: 169\n",
            "Episode 298: reward: 164.000, steps: 164\n",
            "Episode 299: reward: 138.000, steps: 138\n",
            "Episode 300: reward: 159.000, steps: 159\n",
            "Episode 301: reward: 141.000, steps: 141\n",
            "Episode 302: reward: 152.000, steps: 152\n",
            "Episode 303: reward: 147.000, steps: 147\n",
            "Episode 304: reward: 159.000, steps: 159\n",
            "Episode 305: reward: 144.000, steps: 144\n",
            "Episode 306: reward: 154.000, steps: 154\n",
            "Episode 307: reward: 154.000, steps: 154\n",
            "Episode 308: reward: 156.000, steps: 156\n",
            "Episode 309: reward: 149.000, steps: 149\n",
            "Episode 310: reward: 161.000, steps: 161\n",
            "Episode 311: reward: 159.000, steps: 159\n",
            "Episode 312: reward: 149.000, steps: 149\n",
            "Episode 313: reward: 152.000, steps: 152\n",
            "Episode 314: reward: 147.000, steps: 147\n",
            "Episode 315: reward: 157.000, steps: 157\n",
            "Episode 316: reward: 154.000, steps: 154\n",
            "Episode 317: reward: 171.000, steps: 171\n",
            "Episode 318: reward: 157.000, steps: 157\n",
            "Episode 319: reward: 147.000, steps: 147\n",
            "Episode 320: reward: 154.000, steps: 154\n",
            "Episode 321: reward: 140.000, steps: 140\n",
            "Episode 322: reward: 151.000, steps: 151\n",
            "Episode 323: reward: 147.000, steps: 147\n",
            "Episode 324: reward: 149.000, steps: 149\n",
            "Episode 325: reward: 156.000, steps: 156\n",
            "Episode 326: reward: 148.000, steps: 148\n",
            "Episode 327: reward: 146.000, steps: 146\n",
            "Episode 328: reward: 148.000, steps: 148\n",
            "Episode 329: reward: 158.000, steps: 158\n",
            "Episode 330: reward: 146.000, steps: 146\n",
            "Episode 331: reward: 150.000, steps: 150\n",
            "Episode 332: reward: 159.000, steps: 159\n",
            "Episode 333: reward: 140.000, steps: 140\n",
            "Episode 334: reward: 148.000, steps: 148\n",
            "Episode 335: reward: 154.000, steps: 154\n",
            "Episode 336: reward: 154.000, steps: 154\n",
            "Episode 337: reward: 147.000, steps: 147\n",
            "Episode 338: reward: 146.000, steps: 146\n",
            "Episode 339: reward: 139.000, steps: 139\n",
            "Episode 340: reward: 152.000, steps: 152\n",
            "Episode 341: reward: 139.000, steps: 139\n",
            "Episode 342: reward: 145.000, steps: 145\n",
            "Episode 343: reward: 142.000, steps: 142\n",
            "Episode 344: reward: 161.000, steps: 161\n",
            "Episode 345: reward: 144.000, steps: 144\n",
            "Episode 346: reward: 165.000, steps: 165\n",
            "Episode 347: reward: 150.000, steps: 150\n",
            "Episode 348: reward: 149.000, steps: 149\n",
            "Episode 349: reward: 152.000, steps: 152\n",
            "Episode 350: reward: 171.000, steps: 171\n",
            "Episode 351: reward: 150.000, steps: 150\n",
            "Episode 352: reward: 154.000, steps: 154\n",
            "Episode 353: reward: 154.000, steps: 154\n",
            "Episode 354: reward: 157.000, steps: 157\n",
            "Episode 355: reward: 144.000, steps: 144\n",
            "Episode 356: reward: 155.000, steps: 155\n",
            "Episode 357: reward: 145.000, steps: 145\n",
            "Episode 358: reward: 155.000, steps: 155\n",
            "Episode 359: reward: 161.000, steps: 161\n",
            "Episode 360: reward: 152.000, steps: 152\n",
            "Episode 361: reward: 161.000, steps: 161\n",
            "Episode 362: reward: 152.000, steps: 152\n",
            "Episode 363: reward: 143.000, steps: 143\n",
            "Episode 364: reward: 141.000, steps: 141\n",
            "Episode 365: reward: 151.000, steps: 151\n",
            "Episode 366: reward: 149.000, steps: 149\n",
            "Episode 367: reward: 139.000, steps: 139\n",
            "Episode 368: reward: 156.000, steps: 156\n",
            "Episode 369: reward: 134.000, steps: 134\n",
            "Episode 370: reward: 160.000, steps: 160\n",
            "Episode 371: reward: 173.000, steps: 173\n",
            "Episode 372: reward: 149.000, steps: 149\n",
            "Episode 373: reward: 143.000, steps: 143\n",
            "Episode 374: reward: 154.000, steps: 154\n",
            "Episode 375: reward: 161.000, steps: 161\n",
            "Episode 376: reward: 158.000, steps: 158\n",
            "Episode 377: reward: 162.000, steps: 162\n",
            "Episode 378: reward: 161.000, steps: 161\n",
            "Episode 379: reward: 159.000, steps: 159\n",
            "Episode 380: reward: 150.000, steps: 150\n",
            "Episode 381: reward: 148.000, steps: 148\n",
            "Episode 382: reward: 163.000, steps: 163\n",
            "Episode 383: reward: 165.000, steps: 165\n",
            "Episode 384: reward: 146.000, steps: 146\n",
            "Episode 385: reward: 151.000, steps: 151\n",
            "Episode 386: reward: 142.000, steps: 142\n",
            "Episode 387: reward: 149.000, steps: 149\n",
            "Episode 388: reward: 145.000, steps: 145\n",
            "Episode 389: reward: 139.000, steps: 139\n",
            "Episode 390: reward: 157.000, steps: 157\n",
            "Episode 391: reward: 150.000, steps: 150\n",
            "Episode 392: reward: 157.000, steps: 157\n",
            "Episode 393: reward: 165.000, steps: 165\n",
            "Episode 394: reward: 139.000, steps: 139\n",
            "Episode 395: reward: 144.000, steps: 144\n",
            "Episode 396: reward: 143.000, steps: 143\n",
            "Episode 397: reward: 145.000, steps: 145\n",
            "Episode 398: reward: 146.000, steps: 146\n",
            "Episode 399: reward: 129.000, steps: 129\n",
            "Episode 400: reward: 160.000, steps: 160\n",
            "Episode 401: reward: 156.000, steps: 156\n",
            "Episode 402: reward: 155.000, steps: 155\n",
            "Episode 403: reward: 143.000, steps: 143\n",
            "Episode 404: reward: 156.000, steps: 156\n",
            "Episode 405: reward: 161.000, steps: 161\n",
            "Episode 406: reward: 144.000, steps: 144\n",
            "Episode 407: reward: 163.000, steps: 163\n",
            "Episode 408: reward: 155.000, steps: 155\n",
            "Episode 409: reward: 161.000, steps: 161\n",
            "Episode 410: reward: 156.000, steps: 156\n",
            "Episode 411: reward: 144.000, steps: 144\n",
            "Episode 412: reward: 148.000, steps: 148\n",
            "Episode 413: reward: 167.000, steps: 167\n",
            "Episode 414: reward: 144.000, steps: 144\n",
            "Episode 415: reward: 147.000, steps: 147\n",
            "Episode 416: reward: 145.000, steps: 145\n",
            "Episode 417: reward: 170.000, steps: 170\n",
            "Episode 418: reward: 157.000, steps: 157\n",
            "Episode 419: reward: 144.000, steps: 144\n",
            "Episode 420: reward: 152.000, steps: 152\n",
            "Episode 421: reward: 149.000, steps: 149\n",
            "Episode 422: reward: 140.000, steps: 140\n",
            "Episode 423: reward: 153.000, steps: 153\n",
            "Episode 424: reward: 149.000, steps: 149\n",
            "Episode 425: reward: 150.000, steps: 150\n",
            "Episode 426: reward: 147.000, steps: 147\n",
            "Episode 427: reward: 146.000, steps: 146\n",
            "Episode 428: reward: 139.000, steps: 139\n",
            "Episode 429: reward: 136.000, steps: 136\n",
            "Episode 430: reward: 157.000, steps: 157\n",
            "Episode 431: reward: 133.000, steps: 133\n",
            "Episode 432: reward: 151.000, steps: 151\n",
            "Episode 433: reward: 142.000, steps: 142\n",
            "Episode 434: reward: 141.000, steps: 141\n",
            "Episode 435: reward: 150.000, steps: 150\n",
            "Episode 436: reward: 156.000, steps: 156\n",
            "Episode 437: reward: 143.000, steps: 143\n",
            "Episode 438: reward: 146.000, steps: 146\n",
            "Episode 439: reward: 142.000, steps: 142\n",
            "Episode 440: reward: 150.000, steps: 150\n",
            "Episode 441: reward: 140.000, steps: 140\n",
            "Episode 442: reward: 166.000, steps: 166\n",
            "Episode 443: reward: 137.000, steps: 137\n",
            "Episode 444: reward: 155.000, steps: 155\n",
            "Episode 445: reward: 167.000, steps: 167\n",
            "Episode 446: reward: 144.000, steps: 144\n",
            "Episode 447: reward: 157.000, steps: 157\n",
            "Episode 448: reward: 168.000, steps: 168\n",
            "Episode 449: reward: 160.000, steps: 160\n",
            "Episode 450: reward: 146.000, steps: 146\n",
            "Episode 451: reward: 163.000, steps: 163\n",
            "Episode 452: reward: 153.000, steps: 153\n",
            "Episode 453: reward: 144.000, steps: 144\n",
            "Episode 454: reward: 137.000, steps: 137\n",
            "Episode 455: reward: 143.000, steps: 143\n",
            "Episode 456: reward: 146.000, steps: 146\n",
            "Episode 457: reward: 152.000, steps: 152\n",
            "Episode 458: reward: 143.000, steps: 143\n",
            "Episode 459: reward: 156.000, steps: 156\n",
            "Episode 460: reward: 145.000, steps: 145\n",
            "Episode 461: reward: 150.000, steps: 150\n",
            "Episode 462: reward: 144.000, steps: 144\n",
            "Episode 463: reward: 132.000, steps: 132\n",
            "Episode 464: reward: 144.000, steps: 144\n",
            "Episode 465: reward: 143.000, steps: 143\n",
            "Episode 466: reward: 155.000, steps: 155\n",
            "Episode 467: reward: 160.000, steps: 160\n",
            "Episode 468: reward: 149.000, steps: 149\n",
            "Episode 469: reward: 151.000, steps: 151\n",
            "Episode 470: reward: 150.000, steps: 150\n",
            "Episode 471: reward: 152.000, steps: 152\n",
            "Episode 472: reward: 167.000, steps: 167\n",
            "Episode 473: reward: 149.000, steps: 149\n",
            "Episode 474: reward: 159.000, steps: 159\n",
            "Episode 475: reward: 165.000, steps: 165\n",
            "Episode 476: reward: 173.000, steps: 173\n",
            "Episode 477: reward: 157.000, steps: 157\n",
            "Episode 478: reward: 160.000, steps: 160\n",
            "Episode 479: reward: 138.000, steps: 138\n",
            "Episode 480: reward: 145.000, steps: 145\n",
            "Episode 481: reward: 148.000, steps: 148\n",
            "Episode 482: reward: 160.000, steps: 160\n",
            "Episode 483: reward: 159.000, steps: 159\n",
            "Episode 484: reward: 137.000, steps: 137\n",
            "Episode 485: reward: 149.000, steps: 149\n",
            "Episode 486: reward: 137.000, steps: 137\n",
            "Episode 487: reward: 164.000, steps: 164\n",
            "Episode 488: reward: 152.000, steps: 152\n",
            "Episode 489: reward: 147.000, steps: 147\n",
            "Episode 490: reward: 143.000, steps: 143\n",
            "Episode 491: reward: 147.000, steps: 147\n",
            "Episode 492: reward: 163.000, steps: 163\n",
            "Episode 493: reward: 144.000, steps: 144\n",
            "Episode 494: reward: 167.000, steps: 167\n",
            "Episode 495: reward: 149.000, steps: 149\n",
            "Episode 496: reward: 143.000, steps: 143\n",
            "Episode 497: reward: 148.000, steps: 148\n",
            "Episode 498: reward: 150.000, steps: 150\n",
            "Episode 499: reward: 153.000, steps: 153\n",
            "Episode 500: reward: 149.000, steps: 149\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "dsfmG3H5fpLy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "d88dbea1-d881-469a-e969-a1b7a605eaf8"
      },
      "source": [
        "#Importing the necessary plotting libraries\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import seaborn as sns\n",
        "sns.set()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pENuEDkAFjRv",
        "outputId": "64b7bec8-efb0-4831-c900-4222e3a04c29",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        }
      },
      "source": [
        "#Visualizing our resulted rewards\n",
        "plt.plot(scores.history['episode_reward'])\n",
        "plt.xlabel('Episode')\n",
        "plt.ylabel('Testing total reward')\n",
        "plt.title('Total rewards over all episodes in testing') \n",
        "plt.show()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEcCAYAAADUX4MJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOydeZjUVNq3f5XU0ju90E03NJvIjjQN3Q1CszUqINvIvArDDM6AjoPKIN8owjsioOKrgIqKqCCKzsiIOo4oiIKDgIrIvu87Te/0vlVVKsn3R9VJJamkOtU7cO7r4qIrlUpOkpPzrOc5JlEURVAoFAqFUgNMUzeAQqFQKDcGVGBQKBQKxRBUYFAoFArFEFRgUCgUCsUQVGBQKBQKxRBUYFAoFArFEFRg3IJ07doVV65caepmBMSePXswZMiQpm5GvbFixQo89dRTAIBr166ha9eucLlc9X6eMWPGYM+ePfV6zHnz5mH58uX1eswFCxZg5cqV9XrM+uDdd9/FM88809TNaDaYm7oBFC/JycnS39XV1bBarWBZFgDw3HPPYfz48T6/2bNnD+bMmYMff/yx0dpJuXH45ptvmroJhnj++edr/dupU6di/PjxuP/+++vUBq13acaMGXU65s0GFRjNiEOHDkl/Z2RkYPHixRg4cGCTtUcURYiiCIZpXEPU5XLBbL5xuuaN1l4KpbZQl9QNgNPpxIsvvoj09HSkp6fjxRdfhNPpRFVVFf785z8jPz8fycnJSE5ORl5eHo4ePYpJkyYhJSUF6enpeP755+F0Og2da+rUqVi+fDkmT56MpKQkZGZm4sKFC5g2bRrS0tIwcuRIbN68GQCQmZmJlJQUCIIAAJg/fz7uvPNO6Vhz5szBhx9+CAD44osvMHr0aCQnJ2PEiBFYv369tB9xN61evRqDBg3C//7v/8Jut2PevHlITU3Fvffei2PHjinauXr1agwePBjJyckYOXIkdu/erXk95eXlePrppzFgwAAMHz4cb7/9NgRBgNPpREpKCs6ePSvtW1RUhN69e6OwsBAAsH37dkyYMAEpKSmYPHkyTp8+Le2bkZGB1atXY9y4cejTp4+mO2nx4sUYOnQo+vbti4kTJ2L//v2GnoGavLw8/PWvf8WAAQOQkZGBf/zjH9J3K1aswKxZszB79mwkJyfjvvvu82nnL7/8AgA4evQoJk6ciL59+2LgwIF46aWXpP22bduGMWPGICUlBVOnTsWFCxek706ePIn77rsPycnJmD17NhwOh6J9/u6T0eckd3OR/vDBBx/gzjvvRHp6Or744gvN3y1fvhz79+/H888/j+TkZMlS0euzALBz507ce++9SE5OxuDBg/H+++/rvktarsMvv/wSw4YNQ//+/fHOO+9Ix7Xb7Zg7dy5SU1MxevRovPfeezeVGxUAIFKaJcOHDxd37doliqIovv766+L9998vXr9+XSwsLBQnTZokLl++XBRFUfz111/FwYMHK3577Ngx8dChQyLHcWJmZqY4atQoce3atdL3Xbp0ES9fvqx53j/84Q/i0KFDxbNnz4ocx4llZWXikCFDxH//+98ix3HiiRMnxLS0NPHcuXOiKIri0KFDxWPHjomiKIr33HOPmJGRIZ4/f1767sSJE6IoiuL27dvFK1euiIIgiHv27BF79+4tHj9+XLqG7t27i0uXLhUdDodYXV0tLlu2TPzd734nFhcXi9nZ2eKYMWOk67xw4YI4ZMgQMTc3VxRFUczMzBSvXLmieT1z5swRZ8yYIZaXl4uZmZniPffcI3722WeiKIrivHnzxNdee03a9+OPPxanT58uiqIonjhxQhwwYIB4+PBh0eVyif/5z3/E4cOHiw6HQ3o+48ePF7Ozs8Xq6mrNc2/YsEEsKioSOY4T33//fXHgwIGi3W4XRVEU33zzTfHJJ5+U2t+lSxeR4zifY/A8L953333iihUrRIfDIV69elXMyMgQf/zxR+k4PXr0EL/99lvR6XSKa9asEYcPHy46nU6pnaQfPfDAA+KXX34piqIoVlRUiIcOHRJFURQvXrwoJiUliT///LPodDrF1atXi3fddZfocDhEh8MhDhs2TFy7dq3odDrFb7/9VuzRo4d03/zdp0Ce09y5c6Vjkv7w+uuvi06nU9yxY4fYu3dvsaSkRPO3f/jDH6RnKoqiWFlZ6bfPDho0SNy3b58oiqJYUlKi6Ifqd0nrOT3zzDNidXW1eOrUKbFnz55Sf1+2bJn4+9//XiwpKRFzcnLEsWPH+hzvRodaGDcAGzduxOOPP46YmBhER0fj8ccfx9dff627f69evdCnTx+YzWYkJiZi0qRJ2Ldvn+Hz3XfffejcuTPMZjN++ukntGnTBr/97W9hNpvRo0cPjBw5Et999x0AIDU1Ffv27UNBQQEAYOTIkdi7dy8yMzNRUVGBbt26AQCGDRuGdu3awWQyIS0tDYMGDVJo3AzDYNasWbBarQgKCsK3336LGTNmIDIyEgkJCZg6daq0L8uycDqduHDhAjiOQ2JiItq1a+dzHTzPY/PmzXjyyScRFhaGxMRETJs2Tbp348aNU/j4N27ciHHjxgEAPv30U0yaNAlJSUlgWRb33XcfLBYLDh8+LO0/depUJCQkICgoSPM+TpgwAVFRUTCbzZg+fTqcTicuXbpk+DkAwLFjx1BUVISZM2fCarWibdu2eOCBBxQac8+ePTFq1ChYLBZMmzYNTqcTR44c8TmW2WzG1atXUVRUhNDQUPTp0wcAsHnzZgwdOhSDBg2CxWLBQw89BLvdjkOHDuHIkSPgOA5//OMfYbFYMGrUKNxxxx3SMf3dJ6PPSQuz2YzHH38cFosFQ4cORUhIiOF7t2PHDr991mw24/z586ioqECLFi3Qs2dPQ8clzJw5E0FBQejWrRu6desmWVTffvst/vKXv6BFixaIj4/Hgw8+GNBxbwSo4/UGID8/H61bt5Y+t27dGvn5+br7X7p0CS+//DKOHz+O6upq8Dwf0EuRkJAg/Z2VlYWjR48iJSVF2sbzvBSAT0tLw7Zt29CqVSukpqaif//++Oqrr2Cz2ZCSkiLFP3bu3ImVK1fi8uXLEAQBdrsdXbp0kY4ZFRUFm82muGZ5O+TX3759e/z973/HihUrcP78eaSnp2PevHlo1aqV4jqKi4vBcZzPvcvLywMA9O/fH3a7HUeOHEFMTAxOnz6Nu+66CwCQnZ2NDRs24OOPP5Z+y3Gc4r7L26fF+++/j3//+9/Iz8+HyWRCRUUFiouL/f5GTVZWFvLz833uv/xzfHy89DfDMGjVqpVm/3jxxRfx5ptvYvTo0UhMTMTMmTMxfPhwn/7FMAwSEhKQl5cHlmXRqlUrmEwm6Xv5vv7uU1pamqHnpEVkZKQiLhQcHIyqqqoafwfU3GfffPNNvPPOO3j11VfRtWtXPPnkk4qEk5po2bKlZrvUfVb+XG4WqMC4AYiLi0N2djY6d+4MAMjJyUFcXBwAKF5kwqJFi9CjRw+8+uqrCAsLw4cffogtW7YYPp/8mAkJCUhNTcXatWs1901NTcXSpUsRHx+P1NRU9OvXDwsXLoTNZkNqaioAdwxm1qxZWLJkCUaMGAGLxYLHHnsMoqxQsvo6YmNjkZOTo7hmOePGjcO4ceNQUVGBBQsW4JVXXsGyZcsU+0RFRcFisSA7Oxu33367dBwyYLEsi1GjRmHTpk1o2bIlhg0bhrCwMOm6Z8yYgUcffdTQfVKzf/9+rFmzBh9++CE6d+4MhmGQmpqquGYjJCQkIDExEVu3btXdJzc3V/pbEATk5eVJ/UNOhw4d8Nprr0EQBGzduhWzZs3Cnj17EBcXp4jliKIo3SeTyYS8vDyIoihdb3Z2Ntq2bSu1z999MvKc6pua+mzv3r3xzjvvgOM4rFu3DrNnz8bOnTv9Pk8jxMbGIjc3V+pr8udys0BdUjcAY8aMwTvvvIOioiIUFRVh5cqVkuskJiYGJSUlKC8vl/avrKxEaGgoQkNDceHCBXzyySe1PvewYcNw+fJlbNiwARzHgeM4HD16VAqKdujQATabDV9//TXS0tIQFhaGmJgYbNmyRSEwnE4noqOjYTabsXPnTuzatcvveUePHo3Vq1ejtLQUubm5+Oc//yl9d/HiRezevRtOpxNWqxU2m00zk4sIhOXLl6OiogJZWVlYu3atIj153Lhx+Pbbb7Fx40aMHTtW2n7//fdj/fr1OHLkCERRRFVVFXbs2IGKigpD962yshIsyyI6OhoulwtvvfWW4d/K6d27N0JDQ7F69WrY7XbwPI+zZ8/i6NGj0j4nTpzA1q1b4XK58NFHH8FqtSIpKcnnWF999RWKiorAMAwiIiIAuK2J0aNHY+fOndi9ezc4jsMHH3wAq9WK5ORkybX5j3/8AxzHYevWrYoEBH/3yehzqistW7ZEZmam9Nlfn3U6nfj6669RXl4Oi8WC0NBQqU1a71IgjB49GqtWrUJpaSny8vIUVtfNAhUYNwCPPfYYevXqhfHjx2P8+PHo2bMnHnvsMQBAp06dMGbMGNx1111ISUlBXl4e5s6di02bNqFv37549tlnce+999b63GFhYXj//fexefNmDB48GOnp6XjllVcUWVdpaWlSrIF8FkVRcoOFhYVh/vz5mD17NlJTU7Fp0yZkZGT4Pe/MmTPRunVrjBgxAtOnT8eECROk75xOJ1599VX0798f6enpKCoqwt/+9jfN4zz77LMIDg7GXXfdhSlTpmDs2LH47W9/K32flJSE4OBg5OfnKzJa7rjjDrzwwgt4/vnnkZqainvuuQf/+c9/DN+39PR0DB48GCNHjkRGRgZsNluNLiwtWJbFu+++i9OnT2PEiBEYMGAA5s+frxA+I0aMwObNm5GamoqvvvoKK1asgMVi8TnWTz/9hDFjxiA5ORkvvvgili9fjqCgINx2221YtmwZXnjhBQwYMADbt2/Hu+++C6vVCqvVihUrVuDLL79EWloaNm/ejLvvvtvQfQrkOdWFBx98UFJQFi9eXGOf/eqrr5CRkYG+ffti/fr1ksWj9S4FwuOPP474+HiMGDECf/rTnzBy5EhYrdZ6v96mxCQGaiNTKJRmw4oVK3DlyhW88sorTd0Uiop//etf2Lx5801laVALg0KhUOqB/Px8HDhwAIIg4OLFi1i7dq2URHGzQIPeFAqFUg9wHIeFCxfi2rVrCA8Px5gxYzBlypSmbla9Ql1SFAqFQjEEdUlRKBQKxRBUYFAoFArFEFRgUCgUCsUQN33Qu7i4EoIQeJgmJiYMhYWBT7S6kaHXfGtAr/nWoLbXzDAmREWFan530wsMQRBrJTDIb2816DXfGtBrvjWo72umLikKhUKhGIIKDAqFQqEYggoMCoVCoRiCCgwKhUKhGKJRBMaSJUuQkZGBrl27Kurub9++Hb/5zW8wYcIEjB8/XlHz/9KlS5g0aRJGjhyJSZMm4fLly43RVAqFQqHo0CgCY8SIEVi3bh3atGkjbRNFEU8//TSWLl2Kr776CkuXLsXcuXMhCAIAYOHChZgyZQq2bNmCKVOmYMGCBY3RVAqFQqHo0CgCIyUlRXMtAIZhpMVKysvLERcXB4ZhUFhYiJMnT0oL2owdOxYnT55EUVFRYzSXUktKKhyY/vIPOHrhelM3hUKhNABNNg/DZDLh9ddfx2OPPYaQkBBUVlZi9erVALzLaLIsC8C9iExcXBxycnIQHR3dVE2m1MDlXLfw/+FgFnp3alnD3hQK5UajyQSGy+XCqlWr8Pbbb6Nfv344cOAAZs+ejW+++aZezxMTE1br38bGhtdjS24M6nLNkQWVAACr1XxD3bsbqa31Bb3mW4P6vuYmExinTp1Cfn4++vXrBwDo168fgoODceHCBbRp0wZ5eXngeR4sy4LneeTn59dqicvCwopazXaMjQ1HQUHt1va9UanrNZeVVQMA7A7uhrl39DnfGtBrNg7DmHQV7SZLq42Pj0dubi4uXrwIALhw4QIKCwvRrl07xMTEoHv37ti0aRMAYNOmTejevTt1RzV7TO7/br0KDBTKLUGjWBiLFy/G1q1bcf36dUybNg2RkZH45ptvsGjRIjzxxBMwmdwDzf/93/8hMjISALBo0SLMmzcPb7/9NiIiIrBkyZLGaCqlDpiIvKBrclEoNyU3/Yp71CVlnLpe8/FLhXjt0yPo0SEKT01OrseWNRz0Od8a0Gs2TrN0SVFuPoileHOrIBTKrQsVGJR6w9TUDaBQKA0KFRiUeoNYFje5l5NCuWWhAoNSbwgeQUHlBYVyc0IFBqVeOHm5CMs/OwKAZtVS6k5ZlRMPLfkBZzNLmropTUK1w4UZr+zAsYuFTd0UBVRgUOqFb3Zfkf6+0V1SZzNLcOZqcVM345bmXGYJRBHYui+zqZvSJGTmV8DpErDxl8tN3RQFVGBQ6h0tcSGIIqrsXKO3pTa8vO4glvzrUFM3o0mptHOSi7E+qHa44OIFw/vznlR45hbNpOA898rCNq8hunm1hnJzoDHOfLHjAma+/hOqHa7Gbw8lIMoqnfjr6z9h067L9XbMx5f/iFVfnTC8P5k7xdyiEsPlcgsMMxUYlJsdUUNi/HoyDwCowLgBKK10AgD2ncmv1+MeOFtgeF9iYbCNJDB4QcB7G08gq6CiUc5XE8QaM7PNS2BSgXGTcuZqMarsTTQ4a1gYRGMkk/sojYcgiDhy/rrh2JL0hOrJI1WbSguNbWGUVXLYfSIPp640j9iV3ckDACzm5jVEN6/WUOqFaocLS/51CCu/PNZo55TLAa3xgWiMN3pA/EZk28FreOPfR7HvtEGLoZ5rSPKC8diF9BuxkS0Mj0bPuQJva0NALHHqkrpJqHa4MP3lH/DjkeymbooPJGCWmd9U5rXvUEMEhUtDmpRVOjH95R+MD2iUgCipcAAA8ourDe3P8/Ur3F18XSyMxhmiiICqD4GRX1KN6S//gJOXa79CaLXHwqAC4yahqMwOoHmm/TW1Eq91fpJxo+WeuObxG+84lNWg7ZKTX1yFb3ZfrnFQFEURX++6hMJSe+M0rAGwWdwrVzo43tD+ROGohSdJk0CyowhSDKOBXZiZ+RXYduCaJCSddRQY3+25ip+P5gAAfj6WE/Dvr+aV44eD12CXYn01PwRBFPHljxel2FNDQgVGLSFak7kZZnEQ87qpwgWaAsPzHvIagwcZUNhGDPC99tkRfLHzoqR965FXXI0NP13CWwG490oqHDUG9yuqOZRXNfwLDgBWc2ACgzyjprQwyADe0DGMhR/sxbrvz0qKjNNl7B6pKa10oqTCgc+2n8cmz9wJvhbXvWjtPny89axkYXCumo9xLrMEG3+5jLWbTwV8vkBpshX3bnRcQuMPckYhbp/GbJn8XFpZUsTC4DXUVq/wbTz9hQzoNQVkyXhVWW18Dsmr6w+jR4do/O6uzrr7zHrjJwDAB/MyDB+3thCB7AzQwqivIIaWklBjGzwDd+NlSXlcprW0MP7fip91j1kbiIXBGbh3krAz+HzrArUwNPh8+3ls/uWS331Ix2JVPsYfDl7Dii+O+v3tsYuFeOGjfbUy1Y3gtTBqftm+/fUK1mw6Wb8N0HhPRJnAOJtZgnmrdsPh0aIaIoXw3LUSzH33F9id2po+UZ6dnP4zkGvYgbz8JRUOlFcbtx6++vkSPvrutOH9A4VYFpUGs+ZcHq1WS/AHwpmrxfjfVbtRVYtU6ppcQ+9tPIn/7vd1B2/+9Qq+3uX/3dWCDMx1dUnJqU12GIE8KyMCjLzn9eVC9AcVGBocvViIwzXkjJOXUO2S+njrWRw6d93vb1f+5xgu5ZTjYnYZTtQhMKaH5AIwMP5+vuMCfjmeW6/n15zpLZD/RXy2/Tzyi6uR6Yld2BsgwPf5jgsoKLHjap7/wL8/N82RC4XIKawCEJgfnnMJAblhvvr5EnYe1k6esDtd2F3H50Ousdygj9sluaTqdFqs/+E88oqrcSUv8EV8SPDZpZFhxbkE7D6Ri3/995zPd4fOFeDYhcDrL3Gee1SfAkOr7UYprXS7SrX6XZWdw68nvX1C0gsbIXhJXVIasIypxgHCwWlbGEYgnfLldQcB1L9bgrQ9EH2d/MbFC2AYE5g6BEC0fN9yl5R6KVd7A6QQet8h/y+Rw+lCqNmi+d2b//ZaikYFgCiKcLqEWrlhtPjHd2fw68k8xMeEoGNCRK2OQSy5SoOafn0JDKJL1cbNQwSGlmV3NV9fAFU7eFhrmLvg4gWfvuYgAqoeBQbPi5rnMkJphVu4a41DazefxoGzBUiMDUNibJg3Zb1uzTUEtTA0YBlTjS4I8hLq+VgDCRgaDUYahQTbApkkV1zugCCIeGTZDny67XzgJ5Wdy9+V84IoCSNyi4jLoj4DnEaPFGgguCYkzbgWAU8t8ordFk5d3BvkGo0ew+s3r9s1kP5Xm1RVEnzWavOVXLfASIgJ8fmu2uHye53XCirwyLIdOKjyIJA21jborcWpK8WYufxHlNUiuYFkPGnFMEiiht3B45fjOXhl/WEAjZMd2SgCY8mSJcjIyEDXrl1x9uxZabvD4cDChQtxzz33YNy4cXj22Wel7y5duoRJkyZh5MiRmDRpEi5fvtwYTQUAsAzjM0Bs+OmiotSy5JKSaQ/yjqpnoWh15uJy/5k6geLykyVVUuHA2s2npKCivA0kaPa9hm+4JuSnKq1wap4DcE/i8i7l6rEwPMJXPcHr6IXr2Lr3asBtAWpeLpacmwh+gt5gIxcADiePD745JQ0Ep64U45X1h3DicpFkPRqdrCbvJ//6/qyPokGOV5cZvyROYzRrh+xXV5846X9GArdqJAvD05YLWaVYt/UsBFGUEhaCbb4OkiqHy6+yd9XjHjtwRikwnA3gkiLHK6nD++3SyJIiSiovCNh7yjt3qTEmxTaKwBgxYgTWrVuHNm3aKLYvW7YMNpsNW7ZswcaNG/HEE09I3y1cuBBTpkzBli1bMGXKFCxYsKAxmgqAuKSUN//rXZclF5IgijjuqVOfW1SFg2cLIAgiTstKYutpVRUa2TbFZfWb4098p1pa9mfbz+Onozk4eFYZZykqt9fby1JRzeGnozmatYPOXC2R2iWI7gHziMfnrL5nr39+FOt/qIW1A/i4vfSQWxgV1Zyuu0NeufXHo9n4+VgONntKuh84k4+Tl4tx6GyBNPCo+w/nEnAhuxQV1ZyiXlF5lbc//PfANUl4Sr/j6i4wSOBfbyB18QLOXyv1nrMe3GlllU7kFbknCuolFtidLlzJLcfZzBKf58TJBG9RmR0rvzyGbQev4eiFQuk7df8WBBEOJ697nZxLwLX8SgDAxexSRfVk0vcbYqZ3XTwIWooncYOr+1hjuKQaJYaRkpLis62yshIbNmzAzp07JW2wZcuWAIDCwkKcPHkSa9euBQCMHTsWL7zwAoqKihAdHd3g7WVZk19r4eejOdIgl329Em/95xjm/b4vPpEF4QIRGEX1bmH4cUl5Lktdurq0wlm3tDyNU2mlycrXzRBEET8fy0Fekdvt0hAva00vkfyaF63di6Kymp+FOkhPNF5eEGWasfJaPtl2DjsOZcFqYRQDqHouhlq+ERdJXWpwkXibXrnyLXuv4oudFzF3SjK6touql5nec9/dLQ2UegPmOxtOSAsE/XlsD9zZK176zimLYTz19i/S9t3Hc9EyMgiAb9WAaqf/VOkPvz2F3SfcRTDziqvx6qeHvefjyLyHBhAYztq/V1rCm6Ty84KgeKY3jUtKi8zMTERGRuKtt97CxIkTMXXqVOzfvx8AkJOTg1atWoFl3ROOWJZFXFwccnICnzlZGxhV0FutIZVpZJtUOVyKFE4jAoOMAfXtkvLnb5dS8ARR0dk+/eE8ln9y0NDxF7y/B//6/myN+9WUQ+9yCYoCifX5ssqv83xWKaa//IM0W/ufW89IaYvyl7kmYTH95R+w6IO90uBis7r7Z7XD/Xnn4Wy8+M8DAHy1v4vZbg3epy+pBIZ6UCf7k8HbyfFY8P6egBZ4Iu3V6xfkGZDCe1w9BL3lQkIvtVleOoPEagh6Qe+L2aWSm0Z9PeQ65O7A3cdz8fCS7eBcAk5cVt6zSzlea5IIqIaYy6C2GtV8vuM8nnnvV5/t4SEW5BdX48sfL+LoheuY9cZPqLRzkiLG86JKI7qJs6R4nkdmZiZ69OiBuXPn4siRI5gxYwa+//77ej1PTExYwL8JCbKiyl6N2NhwAN4yIAAQGxuO1q18s1WCgq2wO3lEhttQUu5AWESw9HuCk+Px5SfehXnIC8mYWZ9960KIx71gNjM+xw0OdmcEhYUFIaKFMmh49LzXTeWvPdcKKnGtoBJPTOknbbNafbtSdHSo3+OEhNogetJWGcYEMCbN/Wtzb2ye9oSE2nDQU6Z759Ec3NamBbYf9JYgcXB8QMe/ml+B3l1iAQCREUGIjQ0HJxvUJIXApGy3XjadNciq+BwZFYKo8CDpMxm8o6Lc9zK30H3v80odSDfYbnKMsioO+89dx+jYcBy8UIg+XeLQKjpE6s95JXbExobDSmIDJv/3fseBTHRs0wLt4/1nb4km77XLjycXBqGhNsV3Jo+yYfbMUicUljlQ5rnHIpT9pdwp+GzftHsPBFFEQUmVrNyGLxZPfxFEMeD+VpMlZg2y+D3mt7+643TqfSLDbSiv4rDxl8sY3KcNKqo57D9XKL3DIWE2mC3e+8Oyvu97fY4rQBMKjISEBJjNZowdOxYAkJSUhKioKFy6dAmtW7dGXl4eeJ4Hy7LgeR75+flISEgI+DyFhRUBZ5jwLh4uQUBBgVsDyZdpPwUF5SgtrfL5TcH1ClTaXWgbEYSScgfyC8phFgXYnTzCPA9424FrOKNRPrms3C6dS40giiirdCIyzGa4/UUks4YXfY7r9Lw0JaVVuJatv15yQUE5Kqo5BFlZ3bRA+bGdGlpkSUmV7nUBQGFxJXKvVyAq3IY2LUNRUFSF7JwSWFSDhL9j6EEC7kXFVQjzDICbNCZ0OZx8wMcvLnX75otKqlFQUI4yjfIidtVxOR3N9Xqhcp5IQUEFXB7fOufiJU27uLgSQQyQ5Qna5hVWGG63fKB8+4ujSO0Rj7c+P4KWLYKw9Pzprn8AACAASURBVNGBKC6p8hyz0n095W4Fyclp35tqhwsmE/Dqv4ylhZfIFC69NldWOhTfOTxtrtJw4V7Mcvdbh9Ol+E12rltRcrm87Q4Ndj/74jKH3xhdieeZ2h2B94eaEhwKPPe1JtT7VFV7n1uLEPcYcvx8gWS5Xy+sVLx3HCcojhEbG16rd4dhTLqKdpO5pKKjo9G/f3/s2rULgDsrqrCwEO3bt0dMTAy6d++OTZs2AQA2bdqE7t27N0r8AnDfMHlGidqN4NAI4pE0uBahbo2Rcwl4+8vjUgkIALBatG+3P1fMu1+dwN/e2qWZcaSHpLlpeIRIWMHh5P36VkVRxKw3fsKqr5WrpNWqHTq4XCLKqzhEhFhhMTPIK67Ga58e8dmvNiml5NJdvIDQIH29qDYByesldsVvtepGqd0lelegHsTkg8/yz7z3giixxL0RSKkStXuMtLfccwwSJ/GWx/BfufXx5T/ir6//pPmdFnouKX+QtsjvR48OUQC8Vpy6f1U5fIP75H0sqqF4pOSSqoVbtKbsM8Op2yrBU+lRHKLCbTK3ogjW8xI7XYLCbVjXmflGaBSBsXjxYgwZMgS5ubmYNm0axowZAwB47rnnsGrVKowbNw5/+9vfsHTpUkREuM3bRYsW4eOPP8bIkSPx8ccf47nnnmuMpgJwB5XkD0/9wMmgGRXu1frJRBu5wDjscfG8vO4gquwcgjTcNoAysPXdnqv4+me3JuzkeOz3lPyuDiBwpp6498E3p3D0AnE3ubfanbxf3yrx8avTD9XlJSqqOby87qBmNVfvBDD97JzSSiciQq1SzOFMpq/VU5usHXI8Fy9Kpau1qI3AyC6sVPxWS2CQa//sh/P44eA13XgAp1I+iHD8YucFnL7qvRfk5+ScFdXGB2H1jGPSXhJiIgqRg+NRWGqX0qpdvKgbKA+kVAqJ8QDA4fPX8YFGkTz1ach9IL8ddEc8Jmd0VmyTxxkPnCnAW/855tO2yFD3O3q91H9pd292m6DZX6sdLrz08QHkeJ69nJruRU0xDG8blM+JzFdq2SJIEdMhQW+11doYQe9GcUnNnz8f8+fP99netm1b/POf/9T8TadOnfD55583dNM0UafVqgNhTpcAxmRCZJhVCliTyTQRYVZpH8LZzBIcOFOg69qRzy79bLs7jXR8ekdpWVOtNvjDWxrEne3187Ec/HwsB89PT5NeMjvHSwJNCxK3CVHluqtX8dt9PFcxP0XZDv+T2DheQFmlE4ktQ1Eoc1tcyS1H+3iv79XFC1KJblIKPTHWf2yKJBTsPZ2HXh30LdPaZLCQVFink4coiooB0dtm9zV/55lHojXJDPBq9zERQSgss0uDjzybDIA0GpDBp7zKif2n89G3a6zurPysggp36rIql9+bTmpCUZkdJy65g88OjscW1bwXJ8cjyGqGKIo4cKYAyV1aap7LH3ILg8yenza6m2IfueA+faVYep9IFlmn1i0QorIU5f1KvliYfAAnVv31khoEhuwd5FwCrBalW/T4pSKcu1aKL3+8iMfuuwNlVU5k5Vege4doXaEqHVuVun0lrxw9PX1SkVyjsm7m/C4Zz324D8E2s9fCEASpHJHDpRRuN3WWVHOGZRgIcgtD9SCdnACrhYFccSvx1H5p4dFo1OY8L4i6mrLednlNHC03GEHdYYk7hDEpZ64ukGX42J08th24pntMsthOWIi3bIYgij4Co9qPu4G80HrF51y8gEo7h9Bgi+QeAYDnPtyneRwAWPD+Xix4f6/uOdUcv1iEf++8oPu9UQuDuEPk2DkeDo7XHDDU7gU9JdTJCTABuH94J/d+OjuSrSQece5aKd7ecFyzzhQZRJ59fy8WfrAXgigqhEpltdfCmPPOL8i6XulpC48WYcogPOl3+88U4O0Nx7F1b+CTOrU0bF83r3efpZ8ckixZ4n4KCTIjyKocxPWyvuT3kDybQ2f9L84lH9S13FLSwOy5j0vWHcSy9YchiGKNLin59b/x+RG8uv6wdL3yjEuH6l1q1yoMnVpHgOcFb5oxL0pt8HUP3yQuqRsNYmH8dDQb01/+wWdiHedy16uRDxQ+Lile+TB5QZS0iWem9lN8p+crdsg6hJ6F4eIFPLxku+TGItuk36mFneezw+mCKIpI752Ali2CoIZo8uHBXoHx8rqD+L+P3WmjJhOw/eA1bPjJN5BMIC80mWfh03aXAJdLhMXMoKJK3ydfm/o+8gFSy8J54aE0tIkNNWxhaM0qdnC87rrpPufUUf+cLh4WCyP5pXlB1HbhkRiGqh8UqvpmUZkdDy3Z7uNKtFm9rzoZhE0mk6JZdifvo1mTgY3MVlbPGTJSflzLZffoazsVn/Vm+5PtIUFmKY2ZoGe5yu8hOVxmDUUoOZWFoQe5WqkopUuouYyQ7JldK6hUnEM+cVPt7jWZTNJYJHdJecuZq2IY1MJoGljGBJ4X8P0+twaeIxvwBEGEg3ObrHKBQUxoeQxDDi+I0sCn7vh6HVQ+UDo5HtdLq/HFzguK85KaOD8c9FoL8g6sFjRkroHd6daMg6ysZkfL8nTskCC3wLheUq2YDWyzsPh0u/9Z2JznhdYTGE6Xe+KRhWU0JzQSiAA0Wm7jbGaJX3cbAFgtLKxmRtfCSL8jAZMzbpc+q11zgNslpbdQknqyp57bwukSYDWz0sB77GIhtmho8ZKFoRJwau2W9NWNqvL8ckFAAubqey6KvsH0j749DQfHSwOwWjyYTCb8Y8sZ/HuHvhVnJN4hlbrXWTAoxGYByzCKGe+C6B08fWZ9+1nhEQBGD2in+Cx/T7QSO/QGY6dL0DyHTXa/5UqJSYobEUvf23+0lA+WZcALotRPXbz3fE6XoLApanKN1QdUYGjgDnqLUseRD+gOjgfn4mExM6rZ4O6/oyLcLim1VizIXFJq09rFCzibWSK5BgicS0CwjayWJuC9jSfxze4r2H08V3qBL2aXAQC6tI1E9vVKuHhvaW35zOP+PVoB8FoOdicPFy964iq+HS3Xk5rLCwKyCipwOVeZnmezsH7XkgC8FkZucZVmmRK7x/fPsiY8OamP7nHI4EusuJogJVz8YbOwsJpZXcttYK949O0aK33WtjAEKX6h1rRFUTlY6SkFTicPq4WRCi9+sfOiFMdSHs99LJ/aVz4T/dzfZ6v6kk2Wqlxp1xfOROPt1i4SgHsy385DWZJLTS0IXbyAHYeysPnXKwFl0KkhA6deSXDyzvi4pTz7qy0jcu/1Eh7UcS25Je50CeBcgiLATTKQ1OEiziVoKjLyeItcKJDfEwEg7xdaz4Uk4MgtDCKAOY7XlGRZnnGgIaACQwPGYwaSTiR3lzg4XtIK1YpFiM0suXB8LQzvQG5Tde5rBZV4ed1BPLtmj2I75xIQ6tHwnZy3Rs7735zCD57JZ6QzXs2vwPw1e/D1rktSZxEEURrU27QMVRy7yu7ylF42aXo+SdD7Sm45nn1/L7786aLieyOVKkg7cq5XIV6rsqin7RaWQbf2UZJQI233Hsf9d32WULFaGFgtrK5LimVNikFIq5aTneOl+Ex4iG+JdPlLq2fJ2J08LDILoybUKarqfkasBrW7Rn4t8rkN6lTvskongqwsfjP4NmmbyWSSlB1/sbQqjeC/UeyShaF9fPLOqN8dcp3q50PeFT0Lw6oSPEoLQ8D6H87hmff2oLSGJXydLu3aVfJUbuWzN3nO53s/tVKlzQwDnhdlQW9v5prDx8JwV414ds0efFbLGmw1QQWGBizDKKS6PCDrcPJwcm6tUFR1lKgIm9Rx1UXzHJwAFy+AZUw+nVvLrSGKbosk1COAnC5ekWVFLAWi4ZIgdUGJXXJT8IIoBb3lKcCAu9igKLo7pJYSRo5L/KrEZ0soMaDtu3i3L/lidilu01jLgVw3mQUtX8fgybd3SX8/9+E+ODheUUIlq6ACT67chX9sOVNjO7SwmllYLfouKZZhFIOTloB0OF3SNYQFW32+lw/aeqmV1U4XbGamxtLu6nkYBLUbo1In3VYew5BPYIxSTQg9cLYANguruHaLmZGu0591UlbpxFNv78IhP4uPxUT4xssAmcDQiUsQwaZOTSeDtc2iLzC0np1NNTlUbmG88NF+7PVkKF4nMSK9tGgdlxRR9ADvta3fdk66j07Je+F9nuUaAoMsteANenvXWnFyvKJZPC9Ik4zlKdn1CRUYGpgZdzBQWqlMVu/nvweuobTSqQh6EwEQHR6kmzrrcLpn7ZrN3gCnP1y8W2CFeTQVBycoljBVF74jtIoKlsx6nhckTSYiVDmgEddDQ6xJ/pfxPaVrKCi1o6yKw21tWvjsR14kcv+sspdY7X7KK6pSaGDZhVUoLndgx6Es1AaGMcFqZnE1t1w2R8WLlmCXE2Izo7TSKfUNTQtD5qrQcy/bnSTorf0ciCts5+EsVDtcPhaROg6hFwuyqgZIglqRANz3Rh5n238mH7kehcGfWzD7eiWKyhxY91/9OmMxqgSLsQPbo21cGOxOF05dLsK5LO2BjlhIxCVF7hYRTmqXlHfin6g5/8miEjBq1yRRlMorSRkSN4fPXVdUfuB0gt4hGhbG1n2ZPtvkgkorgYRlTXDJFD+5S6rSzikyxVy8gDyP4qg3Sbiu6M7DmDNnjqEKmUuXLq3XBjUHiLanlclAUlHjo0MkgREaZEZJhRNR4TaYTCbc1jpCii0Q7E4XzGYGFpYxNEhXVLvcLimPhVFW6VQIIzNrQrXD5ZPWKs/GklsY4SEWmODu+CE2s+RKMbOMT1aOmfUt764HmT8gp3+PVvjHltNw8SL2nXJrasQnLoe4V4gg9DdAC6IoXau7fXX30RKt9PXPj2LN08MV37Gs/1UHE+PCcDazRFoCVms2uZH1J+xOHi1CrQoLIyrcJllTbu0e2HE4G+VVHBwcD5vV60qrsBsTGGpXjvdcvhq/w8kr9j8pK9pH0se1qPJjfRBatgjCWVlMv3+PeNidPHYdy8Gy9Yd1f0f6hhTLsLGodvBY++1pDE5q7SMQ5RV3g6ysj2Kl3l8vxpRdWIk+nb1zT5wuAfNWeQsFOjleUxkIlWUXark9STptTfOrSAKOtKaJLEuqrNIJm8Xb71y8iFxP0kMdihv7RfcNbd++Pdq1a4d27dohPDwc//3vf8HzPOLj4yEIArZt2ybNyr7ZUA/oWi+hPOhNFIwYT8B7/oMpGNxbWffKwfFwudxWgnwg0huUnly5Cw6Ol0zbr36+hHOyLKXdJ/Lw+PIfcSVXmS4oX0+al8UwrGZWuq62cd5Jb2aWQZe2ysFcbk7XxIt/7q+5nWUYuAQBPx/LRbd2kUiICfXZh7i9jCxhWW13yQLMjKH4QE3ItVJ1wkFNMQVyDy/lloExmXw0XMDYOuB2pwtWs9LC6N7eO+dDvv18VilcvKCoK6b2e1dWc9IkQbn1oM7MI0RH+FoYDo7XFTD+LIwyP6nRBPUys0EWFkFW1u9saKuFkd4TYnHpVU0gCDILQ+ta1Bq4XkmQf++4gDNXi3VjIWVVHJbKCooS9GMYbtZsOoUzV4t9BJW6rW73uLJsPgnkl1Vxij7m4gVDlmBd0L3rM2fOlP5+6KGHsHr1asW6Fvv378c777zTII1qaoy4jORBb6ItpHSLk75XZ9XYnTxYxuQzOAbbWJ/8a/lv5Fkhci2JnFM9KHEuQREgIxaG1cxIpmzbVmFSCQ6WNWH6vd3RumUoNvx0CS1CrQgLtki1sfwRFmyB1cJqLmlrMTPgOAHXS6rRr0us5u+9FoP7nvhLC1z++VH0uT3Gc12CwgIqr3TCFhksfWcUuUWjnq2u1kDVgp0IjKyCSt1aVUZm59sdPCwWVnH8e1LbwsUrV1MD3PXKWkYGITLUKqUqk3OIoohP/nsOh89fR6fWEZgxPQ0xETbM9NR80lvnukWob+yFF0RFzMMo/gLEv0nviF63xfgILpuV9cwk1z+u/Fm0iQ3FvtPKvrJm00k4OB4JMSEYltwGn/z3nDSoCrJSGspjGr++i9llCotBjlZJHMCbjg64NX8t5eHctVKf7Ysf7o8573jX/zCzxMLwTPSr4nD8omdmvpNXWHUuXpDK6JMll+sbQ6VBDh8+jKSkJMW2pKQkHDrkK1lvBoxkrKR2j5Ny/R8a0x0lFQ6FFq1O/ysudyAzv8Jne5DVrCswgMBXWrucWyZpS4JMM7FavPMt4qO9GUtmxp0t1NrT9rZxYX4LsN3VLxF5xdU4drFQqsKrJTBYxoSicnepCy0tFvCm1RKB4W/QcPEC9nsmo7l4UZFNU1rlREuPwNAq06GH3CooUJWOUPu4R6a1Q0mFA78cz4OLF9Ai1AqbhYWD4xFs0x7w9O5jvy6xOHTuunvmvMMd9Jb3OZuVRefESB+BAbjnKYSGedtdUuHErmM56BAfjv963KWR4TaFFam+VjlBVjMmpHdE9/ZR2LT7sjQYGVGaCMQtSRIh1K649N4JGJzUGlHhNuSr7rPNwuhaM6Rfyd+BDp6SMXIN+hfPbPe07nFSliLx7QuiCFbDiteyarX6MeDOzlO/t4Qqh7ZVFaZSIrTW0GEYEziHADPLYFT/tkjq1BIxLYIwcchtktXPMgzsquC24rhVcoEhoqyKw/DkNgi2mWtMpKgNhnpFjx498Nprr8Fud0tTu92O5cuXo3v37vXeoOaAlsCQa5FPTe6DO26LQUZf95Kzad3jMObODor91RZGZn4FissdPgMLmWehR6AC41JOuTTpTu6Skh8nWua3JvEDIkTuSmkrCQI1ZtaEKXd3QZtYt3AhZr18rQciHMwsIw3CWoFVQL4uunKNbyPIzXz5y2jEj06wye6JOvvH4mMJmvGn0d2lYKbVzEh/qyf1kd6jtjDIs45pEYQ/juoqbY9pEaR4uc0MA7133cULirbxgoj3vzmF1RtPSttaRfmmMOsNykFWFhPSO6JL20jMvO8O7ZN60Jq8CACR4VaYWUaySuVas83KYvq93aU+oNbszSyjOxiTRA35IN7JkzzRU6NUi5n1ZpvJXVImjZupFZ9Vt6NNy1C0bxWOvKIq3ZheWaV2fwtRuXXVcT6CO0WfwcQhnaRrGzuwg1dgsCbpHdZ7hmoG3ZGA/xnWydC+gWJoNHrppZdw6NAhpKSkYODAgUhJScHBgwexZMmSBmlUU6MlMOSDKNFOJqR3xJqnh/us3wDo+4zVwbcgz0uolWUDQPPYavwFZ8n55D5b+QBOriUxLgxfLh2H3p1iEBasPTCQfcn9Ia4C8vme1LZY+uhAz74mFHjKgEdrBFa1jhvIRFW5z3vFF8ekAUKvbpUWirkJKitPL8uEdA2LhZUGULdy4G08eaZqvzyJDbGsSSEgJqR3VPQ5s5mRopbqR1tp52A2M1jz9HCMG9hB2n5Ntk643IIkqNNOCfJ7YNWZ60AYmtxac3uwzYzQILOkICi0dNUzVQsMk8mkKzC8VRO89zE0yILVc4ZhUG/ftXHMrElRYgXwuKQMatpqS99iZtAqOhjXCip041FalgPgXYeDoOW6qrRzUoq+HvK2y+cy+SvZr6eg1Qc1Cgye5/Hrr7/io48+wvfff4933nkHW7duxfr165GYmNhgDWtKtHyecoFBtHWTyaRr9qk1VIJaYESEuF+KVhovOTnXvN/3RZzH5aKF2exHYDhdYBmTwsUgT7FlNVJ1ycCmNtvVAoPcB3IMM+sNTsp/G6XjkiKQ4wRSz59knnROdGtlH313GpyL95l93q+rdvwEgOLZHTqnTK3Vc8kQzdTCMgj2vLRhKmFPBjp1sgRRClhG6YJS9yMza9KcGQ+4NVqzR+DIlRK5sNVyAeoJAbUgeXJyHzz/UBoAYO6UZGn7qLR2GNxbR2BYzejZMVpz0qD6mWopQHoBbNJP1c/UzDKo1nDjmlnvfeUFEccvFeL4pSK/CpU/WMaE5M6xKKlw+vQPgl4JmmDVNWlZGJXVnLsyrh+lUP5+Jsom3/oTCnrKZ31Qo8BgWRYvv/wybDYbEhISkJSUhNattTvOzYL85SUdUCEwDGT1qIUO0aLUfm1yrlZR2gLB4sli6tbeNy2VoDcZCnDHCdQajNyC0PLlkmtVa0mSgGCUQsH72Vf4mEzeAobqzDEC+f3dqW0RGxmEli2CMG10N92S4IAnTZk14d4B7QEAPx3Nwa5juYpyDmHBFnRq7Tv/g+Bv7XPChPSOGJ7cRvpMBh+rhZEsjIhQq2JYJC+zer1uEjg1s76KBqtySelBam8B2kLAYmbQQWOSpDqGQbKVzCqNv2eHaMR6lJOu7bxun0F3xKNVVDCSO7fEn8f1QLu4MCR1cichBNtYDJDN0ldo4yodwKyhjOlZ40SZ0oor9O0a57NN7pJycry0GFdtffmCKKJf11iwjAnns0pr/oEMi5lBjw5RGJLkHitzC33rqVVUu+DgeJ94mRyiuDAmE+JkY4S/TEYjWYe1xdCRhw8fjh9++KHBGtHckL+wZDBWuKQMxBXUL/3z09N892FNaN/KHZzsnKgtEMggTXyiPTtE4bbWygFhVFo7n98RSiodkgZH0jXlWp5Z42UiA5v6GzJQsSpBQdxB8gGMDAxBVlbSyqfd212RMqq+xlZRIVgyYyCWPjoQg5Na4zmNe0YGbDvHg2UZhab1jy1n8NF33pnf6oApgQgizsA8iQnpHTF1pDfeQB4ry3hdKREhVsXAGE0EhspdES5LElBrvQoLw2zSXC1R+l5jZjxh0bRUTUGi3kaemVF3jdXifo5//W1v3NkzHoump0lCx2phER7im20F+JZ1NxI7IKhLrSu+C7Xi2T+mKLaxrElS1I7IlgYweo1qeMFda82f+0cPm4XFU5OTMbSPW2Bc0aiWW2HEwvC0PSrcpnhvA4n31SeG7oTD4cCsWbOQnJyM+Ph4xUO/GSfuyTsYkf6hihhGzR1QrTWFq9IXX5+VDjPjDvildW+lO+FKEhgk4GgySQNFr47R+PO4HggLtqBru0i8+cUxqehcTIQNhWUOHL9YhN4eTfCJ/+nt46dl/VgY6rpBZpWFQf6XamRZ5QJDWwvW0n70NCIzy+CVxwbCZDJh3fdncfBsAcJCLCirdMLu5GFmTIj2Y13xgiDdv4hQK+ZP7QeWZaTgs3ogIffMH6FBFhTADpi8y/hGyNJcAXeWkgnwSU0ODdZ2SQFKgcGYTH6tCPIctDKf1MFWgnpfMkgb1Ua1zkWCyVYzK7nnfNrjJ6njjVnpiraoidARQgSfvsUwUkbUd3u8C0EZlRdvzErHE2/+LH0mQjU4yGJojokc9URDeYwpKtyG1jEhOOGZEClfLEwNEYDBNrOizwSy4mF9YkhgdOnSBV26dGnotjQb5C9vRIgVWagM2CXVs2M0xg/qgK93XQbg27nlL0NsZLBiNuhv0juioLQau47lyiwM96PiXILUCWOjgiXNLi4qROEKiGkRLA1+JPvCamF9Xnx/LqnfpHdEpZ1DYakdu47nSoKSdFxyn8jcB/k1EuGh9k9raXv+Bi0iEEb1b4cgK4uocBu+2X3F45Lyr/3xvNd9IwiilHpLGJKUAE4E9p/MxcXsMnRIiEBhmX4dJACYOfEO7D6Ri9gWQVJBvvBgi8LzEmRlERJk1rUwzKxJGmzVwhdwa+H9e7RCXnE12sWF4e0NxxXHkSwMjzJj8VQQqHK4dAdodazikfE9setYjk9RSj20guYktdliZjQzqLq0jcSfVCvrAcC0e7shMTZM6rt6MQy9bD2CWtCYzdpVFBiZtf/4fb2kDMZZ/9MbgNsSbBvnbs+fx/ZATlEVNv1yWRqU9bLDtOh1WzS6JEZKlq88WzIyzIqSCidCgyyKyZdXVJWgFdfEEMVBeW0umcCYOfEOMIxJWs2wITF0J+ST+G4FyIOJCrehdctQnLqinOlpxCXFmEz4zeDb8O2eqxjWx+0Djwi1om9n7SUu5a6T8ekdse3ANbfAYJUWBimtDgAtVBqYPP89VlaCoWOCHw1GYwCPjw5BVLgNXdtFol2rcPzre3dtIB+XFKuyMGQCg7RXbWkRoWMyeQO1Riy229u0wO1tWki1gxxOdzFGk8mkWZ4EULqktDQyi5nF1NHdcfKiO3BpxPUQHREkpVCTGIh7jov3+BYzi9Bgi6JYIuAdAFlZcFYSvipXjZllMHHIbdLyqYp2e+4Xud+MyYRW0SG4VlChm1WnVhSiwm0YK8uyqgkttwkRmFYzo5kePqBnK82MLXXwXC+1nGXdE13vTtFOrvERGBqxIcB9jzNS2uLA6Tz0k8U++tzu+y7e2SseZ64WY9Mvl6V3Xr00LEHLIo0OD1LcV7mw6d4+GrtP5GJU/7Y4n+UtHXSXzvUB3neMTJAl8LyIewe0x/ZDWejrmRgbFmzBgJ6tNI9TXxgWnU6nE5cuXUJxcbHi5bjzzjsN/X7JkiXYsmULsrKysHHjRh+L5a233sKKFSsU3x0+fBgLFiyAw+FAmzZtsGzZMsTExBhtcp2JbRGEbu2isO3ANUXnDCSotOqpYdLfr/81XXc/9TFjI92aNdFQSKd1ugSUeNxOHVWxDHmxu0iZb1/rpfWe1/cFiwi14tXHB0mfyaCvjl14XVK+axKQ9gary0h7tNIQm3fCYiD3kwhru5OX2r7ssYGY/rJvjK1DfLi0v79Z5KI0MASWXZIQE4oTl4t9ZkuH2MzgXALyVQIjVCOGwbK+FoYCjc1EKJD7bTIBrVuGoMxPnSejOfx6aA3EZFKoXkFNI5Y4oJ+RxjIMVs8Zpvs7tTJigklxH0emtcWWvZkwmYD/97u+KCjQ1+TlEIunJgtj2WODfPqd+jZZzIykHMVFBeODeRkAINUgYxkTptyl770h12OzsIr3hBcE/M+wTor5Fm8+MdjI5dUJQ090//79yMjIwNSpUzF9+nQ88cQTePjhhzF//nzDJxoxYgTWrVuHNm3a+Hx34sQJHD58WPGdIAiYM2cOFixY7uwv6QAAIABJREFUgC1btiAlJQWvvPKK4fPVhZJytyuhZWQw+nZpib89kISMfl4twOiLEAjq4Gyv22Lw/x5IQjtPUDzE5llngxMkzVUd/JZbGHJT2J+f38hgTdIb1UXUSGcmY7FcqJKXTG2NkXpYqbIyKgEJDM857U7e7++emtwHf5vUR7qv/sokkK8CDW7eP7wTnprcR+GDHpLUGmnd43ysC0A7S0rKgtERGFpbiaAksSyTyYSJQzph5sTeum1tiOqlnMt3UihhRN9E3Nkr3vCxtARaTcFqlmHw96n9pAm0To6X7qcJvll8RgnyWDxEydCzMABg6QylwqwO6ptMJikhQu5ik7sT/UH6RZBVWT7GSGHLhsDwxL2HH34Ye/fuRWhoKPbu3YtHH30UU6ZMMXyilJQUJCT4plU6nU48//zzWLRokWL78ePHYbPZpPpVkydPxnfffWf4fHWhb5dYDOrdGvcP6wSTyYRet8UoBqeGmHKvHvwYkwl33BYjdUASVOR4AbPvT8Ko/u18Uuv06ij5y0M3UjmXZP2UV5PSD+7zqDVDhUvK0zb1miEPjemOUWntFFlhRlxS3vZ6LQx/be/aLhJhwRZFDEMP8l0gRRcBt6bfw7Ny2wPDb0e/rrH43V2dYbWw+PO4Hj77B1lIsF3DJRWIwDArA+KMye1i0gqezvt9X4zom6hIbnj6Dyk++9UGSWBoCO4BPVsFNP/hf//QFwNVAsbIe3Z7mxaSQmR38tJvQoMtstTuwN5XdZ9RWxgDe8XjdyM6A3ArlXLlTOtUpOfJFRKLatKrHsTda7WwCrdqIDXT6hNDKtXly5fx4IMPKrY98sgjGDFiBB566KE6NeCNN97A+PHjfSYB5uTkKOZ7REdHQxAElJSUIDJSf06CmpiYsJp30mDeH1N1v4uN1Y8J1Bb5bFat47MeC8PFCxgxoANGDOjgs89dae2x0bMyXkiIFSYT0CY2zG97W8VFKCwQrX1v8yzKU1HFITY2HMEhbgESGmpV7B/fKlz63CrWfd9ZM6vYZ2RsOEYC+OVotrQtLs541eNiT1tcvIDgIIvutcW3cgf6SzxuL14QdfdlPQNwfJz3+0CfcWxsOBZ18k4SHD8sHO/JynUAwO0dYsAyJtzWLgpkap7FwvqcS/45R2OGcHRkCGJjw8F4XCcsy+i2NzY2HIP6tpWyuOKigjE42dfK1+O21i1wMbtU8/gDerfG7hO5SO4e7/N9m4QWAd3D2Nhw9OvVGuOe/EraFhMdYugYLT2lUEwsg2jP3xGhVrTw9GsLSRIx2J7wCHdyxIjUdoiNDUesqtLy5JHd0FE2v2fkgPbYsNO9pnloqE33PG3ivfckOtLdToYx+W2XxfOMI1sEISTU62YWDV5PfY9VhgRGeHg4KioqEBERgdjYWJw/fx6RkZGoqvKdjBIIhw4dwvHjx/HUU0/V6Tj+KCysqFXVxtjYcF2fp1FfaCDI40Jaxye1k8KCLbrnn3Bne5gEAV/vugye4/H234aCMZn8tre0pAq8p4Ca7jW7vIu9FBSUo7TMXQLC6XAp9q+qcKDAozC5PJVo7XZO85jVskltgdzP8jJv8TpREGt8RhXldp9tcmJjw+HwtJWXrS3SIM+Yc+GNWYMREmSW1ksxwfca5J9LVcX6APe9KygolxZvigqz1dje657jkHfB6PXN+30yeJ37fEf7SLw1ewhCzL59zF7lQIH/hLMaKS+zG2on43l3eBePguvu2EBcZDAcnvpg1Z6U9UCe6cr/NwQ2K4uCgnKIqrXKy8uqUSBz8Y0d0A4iL+Crny/BXq3d3wHA5fR+5/S8c4KfPgwARZ7FmgQXj9JS73jr5IQar8ffGOYPhjHpKtqGBMbdd9+NnTt3Yty4cfjtb3+LBx98EGazGSNHjgy4MXL27duHCxcuYMSIEQCA3NxcPPTQQ3jppZeQkJCA7GyvFlpUVASGYQKyLm4kajKbbRYWj4zr4bN2hRyGMWHswA4IC7ZgWHJrQxVHjfh31UFdvUlf8tRLYsbryWq92kY1oV5EqiaMFG8k1n1dA8N6PDiqq7QUKvGHs6oYhh7ybhFsM6Pa4ZLuQXiIFQ+N6Y6eHaNrbkQtJ3qZWQb+ypnJ/fsL/5SKrOsVEEUo0kYDYfHD/THfs7a9Uddv366x+P3dXZB+RwKsFgZTR3ZF/+5x2H3CvXhXbS7dXwxQy31MYhL+HqcihmGwqChRFK3mG8gl9cwzz0h/P/TQQ0hKSkJlZSUGD65bVP6RRx7BI488In3OyMjAu+++iy5dukAQBNjtduzfvx8pKSlYv349Ro0aVafzNXfaxoVh0B3a5TMAYEDPmoOIZpbBXSltDZ/TSMCZYdyrCJJANem46hdaniUlr+mjBfGpt2zhvzChmohQq1SGWt72sQPb49TlYlxQrXRoJEGBBDcbYrlaAGgXF+6ToEAEQSDnDAt2CwyLrHaYv/4iJyrchtAgMyZldDZ8vkBpHx/udxKaEVq3DMXtiS1w/prxUhyMyYQRsqQUUsrFYiBDzgjqLEOtd4acwl/cRhn0Nl55dtuBa+jfo5VirfSmCnobEhinT59Gt27eCTjyhZSMsnjxYmzduhXXr1/HtGnTEBkZiW+++UZ3f4ZhsHTpUixcuFCRVnszo1UKo6Ho2TEaJy4VGR6w5j/ofebkBVKvuSB/kYhVcrvGWt6AV5u/04AQlBMWbEFq9zj8eiJPYeFMHNIJ3EAef3llp2J/IxZG5zYtcCW3XCraph7c64qWFScJqQASKMgs89rUCrKYWayYPSTg3zUF5I7UtfyFunRNbVGvQ671zpB3gpT+l9O1bSTOZJYosgiNLlvQumUo3vWk5sfJBFc3jRI7jYEhgTFjxgxUV1ejX79+SEtLQ2pqKnr06BFQ9sH8+fNrTMNV16vq27cvNm7caPgcFOPMvO8OFJXba1XJM6VbHBb+KdWvNtkyMhjPT09TlGSW0zYuDAv/lIq2rQJPSuicGIlfT+Qp1loHtDU/Iy/mAxm3Y0hSa7RsEYyXHhmgqOZbH2i5VoiGWLNLyvtbb1puwxWXaw7Ul51XXxaGT9VmjWfWt0ssFv4pVUqDlzPrf3qjtNKpeJY2gwJDTp/bW2Lhn1JhtTB+U+UbEkMCY8eOHcjMzMS+ffuwb98+fPzxxygpKUG/fv2watWqhm4jpQGwWVnNdbaNIhcW837fF1fyfINriXH+hUFt3Rfxnqqd6tndWgqMEYFhZhmprXpl5uuClhVBXHWBuaTcAiPQRbVuOEzK+T21hQjj+liq9M9je+C9Te6sN4vOcgJ6/TnYZvZZUM1Sy3hZXV1+dcXwTKW2bdvC5XKB4zg4nU78/PPPKCwsrPmHNxFtWoZqLuh+q9OlbaTfYHx9QwZ1rclxagJZarSh0BIYpFT10CT/SwXIZWBibCgOW9gGXSCnOSC5pOp4HDKw10edvjt7xUsCoz76VCBrijcnDAmM2bNn4/Dhw4iLi0NaWhrGjx+P5557DmFhtZvjcKPywsP9m7oJFHjLngQycNZUyK4h0XJJRYbZpDIR/pAvpdQhIQLvPDm0XtvWHImNCsaZzJIaly+uCeJKqqtLSk19TNw1GvRubhgSGCdPngTDMOjWrZv071YTFpTmA2MyYe6UZLRsob8KoZy5U5IRp7HOdUNjZk1w8caXCH3mwX5SRVsJk+afNzW/v7sLkjq1RIf4uiUfSEv/NlEpcH/c1BbG1q1bkZ+fj/3792Pfvn1YvXo1HA4HUlJS8OKLLzZ0GykUH+Srwcm5J7WtT0BQb9+GhmUZuHjesMDQWh3QpPP3zYzNwvpdWtcoJD5UXxbGjAk9sf9MHWcjejBS8bo5YjiGERcXh44dOyI/Px+5ubnYs2cPfvzxx4ZsG4USMJNHNNw8g0AxMyY44F1oqFYoJMatIjLqB5IBWF9z3NK6t0Ja9/opH07SylO6+S4125wxnFZ78OBBhIaGIiUlBRkZGZg7dy46dOjQwM2jUG5cWoTZ3CXc66DgymMYVFwEhiQwmmg5U3/YLCyWzxykWMnzRsCQwLjnnnvwzDPPoG1b4zOIKZRbnb89kISDZwvqbV4HNTACgwSnm6PAANwKxY2GIUfaxIkTER8fj/3792Pz5s0AgKqqqjoXH6RQbmaiI4ICKtOiBRUStYd4AutjHgbFjSEL48yZM3j00UdhtVqRl5eHe++9F/v27cOXX36J119/vaHbSKFQEPi6Drc6ZE2W9q2adrLbzYQhC2PRokWYNWsWvvvuO5jNbhmTmpqKAwcONGjjKJRbHSojak9UuA3PPNgPfxzdreadKYYwZGGcP38eEyZMAODVckJCQuBw1DzTlkKh1B5F0JsKj4DRSlWm1B5DFkabNm1w/PhxxbajR4+iXbt2DdIoCoXii4nmSVGaGEMWxhNPPIG//OUvmDx5MjiOw6pVq7B+/Xq88MILDd0+CuWWxnQrztyjNFsMWRjDhw/HmjVrUFRUhNTUVGRlZWHFihVIT09v6PZRKBQPVF5QmpoaLQye5zFy5Ehs3rwZixYtaoQmUSgUgiIzikoMShNTo4XBsixYlqUBbgqlCVB6pKjEoDQthmIYDz74IGbPno2//OUviI+PV2g9dPY3hdI40CwpSlNjSGCQ4PauXbsU200mE06dOmXoREuWLMGWLVuQlZWFjRs3okuXLiguLsbTTz+Nq1evwmq1on379nj++ecRHR0NADh8+DAWLFigWNM7JiYmkOujUG5sqJCgNCMMBb1Pnz6t+c+osACAESNGYN26dWjTpo20zWQy4eGHH8aWLVuwceNGtG3bFq+88goAQBAEzJkzBwsWLMCWLVuQkpIifUeh3CrQYrWU5kSjFWVPSUlBQkKCYltkZCT69/euYtenTx9kZ2cDAI4fPw6bzYaUlBQAwOTJk/Hdd981VnMplOaBSV6tlkoMStPSbFbxEAQBn3zyCTIy3MtW5uTkoHVr73rH0dHREAQBJSUlTdVECqXRoRYGpTlheAGlhuaFF15ASEgI/vCHP9TrcWNiar+UbGzsrVe0jF5z88IuW/wnOjq03tranK+5oaDXXHeahcBYsmQJrly5gnfffRcM4zZ6EhISJPcUABQVFYFhGERGRgZ07MLCilqVN46NDUdBQXnAv7uRodfc/CgurpT9XYUwS92dAs39mhsCes3GYRiTrqLd5C6p1157DcePH8fKlSthtXoXmunVqxfsdjv2798PAFi/fj1GjRrVVM2kUJoEk4muuEdpPuhaGEOHDjVUf3/Hjh2GTrR48WJs3boV169fx7Rp0xAZGYnXX38dq1atQocOHTB58mQAQGJiIlauXAmGYbB06VIsXLhQkVZLodxKmHQ/UCiNj67AqO/Bef78+Zg/f77P9jNnzuj+pm/fvti4cWO9toNCuVGh8oLS1OgKjLS0tMZsB4VC0YKmSVGaEYaD3qdOncL+/ftRXFwMUbao+hNPPNEgDaNQKOpaUhRK02Io6P3pp5/id7/7HX799Ve89957OHv2LNauXYurV682dPsolFsbE11xj9J8MCQw1qxZgzVr1mDlypUICgrCypUr8cYbb0jre1MolIaByghKc8KQwCgsLJRKdDAMA0EQMHToUGzfvr1BG0eh3OooQxhUfFCaFkMmQnx8PK5du4bExER06NAB27ZtQ1RUFCwWS0O3j0K5taHrJ1GaEYYExsMPP4wLFy4gMTERjz32GJ544glwHIe///3vDd0+CuWWxkQlBqUZYUhgTJw4Ufp76NCh2Lt3LziOQ2hoaIM1jEKhKAPdVF5QmhpDMYzf/OY3is9WqxWhoaEKQUKhUBoYGsOgNDGGBMaVK1d8tomiiGvXrtV7gygUijZUXFCaGr8uqaeffhoAwHGc9DchKysLt99+e8O1jEKh0OKDlGaFX4HRrl07zb8Bd50nWj2WQmlEqMSgNDF+BcbMmTMBAElJSRg8eHCjNIhCoXhRBr2pxKA0LYaypAYPHow9e/Zgw4YNyM/PR1xcHCZMmIABAwY0dPsolFsaWnuQ0pwwFPT+/PPPMXv2bMTGxuLuu+9GXFwcnnzySXz22WcN3T4K5daGSglKM8KQhbFmzRqsXbsW3bp1k7aNHj0as2bNwgMPPNBgjaNQbnVoaRDK/2/v/oOjqO8/jj/vLiSAEkJ+H4FCodVJpRRMGFq1poZSqI1JHG2hGVIGQVGqRBSbFByI/CoJVUmFlHa0zmgZaKn8kCg/WgMWHUihBCoNrYCBSc0lIZdEfhrI3X7/8MuN4cexSe5yl+T1mHEm2U927/2+Db5uP7vZDSamjjCampoYPnx4q2XDhg3js88+80tRIvL/lBESREwFxp133sny5cu5ePEiABcuXKCwsJDRo0f7tTiRnk7nMCSYmJqSeuGFF5gzZw7Jycn079+fzz77jNGjR/Piiy/6uz6RHk1/hyHBxFRgxMbGsnbtWmpqajxXScXHx5t+kYKCAnbs2MGnn37K1q1bue222wCorKwkLy+PpqYmIiIiKCgoYOjQoTcdE+mRdIghAdame0nFx8czcuRIT1iYvZfUuHHjWLt2LQkJCa2WL1y4kKysLHbs2EFWVhYLFiwwNSbSEykuJNA65V5SycnJ2O32VsucTicVFRWkpaUBkJaWRkVFBQ0NDV7HRHqSVgcVSgwJsIDdS8rhcBAXF4fNZgPAZrMRGxuLw+HAMIwbjkVGRrb7NUW6GuWFBJNufy+pqKhb271uTEw/H1bSNajn4HL+4mXP19HR/Qi/JdQn2w3mnv1FPXdcwO4lZbfbqa2txeVyYbPZcLlc1NXVYbfbMQzjhmNt5XSew+022rxeTEw/Tp8+2+b1ujL1HHwuNrd4vnY6z9F8oeOPRQ72nv1BPZtntVpu+EHb1DkMf9x4MCoqisTEREpKSgAoKSkhMTGRyMhIr2MiPZUukpJAM3VZbUctWbKEnTt3Ul9fz7Rp04iIiOCdd94hPz+fvLw8iouLCQ8Pp6CgwLOOtzGRnkKPaJVgYjEMo+3zNV2IpqTMU8/Bp/mSiydeeh+AVU/fS9/eHf+MF+w9+4N6Nq/DU1IiEniakpJAM/Vx5S9/+ct1l4eGhhIfH8+oUaMIDfXN1Rsi8iUKCQkipgJjy5YtlJeXEx0dTXx8PDU1NdTX1zNixAg+/fRTAIqLi/nmN7/p12JFehrdfFCCianA+NrXvsb48eP52c9+5ln2xz/+kU8++YR169bx29/+liVLlvCnP/3Jb4WK9ER6RKsEE1PnMEpKSpgyZUqrZT/96U/ZunUrFouFGTNmcPz4cb8UKNKz6TIpCR6mAiMqKorS0tJWy3bv3u35u4jm5mZCQjrlCl2RHkWX1UowMfV/+eeff56cnBy+/vWvY7fbcTgcHDt2jKKiIgAOHz5Mdna2XwsV6el0DkMCzVRg3HPPPfztb3/j/fffp66ujpSUFFJSUhgwYIBn/J577vFroSI9UeuQUGJIYJmeRxowYIDnuRgi0jm+fKJbRxgSaKYCo6qqipUrV3L06FEuXLjQamz37t3+qEtEQAcVElRMBcbcuXMZPHgwubm59OnTx981ich16AhDAs1UYBw7dox169ZhtepOIiKdqfUDlJQYElimEmDMmDFUVFT4uxYRuYpF19VKEDF1hJGQkMCMGTMYP3480dHRrcZycnL8UpiItKa8kEAzFRgXL17kvvvuo6WlhZqaGn/XJCLXYdFJDAkwU4Hxq1/9yt91iIhIkLthYPzvf/9j0KBBwBeX1d7I4MGDfV+ViIgEnRsGxgMPPEB5eTkA48ePx2KxcPXD+SwWC0ePHvVvhSIiEhRuGBhXwgLgP//5T6cUIyIiwcvUZbVLliy57vKlS5f6pIhdu3aRmZlJRkYG6enp7Ny5E4DKykomTZrEhAkTmDRpEidPnvTJ64mISNuZCoyNGzded/nbb7/d4QIMw+AXv/gFhYWFbNmyhcLCQnJzc3G73SxcuJCsrCx27NhBVlYWCxYs6PDriYhI+3i9SurKs7xdLtc1z/WuqqoiIiLCJ0VYrVbOnj0LwNmzZ4mNjaWxsZGKigpef/11ANLS0li8eDENDQ2e53CIiEjn8RoYW7ZsAeDy5cuer+GLk93R0dEUFBR0uACLxcLKlSuZNWsWffv25fz58/z+97/H4XAQFxeHzWYDwGazERsbi8PhaFNgREXd2u7aYmL6tXvdrko9By9f1tlVevYl9dxxXgPjzTffBODll19mzpw5Pn3hK1paWvjd735HcXExSUlJ/POf/+Tpp5+msLDQJ9t3Os/hdhs3/8GrxMT04/Tpsz6poatQz8HNV3V2pZ59RT2bZ7VabvhB29Q5jKlTp3L+/Hngi+mpt956i82bN+N2u9tczNWOHj1KXV0dSUlJACQlJdGnTx/CwsKora3F5XJ5Xreurg673d7h1xQRkbYzFRgzZ87k1KlTALz00kv84Q9/4PXXX2f58uUdLiA+Pp6amho++eQTAE6cOIHT6WTIkCEkJiZSUlICQElJCYmJiTp/ISISIKZuDXLy5EkSExMB2Lp1K+vXr6dv376kpaUxb968DhUQExNDfn4+OTk5nnvlLFu2jIiICPLz88nLy6O4uJjw8HCfnDMREZH2MRUYVquVy5cvU1lZSb9+/Rg4cCBut9szTdVR6enppKenX7N8+PDhbNiwwSevISIiHWMqMO69915ycnJoamri/vvvB+D48ePExcX5tTgREQkepgJj6dKlbNq0iZCQEDIyMgBobGzkqaee8mtxIiISPEwFRmhoKJMmTcLtdlNfX09sbCxjx471d20iIhJETF0ldebMGZ599llGjhzJD37wAwDee+89Xn75Zb8WJyIiwcNUYCxcuJBbb72V0tJSevXqBcDo0aPZtm2bX4sTEZHgYWpKau/evezZs4devXp5Ln2NjIzE6XT6tTgREQkepo4w+vXrR2NjY6tl1dXVxMTE+KUoEREJPl4D48pfWf/4xz9m9uzZ7Nu3D7fbTXl5Obm5uUyePLlTihQRkcDzGhhXnj/x6KOP8sMf/pBFixbR0tLCvHnzGDduHFOnTu2UIkVEJPC8nsO48gxvi8XC1KlTFRAiIj2Y18Bwu93s27fPExzX853vfMfnRYmISPDxGhiXLl1i/vz5NwwMi8XCe++955fCREQkuHgNjD59+igQREQEMHlZrYiIiNfA8HbuQkREehavgVFeXt5ZdYiISJDTlJSIiJiiwBAREVNM3XzQ35qbm1m2bBl79+4lLCyMUaNGsXjxYiorK8nLy6OpqYmIiAgKCgoYOnRooMsVEemRgiIwVqxYQVhYGDt27MBisVBfXw98cVv1rKwsMjIy2LJlCwsWLOCNN94IcLUiIj1TwKekzp8/z+bNm8nJyfHcOj06Ohqn00lFRQVpaWkApKWlUVFRQUNDQyDLFRHpsQJ+hFFVVUVERASrVq2irKyMW265hZycHHr37k1cXBw2mw0Am81GbGwsDoeDyMjIAFctItLzBDwwXC4XVVVVfOMb3yA3N5fDhw/z+OOPU1RU5JPtR0Xd2u51Y2L6+aSGrkQ9By9f1tlVevYl9dxxAQ8Mu91OSEiIZ+rpW9/6FgMGDKB3797U1tbicrmw2Wy4XC7q6uqw2+1t2r7TeQ63u+1/gBgT04/Tp8+2eb2uTD0HN1/V2ZV69hX1bJ7VarnhB+2An8OIjIxk7NixfPjhhwBUVlbidDoZOnQoiYmJnoc4lZSUkJiYqOkoEZEACfgRBsALL7zAvHnzKCgoICQkhMLCQsLDw8nPzycvL4/i4mLCw8MpKCgIdKkiIj1WUATG4MGDefPNN69ZPnz4cDZs2BCAikRE5GoBn5ISEZGuQYEhIiKmKDBERMQUBYaIiJiiwBAREVMUGCIiYooCQ0RETFFgiIiIKQoMERExRYEhIiKmKDBERMQUBYaIiJiiwBAREVMUGCIiYooCQ0RETFFgiIiIKQoMERExRYEhIiKmKDBERMSUoAqMVatWcfvtt/Pxxx8DcOjQIdLT05kwYQKPPPIITqczwBWKiPRcQRMY//73vzl06BAJCQkAuN1unnvuORYsWMCOHTtITk7m17/+dYCrFBHpuYIiMC5dusSiRYvIz8/3LDty5AhhYWEkJycDMHnyZLZv3x6gCkVEJCgCo6ioiPT0dAYNGuRZ5nA4GDhwoOf7yMhI3G43TU1NgShRRKTHCwl0AeXl5Rw5coS5c+f6ZftRUbe2e92YmH4+rKRrUM/By5d1dpWefUk9d1zAA2P//v2cOHGCcePGAVBTU8P06dPJzs6murra83MNDQ1YrVYiIiLatH2n8xxut9HmumJi+nH69Nk2r9eVqefg5qs6u1LPvqKezbNaLTf8oB3wKanHHnuMDz74gNLSUkpLS4mPj+e1115jxowZfP755xw4cACA9evXM3HixABXK9L5bFYLYb1sgS5DJPBHGDditVopLCxk4cKFNDc3k5CQwIoVKwJdlkinK34mJdAliABBGBilpaWer++88062bt0awGpEAq9XSMAnAkSAIJiSEhGRrkGBISIipigwRETEFAWGiIiYosAQERFTFBgiImJK0F1W62tWqyUg63ZV6rlnUM89Q3t69raOxTCMtt83Q0REehxNSYmIiCkKDBERMUWBISIipigwRETEFAWGiIiYosAQERFTFBgiImKKAkNERExRYIiIiCkKjKtUVlYyadIkJkyYwKRJkzh58mSgS/KJgoICUlNTuf322/n44489y73125Xfi8bGRh599FEmTJjAAw88wJNPPklDQwMAhw4dIj09nQkTJvDII4/gdDo963kb6wpmzZpFeno6mZmZZGVlcfToUaD77ucvW7VqVavf7+68n1NTU5k4cSIZGRlkZGSwZ88eoBN6NqSV7OxsY/PmzYZhGMbmzZuN7OzsAFfkG/v37zeqq6uN++67z/jvf//rWe6t3678XjQ2Nhr79u3zfL98+XLQ2U2FAAAHHUlEQVTjl7/8peFyuYzvf//7xv79+w3DMIzVq1cbeXl5hmEYXse6ijNnzni+/utf/2pkZmYahtF99/MVR44cMaZPn+75/e7u+/nqf8eG4b0vX/WswPiS+vp6IykpyWhpaTEMwzBaWlqMpKQkw+l0Brgy3/nyL5q3frvbe7F9+3Zj6tSpxuHDh40f/ehHnuVOp9MYNWqUYRiG17GuaNOmTcaDDz7Y7fdzc3Oz8ZOf/MSoqqry/H539/18vcDojJ67/d1q28LhcBAXF4fNZgPAZrMRGxuLw+EgMjIywNX5nrd+DcPoNu+F2+1m3bp1pKam4nA4GDhwoGcsMjISt9tNU1OT17GIiIhAlN4u8+fP58MPP8QwDF599dVuv5+LiopIT09n0KBBnmU9YT/PnTsXwzBISkrimWee6ZSedQ5Dur3FixfTt29fpkyZEuhSOsXSpUvZvXs3c+bMobCwMNDl+FV5eTlHjhwhKysr0KV0qrVr1/L222/z1ltvYRgGixYt6pTXVWB8id1up7a2FpfLBYDL5aKurg673R7gyvzDW7/d5b0oKCjg1KlTrFy5EqvVit1up7q62jPe0NCA1WolIiLC61hXlJmZSVlZGfHx8d12P+/fv58TJ04wbtw4UlNTqampYfr06Zw6dapb7+cr+yc0NJSsrCwOHjzYKb/bCowviYqKIjExkZKSEgBKSkpITEzsMofmbeWt3+7wXrz00kscOXKE1atXExoaCsCIESP4/PPPOXDgAADr169n4sSJNx3rCs6fP4/D4fB8X1paSv/+/bv1fn7sscf44IMPKC0tpbS0lPj4eF577TVmzJjRbffzhQsXOHv2LACGYfDuu++SmJjYKb/beoDSVU6cOEFeXh5nzpwhPDycgoIChg0bFuiyOmzJkiXs3LmT+vp6BgwYQEREBO+8847Xfrvye3Hs2DHS0tIYOnQovXv3BmDQoEGsXr2agwcPsnDhQpqbm0lISGDFihVER0cDeB0LdvX19cyaNYuLFy9itVrp378/ubm53HHHHd12P18tNTWVNWvWcNttt3Xb/VxVVcVTTz2Fy+XC7XYzfPhwnn/+eWJjY/3eswJDRERM0ZSUiIiYosAQERFTFBgiImKKAkNERExRYIiIiCkKDBE/mDFjBps2bfLpNl955RXmzp3r022KtIXuJSXiRWpqKvX19Z57LQE8+OCDLFiwwOt6r776qr9LE+l0CgyRm1izZg133XVXoMsQCThNSYm0w8aNG5k8eTKLFi0iKSmJiRMnsnfvXs94dnY2GzZsAODUqVNMmTKFpKQkxo4dy9NPP+35uYMHD/LQQw+RlJTEQw89xMGDBz1jVVVVTJkyhdGjRzNt2jQaGxtb1XDo0CEmT55McnIy6enplJWV+blr6ekUGCLt9K9//YuvfOUr7Nu3j9mzZ/Pkk0/S1NR0zc8VFRVx9913s3//fv7+97977prb1NTEzJkzyc7OpqysjGnTpjFz5kxPMMydO5c77riDsrIyZs2a1eqcSG1tLTNnzuSJJ57gH//4B7m5ucyePdvzVEERf1BgiNzEz3/+c5KTkz3//fnPfwa+eKbA1KlT6dWrF/fffz9f/epX2b179zXrh4SEUF1dTV1dHWFhYSQnJwOwe/duhgwZQmZmJiEhIaSlpTFs2DB27dpFdXU1H330ETk5OYSGhjJmzBhSU1M929yyZQv33nsvKSkpWK1W7r77bkaMGMH777/fKe+J9Ew6hyFyE6tXr77mHMbGjRuJi4vDYrF4lg0cOJC6urpr1n/uuecoKiri4Ycfpn///kybNo2HH36Yurq6Vg+1ubKN2tpa6urqCA8Pp2/fvq3GrtyNtrq6mu3bt7Nr1y7PeEtLC2PHjvVJzyLXo8AQaafa2loMw/CEhsPhaHUUcEVMTAxLliwB4MCBA0ybNo0xY8YQGxvb6hkFV7bx3e9+l5iYGM6cOcOFCxc8oVFdXe15LbvdTkZGhme7Ip1BU1Ii7dTQ0MAbb7zB5cuX2bZtGydOnCAlJeWan9u2bRs1NTUA9O/fH4vFgtVqJSUlhZMnT7J161ZaWlp49913OX78ON/73vdISEhgxIgRvPLKK1y6dIkDBw60OppIT09n165d7NmzB5fLRXNzM2VlZZ7XEfEHHWGI3MTjjz/e6u8w7rrrLsaNG8fIkSM5deoU3/72t4mOjuY3v/kNAwYMuGb9jz76iGXLlnHu3DmioqKYP38+gwcPBr64ZHfZsmXk5+czZMgQ1qxZ43l40Ysvvkhubi5jx45l1KhRZGZmcubMGeCLI4zi4mJWrFjBs88+i9VqZeTIkeTn5/v/DZEeS8/DEGmHjRs3smHDBtatWxfoUkQ6jaakRETEFAWGiIiYoikpERExRUcYIiJiigJDRERMUWCIiIgpCgwRETFFgSEiIqYoMERExJT/A/e0lIt5oJ/WAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wDnK16YvFqzH",
        "outputId": "ea210503-b0a1-4bff-bb20-2988fb488061",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "#Getting the action probabilities vector\n",
        "np.array(sarsa.q_values)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.9960373 , 0.00396272],\n",
              "       [0.9601623 , 0.0398377 ],\n",
              "       [0.9386001 , 0.06139983],\n",
              "       ...,\n",
              "       [0.49630713, 0.50369287],\n",
              "       [0.5007292 , 0.49927077],\n",
              "       [0.49682516, 0.50317484]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WzCQjTRo03DK",
        "colab": {}
      },
      "source": [
        "#Saving our model\n",
        "sarsa.save_weights('sarsa_{}_weights.h5f'.format(env), overwrite=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1yhNZR83zgmo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}