model-based-learning/value-based/async-dqn/asynchronous_n_step_dqn_lunarlander.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Asynchronous N Steps Deep Q-Network (AsyncNDQN)\n",
    "\n",
    "### 0.Background: From One Step to N Step Asynchronous DQN\n",
    "\n",
    "Traditional **[Deep Q-Networks (DQN)](../vanilla-dqn/dqn_lunarlander.ipynb)** have been highly successful in solving reinforcement learning tasks by combining Q-learning with deep neural networks. However, they often require significant computational resources and suffer from stability issues during training. To address these challenges, **[Asynchronous One Step DQN](./asynchronous_one_step_dqn_lunarlander.ipynb)** introduces the concept of **asynchronous multi-threading**, where multiple agents (or threads) interact with their own copies of the environment in parallel. This approach provides several benefits:\n",
    "\n",
    "1. **Improved computational efficiency**:\n",
    "   - By leveraging multiple threads, the algorithm can make better use of multi-core CPUs, enabling faster training without requiring expensive GPUs.\n",
    "\n",
    "2. **Stabilized training**:\n",
    "   - Asynchronous updates reduce the correlation between consecutive updates, which helps mitigate the instability caused by highly correlated training data in traditional DQN.\n",
    "\n",
    "3. **Enhanced exploration**:\n",
    "   - By assigning different exploration policies to each thread, Asynchronous DQN encourages diverse exploration, leading to better coverage of the state-action space.\n",
    "\n",
    "\n",
    "And with **n-step return**, further improvements are achieved by extending the one-step update mechanism to incorporate rewards from multiple future steps. This modification provides several key benefits:\n",
    "1. **Reduced Variance in Updates**:\n",
    "   - By averaging the rewards over `n` steps, the updates become less sensitive to noise in individual rewards, leading to more stable learning.\n",
    " \n",
    "2. **Improved Credit Assignment**:\n",
    "   - n-step returns allow the algorithm to assign credit more effectively across multiple time steps, which is particularly advantageous in environments with delayed rewards or sparse feedback.\n",
    " \n",
    "3. **Faster Convergence**:\n",
    "   - With n-step returns, the algorithm can capture longer-term dependencies in the environment, enabling more efficient learning and faster convergence compared to one-step methods.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Import the Necessary Packages\n",
    "\n",
    "In this notebook, we will implement a DQN agent with OpenAI Gym's LunarLander-v2 environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on: cpu\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import gymnasium as gym\n",
    "from collections import deque\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import random\n",
    "\n",
    "# for game rendering\n",
    "import time\n",
    "from PIL import Image\n",
    "from IPython import display\n",
    "\n",
    "device = torch.device(\"cpu\")\n",
    "print(f\"Training on: {device}\")\n",
    "\n",
    "np.set_printoptions(precision=3, linewidth=120)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Explore Environment\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "    <img src=\"./images/lunar_lander.gif\" alt=\"Mountain Car Environment\" width=\"50%\">\n",
    "</div>\n",
    "\n",
    "#### Discreate Action Space\n",
    "\n",
    "There are four discrete actions available:\n",
    "\n",
    "- 0: do nothing\n",
    "- 1: fire left orientation engine\n",
    "- 2: fire main engine\n",
    "- 3: fire right orientation engine\n",
    "\n",
    "#### Continuous Observation Space\n",
    "\n",
    "The state is an 8-dimensional vector: \n",
    "\n",
    "- the coordinates of the lander in `x`\n",
    "- the coordinates of the lander in`y`, \n",
    "- linear velocities in `x` \n",
    "- linear velocities in `y`, \n",
    "- its angle, \n",
    "- its angular velocity, and \n",
    "- a booleans that represent whether `left` leg is in contact with the ground or not.\n",
    "- a booleans that represent whether `right` leg is in contact with the ground or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make('LunarLander-v3', render_mode=\"rgb_array\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State space:\n",
      " Continuous (8,)\n",
      " - low: [ -2.5    -2.5   -10.    -10.     -6.283 -10.     -0.     -0.   ]\n",
      " - high: [ 2.5    2.5   10.    10.     6.283 10.     1.     1.   ]\n",
      "Action space:\n",
      " Discrete(4)\n",
      "--------------------------------------------------\n",
      "State space 10 samples:\n",
      "[[ 1.388 -2.    -0.38  -3.525 -3.051 -3.775  0.711  0.071]\n",
      " [-2.244 -1.125 -7.72   8.819 -1.307  9.409  0.212  0.828]\n",
      " [ 1.905 -0.419 -5.357  4.421 -0.714  9.898  0.749  0.924]\n",
      " [-0.68  -1.356  6.445  2.214  0.135  4.68   0.191  0.892]\n",
      " [-0.443 -0.242  2.904  0.653  1.091  2.872  0.317  0.43 ]\n",
      " [ 1.436  2.121  0.156 -4.81   6.207 -4.851  0.751  0.138]\n",
      " [ 2.135 -2.151 -8.637  9.423 -2.296 -6.163  0.84   0.767]\n",
      " [-0.942  0.093  0.246  6.68   0.694 -2.961  0.229  0.694]\n",
      " [ 0.982  2.089  9.559 -3.056 -4.143 -6.456  0.398  0.699]\n",
      " [ 1.061 -0.368 -1.596  4.37   2.957 -8.076  0.636  0.998]]\n",
      "Action space 10 samples:\n",
      "[1 2 2 2 2 2 3 1 2 1]\n"
     ]
    }
   ],
   "source": [
    "# Explore state (observation) space\n",
    "print(\"State space:\\n Continuous\", env.observation_space.shape)\n",
    "print(\" - low:\", env.observation_space.low)\n",
    "print(\" - high:\", env.observation_space.high)\n",
    "\n",
    "# Explore action space\n",
    "print(\"Action space:\\n\", env.action_space)\n",
    "\n",
    "print(\"-\"*50)\n",
    "# Generate some samples from the state space \n",
    "print(\"State space 10 samples:\")\n",
    "print(np.array([env.observation_space.sample() for i in range(10)]))\n",
    "\n",
    "# Generate some samples from the action space\n",
    "print(\"Action space 10 samples:\")\n",
    "print(np.array([env.action_space.sample() for i in range(10)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Remove Replay Buffer\n",
    "\n",
    "#### Using Experience Replay Improve Training Stability\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "    <img src=\"./images/reply-buffer.png\" alt=\"Mountain Car Environment\" width=\"70%\">\n",
    "</div>\n",
    "\n",
    "When an agent interacts with the environment, the sequence of experience tuples can be highly correlated, which poses a risk for naive Q-learning algorithms. These algorithms, when learning sequentially from such correlated data, may lead to unstable updates, causing action values to oscillate or diverge.\n",
    "\n",
    "To address this, a replay buffer is introduced to store experience tuples $(S, A, R, S')$ collected during interactions with the environment. By using **experience replay**, small batches of tuples are randomly sampled from the buffer for training. This random sampling breaks harmful correlations, stabilizes learning, and allows the agent to:  \n",
    "1. Reuse individual experience tuples multiple times.  \n",
    "2. Recall rare events.  \n",
    "3. Make better overall use of past experiences.  \n",
    "\n",
    "Experience replay thus improves the efficiency and stability of the learning process.\n",
    "\n",
    "<span style=\"color: red;\">Update: </span> However, asynchronous methods eliminate the need for a replay buffer. In asynchronous one-step Q-learning and related methods, multiple agents (threads) interact with their own copies of the environment in parallel. This naturally introduces diversity into the data collected by each thread, breaking the correlation between consecutive updates. As a result:\n",
    "\n",
    "The updates from different threads act as a form of implicit decorrelation, reducing the risk of instability.\n",
    "The computational complexity of managing a replay buffer is avoided, making the algorithm simpler and more efficient.\n",
    "Thus, asynchronous mechanisms provide a built-in alternative to experience replay, achieving stability and efficiency without the need for a replay buffer.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Define Q Network\n",
    "\n",
    "#### Using Neural Networks as Approximators for Q values\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "    <img src=\"./images/model-view.png\" alt=\"Mountain Car Environment\" width=\"70%\">\n",
    "</div>\n",
    "\n",
    "This image shows the transition from a **Traditional Q Table** to a **Parameterized Q Function** in reinforcement learning. \n",
    "\n",
    "The Q table (left) stores discrete Q-values $ Q(s_t, a_t) $ for each state-action pair but struggles with scalability in high-dimensional spaces. The parameterized Q function (right) replaces the table with a neural network $ Q(s_t, a_t; w) $, where $ w $ are the network's parameters.\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "    <img src=\"./images/model-arch.png\" alt=\"Mountain Car Environment\" width=\"70%\">\n",
    "</div>\n",
    "\n",
    "- Define a neural network architecture that maps states to action values $Q(s_t, a_t; w)$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 4])\n"
     ]
    }
   ],
   "source": [
    "class QNetwork(nn.Module):\n",
    "    \"\"\"Actor (Policy) Model.\"\"\"\n",
    "\n",
    "    def __init__(self, state_size, action_size, hidden_size=64, seed=42):\n",
    "        \"\"\"Initialize parameters and build model.\n",
    "        Params\n",
    "        ======\n",
    "            state_size (int): Dimension of each state\n",
    "            action_size (int): Dimension of each action\n",
    "            seed (int): Random seed\n",
    "        \"\"\"\n",
    "        super(QNetwork, self).__init__()\n",
    "        self.seed = torch.manual_seed(seed)\n",
    "        self.Q = nn.Sequential(\n",
    "            nn.Linear(state_size, hidden_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hidden_size, hidden_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hidden_size, action_size)\n",
    "        )\n",
    "\n",
    "    def forward(self, state):\n",
    "        \"\"\"Build a network that maps state -action values.\"\"\"\n",
    "        actions = self.Q(state)\n",
    "        return actions\n",
    "\n",
    "\n",
    "# test model\n",
    "q_net = QNetwork(8, 4, 16, seed=42)\n",
    "# fake input, by given batch size 4\n",
    "states = torch.rand((4, 8))\n",
    "# fake output size\n",
    "print(q_net(states).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Define Agent\n",
    "\n",
    "##### How to Learn in DQN\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "    <img src=\"./images/dqn-gradient-descent.png\" alt=\"Mountain Car Environment\" width=\"70%\">\n",
    "</div>\n",
    "\n",
    "This image provides a visual summary of the core theory and update mechanism of **Deep Q-Network (DQN)**, showcasing the key formula derivations from **Q-Learning** to **DQN**, as well as how neural networks are used to approximate and update Q-values.\n",
    "\n",
    "---\n",
    "\n",
    "**1. Core Idea Based on the Bellman Equation**\n",
    "- **Bellman Equation:**\n",
    "  $$\n",
    "  Q^*(s_t, a_t) = R_t + \\gamma \\max_a Q^*(s_{t+1}, a)\n",
    "  $$\n",
    "  - This states that the optimal Q-value for the current state-action pair equals the immediate reward $ R_t $ plus the discounted maximum Q-value of the next state.\n",
    "  - The core objective of DQN is to approximate this equation using a neural network.\n",
    "\n",
    "**2. Derivation from Q-Learning to DQN**\n",
    "- **Q-Learning Update Formula:**\n",
    "  $$\n",
    "  Q(s_t, a_t) \\leftarrow Q(s_t, a_t) + \\alpha \\left( R_t + \\gamma \\max_a Q(s_{t+1}, a) - Q(s_t, a_t) \\right)\n",
    "  $$\n",
    "  - **TD Target:**\n",
    "    $$\n",
    "    R_t + \\gamma \\max_a Q(s_{t+1}, a)\n",
    "    $$\n",
    "    This represents the target Q-value for the current state, combining the immediate reward $ R_t $ and the maximum Q-value of the next state.\n",
    "  - **Current Value:**\n",
    "    $$\n",
    "    Q(s_t, a_t)\n",
    "    $$\n",
    "    This is the current estimated Q-value.\n",
    "  - **TD Error:**\n",
    "    $$\n",
    "    \\delta_t = \\left( R_t + \\gamma \\max_a Q(s_{t+1}, a) - Q(s_t, a_t) \\right)\n",
    "    $$\n",
    "    This measures the difference between the target Q-value and the current Q-value.\n",
    "\n",
    "- **Neural Network Introduction:**\n",
    "  - To handle continuous state spaces, DQN replaces the traditional Q-table with a neural network $ Q(s, a; w) $, where $ w $ represents the network parameters.\n",
    "  - The goal is to optimize the network so that its output $ Q(s, a; w) $ approximates the true $ Q^*(s, a) $.\n",
    "\n",
    "\n",
    "**3. Loss Function and Gradient Update in DQN**\n",
    "- **Loss Function:**\n",
    "  $$\n",
    "  L(w) = \\frac{1}{2} \\left[ Q(s_t, a_t; w) - Q^*(s_t, a_t) \\right]^2\n",
    "  $$\n",
    "  - Here, $ Q^*(s_t, a_t) $ is the target Q-value (TD Target) calculated using the Bellman equation.\n",
    "  - The loss function minimizes the squared error between the network's output $ Q(s_t, a_t; w) $ and the target Q-value $ Q^*(s_t, a_t) $.\n",
    "\n",
    "- **Gradient Calculation:**\n",
    "  $$\n",
    "  \\nabla_w L(w) = \\left( Q(s_t, a_t; w) - Q^*(s_t, a_t) \\right) \\nabla_w Q(s_t, a_t; w)\n",
    "  $$\n",
    "  - The gradient consists of two parts:\n",
    "    1. The error term: $ Q(s_t, a_t; w) - Q^*(s_t, a_t) $\n",
    "    2. The gradient of the network output: $ \\nabla_w Q(s_t, a_t; w) $\n",
    "\n",
    "- **Weight Update:**\n",
    "  $$\n",
    "  w \\leftarrow w - \\alpha \\nabla_w L(w)\n",
    "  $$\n",
    "  - Using gradient descent, the network parameters $ w $ are updated to reduce the loss function.\n",
    "\n",
    "- **Final DQN Update Formula:**\n",
    "  $$\n",
    "  L(w) = \\frac{1}{2} \\left[ Q(s_t, a_t; w) - (R_t + \\gamma \\max_a Q(s_{t+1}, a; w^-)) \\right]^2\n",
    "  $$\n",
    "\n",
    "  $$\n",
    "  w' \\leftarrow w + \\alpha \\left( R_t + \\gamma \\max_a Q(s_{t+1}, a; w^-) - Q(s_t, a_t; w) \\right) \\nabla_w Q(s_t, a_t; w)\n",
    "  $$\n",
    "  - Here, $ w^- $ represents the fixed target network parameters during the learning step, which are used to stabilize training.\n",
    "\n",
    "\n",
    "<span style=\"color: red;\">Update: </span> **Asynchronous N Steps Version**\n",
    "\n",
    "Like Paper [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/pdf/1602.01783) illustrated, we can implement Asynchronous N steps DQN like below with few modifications\n",
    "\n",
    "From original paper, the left diagram shows the pseudocode for the standard Asynchronous One Step DQN, while the right diagram highlights the pseudocode for the n-step Asynchronous DQN. The key difference is the use of n-step returns in the update equation, which balances between bias and variance, enabling faster convergence compared to the standard [Asynchronous One Step DQN](./asynchronous_one_step_dqn_lunarlander.ipynb)\n",
    "\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "    <img src=\"./images/one-step-q-and-n-step-q.png\" width=\"80%\">\n",
    "</div>\n",
    "\n",
    "\n",
    "**Key Differences from One-Step Asynchronous DQN**\n",
    " \n",
    "- **Target Calculation**: Instead of using a single reward and the next state-action pair, n-step DQN aggregates rewards over multiple steps, making the target more informative.\n",
    "- **Exploration vs. Exploitation**: n-step returns help balance exploration and exploitation by incorporating longer-term rewards, which can lead to more efficient policy improvement.\n",
    "- **Stability**: The use of n-step returns reduces the high-frequency noise present in one-step updates, improving convergence stability.\n",
    "\n",
    "**How to Choose N**\n",
    "- **Choosing `n`**: The value of `n` is a critical hyperparameter. A small `n` may not capture sufficient future information, while a very large `n` can introduce high variance. Typically, values like `n=5` work well in practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Agent:\n",
    "    \"\"\"Interacts with and learns from the environment.\"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        state_size,\n",
    "        action_size,\n",
    "        Q_network,\n",
    "        Q_target_network=None,\n",
    "        optimizer=None,\n",
    "        gamma=0.99,\n",
    "        target_update_steps=1000,\n",
    "        n_steps=5,\n",
    "        seed=42,\n",
    "        T=None,\n",
    "        optimizer_lock=None\n",
    "    ):\n",
    "        \"\"\"\n",
    "        Initialize the agent.\n",
    "        Args:\n",
    "            state_size (int): Dimension of the state space.\n",
    "            action_size (int): Dimension of the action space.\n",
    "            Q_network (QNetwork): Main Q-Network.\n",
    "            Q_target_network (QNetwork): Target Q-Network.\n",
    "            optimizer (torch.optim.Optimizer): Optimizer for training.\n",
    "            gamma (float): Discount factor.\n",
    "            target_update_steps (int): Steps between target network updates.\n",
    "            n_steps (int): for N Steps return between gradient application.\n",
    "            seed (int): Random seed for reproducibility.\n",
    "            T (multiprocessing.Value): Global step counter shared across processes.\n",
    "            optimizer_lock (multiprocessing.Lock): Lock for synchronizing optimizer updates.\n",
    "        \"\"\"\n",
    "        self.state_size = state_size\n",
    "        self.action_size = action_size\n",
    "        self.Q = Q_network\n",
    "        self.Q_target = Q_target_network\n",
    "        self.optimizer = optimizer\n",
    "        self.gamma = gamma\n",
    "        self.target_update_steps = target_update_steps\n",
    "        self.memory = []  # Store N steps experiences\n",
    "        self.n_steps = n_steps  # Number of steps after which to apply gradients\n",
    "        self.t_step = 0  # Time step counter for this thread\n",
    "        self.T = T  # Shared global counter\n",
    "        self.optimizer_lock = optimizer_lock\n",
    "        self.seed = np.random.seed(seed)\n",
    "        random.seed(seed)\n",
    "\n",
    "        # Initialize accumulated gradients as None\n",
    "        self.reset_gradients()\n",
    "    \n",
    "    def reset_gradients(self):\n",
    "        \"\"\"Reset accumulated gradients.\"\"\"\n",
    "        self.accumulated_grads = [torch.zeros_like(p) for p in self.Q.parameters()]\n",
    "    \n",
    "    def select_action(self, state, epsilon=0.0):\n",
    "        \"\"\"Selects an action using epsilon-greedy policy.\"\"\"\n",
    "        state = torch.from_numpy(state).float().unsqueeze(0).to(device)\n",
    "        self.Q.eval()\n",
    "        with torch.no_grad():\n",
    "            actions = self.Q(state)\n",
    "        self.Q.train()\n",
    "\n",
    "        # Epsilon-greedy action selection\n",
    "        if random.random() > epsilon:\n",
    "            return np.argmax(actions.cpu().data.numpy())\n",
    "        else:\n",
    "            return random.choice(np.arange(self.action_size))\n",
    "\n",
    "    def step(self, state, action, reward, next_state, done):\n",
    "        \"\"\"Processes a step and learns from the experience.\"\"\"\n",
    "        # repeat add experience\n",
    "        state = torch.tensor(state, dtype=torch.float).unsqueeze(0).to(device)\n",
    "        action = torch.tensor(action, dtype=torch.int64).reshape(1, 1).to(device)\n",
    "        reward = torch.tensor(reward, dtype=torch.float).reshape(1, 1).to(device)\n",
    "        next_state = torch.tensor(next_state, dtype=torch.float32).unsqueeze(0).to(device)\n",
    "        done = torch.tensor(done, dtype=torch.float32).reshape(1, 1).to(device)\n",
    "        \n",
    "        self.memory.append((state, action, reward, next_state, done))\n",
    "\n",
    "        # Increment step counters\n",
    "        self.t_step += 1\n",
    "        with self.T.get_lock():\n",
    "            self.T.value += 1\n",
    "\n",
    "        if done or (self.t_step == self.n_steps):\n",
    "            # Q_target is R\n",
    "            Q_target = 0.0\n",
    "            if not done:\n",
    "                with torch.no_grad():\n",
    "                    Q_target = torch.max(self.Q_target(next_state), dim=-1, keepdim=True)[0]\n",
    "            \n",
    "            for i in reversed(range(0, self.t_step)):\n",
    "                # reverse iterate experience from last to first\n",
    "                state, action, reward, next_state, done = self.memory[i]\n",
    "                \n",
    "                # Compute TD target using the target network\n",
    "                Q_target = reward + self.gamma * Q_target\n",
    "                \n",
    "                # Perform learning step\n",
    "                loss = self.compute_loss(state, action, Q_target)\n",
    "\n",
    "                # Backpropagate loss and accumulate gradients\n",
    "                self.Q.zero_grad()\n",
    "                loss.backward()\n",
    "\n",
    "                # Accumulate gradients\n",
    "                with torch.no_grad():\n",
    "                    for acc_grad, param in zip(self.accumulated_grads, self.Q.parameters()):\n",
    "                        acc_grad += param.grad.clone()\n",
    "\n",
    "            # Apply the accumulated gradients\n",
    "            self.apply_gradients()\n",
    "\n",
    "            # Reset accumulated gradients\n",
    "            self.reset_gradients()\n",
    "\n",
    "            # Reset memory and time step counter\n",
    "            self.memory = []\n",
    "            self.t_step = 0\n",
    "        \n",
    "        # Update target network\n",
    "        if self.T.value % self.target_update_steps == 0:\n",
    "            self.hard_update()\n",
    "        \n",
    "    def compute_loss(self, state, action, Q_target):\n",
    "        \"\"\"Computes the loss for a single experience tuple.\"\"\"\n",
    "\n",
    "        # Compute expected Q values using the local network\n",
    "        Q_expected = torch.gather(self.Q(state), dim=-1, index=action)\n",
    "\n",
    "        # Compute loss (mean squared error)\n",
    "        loss = F.mse_loss(Q_expected, Q_target)\n",
    "        return loss\n",
    "\n",
    "    def apply_gradients(self):\n",
    "        \"\"\"Apply accumulated gradients to the shared network.\"\"\"\n",
    "        with self.optimizer_lock:\n",
    "            for param, acc_grad in zip(self.Q.parameters(), self.accumulated_grads):\n",
    "                param.grad = acc_grad  # Set the accumulated gradients\n",
    "\n",
    "            # Perform optimizer step\n",
    "            self.optimizer.step()\n",
    "            # Zero the parameter gradients (in case they weren't zeroed)\n",
    "            self.optimizer.zero_grad()\n",
    " \n",
    "    def hard_update(self):\n",
    "        \"\"\"Hard update: θ_target = θ\"\"\"\n",
    "        with self.optimizer_lock:\n",
    "            self.Q_target.load_state_dict(self.Q.state_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Train the Agent with Async N DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SharedAdam(optim.Adam):\n",
    "    \"\"\"Shared Adam optimizer for multiprocessing\"\"\"\n",
    "\n",
    "    def __init__(self, params, lr=5e-4):\n",
    "        \"\"\"\n",
    "        Initialize the shared Adam optimizer.\n",
    "        Args:\n",
    "            params: Parameters to optimize.\n",
    "            lr (float): Learning rate.\n",
    "        \"\"\"\n",
    "        super(SharedAdam, self).__init__(params, lr=lr)\n",
    "        for group in self.param_groups:\n",
    "            for p in group['params']:\n",
    "                # Initialize optimizer state\n",
    "                state = self.state[p]\n",
    "                state['step'] = torch.tensor(0)\n",
    "                state['exp_avg'] = torch.zeros_like(p.data)\n",
    "                state['exp_avg_sq'] = torch.zeros_like(p.data)\n",
    "                # Share memory for multiprocessing\n",
    "                state['exp_avg'].share_memory_()\n",
    "                state['exp_avg_sq'].share_memory_()\n",
    "                state['step'].share_memory_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def worker_dqn(rank, Q, Q_target, optimizer, T, max_score, env_name, num_episodes=2000, max_t=1000, window=100, optimizer_lock=None):\n",
    "    \"\"\"\n",
    "    Function executed by each worker process.\n",
    "    Args:\n",
    "        rank (int): Process rank.\n",
    "        Q (QNetwork): Shared Q-network.\n",
    "        Q_target (QNetwork): Shared target Q-network.\n",
    "        optimizer (SharedAdam): Shared optimizer.\n",
    "        T (multiprocessing.Value): Global step counter.\n",
    "        max_score (multiprocessing.Value): Global maximum score.\n",
    "        env_name (str): Environment name.\n",
    "        num_episodes (int): Number of training episodes.\n",
    "        max_t (int): Maximum steps per episode.\n",
    "        window (int): Window size for calculating average scores.\n",
    "        optimizer_lock (multiprocessing.Lock): Lock for synchronizing optimizer updates.\n",
    "    \"\"\"\n",
    "    # Create an environment instance for each worker\n",
    "    env = gym.make(env_name)\n",
    "    state_size = env.observation_space.shape[0]\n",
    "    action_size = env.action_space.n\n",
    "\n",
    "    # Initialize the agent with shared networks and optimizer\n",
    "    agent = Agent(\n",
    "        state_size=state_size,\n",
    "        action_size=action_size,\n",
    "        Q_network=Q,\n",
    "        Q_target_network=Q_target,\n",
    "        optimizer=optimizer,\n",
    "        gamma=0.995,\n",
    "        target_update_steps=50,\n",
    "        n_steps=5,\n",
    "        seed=42 + rank,\n",
    "        T=T,\n",
    "        optimizer_lock=optimizer_lock\n",
    "    )\n",
    "    \n",
    "    epsilon = 1.0\n",
    "    eps_min = 0.01\n",
    "    eps_decay = 0.999\n",
    "    \n",
    "    scores_window = deque(maxlen=window)\n",
    "\n",
    "    for i_episode in range(1, num_episodes + 1):\n",
    "        with max_score.get_lock():\n",
    "            if max_score.value >= 200.0:\n",
    "                break\n",
    "            \n",
    "        state, _ = env.reset()\n",
    "        total_reward = 0\n",
    "        agent.t_step = 0\n",
    "        agent.reset_gradients()\n",
    "        agent.memory = []\n",
    "        done = False\n",
    "\n",
    "        for t in range(max_t):\n",
    "            action = agent.select_action(state, epsilon)\n",
    "            next_state, reward, done, _, info = env.step(action)\n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            state = next_state\n",
    "            total_reward += reward\n",
    "            if done:\n",
    "                break\n",
    "        \n",
    "        # Decay epsilon\n",
    "        epsilon = max(eps_min, eps_decay * epsilon)\n",
    "        \n",
    "        scores_window.append(total_reward)\n",
    "        mean_score = np.mean(scores_window)\n",
    "\n",
    "        if i_episode % 10 == 0:\n",
    "            print(f\"Process {rank}, Episode {i_episode}, Total Reward: {total_reward:.2f}, Average Score: {mean_score:.2f}\")\n",
    "\n",
    "        if len(scores_window) >= window and mean_score >= 200.0:\n",
    "            with max_score.get_lock():\n",
    "                max_score.value = mean_score\n",
    "                print(f\"\\nEnvironment solved in {i_episode:d} episodes!\\tAverage Score: {mean_score:.2f}\")\n",
    "                torch.save(agent.Q.state_dict(), \"checkpoint.pth\")\n",
    "                break\n",
    "\n",
    "    env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color: red;\">Update: </span> Below code in this [script](./asynchronous_n_step_dqn_lunarlander.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on: cpu\n",
      "Training on: cpu\n",
      "Training on: cpu\n",
      "Training on: cpu\n",
      "Training on: cpu\n",
      "Training on: cpu\n",
      "Training on: cpu\n",
      "Training on: cpu\n",
      "Training on: cpu\n",
      "Process 7, Episode 10, Total Reward: -211.14, Average Score: -164.47\n",
      "Process 1, Episode 10, Total Reward: -137.78, Average Score: -205.91\n",
      "Process 0, Episode 10, Total Reward: 16.52, Average Score: -153.01\n",
      "Process 6, Episode 10, Total Reward: -194.13, Average Score: -174.23\n",
      "Process 3, Episode 10, Total Reward: -184.83, Average Score: -197.16\n",
      "Process 4, Episode 10, Total Reward: -141.82, Average Score: -138.13\n",
      "Process 5, Episode 10, Total Reward: -111.30, Average Score: -196.16\n",
      "Process 2, Episode 10, Total Reward: -196.07, Average Score: -180.19\n",
      "Process 7, Episode 20, Total Reward: -88.23, Average Score: -151.91\n",
      "Process 3, Episode 20, Total Reward: -91.74, Average Score: -169.60\n",
      "Process 0, Episode 20, Total Reward: -416.43, Average Score: -186.72\n",
      "Process 1, Episode 20, Total Reward: -128.14, Average Score: -229.99\n",
      "Process 6, Episode 20, Total Reward: -311.00, Average Score: -188.45\n",
      "Process 4, Episode 20, Total Reward: -295.61, Average Score: -169.81\n",
      "Process 5, Episode 20, Total Reward: -117.13, Average Score: -191.80\n",
      "Process 2, Episode 20, Total Reward: -135.56, Average Score: -153.47\n",
      "Process 7, Episode 30, Total Reward: -314.78, Average Score: -160.44\n",
      "Process 3, Episode 30, Total Reward: -271.17, Average Score: -185.94\n",
      "Process 5, Episode 30, Total Reward: -353.64, Average Score: -198.97\n",
      "Process 4, Episode 30, Total Reward: -191.22, Average Score: -172.25\n",
      "Process 1, Episode 30, Total Reward: -124.82, Average Score: -207.04\n",
      "Process 2, Episode 30, Total Reward: -450.34, Average Score: -146.12\n",
      "Process 0, Episode 30, Total Reward: -133.64, Average Score: -189.78\n",
      "Process 1, Episode 40, Total Reward: -125.87, Average Score: -195.59\n",
      "Process 7, Episode 40, Total Reward: -56.00, Average Score: -158.42\n",
      "Process 5, Episode 40, Total Reward: -118.77, Average Score: -193.12\n",
      "Process 4, Episode 40, Total Reward: -113.37, Average Score: -165.15\n",
      "Process 3, Episode 40, Total Reward: -102.60, Average Score: -186.40\n",
      "Process 2, Episode 40, Total Reward: -165.16, Average Score: -163.11\n",
      "Process 6, Episode 30, Total Reward: -310.09, Average Score: -162.95\n",
      "Process 0, Episode 40, Total Reward: -173.24, Average Score: -164.89\n",
      "Process 5, Episode 50, Total Reward: -202.87, Average Score: -175.71\n",
      "Process 1, Episode 50, Total Reward: -88.98, Average Score: -191.24\n",
      "Process 7, Episode 50, Total Reward: -292.51, Average Score: -158.88\n",
      "Process 4, Episode 50, Total Reward: -250.16, Average Score: -175.82\n",
      "Process 2, Episode 50, Total Reward: -335.31, Average Score: -165.10\n",
      "Process 3, Episode 50, Total Reward: -103.00, Average Score: -185.52\n",
      "Process 6, Episode 40, Total Reward: -127.77, Average Score: -169.40\n",
      "Process 0, Episode 50, Total Reward: -123.48, Average Score: -156.96\n",
      "Process 5, Episode 60, Total Reward: -287.63, Average Score: -170.17\n",
      "Process 1, Episode 60, Total Reward: -114.32, Average Score: -184.92\n",
      "Process 7, Episode 60, Total Reward: -215.07, Average Score: -156.73\n",
      "Process 4, Episode 60, Total Reward: -115.71, Average Score: -182.06\n",
      "Process 2, Episode 60, Total Reward: -91.99, Average Score: -162.10\n",
      "Process 6, Episode 50, Total Reward: -181.80, Average Score: -179.35\n",
      "Process 3, Episode 60, Total Reward: -161.14, Average Score: -179.36\n",
      "Process 0, Episode 60, Total Reward: -4.71, Average Score: -159.04\n",
      "Process 5, Episode 70, Total Reward: -125.96, Average Score: -166.29\n",
      "Process 1, Episode 70, Total Reward: -121.27, Average Score: -182.35\n",
      "Process 7, Episode 70, Total Reward: -132.10, Average Score: -155.39\n",
      "Process 4, Episode 70, Total Reward: -212.53, Average Score: -179.00\n",
      "Process 2, Episode 70, Total Reward: -291.85, Average Score: -165.17\n",
      "Process 6, Episode 60, Total Reward: -160.17, Average Score: -173.91\n",
      "Process 3, Episode 70, Total Reward: -195.96, Average Score: -184.45\n",
      "Process 0, Episode 70, Total Reward: -81.88, Average Score: -161.55\n",
      "Process 5, Episode 80, Total Reward: 0.65, Average Score: -161.51\n",
      "Process 4, Episode 80, Total Reward: -142.52, Average Score: -175.10\n",
      "Process 7, Episode 80, Total Reward: -397.59, Average Score: -158.88\n",
      "Process 2, Episode 80, Total Reward: -102.78, Average Score: -163.29\n",
      "Process 0, Episode 80, Total Reward: -135.21, Average Score: -158.67\n",
      "Process 6, Episode 70, Total Reward: -275.52, Average Score: -169.88\n",
      "Process 3, Episode 80, Total Reward: -113.31, Average Score: -187.06\n",
      "Process 5, Episode 90, Total Reward: -291.88, Average Score: -161.82\n",
      "Process 1, Episode 80, Total Reward: -223.91, Average Score: -180.96\n",
      "Process 4, Episode 90, Total Reward: -327.78, Average Score: -174.00\n",
      "Process 7, Episode 90, Total Reward: -190.74, Average Score: -160.09\n",
      "Process 2, Episode 90, Total Reward: -77.70, Average Score: -162.77\n",
      "Process 6, Episode 80, Total Reward: -98.43, Average Score: -169.51\n",
      "Process 0, Episode 90, Total Reward: -118.60, Average Score: -154.43\n",
      "Process 3, Episode 90, Total Reward: -154.26, Average Score: -182.16\n",
      "Process 5, Episode 100, Total Reward: -110.89, Average Score: -160.72\n",
      "Process 1, Episode 90, Total Reward: -175.88, Average Score: -180.34\n",
      "Process 4, Episode 100, Total Reward: -233.80, Average Score: -170.82\n",
      "Process 7, Episode 100, Total Reward: -72.79, Average Score: -161.44\n",
      "Process 2, Episode 100, Total Reward: -79.47, Average Score: -159.03\n",
      "Process 6, Episode 90, Total Reward: -322.44, Average Score: -169.21\n",
      "Process 3, Episode 100, Total Reward: -128.22, Average Score: -179.94\n",
      "Process 0, Episode 100, Total Reward: -131.62, Average Score: -151.79\n",
      "Process 1, Episode 100, Total Reward: -100.85, Average Score: -172.57\n",
      "Process 5, Episode 110, Total Reward: -140.56, Average Score: -156.35\n",
      "Process 4, Episode 110, Total Reward: -88.63, Average Score: -168.04\n",
      "Process 7, Episode 110, Total Reward: -82.00, Average Score: -158.49\n",
      "Process 2, Episode 110, Total Reward: -132.24, Average Score: -156.45\n",
      "Process 6, Episode 100, Total Reward: -92.88, Average Score: -165.39\n",
      "Process 0, Episode 110, Total Reward: -130.72, Average Score: -152.41\n",
      "Process 3, Episode 110, Total Reward: -172.18, Average Score: -173.47\n",
      "Process 1, Episode 110, Total Reward: -196.33, Average Score: -167.57\n",
      "Process 5, Episode 120, Total Reward: -187.94, Average Score: -153.83\n",
      "Process 4, Episode 120, Total Reward: -71.94, Average Score: -162.52\n",
      "Process 7, Episode 120, Total Reward: -32.71, Average Score: -161.20\n",
      "Process 2, Episode 120, Total Reward: -134.92, Average Score: -153.34\n",
      "Process 6, Episode 110, Total Reward: -118.50, Average Score: -160.45\n",
      "Process 3, Episode 120, Total Reward: -186.79, Average Score: -178.45\n",
      "Process 1, Episode 120, Total Reward: -119.16, Average Score: -159.07\n",
      "Process 5, Episode 130, Total Reward: -183.51, Average Score: -149.10\n",
      "Process 4, Episode 130, Total Reward: -83.71, Average Score: -162.49\n",
      "Process 2, Episode 130, Total Reward: -122.05, Average Score: -156.06\n",
      "Process 0, Episode 120, Total Reward: -142.90, Average Score: -141.71\n",
      "Process 7, Episode 130, Total Reward: -41.65, Average Score: -152.25\n",
      "Process 6, Episode 120, Total Reward: -141.84, Average Score: -154.94\n",
      "Process 3, Episode 130, Total Reward: -90.59, Average Score: -170.54\n",
      "Process 5, Episode 140, Total Reward: -355.48, Average Score: -146.07\n",
      "Process 1, Episode 130, Total Reward: -124.25, Average Score: -156.62\n",
      "Process 4, Episode 140, Total Reward: -136.52, Average Score: -160.25\n",
      "Process 0, Episode 130, Total Reward: 4.60, Average Score: -132.43\n",
      "Process 7, Episode 140, Total Reward: -40.25, Average Score: -152.01\n",
      "Process 2, Episode 140, Total Reward: -248.59, Average Score: -147.40\n",
      "Process 6, Episode 130, Total Reward: -221.37, Average Score: -157.57\n",
      "Process 3, Episode 140, Total Reward: -171.61, Average Score: -167.01\n",
      "Process 5, Episode 150, Total Reward: -116.67, Average Score: -149.03\n",
      "Process 4, Episode 150, Total Reward: -119.50, Average Score: -153.36\n",
      "Process 1, Episode 140, Total Reward: -146.03, Average Score: -156.73\n",
      "Process 0, Episode 140, Total Reward: -158.61, Average Score: -134.97\n",
      "Process 2, Episode 150, Total Reward: -234.17, Average Score: -147.18\n",
      "Process 7, Episode 150, Total Reward: -154.84, Average Score: -147.19\n",
      "Process 6, Episode 140, Total Reward: -107.60, Average Score: -149.06\n",
      "Process 3, Episode 150, Total Reward: -77.53, Average Score: -163.76\n",
      "Process 5, Episode 160, Total Reward: -121.71, Average Score: -146.79\n",
      "Process 4, Episode 160, Total Reward: -238.81, Average Score: -144.48\n",
      "Process 1, Episode 150, Total Reward: -96.90, Average Score: -152.80\n",
      "Process 0, Episode 150, Total Reward: -125.66, Average Score: -131.67\n",
      "Process 2, Episode 160, Total Reward: -72.74, Average Score: -145.07\n",
      "Process 7, Episode 160, Total Reward: -344.75, Average Score: -150.96\n",
      "Process 6, Episode 150, Total Reward: -242.05, Average Score: -142.03\n",
      "Process 5, Episode 170, Total Reward: -84.03, Average Score: -145.61\n",
      "Process 3, Episode 160, Total Reward: -74.79, Average Score: -158.94\n",
      "Process 4, Episode 170, Total Reward: -187.55, Average Score: -140.02\n",
      "Process 1, Episode 160, Total Reward: -117.41, Average Score: -150.84\n",
      "Process 0, Episode 160, Total Reward: -356.72, Average Score: -131.68\n",
      "Process 7, Episode 170, Total Reward: -34.73, Average Score: -149.54\n",
      "Process 2, Episode 170, Total Reward: -101.07, Average Score: -138.22\n",
      "Process 6, Episode 160, Total Reward: -105.63, Average Score: -142.88\n",
      "Process 5, Episode 180, Total Reward: -72.77, Average Score: -142.26\n",
      "Process 3, Episode 170, Total Reward: -94.42, Average Score: -150.54\n",
      "Process 4, Episode 180, Total Reward: -138.37, Average Score: -136.25\n",
      "Process 1, Episode 170, Total Reward: -215.43, Average Score: -145.48\n",
      "Process 0, Episode 170, Total Reward: -139.91, Average Score: -126.39\n",
      "Process 7, Episode 180, Total Reward: 6.85, Average Score: -141.38\n",
      "Process 2, Episode 180, Total Reward: -96.12, Average Score: -136.89\n",
      "Process 3, Episode 180, Total Reward: -49.75, Average Score: -139.39\n",
      "Process 5, Episode 190, Total Reward: -246.62, Average Score: -142.02\n",
      "Process 1, Episode 180, Total Reward: -159.32, Average Score: -138.77\n",
      "Process 4, Episode 190, Total Reward: -95.02, Average Score: -131.26\n",
      "Process 0, Episode 180, Total Reward: -193.35, Average Score: -122.83\n",
      "Process 2, Episode 190, Total Reward: -83.60, Average Score: -131.09\n",
      "Process 6, Episode 170, Total Reward: -304.83, Average Score: -139.20\n",
      "Process 7, Episode 190, Total Reward: -164.87, Average Score: -133.35\n",
      "Process 5, Episode 200, Total Reward: -85.29, Average Score: -136.91\n",
      "Process 1, Episode 190, Total Reward: -94.79, Average Score: -129.18\n",
      "Process 3, Episode 190, Total Reward: -121.60, Average Score: -135.87\n",
      "Process 4, Episode 200, Total Reward: -88.38, Average Score: -126.11\n",
      "Process 0, Episode 190, Total Reward: -145.01, Average Score: -125.58\n",
      "Process 6, Episode 180, Total Reward: -109.35, Average Score: -132.37\n",
      "Process 7, Episode 200, Total Reward: -98.76, Average Score: -128.03\n",
      "Process 2, Episode 200, Total Reward: -141.75, Average Score: -128.94\n",
      "Process 1, Episode 200, Total Reward: -127.91, Average Score: -127.54\n",
      "Process 3, Episode 200, Total Reward: -113.93, Average Score: -130.39\n",
      "Process 4, Episode 210, Total Reward: -118.66, Average Score: -127.86\n",
      "Process 5, Episode 210, Total Reward: -165.45, Average Score: -132.64\n",
      "Process 0, Episode 200, Total Reward: -95.23, Average Score: -123.36\n",
      "Process 6, Episode 190, Total Reward: -101.56, Average Score: -125.12\n",
      "Process 2, Episode 210, Total Reward: -220.42, Average Score: -125.57\n",
      "Process 7, Episode 210, Total Reward: -134.60, Average Score: -127.43\n",
      "Process 1, Episode 210, Total Reward: -165.56, Average Score: -125.86\n",
      "Process 4, Episode 220, Total Reward: -97.96, Average Score: -123.10\n",
      "Process 3, Episode 210, Total Reward: -75.07, Average Score: -128.84\n",
      "Process 0, Episode 210, Total Reward: -109.45, Average Score: -122.66\n",
      "Process 5, Episode 220, Total Reward: -577.39, Average Score: -132.26\n",
      "Process 6, Episode 200, Total Reward: -66.13, Average Score: -130.80\n",
      "Process 2, Episode 220, Total Reward: -50.70, Average Score: -128.55\n",
      "Process 7, Episode 220, Total Reward: -19.29, Average Score: -120.44\n",
      "Process 1, Episode 220, Total Reward: -104.17, Average Score: -119.80\n",
      "Process 3, Episode 220, Total Reward: -98.58, Average Score: -120.80\n",
      "Process 4, Episode 230, Total Reward: -124.11, Average Score: -122.50\n",
      "Process 0, Episode 220, Total Reward: -43.44, Average Score: -122.60\n",
      "Process 5, Episode 230, Total Reward: -103.01, Average Score: -124.65\n",
      "Process 6, Episode 210, Total Reward: -27.49, Average Score: -130.18\n",
      "Process 2, Episode 230, Total Reward: -173.98, Average Score: -124.98\n",
      "Process 1, Episode 230, Total Reward: -82.00, Average Score: -116.96\n",
      "Process 0, Episode 230, Total Reward: -101.12, Average Score: -123.14\n",
      "Process 4, Episode 240, Total Reward: -93.39, Average Score: -122.16\n",
      "Process 6, Episode 220, Total Reward: -58.50, Average Score: -126.37\n",
      "Process 5, Episode 240, Total Reward: -105.47, Average Score: -120.93\n",
      "Process 7, Episode 230, Total Reward: -107.72, Average Score: -119.46\n",
      "Process 2, Episode 240, Total Reward: -338.96, Average Score: -124.36\n",
      "Process 3, Episode 230, Total Reward: -9.66, Average Score: -114.51\n",
      "Process 1, Episode 240, Total Reward: -251.82, Average Score: -114.65\n",
      "Process 0, Episode 240, Total Reward: -62.11, Average Score: -123.36\n",
      "Process 4, Episode 250, Total Reward: -278.67, Average Score: -120.24\n",
      "Process 6, Episode 230, Total Reward: -93.88, Average Score: -124.96\n",
      "Process 7, Episode 240, Total Reward: -62.69, Average Score: -115.37\n",
      "Process 5, Episode 250, Total Reward: -25.46, Average Score: -117.46\n",
      "Process 2, Episode 250, Total Reward: 26.49, Average Score: -117.61\n",
      "Process 1, Episode 250, Total Reward: -37.36, Average Score: -110.46\n",
      "Process 0, Episode 250, Total Reward: -69.33, Average Score: -123.66\n",
      "Process 6, Episode 240, Total Reward: -27.78, Average Score: -126.42\n",
      "Process 4, Episode 260, Total Reward: -76.44, Average Score: -115.60\n",
      "Process 7, Episode 250, Total Reward: -125.62, Average Score: -115.82\n",
      "Process 5, Episode 260, Total Reward: -71.76, Average Score: -115.59\n",
      "Process 3, Episode 240, Total Reward: -181.75, Average Score: -108.91\n",
      "Process 2, Episode 260, Total Reward: -158.64, Average Score: -115.92\n",
      "Process 1, Episode 260, Total Reward: -108.40, Average Score: -105.96\n",
      "Process 0, Episode 260, Total Reward: -127.69, Average Score: -117.95\n",
      "Process 4, Episode 270, Total Reward: -29.91, Average Score: -112.02\n",
      "Process 6, Episode 250, Total Reward: -177.71, Average Score: -123.83\n",
      "Process 7, Episode 260, Total Reward: -142.62, Average Score: -109.52\n",
      "Process 5, Episode 270, Total Reward: -59.29, Average Score: -111.94\n",
      "Process 3, Episode 250, Total Reward: -59.17, Average Score: -103.06\n",
      "Process 2, Episode 270, Total Reward: -110.48, Average Score: -116.88\n",
      "Process 0, Episode 270, Total Reward: -1.33, Average Score: -115.05\n",
      "Process 1, Episode 270, Total Reward: -209.01, Average Score: -103.61\n",
      "Process 4, Episode 280, Total Reward: -87.46, Average Score: -111.34\n",
      "Process 5, Episode 280, Total Reward: -150.30, Average Score: -112.31\n",
      "Process 7, Episode 270, Total Reward: -110.75, Average Score: -107.70\n",
      "Process 3, Episode 260, Total Reward: -80.18, Average Score: -102.32\n",
      "Process 2, Episode 280, Total Reward: -118.41, Average Score: -110.94\n",
      "Process 0, Episode 280, Total Reward: -63.80, Average Score: -113.99\n",
      "Process 1, Episode 280, Total Reward: -148.92, Average Score: -107.34\n",
      "Process 6, Episode 260, Total Reward: -99.43, Average Score: -113.60\n",
      "Process 4, Episode 290, Total Reward: -131.67, Average Score: -111.60\n",
      "Process 7, Episode 280, Total Reward: -128.58, Average Score: -108.24\n",
      "Process 5, Episode 290, Total Reward: -130.73, Average Score: -108.15\n",
      "Process 3, Episode 270, Total Reward: -207.77, Average Score: -105.45\n",
      "Process 2, Episode 290, Total Reward: -118.65, Average Score: -114.16\n",
      "Process 0, Episode 290, Total Reward: -97.80, Average Score: -110.55\n",
      "Process 6, Episode 270, Total Reward: -83.32, Average Score: -115.93\n",
      "Process 1, Episode 290, Total Reward: -253.88, Average Score: -111.49\n",
      "Process 4, Episode 300, Total Reward: -368.45, Average Score: -115.03\n",
      "Process 5, Episode 300, Total Reward: -105.00, Average Score: -112.27\n",
      "Process 7, Episode 290, Total Reward: -83.29, Average Score: -111.23\n",
      "Process 3, Episode 280, Total Reward: -153.43, Average Score: -110.89\n",
      "Process 2, Episode 300, Total Reward: -170.29, Average Score: -115.28\n",
      "Process 0, Episode 300, Total Reward: -175.97, Average Score: -111.85\n",
      "Process 1, Episode 300, Total Reward: -91.88, Average Score: -113.00\n",
      "Process 4, Episode 310, Total Reward: -90.49, Average Score: -116.71\n",
      "Process 7, Episode 300, Total Reward: -93.26, Average Score: -113.91\n",
      "Process 5, Episode 310, Total Reward: -4.59, Average Score: -108.89\n",
      "Process 3, Episode 290, Total Reward: -57.09, Average Score: -107.92\n",
      "Process 2, Episode 310, Total Reward: 8.45, Average Score: -111.03\n",
      "Process 1, Episode 310, Total Reward: -265.18, Average Score: -112.08\n",
      "Process 6, Episode 280, Total Reward: 76.02, Average Score: -115.86\n",
      "Process 4, Episode 320, Total Reward: -128.79, Average Score: -119.49\n",
      "Process 7, Episode 310, Total Reward: -82.39, Average Score: -113.20\n",
      "Process 5, Episode 320, Total Reward: -103.97, Average Score: -109.60\n",
      "Process 3, Episode 300, Total Reward: -130.58, Average Score: -111.25\n",
      "Process 0, Episode 310, Total Reward: -61.27, Average Score: -107.64\n",
      "Process 1, Episode 320, Total Reward: -136.26, Average Score: -111.71\n",
      "Process 6, Episode 290, Total Reward: -97.36, Average Score: -119.89\n",
      "Process 2, Episode 320, Total Reward: -93.10, Average Score: -109.53\n",
      "Process 4, Episode 330, Total Reward: -313.86, Average Score: -115.89\n",
      "Process 7, Episode 320, Total Reward: -409.06, Average Score: -117.65\n",
      "Process 5, Episode 330, Total Reward: -53.03, Average Score: -108.51\n",
      "Process 3, Episode 310, Total Reward: -100.51, Average Score: -112.25\n",
      "Process 0, Episode 320, Total Reward: -80.51, Average Score: -106.13\n",
      "Process 6, Episode 300, Total Reward: -66.63, Average Score: -111.29\n",
      "Process 4, Episode 340, Total Reward: -98.50, Average Score: -113.30\n",
      "Process 1, Episode 330, Total Reward: -143.84, Average Score: -113.14\n",
      "Process 7, Episode 330, Total Reward: -120.04, Average Score: -122.60\n",
      "Process 5, Episode 340, Total Reward: -110.28, Average Score: -106.74\n",
      "Process 3, Episode 320, Total Reward: -164.91, Average Score: -110.10\n",
      "Process 2, Episode 330, Total Reward: -65.79, Average Score: -105.01\n",
      "Process 6, Episode 310, Total Reward: -48.41, Average Score: -106.68\n",
      "Process 0, Episode 330, Total Reward: -80.28, Average Score: -105.44\n",
      "Process 4, Episode 350, Total Reward: -87.69, Average Score: -107.56\n",
      "Process 7, Episode 340, Total Reward: -83.36, Average Score: -120.66\n",
      "Process 1, Episode 340, Total Reward: -46.17, Average Score: -108.90\n",
      "Process 5, Episode 350, Total Reward: -89.12, Average Score: -105.55\n",
      "Process 3, Episode 330, Total Reward: -49.58, Average Score: -112.14\n",
      "Process 6, Episode 320, Total Reward: -36.23, Average Score: -105.15\n",
      "Process 2, Episode 340, Total Reward: -34.90, Average Score: -100.48\n",
      "Process 0, Episode 340, Total Reward: -87.88, Average Score: -102.15\n",
      "Process 1, Episode 350, Total Reward: -69.65, Average Score: -106.04\n",
      "Process 7, Episode 350, Total Reward: -103.89, Average Score: -118.77\n",
      "Process 4, Episode 360, Total Reward: -60.48, Average Score: -109.05\n",
      "Process 5, Episode 360, Total Reward: -102.03, Average Score: -103.99\n",
      "Process 3, Episode 340, Total Reward: -100.04, Average Score: -110.07\n",
      "Process 6, Episode 330, Total Reward: -29.07, Average Score: -101.53\n",
      "Process 2, Episode 350, Total Reward: -33.95, Average Score: -97.38\n",
      "Process 0, Episode 350, Total Reward: -107.25, Average Score: -101.85\n",
      "Process 7, Episode 360, Total Reward: 7.77, Average Score: -115.57\n",
      "Process 1, Episode 360, Total Reward: -223.27, Average Score: -106.22\n",
      "Process 4, Episode 370, Total Reward: -127.94, Average Score: -113.83\n",
      "Process 5, Episode 370, Total Reward: -151.52, Average Score: -106.06\n",
      "Process 3, Episode 350, Total Reward: -93.61, Average Score: -108.33\n",
      "Process 2, Episode 360, Total Reward: -80.19, Average Score: -95.11\n",
      "Process 0, Episode 360, Total Reward: -24.61, Average Score: -99.80\n",
      "Process 7, Episode 370, Total Reward: -47.74, Average Score: -111.32\n",
      "Process 1, Episode 370, Total Reward: -72.89, Average Score: -105.08\n",
      "Process 4, Episode 380, Total Reward: -62.72, Average Score: -113.67\n",
      "Process 6, Episode 340, Total Reward: -112.33, Average Score: -99.60\n",
      "Process 3, Episode 360, Total Reward: -89.26, Average Score: -107.55\n",
      "Process 5, Episode 380, Total Reward: -32.05, Average Score: -103.15\n",
      "Process 0, Episode 370, Total Reward: -65.22, Average Score: -98.33\n",
      "Process 2, Episode 370, Total Reward: -25.86, Average Score: -89.93\n",
      "Process 7, Episode 380, Total Reward: -99.94, Average Score: -109.85\n",
      "Process 4, Episode 390, Total Reward: -212.19, Average Score: -114.87\n",
      "Process 1, Episode 380, Total Reward: -4.14, Average Score: -100.62\n",
      "Process 6, Episode 350, Total Reward: -52.18, Average Score: -99.08\n",
      "Process 3, Episode 370, Total Reward: -409.66, Average Score: -103.76\n",
      "Process 5, Episode 390, Total Reward: -32.25, Average Score: -100.36\n",
      "Process 2, Episode 380, Total Reward: -114.93, Average Score: -90.67\n",
      "Process 0, Episode 380, Total Reward: -47.52, Average Score: -100.38\n",
      "Process 7, Episode 390, Total Reward: -194.92, Average Score: -108.15\n",
      "Process 4, Episode 400, Total Reward: -48.23, Average Score: -109.97\n",
      "Process 1, Episode 390, Total Reward: -86.52, Average Score: -95.72\n",
      "Process 3, Episode 380, Total Reward: -32.29, Average Score: -94.31\n",
      "Process 6, Episode 360, Total Reward: -81.28, Average Score: -103.99\n",
      "Process 2, Episode 390, Total Reward: -6.21, Average Score: -84.79\n",
      "Process 0, Episode 390, Total Reward: -57.80, Average Score: -94.87\n",
      "Process 5, Episode 400, Total Reward: -109.97, Average Score: -96.34\n",
      "Process 7, Episode 400, Total Reward: -50.16, Average Score: -100.14\n",
      "Process 4, Episode 410, Total Reward: -98.60, Average Score: -100.97\n",
      "Process 1, Episode 400, Total Reward: -45.45, Average Score: -95.04\n",
      "Process 6, Episode 370, Total Reward: -34.06, Average Score: -97.01\n",
      "Process 3, Episode 390, Total Reward: -93.12, Average Score: -94.71\n",
      "Process 2, Episode 400, Total Reward: -43.55, Average Score: -83.86\n",
      "Process 5, Episode 410, Total Reward: -131.97, Average Score: -94.73\n",
      "Process 7, Episode 410, Total Reward: -105.11, Average Score: -94.85\n",
      "Process 6, Episode 380, Total Reward: -62.62, Average Score: -95.37\n",
      "Process 1, Episode 410, Total Reward: -38.44, Average Score: -88.85\n",
      "Process 4, Episode 420, Total Reward: -39.64, Average Score: -95.99\n",
      "Process 0, Episode 400, Total Reward: -43.66, Average Score: -89.07\n",
      "Process 3, Episode 400, Total Reward: -0.86, Average Score: -87.23\n",
      "Process 2, Episode 410, Total Reward: -91.35, Average Score: -85.85\n",
      "Process 5, Episode 420, Total Reward: -73.93, Average Score: -86.35\n",
      "Process 7, Episode 420, Total Reward: -64.72, Average Score: -88.50\n",
      "Process 0, Episode 410, Total Reward: -28.08, Average Score: -85.10\n",
      "Process 6, Episode 390, Total Reward: -68.52, Average Score: -89.91\n",
      "Process 1, Episode 420, Total Reward: -84.75, Average Score: -85.54\n",
      "Process 4, Episode 430, Total Reward: -69.03, Average Score: -91.26\n",
      "Process 5, Episode 430, Total Reward: -43.97, Average Score: -84.44\n",
      "Process 2, Episode 420, Total Reward: -103.89, Average Score: -82.05\n",
      "Process 3, Episode 410, Total Reward: -46.42, Average Score: -79.10\n",
      "Process 6, Episode 400, Total Reward: -61.22, Average Score: -88.58\n",
      "Process 0, Episode 420, Total Reward: -20.72, Average Score: -85.32\n",
      "Process 1, Episode 430, Total Reward: -43.68, Average Score: -81.34\n",
      "Process 4, Episode 440, Total Reward: -95.38, Average Score: -89.53\n",
      "Process 2, Episode 430, Total Reward: -64.18, Average Score: -81.62\n",
      "Process 3, Episode 420, Total Reward: -55.28, Average Score: -77.89\n",
      "Process 7, Episode 430, Total Reward: -102.36, Average Score: -83.94\n",
      "Process 5, Episode 440, Total Reward: 18.98, Average Score: -82.71\n",
      "Process 6, Episode 410, Total Reward: -63.34, Average Score: -89.29\n",
      "Process 0, Episode 430, Total Reward: -92.47, Average Score: -83.12\n",
      "Process 4, Episode 450, Total Reward: -68.68, Average Score: -90.67\n",
      "Process 1, Episode 440, Total Reward: -130.75, Average Score: -84.93\n",
      "Process 3, Episode 430, Total Reward: -85.82, Average Score: -75.33\n",
      "Process 7, Episode 440, Total Reward: -6.21, Average Score: -84.10\n",
      "Process 2, Episode 440, Total Reward: -83.40, Average Score: -80.02\n",
      "Process 5, Episode 450, Total Reward: -84.47, Average Score: -84.65\n",
      "Process 0, Episode 440, Total Reward: -37.50, Average Score: -79.57\n",
      "Process 4, Episode 460, Total Reward: -66.79, Average Score: -89.68\n",
      "Process 1, Episode 450, Total Reward: -22.29, Average Score: -85.22\n",
      "Process 3, Episode 440, Total Reward: -13.90, Average Score: -75.91\n",
      "Process 6, Episode 420, Total Reward: -9.76, Average Score: -85.29\n",
      "Process 7, Episode 450, Total Reward: -59.90, Average Score: -80.76\n",
      "Process 2, Episode 450, Total Reward: -55.82, Average Score: -79.30\n",
      "Process 0, Episode 450, Total Reward: -73.13, Average Score: -77.23\n",
      "Process 5, Episode 460, Total Reward: -47.20, Average Score: -79.99\n",
      "Process 1, Episode 460, Total Reward: -113.96, Average Score: -82.68\n",
      "Process 4, Episode 470, Total Reward: -30.17, Average Score: -82.10\n",
      "Process 6, Episode 430, Total Reward: -63.88, Average Score: -82.75\n",
      "Process 7, Episode 460, Total Reward: -144.83, Average Score: -77.90\n",
      "Process 3, Episode 450, Total Reward: -52.69, Average Score: -75.19\n",
      "Process 0, Episode 460, Total Reward: -55.19, Average Score: -73.93\n",
      "Process 2, Episode 460, Total Reward: -54.06, Average Score: -77.14\n",
      "Process 5, Episode 470, Total Reward: -62.50, Average Score: -76.57\n",
      "Process 4, Episode 480, Total Reward: -83.99, Average Score: -78.43\n",
      "Process 1, Episode 470, Total Reward: -13.46, Average Score: -80.85\n",
      "Process 6, Episode 440, Total Reward: -123.89, Average Score: -80.87\n",
      "Process 7, Episode 470, Total Reward: -91.44, Average Score: -77.87\n",
      "Process 3, Episode 460, Total Reward: -60.39, Average Score: -74.93\n",
      "Process 0, Episode 470, Total Reward: -31.82, Average Score: -72.54\n",
      "Process 2, Episode 470, Total Reward: -68.26, Average Score: -78.38\n",
      "Process 5, Episode 480, Total Reward: -48.44, Average Score: -77.44\n",
      "Process 1, Episode 480, Total Reward: -75.96, Average Score: -76.47\n",
      "Process 6, Episode 450, Total Reward: -78.29, Average Score: -75.73\n",
      "Process 7, Episode 480, Total Reward: -67.25, Average Score: -72.77\n",
      "Process 3, Episode 470, Total Reward: -14.21, Average Score: -70.61\n",
      "Process 2, Episode 480, Total Reward: -18.07, Average Score: -76.72\n",
      "Process 0, Episode 480, Total Reward: 3.88, Average Score: -65.62\n",
      "Process 4, Episode 490, Total Reward: -18.68, Average Score: -71.70\n",
      "Process 5, Episode 490, Total Reward: -41.34, Average Score: -74.15\n",
      "Process 6, Episode 460, Total Reward: -135.36, Average Score: -72.06\n",
      "Process 1, Episode 490, Total Reward: -34.77, Average Score: -74.94\n",
      "Process 7, Episode 490, Total Reward: -37.76, Average Score: -67.91\n",
      "Process 3, Episode 480, Total Reward: -58.14, Average Score: -70.94\n",
      "Process 0, Episode 490, Total Reward: -43.71, Average Score: -67.81\n",
      "Process 2, Episode 490, Total Reward: -138.19, Average Score: -78.21\n",
      "Process 4, Episode 500, Total Reward: -81.27, Average Score: -71.39\n",
      "Process 5, Episode 500, Total Reward: -86.96, Average Score: -69.16\n",
      "Process 6, Episode 470, Total Reward: -112.11, Average Score: -71.62\n",
      "Process 1, Episode 500, Total Reward: -46.86, Average Score: -71.59\n",
      "Process 7, Episode 500, Total Reward: -66.27, Average Score: -69.57\n",
      "Process 3, Episode 490, Total Reward: -90.25, Average Score: -71.25\n",
      "Process 2, Episode 500, Total Reward: -57.56, Average Score: -71.94\n",
      "Process 0, Episode 500, Total Reward: -44.48, Average Score: -66.07\n",
      "Process 4, Episode 510, Total Reward: -8.77, Average Score: -68.62\n",
      "Process 6, Episode 480, Total Reward: -10.21, Average Score: -69.28\n",
      "Process 1, Episode 510, Total Reward: -63.62, Average Score: -71.29\n",
      "Process 3, Episode 500, Total Reward: -64.34, Average Score: -73.55\n",
      "Process 7, Episode 510, Total Reward: -40.66, Average Score: -67.70\n",
      "Process 5, Episode 510, Total Reward: 0.36, Average Score: -68.85\n",
      "Process 2, Episode 510, Total Reward: -29.95, Average Score: -67.31\n",
      "Process 1, Episode 520, Total Reward: -32.69, Average Score: -67.63\n",
      "Process 0, Episode 510, Total Reward: -80.68, Average Score: -64.10\n",
      "Process 4, Episode 520, Total Reward: -7.00, Average Score: -65.65\n",
      "Process 6, Episode 490, Total Reward: -54.80, Average Score: -66.35\n",
      "Process 7, Episode 520, Total Reward: -85.11, Average Score: -64.00\n",
      "Process 5, Episode 520, Total Reward: -23.04, Average Score: -64.17\n",
      "Process 3, Episode 510, Total Reward: -45.27, Average Score: -72.81\n",
      "Process 1, Episode 530, Total Reward: -49.96, Average Score: -64.02\n",
      "Process 4, Episode 530, Total Reward: 3.88, Average Score: -60.63\n",
      "Process 7, Episode 530, Total Reward: -37.26, Average Score: -62.55\n",
      "Process 6, Episode 500, Total Reward: -45.35, Average Score: -62.09\n",
      "Process 3, Episode 520, Total Reward: -55.90, Average Score: -71.05\n",
      "Process 0, Episode 520, Total Reward: -24.96, Average Score: -57.75\n",
      "Process 5, Episode 530, Total Reward: -8.59, Average Score: -62.10\n",
      "Process 4, Episode 540, Total Reward: -55.82, Average Score: -57.35\n",
      "Process 2, Episode 520, Total Reward: -48.81, Average Score: -66.18\n",
      "Process 7, Episode 540, Total Reward: -73.24, Average Score: -61.03\n",
      "Process 5, Episode 540, Total Reward: -66.64, Average Score: -58.69\n",
      "Process 0, Episode 530, Total Reward: -18.98, Average Score: -54.23\n",
      "Process 3, Episode 530, Total Reward: -55.63, Average Score: -68.82\n",
      "Process 1, Episode 540, Total Reward: -46.94, Average Score: -53.23\n",
      "Process 4, Episode 550, Total Reward: -18.80, Average Score: -51.37\n",
      "Process 6, Episode 510, Total Reward: -73.44, Average Score: -57.99\n",
      "Process 0, Episode 540, Total Reward: -65.98, Average Score: -55.87\n",
      "Process 2, Episode 530, Total Reward: 2.65, Average Score: -64.25\n",
      "Process 3, Episode 540, Total Reward: -39.21, Average Score: -63.84\n",
      "Process 1, Episode 550, Total Reward: -52.40, Average Score: -51.47\n",
      "Process 5, Episode 550, Total Reward: -72.69, Average Score: -51.84\n",
      "Process 2, Episode 540, Total Reward: -14.60, Average Score: -62.16\n",
      "Process 3, Episode 550, Total Reward: -228.35, Average Score: -66.96\n",
      "Process 4, Episode 560, Total Reward: -18.15, Average Score: -46.75\n",
      "Process 6, Episode 520, Total Reward: -73.71, Average Score: -56.40\n",
      "Process 7, Episode 550, Total Reward: -34.45, Average Score: -60.91\n",
      "Process 2, Episode 550, Total Reward: -14.51, Average Score: -60.24\n",
      "Process 0, Episode 550, Total Reward: 19.83, Average Score: -50.90\n",
      "Process 6, Episode 530, Total Reward: -63.37, Average Score: -54.69\n",
      "Process 1, Episode 560, Total Reward: -23.64, Average Score: -48.22\n",
      "Process 4, Episode 570, Total Reward: -306.55, Average Score: -56.61\n",
      "Process 3, Episode 560, Total Reward: 56.58, Average Score: -62.43\n",
      "Process 5, Episode 560, Total Reward: 70.14, Average Score: -49.89\n",
      "Process 0, Episode 560, Total Reward: -85.58, Average Score: -51.59\n",
      "Process 2, Episode 560, Total Reward: -86.78, Average Score: -59.68\n",
      "Process 7, Episode 560, Total Reward: -26.89, Average Score: -58.97\n",
      "Process 3, Episode 570, Total Reward: -59.16, Average Score: -59.05\n",
      "Process 5, Episode 570, Total Reward: -97.67, Average Score: -46.74\n",
      "Process 6, Episode 540, Total Reward: 0.12, Average Score: -50.27\n",
      "Process 2, Episode 570, Total Reward: -22.87, Average Score: -56.38\n",
      "Process 4, Episode 580, Total Reward: -88.55, Average Score: -54.19\n",
      "Process 1, Episode 570, Total Reward: -72.12, Average Score: -49.18\n",
      "Process 0, Episode 570, Total Reward: 110.53, Average Score: -51.59\n",
      "Process 7, Episode 570, Total Reward: 6.27, Average Score: -55.28\n",
      "Process 6, Episode 550, Total Reward: -23.39, Average Score: -49.18\n",
      "Process 3, Episode 580, Total Reward: -172.66, Average Score: -58.37\n",
      "Process 5, Episode 580, Total Reward: -15.22, Average Score: -41.11\n",
      "Process 2, Episode 580, Total Reward: -31.71, Average Score: -53.44\n",
      "Process 0, Episode 580, Total Reward: -17.89, Average Score: -51.64\n",
      "Process 1, Episode 580, Total Reward: 4.89, Average Score: -44.82\n",
      "Process 6, Episode 560, Total Reward: -51.78, Average Score: -46.83\n",
      "Process 5, Episode 590, Total Reward: -55.04, Average Score: -41.37\n",
      "Process 7, Episode 580, Total Reward: -62.36, Average Score: -54.41\n",
      "Process 4, Episode 590, Total Reward: 12.62, Average Score: -47.19\n",
      "Process 2, Episode 590, Total Reward: -79.14, Average Score: -48.00\n",
      "Process 3, Episode 590, Total Reward: 27.63, Average Score: -50.55\n",
      "Process 6, Episode 570, Total Reward: -80.15, Average Score: -46.93\n",
      "Process 5, Episode 600, Total Reward: -283.27, Average Score: -44.80\n",
      "Process 0, Episode 590, Total Reward: 7.97, Average Score: -44.58\n",
      "Process 4, Episode 600, Total Reward: -47.04, Average Score: -43.73\n",
      "Process 3, Episode 600, Total Reward: -3.72, Average Score: -48.36\n",
      "Process 6, Episode 580, Total Reward: -61.50, Average Score: -45.75\n",
      "Process 1, Episode 590, Total Reward: -15.44, Average Score: -40.34\n",
      "Process 0, Episode 600, Total Reward: -65.97, Average Score: -45.29\n",
      "Process 4, Episode 610, Total Reward: -32.55, Average Score: -46.55\n",
      "Process 6, Episode 590, Total Reward: -1.64, Average Score: -45.09\n",
      "Process 7, Episode 590, Total Reward: 64.75, Average Score: -50.74\n",
      "Process 5, Episode 610, Total Reward: -55.91, Average Score: -40.87\n",
      "Process 1, Episode 600, Total Reward: -20.98, Average Score: -36.66\n",
      "Process 2, Episode 600, Total Reward: 0.86, Average Score: -47.75\n",
      "Process 0, Episode 610, Total Reward: -31.01, Average Score: -44.73\n",
      "Process 3, Episode 610, Total Reward: 18.45, Average Score: -50.31\n",
      "Process 5, Episode 620, Total Reward: -42.59, Average Score: -42.53\n",
      "Process 1, Episode 610, Total Reward: 6.56, Average Score: -32.60\n",
      "Process 4, Episode 620, Total Reward: -29.39, Average Score: -44.48\n",
      "Process 6, Episode 600, Total Reward: -71.26, Average Score: -45.99\n",
      "Process 7, Episode 600, Total Reward: 79.07, Average Score: -45.44\n",
      "Process 3, Episode 620, Total Reward: 4.90, Average Score: -51.87\n",
      "Process 1, Episode 620, Total Reward: -12.09, Average Score: -36.35\n",
      "Process 0, Episode 620, Total Reward: -6.00, Average Score: -43.56\n",
      "Process 2, Episode 610, Total Reward: -124.12, Average Score: -43.72\n",
      "Process 7, Episode 610, Total Reward: 8.53, Average Score: -47.02\n",
      "Process 6, Episode 610, Total Reward: -26.48, Average Score: -44.26\n",
      "Process 5, Episode 630, Total Reward: -13.78, Average Score: -40.08\n",
      "Process 1, Episode 630, Total Reward: -4.21, Average Score: -33.06\n",
      "Process 7, Episode 620, Total Reward: -24.89, Average Score: -46.81\n",
      "Process 2, Episode 620, Total Reward: -68.19, Average Score: -42.66\n",
      "Process 3, Episode 630, Total Reward: -51.12, Average Score: -51.75\n",
      "Process 0, Episode 630, Total Reward: -45.25, Average Score: -39.88\n",
      "Process 4, Episode 630, Total Reward: -8.10, Average Score: -40.90\n",
      "Process 6, Episode 620, Total Reward: -136.17, Average Score: -43.10\n",
      "Process 1, Episode 640, Total Reward: -16.80, Average Score: -31.66\n",
      "Process 5, Episode 640, Total Reward: -158.43, Average Score: -40.78\n",
      "Process 7, Episode 630, Total Reward: -49.54, Average Score: -45.00\n",
      "Process 0, Episode 640, Total Reward: -205.17, Average Score: -39.08\n",
      "Process 2, Episode 630, Total Reward: -92.05, Average Score: -39.92\n",
      "Process 3, Episode 640, Total Reward: -57.35, Average Score: -52.14\n",
      "Process 1, Episode 650, Total Reward: -27.14, Average Score: -30.42\n",
      "Process 4, Episode 640, Total Reward: 109.39, Average Score: -35.55\n",
      "Process 0, Episode 650, Total Reward: -47.94, Average Score: -38.91\n",
      "Process 6, Episode 630, Total Reward: 10.90, Average Score: -39.96\n",
      "Process 7, Episode 640, Total Reward: -8.55, Average Score: -41.22\n",
      "Process 3, Episode 650, Total Reward: -26.15, Average Score: -43.46\n",
      "Process 2, Episode 640, Total Reward: -132.05, Average Score: -35.29\n",
      "Process 5, Episode 650, Total Reward: 93.27, Average Score: -33.13\n",
      "Process 1, Episode 660, Total Reward: 93.36, Average Score: -27.85\n",
      "Process 3, Episode 660, Total Reward: -6.17, Average Score: -46.35\n",
      "Process 6, Episode 640, Total Reward: -13.87, Average Score: -39.23\n",
      "Process 4, Episode 650, Total Reward: -57.85, Average Score: -34.82\n",
      "Process 7, Episode 650, Total Reward: -22.09, Average Score: -36.18\n",
      "Process 2, Episode 650, Total Reward: -6.74, Average Score: -34.35\n",
      "Process 0, Episode 660, Total Reward: 54.78, Average Score: -34.16\n",
      "Process 5, Episode 660, Total Reward: -164.73, Average Score: -35.96\n",
      "Process 4, Episode 660, Total Reward: -72.84, Average Score: -34.70\n",
      "Process 6, Episode 650, Total Reward: -34.06, Average Score: -40.19\n",
      "Process 2, Episode 660, Total Reward: 102.62, Average Score: -32.25\n",
      "Process 1, Episode 670, Total Reward: 109.33, Average Score: -21.17\n",
      "Process 0, Episode 670, Total Reward: -60.30, Average Score: -32.79\n",
      "Process 3, Episode 670, Total Reward: -44.70, Average Score: -39.39\n",
      "Process 6, Episode 660, Total Reward: 0.55, Average Score: -35.81\n",
      "Process 5, Episode 670, Total Reward: 28.70, Average Score: -30.57\n",
      "Process 7, Episode 660, Total Reward: -35.62, Average Score: -32.15\n",
      "Process 2, Episode 670, Total Reward: 5.38, Average Score: -28.42\n",
      "Process 4, Episode 670, Total Reward: 49.49, Average Score: -18.47\n",
      "Process 1, Episode 680, Total Reward: -99.32, Average Score: -23.13\n",
      "Process 5, Episode 680, Total Reward: -35.19, Average Score: -34.38\n",
      "Process 3, Episode 680, Total Reward: 47.22, Average Score: -35.70\n",
      "Process 2, Episode 680, Total Reward: -20.89, Average Score: -29.08\n",
      "Process 6, Episode 670, Total Reward: 5.38, Average Score: -29.78\n",
      "Process 7, Episode 670, Total Reward: 247.68, Average Score: -25.53\n",
      "Process 1, Episode 690, Total Reward: -26.85, Average Score: -23.40\n",
      "Process 4, Episode 680, Total Reward: 89.06, Average Score: -15.82\n",
      "Process 3, Episode 690, Total Reward: 112.15, Average Score: -39.10\n",
      "Process 5, Episode 690, Total Reward: 107.12, Average Score: -28.58\n",
      "Process 0, Episode 680, Total Reward: -43.72, Average Score: -24.93\n",
      "Process 7, Episode 680, Total Reward: 12.09, Average Score: -24.91\n",
      "Process 2, Episode 690, Total Reward: -14.55, Average Score: -25.48\n",
      "Process 5, Episode 700, Total Reward: -31.17, Average Score: -23.37\n",
      "Process 0, Episode 690, Total Reward: -20.86, Average Score: -27.77\n",
      "Process 6, Episode 680, Total Reward: 47.42, Average Score: -25.15\n",
      "Process 4, Episode 690, Total Reward: 63.12, Average Score: -16.29\n",
      "Process 1, Episode 700, Total Reward: 20.01, Average Score: -23.22\n",
      "Process 7, Episode 690, Total Reward: -47.31, Average Score: -30.44\n",
      "Process 3, Episode 700, Total Reward: -13.75, Average Score: -32.57\n",
      "Process 2, Episode 700, Total Reward: 20.71, Average Score: -24.40\n",
      "Process 5, Episode 710, Total Reward: -22.24, Average Score: -22.84\n",
      "Process 7, Episode 700, Total Reward: -23.80, Average Score: -26.73\n",
      "Process 0, Episode 700, Total Reward: -56.78, Average Score: -26.06\n",
      "Process 3, Episode 710, Total Reward: 71.52, Average Score: -30.58\n",
      "Process 2, Episode 710, Total Reward: -37.95, Average Score: -26.57\n",
      "Process 4, Episode 700, Total Reward: -17.45, Average Score: -16.50\n",
      "Process 6, Episode 690, Total Reward: -7.97, Average Score: -18.58\n",
      "Process 5, Episode 720, Total Reward: -32.70, Average Score: -22.59\n",
      "Process 0, Episode 710, Total Reward: -28.76, Average Score: -22.48\n",
      "Process 1, Episode 710, Total Reward: -48.50, Average Score: -18.96\n",
      "Process 4, Episode 710, Total Reward: -12.01, Average Score: -12.68\n",
      "Process 7, Episode 710, Total Reward: 83.34, Average Score: -19.47\n",
      "Process 3, Episode 720, Total Reward: -40.71, Average Score: -26.57\n",
      "Process 5, Episode 730, Total Reward: 19.66, Average Score: -24.49\n",
      "Process 2, Episode 720, Total Reward: -31.73, Average Score: -21.60\n",
      "Process 6, Episode 700, Total Reward: 102.22, Average Score: -13.35\n",
      "Process 3, Episode 730, Total Reward: -19.04, Average Score: -31.48\n",
      "Process 0, Episode 720, Total Reward: -63.48, Average Score: -19.89\n",
      "Process 7, Episode 720, Total Reward: -129.63, Average Score: -25.10\n",
      "Process 1, Episode 720, Total Reward: 77.18, Average Score: -12.27\n",
      "Process 6, Episode 710, Total Reward: -39.87, Average Score: -13.20\n",
      "Process 5, Episode 740, Total Reward: -67.79, Average Score: -19.39\n",
      "Process 4, Episode 720, Total Reward: 12.08, Average Score: -9.13\n",
      "Process 2, Episode 730, Total Reward: -16.42, Average Score: -22.32\n",
      "Process 3, Episode 740, Total Reward: -51.61, Average Score: -30.76\n",
      "Process 0, Episode 730, Total Reward: -4.02, Average Score: -19.15\n",
      "Process 1, Episode 730, Total Reward: 25.61, Average Score: -15.18\n",
      "Process 4, Episode 730, Total Reward: -85.53, Average Score: -10.24\n",
      "Process 7, Episode 730, Total Reward: 53.25, Average Score: -17.50\n",
      "Process 6, Episode 720, Total Reward: 99.29, Average Score: -13.66\n",
      "Process 1, Episode 740, Total Reward: 25.55, Average Score: -17.15\n",
      "Process 3, Episode 750, Total Reward: 2.65, Average Score: -29.92\n",
      "Process 2, Episode 740, Total Reward: -25.49, Average Score: -25.19\n",
      "Process 0, Episode 740, Total Reward: -2.23, Average Score: -16.80\n",
      "Process 5, Episode 750, Total Reward: 112.74, Average Score: -21.84\n",
      "Process 4, Episode 740, Total Reward: -63.75, Average Score: -9.57\n",
      "Process 3, Episode 760, Total Reward: 129.76, Average Score: -25.27\n",
      "Process 1, Episode 750, Total Reward: -66.52, Average Score: -17.42\n",
      "Process 6, Episode 730, Total Reward: -34.48, Average Score: -13.77\n",
      "Process 2, Episode 750, Total Reward: 13.03, Average Score: -20.29\n",
      "Process 0, Episode 750, Total Reward: 7.58, Average Score: -18.31\n",
      "Process 5, Episode 760, Total Reward: 136.80, Average Score: -16.16\n",
      "Process 4, Episode 750, Total Reward: -129.96, Average Score: -12.46\n",
      "Process 7, Episode 740, Total Reward: -7.59, Average Score: -9.61\n",
      "Process 2, Episode 760, Total Reward: -60.06, Average Score: -18.17\n",
      "Process 3, Episode 770, Total Reward: 20.33, Average Score: -29.16\n",
      "Process 4, Episode 760, Total Reward: -0.25, Average Score: -8.88\n",
      "Process 0, Episode 760, Total Reward: -34.66, Average Score: -16.47\n",
      "Process 7, Episode 750, Total Reward: -43.83, Average Score: -10.19\n",
      "Process 6, Episode 740, Total Reward: -3.42, Average Score: -11.45\n",
      "Process 1, Episode 760, Total Reward: 137.28, Average Score: -12.85\n",
      "Process 5, Episode 770, Total Reward: 32.38, Average Score: -14.74\n",
      "Process 0, Episode 770, Total Reward: -26.61, Average Score: -15.60\n",
      "Process 3, Episode 780, Total Reward: 9.75, Average Score: -28.39\n",
      "Process 0, Episode 780, Total Reward: -14.83, Average Score: -21.05\n",
      "Process 4, Episode 770, Total Reward: -19.43, Average Score: -8.19\n",
      "Process 6, Episode 750, Total Reward: -117.43, Average Score: -7.95\n",
      "Process 1, Episode 770, Total Reward: -39.99, Average Score: -13.73\n",
      "Process 2, Episode 770, Total Reward: 122.24, Average Score: -14.94\n",
      "Process 7, Episode 760, Total Reward: -120.12, Average Score: -9.05\n",
      "Process 5, Episode 780, Total Reward: 117.67, Average Score: -10.14\n",
      "Process 4, Episode 780, Total Reward: -35.36, Average Score: -8.96\n",
      "Process 0, Episode 790, Total Reward: -10.26, Average Score: -16.56\n",
      "Process 2, Episode 780, Total Reward: 66.98, Average Score: -11.49\n",
      "Process 1, Episode 780, Total Reward: 50.30, Average Score: -12.65\n",
      "Process 6, Episode 760, Total Reward: 110.59, Average Score: -6.74\n",
      "Process 7, Episode 770, Total Reward: 1.61, Average Score: -13.62\n",
      "Process 3, Episode 790, Total Reward: 115.90, Average Score: -19.99\n",
      "Process 2, Episode 790, Total Reward: -20.98, Average Score: -13.14\n",
      "Process 6, Episode 770, Total Reward: 8.43, Average Score: -6.83\n",
      "Process 5, Episode 790, Total Reward: 114.85, Average Score: -7.15\n",
      "Process 4, Episode 790, Total Reward: 22.97, Average Score: -8.07\n",
      "Process 0, Episode 800, Total Reward: -54.85, Average Score: -11.32\n",
      "Process 7, Episode 780, Total Reward: -70.71, Average Score: -9.71\n",
      "Process 5, Episode 800, Total Reward: 97.40, Average Score: -7.05\n",
      "Process 1, Episode 790, Total Reward: 118.97, Average Score: -9.28\n",
      "Process 6, Episode 780, Total Reward: -114.20, Average Score: -9.79\n",
      "Process 3, Episode 800, Total Reward: 68.79, Average Score: -18.12\n",
      "Process 2, Episode 800, Total Reward: 86.04, Average Score: -7.06\n",
      "Process 7, Episode 790, Total Reward: -60.68, Average Score: -7.27\n",
      "Process 0, Episode 810, Total Reward: 38.15, Average Score: -9.75\n",
      "Process 4, Episode 800, Total Reward: -87.45, Average Score: -1.59\n",
      "Process 1, Episode 800, Total Reward: -52.86, Average Score: -7.07\n",
      "Process 5, Episode 810, Total Reward: 16.75, Average Score: -6.89\n",
      "Process 6, Episode 790, Total Reward: 10.21, Average Score: -10.64\n",
      "Process 1, Episode 810, Total Reward: 106.81, Average Score: -8.88\n",
      "Process 3, Episode 810, Total Reward: 159.67, Average Score: -12.99\n",
      "Process 0, Episode 820, Total Reward: -55.43, Average Score: -11.12\n",
      "Process 7, Episode 800, Total Reward: -34.70, Average Score: -4.92\n",
      "Process 4, Episode 810, Total Reward: 104.70, Average Score: 4.79\n",
      "Process 2, Episode 810, Total Reward: -6.90, Average Score: 1.29\n",
      "Process 5, Episode 820, Total Reward: -15.88, Average Score: 2.47\n",
      "Process 6, Episode 800, Total Reward: 6.14, Average Score: -7.75\n",
      "Process 7, Episode 810, Total Reward: 116.10, Average Score: -4.17\n",
      "Process 0, Episode 830, Total Reward: 87.67, Average Score: -9.10\n",
      "Process 1, Episode 820, Total Reward: 142.66, Average Score: -4.37\n",
      "Process 2, Episode 820, Total Reward: -3.69, Average Score: 5.02\n",
      "Process 3, Episode 820, Total Reward: 101.73, Average Score: -4.35\n",
      "Process 5, Episode 830, Total Reward: 169.30, Average Score: 10.10\n",
      "Process 6, Episode 810, Total Reward: -56.11, Average Score: -6.95\n",
      "Process 7, Episode 820, Total Reward: -22.16, Average Score: 1.09\n",
      "Process 4, Episode 820, Total Reward: 49.98, Average Score: 9.58\n",
      "Process 3, Episode 830, Total Reward: 128.00, Average Score: 3.09\n",
      "Process 1, Episode 830, Total Reward: -68.36, Average Score: 3.50\n",
      "Process 2, Episode 830, Total Reward: 57.29, Average Score: 9.49\n",
      "Process 5, Episode 840, Total Reward: 100.60, Average Score: 14.12\n",
      "Process 6, Episode 820, Total Reward: -10.35, Average Score: -4.40\n",
      "Process 0, Episode 840, Total Reward: 106.88, Average Score: 1.52\n",
      "Process 7, Episode 830, Total Reward: 72.69, Average Score: 3.13\n",
      "Process 4, Episode 830, Total Reward: 30.86, Average Score: 13.66\n",
      "Process 2, Episode 840, Total Reward: 116.83, Average Score: 12.13\n",
      "Process 3, Episode 840, Total Reward: 110.87, Average Score: 7.09\n",
      "Process 1, Episode 840, Total Reward: 6.76, Average Score: 13.37\n",
      "Process 0, Episode 850, Total Reward: 102.20, Average Score: 9.11\n",
      "Process 5, Episode 850, Total Reward: -161.81, Average Score: 16.65\n",
      "Process 2, Episode 850, Total Reward: -208.64, Average Score: 3.80\n",
      "Process 4, Episode 840, Total Reward: -35.22, Average Score: 14.52\n",
      "Process 6, Episode 830, Total Reward: 90.02, Average Score: 3.98\n",
      "Process 3, Episode 850, Total Reward: 125.44, Average Score: 11.88\n",
      "Process 7, Episode 840, Total Reward: 42.18, Average Score: 2.12\n",
      "Process 0, Episode 860, Total Reward: 2.05, Average Score: 6.63\n",
      "Process 2, Episode 860, Total Reward: -245.16, Average Score: 6.59\n",
      "Process 1, Episode 850, Total Reward: -39.32, Average Score: 18.11\n",
      "Process 6, Episode 840, Total Reward: -21.03, Average Score: 5.97\n",
      "Process 4, Episode 850, Total Reward: 51.63, Average Score: 21.69\n",
      "Process 5, Episode 860, Total Reward: 125.38, Average Score: 15.97\n",
      "Process 0, Episode 870, Total Reward: 75.21, Average Score: -1.80\n",
      "Process 7, Episode 850, Total Reward: -21.66, Average Score: 6.93\n",
      "Process 3, Episode 860, Total Reward: 72.69, Average Score: 18.76\n",
      "Process 6, Episode 850, Total Reward: 23.12, Average Score: 6.64\n",
      "Process 1, Episode 860, Total Reward: 40.07, Average Score: 18.28\n",
      "Process 2, Episode 870, Total Reward: -13.91, Average Score: 11.19\n",
      "Process 5, Episode 870, Total Reward: 9.02, Average Score: 19.60\n",
      "Process 4, Episode 860, Total Reward: 49.27, Average Score: 29.62\n",
      "Process 7, Episode 860, Total Reward: -317.06, Average Score: 7.71\n",
      "Process 0, Episode 880, Total Reward: 67.81, Average Score: 1.81\n",
      "Process 1, Episode 870, Total Reward: -43.39, Average Score: 23.09\n",
      "Process 6, Episode 860, Total Reward: 33.37, Average Score: 10.34\n",
      "Process 3, Episode 870, Total Reward: 106.67, Average Score: 26.97\n",
      "Process 2, Episode 880, Total Reward: 54.45, Average Score: 17.30\n",
      "Process 5, Episode 880, Total Reward: 23.29, Average Score: 23.47\n",
      "Process 4, Episode 870, Total Reward: -153.17, Average Score: 33.97\n",
      "Process 1, Episode 880, Total Reward: 97.84, Average Score: 31.23\n",
      "Process 0, Episode 890, Total Reward: -36.86, Average Score: 5.67\n",
      "Process 6, Episode 870, Total Reward: -37.99, Average Score: 16.52\n",
      "Process 7, Episode 870, Total Reward: 148.58, Average Score: 14.62\n",
      "Process 3, Episode 880, Total Reward: 133.20, Average Score: 33.05\n",
      "Process 5, Episode 890, Total Reward: -14.57, Average Score: 26.08\n",
      "Process 2, Episode 890, Total Reward: 132.12, Average Score: 22.31\n",
      "Process 4, Episode 880, Total Reward: 97.62, Average Score: 35.27\n",
      "Process 7, Episode 880, Total Reward: -16.33, Average Score: 15.70\n",
      "Process 1, Episode 890, Total Reward: 158.80, Average Score: 36.68\n",
      "Process 6, Episode 880, Total Reward: 101.40, Average Score: 26.09\n",
      "Process 0, Episode 900, Total Reward: 136.18, Average Score: 11.61\n",
      "Process 2, Episode 900, Total Reward: 10.07, Average Score: 22.51\n",
      "Process 3, Episode 890, Total Reward: 96.99, Average Score: 35.74\n",
      "Process 4, Episode 890, Total Reward: 96.29, Average Score: 37.34\n",
      "Process 5, Episode 900, Total Reward: 109.03, Average Score: 38.35\n",
      "Process 7, Episode 890, Total Reward: -9.40, Average Score: 23.94\n",
      "Process 1, Episode 900, Total Reward: -145.29, Average Score: 40.56\n",
      "Process 6, Episode 890, Total Reward: 145.72, Average Score: 34.50\n",
      "Process 4, Episode 900, Total Reward: 168.04, Average Score: 36.80\n",
      "Process 0, Episode 910, Total Reward: 163.96, Average Score: 17.85\n",
      "Process 2, Episode 910, Total Reward: -44.11, Average Score: 21.31\n",
      "Process 7, Episode 900, Total Reward: 18.71, Average Score: 27.62\n",
      "Process 3, Episode 900, Total Reward: 163.46, Average Score: 38.39\n",
      "Process 1, Episode 910, Total Reward: 128.28, Average Score: 42.18\n",
      "Process 5, Episode 910, Total Reward: 111.73, Average Score: 46.63\n",
      "Process 6, Episode 900, Total Reward: 66.70, Average Score: 35.68\n",
      "Process 2, Episode 920, Total Reward: 1.20, Average Score: 21.42\n",
      "Process 7, Episode 910, Total Reward: 131.34, Average Score: 28.89\n",
      "Process 4, Episode 910, Total Reward: 48.51, Average Score: 36.99\n",
      "Process 0, Episode 920, Total Reward: 102.54, Average Score: 26.61\n",
      "Process 3, Episode 910, Total Reward: 87.74, Average Score: 41.79\n",
      "Process 5, Episode 920, Total Reward: 22.82, Average Score: 48.91\n",
      "Process 6, Episode 910, Total Reward: 27.33, Average Score: 38.97\n",
      "Process 2, Episode 930, Total Reward: 15.68, Average Score: 23.37\n",
      "Process 4, Episode 920, Total Reward: 20.46, Average Score: 31.33\n",
      "Process 1, Episode 920, Total Reward: -52.84, Average Score: 46.12\n",
      "Process 7, Episode 920, Total Reward: 210.91, Average Score: 39.73\n",
      "Process 0, Episode 930, Total Reward: 147.44, Average Score: 30.75\n",
      "Process 6, Episode 920, Total Reward: -2.16, Average Score: 45.53\n",
      "Process 2, Episode 940, Total Reward: 32.87, Average Score: 27.62\n",
      "Process 3, Episode 920, Total Reward: 90.94, Average Score: 39.19\n",
      "Process 5, Episode 930, Total Reward: -46.18, Average Score: 52.04\n",
      "Process 4, Episode 930, Total Reward: 103.19, Average Score: 34.77\n",
      "Process 1, Episode 930, Total Reward: 2.86, Average Score: 48.69\n",
      "Process 7, Episode 930, Total Reward: 46.83, Average Score: 41.21\n",
      "Process 6, Episode 930, Total Reward: 22.48, Average Score: 39.07\n",
      "Process 0, Episode 940, Total Reward: 21.88, Average Score: 29.32\n",
      "Process 2, Episode 950, Total Reward: -57.62, Average Score: 38.84\n",
      "Process 5, Episode 940, Total Reward: -65.44, Average Score: 51.16\n",
      "Process 4, Episode 940, Total Reward: 226.45, Average Score: 36.25\n",
      "Process 6, Episode 940, Total Reward: 38.98, Average Score: 39.87\n",
      "Process 1, Episode 940, Total Reward: 129.81, Average Score: 44.29\n",
      "Process 3, Episode 930, Total Reward: -28.58, Average Score: 45.76\n",
      "Process 7, Episode 940, Total Reward: 101.43, Average Score: 44.02\n",
      "Process 0, Episode 950, Total Reward: 6.40, Average Score: 32.89\n",
      "Process 2, Episode 960, Total Reward: -29.80, Average Score: 40.63\n",
      "Process 1, Episode 950, Total Reward: 4.59, Average Score: 48.18\n",
      "Process 6, Episode 950, Total Reward: 74.61, Average Score: 48.30\n",
      "Process 4, Episode 950, Total Reward: 100.72, Average Score: 40.74\n",
      "Process 3, Episode 940, Total Reward: -179.64, Average Score: 45.34\n",
      "Process 5, Episode 950, Total Reward: 67.90, Average Score: 55.74\n",
      "Process 7, Episode 950, Total Reward: 164.01, Average Score: 46.80\n",
      "Process 2, Episode 970, Total Reward: 19.99, Average Score: 37.81\n",
      "Process 0, Episode 960, Total Reward: 118.39, Average Score: 39.83\n",
      "Process 3, Episode 950, Total Reward: -7.99, Average Score: 42.39\n",
      "Process 5, Episode 960, Total Reward: -6.66, Average Score: 56.57\n",
      "Process 6, Episode 960, Total Reward: -40.97, Average Score: 51.27\n",
      "Process 1, Episode 960, Total Reward: 128.87, Average Score: 53.49\n",
      "Process 4, Episode 960, Total Reward: 17.12, Average Score: 41.47\n",
      "Process 7, Episode 960, Total Reward: -67.59, Average Score: 51.22\n",
      "Process 2, Episode 980, Total Reward: 135.33, Average Score: 40.76\n",
      "Process 0, Episode 970, Total Reward: -14.02, Average Score: 57.86\n",
      "Process 3, Episode 960, Total Reward: -22.58, Average Score: 43.55\n",
      "Process 5, Episode 970, Total Reward: 6.29, Average Score: 56.70\n",
      "Process 4, Episode 970, Total Reward: 109.49, Average Score: 41.89\n",
      "Process 6, Episode 970, Total Reward: 98.41, Average Score: 48.16\n",
      "Process 1, Episode 970, Total Reward: 146.27, Average Score: 55.22\n",
      "Process 7, Episode 970, Total Reward: -15.11, Average Score: 54.26\n",
      "Process 2, Episode 990, Total Reward: 25.26, Average Score: 36.46\n",
      "Process 0, Episode 980, Total Reward: 129.51, Average Score: 60.98\n",
      "Process 3, Episode 970, Total Reward: 91.64, Average Score: 45.07\n",
      "Process 5, Episode 980, Total Reward: 94.63, Average Score: 62.45\n",
      "Process 4, Episode 980, Total Reward: 65.39, Average Score: 51.08\n",
      "Process 6, Episode 980, Total Reward: 115.64, Average Score: 50.30\n",
      "Process 2, Episode 1000, Total Reward: 107.07, Average Score: 42.10\n",
      "Process 1, Episode 980, Total Reward: 127.64, Average Score: 57.01\n",
      "Process 7, Episode 980, Total Reward: 90.76, Average Score: 61.38\n",
      "Process 3, Episode 980, Total Reward: -41.56, Average Score: 47.89\n",
      "Process 0, Episode 990, Total Reward: 262.66, Average Score: 67.93\n",
      "Process 4, Episode 990, Total Reward: 103.57, Average Score: 54.90\n",
      "Process 5, Episode 990, Total Reward: 74.87, Average Score: 66.63\n",
      "Process 6, Episode 990, Total Reward: 101.85, Average Score: 49.56\n",
      "Process 2, Episode 1010, Total Reward: 116.04, Average Score: 46.80\n",
      "Process 3, Episode 990, Total Reward: 140.21, Average Score: 45.52\n",
      "Process 1, Episode 990, Total Reward: 134.12, Average Score: 59.26\n",
      "Process 7, Episode 990, Total Reward: 55.72, Average Score: 65.76\n",
      "Process 0, Episode 1000, Total Reward: 8.99, Average Score: 62.99\n",
      "Process 5, Episode 1000, Total Reward: 114.15, Average Score: 60.58\n",
      "Process 4, Episode 1000, Total Reward: -2.27, Average Score: 56.12\n",
      "Process 1, Episode 1000, Total Reward: -15.64, Average Score: 58.52\n",
      "Process 2, Episode 1020, Total Reward: 93.44, Average Score: 46.09\n",
      "Process 3, Episode 1000, Total Reward: 118.59, Average Score: 48.44\n",
      "Process 6, Episode 1000, Total Reward: 103.00, Average Score: 55.28\n",
      "Process 7, Episode 1000, Total Reward: -40.56, Average Score: 69.09\n",
      "Process 0, Episode 1010, Total Reward: 72.43, Average Score: 63.34\n",
      "Process 5, Episode 1010, Total Reward: -8.87, Average Score: 63.48\n",
      "Process 4, Episode 1010, Total Reward: -21.25, Average Score: 59.57\n",
      "Process 3, Episode 1010, Total Reward: 121.14, Average Score: 46.87\n",
      "Process 6, Episode 1010, Total Reward: -61.38, Average Score: 58.12\n",
      "Process 2, Episode 1030, Total Reward: 123.97, Average Score: 46.67\n",
      "Process 1, Episode 1010, Total Reward: 126.88, Average Score: 67.17\n",
      "Process 7, Episode 1010, Total Reward: 108.06, Average Score: 72.42\n",
      "Process 0, Episode 1020, Total Reward: 155.91, Average Score: 60.61\n",
      "Process 5, Episode 1020, Total Reward: 138.13, Average Score: 65.87\n",
      "Process 4, Episode 1020, Total Reward: 120.34, Average Score: 67.97\n",
      "Process 6, Episode 1020, Total Reward: -2.81, Average Score: 59.01\n",
      "Process 3, Episode 1020, Total Reward: 26.38, Average Score: 54.47\n",
      "Process 1, Episode 1020, Total Reward: 53.66, Average Score: 64.74\n",
      "Process 2, Episode 1040, Total Reward: 156.18, Average Score: 53.66\n",
      "Process 7, Episode 1020, Total Reward: 104.52, Average Score: 73.86\n",
      "Process 0, Episode 1030, Total Reward: -10.84, Average Score: 60.40\n",
      "Process 5, Episode 1030, Total Reward: 10.22, Average Score: 65.25\n",
      "Process 4, Episode 1030, Total Reward: -267.43, Average Score: 68.41\n",
      "Process 1, Episode 1030, Total Reward: 173.78, Average Score: 61.51\n",
      "Process 3, Episode 1030, Total Reward: 139.10, Average Score: 57.79\n",
      "Process 6, Episode 1030, Total Reward: 120.49, Average Score: 68.23\n",
      "Process 2, Episode 1050, Total Reward: 11.32, Average Score: 55.78\n",
      "Process 0, Episode 1040, Total Reward: 84.32, Average Score: 60.54\n",
      "Process 5, Episode 1040, Total Reward: 118.79, Average Score: 71.45\n",
      "Process 7, Episode 1030, Total Reward: 131.27, Average Score: 79.44\n",
      "Process 4, Episode 1040, Total Reward: 39.72, Average Score: 71.14\n",
      "Process 1, Episode 1040, Total Reward: 111.08, Average Score: 66.27\n",
      "Process 3, Episode 1040, Total Reward: 128.41, Average Score: 64.15\n",
      "Process 6, Episode 1040, Total Reward: 140.89, Average Score: 74.97\n",
      "Process 0, Episode 1050, Total Reward: 137.48, Average Score: 59.17\n",
      "Process 2, Episode 1060, Total Reward: 144.25, Average Score: 61.87\n",
      "Process 5, Episode 1050, Total Reward: 13.20, Average Score: 70.72\n",
      "Process 4, Episode 1050, Total Reward: -202.89, Average Score: 65.34\n",
      "Process 7, Episode 1040, Total Reward: 130.71, Average Score: 82.46\n",
      "Process 3, Episode 1050, Total Reward: 1.05, Average Score: 69.14\n",
      "Process 1, Episode 1050, Total Reward: 137.59, Average Score: 65.17\n",
      "Process 0, Episode 1060, Total Reward: 155.16, Average Score: 61.69\n",
      "Process 2, Episode 1070, Total Reward: 68.25, Average Score: 67.01\n",
      "Process 6, Episode 1050, Total Reward: 11.65, Average Score: 77.40\n",
      "Process 5, Episode 1060, Total Reward: 13.33, Average Score: 79.96\n",
      "Process 4, Episode 1060, Total Reward: 100.01, Average Score: 65.82\n",
      "Process 3, Episode 1060, Total Reward: 163.75, Average Score: 71.19\n",
      "Process 7, Episode 1050, Total Reward: 145.99, Average Score: 87.50\n",
      "Process 1, Episode 1060, Total Reward: 117.12, Average Score: 67.34\n",
      "Process 0, Episode 1070, Total Reward: 10.62, Average Score: 61.96\n",
      "Process 5, Episode 1070, Total Reward: 144.44, Average Score: 80.90\n",
      "Process 6, Episode 1060, Total Reward: 30.75, Average Score: 79.19\n",
      "Process 2, Episode 1080, Total Reward: 98.07, Average Score: 67.34\n",
      "Process 4, Episode 1070, Total Reward: -127.06, Average Score: 66.68\n",
      "Process 3, Episode 1070, Total Reward: -1.82, Average Score: 70.12\n",
      "Process 1, Episode 1070, Total Reward: 119.22, Average Score: 70.87\n",
      "Process 0, Episode 1080, Total Reward: 140.30, Average Score: 62.83\n",
      "Process 7, Episode 1060, Total Reward: 136.57, Average Score: 91.88\n",
      "Process 5, Episode 1080, Total Reward: 62.77, Average Score: 81.90\n",
      "Process 6, Episode 1070, Total Reward: 133.57, Average Score: 81.45\n",
      "Process 2, Episode 1090, Total Reward: 169.95, Average Score: 81.40\n",
      "Process 4, Episode 1080, Total Reward: 164.28, Average Score: 67.25\n",
      "Process 3, Episode 1080, Total Reward: 159.03, Average Score: 70.78\n",
      "Process 1, Episode 1080, Total Reward: 148.14, Average Score: 71.94\n",
      "Process 7, Episode 1070, Total Reward: 74.86, Average Score: 90.80\n",
      "Process 6, Episode 1080, Total Reward: 146.17, Average Score: 78.48\n",
      "Process 0, Episode 1090, Total Reward: 170.66, Average Score: 63.91\n",
      "Process 5, Episode 1090, Total Reward: 106.55, Average Score: 84.95\n",
      "Process 2, Episode 1100, Total Reward: 32.78, Average Score: 84.11\n",
      "Process 3, Episode 1090, Total Reward: 171.40, Average Score: 76.43\n",
      "Process 4, Episode 1090, Total Reward: 154.09, Average Score: 70.71\n",
      "Process 1, Episode 1090, Total Reward: 152.24, Average Score: 72.77\n",
      "Process 7, Episode 1080, Total Reward: 124.54, Average Score: 88.08\n",
      "Process 6, Episode 1090, Total Reward: 107.45, Average Score: 77.72\n",
      "Process 0, Episode 1100, Total Reward: -24.36, Average Score: 68.36\n",
      "Process 5, Episode 1100, Total Reward: 118.45, Average Score: 89.69\n",
      "Process 2, Episode 1110, Total Reward: 14.60, Average Score: 85.55\n",
      "Process 4, Episode 1100, Total Reward: 125.14, Average Score: 73.84\n",
      "Process 3, Episode 1100, Total Reward: 77.52, Average Score: 78.84\n",
      "Process 0, Episode 1110, Total Reward: -19.28, Average Score: 63.01\n",
      "Process 7, Episode 1090, Total Reward: 76.71, Average Score: 90.84\n",
      "Process 1, Episode 1100, Total Reward: 86.46, Average Score: 77.21\n",
      "Process 6, Episode 1100, Total Reward: 115.88, Average Score: 78.61\n",
      "Process 5, Episode 1110, Total Reward: 167.24, Average Score: 89.33\n",
      "Process 2, Episode 1120, Total Reward: 160.19, Average Score: 94.06\n",
      "Process 4, Episode 1110, Total Reward: 118.76, Average Score: 73.61\n",
      "Process 0, Episode 1120, Total Reward: -50.46, Average Score: 65.97\n",
      "Process 1, Episode 1110, Total Reward: -27.05, Average Score: 68.75\n",
      "Process 3, Episode 1110, Total Reward: 81.48, Average Score: 86.60\n",
      "Process 7, Episode 1100, Total Reward: 119.87, Average Score: 92.75\n",
      "Process 6, Episode 1110, Total Reward: 136.41, Average Score: 83.24\n",
      "Process 5, Episode 1120, Total Reward: 134.93, Average Score: 90.00\n",
      "Process 4, Episode 1120, Total Reward: 101.24, Average Score: 74.29\n",
      "Process 2, Episode 1130, Total Reward: 145.98, Average Score: 98.56\n",
      "Process 0, Episode 1130, Total Reward: 144.96, Average Score: 66.46\n",
      "Process 3, Episode 1120, Total Reward: 12.02, Average Score: 80.80\n",
      "Process 1, Episode 1120, Total Reward: 124.88, Average Score: 71.06\n",
      "Process 7, Episode 1110, Total Reward: 128.92, Average Score: 97.34\n",
      "Process 5, Episode 1130, Total Reward: 84.72, Average Score: 87.51\n",
      "Process 3, Episode 1130, Total Reward: -0.08, Average Score: 78.31\n",
      "Process 6, Episode 1120, Total Reward: 106.44, Average Score: 85.94\n",
      "Process 4, Episode 1130, Total Reward: -4.37, Average Score: 76.76\n",
      "Process 0, Episode 1140, Total Reward: 167.47, Average Score: 73.34\n",
      "Process 1, Episode 1130, Total Reward: -4.61, Average Score: 73.30\n",
      "Process 2, Episode 1140, Total Reward: 101.90, Average Score: 99.44\n",
      "Process 7, Episode 1120, Total Reward: 151.17, Average Score: 101.66\n",
      "Process 6, Episode 1130, Total Reward: -14.50, Average Score: 87.52\n",
      "Process 3, Episode 1140, Total Reward: 156.69, Average Score: 82.86\n",
      "Process 5, Episode 1140, Total Reward: 148.24, Average Score: 90.37\n",
      "Process 0, Episode 1150, Total Reward: -16.91, Average Score: 76.71\n",
      "Process 2, Episode 1150, Total Reward: 105.76, Average Score: 98.62\n",
      "Process 4, Episode 1140, Total Reward: 144.10, Average Score: 79.86\n",
      "Process 1, Episode 1140, Total Reward: 148.90, Average Score: 79.82\n",
      "Process 3, Episode 1150, Total Reward: 10.53, Average Score: 85.12\n",
      "Process 5, Episode 1150, Total Reward: -275.93, Average Score: 87.69\n",
      "Process 7, Episode 1130, Total Reward: -31.59, Average Score: 99.68\n",
      "Process 2, Episode 1160, Total Reward: 118.18, Average Score: 98.27\n",
      "Process 6, Episode 1140, Total Reward: 85.41, Average Score: 88.73\n",
      "Process 0, Episode 1160, Total Reward: 228.42, Average Score: 82.69\n",
      "Process 1, Episode 1150, Total Reward: -43.65, Average Score: 84.83\n",
      "Process 4, Episode 1150, Total Reward: 69.62, Average Score: 89.64\n",
      "Process 3, Episode 1160, Total Reward: 116.80, Average Score: 87.71\n",
      "Process 5, Episode 1160, Total Reward: 121.84, Average Score: 82.01\n",
      "Process 7, Episode 1140, Total Reward: -173.60, Average Score: 97.72\n",
      "Process 2, Episode 1170, Total Reward: 5.80, Average Score: 99.50\n",
      "Process 6, Episode 1150, Total Reward: 138.01, Average Score: 90.04\n",
      "Process 0, Episode 1170, Total Reward: 165.84, Average Score: 88.86\n",
      "Process 4, Episode 1160, Total Reward: 44.85, Average Score: 90.23\n",
      "Process 1, Episode 1160, Total Reward: 129.76, Average Score: 82.18\n",
      "Process 5, Episode 1170, Total Reward: 79.37, Average Score: 79.75\n",
      "Process 7, Episode 1150, Total Reward: 148.24, Average Score: 93.10\n",
      "Process 3, Episode 1170, Total Reward: 78.10, Average Score: 89.10\n",
      "Process 6, Episode 1160, Total Reward: 141.27, Average Score: 85.38\n",
      "Process 2, Episode 1180, Total Reward: 126.92, Average Score: 96.53\n",
      "Process 0, Episode 1180, Total Reward: -121.21, Average Score: 95.30\n",
      "Process 4, Episode 1170, Total Reward: 153.05, Average Score: 96.82\n",
      "Process 5, Episode 1180, Total Reward: 178.30, Average Score: 83.11\n",
      "Process 1, Episode 1170, Total Reward: 115.77, Average Score: 85.11\n",
      "Process 7, Episode 1160, Total Reward: 152.94, Average Score: 96.20\n",
      "Process 3, Episode 1180, Total Reward: 103.99, Average Score: 90.51\n",
      "Process 6, Episode 1170, Total Reward: 10.66, Average Score: 87.48\n",
      "Process 4, Episode 1180, Total Reward: 18.93, Average Score: 94.97\n",
      "Process 2, Episode 1190, Total Reward: 23.86, Average Score: 93.08\n",
      "Process 1, Episode 1180, Total Reward: 128.59, Average Score: 84.04\n",
      "Process 0, Episode 1190, Total Reward: 146.19, Average Score: 93.23\n",
      "Process 5, Episode 1190, Total Reward: 104.48, Average Score: 81.78\n",
      "Process 4, Episode 1190, Total Reward: 120.57, Average Score: 90.56\n",
      "Process 7, Episode 1170, Total Reward: -2.16, Average Score: 100.45\n",
      "Process 2, Episode 1200, Total Reward: -133.17, Average Score: 91.21\n",
      "Process 6, Episode 1180, Total Reward: 153.46, Average Score: 93.36\n",
      "Process 3, Episode 1190, Total Reward: 162.95, Average Score: 92.34\n",
      "Process 5, Episode 1200, Total Reward: 88.57, Average Score: 81.86\n",
      "Process 1, Episode 1190, Total Reward: 40.71, Average Score: 87.10\n",
      "Process 0, Episode 1200, Total Reward: 59.66, Average Score: 97.10\n",
      "Process 4, Episode 1200, Total Reward: -0.74, Average Score: 95.04\n",
      "Process 7, Episode 1180, Total Reward: 17.21, Average Score: 101.94\n",
      "Process 2, Episode 1210, Total Reward: 259.18, Average Score: 89.05\n",
      "Process 3, Episode 1200, Total Reward: 124.77, Average Score: 89.80\n",
      "Process 6, Episode 1190, Total Reward: 94.82, Average Score: 95.08\n",
      "Process 5, Episode 1210, Total Reward: 75.22, Average Score: 83.68\n",
      "Process 1, Episode 1200, Total Reward: 161.09, Average Score: 89.43\n",
      "Process 0, Episode 1210, Total Reward: 108.90, Average Score: 109.67\n",
      "Process 2, Episode 1220, Total Reward: -122.58, Average Score: 85.73\n",
      "Process 7, Episode 1190, Total Reward: 3.13, Average Score: 102.46\n",
      "Process 4, Episode 1210, Total Reward: 129.46, Average Score: 98.71\n",
      "Process 3, Episode 1210, Total Reward: 129.77, Average Score: 88.28\n",
      "Process 6, Episode 1200, Total Reward: 127.73, Average Score: 93.56\n",
      "Process 1, Episode 1210, Total Reward: 131.09, Average Score: 99.04\n",
      "Process 5, Episode 1220, Total Reward: 99.14, Average Score: 84.51\n",
      "Process 2, Episode 1230, Total Reward: -144.47, Average Score: 85.83\n",
      "Process 0, Episode 1220, Total Reward: 118.00, Average Score: 112.46\n",
      "Process 4, Episode 1220, Total Reward: 128.71, Average Score: 99.78\n",
      "Process 7, Episode 1200, Total Reward: -105.82, Average Score: 99.36\n",
      "Process 3, Episode 1220, Total Reward: 144.75, Average Score: 93.56\n",
      "Process 6, Episode 1210, Total Reward: 146.89, Average Score: 91.50\n",
      "Process 1, Episode 1220, Total Reward: 86.05, Average Score: 97.11\n",
      "Process 5, Episode 1230, Total Reward: -242.07, Average Score: 86.18\n",
      "Process 2, Episode 1240, Total Reward: 161.11, Average Score: 84.98\n",
      "Process 0, Episode 1230, Total Reward: 129.21, Average Score: 114.71\n",
      "Process 4, Episode 1230, Total Reward: 115.67, Average Score: 102.38\n",
      "Process 6, Episode 1220, Total Reward: 107.12, Average Score: 85.74\n",
      "Process 7, Episode 1210, Total Reward: 111.72, Average Score: 100.01\n",
      "Process 3, Episode 1230, Total Reward: 79.65, Average Score: 100.47\n",
      "Process 1, Episode 1230, Total Reward: -14.51, Average Score: 97.77\n",
      "Process 5, Episode 1240, Total Reward: 143.62, Average Score: 81.67\n",
      "Process 2, Episode 1250, Total Reward: 112.93, Average Score: 89.72\n",
      "Process 0, Episode 1240, Total Reward: 153.78, Average Score: 116.78\n",
      "Process 4, Episode 1240, Total Reward: 122.67, Average Score: 103.41\n",
      "Process 7, Episode 1220, Total Reward: 108.69, Average Score: 95.61\n",
      "Process 6, Episode 1230, Total Reward: 129.52, Average Score: 86.62\n",
      "Process 1, Episode 1240, Total Reward: 70.30, Average Score: 92.64\n",
      "Process 3, Episode 1240, Total Reward: 138.70, Average Score: 100.61\n",
      "Process 4, Episode 1250, Total Reward: 140.79, Average Score: 101.45\n",
      "Process 2, Episode 1260, Total Reward: 85.86, Average Score: 92.94\n",
      "Process 7, Episode 1230, Total Reward: 150.30, Average Score: 90.42\n",
      "Process 5, Episode 1250, Total Reward: 104.96, Average Score: 90.92\n",
      "Process 0, Episode 1250, Total Reward: 87.46, Average Score: 116.00\n",
      "Process 6, Episode 1240, Total Reward: 151.78, Average Score: 85.14\n",
      "Process 3, Episode 1250, Total Reward: -31.22, Average Score: 100.33\n",
      "Process 1, Episode 1250, Total Reward: 117.25, Average Score: 92.53\n",
      "Process 4, Episode 1260, Total Reward: 154.15, Average Score: 102.91\n",
      "Process 7, Episode 1240, Total Reward: 81.28, Average Score: 92.31\n",
      "Process 5, Episode 1260, Total Reward: 110.51, Average Score: 100.63\n",
      "Process 6, Episode 1250, Total Reward: 135.95, Average Score: 83.94\n",
      "Process 2, Episode 1270, Total Reward: 71.21, Average Score: 90.79\n",
      "Process 0, Episode 1260, Total Reward: -21.44, Average Score: 112.88\n",
      "Process 3, Episode 1260, Total Reward: 160.02, Average Score: 100.01\n",
      "Process 1, Episode 1260, Total Reward: 266.19, Average Score: 94.85\n",
      "Process 7, Episode 1250, Total Reward: 189.93, Average Score: 94.80\n",
      "Process 4, Episode 1270, Total Reward: 125.41, Average Score: 101.62\n",
      "Process 5, Episode 1270, Total Reward: 100.93, Average Score: 103.35\n",
      "Process 6, Episode 1260, Total Reward: 97.94, Average Score: 88.03\n",
      "Process 0, Episode 1270, Total Reward: 88.17, Average Score: 110.95\n",
      "Process 2, Episode 1280, Total Reward: 156.69, Average Score: 94.53\n",
      "Process 3, Episode 1270, Total Reward: 94.84, Average Score: 104.61\n",
      "Process 7, Episode 1260, Total Reward: -125.17, Average Score: 85.97\n",
      "Process 1, Episode 1270, Total Reward: 134.60, Average Score: 95.19\n",
      "Process 4, Episode 1280, Total Reward: 257.42, Average Score: 105.69\n",
      "Process 5, Episode 1280, Total Reward: 95.85, Average Score: 100.21\n",
      "Process 6, Episode 1270, Total Reward: -90.52, Average Score: 89.25\n",
      "Process 0, Episode 1280, Total Reward: 148.54, Average Score: 111.67\n",
      "Process 2, Episode 1290, Total Reward: 148.71, Average Score: 95.33\n",
      "Process 3, Episode 1280, Total Reward: 183.32, Average Score: 109.60\n",
      "Process 1, Episode 1280, Total Reward: 9.25, Average Score: 94.40\n",
      "Process 7, Episode 1270, Total Reward: 137.32, Average Score: 86.14\n",
      "Process 4, Episode 1290, Total Reward: 150.15, Average Score: 113.68\n",
      "Process 5, Episode 1290, Total Reward: 124.73, Average Score: 97.26\n",
      "Process 6, Episode 1280, Total Reward: 112.78, Average Score: 89.90\n",
      "Process 0, Episode 1290, Total Reward: 76.83, Average Score: 111.38\n",
      "Process 1, Episode 1290, Total Reward: 266.86, Average Score: 97.17\n",
      "Process 7, Episode 1280, Total Reward: 13.66, Average Score: 89.75\n",
      "Process 3, Episode 1290, Total Reward: 123.00, Average Score: 107.25\n",
      "Process 2, Episode 1300, Total Reward: 112.04, Average Score: 99.91\n",
      "Process 4, Episode 1300, Total Reward: 141.35, Average Score: 114.34\n",
      "Process 5, Episode 1300, Total Reward: 91.58, Average Score: 99.28\n",
      "Process 1, Episode 1300, Total Reward: 142.12, Average Score: 101.14\n",
      "Process 3, Episode 1300, Total Reward: 137.21, Average Score: 110.94\n",
      "Process 2, Episode 1310, Total Reward: 44.20, Average Score: 101.97\n",
      "Process 4, Episode 1310, Total Reward: 134.48, Average Score: 121.49\n",
      "Process 7, Episode 1290, Total Reward: 244.17, Average Score: 91.23\n",
      "Process 6, Episode 1290, Total Reward: 281.77, Average Score: 94.77\n",
      "Process 0, Episode 1300, Total Reward: 63.14, Average Score: 111.09\n",
      "Process 1, Episode 1310, Total Reward: 112.53, Average Score: 96.63\n",
      "Process 2, Episode 1320, Total Reward: -144.90, Average Score: 101.04\n",
      "Process 7, Episode 1300, Total Reward: 10.10, Average Score: 91.58\n",
      "Process 6, Episode 1300, Total Reward: -18.47, Average Score: 94.80\n",
      "Process 4, Episode 1320, Total Reward: -16.33, Average Score: 119.25\n",
      "Process 5, Episode 1310, Total Reward: 76.82, Average Score: 98.31\n",
      "Process 3, Episode 1310, Total Reward: 75.14, Average Score: 113.47\n",
      "Process 0, Episode 1310, Total Reward: 132.07, Average Score: 111.54\n",
      "Process 1, Episode 1320, Total Reward: 64.45, Average Score: 101.81\n",
      "Process 2, Episode 1330, Total Reward: 49.93, Average Score: 103.64\n",
      "Process 4, Episode 1330, Total Reward: 158.58, Average Score: 120.66\n",
      "Process 6, Episode 1310, Total Reward: 23.38, Average Score: 103.12\n",
      "Process 7, Episode 1310, Total Reward: 154.47, Average Score: 93.43\n",
      "Process 5, Episode 1320, Total Reward: 117.61, Average Score: 105.32\n",
      "Process 3, Episode 1320, Total Reward: 30.23, Average Score: 113.81\n",
      "Process 1, Episode 1330, Total Reward: 14.79, Average Score: 105.37\n",
      "Process 0, Episode 1320, Total Reward: 149.07, Average Score: 114.04\n",
      "Process 6, Episode 1320, Total Reward: -289.96, Average Score: 103.08\n",
      "Process 2, Episode 1340, Total Reward: 129.67, Average Score: 104.30\n",
      "Process 4, Episode 1340, Total Reward: 172.20, Average Score: 123.45\n",
      "Process 5, Episode 1330, Total Reward: 201.60, Average Score: 112.87\n",
      "Process 7, Episode 1320, Total Reward: 243.40, Average Score: 97.46\n",
      "Process 3, Episode 1330, Total Reward: 119.10, Average Score: 114.42\n",
      "Process 0, Episode 1330, Total Reward: 167.92, Average Score: 120.69\n",
      "Process 1, Episode 1340, Total Reward: 110.83, Average Score: 110.97\n",
      "Process 6, Episode 1330, Total Reward: 144.73, Average Score: 103.01\n",
      "Process 4, Episode 1350, Total Reward: 271.76, Average Score: 125.34\n",
      "Process 3, Episode 1340, Total Reward: 29.59, Average Score: 115.34\n",
      "Process 5, Episode 1340, Total Reward: 74.40, Average Score: 120.59\n",
      "Process 7, Episode 1330, Total Reward: 4.15, Average Score: 106.10\n",
      "Process 0, Episode 1340, Total Reward: -0.81, Average Score: 111.44\n",
      "Process 2, Episode 1350, Total Reward: 156.00, Average Score: 106.61\n",
      "Process 1, Episode 1350, Total Reward: 124.63, Average Score: 119.17\n",
      "Process 6, Episode 1340, Total Reward: 158.77, Average Score: 103.66\n",
      "Process 4, Episode 1360, Total Reward: 83.28, Average Score: 119.57\n",
      "Process 7, Episode 1340, Total Reward: 13.27, Average Score: 109.39\n",
      "Process 3, Episode 1350, Total Reward: -188.51, Average Score: 115.12\n",
      "Process 2, Episode 1360, Total Reward: 98.40, Average Score: 106.87\n",
      "Process 5, Episode 1350, Total Reward: 125.36, Average Score: 120.85\n",
      "Process 0, Episode 1350, Total Reward: 198.46, Average Score: 120.64\n",
      "Process 1, Episode 1360, Total Reward: 273.86, Average Score: 128.54\n",
      "Process 6, Episode 1350, Total Reward: 142.89, Average Score: 109.84\n",
      "Process 4, Episode 1370, Total Reward: 114.04, Average Score: 117.07\n",
      "Process 0, Episode 1360, Total Reward: 196.90, Average Score: 117.97\n",
      "Process 2, Episode 1370, Total Reward: 250.27, Average Score: 112.96\n",
      "Process 7, Episode 1350, Total Reward: 144.12, Average Score: 117.45\n",
      "Process 3, Episode 1360, Total Reward: 105.15, Average Score: 118.67\n",
      "Process 5, Episode 1360, Total Reward: 149.10, Average Score: 120.07\n",
      "Process 1, Episode 1370, Total Reward: 98.88, Average Score: 133.99\n",
      "Process 6, Episode 1360, Total Reward: 126.89, Average Score: 116.29\n",
      "Process 4, Episode 1380, Total Reward: 241.11, Average Score: 121.36\n",
      "Process 0, Episode 1370, Total Reward: 0.37, Average Score: 121.53\n",
      "Process 2, Episode 1380, Total Reward: 28.92, Average Score: 117.42\n",
      "Process 3, Episode 1370, Total Reward: 94.20, Average Score: 115.42\n",
      "Process 7, Episode 1360, Total Reward: 209.70, Average Score: 131.26\n",
      "Process 1, Episode 1380, Total Reward: 295.06, Average Score: 143.51\n",
      "Process 5, Episode 1370, Total Reward: 111.41, Average Score: 130.42\n",
      "Process 6, Episode 1370, Total Reward: 156.14, Average Score: 124.09\n",
      "Process 0, Episode 1380, Total Reward: 150.17, Average Score: 122.47\n",
      "Process 4, Episode 1390, Total Reward: 149.70, Average Score: 118.88\n",
      "Process 2, Episode 1390, Total Reward: 239.47, Average Score: 116.20\n",
      "Process 3, Episode 1380, Total Reward: 18.28, Average Score: 115.05\n",
      "Process 5, Episode 1380, Total Reward: 273.83, Average Score: 136.91\n",
      "Process 1, Episode 1390, Total Reward: 144.25, Average Score: 139.01\n",
      "Process 7, Episode 1370, Total Reward: 271.24, Average Score: 138.96\n",
      "Process 4, Episode 1400, Total Reward: -88.55, Average Score: 120.84\n",
      "Process 6, Episode 1380, Total Reward: 249.82, Average Score: 126.22\n",
      "Process 0, Episode 1390, Total Reward: 117.41, Average Score: 129.98\n",
      "Process 3, Episode 1390, Total Reward: 261.10, Average Score: 119.80\n",
      "Process 2, Episode 1400, Total Reward: 290.63, Average Score: 121.07\n",
      "Process 1, Episode 1400, Total Reward: 269.67, Average Score: 140.12\n",
      "Process 4, Episode 1410, Total Reward: 154.70, Average Score: 111.36\n",
      "Process 5, Episode 1390, Total Reward: 99.19, Average Score: 144.72\n",
      "Process 0, Episode 1400, Total Reward: 221.13, Average Score: 136.02\n",
      "Process 7, Episode 1380, Total Reward: 234.31, Average Score: 138.44\n",
      "Process 3, Episode 1400, Total Reward: 243.39, Average Score: 127.70\n",
      "Process 2, Episode 1410, Total Reward: 148.38, Average Score: 130.34\n",
      "Process 6, Episode 1390, Total Reward: 99.25, Average Score: 123.58\n",
      "Process 7, Episode 1390, Total Reward: 256.92, Average Score: 145.18\n",
      "Process 4, Episode 1420, Total Reward: 216.56, Average Score: 117.25\n",
      "Process 2, Episode 1420, Total Reward: 247.47, Average Score: 137.13\n",
      "Process 1, Episode 1410, Total Reward: -30.68, Average Score: 145.63\n",
      "Process 3, Episode 1410, Total Reward: 146.08, Average Score: 130.96\n",
      "Process 0, Episode 1410, Total Reward: 161.77, Average Score: 135.16\n",
      "Process 5, Episode 1400, Total Reward: 197.19, Average Score: 147.98\n",
      "Process 6, Episode 1400, Total Reward: 257.57, Average Score: 131.68\n",
      "Process 7, Episode 1400, Total Reward: 264.45, Average Score: 146.03\n",
      "Process 4, Episode 1430, Total Reward: -6.18, Average Score: 116.98\n",
      "Process 3, Episode 1420, Total Reward: 128.40, Average Score: 136.27\n",
      "Process 2, Episode 1430, Total Reward: 90.85, Average Score: 137.02\n",
      "Process 0, Episode 1420, Total Reward: 250.60, Average Score: 139.23\n",
      "Process 1, Episode 1420, Total Reward: 153.88, Average Score: 145.57\n",
      "Process 5, Episode 1410, Total Reward: 141.28, Average Score: 152.84\n",
      "Process 6, Episode 1410, Total Reward: 279.38, Average Score: 128.69\n",
      "Process 7, Episode 1410, Total Reward: 139.76, Average Score: 152.26\n",
      "Process 4, Episode 1440, Total Reward: 85.67, Average Score: 116.57\n",
      "Process 1, Episode 1430, Total Reward: 250.65, Average Score: 155.82\n",
      "Process 6, Episode 1420, Total Reward: 51.67, Average Score: 141.84\n",
      "Process 5, Episode 1420, Total Reward: 81.00, Average Score: 146.29\n",
      "Process 3, Episode 1430, Total Reward: 115.09, Average Score: 136.97\n",
      "Process 0, Episode 1430, Total Reward: 128.62, Average Score: 139.14\n",
      "Process 2, Episode 1440, Total Reward: 155.72, Average Score: 140.30\n",
      "Process 7, Episode 1420, Total Reward: 136.96, Average Score: 149.76\n",
      "Process 4, Episode 1450, Total Reward: 142.86, Average Score: 124.86\n",
      "Process 1, Episode 1440, Total Reward: 146.22, Average Score: 152.71\n",
      "Process 5, Episode 1430, Total Reward: 61.84, Average Score: 146.57\n",
      "Process 3, Episode 1440, Total Reward: 234.39, Average Score: 139.98\n",
      "Process 6, Episode 1430, Total Reward: 5.73, Average Score: 140.59\n",
      "Process 0, Episode 1440, Total Reward: 127.00, Average Score: 151.28\n",
      "Process 7, Episode 1430, Total Reward: 125.71, Average Score: 149.12\n",
      "Process 2, Episode 1450, Total Reward: 143.29, Average Score: 144.73\n",
      "Process 4, Episode 1460, Total Reward: 9.04, Average Score: 135.69\n",
      "Process 1, Episode 1450, Total Reward: 243.31, Average Score: 149.83\n",
      "Process 6, Episode 1440, Total Reward: 266.83, Average Score: 145.44\n",
      "Process 5, Episode 1440, Total Reward: 283.47, Average Score: 146.58\n",
      "Process 0, Episode 1450, Total Reward: 254.28, Average Score: 150.17\n",
      "Process 3, Episode 1450, Total Reward: 275.72, Average Score: 147.54\n",
      "Process 7, Episode 1440, Total Reward: 131.67, Average Score: 149.34\n",
      "Process 1, Episode 1460, Total Reward: 186.79, Average Score: 142.88\n",
      "Process 2, Episode 1460, Total Reward: 158.29, Average Score: 148.88\n",
      "Process 4, Episode 1470, Total Reward: 250.99, Average Score: 141.40\n",
      "Process 5, Episode 1450, Total Reward: 152.06, Average Score: 142.57\n",
      "Process 0, Episode 1460, Total Reward: 207.41, Average Score: 158.25\n",
      "Process 6, Episode 1450, Total Reward: 147.10, Average Score: 145.21\n",
      "Process 3, Episode 1460, Total Reward: 155.28, Average Score: 145.84\n",
      "Process 1, Episode 1470, Total Reward: 120.90, Average Score: 138.19\n",
      "Process 2, Episode 1470, Total Reward: 145.41, Average Score: 149.98\n",
      "Process 7, Episode 1450, Total Reward: 231.44, Average Score: 150.42\n",
      "Process 5, Episode 1460, Total Reward: 64.44, Average Score: 150.43\n",
      "Process 4, Episode 1480, Total Reward: 136.57, Average Score: 141.39\n",
      "Process 0, Episode 1470, Total Reward: 269.13, Average Score: 157.67\n",
      "Process 6, Episode 1460, Total Reward: 102.68, Average Score: 144.64\n",
      "Process 3, Episode 1470, Total Reward: 139.46, Average Score: 152.49\n",
      "Process 1, Episode 1480, Total Reward: 249.54, Average Score: 137.19\n",
      "Process 7, Episode 1460, Total Reward: 240.16, Average Score: 148.79\n",
      "Process 2, Episode 1480, Total Reward: 240.04, Average Score: 150.41\n",
      "Process 6, Episode 1470, Total Reward: 230.50, Average Score: 144.63\n",
      "Process 5, Episode 1470, Total Reward: 231.82, Average Score: 147.07\n",
      "Process 0, Episode 1480, Total Reward: 261.63, Average Score: 158.35\n",
      "Process 4, Episode 1490, Total Reward: 79.79, Average Score: 143.51\n",
      "Process 3, Episode 1480, Total Reward: 287.22, Average Score: 152.64\n",
      "Process 1, Episode 1490, Total Reward: 293.83, Average Score: 142.77\n",
      "Process 7, Episode 1470, Total Reward: 118.08, Average Score: 152.32\n",
      "Process 2, Episode 1490, Total Reward: 194.50, Average Score: 159.07\n",
      "Process 4, Episode 1500, Total Reward: 115.74, Average Score: 147.91\n",
      "Process 5, Episode 1480, Total Reward: 150.65, Average Score: 141.23\n",
      "Process 6, Episode 1480, Total Reward: 219.52, Average Score: 146.09\n",
      "Process 0, Episode 1490, Total Reward: 124.17, Average Score: 154.21\n",
      "Process 1, Episode 1500, Total Reward: 282.24, Average Score: 144.48\n",
      "Process 3, Episode 1490, Total Reward: 158.65, Average Score: 153.58\n",
      "Process 7, Episode 1480, Total Reward: 72.25, Average Score: 157.59\n",
      "Process 6, Episode 1490, Total Reward: 146.32, Average Score: 151.92\n",
      "Process 4, Episode 1510, Total Reward: 152.96, Average Score: 154.43\n",
      "Process 0, Episode 1500, Total Reward: 257.80, Average Score: 159.86\n",
      "Process 2, Episode 1500, Total Reward: 157.67, Average Score: 157.63\n",
      "Process 5, Episode 1490, Total Reward: 156.62, Average Score: 142.94\n",
      "Process 1, Episode 1510, Total Reward: 269.55, Average Score: 149.26\n",
      "Process 6, Episode 1500, Total Reward: 261.05, Average Score: 149.22\n",
      "Process 2, Episode 1510, Total Reward: 286.88, Average Score: 158.35\n",
      "Process 4, Episode 1520, Total Reward: 254.04, Average Score: 155.01\n",
      "Process 7, Episode 1490, Total Reward: 31.63, Average Score: 158.92\n",
      "Process 5, Episode 1500, Total Reward: 14.33, Average Score: 147.16\n",
      "Process 3, Episode 1500, Total Reward: 96.81, Average Score: 147.93\n",
      "Process 0, Episode 1510, Total Reward: 241.23, Average Score: 159.76\n",
      "Process 6, Episode 1510, Total Reward: 21.75, Average Score: 153.21\n",
      "Process 1, Episode 1520, Total Reward: 191.95, Average Score: 161.02\n",
      "Process 4, Episode 1530, Total Reward: 277.51, Average Score: 165.42\n",
      "Process 2, Episode 1520, Total Reward: 249.17, Average Score: 166.93\n",
      "Process 3, Episode 1510, Total Reward: 248.18, Average Score: 152.83\n",
      "Process 7, Episode 1500, Total Reward: 272.75, Average Score: 165.32\n",
      "Process 0, Episode 1520, Total Reward: 273.29, Average Score: 158.80\n",
      "Process 5, Episode 1510, Total Reward: 245.38, Average Score: 152.75\n",
      "Process 1, Episode 1530, Total Reward: 301.71, Average Score: 165.15\n",
      "Process 6, Episode 1520, Total Reward: 211.69, Average Score: 155.68\n",
      "Process 2, Episode 1530, Total Reward: 267.00, Average Score: 176.05\n",
      "Process 3, Episode 1520, Total Reward: 278.35, Average Score: 152.47\n",
      "Process 4, Episode 1540, Total Reward: 69.79, Average Score: 166.88\n",
      "Process 7, Episode 1510, Total Reward: 269.51, Average Score: 162.99\n",
      "Process 0, Episode 1530, Total Reward: 77.24, Average Score: 161.29\n",
      "Process 5, Episode 1520, Total Reward: 274.69, Average Score: 162.04\n",
      "Process 1, Episode 1540, Total Reward: 249.46, Average Score: 161.60\n",
      "Process 6, Episode 1530, Total Reward: 27.09, Average Score: 158.55\n",
      "Process 4, Episode 1550, Total Reward: 263.31, Average Score: 168.49\n",
      "Process 2, Episode 1540, Total Reward: 284.73, Average Score: 181.58\n",
      "Process 3, Episode 1530, Total Reward: 25.11, Average Score: 156.84\n",
      "Process 0, Episode 1540, Total Reward: 7.64, Average Score: 166.06\n",
      "Process 5, Episode 1530, Total Reward: 115.43, Average Score: 168.00\n",
      "Process 7, Episode 1520, Total Reward: 170.00, Average Score: 176.07\n",
      "Process 1, Episode 1550, Total Reward: 253.56, Average Score: 169.40\n",
      "Process 6, Episode 1540, Total Reward: 77.09, Average Score: 159.58\n",
      "Process 4, Episode 1560, Total Reward: 33.55, Average Score: 173.41\n",
      "Process 2, Episode 1550, Total Reward: 146.72, Average Score: 187.74\n",
      "Process 5, Episode 1540, Total Reward: 298.44, Average Score: 176.92\n",
      "Process 3, Episode 1540, Total Reward: 114.43, Average Score: 158.61\n",
      "Process 7, Episode 1530, Total Reward: 256.10, Average Score: 175.32\n",
      "Process 0, Episode 1550, Total Reward: 281.81, Average Score: 172.62\n",
      "Process 1, Episode 1560, Total Reward: 252.91, Average Score: 176.55\n",
      "Process 6, Episode 1550, Total Reward: 259.12, Average Score: 166.45\n",
      "Process 4, Episode 1570, Total Reward: 252.31, Average Score: 174.73\n",
      "Process 5, Episode 1550, Total Reward: 138.05, Average Score: 179.93\n",
      "Process 3, Episode 1550, Total Reward: 307.10, Average Score: 159.11\n",
      "Process 2, Episode 1560, Total Reward: 267.55, Average Score: 186.78\n",
      "Process 1, Episode 1570, Total Reward: 246.25, Average Score: 183.90\n",
      "Process 7, Episode 1540, Total Reward: 160.18, Average Score: 172.95\n",
      "Process 0, Episode 1560, Total Reward: 109.77, Average Score: 180.85\n",
      "Process 6, Episode 1560, Total Reward: 244.62, Average Score: 160.44\n",
      "Process 1, Episode 1580, Total Reward: 250.76, Average Score: 183.29\n",
      "Process 4, Episode 1580, Total Reward: 295.81, Average Score: 180.04\n",
      "Process 5, Episode 1560, Total Reward: 271.89, Average Score: 178.96\n",
      "Process 2, Episode 1570, Total Reward: 277.43, Average Score: 194.35\n",
      "Process 0, Episode 1570, Total Reward: 276.83, Average Score: 185.85\n",
      "Process 3, Episode 1560, Total Reward: 248.98, Average Score: 164.51\n",
      "Process 7, Episode 1550, Total Reward: 246.33, Average Score: 178.03\n",
      "Process 6, Episode 1570, Total Reward: 280.98, Average Score: 167.27\n",
      "Process 1, Episode 1590, Total Reward: 272.62, Average Score: 188.45\n",
      "\n",
      "Environment solved in 1576 episodes!\tAverage Score: 200.85\n",
      "Process 5, Episode 1570, Total Reward: 269.02, Average Score: 183.28\n",
      "Total Training Time: 4.90\n"
     ]
    }
   ],
   "source": [
    "%run asynchronous_n_step_dqn_lunarlander.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAGQAlgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD5/ooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAor6TvfAnhe/VRNolqm3keQph/8AQMZrm774OaHP5jWd5eWrt90ErIi/gQCf++q+fpcSYSfxpx+V/wAv8jV0ZHiFFel3/wAGdUg+az1SznjAyxnDREfluH6155f2UunX01pM0TSRNtYxSLIp+jKSP8Oh5r1sNjsPif4MrkOLjuV6KKK6iQooooAKKKKACiiigAooooAKKK+k73wJ4Xv1UTaJapt5HkKYf/QMZrzcwzOngXD2ib5r7eVv8y4Qctj5sor2+++Dmhz+Y1neXlq7fdBKyIv4EAn/AL6rm7/4M6pB81nqlnPGBljOGiI/LcP1rKlnuBqfbs/NP/hhulJHmlFWL+yl06+mtJmiaSJtrGKRZFP0ZSR/h0PNV69ZNSV1sZhRRRTAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAmu7O60+6ktb22mtrmM4eGZCjofQg8ioa+69c8OaN4kszaazptvexEEDzUyyZ7q3VT7gg15L4m/Zz0q7Es/hvUZbGY5K211mSLPYBvvKPc7qAPm6iuu8T/DLxb4TZ31DSZZLVAWN3agyw7RjJJH3Rz/EBXI0AFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUV3OjeBIL3T4bi5unDyIsm2PGAGGQOe+MVtRoTrPlggOGor1GLwBpCf6wzSf8CxVqPwVocZyLUt/vOTXYsrrvew7Mk8V/FbUNI8QajpNjp1sDY3ctsZp2Z/M2MVyFG3GcZ6muMvviT4qv/MU6m0Eb/wW8apt+jAbv1r0vxN4d0ubxJrMpsYGmkvJ2LsgJ3F25/OvDbmE211LA3WNyv5GvHlkOFwajJU1r8/zKc5Pdkl5qN7qLh768uLl14DTys5H5mq1FFaqKirIgKKKKYBRRRQAUUUUAFFFFABRRRQAV6/4r+K2oaR4g1HSbHTrYGxu5bYzTsz+ZsYrkKNuM4z1NeQV0Hjv/kofiX/sK3X/AKNauXE4KhiXF1o3tt8ylJx2Ll98SfFV/wCYp1NoI3/gt41Tb9GA3frXOXmo3uouHvry4uXXgNPKzkfmarUVdLDUaP8ADgl6ITbe4UUUVuIKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAPv8AooooAK4zxN8K/CHisvLe6UkF2wP+lWZ8qTJ7nHysfdga7OigD5q8Sfs6a1Zb5vD+owalF8xEE48mUDPCg5KscdSSv0ryXV9C1bQLr7Nq+m3VlLkhVniK7sHBKk8MPcZFfd9Vr/TrLVLRrTULO3u7ZyC0NxEsiEg5GVII60AfBNFfUvib9n/wvq4eXR5JtGuSc/u8ywnJycoxyPbDAD0rx7xN8FPGXh0PNHZDVLVT/rbDMjAZ4zHjd9cAgetAHndFOdHikaORGR1JVlYYII6gim0AFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAV7P4d50OzP/TCIf8AkNa8e+yXH2cT+RJ5J6OFO30617D4b/5F2wPfyE/9BFetlK/ev0GjVooor3yjQ17/AJGLU/8Ar7l/9DNeKeNLP7J4kmIHyzASDj8P6V7Xr3/Ixan/ANfcv/oZrzjxtpkmp6tplvCVV5FkG4+w3H9BXl46nz4aNt1b/IlnnVFb3iLw2+gpbN5vnJKDucDA3en5Vg14NSnKnLlktRBRRRUAFFFFABRRRQAUUUUAFFFFABXQeO/+Sh+Jf+wrdf8Ao1q5+ug8d/8AJQ/Ev/YVuv8A0a1AHP0UUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQB9/0UUUAFFFFABRRRQAUUUUAc/4i8D+GvFat/bOj21zKQF8/bslAHQCRcNj2zivIPEv7OGWkn8M6vgdVtb8fniRR+QK/U17/RQB8P8AiDwT4l8LN/xOtGurVOP320PFz0HmLlc+2c1gV9/OiyIyOoZWGCpGQRXnfiT4J+DPEP7yKxbSrj/npp+I1PBwCmCuOc8AHjrQB8i0V6z4l/Z/8U6RmXSJINZg7iPEUo46lGOCPoxPtXlt5ZXWnXclpfW01tcxnDwzxlHU4zyp5HBoAgooooAKKKKACiiigAooooAKKKKACpIIZLm4jghXdLK4RFzjJJwBzUddB4NtFuteDtgiCJpQCm7J4UfTBbdn2q6UOeaj3A67xHJHpPgxbaNid6iGMvySuMfntFaeh3cFroNisz7P9HjIyD/cWuS+IV+st1a2cTZSNN5A6ZPA/rXK2mm31+22zsrm5b0hiZ/5CvTq4z2Nd8qvZWA9fk8Q6VF969gB9DKoP6mqcnjHRout0p+nzfyzXG2Xwz8bX+PI8MakAehmhMQ/8fxXRWXwF8d3WPOs7Szz/wA97pTj/vjdUvNqr2SHc2/FfjXS7LxXrNuzM0kV9OjAK3UOwPauD8SeK4NUezlsklintpRIrsoxkcj9cdq9E134CeMNb8Q6nq32zQ4ft13Lc+V9pmbZvcttz5QzjOM4FZ//AAzj4w/6CWh/9/5v/jVc9THVpw5G9BFK8ji8T+EC8SgPs8yMf3SOoyQOhBGe+K8tIIJBBBHUGvf/AA58GPGWhQXERvdElV/mjHnygBuhz+67gD6Y6cmsS6/Z48Y3V1JO1/oCGRtxVJpsZ/790YqtCtGM/tW1A8aor2D/AIZx8Yf9BLQ/+/8AN/8AGqP+GcfGH/QS0P8A7/zf/Gq4wPH6K9g/4Zx8Yf8AQS0P/v8Azf8Axqj/AIZx8Yf9BLQ/+/8AN/8AGqAPH6K9g/4Zx8Yf9BLQ/wDv/N/8ao/4Zx8Yf9BLQ/8Av/N/8aoA8for2D/hnHxh/wBBLQ/+/wDN/wDGqP8AhnHxh/0EtD/7/wA3/wAaoA8for2D/hnHxh/0EtD/AO/83/xqj/hnHxh/0EtD/wC/83/xqgDx+ug8d/8AJQ/Ev/YVuv8A0a1egf8ADOPjD/oJaH/3/m/+NVB4x+Dnjq68Q6pq8Wl21yl5dy3JS0u1fZvctj5whbGcdPwoA8morW1Pwtr+i7jqei6haKOrzW7Kv4NjBrJoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigD7/ooooAKKKKACiiigAooooAKKKKACiiigArP1fQdJ1+1Ftq+nWt7CCSqzxBtp6ZUnkH3HNaFFAHiviT9nTRb0vN4e1GfTpCciCf99F06A/eHPOSWriv+GcfGH/QS0P8A7/zf/Gq+n6KAPmD/AIZx8Yf9BLQ/+/8AN/8AGqP+GcfGH/QS0P8A7/zf/Gq+n6KAPmD/AIZx8Yf9BLQ/+/8AN/8AGqP+GcfGH/QS0P8A7/zf/Gq+n6KAPmD/AIZx8Yf9BLQ/+/8AN/8AGqP+GcfGH/QS0P8A7/zf/Gq+n6KAPmOL9m/xWZAJtV0ZE7lJJWP5GMfzrdsv2aOjX3if6pBaf+zFv6V7/RQB5FZfs6+EbfBubzVbpu4MqIv5Bc/rXR2Hwd8B6cyvFoSPIvR5p5HP5FsfpXdUUAZMHhfQLaTzYdE05JcAeYLZN5x0y2MmtVVVFCqAFHQAdKWihu4BRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABWFqfgvwxrO46joGnXDnrI1uu/wD76Az+tbtFAHl+p/ALwPf7jbQXunsf+fa4JGfpJurjNT/ZqlG5tJ8Ro3pHd25H/jyk/wDoNfQdFAHyTqfwJ8dadkxWNtfoOrWlwp/R9pP5Vxmp+Ftf0XcdT0XULRR1ea3ZV/BsYNfdNFAHwBRX27rvgPwt4kgePU9Es5WbnzkjEcoPs64b9cHvXjfif9nCeIed4X1UTKOttf4V+nZ1GCenBUfWgDwaitXXvDOteGLz7JrWm3FlKc7fMX5XxjO1h8rDkcgmsqgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKt6bpd/rF6lnptnPd3L/digjLsffA7e9ew+E/2d9VvmS48T3i6fbnBNtbkSTH2LfdXtz830FAHi0UUk8yQwxtJLIwVEQZLE8AADqa9S8KfAXxPr0a3GqMmiWrDK+em+Y/9swRj/gRB9q+iPDXgXw34RjA0fSoIZgpVrlhvmYHGcuecHA4HHtXRUAcV4R+FXhXwcUms7H7TfLyLy8xJIDz93janXGVAOOpNFdrRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBDd2dtf2slreW8NxbyDDxTIHRh6EHg15V4o/Z/8M6uHm0aSXR7piWwmZYSScn5Ccj22kAelet0UAfHPif4ReMfCw82fTTe2o63FhmZBxk5GAygepUD3rhq+/wCuO8U/C/wl4ud59Q0xYrxwc3dqfKlyccnHDnj+IGgD4xor2XxR+zxr2mh7jw/dxarCCSIHxDMBnjqdrYHU5GewryS/02+0q6NrqNlc2dwACYriJo3APQ4YA0AVaKKKACirNrp19fHFpZXFwfSGJn/kK3rT4deM77HkeGNUwehktmjB/FgKAOYor0S0+B3j66wW0iO3U95rqMfoGJ/St60/Zx8VS4N1qWk26nsryOw/DYB+tAHjtFfQFp+zQODe+KPqsNn/AFL/ANK6bRv2e/COnXSXF7Nfalt/5ZTSBIyfUhQD+uKAPm/QfDGt+J7v7NoumXF5IPvGNflT/eY4VfxIr23wn+zoi7LnxXqG89fsdkcD6NIRn8AB9a90sbCz0y0S0sLWG1toxhYoUCKPoBVigDM0Pw7o/hqxFno2nQWUHGREvLkd2Y8sfckmtOiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKz9Y0LSfEFp9l1fTra9hByqzxhtpxjKk8g+45rQooA4y0+E/gSyx5XhqzbH/PYtL/AOhk1vWnhjQNPx9j0PTbfHTybRE/kK1aKAEAAAAAAHQCloooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA//Z",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAn20lEQVR4AWIYBaMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyGwGgIjIbAKBgNgdEQGA2B0RAYDYHREBgFoyEwGgKjITAaAqMhMBoCo2A0BEZDYDQERkNgNARGQ2AUjIbAaAiMhsBoCIyGwGgIjILREBgNgdEQGA2B0RAYDYFRMBoCoyEwGgKjITAaAqMhMApGQ2A0BEZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyGwGgIjIbAKBgNgdEQGA2B0RAYDYHREBgFoyEwCkZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgFoyEwGgKjITAaAqMhMBoCo2A0BEZDYDQERkNgNARGQ2AUjIbAaAiMhsBoCIyGwGgIjILREBgNgdEQGA2BUTAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyGwGgIjIbAKBgNgdEQGA2B0RAYDYHREBgFoyEwGgKjITAaAqMhMBoCo2A0BEZDYDQERkNgNARGQ2AUjIbAaAiMhsBoCIyGwCgYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyGwGgIjIbAKBgNgdEQGA2B0RAYDYHREBgFoyEwGgKjITAaAqMhMBoCo2A0BEZDYDQERkNgNARGQ2AUjIbAaAiMhsBoCIyGwGgIjILREBgNgdEQGA2B0RAYDYFRMBoCoyEwGgKjITAaAqMhMApGQ2A0BEZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyGwGgIjIbAKBgNgdEQGA2B0RAYDYHREBgFoyEwCkZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyGwGgIjIbAKBgNgdEQGAWjITAaAqMhMApGQ2A0BEZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyC0RAYDYFRMBoCoyEwGgKjITAaAqMhMApGQ2A0BEZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyGwGgIjIbAKBgNgdEQGA2B0RAYDYHREBgFoyEwGgKjITAaAqMhMBoCo2A0BEZDYDQERkNgNARGwSgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsBoCIyGwGgIjIbAKBgNgdEQGA2B0RAYDYHREBgFoyEwGgKjITAaAqMhMBoCo2A0BEZDYDQERkNgNARGQ2AUjIbAaAiMhsBoCIyGwGgIjILREBgNgdEQGA2B0RAYDYFRMBoCoyEwGgKjITAaAqMhMApGQ2A0BEZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMglEwGgKjITAaAqMhMBoCo2A0BEZDYDQERkNgNARGQ2AUjIbAaAiMhsBoCIyGwGgIjILREBgNgdEQGA2B0RAYDYFRMBoCoyEwGgKjITAaAqMhMApGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgNARGQ2A0BEbBaAiMhsAoGA2B0RAYDYFRMBoCoyEwGgKjITAaAqMhMApGQ2A0BEZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhsAoGA2B0RAYDYHREBgNgdEQGAWjITAaAqMhMBoCoyEwGgKjYDQERkNgNARGQ2A0BEZDYBSMhsBoCIyGwGgIjIbAaAiMgtEQGA2B0RAYDYHREBgNgVEwGgKjITAaAqMhMBoCoyEwCkZDYDQERkNgFIyGwGgIjILREBgNgdEQGA2B0RAYDYFRMBoCoyEwGgKjITAaAqMhMApGQ2A0BEZDYDQERkNgNARGwWgIjIbAaAiMhsBoCIyGwCgYDYHREBgNgdEQGA2B0RAYBaMhMBoCoyEwGgKjITAaAqNgNARGQ2A0BEZDYDQERkNgFIyGwGgIjIbAaAiMhsBoCIyC0RAYDYHREBgNgdEQGA2BUTAaAqMhMBoCoyEwGgKjITAKRkNgNARGQ2A0BEZDYDQERsFoCIyGwGgIjIbAaAiMhgApgJEUxaNqR0NgsIfAmTMM//4xfPnCcP8+w+HDDAsWDHYHD1H3jYbzEI24UWdjBaMVIdZgGRUcqiFw5gy6y0frRfQQoQZ/NJypEYqjZgwWMFoRDpaYGHUHVUIAs4BGM3a0XkQLEPK4o+FMXriN6hqcYLQiHJzxMuoqMkOAYAGNZu5ovYgWIERyR8OZyIAaVTYkwGhFOCSiadSRxIYAqQU0mrmj9SJagODijoYzrpAZFR+KYLQiHIqxNupmnCFAYQGNZq6JCZrAKBcKRsMZGhCj1LAALMPCF6OeGA0B6oTAaI+QOuFIyJTRcCYUQqPydAWjFSFdg3vUssEWAqMlMn1iZDSc6RPOo7aQB0YrQvLCbVTXUA2B0RKZPjE3Gs70CedRW6gCRitCqgTjqCGDNwRGS2T6xM1oONMnnEdtoQUYrQhpEaqjZg5kCIyWyPQJ/dFwpk84j9oyCkZDYDQESAuB////k6ZhVDVZITAazmQF26imQQqYBqm7Rp01GgKjITAaAqMhMBoCdAGjFSFdgnnUktEQGA2B0RAYDYHBCpj+//8/b968weq8UXeNhsBoCIyGwGgIjIYAjcF/GLh161ZFRQWNbRs1fjQEaBsCmHNXTEzMbGyctLV15JmOGc4jLwxGfTx8ACNmgt66dauPj8/w8eKoT4ZpCDAyMrKycrGxcbGxcXJy8PPySPBwiZw+v8zNqYKNlZOFhZWRmek/6HbC3z9/ft21q3uYBsPAeOv///+MjKMHNA5M4I/aSnWAZfuEt7f3////P3z4MG/evOLiYqpbOWrgaAjgDwFwH44LXMlxgus5UFXHxsbFwcHPxSHAzs7DysrBzMLKCCqJmZkZ2TjY+FkYOViYOJiZ2BkYGNSlPViY2FmYQCIgQUb2s89n4bdxVHY0BEZDYCQDLBUhJDgEBASKwODUqVPz5s2bOXMmRHyUHA0B2oWAuLi6qWn4//////39+/8fAxMjCwsjOxsLDwcrPy+nJLx6Y4bWc6BqD9MxIlzq6IJ/mISFFd6+fYAuPsofDYHREBgFDAw4K0J44JiBwYwZM5YsWRIbGwsXH2WMhgDVQyDSfcGrr1fUhL2pa7IQt4qIiOJoRUjdUB01bTQEhg0gYftETEzM////7927V1NTM2z8P+qRwRMCDWnP/vz7wcbMTXUnifFqSkpoUd3YUQNHQ2A0BIYHIKEihHhYUVGxubn5////O3bsgIiMkqMhQK0Q+P33KysNKkIeNgkeXhFGRpJTO7X8NWrOaAiMhsBgBuQXDe7u7v/////y5cukSZMGsw9H3TaEQuD3v2+sTFy0cDAbI6+IiCItTB41czQERkNgqAPyK0KIz7m5uXNzc////3/27Nns7GyI4Cg5GgLkhQDtKkIxXi1xMTXyXDWqazQERkNgeANKK0J46BgZGU2ZMuX////Lly+HC44yRkOApBD4/fcbKzNNeoS8bFJSkjokOWZU8WgIjIbACAFUqwjh4RUREfH////Hjx83NDTABUcZoyFATAjQrkfIzSbGzsHNzMxKjDNG1YyGwGgIjChA/YoQEnwyMjL19fX////fs2cPRGSUHA0BgiFAux4hAwMDD5vk6DQhwSgYVTAaAiMQ0KoihAels7Pz////f/z4MW3aNLjgKGM0BDBD4M+/H0yMrEyMzJhSVBER4JBXkLOgilGjhoyGwGgIDCdA84oQEljs7OyZmZn///+/dOlSfn4+RHCUHA0B5BD49fcrLTYRwq3gZZcSEVaAc0cZoyEwGgKjIQABdKoIIZYxMDDo6upOmDDh////q1evhguOMkZDgIGBgXYThJDg5WIVYWZlY2XlgHBHydEQGA2B0RCAAHpXhBBbGRgYQkJC/v////z589bWVrjgKGMkhwBNJwghAcvPISsrbQxhj5KjITAaAqMhAAEDVhFCrJeQkKiqqvr///+BAwcgIqPkiA0BWvcIGRgYeNmkFGTNRmwIj3p8NARGQwArGOCKEO4me3v7//////nzZ9as0Rtz4KEyshh06BGCzlrjFxpZwTrq29EQGA0BQmCwVIQQdzIzM6empv7////atWslJSUQwVFyhIQAHXqEXKzC/xn/sbFhOdebiYlWq1VHSPSNenM0BIYuGFwVITwcNTU1u7u7////v379erjgKGN4hwAdeoQMDAysTJxMTCjJXlhYwdW1WEhIfngH76jvRkNgNARwAcL3EeLSSR/xgICA////v3nzZt68eeXl5fSxdNSWAQmBb7/fMBFxQSaFbmNh4mBj4/rx4zMXl5C6sqOklBY7K68Al9wz8Wtv3tyj0PBR7aMhMBoCQxGgNI0HrQdERETKysr+//9/9OjR5OTkQevOUYdREgKszJwMtL8piYWJg5UVdJxpmPcUJWUrHalQfakYeQE7RUUzBgZGStw/qnc0BEZDYIiCoVERwgPXyspqzpw5////nzdvHlxwlDHkQkBe3piHRwTN2WxMPH///0QTpDoX3CPkZGBgkBOw+ff/NysTdL6Qn1VeScmS6taNGjgaAqMhMPjBEKsI4QGamJj4////W7duVVRUwAVHGUMlBAz0Ax0dcj3da8yMY6SkdFhY2BkYGJiZ2P/+o3lFyMzEzsYG6hEyMbKIceu++noZEmjyQrZqqvYQ9ig5GgKjITCiwFCtCCGRpKqq2t7e/v///y1btkBERsnBHwLliVfYmHmNpVJ1JEP01cKMjUP+/fvLwMDAwsTx598PWrsf3CMEVYQNs6TEuXVfwipCNmZuThYhKSltWjtg1PzREBgNgcEGhnZFCA9Nb2/v////v3//vre3Fy44yhicIfDx52N+dlkGBgZuNjFuVlHmvxz//v2hb0UIGhplYGBgYmQR5lJ7/fUaJKCURV1UVUY7hZDAGCVHQ2AEgWFSEUJiTEBAoKio6P///ydPnkxPT4cIjpKDLQQ+/XzCxy4DcdWnn09ev74PYdOpR8gIWjUKsRGtU8jDJs7DJSYoCHUbRM0oORoCoyEw7MGwqgjhsWVmZjZjxoz///8vXrwYLjjKGAwhUJ/69POvp0gV4eNrt7ZBHMbCxP6HZnOEf/79+PHnw5dfL3/++cjNjVinw8EiwM7C9/HHI4gblEWdR5fMQIJilBwNgZEDBvs+QgpjIgYM7t+/P2/evJaWFgpNG9VOeQggdwd//vn0+8/39x8fQ4wlo0f4//+/P/9+/vn348//H3/+/QCvtZF88ukESAQi/g8s/v8nMyM7CxMHCxM7EyPL37+IJTkNs6SK4k+9+HKRn0OOgYFBiFNZXEydg4P3x4/PEFeNkqMhMBoCwx4M84oQEn+KiorNYLBz504PDw+I4Cg5ICHw6edjPvAEIQMDw6efT758fg93BnJF+O//X3BlBqrbIAwQ+R9c50HqNnA99/f/bxYmcA3HyMHCxMHMBFp9+vv3zz+/f/369f3Hzy/fv7//+u3t568vv35/8/X7my/fX3/78Q5uI4TBxy7z+NPx77/fcbKCjiEV4dBQUrK8dm0XRHaUHA2B0RAY9oDx////w96TaB78+vXrvHnz8vLy0MRHubQOgYa0Z1dfr1IScOVkFWRgYLjzbuexM7PvPTgGsbc4/uztd9tYmTn//Pvx//8/cAcOVLdBGCCSkf3fv3+/f//49evbjx+fvn1/9/Xbu6/fX3/5/vrr97dfQeSbH78+MzKSvC8+J2bfl18vFQRAK2X+//9/9vmsLVsaIa4aJbGGwP///8kIZ6xGjQqOhsCAgxHRI0QLZW5u7lwwOHfu3Lx586ZOnYqmYJRLoxD49ffL33+/IbUgAwPDx5+PHzw6BbeLi1VYnNPg/x+GH78+fwNVbG++fH8A78lBqrq//37D1VORIcKlAR5QtWBhYmdkZGT7x6+oaH7//kkqWjFq1GgIjIbAoAUjsUeIGRkrVqyIjIzEFB8VoW4IZMfs/fbrtbyAHQMDw+efz689X799F5UnbsnuqaRFbmZg+C/Fa8LAwPDzz+czj2bt3jO6FQdn/JMdzjhNHJUYDYGBA8Nz1Sip4RkREfH////Hjx83NDSQqndUPZEh0JD2DHmlzKdfT56/uE6kXjooQ95cz87Cy8bEq6RkRQd7R60YDYHREBhwMFoRIqJARkamvr7+////e/bsQYiOsqgXAsgV4btvd+8/PE49syk1qXmOgiCH0ptvNxgYGF59vcrOyS0jo0epoaP6R0NgNASGAhiJc4QE48XZ2fn///8/f/6cN29eVlYWQfWjCogJgS+/XnKyCDIzsTEwMPz59+Pnn08fPz4jRiPd1Ihz6956t+XRx6OMv9n2HO78+hV9fSndXDJq0WgIjIYAPcFojxBnaLOzs2dmZv7////SpUv5+fk41Y1KEBEC4HFRlI0TH97TpBakZClj53wdXjaZh7fOb9xRPloL4o9VSsIZv8mjsqMhQH8wuliGhDBfs2ZNaGgoCRpGlcJCoD71yc23m2T4LHnYxEEbJ97uPHBywpMnF2Hy5NBCQkJKGEBRUZEcs0b1kB4C586dOwsDZ86cId2AUR2jITBYwGhFSHJMvHjxYt68edXV1STrHMEaKhKvX32zUoxLh49dho9d5tyzubv2dP/8+YXIIFFWVsao8pQEBASI1D6qjA4hMFov0iGQR62gERitCMkP2IMHDzo4OJCvfyTprE15+P3P26+/X3/6+eTTz8d//vzevr0VMwBGO3mYYTJERUbrxSEacSMTjFaElMb7379/582bl5aWRqlBw12/vIS5gpSVgqSljITRrE0efHz/Rzt5wz3OEf4brRcRYTHKGkygu7u7pKRktCKkWpxcv3593rx5PT09VDNxWBg02skbFtFIZU+M1otUDtBR40gHCxcujIuLg+gbrQgh4UBNcsOGDYGBgdQ0cYiYNTqTN0QiatA5c7ReHHRRMqzB9u3b0W5fGK0IaRXhb968mTdvXnl5Oa0sGDhzRzt5Axf2I8Lm0XpxRETzQIAzZ84YGxtj2jxaEWKGCZVFjh07Nm/evLlz51LZXLoYN9rJo0swj1qCLwRG60V8oTMqRwTIzc0tLS2VlZXFpXa0IsQVMqPio2A0BAZjCIzWi4MxVgYraG5uLi0tZWcH3VSKx42jFSGewBmVGg2B0RAY7CEwWi8O9hgaIDBjxoz09HQiLR+tCIkMqFFloyEwGgJDIARG68UhEEk0BuvWrSN1ueJoRUjjOBk1fjQERkNg4EJgtF4cuLAfAHDkyBFra2syLB6tCMkItFEtoyEwGgJDMgSmT58+ep/MkIw5vCAlJaW0tFRNTQ2vKnySoxUhvtAZlRsNgdEQGH4hMHo44rCJ06qqqtLSUsqPHR69hmnYJIlRj4yGwGgIEBUC9vb2////f/36dUNDA1EaRhUNPjBhwoT///+3trZSXgsyMDCM9ggHXwyPumg0BEZDgI4hMHq92tACS5cujYqKoq6bR3uE1A3PUdNGQ2A0BIZYCISEhPz////atWvZ2dlDzOkjyblubm67d+/+//8/1WvB0R7hSEpHo34dDYHRECAiBEYX1Aw2EBMTU1paqqenRzuHjQ6N0i5sR00eDYHREBiqITC6oGYwgMLCwtLSUklJSVo7ZnRolNYhPGr+aAiMhsDQC4HRBTUDC9rb2//8+dPX10eHWnB0aHRg43rU9tEQGA2BoRECowtq6Abmzp2blJREN+sgFo32CCHhMEqOhsBoCIyGAM4QGF1QQ2tga2u7adOm////078WHO0R0jpyR80fDYHREBiGITC6oIaKIDg4uLS01NzcnIpmkmrU6GIZUkNsVP1oCIyGwGgIgEJgdEENhSAzM7O0tFRRUZFCcyjXPjo0SnkYjpowGgKjITASQ2B0QQ3ZoK6u7vPnz9OmTRsMteDo0CjZ8TiqcTQERkNgNARQQmB0QQ1BIC8vX1paOggPLhgdGiUYd6MKRkNgFIyGALEhcP369algQKyGkaHO1NS0tLQ0NDR0cHp3tCIcnPEy6qrREBgNgaEdAqMLaiDAy8urtLTUwcEBwh2c5GhFODjjZdRVoyEwGgLDIQRG8oKahISE0tJSLS2twR+Ro4tlBn8cjbpwNARGQ2CohsDIXFBTVlb2+vXr+fPnD4lacHSxzFDNXaPuHg2B0RAYiiEwnBbUCAoKymEAWTAYclEzOjQ65KJs1MGjITAaAkM7BIbQghpWVlZZWVmM+g4kwM3NPbSjAcn1oxUhUmCMMkdDYDQERkOAjiEweBbUiImJycnJYdZ5EhISdAyPAbNqtCIcsKAftXg0BEZDYDQEGBgY6LaghpOTE9SVA2O0Oo+dnX0kx8VoRTiSY3/U76MhMBoCgyUE3rx5M3Xq1IaGBsodJCUlBa7sQARyhSciIkK54cMSjFaEwzJaRz01GgKjITBUQ4DIBTW8vLygig4Jw+s8Zmbmoer5AQKjFeEABfyotaMhMBoCoyGAOwTgC2qQajoEU1ZWVkBAALfuURnSwGhFSFp4jaoeDYHREBgNgdEQGGZgdEP9MIvQUe+MhsBoCIyGwGgIkAZGK0LSwmtU9WgIjIbAaAiMhsAwA6MV4TCL0FHvjIbAaAiMhsBoCJAGRitC0sJrVPVoCIyGwGgIjIbAMAOjFeEwi9BR74yGwGgIjIbAaAiQBkYrQtLCa1T1aAiMhsBoCIyGwDADoxXhMIvQUe+MhsBoCIyC0RAgDYxWhKSF16jq0RAYDYHREBgNgWEGRivCYRaho94ZDYHREBgNgdEQIA2MVoSkhdeo6tEQGA2B0RAYDYFhBkYrwmEWoaPeGQ2B0RAYDYHRECANjFaEpIXXqOrREBgNgdEQGA2BYQZGK8JhFqGj3hkNgdEQGA2B0RAgDYxWhKSF16jq0RAYDYHREBgNgWEGRivCYRaho94ZDYHREBgNgdEQIA2MVoSkhdeo6tEQGA2B0RAYDYFhBkYrwmEWoaPeGQ2B0RAYDYHRECANjFaEpIXXqOrREBgNgdEQGA2BYQZGK8JhFqGj3hkNgdEQGA2B0RAgDYxWhKSF16jq0RAYDYHREBgNgWEGRivCYRaho94ZDYHREBgNgdEQIA2MVoSkhdeo6tEQGA2B0RAYDYFhBkYrwmEWoaPeGQ2B0RAYDYHRECANjFaEpIXXqOrREBgNgdEQGA2BYQZGK8JhFqGj3hkNgdEQGA2BUUAaGK0ISQuvUdWjITAaAqMhMBoCwwyMVoTDLEJHvTMaAqMhMBoCoyFAGhitCEkLr1HVoyEwGgKjITAaAsMMjFaEwyxCR70zGgKjITAaAqMhQBoYrQhJC69R1aMhMBoCoyEwGgLDDIxWhMMsQke9MxoCoyEwGgKjIUAaGK0ISQuvUdWjITAaAqMhMBoCwwyMVoTDLEJHvTMaAqMhMBoCoyFAGhitCEkLr1HVoyEwGgKjITAaAsMMjFaEwyxCR70zGgKjITAaAqMhQBoYrQhJC69R1aMhMBoCoyEwGgLDDIxWhMMsQke9MxoCoyEwGgKjIUAaGK0ISQuvUdWjITAaAqMhMBoCwwyMVoTDLEJHvTMaAqMhMBoCoyFAGhitCEkLr1HVoyEwGgKjITAaAsMMjFaEwyxCR70zGgKjITAaAqMhQBoYrQhJC69R1aMhMBoCoyEwGgLDDIxWhMMsQke9MxoCoyEwGgKjIUAaGK0ISQuvUdWjITAaAqMhMBoCwwyMVoTDLEJHvTMaAqMhMBoCoyFAGhitCEkLr1HVoyEwGgKjITAaAsMMjFaEwyxCR70zGgKjITAaAqMhQBoYrQhJC69R1aMhMBoCoyEwGgLDDIxWhMMsQke9MxoCoyEwGgKjIUAaGK0ISQuvUdWjITAaAqMhMBoCwwyMVoTDLEJHvTMaAqMhMBoCoyFAGhitCEkLr1HVoyEwGgKjITAaAsMMjFaEwyxCR70zGgKjITAaAqMhQBoYrQhJC69R1aMhMBoCoyEwGgLDDIxWhMMsQke9MxoCoyEwGgKjIUAaGK0ISQuvUdWjITAaAqMhMBoCwwyMVoTDLEJHvTMaAqMhMBoCoyFAGhitCEkLr1HVoyEwGgKjITAaAsMMjFaEwyxCR70zGgKjITAaAqMhQBoYrQhJC69R1aMhMBoCoyEwGgLDDIxWhMMsQke9MxoCoyEwGgKjIQAYaSEAAJ3bknlB8Jv3AAAAAElFTkSuQmCC",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=600x400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score: 250.53932903934313\n",
      "Average score: 273.1193242925088\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('LunarLander-v3', render_mode=\"rgb_array\")\n",
    "\n",
    "state_size = env.observation_space.shape[0]\n",
    "action_size = env.action_space.n\n",
    "\n",
    "# init agent\n",
    "Q = QNetwork(state_size, action_size, hidden_size=128).to(device)\n",
    "\n",
    "agent = Agent(\n",
    "    state_size=state_size,\n",
    "    action_size=action_size,\n",
    "    Q_network=Q,\n",
    "    gamma=0.995,\n",
    ")\n",
    "\n",
    "# load the weights from file\n",
    "agent.Q.load_state_dict(torch.load('checkpoint.pth'))\n",
    "\n",
    "env = gym.make(\"LunarLander-v3\", render_mode=\"rgb_array\")\n",
    "\n",
    "avg_score = 0\n",
    "for i in range(3):\n",
    "\tstate, info = env.reset(seed=123+i)\n",
    "\ttotal_reward = 0\n",
    "\tfor t in range(500):\n",
    "\t\t# clear frame\n",
    "\t\tdisplay.clear_output(wait=True)\n",
    "\n",
    "\t\taction = agent.select_action(state)\n",
    "\n",
    "\t\t# render game\n",
    "\t\tframe = env.render()\n",
    "\t\tdisplay.display(Image.fromarray(frame))\n",
    "\t\ttime.sleep(0.05)\n",
    "\n",
    "\t\tstate, reward, done, _, info = env.step(action)\n",
    "\t\ttotal_reward += reward\n",
    "\t\tif done:\n",
    "\t\t\tbreak\n",
    "\t\n",
    "\tavg_score += total_reward\n",
    "\tprint('Final score:', total_reward)\n",
    "env.close()\n",
    "print(f\"Average score: {avg_score / 3}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Summary\n",
    "\n",
    "The **Asynchronous n-step Deep Q-Network (AsyncNDQN)** is a powerful extension of the original Asynchronous One Step DQN algorithm, leveraging the benefits of multi-threading and n-step returns. By incorporating longer-term rewards and reducing the correlation between updates, AsyncNDQN achieves faster convergence, improved stability, and better overall performance.\n",
    "\n",
    "Compared to **[AsyncDQN](./asynchronous_one_step_dqn_lunarlander.ipynb)** (4.92 minutes), AsyncNDQN achieves similar training times (4.90 minutes) but often converges to higher-quality policies due to the use of N-step returns.\n",
    "\n",
    "Under the three environments (same seeds) with average score:\n",
    "- AsyncDQN: 260.91916800965765\n",
    "- AsyncSARSA: 291.84418794684774\n",
    "- AsyncNDQN: 273.1193242925088"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "DRL",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}