Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "\n",
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='./Reacher-2.app')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  collections  import deque\n",
    "from itertools import count\n",
    "import time\n",
    "import torch\n",
    "from ddpg_agent import Agent\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1, Average Score: 0.00, Max: 0.00, Min: 0.00, Time: 13.72\n",
      "Episode 2, Average Score: 0.01, Max: 0.13, Min: 0.00, Time: 14.10\n",
      "Episode 3, Average Score: 0.01, Max: 0.08, Min: 0.00, Time: 14.37\n",
      "Episode 4, Average Score: 0.09, Max: 0.85, Min: 0.00, Time: 14.45\n",
      "Episode 5, Average Score: 0.24, Max: 1.39, Min: 0.02, Time: 15.84\n",
      "Episode 6, Average Score: 0.21, Max: 0.99, Min: 0.00, Time: 15.59\n",
      "Episode 7, Average Score: 0.22, Max: 0.93, Min: 0.00, Time: 15.70\n",
      "Episode 8, Average Score: 0.26, Max: 1.06, Min: 0.00, Time: 15.21\n",
      "Episode 9, Average Score: 0.29, Max: 1.68, Min: 0.12, Time: 15.83\n",
      "Episode 10, Average Score: 0.33, Max: 1.46, Min: 0.00, Time: 16.63\n",
      "Episode 10\tAverage Score: 0.33\n",
      "Episode 11, Average Score: 0.36, Max: 2.05, Min: 0.00, Time: 17.20\n",
      "Episode 12, Average Score: 0.41, Max: 2.33, Min: 0.05, Time: 16.35\n",
      "Episode 13, Average Score: 0.44, Max: 1.54, Min: 0.12, Time: 16.55\n",
      "Episode 14, Average Score: 0.46, Max: 2.56, Min: 0.05, Time: 17.22\n",
      "Episode 15, Average Score: 0.51, Max: 2.11, Min: 0.23, Time: 17.26\n",
      "Episode 16, Average Score: 0.51, Max: 1.80, Min: 0.07, Time: 17.53\n",
      "Episode 17, Average Score: 0.54, Max: 1.64, Min: 0.18, Time: 18.06\n",
      "Episode 18, Average Score: 0.55, Max: 1.46, Min: 0.30, Time: 17.65\n",
      "Episode 19, Average Score: 0.58, Max: 2.74, Min: 0.29, Time: 17.91\n",
      "Episode 20, Average Score: 0.60, Max: 1.71, Min: 0.30, Time: 21.31\n",
      "Episode 20\tAverage Score: 0.60\n",
      "Episode 21, Average Score: 0.61, Max: 1.86, Min: 0.25, Time: 18.48\n",
      "Episode 22, Average Score: 0.62, Max: 1.56, Min: 0.00, Time: 19.43\n",
      "Episode 23, Average Score: 0.62, Max: 1.52, Min: 0.21, Time: 19.85\n",
      "Episode 24, Average Score: 0.63, Max: 1.51, Min: 0.20, Time: 19.34\n",
      "Episode 25, Average Score: 0.64, Max: 1.65, Min: 0.09, Time: 20.42\n",
      "Episode 26, Average Score: 0.65, Max: 1.78, Min: 0.14, Time: 21.14\n",
      "Episode 27, Average Score: 0.66, Max: 2.09, Min: 0.38, Time: 20.27\n",
      "Episode 28, Average Score: 0.69, Max: 2.35, Min: 0.24, Time: 21.36\n",
      "Episode 29, Average Score: 0.71, Max: 2.99, Min: 0.59, Time: 21.80\n",
      "Episode 30, Average Score: 0.75, Max: 2.77, Min: 0.55, Time: 20.79\n",
      "Episode 30\tAverage Score: 0.75\n",
      "Episode 31, Average Score: 0.77, Max: 2.70, Min: 0.27, Time: 20.95\n",
      "Episode 32, Average Score: 0.79, Max: 2.65, Min: 0.62, Time: 21.30\n",
      "Episode 33, Average Score: 0.80, Max: 2.19, Min: 0.10, Time: 21.80\n",
      "Episode 34, Average Score: 0.83, Max: 2.63, Min: 0.88, Time: 23.10\n",
      "Episode 35, Average Score: 0.85, Max: 3.33, Min: 0.11, Time: 22.79\n",
      "Episode 36, Average Score: 0.87, Max: 2.47, Min: 0.33, Time: 23.02\n",
      "Episode 37, Average Score: 0.88, Max: 2.28, Min: 0.17, Time: 22.86\n",
      "Episode 38, Average Score: 0.89, Max: 2.52, Min: 0.49, Time: 23.25\n",
      "Episode 39, Average Score: 0.91, Max: 2.89, Min: 0.53, Time: 23.19\n",
      "Episode 40, Average Score: 0.92, Max: 3.13, Min: 0.54, Time: 23.43\n",
      "Episode 40\tAverage Score: 0.92\n",
      "Episode 41, Average Score: 0.93, Max: 2.76, Min: 0.60, Time: 23.50\n",
      "Episode 42, Average Score: 0.97, Max: 4.26, Min: 0.80, Time: 23.88\n",
      "Episode 43, Average Score: 0.99, Max: 5.91, Min: 0.61, Time: 26.26\n",
      "Episode 44, Average Score: 1.02, Max: 3.91, Min: 0.58, Time: 27.00\n",
      "Episode 45, Average Score: 1.03, Max: 3.26, Min: 0.68, Time: 25.52\n",
      "Episode 46, Average Score: 1.04, Max: 3.15, Min: 0.30, Time: 25.31\n",
      "Episode 47, Average Score: 1.06, Max: 3.73, Min: 0.68, Time: 25.21\n",
      "Episode 48, Average Score: 1.08, Max: 4.90, Min: 0.40, Time: 25.28\n",
      "Episode 49, Average Score: 1.11, Max: 4.52, Min: 0.37, Time: 25.92\n",
      "Episode 50, Average Score: 1.14, Max: 5.95, Min: 0.89, Time: 26.21\n",
      "Episode 50\tAverage Score: 1.14\n",
      "Episode 51, Average Score: 1.17, Max: 6.41, Min: 1.40, Time: 25.47\n",
      "Episode 52, Average Score: 1.21, Max: 4.98, Min: 0.93, Time: 25.35\n",
      "Episode 53, Average Score: 1.24, Max: 5.04, Min: 2.11, Time: 25.30\n",
      "Episode 54, Average Score: 1.26, Max: 4.33, Min: 0.67, Time: 25.47\n",
      "Episode 55, Average Score: 1.30, Max: 5.66, Min: 1.80, Time: 25.29\n",
      "Episode 56, Average Score: 1.35, Max: 5.99, Min: 1.80, Time: 25.40\n",
      "Episode 57, Average Score: 1.40, Max: 6.97, Min: 2.94, Time: 26.98\n",
      "Episode 58, Average Score: 1.45, Max: 6.84, Min: 3.00, Time: 26.32\n",
      "Episode 59, Average Score: 1.50, Max: 6.38, Min: 3.20, Time: 26.11\n",
      "Episode 60, Average Score: 1.55, Max: 7.02, Min: 1.15, Time: 26.07\n",
      "Episode 60\tAverage Score: 1.55\n",
      "Episode 61, Average Score: 1.60, Max: 7.16, Min: 2.43, Time: 25.59\n",
      "Episode 62, Average Score: 1.66, Max: 11.04, Min: 2.74, Time: 25.85\n",
      "Episode 63, Average Score: 1.69, Max: 6.25, Min: 1.49, Time: 26.19\n",
      "Episode 64, Average Score: 1.73, Max: 6.64, Min: 1.52, Time: 26.50\n",
      "Episode 65, Average Score: 1.78, Max: 7.99, Min: 2.82, Time: 28.75\n",
      "Episode 66, Average Score: 1.83, Max: 8.53, Min: 1.91, Time: 28.09\n",
      "Episode 67, Average Score: 1.87, Max: 7.22, Min: 1.81, Time: 27.27\n",
      "Episode 68, Average Score: 1.92, Max: 7.85, Min: 3.17, Time: 28.38\n",
      "Episode 69, Average Score: 1.95, Max: 9.18, Min: 1.77, Time: 28.96\n",
      "Episode 70, Average Score: 1.99, Max: 7.26, Min: 0.66, Time: 27.21\n",
      "Episode 70\tAverage Score: 1.99\n",
      "Episode 71, Average Score: 2.03, Max: 11.86, Min: 2.61, Time: 27.68\n",
      "Episode 72, Average Score: 2.09, Max: 10.86, Min: 2.89, Time: 26.64\n",
      "Episode 73, Average Score: 2.15, Max: 9.18, Min: 1.93, Time: 26.77\n",
      "Episode 74, Average Score: 2.19, Max: 8.67, Min: 2.96, Time: 28.70\n",
      "Episode 75, Average Score: 2.25, Max: 9.52, Min: 3.35, Time: 31.60\n",
      "Episode 76, Average Score: 2.29, Max: 10.54, Min: 2.86, Time: 31.06\n",
      "Episode 77, Average Score: 2.34, Max: 12.40, Min: 2.85, Time: 31.61\n",
      "Episode 78, Average Score: 2.40, Max: 12.72, Min: 3.39, Time: 28.52\n",
      "Episode 79, Average Score: 2.45, Max: 12.43, Min: 3.81, Time: 28.98\n",
      "Episode 80, Average Score: 2.52, Max: 18.08, Min: 4.13, Time: 27.77\n",
      "Episode 80\tAverage Score: 2.52\n",
      "Episode 81, Average Score: 2.57, Max: 11.78, Min: 3.06, Time: 29.56\n",
      "Episode 82, Average Score: 2.64, Max: 16.63, Min: 3.93, Time: 30.48\n",
      "Episode 83, Average Score: 2.69, Max: 10.96, Min: 4.75, Time: 31.93\n",
      "Episode 84, Average Score: 2.75, Max: 11.19, Min: 2.10, Time: 31.26\n",
      "Episode 85, Average Score: 2.80, Max: 10.74, Min: 2.78, Time: 31.25\n",
      "Episode 86, Average Score: 2.86, Max: 12.43, Min: 5.10, Time: 35.00\n",
      "Episode 87, Average Score: 2.91, Max: 11.83, Min: 2.06, Time: 32.96\n",
      "Episode 88, Average Score: 2.97, Max: 12.58, Min: 5.00, Time: 29.82\n",
      "Episode 89, Average Score: 3.03, Max: 25.26, Min: 2.05, Time: 30.65\n",
      "Episode 90, Average Score: 3.09, Max: 16.05, Min: 4.03, Time: 35.18\n",
      "Episode 90\tAverage Score: 3.09\n",
      "Episode 91, Average Score: 3.15, Max: 15.87, Min: 4.99, Time: 33.95\n",
      "Episode 92, Average Score: 3.20, Max: 13.89, Min: 5.20, Time: 33.18\n",
      "Episode 93, Average Score: 3.27, Max: 18.12, Min: 5.43, Time: 32.39\n",
      "Episode 94, Average Score: 3.32, Max: 13.19, Min: 4.30, Time: 33.18\n",
      "Episode 95, Average Score: 3.38, Max: 12.68, Min: 5.44, Time: 30.62\n",
      "Episode 96, Average Score: 3.43, Max: 17.05, Min: 4.52, Time: 34.42\n",
      "Episode 97, Average Score: 3.50, Max: 15.74, Min: 6.02, Time: 32.89\n",
      "Episode 98, Average Score: 3.57, Max: 15.87, Min: 5.89, Time: 35.12\n",
      "Episode 99, Average Score: 3.63, Max: 17.47, Min: 7.17, Time: 33.43\n",
      "Episode 100, Average Score: 3.69, Max: 13.94, Min: 3.92, Time: 34.16\n",
      "Episode 100\tAverage Score: 3.69\n",
      "Episode 101, Average Score: 3.80, Max: 13.19, Min: 5.57, Time: 37.44\n",
      "Episode 102, Average Score: 3.91, Max: 14.99, Min: 5.43, Time: 25.68\n",
      "Episode 103, Average Score: 4.03, Max: 19.71, Min: 5.60, Time: 25.85\n",
      "Episode 104, Average Score: 4.16, Max: 21.36, Min: 8.30, Time: 27.31\n",
      "Episode 105, Average Score: 4.29, Max: 18.96, Min: 8.90, Time: 26.49\n",
      "Episode 106, Average Score: 4.43, Max: 18.53, Min: 8.61, Time: 26.72\n",
      "Episode 107, Average Score: 4.55, Max: 19.65, Min: 6.55, Time: 28.48\n",
      "Episode 108, Average Score: 4.68, Max: 22.94, Min: 8.21, Time: 27.95\n",
      "Episode 109, Average Score: 4.81, Max: 18.67, Min: 9.39, Time: 26.46\n",
      "Episode 110, Average Score: 4.93, Max: 19.39, Min: 5.33, Time: 26.24\n",
      "Episode 110\tAverage Score: 4.93\n",
      "Episode 111, Average Score: 5.06, Max: 21.19, Min: 8.90, Time: 26.10\n",
      "Episode 112, Average Score: 5.19, Max: 20.53, Min: 8.01, Time: 26.21\n",
      "Episode 113, Average Score: 5.32, Max: 20.06, Min: 7.20, Time: 26.57\n",
      "Episode 114, Average Score: 5.47, Max: 20.05, Min: 11.25, Time: 26.90\n",
      "Episode 115, Average Score: 5.60, Max: 19.90, Min: 6.42, Time: 26.95\n",
      "Episode 116, Average Score: 5.74, Max: 21.69, Min: 3.56, Time: 28.57\n",
      "Episode 117, Average Score: 5.88, Max: 19.76, Min: 8.32, Time: 27.05\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 118, Average Score: 6.04, Max: 23.16, Min: 8.87, Time: 26.93\n",
      "Episode 119, Average Score: 6.19, Max: 25.07, Min: 6.20, Time: 27.31\n",
      "Episode 120, Average Score: 6.33, Max: 21.98, Min: 7.73, Time: 27.32\n",
      "Episode 120\tAverage Score: 6.33\n",
      "Episode 121, Average Score: 6.49, Max: 25.60, Min: 7.86, Time: 26.52\n",
      "Episode 122, Average Score: 6.67, Max: 36.91, Min: 6.63, Time: 25.92\n",
      "Episode 123, Average Score: 6.86, Max: 24.46, Min: 9.71, Time: 25.94\n",
      "Episode 124, Average Score: 7.06, Max: 28.92, Min: 9.04, Time: 25.93\n",
      "Episode 125, Average Score: 7.24, Max: 26.74, Min: 5.42, Time: 26.08\n",
      "Episode 126, Average Score: 7.43, Max: 28.69, Min: 6.04, Time: 26.02\n",
      "Episode 127, Average Score: 7.64, Max: 29.16, Min: 13.60, Time: 26.05\n",
      "Episode 128, Average Score: 7.83, Max: 29.91, Min: 11.80, Time: 26.07\n",
      "Episode 129, Average Score: 8.04, Max: 35.12, Min: 14.10, Time: 26.38\n",
      "Episode 130, Average Score: 8.31, Max: 39.01, Min: 17.90, Time: 26.70\n",
      "Episode 130\tAverage Score: 8.31\n",
      "Episode 131, Average Score: 8.58, Max: 36.30, Min: 23.87, Time: 26.83\n",
      "Episode 132, Average Score: 8.83, Max: 36.80, Min: 18.11, Time: 29.19\n",
      "Episode 133, Average Score: 9.11, Max: 35.08, Min: 17.79, Time: 28.29\n",
      "Episode 134, Average Score: 9.40, Max: 38.06, Min: 19.47, Time: 29.03\n",
      "Episode 135, Average Score: 9.70, Max: 36.55, Min: 19.35, Time: 30.30\n",
      "Episode 136, Average Score: 10.03, Max: 39.15, Min: 21.46, Time: 29.23\n",
      "Episode 137, Average Score: 10.35, Max: 37.98, Min: 23.37, Time: 31.21\n",
      "Episode 138, Average Score: 10.66, Max: 39.49, Min: 24.46, Time: 35.53\n",
      "Episode 139, Average Score: 10.98, Max: 38.59, Min: 24.84, Time: 35.81\n",
      "Episode 140, Average Score: 11.30, Max: 38.14, Min: 28.59, Time: 32.75\n",
      "Episode 140\tAverage Score: 11.30\n",
      "Episode 141, Average Score: 11.61, Max: 39.43, Min: 24.94, Time: 32.72\n",
      "Episode 142, Average Score: 11.90, Max: 37.03, Min: 27.37, Time: 34.01\n",
      "Episode 143, Average Score: 12.22, Max: 39.39, Min: 27.75, Time: 33.83\n",
      "Episode 144, Average Score: 12.55, Max: 38.90, Min: 27.42, Time: 32.19\n",
      "Episode 145, Average Score: 12.88, Max: 38.57, Min: 29.49, Time: 30.14\n",
      "Episode 146, Average Score: 13.21, Max: 38.30, Min: 29.85, Time: 29.98\n",
      "Episode 147, Average Score: 13.53, Max: 38.80, Min: 30.66, Time: 32.37\n",
      "Episode 148, Average Score: 13.84, Max: 38.03, Min: 22.08, Time: 34.21\n",
      "Episode 149, Average Score: 14.16, Max: 38.17, Min: 26.26, Time: 35.42\n",
      "Episode 150, Average Score: 14.46, Max: 37.55, Min: 26.58, Time: 30.23\n",
      "Episode 150\tAverage Score: 14.46\n",
      "Episode 151, Average Score: 14.77, Max: 39.04, Min: 24.18, Time: 27.60\n",
      "Episode 152, Average Score: 15.05, Max: 38.58, Min: 19.32, Time: 26.32\n",
      "Episode 153, Average Score: 15.34, Max: 38.87, Min: 16.94, Time: 26.05\n",
      "Episode 154, Average Score: 15.65, Max: 39.39, Min: 28.33, Time: 26.01\n",
      "Episode 155, Average Score: 15.92, Max: 38.56, Min: 19.93, Time: 26.45\n",
      "Episode 156, Average Score: 16.22, Max: 39.34, Min: 22.33, Time: 26.58\n",
      "Episode 157, Average Score: 16.52, Max: 39.08, Min: 27.93, Time: 25.93\n",
      "Episode 158, Average Score: 16.83, Max: 39.33, Min: 29.59, Time: 27.31\n",
      "Episode 159, Average Score: 17.14, Max: 39.33, Min: 30.98, Time: 26.40\n",
      "Episode 160, Average Score: 17.43, Max: 37.82, Min: 28.92, Time: 27.20\n",
      "Episode 160\tAverage Score: 17.43\n",
      "Episode 161, Average Score: 17.73, Max: 37.97, Min: 25.32, Time: 27.80\n",
      "Episode 162, Average Score: 18.04, Max: 39.48, Min: 29.23, Time: 26.02\n",
      "Episode 163, Average Score: 18.36, Max: 38.80, Min: 32.84, Time: 26.67\n",
      "Episode 164, Average Score: 18.69, Max: 39.46, Min: 33.87, Time: 26.21\n",
      "Episode 165, Average Score: 19.02, Max: 38.94, Min: 33.50, Time: 26.03\n",
      "Episode 166, Average Score: 19.34, Max: 39.52, Min: 32.15, Time: 26.96\n",
      "Episode 167, Average Score: 19.65, Max: 38.32, Min: 32.18, Time: 26.77\n",
      "Episode 168, Average Score: 19.96, Max: 38.83, Min: 33.31, Time: 26.97\n",
      "Episode 169, Average Score: 20.28, Max: 38.65, Min: 34.33, Time: 26.82\n",
      "Episode 170, Average Score: 20.61, Max: 39.56, Min: 35.20, Time: 26.40\n",
      "Episode 170\tAverage Score: 20.61\n",
      "Episode 171, Average Score: 20.92, Max: 38.52, Min: 34.15, Time: 28.15\n",
      "Episode 172, Average Score: 21.22, Max: 39.10, Min: 31.20, Time: 26.84\n",
      "Episode 173, Average Score: 21.53, Max: 38.99, Min: 33.56, Time: 26.97\n",
      "Episode 174, Average Score: 21.84, Max: 39.59, Min: 31.21, Time: 27.59\n",
      "Episode 175, Average Score: 22.16, Max: 39.53, Min: 33.90, Time: 27.29\n",
      "Episode 176, Average Score: 22.48, Max: 39.61, Min: 32.61, Time: 27.32\n",
      "Episode 177, Average Score: 22.80, Max: 39.32, Min: 33.58, Time: 26.45\n",
      "Episode 178, Average Score: 23.09, Max: 38.80, Min: 30.71, Time: 27.11\n",
      "Episode 179, Average Score: 23.42, Max: 39.63, Min: 36.62, Time: 27.46\n",
      "Episode 180, Average Score: 23.72, Max: 39.34, Min: 34.59, Time: 27.82\n",
      "Episode 180\tAverage Score: 23.72\n",
      "Episode 181, Average Score: 24.02, Max: 39.51, Min: 35.09, Time: 29.74\n",
      "Episode 182, Average Score: 24.33, Max: 39.36, Min: 36.74, Time: 28.29\n",
      "Episode 183, Average Score: 24.63, Max: 39.65, Min: 33.70, Time: 28.21\n",
      "Episode 184, Average Score: 24.93, Max: 39.50, Min: 35.22, Time: 29.25\n",
      "Episode 185, Average Score: 25.25, Max: 39.46, Min: 35.14, Time: 26.49\n",
      "Episode 186, Average Score: 25.55, Max: 39.54, Min: 34.71, Time: 29.21\n",
      "Episode 187, Average Score: 25.86, Max: 39.60, Min: 37.20, Time: 27.73\n",
      "Episode 188, Average Score: 26.16, Max: 39.43, Min: 36.78, Time: 27.89\n",
      "Episode 189, Average Score: 26.45, Max: 39.08, Min: 34.76, Time: 29.87\n",
      "Episode 190, Average Score: 26.75, Max: 39.59, Min: 35.14, Time: 28.29\n",
      "Episode 190\tAverage Score: 26.75\n",
      "Episode 191, Average Score: 27.05, Max: 39.58, Min: 37.21, Time: 28.06\n",
      "Episode 192, Average Score: 27.35, Max: 39.65, Min: 35.95, Time: 28.30\n",
      "Episode 193, Average Score: 27.63, Max: 39.35, Min: 34.43, Time: 27.84\n",
      "Episode 194, Average Score: 27.93, Max: 39.20, Min: 34.10, Time: 28.88\n",
      "Episode 195, Average Score: 28.22, Max: 39.64, Min: 34.52, Time: 29.24\n",
      "Episode 196, Average Score: 28.49, Max: 39.45, Min: 25.45, Time: 31.03\n",
      "Episode 197, Average Score: 28.76, Max: 39.07, Min: 35.14, Time: 30.31\n",
      "Episode 198, Average Score: 29.05, Max: 39.64, Min: 36.98, Time: 29.72\n",
      "Episode 199, Average Score: 29.32, Max: 39.04, Min: 35.28, Time: 29.59\n",
      "Episode 200, Average Score: 29.60, Max: 39.37, Min: 37.10, Time: 28.68\n",
      "Episode 200\tAverage Score: 29.60\n",
      "Episode 201, Average Score: 29.88, Max: 39.56, Min: 34.20, Time: 27.43\n",
      "Episode 202, Average Score: 30.13, Max: 39.07, Min: 32.70, Time: 27.06\n",
      "Episode 203, Average Score: 30.38, Max: 39.09, Min: 31.02, Time: 26.78\n",
      "Episode 204, Average Score: 30.63, Max: 39.53, Min: 34.55, Time: 27.48\n",
      "Episode 205, Average Score: 30.88, Max: 39.52, Min: 36.30, Time: 26.61\n",
      "Episode 206, Average Score: 31.12, Max: 39.59, Min: 35.48, Time: 27.11\n",
      "Episode 207, Average Score: 31.39, Max: 39.57, Min: 36.37, Time: 27.25\n",
      "Episode 208, Average Score: 31.62, Max: 39.21, Min: 35.11, Time: 27.11\n",
      "Episode 209, Average Score: 31.88, Max: 39.62, Min: 36.19, Time: 28.03\n",
      "Episode 210, Average Score: 32.13, Max: 39.39, Min: 36.18, Time: 27.75\n",
      "Episode 210\tAverage Score: 32.13\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8XHW9//HXZyb7vqdp0jRpm+5L2oZSoOw7IhQFARHwulSu4HIV/QHqVe/1/q7XH4pXRaQCiggFZBdEraVQ2UrTfW+attmbTPY9meX7++OchKQkTRqYTDLzeT4eeWTmzJk535xO5z3f5Xy/YoxBKaVU6HIEugBKKaUCS4NAKaVCnAaBUkqFOA0CpZQKcRoESikV4jQIlFIqxGkQKKVUiNMgUEqpEKdBoJRSIS7M3wcQESdQDFQZY64UkXzgSSAF2AbcbIzpPdlrpKWlmby8PH8XVSmlgsrWrVvrjTHpI+3n9yAAvgbsBxLs+/8D3GeMeVJEfgN8HnjgZC+Ql5dHcXGxf0uplFJBRkTKRrOfX5uGRCQH+BjwkH1fgAuAZ+xdHgVW+7MMSimlTs7ffQQ/B74N+Oz7qUCzMcZj368Esv1cBqWUUifhtyAQkSuBOmPM1oGbh9h1yOlPRWSNiBSLSLHL5fJLGZVSSvm3RnAWcJWIHMPqHL4Aq4aQJCJ9fRM5QPVQTzbGrDXGFBljitLTR+zrUEopNUZ+CwJjzN3GmBxjTB5wA/CaMeYmYCNwrb3brcCL/iqDUkqpkQXiOoL/A3xDRA5j9Rk8HIAyKKWUso3H8FGMMa8Dr9u3jwArxuO4SimlRqZXFiul1Am2ljXxdml9wI5f1tDB7sqWcTueBoFSKuh19Xr5297jjGaN9tZuN2v+UMyXHttKa7d7TMera+umorETAGMM3W4vXt/Jj13d3EXxsUYAvvfiXj714DsccbWP6finSoNAKRX0XtpZxZce28rf9h4fcd9fbyyloaOXtm4Pj70z+MLcP++s5uniChrae/q39Xp8fOOpHYO+wX/9yR186TFr5PztT2xj7vf+ypIf/p3a1u5Br9fW7WZ7eROt3W4+/dt3ueWR9+h2e9le1kSX28u/Pb0Tt9eHv2kQKKWC3hFXBwD3rS/B5zMYY/jvV/fz3tHG/n3aut38ckMJj7x5lE8sy+b8Oek89M8jdPZa17++dbier6zbzref2cXF923qry28ccjFc9ur2FRiXe/U0ulm89FGDta20dXrZdOhevJSY2jv8bBrQFh09nr4zMPvcc2v3+b8//c6xxo66ez18ty2Ktp6PFw0L5OdFc38dc/I4fVhaRAopYJKqaudjh7PoG3HGjpwOoSDtW28sruGA8fbePCNI9zxxDZaOt109nr49G8389P1h1hVkMY9V8zjX8+bRVOnm7/sPk5zZy/ffHonM9Nj+dmnltDY0dvfjPPCjioAmjqsuTM3lbjw+gxen2HDgVraezzcuCIXYFBTz9ee3MHuymY+d1Y+0RFO7rxkNiLw4KZSAO6+Yi5Pf+kMrlyc5fdzNi6jhpRSajzsqWrhyl++SZhDyEuLJTspmp9fX0hZQyfnzk6nrKGDtZuOcP7cDESgoaOX2/64FRHYU93C2puXc8mCKQCkxkaQnRTNy7uqOVbfQW1bNy/dsoqCzDjuenY3m482clpeCv/YVwtAU6dVQ3jtQB3hTsHtNTy7tRKAM2amkhYXSakdBMdbulm/r5avXljANy6ezb9/fD4Ar+45zt7qVhKjw8lPjWVmety4nDetESilgsY/9tciAp9blU9OcjRvHHLxxiEXxxo6mJEWy2fPymd3VQuPvn2M0/JSuPvyuWyvaKK4rIkffHxBfwgAiAhXLs7izZJ6Hnu3jIvnZbIoJ5GocCdLpiXy3tFGXt19nB6Pj6hwB82dvXh9htcP1nH5wiyiw528cciF0yHMzoxnRnospXYTVd+IpEsXZA4q/5kzUwEonJaEwzHUjDz+oUGglJpw9lS1sL+mdcjHSmrb+OWGkiFHAL1+0MWSnCTuuWIeD91SRHS4k7/uOU6328f0tFg+sTSb+KgwWrrcXLZgCl84ewYH/vNyDv3ocm49M+8Dr3fl4ql4fIaWLjdfPGdG//YV+SnsrmzhF6+VMHdKPMtyk2nq7OWIq52mTjfnzk5nzpR4fAZmpccRFe5kZnpcf9PQW4cbSI4JZ96UhEHHO3NmGmAFwXjSIFBKTSh7q1u49jdv88U/FA/5Yf/AG6X8dP0hjtZ3DNre1NHLzspmzp1tzU0W5nSwOCeR1w7UAZCXGkNsZBjXF03DIXDpwikfeO0TLcxOYEZ6LEtzkyianty/fUV+Kh6fobKpi7uvmEdybATNnW5qW63RRNnJ0czLsj7kF0y1fs9Mj6Wp001jRy/vlNZzxszUD3zrP2NmKqsLp3J14dTRnq6PhAaBUmrCsMbwb8Vrf8huLWsa9LjH62Oj/cE+cMQPwD8P12MMnDvn/Ukql01PptcefpmXGgvAnZfO4YXbzyI7KXrE8ogI6764kodvPQ1rORXL8unJhDmEc2anc+7sdJJjwmnq7MXVbg0PzYiPZH5WPADz+4PAau/fsL+W6pZuzrC//Q8UFe7k5zcsZcY49Q300SBQSk0YL+2opqq5i7V2s85z26sGPb61rKm/U3ZgEOyoaOYXG0pIiglnSc77zSpL7SaWcKeQlRgFWB+2i3NG3/SSmRBFSmzEoG1xkWE88cWV3PepJQAkx0TQ0vV+jSA9PpIV+amEO4WVM6x2/74g+OVrhwE4y+4PmAg0CJRSE8YL26uYnRnHebPTuWRBJq/sqqHH4+1//B/7a4lwOjh3djqb7SA4Vt/Bdb95m7ZuNz+/vhDngOaWwlzrA39aSgxhzo/2425FfgqpcZEAJEaH4zNQWtdOVLiDuMgw5kyJZ88PL2VhdiJgNRdFhDkob+zki2fnj/u3/pPRIFBKTQjlDZ0UlzWxemk2IsI1S7Np6XLz973W8ExXWw8v76ph5cxULpibQVVzF5VNnfz+7WMAvHTHKs6bkzHoNTPio5ieGuP3YZjJMVaN4VBdOxnxUf3NSJFhzv59nA7h6iVTue3cmdxzxTy/ludU6XUESqkJ4ZmtFQBcXWitXnt2QTrTUqL547tlLM1N4qaHNtPc6eaO82cRH2V9dK17r5xntlby8cVTyUyIGvJ1H7qliOgI55CPfVSSY8MBa0RTXyfxUP7fdUv8Wo6x0iBQSgXc3/Ye5/7XS7l4fmZ/J67TIdx0+nR+/OoBrn/wXdq63Tz+xdNZlpuMz2dYkZfC/Rutq3D/5az8YV+7IDPe7+VPsmsEnb1e0u3moslEg0ApFVBVzV18Zd12FmUnct/1hYMe+1TRNH62/hCuth4e+/wKluVaQzgdDmHdmpU8XVxBfVsPi3ISA1H0fn1NQwAZCRoESil1Sp7aUoHb6+OXNy4lLnLwR1JKbAS/uKGQxOgITp8xeJSN0yH9c/gEWnJMeP9trREMICJRwCYg0j7OM8aY74vI74Fzgb5p+D5rjNnhr3IopSaOP++sZlt5E1MSovjSuTPx+gx/Kq6w+wNihnzOZQv9P+nah5UQFY5DwGesoaOTjT9rBD3ABcaYdhEJB94UkVftx75ljHnGj8dWSn1IHq+PrWVNH/gmPlZur49vPL0Dj89gDNywIpdtZU3UtHTz71fO/0iOESgOh5AYHU5Tp3tSBoHfho8aS9+cq+H2z8jLAymlJoQXdlRz/dp32Vc99Jw/p6qsoQO313DhXGuitermLv6+r5b4qDAunJc5wrMnvr5+goz4oUcvTWR+vY5ARJwisgOoA9YbYzbbD/2XiOwSkftEZPLFp1Ih4J3SBsCa++dofQf//Zf9Iy63eDIltdb3wr4pIKqbuyhv7GBmehwRYZP/kqZEu59AawQnMMZ4jTGFQA6wQkQWAncDc4HTgBTg/wz1XBFZIyLFIlLscrn8WUyl1BDeO2YFwcHjbTy5pZwHNx3pn/vH4/XxlXXb2VbedLKXGKSkrh0ROKfAmmOnqrmL8sZOcofpG5hskmMiEIHUuIiRd55gxiWGjTHNwOvAZcaYGrvZqAf4HbBimOesNcYUGWOK0tPTh9pFKeUn1c1dVDR2AXCwto3t5c0AbDhgXeVb1tjJn3dWj2oN4D4lde3kJEczLTmGCKeD8oZOqpu7gyYI0uMiSYuLJPwjnspiPPitxCKSLiJJ9u1o4CLggIhk2dsEWA3s8VcZlFJjs8VehnHulHj217Syq9IOgv3WzJ99awCX1XeO+jVLatuYlR6HwyFkJUWxpawJr88ETRB85cJZ/OYzywNdjDHxZ3RlARtFZBewBauP4GXgcRHZDewG0oAf+bEMSqkx2Hy0kfjIMFYvzaa+vZdut49luUkcrmunrKGjf8nFYw0dI7wS+Oz1e4/Ud/Rf5ZudFM2eKmsE+XDDRiebnOQYlg9Ys2Ay8dvwUWPMLmDpENsv8NcxlVIfjZ0VzRTmJg2aN+fOS+bw6Yc2s2F/Xf9KW8caOjDGsK28icU5Sf3NIi1dbhwC8VHh3PzIZrp6vfR6fMzKsCZ/m5oU3d/xnJsaHEEwmU2+xiyl1Jite6+cw3VtI+5X1dzF9NQY5k6xvsGnx0dyxsxUZqTFsqnE1b/2brfbxxuHXHzygXf4y+6a/uff8cQ2vv3MLgD2Vreyze5jKLCDoG8+oXCnMGWYyeLU+NEgUCpEtHW7ufu53dz++Hbc9qpdYF3oVXyssf8bemevh+ZON1mJ0WTER5ISG8Gy3CREhFUFaWw+0sjhunampVgf5o9vLgegsqmr/zX317RyrKETj9dHS5ebRdmJLMlJZK69Rm9fEOQkxwxaP0AFhgaBUiHicJ3VnHOwto1H7Tn8Nx1yce5PNnLtb97h6WJrGujqZmu5xeykaESEh24t4rsfs678XTUrjS63l5YuNxfYc//3rQnsarNW5+ro8VDf3ktjRw8tXW6MgWuX5/DiHav6p4POTraCIFj6ByY7DQKlQkSJHQTzsxK4b/0halu7+c4Lu4kMd5KdFM1LO6oBqGmxvtn3Le24LDe5/wN75czU/m/wqwrSCXdKf02irs0KkIomayRRY0cvjR29ACSfsNTjVLtGkJsy8rrByv80CJQKESW1bUSGOfjFjYV0e3x85qHNVDR2cdflc/nk8hw2H22grq2b6mYrCKYOsbh7QlQ4hfY6wLMz4/oDQgTq7PV6yxqsIHB7DeWN1u2UmBODIIrU2IhB6wurwNEgUCpElNS1MzM9jlkZ8dxw2jRK6tqZkRbLxfMy+fjiLHwGXt19nOrmbkQYdsWvyxdOIT0+kpzkGPJSYwFYkZeCq90KgorG968t6GuO6lvBq09kmJN37r6Qa5fn+ONPVadI1yNQKkSU1LZTlGeNc//aRQW8dqCOr11UgMMhFGTGMycznld215CXGkN6XOSw8/987qx8bj5jOk6HcP7cDBwCeamx7KpswZj3awHwfhCkxH5w2oVgmF8oWOi/hFIhoL3HQ1VzV//wzYz4KN65+8L+9YEBLpqfwdayJg7Wtg/ZLNTH4ZD+RdlvXjmdh249jYyESLrcXtp7PJQ1dBJhX0/Q1y+RHDP55t8JJRoESoWAvm/mszKGX7/37IJ0vD7Dzopmpiad2tj+vqmX69p6qGjsZN5Ua5hoaV07MRFOosL9u3i8+nA0CJQKAYdqrYvIZmfGDbvPstxkYuzhnVMTT200T9/Uy7Ut3VQ0dbLU7lBu6/EM2SykJhYNAqVCwPbyZuIjw5hud+4OJSLMwRn2amRZJ2kaGkqGHQS7qlpwew1zpsQTbdcCNAgmPg0CpULA5qMNFOUlj3gV79n2WgHZY2waWr/PmqZ6ZnpcfwBo/8DEp0GgVJCra+vmiKtjVGsPf2zxVC5dkMlpeSmndIyE6DAiwhxsLWsiIz6S5dOT+xdo0RrBxKdBoFSQe++otbbAylEEQXp8JA/eXERq3Kkttygi/c1DVxdOxekQrRFMIhoESgW5d480EBvhZOHUhJF3/hD6guCapdZFYn1BkHLCxWRq4tELypQKcu8dbaQoL4UwPy+hOMeeWXS+HTipfTUCbRqa8PwWBCISBWwCIu3jPGOM+b6I5ANPYi1cvw242RjT669yKBXKej0+Sl0dXDJ/it+P9V+rF+KxJ6ADSIm1aggnzjOkJh5/fkXoAS4wxiwBCoHLRGQl8D/AfcaYAqAJ+Lwfy6BUSCtv7MDrM8zMGH7Y6EfF4ZBB00ZojWDy8FsQGEu7fTfc/jHABcAz9vZHsRawV0r5Qd9KYjPShr+QzF8WT0skNyWmf3lKNXH5tdFQRJwisgOoA9YDpUCzMcZj71IJZA/3fKXUh9O3yPyMdP/XCE40d0oCm759PmmnOAJJjT+/BoExxmuMKQRygBXAvKF2G+q5IrJGRIpFpNjlcvmzmEoFrSOuDjLiI4mP0pE7anjjMnzUGNMMvA6sBJJEpK+TOgeoHuY5a40xRcaYovT09PEoplJBp9RlrUGg1Mn4LQhEJF1Ekuzb0cBFwH5gI3CtvdutwIv+KoNSocwYwxFXR0CahdTk4s/rCLKAR0XEiRU4TxtjXhaRfcCTIvIjYDvwsB/LoFTIaujopaXLrTUCNSK/BYExZhewdIjtR7D6C5RSfnSkb8SQ1gjUCHSKCaWC1BF7xJDWCNRINAiUClKlrnYiwxxkn+LaAir0aBAoFSSMMXi8vv77R1wd5KfF4hhhDQKlNAiUChLffmYXhf+xnruf202326tDR9WoaRAoFSSO1HfgM4Z175Xz8q4aKpq6mKkdxWoUNAiUChIdPR7OnJlKQlQY694rx+szzNAagRoFDQKlgkRbt4eE6HBWzkhla1kToCOG1OhoECgVJDp6PcRFhnHmzPeXpMzXpiE1ChoESgUBYwzt3VYQnDUrDYApCVHEReoihGpkGgRKBYEejw+PzxAbGcasjDgy4iPHZTEaFRz064JSQaC9x1riIz4qDBHh/puWERuh/73V6Og7Rakg0N5tBUFfU9BpeSmBLI6aZLRpSKkg0FcjiNU+ATUGGgRKBYH+piENAjUGGgRKBYEOrRGoD0GDQKkg0FcjiIvSIFCnToNAqSDQHwRaI1Bj4M81i6eJyEYR2S8ie0Xka/b2H4hIlYjssH+u8FcZlAoVJ44aUupU+PNd4wG+aYzZJiLxwFYRWW8/dp8x5l4/HlupkNLe40EEYiKcgS6KmoT8uWZxDVBj324Tkf1Atr+Op1Qoa+/xEBdhXUym1Kkalz4CEcnDWsh+s73pDhHZJSKPiEjyeJRBqWDW3u3RjmI1Zn4PAhGJA54Fvm6MaQUeAGYChVg1hp8O87w1IlIsIsUul8vfxVRqUuvo9ejQUTVmfg0CEQnHCoHHjTHPARhjao0xXmOMD/gtsGKo5xpj1hpjiowxRenp6f4splKTXps986hSY+HPUUMCPAzsN8b8bMD2rAG7XQPs8VcZlAoVHT0aBGrs/PnOOQu4GdgtIjvsbfcAN4pIIWCAY8CX/FgGpUJCe4+HjPioQBdDTVL+HDX0JjDUEIa/+OuYSoWqjh6v9hGoMdMri5UKAm3dbuJ11JAaIw0CpSY5YwztPR5iI/ViMjU2GgRKTXLdbh8+A3GR4YEuipqkNAiUmuTaetyAzjyqxk6DQKlJrqPHC0CcNg2pMdIgUGqSa+rsBSAxWpuG1NhoECg1yR1v6QZgSkJ0gEuiJisNAqUmqcqmTrw+Q40dBFmJekGZGhsNAqUmoebOXi649w2e21ZJbWs3kWEOkmK0aUiNjQ4zUGoSqm7uptfrY19NK/XtvWQlRulaBGrMNAiUmoQaOnoAOFbfQXuPh8wEbRZSYzfqpiERWSUi/2LfTheRfP8VSyl1MvXtdhA0dFLT0q39A+pDGVWNQES+DxQBc4DfAeHAH7FmGFVKjbOGdmvIaEVjJyIwJVFHDKmxG22N4BrgKqADwBhTDcT7q1BKqcG6er3sqWrpv++yawQen8HtNVojUB/KaIOg1xhjsNYQQERi/VckpdSJHn3nGKvvf4uWTms6ib4aQZ8pGgTqQxhtEDwtIg8CSSLyReAfWMtMKqXGwf6aVjw+w2FXO2D1EUwd8OGvNQL1YYwqCIwx9wLPYK0/PAf4d2PML0/2HBGZJiIbRWS/iOwVka/Z21NEZL2IlNi/kz/sH6FUsDtcZwVAad37QTBnSnz/8pRTdNSQ+hBGDAIRcYrIP4wx640x3zLG3GmMWT+K1/YA3zTGzANWAreLyHzgLmCDMaYA2GDfV0oNw+czHHF1APTXCBrae0mLiyQvLYYwh5AaFxnIIqpJbsQgMMZ4gU4RSTyVFzbG1Bhjttm324D9QDZwNfCovdujwOpTKrFSIaa6pYsutzXD6OG6dowxNLT3khoXyZzMBHJTYnA69GIyNXajvaCsG2sR+vXYI4cAjDFfHc2TRSQPWApsBjKNMTX282tEJONUCqxUqCm1awOZCZGUutpp7fbQ6/WRFhfBmnNm0N7tCXAJ1WQ32iB4xf45ZSISh9W38HVjTOtoL4MXkTXAGoDc3NyxHFqpoNDXP3DJ/Ck8vrmMqqYuANLiIkmJjSAlNiKQxVNBYLSdxY8C64Ct9s8T9raTEpFwrBB43BjznL25VkSy7MezgLphjrnWGFNkjClKT08fTTGVCkqH69pJiglnRX4KPgPFZY2AFQRKfRRGFQQich5QAtwP/Bo4JCLnjPAcAR4G9htjfjbgoZeAW+3btwIvnmKZlQoppa52ZqXHMSsjDoDNR+wgiNeagPpojLZp6KfAJcaYgwAiMhurhrD8JM85C7gZq29hh73tHuDHWNclfB4oB64bS8GVCgW9Hh8Halq5YlEW+WmxRIQ52HjQqkSnxmqNQH00RhsE4X0hAGCMOWQ3+wzLGPMmMFyHwIWjPK5SIe0f+2tp7fZw6YIpRIU7+a/VC/nWM7sQQfsG1EdmtEFQLCIPA4/Z92/C6itQSvnRU1sqyEqM4pzZVj/ZdUXTcDqEnRXNOmRUfWRGGwT/CtwOfBXrW/4mrL4CpZSfVDV3sanExVcuKBj0of+JZTl8YllOAEumgs1ogyAM+N++Tl8RcQLaQKmUH722vxZj4BNLswNdFBXkRjvp3AZg4ITn0VgTzyml/ORgbRvxUWFMT40JdFFUkBttEEQZY9r77ti39d2plB8dqm1nTma8rkWs/G60QdAhIsv67ohIEdDlnyIppYwxHKptoyBT139S/jfaPoKvA38SkWqsxWmmAtf7rVRKhThXWw/NnW7mZMYFuigqBJy0RiAip4nIFGPMFmAu8BTW9NJ/BY6OQ/mUCnpt3W6+/PhWalrer2QfrG0DYPYUrREo/xupaehBoG9NvDOwrgy+H2gC1vqxXEqFjO3lzfxl93E2HXL1bzt43A4CbRpS42CkpiGnMabRvn09sNYY8yzw7IBpI5RSH0KlPZtoWUNn/7aS2nZSYyN0Yjk1LkaqEThFpC8sLgReG/DYaPsXlFInUdlkBUBZ4/tBcKC2jQLtH1DjZKQgWAe8ISIvYo0S+ieAiMwCWvxcNqVCQl+NoNyuEbi91kRzC6ae0qKASo3ZSb/VG2P+S0Q2AFnA340xxn7IAXzF34VTKhT01wgarJXIDtW20ePxsThHg0CNjxGbd4wx7w6x7ZB/iqNU6Kls6kIEWrs9NHf2sqvSqmwvyUkKcMlUqBjtBWVKKT/odnupa+thwdQEwOow3lXZQoJOLaHGkQaBUuPoj++WUTGgU7i62eofOGtWGmB1GO+qbGZxTpJOLaHGjQaBUuOkpdPNd1/Ywx/eOda/ra+j+MyZVhCU1LZx8Hib9g+oceW3IBCRR0SkTkT2DNj2AxGpEpEd9s8V/jq+UhNNbVs3YE0m16cvCGZnxpEeH8kzWyvx+AyLtX9AjSN/1gh+D1w2xPb7jDGF9s9f/Hh8pSaU4y19QdDWv62yqZNwp5ARH0VBRhw1Ld2cNyedc2anBaqYKgT57aIwY8wmEcnz1+srNdnUtlpBUNPSTWu3m4SocA4cbyM7KRqnQ7j3uiW093h0Wgk17gLRR3CHiOyym46Sh9tJRNaISLGIFLtcruF2U2rS6AsCsKaQqGru4vWDdVyxKAuAqUnRGgIqIMY7CB4AZgKFQA3w0+F2NMasNcYUGWOK0tPTx6t8SvlNbWsPfQOBSmrbeGJzGQA3rZwewFIpNc5BYIypNcZ4jTE+4LfAivE8vlKBdLy1m1npcUSHO9le3sxTWyq4cF4m2UnRIz9ZKT8a14njRCTLGFNj370G2HOy/ZUKJnWt3UxJjCI6wslTxRU4HcJt584IdLGU8l8QiMg64DwgTUQqge8D54lIIdYqZ8eAL/nr+EpNBMYY3jrcwJwp8dS29jA7M57MhCh2VbbwH1cvYPn0lEAXUSm/jhq6cYjND/vreEpNNLWt3ax5bCs7K5r5+JKpuNp7yEyI4ppl2ZxdkMbVhdmBLqJSgK4poNSYtHa7ufu53eytamFFfgo/uXYJrrYeSl3tJMdEMGdKPE9tqbCni0jkb3uP4/UZMhOjmJkex8x0XWtATRw6xYRSp8gYw3ee38Nf9xzH4RBe3FGN2+vjy49v5Ya173Lpzzexu7KFHRXNzEqP41/PnUmvxwdAZryuOKYmHg0CpU7RSzur+fPOar5x8Wy+dmEBPR4fu6usD/6PLcpCBF47UMf28iaW5iaxqiCNcKc1bnRKYlSAS6/UB2kQKHWKXtxRzfTUGG47d2b/nEBPbC7H7TWsXprNouxEni6uoKnTTeG0ZOKjwjk9PxWAzAQNAjXxaBAodQqMMeyqbKZoegpOh5CXGkN8VBgv7awGYFluEmcXpFFlTy+9NNcKihtWTGNhdoIuRq8mJA0CpU5BVXMX9e29FE6zpokWERbnJNLr8ZGXGkNqXCRnF1hXwsdEOPunjLhy8VRe/srZOB26xoCaeDQIlDoFfctIDpwmelG2dXvZdGvqrGW5ycREOFmck6gf/GpS0OGjSp2CnRXNRDgdzM16f3K4vkVklttBEBHm4CfXLmaK9geoSUKDQKlTsKOimXlTE4gMc/ZvO2d2Op8+PZfLF2b1b7ty8dRAFE+pMdGmIaVGqbmzlz1VLRQf2j3BAAAVJUlEQVSesIxkXGQY//eaRaTERgSoZEp9OBoESo1Cc2cvNz20GbfXcFWhfttXwUWbhpQahd+8cYQDx9t46NYinShOBR2tESg1Cu+U1rM8N5nz52QEuihKfeQ0CJQaQVu3m91VLaycoTUBFZw0CJQaQXFZEz4Dp89IDXRRlPILDQKlRvDukQbCncKy3ORAF0Upv/BbEIjIIyJSJyJ7BmxLEZH1IlJi/9b/WWrCe/dII4XTkoiOcI68s1KTkD9rBL8HLjth213ABmNMAbDBvq/UhFHb2s2PXz1At9sLQHlDJ7sqm1k1Kz3AJVPKf/wWBMaYTUDjCZuvBh61bz8KrPbX8ZUai+88v4ffvFHKn7ZWAvD7t4/hFOGGFdMCXDKl/Ge8+wgyjTE1APZvHYunJoz1+2r5x/5aIsMc/O6to7R2u3m6uIIrF2fpOgIqqE3YzmIRWSMixSJS7HK5Al0cFQJ+/fphZqTH8p+rF3LE1cHq+9+ivcfDv5yVH+iiKeVX4x0EtSKSBWD/rhtuR2PMWmNMkTGmKD1d22fVR+f+jYe5Ye07g7b5fIb9Na2cNzuD1YXZTE2MoqXTzU+vW8KSaUnDvJJSwWG8p5h4CbgV+LH9+8VxPr5SvLqnhr3VrXS7vUSFWyOBKpo66Xb7mDMljogwBy9/9WwiwxzERuosLCr4+XP46DrgHWCOiFSKyOexAuBiESkBLrbvKzVuOno87K9pwxg41tDRv/3A8TYA5kxJACAlNkJDQIUMv73TjTE3DvPQhf46plIj2VnZjNdnACit62Cu/cF/yA6Cgoy4gJVNqUDRrzwqpGwra+q/fcTVzmPvlnG8pYuyhk5yU2K0FqBCkr7rVUjZWtZEQUYcHT0eSl3tbNnSRFVzF0kx4RTp9NIqRE3Y4aNKfVh7qlq4928H6XZ7cXt9vHHIxdayJpZPT2ZGehybSuqpau4CoLnTzZwp2iykQpPWCNSkU9/eQ0uXm5npw39wby9v4pZH3qOt28OB4600dPSyvbyZqHAHq5dm8+ruGt48XA/AVUum8tLO6v6OYqVCjQaBmnR+9PI+Nh50sfmeC/uHfw5U1tDBZ3+3hZTYCG49I49fbTxMbISTe69bwpWLs4gKd3LQ7hzOT7MuIMtMiOS8OXq9igpNGgRqUthR0cxbh+u5/fxZbK9opqXLzd/2Hufqwuz+fX71Wgn7alopqW0H4LHPnU5uagwLsxOZnRnHjAE1iL7axDkFaSRGh/Odj80f3z9IqQlEg0BNCn945xjPbavikvmZlDV0AvDUlor+IHB7fTzweim9Xh/GwCOfPY3c1BgALls45QOvtyg7kYKMOFYvzf7AY0qFGg0CNSnsr7GactZuOgLAirwU3i5t4BtP7+DMmWnkp8XS0evl1zctY1VBGglR4Sd9vcSYcNZ/41y/l1upyUBHDakJz+31cbjOCoIXdlQB8MOrF5CXGsP6vbV874U9/G3vcQDOmJE6YggopQbTIFATXqmrHbfXEB3uxO01ZCdFMy8rgde/dT7r1qyky+3lkTePMj8rgeTYiEAXV6lJR4NATVilrnZ+u+kI+6pbAfjkcqs9f1F2Yv8+C7MTWZqbhMdnOHOmLi6v1FhoH4GakNp7PHz+91s41tDJ7Mw4IpwObl6Zxx/fLWfxtMRB+956Rh7by3ewqiAtQKVVanLTIFABY4xBRIZ87Psv7qW8sZP0+EgO1bYzPyuBOVPieeiWIk6fMXgqiKsLpzI1KZrT8pLHo9hKBR1tGlIBUdHYyeIf/J3XDtQCVih84+kd3L/xMO09Hl7YUcUtZ+Rx12VzAZiXZV31e9H8TOJP6AwWEVbkpwwbKkqpk9MagQqINw/X09bj4Yd/3seqWen8fd9xnttWRUZ8JPOnJuD1GS6cl8EZM1J5/ZCLqwunBrrISgUtDQIVEFuONRLuFMoaOvnWMzvZfKSRyDAHdW09/O6tY4Q5hOXTkwlzOvjljUsDXVylgpo2DamAKD7WxAVzM7hmaTYv7qims9fDL+wP/E2HXCyZlkRMhH5PUWo8BOR/mogcA9oAL+AxxhQFohwqMOpauylv7OTmldP5wtn5/OTaxThEcDqEhdkJ7Klq5YwZOhRUqfESyBrB+caYQg2B4PfW4XqaOnr77xfbq4QV5SUjIoQ7HTgdVkfvebMzAFipQaDUuNGmITUmxhjW76vlgddLee9o46DtAz/03ylt4KaHNvOL10oGbYsMc7Bg6uDrAQBuWpnLmnNmsCJfVwtTarwEKggM8HcR2Soia4baQUTWiEixiBS7XK5xLp4aSo/Hy3+/up/Kpk7eLm3gi38o5n/+eoB7nt/dv8+LO6pZ+p/ruef53Rw83sZ3X7Aee/2g9W94qLaNJ7eU87FFWUSEffDtl5UYzT1XzBvyMaWUfwSqN+4sY0y1iGQA60XkgDFm08AdjDFrgbUARUVFJhCFVIM9+V4FD75xhB63D4DIMAefW5XPA6+XUt/eQ1pcJH/eWU1MhJMn3yvnic3lAFwwN4PXDtRxxNXOt5/ZRXxUON/52LxA/ilKqQECEgTGmGr7d52IPA+sADad/FkqkLp6vfxq42EAXt5VTWSYk7ML0rhoXiYPvF5K8bFGzp2dwZuH67lxRS63nDGd3VUtOB3CgqmJvHagjq8+uZ09Va387w2FpMZFBvgvUkr1GfcgEJFYwGGMabNvXwL8x3iXQ42OMYYN++v4w7tluNp6+MKqfB568ygAd1wwi0XZiUSFO3j3SCNOh4Mej4+L5mUyI33wimDTU2PYU9XKhXMzuGqJXhym1EQSiBpBJvC8PR1AGPCEMeavASiHGkFTRy93/mknGw7UkRobwb9dNJvbzpvBU8UVtHV7uHBuBhFhDpblJvPe0UbaezzER4YN2dF7yfxMntxSwY+uWahTQSg1wYx7EBhjjgBLxvu46tTd949DbCpx8d2PzeOzZ+YR5rQ6cG89I49SVzsZCVEAnJ6fyn3/OMS+mlauXZ4zZEfvnZfO4bZzZ2qTkFITkF66qYbk9vp4eVcNly6YwhfOnjHosTsvnTPo/qULM3lqSzk3rLCGfg4lMsxJZJzTb+VVSo2dBoEa0j9LXDR29LK6cOTF3edOSeDtuy8ch1IppfxBB2urIT2/vZqkmHDOmZ0e6KIopfxMg0B9QHuPh/X7jg970ZdSKrho01AIM8aw7r0KOns9nDkzjcfeLeP0/BR8xtDt9rF66cjNQkqpyU+DIES52nr4wUt7eWV3zaDtL+2oYlZmPNlJ0SzP1aUflQoFGgQh6NmtlXzvxT30eHzcdflcVs1K490jDSzMTuTmhzezs6KZL583E4dDx/srFQo0CILIodo2jIE5U+KH3aet280P/7yXuVPiufe6Jf1X/y7MtmYC/eyZefz2n0e1WUipEKJBECS63V6u+807tHS5WTA1gTsvmcN5c9L7r+LddMjFW4frcTqE1m4PP7xq4aApIPp8+7K5XF2YzezM4cNEKRVcNAgmMGMMXp/pv6L3ZNbvq6Wly81nz8zj9YN1/Mvvt3DnJbO544ICPF4f9zy/m8qmLgDOnZ3OopwPrgUAEO509NcOlFKhQccGTgC9Hh+H69oxZvBs2/dvPMzK/36Nw3VtI77Gs9sqmZoYxfeunM/f/+1cLpmfya9fL8XV1sMru2uobOri6xcVcMHcDO66fK6//hSl1CSkNYIA8nh9PL+9iv/dUEJlUxfLpyfz6RW5rMhPISU2grWbjtDa7eHWR7bwwGeWsSg7ERHBGEN7j4f69l5Katuoaelm0yEX/3reTJwOa+3fu6+Yx0U/e4PvvrCbw3XtzEyP5asXFGgHsFLqAzQIAqTU1c4X/1DMEVcHi7ITuXFFLo+9U8Y3/7QTp0O4YG6G3Za/gHv/dpCrfvUWeakxLM5JYmtZE1XNXYNeLyLMwXXLp/Xfz0+L5YbTpvH45nIiwxz88salGgJKqSHJic0RE1FRUZEpLi4OdDEoqW3jO8/v4T9XLzzpyJyRuNp6uObXb9Ht9vJ/r1nExfMzERG8PsPhunZ+8NJe3jnSwNLcJJ7/8lk0d/byyu4aNuyvY3dVC4uzE1mRn0JqXCSzMuKYmhhFQnQ4UeGDJ3Xr8Xg5Vt9JflqsXiGsVAgSka3GmKIR9wvFIPD5DCKMOC/+n4or2FXZwj1XzMNrDFf96k2OuDr4xNJsblqZy1fX7eArF8wiLiqMJzaXU9fWw+LsRL507sz+oGjtdlPe0MmsjDiiwp1sLWvkG0/vpK61h6e+tJLFOUkfOG6328svNpRw+cKsYTt1lVJqJBoEwzDGcNNDm4mPCuPBmz94frrdXnZUNPP8tiqeKq4AYElOIp29Xkpd7RRNT2FHRTMz0mM5aI/bB5iRHsuMtDjeOlxPl9vL9NQYej0+alq6AUiJjSAnOZpdlS3kJEfz8+sLKcr74AIuSin1URltEASkj0BELgP+F3ACDxljfjxex35pZzVvlzYgApVNneQkx1De0MmvNpZwuK6dPdWt9Hp8iMAXVuWzeFoSdz69k4LMOB74zHLy02K55L5NHDjexr3XLaGly01MhJNPFU3D6RAaO3r5885q/llST3xUGLMz45maFMUru2qoa+vhrsvnctPpucRHhY/Xn6yUUic17jUCEXECh4CLgUpgC3CjMWbfcM/5qGoE7T0eLvrpG8REOjla38HXLizgptOn84kH3qKhvZeF2Ykszk5k5YxUTstPITHa+rDu6vUSFe7ob0q65ZH3aOt28+xtZ2oHrFJqwprINYIVwGF7yUpE5EngamDYIPgoeLw+bn98G672Hp6+6QzuW3+IP75bznPbqqhv62XdmpUUTvtgez1AdMTgTtiHbinCYDQElFJBIRBBkA1UDLhfCZzujwP9YkMJz26rpLXLjc9AS5ebH39iEcunJ/OZlbnc9sdtLJiawP98cvGwITAUHYGjlAomgQiCob5Gf6B9SkTWAGsAcnNzx3SgzIRICqclER8Vhs9A4bQkPlVkjbW/bGEW//z2+eQkR484ekgppYJZIIKgEpg24H4OUH3iTsaYtcBasPoIxnKg60/L5frThg+RaSkxY3lZpZQKKoFo49gCFIhIvohEADcALwWgHEoppQhAjcAY4xGRO4C/YQ0ffcQYs3e8y6GUUsoSkOsIjDF/Af4SiGMrpZQaTIe/KKVUiNMgUEqpEKdBoJRSIU6DQCmlQpwGgVJKhbhJMQ21iLiAsjE8NQ2o/4iLE2z0HJ2cnp+R6TkaWaDO0XRjTPpIO02KIBgrESkezcx7oUzP0cnp+RmZnqORTfRzpE1DSikV4jQIlFIqxAV7EKwNdAEmAT1HJ6fnZ2R6jkY2oc9RUPcRKKWUGlmw1wiUUkqNICiDQEQuE5GDInJYRO4KdHkmChE5JiK7RWSHiBTb21JEZL2IlNi/kwNdzvEkIo+ISJ2I7BmwbchzIpZf2O+rXSKyLHAlHz/DnKMfiEiV/V7aISJXDHjsbvscHRSRSwNT6vEjItNEZKOI7BeRvSLyNXv7pHkfBV0QiIgTuB+4HJgP3Cgi8wNbqgnlfGNM4YChbHcBG4wxBcAG+34o+T1w2QnbhjsnlwMF9s8a4IFxKmOg/Z4PniOA++z3UqE9ozD2/7UbgAX2c35t/58MZh7gm8aYecBK4Hb7PEya91HQBQGwAjhsjDlijOkFngSuDnCZJrKrgUft248CqwNYlnFnjNkENJ6webhzcjXwB2N5F0gSkazxKWngDHOOhnM18KQxpscYcxQ4jPV/MmgZY2qMMdvs223Afqy12SfN+ygYgyAbqBhwv9Lepqy1of8uIlvtNaEBMo0xNWC9oYGMgJVu4hjunOh7a7A77KaNRwY0KYb0ORKRPGApsJlJ9D4KxiAYaiV6HRplOcsYswyranq7iJwT6AJNMvreet8DwEygEKgBfmpvD9lzJCJxwLPA140xrSfbdYhtAT1HwRgElcC0AfdzgOoAlWVCMcZU27/rgOexquy1fdVS+3dd4Eo4YQx3TvS9ZTPG1BpjvMYYH/Bb3m/+CclzJCLhWCHwuDHmOXvzpHkfBWMQbAEKRCRfRCKwOq5eCnCZAk5EYkUkvu82cAmwB+vc3GrvdivwYmBKOKEMd05eAm6xR32sBFr6qv6h5oQ27Wuw3ktgnaMbRCRSRPKxOkTfG+/yjScREeBhYL8x5mcDHpo87yNjTND9AFcAh4BS4DuBLs9E+AFmADvtn7195wVIxRrRUGL/Tgl0Wcf5vKzDatpwY31T+/xw5wSrSn+//b7aDRQFuvwBPEeP2edgF9YHW9aA/b9jn6ODwOWBLv84nJ9VWE07u4Ad9s8Vk+l9pFcWK6VUiAvGpiGllFKnQINAKaVCnAaBUkqFOA0CpZQKcRoESikV4jQIVFATEe+AGTJ3jDQbrYjcJiK3fATHPSYiaWN43qX2zJ7JIvKXD1sOpUYjLNAFUMrPuowxhaPd2RjzG38WZhTOBjYC5wBvBbgsKkRoEKiQJCLHgKeA8+1NnzbGHBaRHwDtxph7ReSrwG1Y0wzvM8bcICIpwCNYF+h1AmuMMbtEJBXrwqt0rCtpZcCxPgN8FYjAmozsy8YY7wnluR64237dq4FMoFVETjfGXOWPc6BUH20aUsEu+oSmoesHPNZqjFkB/Ar4+RDPvQtYaoxZjBUIAD8Ettvb7gH+YG//PvCmMWYp1pW2uQAiMg+4HmvCv0LAC9x04oGMMU8By4A9xphFWFM2LNUQUONBawQq2J2saWjdgN/3DfH4LuBxEXkBeMHetgr4JIAx5jURSRWRRKymnE/Y218RkSZ7/wuB5cAWa0oaohl+Yr8CrGkHAGKMNbe9Un6nQaBCmRnmdp+PYX3AXwV8T0QWcPIphId6DQEeNcbcfbKC2EuHpgFhIrIPyBKRHcBXjDH/PPmfodSHo01DKpRdP+D3OwMfEBEHMM0YsxH4NpAExAGbsJt2ROQ8oN5Yc88P3H450LdQywbgWhHJsB9LEZHpJxbEWEuHvoLVP/ATrEkBCzUE1HjQGoEKdtH2N+s+fzXG9A0hjRSRzVhfiG484XlO4I92s49grc/bbHcm/05EdmF1FvdNM/xDYJ2IbAPeAMoBjDH7ROS7WCvDObBm8LwdKBuirMuwOpW/DPxsiMeV8gudfVSFJHvUUJExpj7QZVEq0LRpSCmlQpzWCJRSKsRpjUAppUKcBoFSSoU4DQKllApxGgRKKRXiNAiUUirEaRAopVSI+/9cSF7lRMf3iQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def ddpg(n_episodes=3000):\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores_global = []\n",
    "        \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]      # reset the environment    \n",
    "        states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "        scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "        agent.reset()\n",
    "        \n",
    "        score_average = 0\n",
    "        timestep = time.time()\n",
    "        for t in range(3000):\n",
    "            actions = agent.act(states)\n",
    "            env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "            next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "            rewards = env_info.rewards                         # get reward (for each agent)\n",
    "            dones = env_info.local_done                        # see if episode finished\n",
    "            agent.step(states, actions, rewards, next_states, dones, t)\n",
    "            states = next_states                               # roll over states to next time step\n",
    "            scores += rewards                                  # update the score (for each agent)            \n",
    "            if np.any(dones):                                  # exit loop if episode finished\n",
    "                break\n",
    "        \n",
    "        score = np.mean(scores)\n",
    "        scores_deque.append(score)\n",
    "        score_average = np.mean(scores_deque)\n",
    "        scores_global.append(score)\n",
    "        \n",
    "        print('\\rEpisode {}, Average Score: {:.2f}, Max: {:.2f}, Min: {:.2f}, Time: {:.2f}'\\\n",
    "              .format(i_episode, score_average, np.max(scores), np.min(scores), time.time() - timestep), end=\"\\n\")        \n",
    "        \n",
    "        if i_episode % 10 == 0:\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))   \n",
    "        if score_average>32:\n",
    "            break\n",
    "            \n",
    "    return scores_global\n",
    "\n",
    "scores = ddpg()\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}