Report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the Agent if Necessary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to train the agent, or skip it to just used the saved model weights. If you train the agent, you will have to restart the kernel afterward because of https://github.com/Unity-Technologies/ml-agents/issues/1167\n",
    "\n",
    "The trainer uses **DDGP** (see [original paper](https://arxiv.org/abs/1509.02971) for details) with actor and critic networks both with two hidden layers of 128 nodes with ReLu activations. The final layer uses tanh activations to enable The actor network uses batchnorm after both the first and second layers. Gradient clipping is applied to the critic network.\n",
    "\n",
    "* BATCH_SIZE = 128        # minibatch size\n",
    "* BUFFER_SIZE = int(1e5)  # replay buffer size\n",
    "* GAMMA = 0.99            # discount factor\n",
    "* TAU = 1e-3              # for soft update of target parameters\n",
    "* LR_ACTOR = 1e-4         # learning rate of the actor\n",
    "* LR_CRITIC = 1e-4        # learning rate of the critic\n",
    "* WEIGHT_DECAY = 0        # L2 weight decay\n",
    "* OU_THETA = 0.15         # Ornstein-Uhlenbeck noise theta\n",
    "* OU_SIGMA = 0.01         # Ornstein-Uhlenbeck noise sigma\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run ./DDGPTrainer.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Agent Visualization\n",
    "\n",
    "-----\n",
    "### Setup the Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using GPU :-)\n"
     ]
    }
   ],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import DDGPTrainer\n",
    "import torch\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Tennis.app\"`\n",
    "- **Windows** (x86): `\"path/to/Tennis_Windows_x86/Tennis.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Tennis_Windows_x86_64/Tennis.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Tennis_Linux/Tennis.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Tennis_Linux/Tennis.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Tennis.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Tennis.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=\"Tennis_Linux/Tennis.x86_64\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Size of each action: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.         -6.65278625 -1.5\n",
      " -0.          0.          6.83172083  6.         -0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# get the default brain\n",
    "\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]\n",
    "\n",
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents \n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the Saved Model and Visualize the Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot the training scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f16c0245550>]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmYZFV9//H3t7bep2frWZiFnpFhUxGGERgXQlARECERFYyPUaOiRqMmJj6iCSEkcU34RYMJEATEBRckCDiAKEREFhmGYZgVevYZZu2Z3pfavr8/6nZP9VLV3TPV1V3F5/U89fSte0/de27d6m+d+t5zzzV3R0REyktooisgIiKFp+AuIlKGFNxFRMqQgruISBlScBcRKUMK7iIiZUjBXUSkDCm4i4iUIQV3EZEyFJmoDc+cOdMbGxsnavMiIiXp2WefPejuDSOVm7Dg3tjYyMqVKydq8yIiJcnMto+mnNIyIiJlSMFdRKQMKbiLiJQhBXcRkTKk4C4iUoYU3EVEypCCu4hIGVJwFxEpknTa+ekzO0mk0uO+LQV3EZEi+fmqXXzh52u45Xdbx31bCu4iIkXS0pUAoLmjd9y3peAuIlKGFNxFRMqQgruISBlScBcRKUMK7iIi4yCeTLPlQAc9iRQtXXEAuuKpom1/wsZzFxEpZ++56Ume39nS//znn1zOg+v2Fm37armLiIyD7MAO8MKuVqZWRQF47+sXjPv2FdxFRIrAg79nNU7nxNl14749BXcRkSLx/hA//kYM7ma2wMweNbP1ZrbOzD47TJnzzKzVzFYHj2vGp7oiIqXLHbDibGs0J1STwOfdfZWZ1QHPmtnD7r5+ULnfufslha+iiEj5KFJsH7nl7u573H1VMN0ObADmjXfFRETKiTtFTMqMMeduZo3AGcDTwyxebmbPm9kDZvbqAtRNRKS8ONgkSssAYGa1wM+Bz7l726DFq4Dj3b3DzC4G7gGWDLOOq4CrABYuXHjUlRYRKVVWpMTMqFruZhYlE9h/6O53D17u7m3u3hFMrwCiZjZzmHI3u/syd1/W0NBwjFUXESktk623jAHfBTa4+/U5yswJymFmZwXrbS5kRUVESp1PsrTMG4EPAC+Y2epg3peAhQDufiPwbuCTZpYEuoEr3b2Y5w5ERCY1Dx6TJri7++OM0HvH3W8AbihUpUREytWkyrmLiMixK2ZCQ8FdRKQI3L2oaRkFdxGRMqTgLiJSJMXsZqLgLiJSJJm0jE6oioiUF/fJM3CYiIgUjk6oioiUmUk7KqSIiBw990k0nruIiBy7vp4yOqEqIlJmJtWokCIiUhhKy4iIlBnHizrkr4K7iEhRKecuIlJW1BVSRKTMuGdGhlRaRkSkDOmEqoiIHDUFdxGRIlFvGRGRMuS47qEqIlJO+nrKqOUuIlJmdCcmEZEypBtki4iUKeXcRUTKjBcxL6PgLiJSBO7BSVWlZUREyoyG/BURKU+T5k5MZrbAzB41s/Vmts7MPjtMGTOzb5tZk5mtMbOl41NdEZHS5EW9DxNERlEmCXze3VeZWR3wrJk97O7rs8pcBCwJHmcD/x38FRGRgLtPnrSMu+9x91XBdDuwAZg3qNhlwB2e8RQw1czmFry2IiIlblL2czezRuAM4OlBi+YBO7Oe72LoFwBmdpWZrTSzlQcOHBhbTUVESsS+tp5h50/Km3WYWS3wc+Bz7t52NBtz95vdfZm7L2toaDiaVYiITHrNHfFh50+6G2SbWZRMYP+hu989TJHdwIKs5/ODeSIirzjDpV76rl+aTL1lDPgusMHdr89R7F7gz4NeM+cAre6+p4D1FBEpecXsLzOa3jJvBD4AvGBmq4N5XwIWArj7jcAK4GKgCegCPlz4qoqIlLZipmVGDO7u/jgj1MczAyZ8qlCVEhEpR+5o+AERkVKVL62uUSFFROSoKbiLiBRYrta5u0/Oi5hEROTYTKp+7iIiMnq5WueT8gpVERE5eu6e6QqptIyISGnKFb8dV28ZEZFypJa7iEgZufMPOyni/bEV3EVEimF3SzeOWu4iIiUrfwBXzl1EpKwoLSMiUtJy95dRWkZEpMxMujsxiYjI6OUdFVItdxGR8qLhB0REypC7rlAVESlLSsuIiJSo3H1likfBXUSkSNRbRkSkhFmO3EvaPeeyQlNwFxEpFl2hKiJSujzHOAPKuYuIlCHdIFtEpITlaqE7qJ+7iEi50aiQIiIlLFcQT0+mtIyZ3Wpm+81sbY7l55lZq5mtDh7XFL6aIiKlL5OWKY7IKMrcDtwA3JGnzO/c/ZKC1EhEpOTl6C1TxLzMiC13d38MOFSEuoiIlDX30htbZrmZPW9mD5jZqwu0ThGRkpSrgZ65QXZxovto0jIjWQUc7+4dZnYxcA+wZLiCZnYVcBXAwoULC7BpEZHSkRnytziOueXu7m3u3hFMrwCiZjYzR9mb3X2Zuy9raGg41k2LiExK+fq5F8sxB3czm2PB7wwzOytYZ/OxrldEpNx4EbvLjJiWMbM7gfOAmWa2C/hHIArg7jcC7wY+aWZJoBu40ot5SlhEZJLJFwGLdYXqiMHd3d83wvIbyHSVFBGRSUJXqIqIFJjnya6XWldIEREZhZLpLSMiIgPlzbmr5S4iIkdLwV1EpMAmQ28ZBXcRkSJSWkZEpETl6y1TLAruIiJFpN4yIiIlKu81+kXKyyi4i4gUkVruIiJy1BTcRUSKSL1lRERKlPq5i4jIuFBwFxEpMI0KKSLyCqPeMiIiJUqjQoqIyLhQcBcRKbD8F6iqt4yIiBwlBXcRkQLzvIPLFIeCu4hIEemEqohIicqbc9cVqiIi5UctdxGREjUJUu4K7iIixaQrVEVESpbGlhERkXEwYnA3s1vNbL+Zrc2x3Mzs22bWZGZrzGxp4aspIlI6SmU899uBC/MsvwhYEjyuAv772KslIlKeJk1axt0fAw7lKXIZcIdnPAVMNbO5haqgiEipmQSdZQqSc58H7Mx6viuYJyIiE6SoJ1TN7CozW2lmKw8cOFDMTYuIFE3+8dwnT859JLuBBVnP5wfzhnD3m919mbsva2hoKMCmRURKSyn1c78X+POg18w5QKu77ynAekVEStJkGBUyMlIBM7sTOA+YaWa7gH8EogDufiOwArgYaAK6gA+PV2VFREpdsXrLjBjc3f19Iyx34FMFq5GISInLPypkcegKVRGRMqTgLiJSYOXSW0ZEREZp0lyhKiIiY+P5RoUsUh0U3EVEypCCu4hIoeXtLqOcu4hI2VFaRkSkRE389akK7iIiRaXeMiIiJapU7sQkIiIFopa7iEiJytfPvVgU3EVEiki9ZURESlT+sWWKUwcFdxGRMqTgLiJSYPnHcy9O033Em3WIiEh+t/xuC//yyw0A3PBnZ1BXGc1dWGkZEZHScMeT2/un73nu5bz3UNUJVRGREpF9krRYJ0xHouAuIlJg+QeF1BWqIiIlYSzhWmkZEZESkd0aN5gUw0IquIuIHKPs1vhIWRddxCQiUkQdvUk+9+PnaOmKj/3FgwJ23nuoKriLiBTPD57azj2rX+a//m/zmF87oOU+QlZdQ/6KiJSIwT1g8o0tUywK7iIiHAnIR9OuVs5dRGSyO4rgOzhgl0zL3cwuNLNNZtZkZl8cZvmHzOyAma0OHh8tfFVFRMZPoW6wUTJXqJpZGPgOcBFwKvA+Mzt1mKI/cffTg8ctBa6niMi4OpKWGXt0HvyafF8TJ238L9j8yJi3MVajabmfBTS5+xZ3jwM/Bi4b32qJiEyMsbS8n97SzMGO3oFjy+T5coiRyAT3HU8fQw1HZzRD/s4DdmY93wWcPUy5y83sXOBF4K/dfefgAmZ2FXAVwMKFC8deWxGRcTaWdvsVNz/Fopk1VEQGtpNzjQq5wPZjpGH64mOo4egU6oTqfUCju58GPAx8b7hC7n6zuy9z92UNDQ0F2rSIyLHLN0xvvvJbD3YOXJDn2+F425eZmCTBfTewIOv5/GBeP3dvdvfe4OktwJmFqZ6ISHH059xH2XRP5/kuyLWocZIF92eAJWa2yMxiwJXAvdkFzGxu1tNLgQ2Fq6KIyPjrC8ijPaGayoruiVS6fzrfq4+3vSSidVA9/ShqODYj5tzdPWlmnwYeAsLAre6+zsyuA1a6+73AZ8zsUiAJHAI+NI51FhEZN6NvuR8J7vGs4A65+7k32j66ahZSX4T+kqO6h6q7rwBWDJp3Tdb01cDVha2aiEjxjPXCo+zg3pPIarnnCdzH2z46a8+kfsy1GztdoSoiBZcc1JI9Fu5e0PXl2kYqndnGaNrUyVSaZFZapjeRGrK+wd4cWkNjaB9ddY3HUtVRU3AXkYLa0dzFCV9+gJ8/u6sg67v23nWc8OUHCrKuXP7pvvV8+5GmzBMzXm7ppml/x7BlE6k0J3z5Ab66YmP/vLae5IAyn/zhqgHPLwk9yfeiXyflxt75FxW28jkouItIQb24rx2AFS/sKcj6vvfkdgDS+bqnHKPbn9jWP23AG772CG+9/rfDlu2KZ1rpd/5hx7DLh2v5fyDyMM3UcXrv/9A+Zckx1nZ0FNxFpCSkJsNoXEA8mT9FNDjlPpV2ltkmfpR6C+1U5+1CWUgK7iJSEtJFCu4jdWTpGZRfH2xw8D4rtJGwOY+lTgOK9yWl4C4iJSE9vudU+43Uz703OVJwPxK859DMv0RvA2CTZ64FHeuVsEdLwV1ESkKxWrwjt9xH+JbJquZHIg8wy1r4ZeosOqgGivcLRMFdREpCqljJ6hH0pWVyfQlkjwu/xHazNt3IpxKf6583zr06+ym4i8iE60mk+NgdK9lyYPjuhzB+vWUGp0n6esNkS6bSfPpHq/jOo01cffcLAIRyRPd48sj6Gm0v23zOgOXFarmP6gpVEZHx9PTWQzy8fh89iRTf/8hwI4qPT1rm7K/8mll1lQPm/Wrd3v7pv/nJaq6/4nS2NXdy/5o93L/mSPfOXNmbNbtaAIiQZL4d4L708gHLlXMXEckyHi33fW29vLC7dcC87K3c/VxmANxIaGiozNVy76vmAjtAxNJsSw9uuR99fcdCwV1ESkKxTqgOl9sfNpDnyrkH9Xx/+NcAbB2UlinWuQMFdxGZcKNJVRQrKA6XE08O0w8zlCO49w3/u9D2A7DaTxiwXGkZEZEsxernPlzsHXVrHuhJpgHnZNvBQ6llpAeFWaVlROQVI98wuX2KlZZJDNNXMTlMRM5V43gyTaPtZWHoAI+lTxuyXGkZkVcAd+cXq3eP+5C2x2rbwU5WbjvEg2v30t6TOOr17G/r4abfbmb9y208/tJB9rX1DFvuyc3NXPOLtRzs6O2flx0UtxzoYNWOw8O+NpXOvKfptHO4M86Da/ew+UAH/3z/enoSKR54YQ+dvUl2Huoavo7tvQOeX333Gv7mp88PKdc5qMukkeb80Co+FH6Q26PfAGB1+oQhr1NXSJFXgPvW7OGzP17NrsPdfOqPhwaCyeK8f/u//ukLXz2HGz8w+tskp9POdfev5/1nL+Qj31vJjkNdfPWBzHC5s6dU8PSX3jrkNe/7n6cAuCMYERIGBsXz/z0zYuO2r71jyGu//+Q2rr1vPXf+YQdPbTk0YNm6l1t5assh3vCqGTyxuXlU9b/zDztHVe4LkZ/wych9ALzs07km8UHW+fFDyhVr/DMFd5EJ1NoVB+Dllu4JrsnobT3YOaby25o7uf2JbTz20gF2DGot72vLtJJH081xtOmMfUHLe3BgB/rHaB9tYB+N062J94Yf5c8ij3JP6g18JfF+9jOV5YtnwpYj27nq3MXc/NgWtdxFXgki4UxmdLJcWj8aw/Ucyadv33KdgMyss1i9ZQpz79IpdHCi7eJjkRW8PbwSgM4T3snn176XFGEAKqIDs959u1+scwcK7iITKBz0p0ukSim456/r4KV9+xbJ1XeQ0d2Wb7Qt3vGJnc4cDnFaaAtvCT3HxeGnqbNu0m58K/Vubk+8lfvecRmptY/2v6IyEh6whnAQ3ZWWEXkFiIYz//BjbQ1PpOQIX0R9Ley+Un29TyLhwrfc3X1ITxsf8vWSvZ3Rvc8LbB8XhJ5lWWgTc+0Qi2wP9ZZJKbV7Fb9Kn8n9qeW8mJ5Pc3Q2PaSHXMU6eH/7frmM5x2lBmy/KFsRkWGFg4AwmuA2WYwUIAdfpBPvC+7DXMLfJztw50q/DNdyj6fSVAxqIeeK7VGSxHvTZHcSDJGp2wxaOTP0EmeHNrDY9rA8tI6YpdjnU3kxPZ/70svZ7rN5Nn0i67yRXmL966gLhTLBfVAwH/ylE1JaZmRd8SRV0TBmRmdvkpqKktyNY9KbTOGe+YlXFQsPWd53d/bK6MBl8WSarniSqdUxehIpIiGjM56iN5GiJ5GmvipKfXUUgMOd8f7XV0RCdCdSmEF1LPN+d8dTJNNpehJpplZH6QhuElxfFeVgZy8VkTCRkFFTESGeTLO3tYeZdTHiycx24qk07pnAVh0N05NMkUo7tRURmjvj1FZEONjRSyQUYkZtDAO6EynSDrFwiJQ7PYkUYTN6kimqoxFqKsLEU2mqYxHaexKEQ0ZLV4L6qighsyHvVTKVZl97L7FwiOpYmFgkRDyZJmRGLBKiuaOXmbUVhEJGIpUmlXbae5JURkOEzGjtTuBkWmNVsTAGHO5KUBkNkUg5NRVhGmorONDRS8gyt4Ews8x7CbR1Z7oVdvYm2dPaTWdvilg4RF1lhLaeBFWxMFXRMNFwKLMtz7wHC6ZVsfNwN7UVkf5gmvbML4HW7gRmxvSaGM0dvTiZz0lFJEQq7f1fJJGQ9R/PnkSKqliYZMrpTqSoq4zQ3pNkZm2MqkGfoZauBO09Ceoqo8STadKe+Zz1vz+9mc9Be0+Czt5k/0iPbTm6UG472Mmm4L6rbd0JNuxpG7bcrsPdzKjp7A+OdXSxafsepndvJbb/BWKdLxNJdfPOzRu5PLaVHmJESRIjyVTrYBodhMzZE5lOiDRVxKmil6gN7NK4Nt3Ij1Pn86PUW9joCxgpT98X1EfK5oeC6F6s73Er1qWwgy1btsxXrlw55te1dMU5/bqH+fzbTuS18+v50G3PcNcnlrOscfqQsr3JFE9ubua8k2YVosqTysn/8ED/TQM2f+Xi/txtn/ff8hS/b2oe0lWs8Yu/BOC2D72eD9/+DOefPItNe9vZndVb47YPv55zlzTwqi+t6J930uy6/n/A56+5gClVERZdvYLh1FdFae0+8o982enHsWlvOxv3tvfP+/i5i/nV+n20dMU53DXwn37OlEr25uj/PForPvNmLv7274bMb/rXi/pPYgJc9K3f5QwmbzxhBr9vaubjf7SYqy86hUtveJw1u1qHLZvP4pk1bBljD5NCqyBOmDRz7BDTaaPCElQRp4ZueonRS5ReorR6Db1E6fQqmplCnGjOdf72787jY3esZPfhbtZddyFX3PQkT28d2kOlUPV/tW3jnNB6Tg1tp9H28ZrQtiHlEh5mq89ht8/EMRJESBCmxWs5SD1Rksyzg3R7Bd1U0E2MJGE6vIo0xnPpJazyE0ddr+WLZzC1OsoDa/ey5toLOO3aX/Uv+7u3n8Q3H9oEwAmzavnMW5bwmTuf4/r3vo53LZ1/1O+FmT3r7stGKldyTd6+ixruWb27v4Xw7PbDwwb3r67YyO1PbOO+T7+J186vL2o9x1v23WB6k6n+1nSf3zfl7+r1+6aDADyycf+QZR++7RnW/tPbB8zrC+wAh7ridMaTOdedHdgBfrH65SFlbnpsS87XH2tgB4YN7AC9yfSA4J4rsMOR9/C3mw5w9UWnHFVgB0Yd2GfQyjmhDZwd2kCDtRAjyRTrZCat9FBBG9X0ehTDCeGZv5b52/cIZf2to4vp1o7h1FsXlicXnUunV7DOG9nr0+nyCrb7HOJE6KSSju31vLSvDSdEOu3HHNj/6vwTmFod45/vXw9kAvry0DreFX6cS8JPEQrqvzU9m5d9JrcmL+Sw17LJF3D6WW/mm091YcC86TXsPJS/a+kXLjyJbzyYCbx/cvpx3LP6Zf7+Hafw1uPq2bS3jYa6Sj71o1Vcde5iFk6vDjIFMK0mk4559dwpPLX1EOefPItIyPj0+R1MqTzyRfjg597MibPqeMOrZpBMO/OmVjG3vpL506o4Y8HUY3qfRqvkgvtYfmhsDn4OtnTHx6k2k0NPIk11bORy2UZ6G/PdBNjdh72hQSnoSaQKnsbbcN2FnHLNgwD86KNnc87iGSwOfvW8cO0FvDarNWekOd720Wj7+LtTW9my6XlOte3Mt4NUWOZLscMr2e0ziROhi0rW+iIqSTDFOqm1bk6cU09FNEzKjed3tZL2TNhLByF+2aIZVEYjJMPVhKfMBjOsejrJcCXUziE8ZTYWrSIZquCC/1pFBQliJKgkQb11UEGCWutmGu00WCtnhJpYGnqJWrqZallfVPfewnMVNaxJLyb16PMsD0VYlV4yIB+dS9O/XgRkegul0o71thLe8SQc3MSHz30ZOpsJvfggJDrxaA126vvYOes8Lr0vzWGmAPCDj5zN8lfN6F/PN576JR7Mnz2lknDI+nuopD2TTjvjnx8G4GNvXtwf3P/jyjO4/r2n96dN+tZ50Wsu7p83nEtfd1z/9KuPG9h4PHlOpo5nLJw2YP7SQc/HU8kF90TKuST0JDv9HGD2RFdnUhgpEA83bkc8mf+kWL51xlPpEXtMFEKMBCfaTmZZC3GiREgxhU56iNFNBdPooMp6iZGgml7mWKbl2E4V3p8BNSqI02At9HqMyBMvwMx5UDcX6udhpPFjGIWjrjJCVSxMyDK51Jl1mfx8mBSvsa3UrXyJW6P/y0Lbzwxro5oeKizzq8c3h6i1mbzoC3g4vYyDPoXL//Ry3nlXB8k8/5q/uvxcTpxdR3tnnCuCYJVt1RVvo6omNiShMniNEWCL7x04M+9h9eCLIMlUa+emc5Os+f0vOS20lcjvvsGdMSfuYdZ7I0nCbE3PYaMvZJMvIBmkRsKkmWvNRB5fBx374OCLRLoOw/514JnPZChSCRV1cOql8Jp3Y41vgmglkdZuDt/3SH9tKqOhIenIzPzwkHNNIQaeb4mGBx7z4YJ4vsBeCkYV3M3sQuBbQBi4xd2/Nmh5BXAHcCbQDFzh7tsKW9WMZNseboj9J2u6f8N9/t28Zfta+RN0WqFo8gXi3mR6yAd9pNdklucO/j2J9LCDK4HTaHtZYAc4zpqZTjuz7DCVxKmyXmro5bDXst1nE7MkYVKESRMmTRW9VBKnwhJUkGCx7aHR9g452ZVPm1cBRi2Zn+Qhyxz4Xo9wkHpiJKh/4pEBr9lQEWWfT6ONalq8lv1MY0t6Lnt8Om3UcNDrqU4PvYS8T2U0DD2tLLON1Fg3Dc+vh/1Ps7riceqsG34NC2wem3w+B9P1dFPJZp/L9vRsrvnYlVxy88AxS/5k7jKSPJ53P/v6T1dGh/9SyjX/2FmQn4/R7tVsn7+ULybnAfDE587k77/9P5wV2sgZoSbm2UGOD+/lPfbY8Kt6FAhFYe7roG42nHwxLDoX5p4OsZphb1A6uN/4cJ/r4cr1qYi8sobSGjG4m1kY+A7wNmAX8IyZ3evu67OKfQQ47O4nmNmVwNeBK8ajwqnOTB70tNR67htl1B4+EJWP/IE4New/QXee4B4iTW93J9NpY5q1kyZEjATTgp/slVs6iSbjvCf8PEtsN8fbPubbARbYAabYwMvL272KHqJ0ewWdVHF66CUaLJPnTnqIFGGShEgTop0qejxGnCjbfDa/Sp/JunQjBzyTo0wSpo1qauihxno46PV0eBVxIvQS7b+7fD4rPrmUU6f0Qvs+OLSFO+66j9l2mBq6mWHtnGDruDw8KF/fBvy/BfwsVsVun4lB0AsjwUnJvfC1Pfy0LxPxBDDzJO5JvZGn0qfynS/9FW/7l2eGrcuU+qHniXIFrAFlYpkglSuI5ZpfaD3JI5+hrlAtj6SX8kh66YAyDbSw2PYwzdqJkCJOhINez93XfQIilbnvMj2MwVd8xnIE68Hl+oxm5MlyMpqW+1lAk7tvATCzHwOXAdnB/TLg2mD6LuAGMzMfh6443nlkJLhPr3kXn61ooWX1adC1FOoXQM3MzE+6ijoWx7cSC+2gbmcbxOZApALCFRCJZf6GoxAKg4UH/Q0NfB6KZOZN0g9H/z9ZX99ITxMhSQinp7sLIkkgM38KHcyzZo5rP8jr7SARSzGdds4MvciZoRc5xbYTsxTcBqsqc2wwuAjvm1Ho8Si7fSbbfE5/H+At6bkcoJ5DPoU2aga92Kmil24qOKZLwY/yk9VNJUybC9MaYeHZfOXHU4aUqaKHWdZCPZ3MsDbOmXKIjy9oJXV4HUvtJZKEg0eE7aEFLPijj/IXD3bT6jV872+voHbGPP4h6JX0ndrcPbWm1gztiTKaVndf18RcaYNipROyGxW5fgkeYGrmyznreFVGQxCtGvP2Bn9p5RrO4JXWQs9lNMF9HpA9LNouYPAdbPvLuHvSzFqBGcDBQlQym3dngvtdqXN5Y+865loPew/vpevp71HNwDPk1wHEyLSmnjj2baeCFmb2I0WItA03L0yKEE4o6KWQ+XQb9Pds6MsX9c0jyBRbf9kjfy3r9eA8U5G5IcAUuoncmgIGtt6b+gLzfw7chzV98/cAFUfmd3uM5/1V3Ja6kE6voqqqkv3dxmGvw4E4UdqoptsrmFpbTdzD7OgMsdMbxpiztkyAnSCfufM5qoe5LiBbN5Vs77s1msOjLXDXjlpeincMKXvqjCmsOPfN/OE3D9HRm6Ri6ty8666MhvqDYm1s6L9fZTQ8YlfQ0bTui+H6h1/sn/7ED54d9evqq3J3r8xn8JdWru+wV1oLPZeinlA1s6uAqwAWLlx4VOuomTaLFamzeGDWVTwybS4PrN3DRa89DtypSbdRneqgMt1JZbqLWKqLx3d0c8arjiPsCSKeIOoJIh4n4gnCniTkQTj2NNYXnj0Txs37nveVORLeLet1A8sGryVN2FPBCbsjIR2OhHm3zPwB87LKAf1lBr/2cHeSVNppTlQwd0b9gNc6Idp7UxzuTjJvWnVmXrCeg51Jnm+t4ZTG41i7t4vZU+ue3UvOAAAHpElEQVSYO2smMxadTnOP87UHNvK6BVOZN7WS7c1d1FZEeH5XS39ACoeMtzfOphKoPdjF5cdN4a5ndwGZPr9TqiI8tG4f73jtXLY1d7Lu5TYiIWP+tCo++9Yl/NtDL9LaneANr5pBTzJNa1ecUMh4bkdL/z4vbqghmXLef/ZCdh7u4onNzew63M1bT5nFvKlV3L9mD+09ST7xR4uJhEN855Em/vjkWVREQty/Zg/diRRLF05l1Y4WplRGuOL1C3i5pYffbNzH6xYM7NUwrSbGH4IufNGwccKsOjbsaaM6FubM46exp7WHE2fXAlAdC9O0v4PXLZjK/vZemvZ3cFMw9O3df/kG/m/T/v4TdT/4yNk0d2a67f7sE8vZvL+DF/d18OYlMzncFWfX4W5CIePPzl5IS1ecl/Z1cMbCqcyoiXH9e1/HA2v38idnHMeGPe0c7ozzllNm882HNrJ04bQBJwP/9U9fw8lzprDlQAfNnXHqKsf2L/2/f/kGPv79Z7ni9Qv4z0eaOGFWLRe9Zg7723p5/aLp/O3PnufCV8/hhd2ttHTFWdRQw+KZtazf08aJs2vpiqc41Bln/rQqTp4zhV2HuwZczxANG3WVUabXxKiOhdne3MWPPnbOmOqY7dp3nsqc+io27m1j0cyBvwrv+/Sbco7x3ucbl5/GoobM67515elMrxljN7MRfO1dr2XJ7LqCrvNojXgRk5ktB65197cHz68GcPevZpV5KCjzpJlFgL1AQ760zNFexCQi8ko22ouYRvN7+hlgiZktMrMYcCVw76Ay9wIfDKbfDTwyHvl2EREZnRF/wwU59E8DD5HpCnmru68zs+uAle5+L/Bd4Ptm1gQcIvMFICIiE2RUCTp3XwGsGDTvmqzpHuA9ha2aiIgcLfUZEhEpQwruIiJlSMFdRKQMKbiLiJQhBXcRkTI0YXdiMrMDwPajfPlMxmFogwmifZmcymlfoLz255W+L8e7e8NIhSYsuB8LM1s5miu0SoH2ZXIqp32B8tof7cvoKC0jIlKGFNxFRMpQqQb3mye6AgWkfZmcymlfoLz2R/syCiWZcxcRkfxKteUuIiJ5lFxwN7MLzWyTmTWZ2Rcnuj4jMbMFZvaoma03s3Vm9tlg/nQze9jMXgr+Tgvmm5l9O9i/NWa2NP8Wis/Mwmb2nJndHzxfZGZPB3X+STA0NGZWETxvCpY3TmS9BzOzqWZ2l5ltNLMNZra8VI+Lmf118Plaa2Z3mlllqRwXM7vVzPab2dqseWM+Dmb2waD8S2b2weG2NUH78s3gM7bGzP7XzKZmLbs62JdNZvb2rPnHHufcvWQeZIYc3gwsJnMDveeBUye6XiPUeS6wNJiuA14ETgW+AXwxmP9F4OvB9MXAA2Ru0XQO8PRE78Mw+/Q3wI+A+4PnPwWuDKZvBD4ZTP8lcGMwfSXwk4mu+6D9+B7w0WA6BkwtxeNC5jaXW4GqrOPxoVI5LsC5wFJgbda8MR0HYDqwJfg7LZieNkn25QIgEkx/PWtfTg1iWAWwKIht4ULFuQn/YI7xjVsOPJT1/Grg6omu1xj34RfA24BNwNxg3lxgUzB9E/C+rPL95SbDA5gP/AY4H7g/+Cc7mPXh7T9GZO4BsDyYjgTlbKL3IahPfRAQbdD8kjsuHLmH8fTgfb4feHspHRegcVBAHNNxAN4H3JQ1f0C5idyXQcv+FPhhMD0gfvUdl0LFuVJLywx3s+55E1SXMQt+/p4BPA3Mdvc9waK9wOxgerLv438AX+DIHblnAC3ungyeZ9d3wI3Tgb4bp08Gi4ADwG1BiukWM6uhBI+Lu+8G/g3YQebW563As5Tmcekz1uMwaY/PIH9B5pcHjPO+lFpwL1lmVgv8HPicu7dlL/PM1/Ok77ZkZpcA+9199Le6n7wiZH4+/7e7nwF0kvn536+Ejss04DIyX1jHATXAhRNaqQIqleMwEjP7MpAEfliM7ZVacN8NLMh6Pj+YN6mZWZRMYP+hu98dzN5nZnOD5XOB/cH8ybyPbwQuNbNtwI/JpGa+BUy1zI3RYWB9+/clWF4PNBezwnnsAna5+9PB87vIBPtSPC5vBba6+wF3TwB3kzlWpXhc+oz1OEzm44OZfQi4BHh/8GUF47wvpRbcR3Oz7knFzIzMPWY3uPv1WYuybyr+QTK5+L75fx70CjgHaM36eTqh3P1qd5/v7o1k3vtH3P39wKNkbowOQ/dlUt443d33AjvN7KRg1luA9ZTgcSGTjjnHzKqDz1vfvpTcccky1uPwEHCBmU0LfslcEMybcGZ2IZlU5qXu3pW16F7gyqD30iJgCfAHChXnJvIkylGerLiYTI+TzcCXJ7o+o6jvm8j8pFwDrA4eF5PJcf4GeAn4NTA9KG/Ad4L9ewFYNtH7kGO/zuNIb5nFwYeyCfgZUBHMrwyeNwXLF090vQftw+nAyuDY3EOml0VJHhfgn4CNwFrg+2R6YJTEcQHuJHOuIEHmF9VHjuY4kMlnNwWPD0+ifWkik0Pv+/+/Mav8l4N92QRclDX/mOOcrlAVESlDpZaWERGRUVBwFxEpQwruIiJlSMFdRKQMKbiLiJQhBXcRkTKk4C4iUoYU3EVEytD/B/OfpsGyZheNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores = pickle.load(open(\"tennis_scores.pkl\", \"rb\"))\n",
    "plt.plot(scores[\"scores\"])\n",
    "plt.plot(scores[\"avg_scores\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DDGP solved the environment in 1194 episodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = DDGPTrainer.Agent(state_size=state_size, action_size=action_size, random_seed=2)\n",
    "agent.actor_local.load_state_dict(torch.load(\"actor.torch\"))\n",
    "agent.critic_local.load_state_dict(torch.load(\"critic.torch\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score (max over agents) from episode: 2.600000038743019\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = [agent.act(state) for state in states]\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Score (max over agents) from episode: {}'.format(np.max(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Future work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Future work could include varying the hyperparameters to see which most affect training, trying multiple random seeds and reporting confidence intervals, trying reward scaling, and trying other algorithms such as PPO or A3C."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}