GAIL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imitation Learning via Reinforcecment Learning (ILRL)\n",
    "\n",
    "This project uses policy gradient methods such as PPO or TRPO along with Generative Adversarial Networks to achieve Imitation Learning on discrete gym environments.\n",
    "\n",
    "*Methodology used here is explained in **Generative Adversarial Imitation Learning (GAIL)** [[paper]](https://arxiv.org/pdf/1606.03476.pdf)*\n",
    "\n",
    "### Gist of it:\n",
    "\n",
    "**Given an Expert Policy as input the GAIL algorithm uses Policy Gradient method like PPO (in this case) to achieve Imitation Learning and in most cases the learned policy gets better than the input Expert Policy.**\n",
    "\n",
    "**For more information why we choose this methodology over other algorithms read the report:** [GAIL to solve Discrete environments](https://github.com/prasadchelsea33/imitation-learning/blob/master/Report/PPO(Prasad%20Madhale).pdf)\n",
    "\n",
    "## Overview of steps:\n",
    "1. **Run PPO algorithm** - run the PPO algorithm on an environment\n",
    "    1. *Create Actor-Critic* architecture which represents the two policy networks\n",
    "    2. *Code the PPO algorithm*\n",
    "    3. *Train an agent using the PPO algorithm*\n",
    "2. **Sample trajectories** - Sample some trajectories which represents the Expert Policy which we later use to train our agent for Imitation learning\n",
    "    1. *Restore the agent policy network weights*\n",
    "    2. *Sample some state and action using the expert policy*\n",
    "    3. *Save the sampled states and actions into csv files*\n",
    "3. **Test Expert Policy** - Test the learned expert policy to see if it satisfies the criteria for solving the environment (render the runs if you want)  \n",
    "4. **Train agent using GAIL for imitation learning** - given the expert trajectories as input we use Generative Adversarial Imitation Learning to train the agent\n",
    "    1. *Create a Discriminator* that differentiates between the Expert Policy and Generated Policy (same as in a conventional Generative Adversarial Network)\n",
    "    2. *Train the agent* to learn by imitating the given expert policy (uses GAIL algorithm)\n",
    "5. **Run Baseline implementations of PPO and TRPO to compare performance with our implementations**\n",
    "6. **Observe reward plots on Tensorboard** - the tensorboard contains the following plots :-\n",
    "    1. Our PPO implementation's Rewards and Lengths\n",
    "    2. Expert Policy Testing plot\n",
    "    3. GAIL reward and lengths plot (final agent)\n",
    "    4. Baseline reward, length and loss plots for comparison\n",
    "    \n",
    "**Note** - We can use any algorithm to obtain expert policy for GAIL agent training. Also, we can use other policy gradient methods like TRPO in place of PPO in the GAIL algorithm to obtain our imitating agent. However, the performance may vary depending on the algorithm choosen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import gym\n",
    "import numpy as np\n",
    "import copy\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A2C\n",
    "\n",
    "Actor-Critic Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class A2C:\n",
    "    def __init__(self, name : str, obs_space, action_space, sess):\n",
    "        self.observation = obs_space\n",
    "        self.action = action_space\n",
    "        self.scope_name = name\n",
    "        self.sess = sess\n",
    "        \n",
    "        with(tf.variable_scope(self.scope_name)):\n",
    "            # placeholder for inputs to the network\n",
    "            self.inputs = tf.placeholder(shape = [None] + list(self.observation.shape) , dtype = tf.float32)\n",
    "        \n",
    "            # build the two networks\n",
    "            self.build_network()\n",
    "            \n",
    "            # stochastic action\n",
    "            self.act = tf.multinomial(tf.log(self.act_probs),1)\n",
    "            self.act = tf.reshape(self.act, shape = [-1])\n",
    "            \n",
    "    def build_network(self):\n",
    "        # critic network gives out value prediction for the given inputs\n",
    "        with tf.variable_scope('critic', reuse = tf.AUTO_REUSE):\n",
    "            cout = tf.layers.dense(self.inputs, units = 16, activation = tf.tanh)\n",
    "            cout = tf.layers.dense(cout, units = 32, activation = tf.tanh)\n",
    "            self.value = tf.layers.dense(cout, units = 1, activation = None)\n",
    "        \n",
    "        # actor network spits out action probabilites\n",
    "        with tf.variable_scope('actor', reuse = tf.AUTO_REUSE):\n",
    "            aout = tf.layers.dense(self.inputs, 16,activation = tf.tanh)\n",
    "            aout = tf.layers.dense(aout, units = 32, activation = tf.tanh)\n",
    "            aout = tf.layers.dense(aout, units = 16, activation = tf.tanh)\n",
    "            self.act_probs = tf.layers.dense(aout, self.action.n , activation = tf.nn.softmax)\n",
    "                    \n",
    "    # get actions based on the given inputs\n",
    "    def get_action(self, inputs):\n",
    "        return self.sess.run(self.act, feed_dict = {self.inputs : inputs})\n",
    "    \n",
    "    # get value prediction for the given inputs\n",
    "    def get_value(self, inputs):\n",
    "        return self.sess.run(self.value, feed_dict = {self.inputs : inputs})\n",
    "    \n",
    "    # get all trainable variables required for policy update later\n",
    "    def trainable_vars(self):\n",
    "        return tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,self.scope_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PPO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PPO:\n",
    "    \n",
    "    def __init__(self, env, sess, eps = 0.2, gamma = 0.95, clip1=1, clip2=0.01, learning_rate = 5e-5):\n",
    "        self.sess = sess\n",
    "        self.eps = eps\n",
    "        self.gamma = gamma\n",
    "        self.learning_rate = learning_rate\n",
    "        self.clip1 = clip1\n",
    "        self.clip2 = clip2\n",
    "        self.act_clip_max = 1\n",
    "        self.act_clip_min = 1e-10\n",
    "        \n",
    "        self.pi = A2C(\"pi\", env.observation_space, env.action_space, self.sess)\n",
    "        self.old_pi = A2C(\"old_pi\", env.observation_space, env.action_space, self.sess)\n",
    "        \n",
    "        self.pi_trainable_params = self.pi.trainable_vars()\n",
    "        self.old_pi_trainable_params = self.old_pi.trainable_vars()\n",
    "        \n",
    "        with tf.variable_scope('update_policy'):\n",
    "            self.update_ops = [old_pi_vals.assign(pi_vals) for pi_vals, old_pi_vals in zip(self.pi_trainable_params, self.old_pi_trainable_params)]\n",
    "        \n",
    "        with tf.variable_scope('training_inputs'):\n",
    "            self.actions = tf.placeholder(shape = [None], dtype=tf.int32)\n",
    "            self.rewards = tf.placeholder(shape = [None], dtype=tf.float32)\n",
    "            self.v_next = tf.placeholder(shape = [None], dtype=tf.float32)\n",
    "            self.adv = tf.placeholder(shape = [None], dtype=tf.float32)\n",
    "            \n",
    "        act_probs = self.hotify_action(self.pi.act_probs)\n",
    "        act_old_probs = self.hotify_action(self.old_pi.act_probs)\n",
    "            \n",
    "        with tf.variable_scope(\"loss\"):\n",
    "        \n",
    "            # loss calculations\n",
    "            clipped_act_probs = tf.log(tf.clip_by_value(act_probs, self.act_clip_min, self.act_clip_max))\n",
    "            clipped_old_act_probs = tf.log(tf.clip_by_value(act_old_probs, self.act_clip_min, self.act_clip_max))\n",
    "            \n",
    "            ratio = tf.exp(clipped_act_probs - clipped_old_act_probs)\n",
    "    \n",
    "            clipped_ratio = tf.clip_by_value(ratio, 1 -self.eps, 1 + self.eps)\n",
    "            surrogate = tf.multiply(ratio, self.adv)\n",
    "            surrogate_clipped = tf.multiply(clipped_ratio, self.adv)\n",
    "            \n",
    "            clipped_loss = tf.minimum(surrogate, surrogate_clipped)\n",
    "            clipped_loss = tf.reduce_mean(clipped_loss)\n",
    "            \n",
    "            entropy = -tf.reduce_sum(self.pi.act_probs * tf.log(tf.clip_by_value(self.pi.act_probs, self.act_clip_min, self.act_clip_max)), 1)\n",
    "            entropy = tf.reduce_mean(entropy, 0)\n",
    "            \n",
    "            value = self.pi.value\n",
    "            error = self.rewards + self.gamma * self.v_next\n",
    "            loss_value = tf.squared_difference(error, value)\n",
    "            loss_value = tf.reduce_sum(loss_value)\n",
    "            \n",
    "            self.loss = -(clipped_loss - self.clip1 * loss_value + self.clip2 * entropy)\n",
    "            self.loss_plot = tf.summary.scalar('loss', self.loss)\n",
    "        \n",
    "        opt = tf.train.AdamOptimizer(self.learning_rate, epsilon=1e-5)\n",
    "        self.gradients = opt.compute_gradients(self.loss, var_list = self.pi_trainable_params)\n",
    "        self.train_op = opt.minimize(self.loss, var_list = self.pi_trainable_params)\n",
    "\n",
    "    # get action given state\n",
    "    def get_action(self, inputs):\n",
    "        return self.pi.get_action(inputs)\n",
    "\n",
    "    # get value estimate given state\n",
    "    def get_value(self, inputs):\n",
    "        return self.pi.get_value(inputs)\n",
    "    \n",
    "    # update old policy network to the new network parameters    \n",
    "    def update_old_policy(self):\n",
    "        self.sess.run(self.update_ops)\n",
    "    \n",
    "    def train_policy(self, inputs, actions, rewards, v_next, advantages):\n",
    "        self.sess.run(self.train_op, feed_dict = {self.pi.inputs : inputs,\n",
    "                                                  self.old_pi.inputs : inputs,\n",
    "                                                  self.actions: actions,\n",
    "                                                  self.rewards: rewards,\n",
    "                                                  self.v_next: v_next, \n",
    "                                                  self.adv: advantages})\n",
    "    \n",
    "    # get advantage estimates\n",
    "    def get_gaes(self, rewards, v_preds, v_preds_next):\n",
    "        deltas = [r_t + self.gamma * v_next - v for r_t, v_next, v in zip(rewards, v_preds_next, v_preds)]\n",
    "        gaes = copy.deepcopy(deltas)\n",
    "        for t in reversed(range(len(gaes) - 1)):\n",
    "            gaes[t] = gaes[t] + self.gamma * gaes[t + 1]\n",
    "        return gaes\n",
    "    \n",
    "    def hotify_action(self, action):\n",
    "        action *= tf.one_hot(self.actions, action.shape[1])\n",
    "        action = tf.reduce_sum(action, 1)\n",
    "        return action\n",
    "    \n",
    "    def get_entropy(self, act_probs):\n",
    "        entropy = -tf.reduce_sum(act_probs * tf.log(tf.clip_by_value(act_probs, self.act_clip_min, self.act_clip_max)), 1)\n",
    "        return tf.reduce_mean(entropy, 0)        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_train(num_epochs, ppo, obs, actions, adv, rewards, v_preds_next):\n",
    "    transitions = [obs, actions, adv, rewards, v_preds_next]\n",
    "    \n",
    "    for epochs in range(num_epochs):\n",
    "        # random sampling\n",
    "        index = indices = np.random.randint(0, obs.shape[0], size = 32)\n",
    "        samples = [np.take(transition, index, axis=0) for transition in transitions]\n",
    "\n",
    "        # training\n",
    "        ppo.train_policy(inputs = samples[0],\n",
    "                     actions = samples[1],\n",
    "                     advantages = samples[2],\n",
    "                     rewards = samples[3],\n",
    "                     v_next = samples[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(input):\n",
    "    return np.stack([input])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def z_score(input):\n",
    "    return (input - input.mean())/input.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterations = 8000\n",
    "num_of_epochs = 6\n",
    "success_threshold = 195"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PPO runner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/prasad/anaconda3/envs/tf/lib/python3.6/site-packages/gym/envs/registration.py:14: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.\n",
      "  result = entry_point.load(False)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode number: 0\n",
      "Episode number: 1000\n",
      "Episode number: 2000\n",
      "Episode number: 3000\n",
      "Episode number: 4000\n",
      "Episode number: 5000\n",
      "Episode number: 6000\n",
      "Episode number: 7000\n",
      "Threshold reached so we end training early. Model saved at trained_model/ppo.ckpt\n",
      "Model saved at trained_model/ppo.ckpt\n",
      "Training complete. Check the tensorboard for plots\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()  \n",
    "\n",
    "with tf.Session() as sess:\n",
    "    tensor_plot = tf.summary.FileWriter('log/ppo', graph = sess.graph)\n",
    "    env = gym.make('CartPole-v0')\n",
    "    ppo = PPO(env, sess)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    state = env.reset()\n",
    "    succ_run = 0\n",
    "    \n",
    "    # save trained model\n",
    "    saver = tf.train.Saver()\n",
    "    \n",
    "    for i in range(iterations+1):\n",
    "        obs = []\n",
    "        actions = []\n",
    "        rewards = []\n",
    "        values = []\n",
    "        length = 0\n",
    "        \n",
    "        if i % 1000 == 0:\n",
    "                print('Episode number: {}'.format(i))\n",
    "             \n",
    "        while True:\n",
    "            length += 1\n",
    "            \n",
    "            state = preprocess(state)\n",
    "            \n",
    "            action = ppo.get_action(state)\n",
    "            action = np.asscalar(action)\n",
    "            \n",
    "            value = ppo.get_value(state)\n",
    "            value = np.asscalar(value)\n",
    "            \n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            obs.append(state)\n",
    "            actions.append(action)\n",
    "            rewards.append(reward)\n",
    "            values.append(value)\n",
    "               \n",
    "            if done:\n",
    "                next_state = preprocess(next_state)\n",
    "                next_value = ppo.get_value(next_state)\n",
    "                next_value = np.asscalar(next_value)\n",
    "                v_preds_next = values[1:] + [next_value]\n",
    "                state = env.reset()\n",
    "                break\n",
    "            else:\n",
    "                state = next_state\n",
    "        \n",
    "        tensor_plot.add_summary(tf.Summary(value = [tf.Summary.Value(tag=\"my_ppo_episode_rewards\", simple_value = sum(rewards))]), i)\n",
    "        tensor_plot.add_summary(tf.Summary(value = [tf.Summary.Value(tag=\"my_ppo_episode_length\", simple_value = length)]), i)\n",
    "                \n",
    "        if sum(rewards) >= success_threshold:\n",
    "            succ_run += 1\n",
    "            \n",
    "            if succ_run >= 100:\n",
    "                path = saver.save(sess, \"trained_model/ppo.ckpt\")\n",
    "                print(\"Threshold reached so we end training early. Model saved at {}\".format(path))\n",
    "                tensor_plot.close()\n",
    "                break\n",
    "        else:\n",
    "            succ_run = 0\n",
    "            \n",
    "        adv = ppo.get_gaes(rewards, values, v_preds_next)\n",
    "\n",
    "        obs = np.reshape(obs, newshape=(-1,) + env.observation_space.shape)\n",
    "        \n",
    "        rewards = np.array(rewards)\n",
    "        v_preds_next = np.array(v_preds_next)\n",
    "        actions = np.array(actions)\n",
    "        adv = z_score(np.array(adv))\n",
    "                \n",
    "        ppo.update_old_policy()\n",
    "\n",
    "        epoch_train(num_of_epochs, ppo, obs, actions, adv, rewards, v_preds_next)\n",
    "    \n",
    "    path = saver.save(sess, \"trained_model/ppo.ckpt\")\n",
    "    print(\"Model saved at {}\".format(path))\n",
    "    tensor_plot.close()\n",
    "env.close()\n",
    "\n",
    "print(\"Training complete. Check the tensorboard for plots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract Expert Trajectories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_to_csv(file_path, data):\n",
    "    with open(file_path, 'ab') as f_handle:\n",
    "        np.savetxt(f_handle, data, fmt='%s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from trained_model/ppo.ckpt\n",
      "Model restored!\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "num_samples = 20\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    env = gym.make('CartPole-v0')\n",
    "    ppo = PPO(env, sess)\n",
    "    \n",
    "    saver = tf.train.Saver()\n",
    "    \n",
    "    saver.restore(sess, \"trained_model/ppo.ckpt\")\n",
    "    print(\"Model restored!\")\n",
    "    \n",
    "    state = env.reset()\n",
    "    \n",
    "    for itr in range(num_samples):\n",
    "        obs = []\n",
    "        actions = []\n",
    "        length = 0\n",
    "        done = False\n",
    "        \n",
    "        while True:\n",
    "            length += 1\n",
    "            state = preprocess(state)\n",
    "            \n",
    "            action = ppo.get_action(state)\n",
    "            action = np.asscalar(action)\n",
    "            \n",
    "            # take action\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            obs.append(state)\n",
    "            actions.append(action)\n",
    "            \n",
    "            if done:\n",
    "                state = env.reset()\n",
    "                break\n",
    "            else:\n",
    "                state = next_state\n",
    "        \n",
    "        obs = np.reshape(obs, newshape=[-1] + list(env.observation_space.shape))\n",
    "        actions = np.array(actions)\n",
    "        \n",
    "        save_to_csv(\"trajectories/expert_obs.csv\", obs)\n",
    "        save_to_csv(\"trajectories/expert_actions.csv\", actions)\n",
    "        \n",
    "env.close()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Render trained model (Demo)\n",
    "## Condition for environment to be considered as solved:\n",
    "Considered solved when the average reward is **greater than or equal to 195.0 over 100 consecutive trials**.\n",
    "So, we check our test for the above condition and break the testing once this condition is satisfied.\n",
    "\n",
    "Observe the rewards obtained in the demo on tensorboard under the tag *'test_episode_rewards'*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from trained_model/ppo.ckpt\n",
      "Model restored!\n",
      "Episode: 0\n",
      "Episode: 50\n",
      "Episode: 100\n",
      "Solved at Episode: 100\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "success_threshold = 195\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    tensor_plot = tf.summary.FileWriter('log/ppo', graph = sess.graph)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    env = gym.make('CartPole-v0')\n",
    "    ppo = PPO(env, sess)\n",
    "    \n",
    "    saver = tf.train.Saver()\n",
    "    succ_runs = 0\n",
    "    \n",
    "    saver.restore(sess, \"trained_model/ppo.ckpt\")\n",
    "    print(\"Model restored!\")\n",
    "            \n",
    "    state = env.reset()\n",
    "    done = False\n",
    "        \n",
    "    for itr in range(1001):\n",
    "        rewards = []\n",
    "        \n",
    "        if itr % 50 == 0:\n",
    "            print('Episode: {}'.format(itr))\n",
    "            \n",
    "        while True:\n",
    "            state = preprocess(state)\n",
    "            # if you want to render the solved environment consider uncommenting the time.sleep statement below to facilitate slow rendering \n",
    "            # env.render()\n",
    "\n",
    "            action = ppo.get_action(state)\n",
    "            action = np.asscalar(action)\n",
    "\n",
    "            state,reward,done,_ = env.step(action)\n",
    "            rewards.append(reward)\n",
    "            \n",
    "            # to demonstrate slow rendering\n",
    "            # time.sleep(0.025)\n",
    "            \n",
    "            if done:\n",
    "                state = env.reset()\n",
    "                break\n",
    "                \n",
    "        tensor_plot.add_summary(tf.Summary(value = [tf.Summary.Value(tag=\"test_episode_rewards\", simple_value = sum(rewards))]), itr) \n",
    "        \n",
    "        if sum(rewards) >= success_threshold:\n",
    "            succ_runs  += 1\n",
    "            \n",
    "            if succ_runs > 100:\n",
    "                print(\"Solved at Episode: {}\".format(itr))\n",
    "                break\n",
    "        else:\n",
    "            succ_runs = 0\n",
    "                    \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discriminator\n",
    "Discriminator represensted by a Generative Adversarial Network to discriminate between the generated trajectory and the expert trajectory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Discriminator:\n",
    "    def __init__(self, name : str, env, sess):\n",
    "        self.sess = sess\n",
    "        self.env = env\n",
    "        self.scope_name = name\n",
    "        \n",
    "        with tf.variable_scope('discriminator'):\n",
    "            with(tf.variable_scope('expert_inputs')):\n",
    "                self.expert_act = tf.placeholder(shape = [None], dtype=tf.int32)\n",
    "                self.expert_state = tf.placeholder(shape = [None] + list(self.env.observation_space.shape), dtype=tf.float32)\n",
    "                self.expert_one_hot_a = tf.one_hot(self.expert_act, depth = self.env.action_space.n)\n",
    "                self.expert_one_hot_a = tf.random_normal(tf.shape(self.expert_one_hot_a), mean = 0.2, stddev=0.1, dtype = tf.float32)/1.2\n",
    "                self.expert_SA = tf.concat([self.expert_state, self.expert_one_hot_a], axis=1)\n",
    "\n",
    "            with(tf.variable_scope('agent_inputs')):\n",
    "                self.agent_act = tf.placeholder(shape = [None], dtype=tf.int32)\n",
    "                self.agent_state = tf.placeholder(shape = [None] + list(self.env.observation_space.shape), dtype=tf.float32)\n",
    "                self.agent_one_hot_a = tf.one_hot(self.agent_act, depth = self.env.action_space.n)\n",
    "                self.agent_one_hot_a = tf.random_normal(tf.shape(self.agent_one_hot_a), mean = 0.2, stddev = 0.1, dtype = tf.float32)/1.2\n",
    "                self.agent_SA = tf.concat([self.agent_state, self.agent_one_hot_a], axis=1)\n",
    "\n",
    "            with tf.variable_scope('network') as scope:\n",
    "                self.prob1 = self.build_network(input = self.expert_SA)\n",
    "                scope.reuse_variables()\n",
    "                self.prob2 = self.build_network(input = self.agent_SA)\n",
    "\n",
    "            with tf.variable_scope('loss'):            \n",
    "                expert_loss = tf.reduce_mean(tf.log(tf.clip_by_value(self.prob1, 0.01, 1)))\n",
    "                agent_loss = tf.reduce_mean(tf.log(tf.clip_by_value(1 - self.prob2, 0.01, 1)))\n",
    "                \n",
    "                # as we have to perform maximizing\n",
    "                loss = -(expert_loss + agent_loss)\n",
    "                # gradient ascent is achieved by minimizing negative of the loss which is the same as maximizing the calculated loss\n",
    "                self.train_op = tf.train.AdamOptimizer().minimize(loss)\n",
    "\n",
    "            self.rewards = tf.log(tf.clip_by_value(self.prob2, 1e-10, 1))\n",
    "        \n",
    "    \n",
    "    def train_dis(self, expert_s, expert_a, agent_s, agent_a):\n",
    "        return self.sess.run(self.train_op, feed_dict = {self.expert_state: expert_s,\n",
    "                                                     self.expert_act: expert_a,\n",
    "                                                     self.agent_state: agent_s,\n",
    "                                                     self.agent_act: agent_a})\n",
    "    \n",
    "    def get_rewards(self, agent_s, agent_a):\n",
    "        return self.sess.run(self.rewards, feed_dict = {self.agent_state: agent_s,\n",
    "                                                       self.agent_act: agent_a})\n",
    "    \n",
    "    def build_network(self, input):\n",
    "        prob = tf.layers.dense(input, units = 20,activation = tf.nn.leaky_relu, name = 'layer1')\n",
    "        prob = tf.layers.dense(prob, units = 20, activation = tf.nn.leaky_relu, name = 'layer2')\n",
    "        prob = tf.layers.dense(prob, units = 20, activation = tf.nn.leaky_relu, name = 'layer3')\n",
    "        prob = tf.layers.dense(prob, units = 1, activation = tf.sigmoid, name = 'prob')       \n",
    "        return prob\n",
    "    \n",
    "    def trainable_vars(self):\n",
    "        return tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,self.scope_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generative Adversarial Imitation Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyperparameters\n",
    "iterations = 15000\n",
    "dis_epochs = 3\n",
    "success_threshold = 195"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_train(num_epochs, ppo, obs, actions, adv, rewards, v_preds_next):\n",
    "    transitions = [obs, actions, adv, rewards, v_preds_next]\n",
    "    \n",
    "    for epochs in range(num_epochs):\n",
    "        # random sampling\n",
    "        index = indices = np.random.randint(0, obs.shape[0], size = 32)\n",
    "        samples = [np.take(transition, index, axis=0) for transition in transitions]\n",
    "\n",
    "        # training\n",
    "        ppo.train_policy(inputs = samples[0],\n",
    "                     actions = samples[1],\n",
    "                     advantages = samples[2],\n",
    "                     rewards = samples[3],\n",
    "                     v_next = samples[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode number: 0\n",
      "Episode number: 1000\n",
      "Episode number: 2000\n",
      "Episode number: 3000\n",
      "Episode number: 4000\n",
      "Episode number: 5000\n",
      "Episode number: 6000\n",
      "Episode number: 7000\n",
      "Episode number: 8000\n",
      "Episode number: 9000\n",
      "Episode number: 10000\n",
      "Episode number: 11000\n",
      "Episode number: 12000\n",
      "Episode number: 13000\n",
      "Episode number: 14000\n",
      "Episode number: 15000\n",
      "GAIL training complete!\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()  \n",
    "\n",
    "with tf.Session() as sess:\n",
    "    tensor_plot = tf.summary.FileWriter('log/ppo', graph = sess.graph)\n",
    "    env = gym.make('CartPole-v0')\n",
    "\n",
    "    ppo = PPO(env, sess)\n",
    "    gan = Discriminator(\"Discriminator\", env, sess)\n",
    "\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    expert_state = np.genfromtxt('trajectories/expert_obs.csv')\n",
    "    expert_act = np.genfromtxt('trajectories/expert_actions.csv', dtype = np.int32)\n",
    "    \n",
    "    state = env.reset()\n",
    "    succ_runs = 0\n",
    "    \n",
    "    # save trained model\n",
    "    saver = tf.train.Saver()\n",
    "    \n",
    "    for i in range(iterations+1):\n",
    "        obs = []\n",
    "        actions = []\n",
    "        values = []\n",
    "        length = 0\n",
    "        rewards = []\n",
    "        \n",
    "        if i % 1000 == 0:\n",
    "            print('Episode number: {}'.format(i))\n",
    "             \n",
    "        while True:\n",
    "            length += 1\n",
    "            \n",
    "            state = preprocess(state)\n",
    "            \n",
    "            action = ppo.get_action(state)\n",
    "            action = np.asscalar(action)\n",
    "            \n",
    "            value = ppo.get_value(state)\n",
    "            value = np.asscalar(value)\n",
    "            \n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            obs.append(state)\n",
    "            actions.append(action)\n",
    "            values.append(value)\n",
    "            reward = rewards.append(reward)\n",
    "            \n",
    "            if done:\n",
    "                next_state = preprocess(next_state)\n",
    "                next_value = ppo.get_value(next_state)\n",
    "                next_value = np.asscalar(next_value)\n",
    "                v_preds_next = values[1:] + [next_value]\n",
    "                state = env.reset()\n",
    "                break\n",
    "            else:\n",
    "                state = next_state\n",
    "                \n",
    "        tensor_plot.add_summary(tf.Summary(value = [tf.Summary.Value(tag=\"gail_episode_rewards\", simple_value = sum(rewards))]), i)\n",
    "        tensor_plot.add_summary(tf.Summary(value = [tf.Summary.Value(tag=\"gail_episode_length\", simple_value = length)]), i)\n",
    "        \n",
    "        if sum(rewards) >= success_threshold:\n",
    "            succ_runs  += 1\n",
    "            \n",
    "            if succ_runs > 100:\n",
    "                print(\"Solved at Episode: {}\".format(itr))\n",
    "                print(\"GAIL training completed early!\")\n",
    "                break\n",
    "        else:\n",
    "            succ_runs = 0\n",
    "            \n",
    "        # preprocess inputs\n",
    "        obs = np.reshape(obs, newshape=(-1,) + env.observation_space.shape)\n",
    "        v_preds_next = np.array(v_preds_next)\n",
    "        actions = np.array(actions)\n",
    "        \n",
    "        # train discriminator\n",
    "        for _ in range(dis_epochs):\n",
    "            gan.train_dis(expert_state, expert_act, obs, actions)\n",
    "        \n",
    "        gan_rewards = gan.get_rewards(obs, actions)\n",
    "        gan_rewards = np.reshape(gan_rewards, newshape=(-1,))\n",
    "        \n",
    "        adv = ppo.get_gaes(gan_rewards, values, v_preds_next)\n",
    "                \n",
    "        ppo.update_old_policy()\n",
    "\n",
    "        epoch_train(dis_epochs, ppo, obs, actions, adv, gan_rewards, v_preds_next)\n",
    "    \n",
    "    print(\"GAIL training complete!\")\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baselines Implementation of PPO\n",
    "\n",
    "From the stable-baselines repo to compare my implementation with the baselines implementation of PPO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating environment from the given name, wrapped in a DummyVecEnv.\n",
      "********** Iteration 0 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00179 |      -0.00693 |      76.23267 |       0.00015 |       0.69303\n",
      "     -0.00902 |      -0.00692 |      74.91623 |       0.00110 |       0.69216\n",
      "     -0.01602 |      -0.00690 |      73.50521 |       0.00339 |       0.68998\n",
      "     -0.02153 |      -0.00685 |      72.08931 |       0.00834 |       0.68524\n",
      "Evaluating losses...\n",
      "     -0.02223 |      -0.00682 |      71.12242 |       0.01187 |       0.68188\n",
      "----------------------------------\n",
      "| EpLenMean       | 21.9         |\n",
      "| EpRewMean       | 21.9         |\n",
      "| EpThisIter      | 11           |\n",
      "| EpisodesSoFar   | 11           |\n",
      "| TimeElapsed     | 0.374        |\n",
      "| TimestepsSoFar  | 256          |\n",
      "| ev_tdlam_before | -0.0104      |\n",
      "| loss_ent        | 0.6818789    |\n",
      "| loss_kl         | 0.011868857  |\n",
      "| loss_pol_entpen | -0.006818789 |\n",
      "| loss_pol_surr   | -0.022231959 |\n",
      "| loss_vf_loss    | 71.12242     |\n",
      "----------------------------------\n",
      "********** Iteration 1 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00206 |      -0.00682 |      90.79230 |       0.00012 |       0.68200\n",
      "     -0.00931 |      -0.00675 |      88.93327 |       0.00153 |       0.67507\n",
      "     -0.01555 |      -0.00664 |      86.53964 |       0.00604 |       0.66359\n",
      "     -0.01317 |      -0.00652 |      83.87897 |       0.01213 |       0.65223\n",
      "Evaluating losses...\n",
      "     -0.01320 |      -0.00649 |      81.83924 |       0.01419 |       0.64884\n",
      "-----------------------------------\n",
      "| EpLenMean       | 23            |\n",
      "| EpRewMean       | 23            |\n",
      "| EpThisIter      | 10            |\n",
      "| EpisodesSoFar   | 21            |\n",
      "| TimeElapsed     | 0.549         |\n",
      "| TimestepsSoFar  | 527           |\n",
      "| ev_tdlam_before | -0.0592       |\n",
      "| loss_ent        | 0.648842      |\n",
      "| loss_kl         | 0.014189997   |\n",
      "| loss_pol_entpen | -0.0064884196 |\n",
      "| loss_pol_surr   | -0.013196163  |\n",
      "| loss_vf_loss    | 81.83924      |\n",
      "-----------------------------------\n",
      "********** Iteration 2 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     7.91e-05 |      -0.00660 |     134.20270 |      3.21e-05 |       0.65987\n",
      "     -0.00130 |      -0.00656 |     128.28532 |       0.00024 |       0.65629\n",
      "     -0.00271 |      -0.00652 |     121.24988 |       0.00078 |       0.65160\n",
      "     -0.00379 |      -0.00644 |     113.51073 |       0.00218 |       0.64431\n",
      "Evaluating losses...\n",
      "     -0.00493 |      -0.00639 |     108.49438 |       0.00344 |       0.63926\n",
      "-----------------------------------\n",
      "| EpLenMean       | 26.4          |\n",
      "| EpRewMean       | 26.4          |\n",
      "| EpThisIter      | 8             |\n",
      "| EpisodesSoFar   | 29            |\n",
      "| TimeElapsed     | 0.721         |\n",
      "| TimestepsSoFar  | 813           |\n",
      "| ev_tdlam_before | 0.0153        |\n",
      "| loss_ent        | 0.6392576     |\n",
      "| loss_kl         | 0.0034351444  |\n",
      "| loss_pol_entpen | -0.006392576  |\n",
      "| loss_pol_surr   | -0.0049295574 |\n",
      "| loss_vf_loss    | 108.494385    |\n",
      "-----------------------------------\n",
      "********** Iteration 3 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00049 |      -0.00650 |     125.53104 |      3.90e-05 |       0.65002\n",
      "     -0.00218 |      -0.00645 |     116.98804 |       0.00035 |       0.64484\n",
      "     -0.00368 |      -0.00638 |     107.91806 |       0.00119 |       0.63848\n",
      "     -0.00462 |      -0.00634 |      98.35778 |       0.00216 |       0.63356\n",
      "Evaluating losses...\n",
      "     -0.00487 |      -0.00631 |      92.07527 |       0.00276 |       0.63109\n",
      "-----------------------------------\n",
      "| EpLenMean       | 28.9          |\n",
      "| EpRewMean       | 28.9          |\n",
      "| EpThisIter      | 5             |\n",
      "| EpisodesSoFar   | 34            |\n",
      "| TimeElapsed     | 0.887         |\n",
      "| TimestepsSoFar  | 1071          |\n",
      "| ev_tdlam_before | -0.0145       |\n",
      "| loss_ent        | 0.63108945    |\n",
      "| loss_kl         | 0.0027620236  |\n",
      "| loss_pol_entpen | -0.0063108937 |\n",
      "| loss_pol_surr   | -0.004873166  |\n",
      "| loss_vf_loss    | 92.07527      |\n",
      "-----------------------------------\n",
      "********** Iteration 4 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00078 |      -0.00644 |     183.33702 |      6.58e-05 |       0.64360\n",
      "     -0.01096 |      -0.00635 |     167.43509 |       0.00086 |       0.63458\n",
      "     -0.01675 |      -0.00622 |     150.75595 |       0.00336 |       0.62167\n",
      "     -0.01568 |      -0.00612 |     133.72420 |       0.00606 |       0.61250\n",
      "Evaluating losses...\n",
      "     -0.01545 |      -0.00610 |     123.30197 |       0.00703 |       0.60974\n",
      "-----------------------------------\n",
      "| EpLenMean       | 31.4          |\n",
      "| EpRewMean       | 31.4          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 36            |\n",
      "| TimeElapsed     | 1.05          |\n",
      "| TimestepsSoFar  | 1367          |\n",
      "| ev_tdlam_before | -0.0674       |\n",
      "| loss_ent        | 0.6097411     |\n",
      "| loss_kl         | 0.007034207   |\n",
      "| loss_pol_entpen | -0.0060974113 |\n",
      "| loss_pol_surr   | -0.015447583  |\n",
      "| loss_vf_loss    | 123.30197     |\n",
      "-----------------------------------\n",
      "********** Iteration 5 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00088 |      -0.00613 |     141.66794 |      4.85e-05 |       0.61282\n",
      "     -0.00485 |      -0.00605 |     126.79100 |       0.00082 |       0.60471\n",
      "     -0.01040 |      -0.00596 |     113.48085 |       0.00260 |       0.59632\n",
      "     -0.01158 |      -0.00593 |     101.16899 |       0.00379 |       0.59285\n",
      "Evaluating losses...\n",
      "     -0.01164 |      -0.00592 |      94.19684 |       0.00409 |       0.59176\n",
      "----------------------------------\n",
      "| EpLenMean       | 37.2         |\n",
      "| EpRewMean       | 37.2         |\n",
      "| EpThisIter      | 3            |\n",
      "| EpisodesSoFar   | 39           |\n",
      "| TimeElapsed     | 1.21         |\n",
      "| TimestepsSoFar  | 1774         |\n",
      "| ev_tdlam_before | -0.0224      |\n",
      "| loss_ent        | 0.59176385   |\n",
      "| loss_kl         | 0.0040874816 |\n",
      "| loss_pol_entpen | -0.005917638 |\n",
      "| loss_pol_surr   | -0.011635844 |\n",
      "| loss_vf_loss    | 94.19684     |\n",
      "----------------------------------\n",
      "********** Iteration 6 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00249 |      -0.00610 |     120.56087 |      4.60e-05 |       0.61000\n",
      "     -0.00999 |      -0.00605 |     109.29431 |       0.00103 |       0.60467\n",
      "     -0.01201 |      -0.00599 |      99.15992 |       0.00378 |       0.59904\n",
      "     -0.01030 |      -0.00596 |      90.81062 |       0.00624 |       0.59591\n",
      "Evaluating losses...\n",
      "     -0.01065 |      -0.00596 |      85.96743 |       0.00660 |       0.59581\n",
      "-----------------------------------\n",
      "| EpLenMean       | 41.6          |\n",
      "| EpRewMean       | 41.6          |\n",
      "| EpThisIter      | 3             |\n",
      "| EpisodesSoFar   | 42            |\n",
      "| TimeElapsed     | 1.37          |\n",
      "| TimestepsSoFar  | 2116          |\n",
      "| ev_tdlam_before | -0.0146       |\n",
      "| loss_ent        | 0.5958133     |\n",
      "| loss_kl         | 0.006597397   |\n",
      "| loss_pol_entpen | -0.0059581324 |\n",
      "| loss_pol_surr   | -0.010654109  |\n",
      "| loss_vf_loss    | 85.96743      |\n",
      "-----------------------------------\n",
      "********** Iteration 7 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00063 |      -0.00605 |     149.72021 |      2.25e-05 |       0.60524\n",
      "     -0.00426 |      -0.00598 |     137.32722 |       0.00056 |       0.59831\n",
      "     -0.00613 |      -0.00590 |     125.23835 |       0.00181 |       0.58963\n",
      "     -0.00495 |      -0.00582 |     114.42767 |       0.00325 |       0.58201\n",
      "Evaluating losses...\n",
      "     -0.00523 |      -0.00581 |     108.33158 |       0.00351 |       0.58085\n",
      "-----------------------------------\n",
      "| EpLenMean       | 44.6          |\n",
      "| EpRewMean       | 44.6          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 44            |\n",
      "| TimeElapsed     | 1.54          |\n",
      "| TimestepsSoFar  | 2417          |\n",
      "| ev_tdlam_before | -0.00429      |\n",
      "| loss_ent        | 0.5808452     |\n",
      "| loss_kl         | 0.0035117045  |\n",
      "| loss_pol_entpen | -0.0058084517 |\n",
      "| loss_pol_surr   | -0.0052278526 |\n",
      "| loss_vf_loss    | 108.33158     |\n",
      "-----------------------------------\n",
      "********** Iteration 8 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "    -6.80e-05 |      -0.00588 |     127.85952 |       0.00023 |       0.58810\n",
      "     -0.00661 |      -0.00583 |     120.36436 |       0.00194 |       0.58299\n",
      "     -0.00691 |      -0.00577 |     112.62624 |       0.00293 |       0.57665\n",
      "     -0.00851 |      -0.00571 |     105.55313 |       0.00254 |       0.57147\n",
      "Evaluating losses...\n",
      "     -0.00939 |      -0.00569 |     101.62038 |       0.00227 |       0.56864\n",
      "-----------------------------------\n",
      "| EpLenMean       | 47.4          |\n",
      "| EpRewMean       | 47.4          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 46            |\n",
      "| TimeElapsed     | 1.71          |\n",
      "| TimestepsSoFar  | 2760          |\n",
      "| ev_tdlam_before | -0.00112      |\n",
      "| loss_ent        | 0.5686421     |\n",
      "| loss_kl         | 0.002273471   |\n",
      "| loss_pol_entpen | -0.0056864205 |\n",
      "| loss_pol_surr   | -0.009393175  |\n",
      "| loss_vf_loss    | 101.620384    |\n",
      "-----------------------------------\n",
      "********** Iteration 9 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     8.56e-06 |      -0.00593 |     140.86620 |      1.15e-05 |       0.59316\n",
      "     -0.00441 |      -0.00588 |     132.18939 |       0.00039 |       0.58776\n",
      "     -0.01148 |      -0.00580 |     123.14722 |       0.00239 |       0.57951\n",
      "     -0.01397 |      -0.00573 |     115.48564 |       0.00633 |       0.57266\n",
      "Evaluating losses...\n",
      "     -0.01444 |      -0.00569 |     111.15245 |       0.00943 |       0.56934\n",
      "----------------------------------\n",
      "| EpLenMean       | 53.2         |\n",
      "| EpRewMean       | 53.2         |\n",
      "| EpThisIter      | 2            |\n",
      "| EpisodesSoFar   | 48           |\n",
      "| TimeElapsed     | 1.9          |\n",
      "| TimestepsSoFar  | 3141         |\n",
      "| ev_tdlam_before | 0.196        |\n",
      "| loss_ent        | 0.56934327   |\n",
      "| loss_kl         | 0.009427521  |\n",
      "| loss_pol_entpen | -0.005693433 |\n",
      "| loss_pol_surr   | -0.014437607 |\n",
      "| loss_vf_loss    | 111.15245    |\n",
      "----------------------------------\n",
      "********** Iteration 10 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "      0.00127 |      -0.00555 |     142.72057 |      9.10e-05 |       0.55503\n",
      "      0.00135 |      -0.00553 |     136.43993 |       0.00031 |       0.55320\n",
      "      0.00041 |      -0.00556 |     130.22527 |       0.00038 |       0.55616\n",
      "     -0.00097 |      -0.00561 |     124.50304 |       0.00043 |       0.56135\n",
      "Evaluating losses...\n",
      "     -0.00258 |      -0.00566 |     121.05017 |       0.00051 |       0.56565\n",
      "-----------------------------------\n",
      "| EpLenMean       | 56.2          |\n",
      "| EpRewMean       | 56.2          |\n",
      "| EpThisIter      | 1             |\n",
      "| EpisodesSoFar   | 49            |\n",
      "| TimeElapsed     | 2.06          |\n",
      "| TimestepsSoFar  | 3403          |\n",
      "| ev_tdlam_before | -0.00422      |\n",
      "| loss_ent        | 0.56565243    |\n",
      "| loss_kl         | 0.0005075048  |\n",
      "| loss_pol_entpen | -0.005656524  |\n",
      "| loss_pol_surr   | -0.0025824122 |\n",
      "| loss_vf_loss    | 121.05017     |\n",
      "-----------------------------------\n",
      "********** Iteration 11 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "      0.00022 |      -0.00583 |     138.74347 |      8.35e-05 |       0.58293\n",
      "     -0.00123 |      -0.00589 |     133.39716 |       0.00060 |       0.58912\n",
      "     -0.00238 |      -0.00595 |     128.12556 |       0.00163 |       0.59467\n",
      "     -0.00071 |      -0.00599 |     123.15498 |       0.00232 |       0.59879\n",
      "Evaluating losses...\n",
      "     -0.00355 |      -0.00601 |     120.23128 |       0.00223 |       0.60116\n",
      "-----------------------------------\n",
      "| EpLenMean       | 57.8          |\n",
      "| EpRewMean       | 57.8          |\n",
      "| EpThisIter      | 1             |\n",
      "| EpisodesSoFar   | 50            |\n",
      "| TimeElapsed     | 2.22          |\n",
      "| TimestepsSoFar  | 3721          |\n",
      "| ev_tdlam_before | 0.00817       |\n",
      "| loss_ent        | 0.6011628     |\n",
      "| loss_kl         | 0.002231727   |\n",
      "| loss_pol_entpen | -0.006011628  |\n",
      "| loss_pol_surr   | -0.0035511805 |\n",
      "| loss_vf_loss    | 120.23128     |\n",
      "-----------------------------------\n",
      "********** Iteration 12 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "      0.00014 |      -0.00578 |     134.17792 |      1.59e-05 |       0.57831\n",
      "     -0.00156 |      -0.00582 |     130.18593 |       0.00020 |       0.58247\n",
      "     -0.00460 |      -0.00587 |     126.56305 |       0.00075 |       0.58656\n",
      "     -0.00603 |      -0.00591 |     123.21868 |       0.00183 |       0.59055\n",
      "Evaluating losses...\n",
      "     -0.00670 |      -0.00592 |     121.12634 |       0.00270 |       0.59168\n",
      "-----------------------------------\n",
      "| EpLenMean       | 63.3          |\n",
      "| EpRewMean       | 63.3          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 52            |\n",
      "| TimeElapsed     | 2.38          |\n",
      "| TimestepsSoFar  | 4157          |\n",
      "| ev_tdlam_before | 0.0019        |\n",
      "| loss_ent        | 0.591684      |\n",
      "| loss_kl         | 0.0027004546  |\n",
      "| loss_pol_entpen | -0.00591684   |\n",
      "| loss_pol_surr   | -0.0067006145 |\n",
      "| loss_vf_loss    | 121.126335    |\n",
      "-----------------------------------\n",
      "********** Iteration 13 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00046 |      -0.00567 |     115.61295 |      2.81e-05 |       0.56726\n",
      "     -0.00454 |      -0.00563 |     113.12674 |       0.00032 |       0.56293\n",
      "     -0.00800 |      -0.00555 |     110.68002 |       0.00131 |       0.55505\n",
      "     -0.00750 |      -0.00551 |     108.28281 |       0.00222 |       0.55129\n",
      "Evaluating losses...\n",
      "     -0.00760 |      -0.00551 |     106.84557 |       0.00269 |       0.55124\n",
      "-----------------------------------\n",
      "| EpLenMean       | 65.3          |\n",
      "| EpRewMean       | 65.3          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 54            |\n",
      "| TimeElapsed     | 2.54          |\n",
      "| TimestepsSoFar  | 4449          |\n",
      "| ev_tdlam_before | 0.00664       |\n",
      "| loss_ent        | 0.55124176    |\n",
      "| loss_kl         | 0.0026882736  |\n",
      "| loss_pol_entpen | -0.0055124178 |\n",
      "| loss_pol_surr   | -0.0076028686 |\n",
      "| loss_vf_loss    | 106.84557     |\n",
      "-----------------------------------\n",
      "********** Iteration 14 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "    -6.12e-05 |      -0.00584 |     112.49179 |      7.72e-06 |       0.58373\n",
      "     -0.00232 |      -0.00580 |     110.45225 |       0.00010 |       0.57956\n",
      "     -0.00577 |      -0.00571 |     108.26415 |       0.00070 |       0.57136\n",
      "     -0.00657 |      -0.00564 |     106.32086 |       0.00181 |       0.56358\n",
      "Evaluating losses...\n",
      "     -0.00673 |      -0.00560 |     105.08083 |       0.00251 |       0.56016\n",
      "----------------------------------\n",
      "| EpLenMean       | 67.3         |\n",
      "| EpRewMean       | 67.3         |\n",
      "| EpThisIter      | 2            |\n",
      "| EpisodesSoFar   | 56           |\n",
      "| TimeElapsed     | 2.72         |\n",
      "| TimestepsSoFar  | 4763         |\n",
      "| ev_tdlam_before | 0.0395       |\n",
      "| loss_ent        | 0.5601573    |\n",
      "| loss_kl         | 0.002510219  |\n",
      "| loss_pol_entpen | -0.005601573 |\n",
      "| loss_pol_surr   | -0.006728638 |\n",
      "| loss_vf_loss    | 105.080826   |\n",
      "----------------------------------\n",
      "********** Iteration 15 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00089 |      -0.00576 |     125.38103 |      5.18e-05 |       0.57607\n",
      "     -0.00459 |      -0.00571 |     123.47046 |       0.00079 |       0.57109\n",
      "     -0.00361 |      -0.00569 |     121.38974 |       0.00134 |       0.56922\n",
      "     -0.00424 |      -0.00572 |     119.47952 |       0.00067 |       0.57167\n",
      "Evaluating losses...\n",
      "     -0.00417 |      -0.00574 |     118.34568 |       0.00029 |       0.57381\n",
      "-----------------------------------\n",
      "| EpLenMean       | 70.2          |\n",
      "| EpRewMean       | 70.2          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 58            |\n",
      "| TimeElapsed     | 2.9           |\n",
      "| TimestepsSoFar  | 5089          |\n",
      "| ev_tdlam_before | 0.00584       |\n",
      "| loss_ent        | 0.57381225    |\n",
      "| loss_kl         | 0.000287401   |\n",
      "| loss_pol_entpen | -0.0057381224 |\n",
      "| loss_pol_surr   | -0.0041719982 |\n",
      "| loss_vf_loss    | 118.34568     |\n",
      "-----------------------------------\n",
      "********** Iteration 16 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "      0.00038 |      -0.00576 |     131.21524 |      4.97e-06 |       0.57560\n",
      "      0.00034 |      -0.00576 |     128.68576 |      1.82e-05 |       0.57594\n",
      "     9.80e-05 |      -0.00575 |     126.01968 |      3.19e-05 |       0.57536\n",
      "    -4.98e-05 |      -0.00575 |     123.39338 |      4.69e-05 |       0.57492\n",
      "Evaluating losses...\n",
      "     -0.00027 |      -0.00574 |     121.75385 |      6.82e-05 |       0.57431\n",
      "-----------------------------------\n",
      "| EpLenMean       | 72.4          |\n",
      "| EpRewMean       | 72.4          |\n",
      "| EpThisIter      | 1             |\n",
      "| EpisodesSoFar   | 59            |\n",
      "| TimeElapsed     | 3.08          |\n",
      "| TimestepsSoFar  | 5367          |\n",
      "| ev_tdlam_before | 0.00276       |\n",
      "| loss_ent        | 0.5743083     |\n",
      "| loss_kl         | 6.824043e-05  |\n",
      "| loss_pol_entpen | -0.0057430826 |\n",
      "| loss_pol_surr   | -0.0002656486 |\n",
      "| loss_vf_loss    | 121.75385     |\n",
      "-----------------------------------\n",
      "********** Iteration 17 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "      0.00010 |      -0.00579 |     128.34184 |      4.79e-06 |       0.57945\n",
      "     -0.00035 |      -0.00580 |     125.78346 |      4.99e-05 |       0.58008\n",
      "     -0.00140 |      -0.00581 |     123.33639 |       0.00014 |       0.58150\n",
      "     -0.00275 |      -0.00583 |     120.81356 |       0.00032 |       0.58307\n",
      "Evaluating losses...\n",
      "     -0.00366 |      -0.00584 |     119.30229 |       0.00053 |       0.58429\n",
      "-----------------------------------\n",
      "| EpLenMean       | 73.8          |\n",
      "| EpRewMean       | 73.8          |\n",
      "| EpThisIter      | 1             |\n",
      "| EpisodesSoFar   | 60            |\n",
      "| TimeElapsed     | 3.28          |\n",
      "| TimestepsSoFar  | 5701          |\n",
      "| ev_tdlam_before | 0.0639        |\n",
      "| loss_ent        | 0.5842918     |\n",
      "| loss_kl         | 0.0005294614  |\n",
      "| loss_pol_entpen | -0.0058429176 |\n",
      "| loss_pol_surr   | -0.003660962  |\n",
      "| loss_vf_loss    | 119.30229     |\n",
      "-----------------------------------\n",
      "********** Iteration 18 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     3.54e-05 |      -0.00578 |     124.76637 |      2.54e-05 |       0.57769\n",
      "     -0.00095 |      -0.00579 |     123.11864 |       0.00020 |       0.57868\n",
      "     -0.00068 |      -0.00578 |     121.67198 |       0.00056 |       0.57840\n",
      "     -0.00091 |      -0.00578 |     120.13757 |       0.00065 |       0.57824\n",
      "Evaluating losses...\n",
      "     -0.00167 |      -0.00579 |     119.25534 |       0.00046 |       0.57870\n",
      "-----------------------------------\n",
      "| EpLenMean       | 76.5          |\n",
      "| EpRewMean       | 76.5          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 62            |\n",
      "| TimeElapsed     | 3.44          |\n",
      "| TimestepsSoFar  | 6136          |\n",
      "| ev_tdlam_before | 0.0252        |\n",
      "| loss_ent        | 0.57869726    |\n",
      "| loss_kl         | 0.0004619496  |\n",
      "| loss_pol_entpen | -0.0057869726 |\n",
      "| loss_pol_surr   | -0.00167262   |\n",
      "| loss_vf_loss    | 119.25534     |\n",
      "-----------------------------------\n",
      "********** Iteration 19 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     -0.00083 |      -0.00573 |     125.37040 |      7.05e-05 |       0.57313\n",
      "     -0.00345 |      -0.00575 |     124.05255 |       0.00078 |       0.57510\n",
      "     -0.00188 |      -0.00576 |     122.89958 |       0.00168 |       0.57604\n",
      "     -0.00270 |      -0.00577 |     121.73227 |       0.00163 |       0.57662\n",
      "Evaluating losses...\n",
      "     -0.00371 |      -0.00576 |     121.02550 |       0.00105 |       0.57637\n",
      "----------------------------------\n",
      "| EpLenMean       | 79.7         |\n",
      "| EpRewMean       | 79.7         |\n",
      "| EpThisIter      | 2            |\n",
      "| EpisodesSoFar   | 64           |\n",
      "| TimeElapsed     | 3.61         |\n",
      "| TimestepsSoFar  | 6510         |\n",
      "| ev_tdlam_before | 0.00677      |\n",
      "| loss_ent        | 0.5763704    |\n",
      "| loss_kl         | 0.0010542639 |\n",
      "| loss_pol_entpen | -0.005763704 |\n",
      "| loss_pol_surr   | -0.003706148 |\n",
      "| loss_vf_loss    | 121.0255     |\n",
      "----------------------------------\n",
      "********** Iteration 20 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "    -4.53e-05 |      -0.00575 |     129.72104 |      8.45e-05 |       0.57528\n",
      "     -0.00189 |      -0.00574 |     128.21004 |       0.00053 |       0.57405\n",
      "     -0.00313 |      -0.00571 |     126.65305 |       0.00086 |       0.57130\n",
      "     -0.00381 |      -0.00570 |     125.06973 |       0.00056 |       0.56974\n",
      "Evaluating losses...\n",
      "     -0.00376 |      -0.00569 |     124.07959 |       0.00048 |       0.56939\n",
      "-----------------------------------\n",
      "| EpLenMean       | 79.7          |\n",
      "| EpRewMean       | 79.7          |\n",
      "| EpThisIter      | 1             |\n",
      "| EpisodesSoFar   | 65            |\n",
      "| TimeElapsed     | 3.79          |\n",
      "| TimestepsSoFar  | 6787          |\n",
      "| ev_tdlam_before | 0.00106       |\n",
      "| loss_ent        | 0.56938595    |\n",
      "| loss_kl         | 0.0004832453  |\n",
      "| loss_pol_entpen | -0.0056938594 |\n",
      "| loss_pol_surr   | -0.0037583113 |\n",
      "| loss_vf_loss    | 124.07959     |\n",
      "-----------------------------------\n",
      "********** Iteration 21 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     1.91e-05 |      -0.00587 |     125.14951 |      2.81e-06 |       0.58743\n",
      "    -8.52e-05 |      -0.00588 |     123.57574 |      1.58e-05 |       0.58806\n",
      "      0.00012 |      -0.00589 |     122.08537 |      2.61e-05 |       0.58937\n",
      "     -0.00047 |      -0.00591 |     120.55425 |      6.56e-05 |       0.59140\n",
      "Evaluating losses...\n",
      "     -0.00090 |      -0.00593 |     119.64975 |       0.00011 |       0.59274\n",
      "------------------------------------\n",
      "| EpLenMean       | 82.5           |\n",
      "| EpRewMean       | 82.5           |\n",
      "| EpThisIter      | 2              |\n",
      "| EpisodesSoFar   | 67             |\n",
      "| TimeElapsed     | 3.95           |\n",
      "| TimestepsSoFar  | 7241           |\n",
      "| ev_tdlam_before | 0.0563         |\n",
      "| loss_ent        | 0.5927439      |\n",
      "| loss_kl         | 0.000108841035 |\n",
      "| loss_pol_entpen | -0.005927439   |\n",
      "| loss_pol_surr   | -0.0009044274  |\n",
      "| loss_vf_loss    | 119.64975      |\n",
      "------------------------------------\n",
      "********** Iteration 22 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "      0.00037 |      -0.00596 |     127.29877 |      6.01e-06 |       0.59646\n",
      "    -8.34e-05 |      -0.00598 |     126.10747 |      1.26e-05 |       0.59751\n",
      "     -0.00013 |      -0.00598 |     124.82404 |      4.34e-05 |       0.59823\n",
      "     -0.00039 |      -0.00599 |     123.59561 |       0.00015 |       0.59883\n",
      "Evaluating losses...\n",
      "     -0.00057 |      -0.00599 |     122.83302 |       0.00025 |       0.59905\n",
      "------------------------------------\n",
      "| EpLenMean       | 84.2           |\n",
      "| EpRewMean       | 84.2           |\n",
      "| EpThisIter      | 1              |\n",
      "| EpisodesSoFar   | 68             |\n",
      "| TimeElapsed     | 4.12           |\n",
      "| TimestepsSoFar  | 7604           |\n",
      "| ev_tdlam_before | -0.000518      |\n",
      "| loss_ent        | 0.5990509      |\n",
      "| loss_kl         | 0.00024647068  |\n",
      "| loss_pol_entpen | -0.005990509   |\n",
      "| loss_pol_surr   | -0.00057125185 |\n",
      "| loss_vf_loss    | 122.83302      |\n",
      "------------------------------------\n",
      "********** Iteration 23 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "      0.00019 |      -0.00581 |     129.80707 |      1.53e-06 |       0.58135\n",
      "     -0.00018 |      -0.00582 |     128.83949 |      1.36e-05 |       0.58214\n",
      "     -0.00179 |      -0.00584 |     127.87000 |       0.00011 |       0.58354\n",
      "     -0.00235 |      -0.00585 |     126.92532 |       0.00035 |       0.58494\n",
      "Evaluating losses...\n",
      "     -0.00281 |      -0.00586 |     126.31238 |       0.00042 |       0.58582\n",
      "-----------------------------------\n",
      "| EpLenMean       | 86.1          |\n",
      "| EpRewMean       | 86.1          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 70            |\n",
      "| TimeElapsed     | 4.28          |\n",
      "| TimestepsSoFar  | 8025          |\n",
      "| ev_tdlam_before | 0.0495        |\n",
      "| loss_ent        | 0.5858217     |\n",
      "| loss_kl         | 0.00041907898 |\n",
      "| loss_pol_entpen | -0.005858217  |\n",
      "| loss_pol_surr   | -0.0028111301 |\n",
      "| loss_vf_loss    | 126.31238     |\n",
      "-----------------------------------\n",
      "********** Iteration 24 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     9.30e-05 |      -0.00576 |     126.98463 |      6.91e-06 |       0.57560\n",
      "     -0.00099 |      -0.00575 |     126.22601 |      5.75e-05 |       0.57487\n",
      "     -0.00147 |      -0.00574 |     125.37617 |       0.00017 |       0.57373\n",
      "     -0.00113 |      -0.00573 |     124.54818 |       0.00027 |       0.57293\n",
      "Evaluating losses...\n",
      "     -0.00118 |      -0.00573 |     124.01172 |       0.00030 |       0.57316\n",
      "-----------------------------------\n",
      "| EpLenMean       | 87.7          |\n",
      "| EpRewMean       | 87.7          |\n",
      "| EpThisIter      | 1             |\n",
      "| EpisodesSoFar   | 71            |\n",
      "| TimeElapsed     | 4.44          |\n",
      "| TimestepsSoFar  | 8400          |\n",
      "| ev_tdlam_before | 0.00391       |\n",
      "| loss_ent        | 0.573158      |\n",
      "| loss_kl         | 0.0002952306  |\n",
      "| loss_pol_entpen | -0.005731581  |\n",
      "| loss_pol_surr   | -0.0011836514 |\n",
      "| loss_vf_loss    | 124.01172     |\n",
      "-----------------------------------\n",
      "********** Iteration 25 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     7.85e-05 |      -0.00573 |     125.42858 |      2.60e-06 |       0.57262\n",
      "     -0.00021 |      -0.00573 |     124.97832 |      2.22e-05 |       0.57335\n",
      "     -0.00071 |      -0.00575 |     124.58188 |      5.43e-05 |       0.57469\n",
      "     -0.00086 |      -0.00576 |     124.14428 |      9.01e-05 |       0.57608\n",
      "Evaluating losses...\n",
      "     -0.00090 |      -0.00577 |     123.89259 |       0.00011 |       0.57684\n",
      "------------------------------------\n",
      "| EpLenMean       | 89.6           |\n",
      "| EpRewMean       | 89.6           |\n",
      "| EpThisIter      | 2              |\n",
      "| EpisodesSoFar   | 73             |\n",
      "| TimeElapsed     | 4.62           |\n",
      "| TimestepsSoFar  | 8831           |\n",
      "| ev_tdlam_before | -0.000629      |\n",
      "| loss_ent        | 0.57684165     |\n",
      "| loss_kl         | 0.000106907944 |\n",
      "| loss_pol_entpen | -0.0057684164  |\n",
      "| loss_pol_surr   | -0.00089972373 |\n",
      "| loss_vf_loss    | 123.89259      |\n",
      "------------------------------------\n",
      "********** Iteration 26 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     9.64e-05 |      -0.00582 |     136.39577 |      4.62e-06 |       0.58157\n",
      "     -0.00097 |      -0.00582 |     136.05002 |      4.78e-05 |       0.58163\n",
      "     -0.00082 |      -0.00582 |     135.70447 |      7.62e-05 |       0.58204\n",
      "     -0.00076 |      -0.00583 |     135.33125 |      3.81e-05 |       0.58295\n",
      "Evaluating losses...\n",
      "     -0.00073 |      -0.00583 |     135.11325 |      2.69e-05 |       0.58341\n",
      "-----------------------------------\n",
      "| EpLenMean       | 91.6          |\n",
      "| EpRewMean       | 91.6          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 75            |\n",
      "| TimeElapsed     | 4.78          |\n",
      "| TimestepsSoFar  | 9200          |\n",
      "| ev_tdlam_before | -0.000604     |\n",
      "| loss_ent        | 0.5834116     |\n",
      "| loss_kl         | 2.6943635e-05 |\n",
      "| loss_pol_entpen | -0.0058341157 |\n",
      "| loss_pol_surr   | -0.0007331455 |\n",
      "| loss_vf_loss    | 135.11325     |\n",
      "-----------------------------------\n",
      "********** Iteration 27 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     8.59e-05 |      -0.00592 |     126.39661 |      1.63e-07 |       0.59216\n",
      "     2.93e-05 |      -0.00592 |     126.10086 |      1.81e-07 |       0.59220\n",
      "     8.37e-06 |      -0.00592 |     125.76957 |      3.89e-07 |       0.59199\n",
      "     -0.00014 |      -0.00592 |     125.44743 |      1.68e-06 |       0.59162\n",
      "Evaluating losses...\n",
      "     -0.00018 |      -0.00591 |     125.24962 |      3.34e-06 |       0.59137\n",
      "------------------------------------\n",
      "| EpLenMean       | 93             |\n",
      "| EpRewMean       | 93             |\n",
      "| EpThisIter      | 1              |\n",
      "| EpisodesSoFar   | 76             |\n",
      "| TimeElapsed     | 4.95           |\n",
      "| TimestepsSoFar  | 9501           |\n",
      "| ev_tdlam_before | 0.00207        |\n",
      "| loss_ent        | 0.59137124     |\n",
      "| loss_kl         | 3.3367635e-06  |\n",
      "| loss_pol_entpen | -0.005913712   |\n",
      "| loss_pol_surr   | -0.00018083537 |\n",
      "| loss_vf_loss    | 125.24962      |\n",
      "------------------------------------\n",
      "********** Iteration 28 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "    -5.46e-05 |      -0.00597 |     122.39555 |      1.49e-07 |       0.59732\n",
      "     -0.00026 |      -0.00597 |     122.16752 |      3.06e-06 |       0.59705\n",
      "     -0.00025 |      -0.00597 |     121.92773 |      1.62e-05 |       0.59677\n",
      "     -0.00023 |      -0.00596 |     121.67959 |      2.36e-05 |       0.59649\n",
      "Evaluating losses...\n",
      "     -0.00037 |      -0.00596 |     121.52153 |      1.78e-05 |       0.59632\n",
      "-----------------------------------\n",
      "| EpLenMean       | 94.4          |\n",
      "| EpRewMean       | 94.4          |\n",
      "| EpThisIter      | 1             |\n",
      "| EpisodesSoFar   | 77            |\n",
      "| TimeElapsed     | 5.11          |\n",
      "| TimestepsSoFar  | 9858          |\n",
      "| ev_tdlam_before | 0.092         |\n",
      "| loss_ent        | 0.5963151     |\n",
      "| loss_kl         | 1.7756685e-05 |\n",
      "| loss_pol_entpen | -0.0059631504 |\n",
      "| loss_pol_surr   | -0.0003733514 |\n",
      "| loss_vf_loss    | 121.52153     |\n",
      "-----------------------------------\n",
      "********** Iteration 29 ************\n",
      "Optimizing...\n",
      "     pol_surr |    pol_entpen |       vf_loss |            kl |           ent\n",
      "     1.73e-05 |      -0.00605 |     119.20558 |      2.08e-08 |       0.60514\n",
      "     2.28e-05 |      -0.00605 |     119.16640 |      4.19e-08 |       0.60515\n",
      "    -9.55e-06 |      -0.00605 |     119.12991 |      3.72e-08 |       0.60517\n",
      "    -6.73e-05 |      -0.00605 |     119.09962 |      2.79e-07 |       0.60520\n",
      "Evaluating losses...\n",
      "    -9.97e-05 |      -0.00605 |     119.07908 |      6.04e-07 |       0.60522\n",
      "-----------------------------------\n",
      "| EpLenMean       | 96.3          |\n",
      "| EpRewMean       | 96.3          |\n",
      "| EpThisIter      | 2             |\n",
      "| EpisodesSoFar   | 79            |\n",
      "| TimeElapsed     | 5.27          |\n",
      "| TimestepsSoFar  | 10273         |\n",
      "| ev_tdlam_before | 0.0119        |\n",
      "| loss_ent        | 0.6052187     |\n",
      "| loss_kl         | 6.037436e-07  |\n",
      "| loss_pol_entpen | -0.0060521867 |\n",
      "| loss_pol_surr   | -9.965524e-05 |\n",
      "| loss_vf_loss    | 119.07908     |\n",
      "-----------------------------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<stable_baselines.ppo1.pposgd_simple.PPO1 at 0x7f857872e400>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from stable_baselines import PPO1\n",
    "\n",
    "model = PPO1('MlpPolicy', 'CartPole-v0', verbose=1, tensorboard_log=\"log/ppo/\")\n",
    "model.learn(total_timesteps=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baselines Implementation of TRPO\n",
    "\n",
    "From the stable-baselines repo to compare my implementation with the baselines implementation of TRPO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating environment from the given name, wrapped in a DummyVecEnv.\n",
      "********** Iteration 0 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.903 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.110 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0298          0\n",
      "         1   0.000116      0.131\n",
      "         2   0.000221      0.153\n",
      "         3   1.99e-05       0.38\n",
      "         4   2.86e-08      0.565\n",
      "         5   5.31e-08      0.565\n",
      "         6   2.69e-10      0.565\n",
      "         7   2.35e-09      0.565\n",
      "         8   5.58e-13      0.565\n",
      "\u001b[35mdone in 0.149 seconds\u001b[0m\n",
      "Expected: 0.023 Actual: 0.023\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.096 seconds\u001b[0m\n",
      "---------------------------------\n",
      "| EpLenMean       | 17.8        |\n",
      "| EpRewMean       | 17.8        |\n",
      "| EpThisIter      | 53          |\n",
      "| EpisodesSoFar   | 53          |\n",
      "| TimeElapsed     | 1.33        |\n",
      "| TimestepsSoFar  | 1024        |\n",
      "| entloss         | 0.0         |\n",
      "| entropy         | 0.6846268   |\n",
      "| ev_tdlam_before | -0.00358    |\n",
      "| meankl          | 0.008726809 |\n",
      "| optimgain       | 0.023063311 |\n",
      "| surrgain        | 0.023063311 |\n",
      "---------------------------------\n",
      "********** Iteration 1 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.840 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.009 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0321          0\n",
      "         1   0.000401      0.135\n",
      "         2   0.000184      0.387\n",
      "         3   5.85e-05      0.403\n",
      "         4   1.97e-06      0.619\n",
      "         5    2.1e-06      0.638\n",
      "         6   1.96e-07      0.638\n",
      "         7   2.29e-09      0.642\n",
      "         8   8.53e-09      0.642\n",
      "         9   7.06e-12      0.642\n",
      "\u001b[35mdone in 0.023 seconds\u001b[0m\n",
      "Expected: 0.026 Actual: 0.029\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.061 seconds\u001b[0m\n",
      "---------------------------------\n",
      "| EpLenMean       | 24.4        |\n",
      "| EpRewMean       | 24.4        |\n",
      "| EpThisIter      | 42          |\n",
      "| EpisodesSoFar   | 95          |\n",
      "| TimeElapsed     | 2.28        |\n",
      "| TimestepsSoFar  | 2056        |\n",
      "| entloss         | 0.0         |\n",
      "| entropy         | 0.65994203  |\n",
      "| ev_tdlam_before | 0.00582     |\n",
      "| meankl          | 0.010105519 |\n",
      "| optimgain       | 0.02863079  |\n",
      "| surrgain        | 0.02863079  |\n",
      "---------------------------------\n",
      "********** Iteration 2 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.847 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.010 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0415          0\n",
      "         1    0.00192      0.134\n",
      "         2   0.000659      0.349\n",
      "         3   0.000929      0.388\n",
      "         4   6.23e-05      0.576\n",
      "         5   3.35e-06      0.813\n",
      "         6   3.21e-05      0.832\n",
      "         7   2.92e-07      0.837\n",
      "         8   1.49e-07      0.839\n",
      "         9    4.4e-08      0.839\n",
      "        10   1.34e-09       0.84\n",
      "\u001b[35mdone in 0.029 seconds\u001b[0m\n",
      "Expected: 0.032 Actual: 0.030\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.053 seconds\u001b[0m\n",
      "----------------------------------\n",
      "| EpLenMean       | 38.2         |\n",
      "| EpRewMean       | 38.2         |\n",
      "| EpThisIter      | 20           |\n",
      "| EpisodesSoFar   | 115          |\n",
      "| TimeElapsed     | 3.23         |\n",
      "| TimestepsSoFar  | 3101         |\n",
      "| entloss         | 0.0          |\n",
      "| entropy         | 0.63891697   |\n",
      "| ev_tdlam_before | -0.0163      |\n",
      "| meankl          | 0.0087044425 |\n",
      "| optimgain       | 0.029601578  |\n",
      "| surrgain        | 0.029601578  |\n",
      "----------------------------------\n",
      "********** Iteration 3 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.877 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.010 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0147          0\n",
      "         1    0.00446       0.09\n",
      "         2   0.000703      0.233\n",
      "         3    0.00117      0.261\n",
      "         4    0.00029      0.313\n",
      "         5    8.5e-06      0.791\n",
      "         6    9.2e-07      0.837\n",
      "         7   5.43e-06      0.845\n",
      "         8   1.01e-07      0.846\n",
      "         9   1.73e-06      0.846\n",
      "        10   2.31e-08      0.847\n",
      "\u001b[35mdone in 0.027 seconds\u001b[0m\n",
      "Expected: 0.026 Actual: 0.025\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.052 seconds\u001b[0m\n",
      "---------------------------------\n",
      "| EpLenMean       | 55.4        |\n",
      "| EpRewMean       | 55.4        |\n",
      "| EpThisIter      | 12          |\n",
      "| EpisodesSoFar   | 127         |\n",
      "| TimeElapsed     | 4.22        |\n",
      "| TimestepsSoFar  | 4136        |\n",
      "| entloss         | 0.0         |\n",
      "| entropy         | 0.6101633   |\n",
      "| ev_tdlam_before | -0.0123     |\n",
      "| meankl          | 0.009158809 |\n",
      "| optimgain       | 0.024793841 |\n",
      "| surrgain        | 0.024793841 |\n",
      "---------------------------------\n",
      "********** Iteration 4 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.792 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.010 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0689          0\n",
      "         1    0.00297     0.0936\n",
      "         2    0.00316      0.118\n",
      "         3   0.000667      0.217\n",
      "         4   0.000148      0.347\n",
      "         5   2.34e-05      0.557\n",
      "         6   4.32e-05       0.62\n",
      "         7   6.14e-06      0.622\n",
      "         8   9.13e-05      0.645\n",
      "         9   1.29e-06      0.654\n",
      "        10   3.89e-08      0.667\n",
      "\u001b[35mdone in 0.027 seconds\u001b[0m\n",
      "Expected: 0.029 Actual: 0.030\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.056 seconds\u001b[0m\n",
      "---------------------------------\n",
      "| EpLenMean       | 70.7        |\n",
      "| EpRewMean       | 70.7        |\n",
      "| EpThisIter      | 11          |\n",
      "| EpisodesSoFar   | 138         |\n",
      "| TimeElapsed     | 5.12        |\n",
      "| TimestepsSoFar  | 5161        |\n",
      "| entloss         | 0.0         |\n",
      "| entropy         | 0.5899418   |\n",
      "| ev_tdlam_before | 0.00195     |\n",
      "| meankl          | 0.011018865 |\n",
      "| optimgain       | 0.029500492 |\n",
      "| surrgain        | 0.029500492 |\n",
      "---------------------------------\n",
      "********** Iteration 5 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.821 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.010 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0147          0\n",
      "         1    0.00277      0.051\n",
      "         2    0.00132      0.071\n",
      "         3    0.00133     0.0898\n",
      "         4   0.000263       0.28\n",
      "         5   1.66e-05      0.634\n",
      "         6   2.02e-06      0.722\n",
      "         7   2.76e-06      0.722\n",
      "         8   3.68e-05      0.728\n",
      "         9   3.24e-07      0.732\n",
      "        10   6.38e-07      0.732\n",
      "\u001b[35mdone in 0.027 seconds\u001b[0m\n",
      "Expected: 0.020 Actual: 0.015\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.063 seconds\u001b[0m\n",
      "---------------------------------\n",
      "| EpLenMean       | 91.2        |\n",
      "| EpRewMean       | 91.2        |\n",
      "| EpThisIter      | 6           |\n",
      "| EpisodesSoFar   | 144         |\n",
      "| TimeElapsed     | 6.06        |\n",
      "| TimestepsSoFar  | 6267        |\n",
      "| entloss         | 0.0         |\n",
      "| entropy         | 0.6121911   |\n",
      "| ev_tdlam_before | 0.0389      |\n",
      "| meankl          | 0.004211154 |\n",
      "| optimgain       | 0.014927611 |\n",
      "| surrgain        | 0.014927611 |\n",
      "---------------------------------\n",
      "********** Iteration 6 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.845 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.010 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0155          0\n",
      "         1    0.00187      0.055\n",
      "         2    0.00283     0.0725\n",
      "         3    0.00167      0.101\n",
      "         4   8.09e-05       0.26\n",
      "         5   3.32e-05      0.404\n",
      "         6    1.1e-05      0.507\n",
      "         7   0.000382      0.546\n",
      "         8    8.5e-06      0.566\n",
      "         9   6.62e-07      0.566\n",
      "        10   4.29e-06      0.567\n",
      "\u001b[35mdone in 0.028 seconds\u001b[0m\n",
      "Expected: 0.019 Actual: 0.016\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.052 seconds\u001b[0m\n",
      "---------------------------------\n",
      "| EpLenMean       | 105         |\n",
      "| EpRewMean       | 105         |\n",
      "| EpThisIter      | 7           |\n",
      "| EpisodesSoFar   | 151         |\n",
      "| TimeElapsed     | 7.01        |\n",
      "| TimestepsSoFar  | 7338        |\n",
      "| entloss         | 0.0         |\n",
      "| entropy         | 0.56996375  |\n",
      "| ev_tdlam_before | 0.0588      |\n",
      "| meankl          | 0.009382002 |\n",
      "| optimgain       | 0.016168624 |\n",
      "| surrgain        | 0.016168624 |\n",
      "---------------------------------\n",
      "********** Iteration 7 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.812 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.009 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0    0.00999          0\n",
      "         1    0.00568     0.0397\n",
      "         2   0.000177     0.0587\n",
      "         3   0.000159     0.0622\n",
      "         4   4.23e-05      0.216\n",
      "         5   6.26e-06      0.276\n",
      "         6   0.000354       0.33\n",
      "         7   2.37e-07      0.351\n",
      "         8   6.07e-06      0.355\n",
      "         9   5.14e-07      0.358\n",
      "        10   1.41e-08      0.358\n",
      "\u001b[35mdone in 0.027 seconds\u001b[0m\n",
      "Expected: 0.013 Actual: 0.010\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.052 seconds\u001b[0m\n",
      "----------------------------------\n",
      "| EpLenMean       | 122          |\n",
      "| EpRewMean       | 122          |\n",
      "| EpThisIter      | 5            |\n",
      "| EpisodesSoFar   | 156          |\n",
      "| TimeElapsed     | 7.92         |\n",
      "| TimestepsSoFar  | 8514         |\n",
      "| entloss         | 0.0          |\n",
      "| entropy         | 0.595799     |\n",
      "| ev_tdlam_before | 0.0142       |\n",
      "| meankl          | 0.0055363066 |\n",
      "| optimgain       | 0.009893656  |\n",
      "| surrgain        | 0.009893656  |\n",
      "----------------------------------\n",
      "********** Iteration 8 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.832 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.010 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0363          0\n",
      "         1     0.0163      0.081\n",
      "         2   0.000925      0.124\n",
      "         3    0.00275      0.149\n",
      "         4   8.62e-05      0.298\n",
      "         5   2.07e-05      0.581\n",
      "         6   0.000218      0.699\n",
      "         7   2.55e-06      0.707\n",
      "         8   6.67e-05      0.717\n",
      "         9   7.78e-07      0.728\n",
      "        10   9.48e-08      0.728\n",
      "\u001b[35mdone in 0.028 seconds\u001b[0m\n",
      "Expected: 0.026 Actual: 0.020\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.054 seconds\u001b[0m\n",
      "----------------------------------\n",
      "| EpLenMean       | 133          |\n",
      "| EpRewMean       | 133          |\n",
      "| EpThisIter      | 8            |\n",
      "| EpisodesSoFar   | 164          |\n",
      "| TimeElapsed     | 8.86         |\n",
      "| TimestepsSoFar  | 9737         |\n",
      "| entloss         | 0.0          |\n",
      "| entropy         | 0.5508223    |\n",
      "| ev_tdlam_before | 0.0178       |\n",
      "| meankl          | 0.0077643963 |\n",
      "| optimgain       | 0.020191714  |\n",
      "| surrgain        | 0.020191714  |\n",
      "----------------------------------\n",
      "********** Iteration 9 ************\n",
      "Optimizing Policy...\n",
      "\u001b[35msampling\u001b[0m\n",
      "\u001b[35mdone in 0.793 seconds\u001b[0m\n",
      "\u001b[35mcomputegrad\u001b[0m\n",
      "\u001b[35mdone in 0.010 seconds\u001b[0m\n",
      "\u001b[35mcg\u001b[0m\n",
      "      iter residual norm  soln norm\n",
      "         0     0.0116          0\n",
      "         1    0.00825     0.0367\n",
      "         2   0.000305     0.0715\n",
      "         3   0.000136     0.0765\n",
      "         4   1.81e-05      0.152\n",
      "         5   1.97e-06      0.224\n",
      "         6   0.000183      0.233\n",
      "         7   1.54e-07       0.27\n",
      "         8   1.39e-06      0.271\n",
      "         9    1.3e-06      0.273\n",
      "        10   7.08e-09      0.274\n",
      "\u001b[35mdone in 0.028 seconds\u001b[0m\n",
      "Expected: 0.013 Actual: 0.013\n",
      "Stepsize OK!\n",
      "\u001b[35mvf\u001b[0m\n",
      "\u001b[35mdone in 0.055 seconds\u001b[0m\n",
      "---------------------------------\n",
      "| EpLenMean       | 151         |\n",
      "| EpRewMean       | 151         |\n",
      "| EpThisIter      | 5           |\n",
      "| EpisodesSoFar   | 169         |\n",
      "| TimeElapsed     | 9.77        |\n",
      "| TimestepsSoFar  | 10781       |\n",
      "| entloss         | 0.0         |\n",
      "| entropy         | 0.5866855   |\n",
      "| ev_tdlam_before | 0.0175      |\n",
      "| meankl          | 0.009008175 |\n",
      "| optimgain       | 0.013332371 |\n",
      "| surrgain        | 0.013332371 |\n",
      "---------------------------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<stable_baselines.trpo_mpi.trpo_mpi.TRPO at 0x7fa2b80c8630>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from stable_baselines import TRPO\n",
    "\n",
    "model = TRPO('MlpPolicy', 'CartPole-v0', verbose=1, tensorboard_log=\"log/trpo/\")\n",
    "model.learn(total_timesteps=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitor Tensorboard for plots\n",
    "\n",
    "1. Change directory into this project directory\n",
    "2. Execute the following command\n",
    "    `tensorboard --logdir=log/`\n",
    "3. Visit the localhost page with the provided port number to monitor tensorboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results on CartPole-v0 environment:\n",
    "\n",
    "## Our PPO implementation Rewards\n",
    "![Our PPO implementation Rewards](./plots/my_ppo.png)\n",
    "\n",
    "## GAIL learned agent Rewards\n",
    "![GAIL rewards](./plots/gail_rewards.png)\n",
    "\n",
    "## Baseline PPO Rewards\n",
    "![Baselines PPO rewards](./plots/baseline_ppo.png)\n",
    "\n",
    "## Baselines TRPO Rewards\n",
    "![Baselines TRPO rewards](./plots/baseline_trpo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References:\n",
    "1. Generative Adversarial Imitation Learning [[paper]](https://arxiv.org/pdf/1606.03476.pdf)\n",
    "2. [OpenAI baselines GAIL](https://github.com/openai/baselines/tree/master/baselines/gail)\n",
    "3. [Tensorflow implementation of Generative Adversarial Imitation Learning(GAIL) with discrete action](https://github.com/uidilr/gail_ppo_tf)\n",
    "4. [Simple GAIL implementation using Tensorflow](https://github.com/AndyHsiao26/GAIL-Tensorflow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}