getting_started/04_TrainingAnAgent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Agent, action converters and l2rpn_baselines\n",
    "Try me out interactively with: [![Binder](./img/badge_logo.svg)](https://mybinder.org/v2/gh/rte-france/Grid2Op/master)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is recommended to have a look at the [00_SmallExample](00_SmallExample.ipynb), [02_Observation](02_Observation.ipynb) and [03_Action](03_Action.ipynb) notebooks before getting into this one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Objectives**\n",
    "\n",
    "In this notebook we will expose :\n",
    "* how to  make grid2op compatible with *gym* RL framework (short introduction to *gym_compat* module)\n",
    "* how to transform grid2op actions / observations with gym \"spaces\" (https://gym.openai.com/docs/#spaces)\n",
    "* how to train a (naive) Agent using reinforcement learning.\n",
    "* how to inspect (rapidly) the actions taken by the Agent.\n",
    "\n",
    "**NB** In this tutorial, we will use the \n",
    "\n",
    "<font size=\"3\" color=\"red\">This notebook do not cover the use of existing RL frameworks. Please consult the [11_IntegrationWithExistingRLFrameworks](11_IntegrationWithExistingRLFrameworks.ipynb) for such information! </font>\n",
    "\n",
    "\n",
    "**Don't hesitate to check the grid2op module grid2op.gym_compat for a closer integration between grid2op and openAI gym. This module is documented at https://grid2op.readthedocs.io/en/latest/gym.html** \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" width=\"200\">\n",
    "Execute the cell below by removing the # character if you use google colab !\n",
    "\n",
    "Cell will look like:\n",
    "```python\n",
    "!pip install grid2op[optional]  # for use with google colab (grid2op is not installed by default)\n",
    "```\n",
    "<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" width=\"200\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install grid2op[optional]  # for use with google colab (grid2Op is not installed by default)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "LEARNING_ITERATION = 50\n",
    "EVAL_EPISODE = 1\n",
    "MAX_EVAL_STEPS = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Impossible to automatically add a menu / table of content to this notebook.\n",
      "You can download \"jyquickhelper\" package with: \n",
      "\"pip install jyquickhelper\"\n"
     ]
    }
   ],
   "source": [
    "res = None\n",
    "try:\n",
    "    from jyquickhelper import add_notebook_menu\n",
    "    res = add_notebook_menu()\n",
    "except ModuleNotFoundError:\n",
    "    print(\"Impossible to automatically add a menu / table of content to this notebook.\\nYou can download \\\"jyquickhelper\\\" package with: \\n\\\"pip install jyquickhelper\\\"\")\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0) Good practice\n",
    "\n",
    "### A. defining a training, validation and test sets\n",
    "\n",
    "As in other machine learning tasks, we highly recommend, before even trying to train an agent, to split the the \"episode data\" (*eg* what are the loads / generations for each load / generator) into 3 datasets:\n",
    "- \"train\" use to train the agent\n",
    "- \"val\" use to validate the hyper parameters\n",
    "- \"test\" at which you would look **only once** to report the agent performance in a scientific paper (for example)\n",
    "\n",
    "Grid2op lets you do that with relative ease:\n",
    "\n",
    "```python\n",
    "import grid2op\n",
    "env_name = \"l2rpn_case14_sandbox\"  # or any other...\n",
    "env = grid2op.make(env_name)\n",
    "\n",
    "# extract 1% of the \"chronics\" to be used in the validation environment. The other 99% will\n",
    "# be used for test\n",
    "nm_env_train, nm_env_val, nm_env_test = env.train_val_split_random(pct_val=1., pct_test=1.)\n",
    "\n",
    "# and now you can use the training set only to train your agent:\n",
    "print(f\"The name of the training environment is \\\\\"{nm_env_train}\\\\\"\")\n",
    "print(f\"The name of the validation environment is \\\\\"{nm_env_val}\\\\\"\")\n",
    "print(f\"The name of the test environment is \\\\\"{nm_env_test}\\\\\"\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, you can use the training environment to train your agent:\n",
    "\n",
    "```python\n",
    "import grid2op\n",
    "env_name = \"l2rpn_case14_sandbox\"\n",
    "env = grid2op.make(env_name+\"_train\")\n",
    "```\n",
    "\n",
    "Be carefull, on windows you might run into issues. Don't hesitate to have a look at the documentation of this funciton if this the case (see https://grid2op.readthedocs.io/en/latest/environment.html#grid2op.Environment.Environment.train_val_split and https://grid2op.readthedocs.io/en/latest/environment.html#grid2op.Environment.Environment.train_val_split_random)\n",
    "\n",
    "### B. Not spending all of your time loading data...\n",
    "\n",
    "In most grid2op environment, the \"data\" are loaded from the hard drive.\n",
    "\n",
    "From experience, what happens (especially at the beginning of training) is that your agent survives a few steps (so taking a few milliseconds) before a game over. At this stage you will call `env.reset()` which will load the data of the next scenario.\n",
    "\n",
    "This is the default behaviour and it is far from \"optimal\" (more time is spent loading data than performing actual useful computation). To that end, we encourage you:\n",
    "- to use a \"caching\" mechanism, for example with `MultifolderWithCache` class\n",
    "- to read the data by small \"chunk\" (`env.chronics_handler.set_chunk_size(...)`). \n",
    "\n",
    "More information is provided in https://grid2op.readthedocs.io/en/latest/environment.html#optimize-the-data-pipeline\n",
    "\n",
    "### C. Use a fast simulator\n",
    "\n",
    "Grid2op will use a \"backend\" to compute the powerflows and be able to return the next observation (after `env.step(...)`). These \"backends\" can be faster. For example, we strongly encourage you to use the \"lightsim2grid\" backend.\n",
    "\n",
    "You can install it with `pip install lightsim2grid`\n",
    "\n",
    "And use it with:\n",
    "```python\n",
    "import grid2op\n",
    "from lightsim2grid import LightSimBackend\n",
    "env_name = \"l2rpn_case14_sandbox\"\n",
    "env = grid2op.make(env_name+\"_train\", backend=LightSimBackend(), ...)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I) Action representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What's this ?\n",
    "\n",
    "The Grid2op package has been built with an \"object-oriented\" perspective: almost everything is encapsulated in a dedicated `class`. This allows for more customization of the plateform.\n",
    "\n",
    "The downside of this approach is that machine learning methods, especially in deep learning, often prefer to deal with vectors rather than with \"complex\" objects. Indeed, as we covered in the previous tutorials on the platform, we saw that building our own actions can be tedious and can sometime require important knowledge of the powergrid.\n",
    "\n",
    "On the contrary, in most of the standard Reinforcement Learning environments, actions have a higher level representation. For example in pacman, there are 4 different types of actions: \"turn left\", \"turn right\", \"go up\" and \"go down\". This allows for easy sampling (if you need to achieve an uniform sampling, you simply need to randomly pick a number between 0 and 3 included) and an easy representation: each action can be represented as a different component of a vector of dimension 4 [because there are 4 actions]. \n",
    "\n",
    "On the other hand, this representation is not \"human friendly\". It is quite convenient in the case of pacman because the action space is rather small, making it possible to remember which action corresponds to which component, but in the case of the grid2op package, there are hundreds or even thousands of actions. We suppose that we do not really care about this here, as tutorials on Reinforcement Learning with discrete action space often assume that actions are labeled with integers (such as in pacman for example).\n",
    "\n",
    "Converting grid2op actions into \"machine readable\" ones is the major difficulty as there is no unique ways to do so. In grid2op we offer some pre defined \"functions\" to do so:\n",
    "\n",
    "- `BoxGymObsSpace` will convert the action space into a gym \"Box\". It is rather straightforward, especially for **continuous** type of actions (such as *redispatching*, *curtailment* or actions on *storage units*). Representing the discrete actions (on powerlines and on substation) is not an easy task with them. We would not recommend to use them if your focus is on topology. More information on https://grid2op.readthedocs.io/en/latest/gym.html#grid2op.gym_compat.BoxGymActSpace\n",
    "- `MultiDiscreteActSpace` is similar to `BoxGymObsSpace` but mainly focused on the **discrete** actions (*lines status* and *substation reconfiguration*). Actions are represented with a gym \"MultiDiscrete\" space. It allows to perform any number of actions you want (which might be illegal) but comes with little restrictions. It handles continuous actions through \"binning\" (which is not ideal but doable). We recommend using this transformation if the algorithm you want to use is able to deal with \"MultiDiscrete\" gym action type. More information is given at https://grid2op.readthedocs.io/en/latest/gym.html#grid2op.gym_compat.MultiDiscreteActSpace\n",
    "- `DiscreteActSpace` is similar to `MultiDiscreteActSpace` in the sense that it focuses on **discrete** actions. It comes with a main restriction though: you can only do one action. For example, you cannot \"modify a substation\" AND \"disconnect a powerline\" with the same action. More information is provided at https://grid2op.readthedocs.io/en/latest/gym.html#grid2op.gym_compat.DiscreteActSpace. We recommend to use it if you want to focus on **discrete** actions and the algorithm you want to use is not able to deal with `MultiDiscreteActSpace`.\n",
    "- You can also fully customize the way you \"represent\" the action. More information is given in the notebook [11_IntegrationWithExistingRLFrameworks](11_IntegrationWithExistingRLFrameworks.ipynb)\n",
    "\n",
    "In the next section we will show an agent working with `DiscreteActSpace`. The code showed can be easily adapted with the other type of actions.\n",
    "\n",
    "### Create a gym compatible environment\n",
    "\n",
    "The first step is to \"convert\" your environment into a gym environment, for easier manipulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid2op.gym_compat import GymEnv\n",
    "import grid2op\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gymnasium import Env\n",
    "from gymnasium.utils.env_checker import check_env\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from lightsim2grid import LightSimBackend\n",
    "    bk_cls = LightSimBackend\n",
    "except ImportError as exc:\n",
    "    print(f\"Error: {exc} when importing faster LightSimBackend\")\n",
    "    from grid2op.Backend import PandaPowerBackend\n",
    "    bk_cls = PandaPowerBackend\n",
    "    \n",
    "env_name = \"l2rpn_case14_sandbox\"\n",
    "training_env = grid2op.make(env_name, backend=bk_cls())  # we put \"test=True\" in this notebook because...\n",
    "# it's a notebook to explain things. Of course, do not put \"test=True\" if you really want\n",
    "# to train an agent...\n",
    "gym_env = GymEnv(training_env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isinstance(gym_env, Env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\env_checker.py:270: UserWarning: \u001b[33mWARN: `check_env(warn=...)` parameter is now ignored.\u001b[0m\n",
      "  logger.warn(\"`check_env(warn=...)` parameter is now ignored.\")\n",
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\passive_env_checker.py:81: UserWarning: \u001b[33mWARN: A Box action space maximum and minimum values are equal. Actual equal coordinates: [(0,), (1,), (5,)]\u001b[0m\n",
      "  logger.warn(\n",
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\passive_env_checker.py:81: UserWarning: \u001b[33mWARN: A Box action space maximum and minimum values are equal. Actual equal coordinates: [(2,), (3,), (4,)]\u001b[0m\n",
      "  logger.warn(\n",
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\env_checker.py:243: UserWarning: \u001b[33mWARN: For Box action spaces, we recommend using a symmetric and normalized space (range=[-1, 1] or [0, 1]). See https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html for more information.\u001b[0m\n",
      "  logger.warn(\n",
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\passive_env_checker.py:56: UserWarning: \u001b[33mWARN: A Box observation space maximum and minimum values are equal. Actual equal coordinates: [(2,), (3,), (4,)]\u001b[0m\n",
      "  logger.warn(\n",
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\passive_env_checker.py:56: UserWarning: \u001b[33mWARN: A Box observation space maximum and minimum values are equal. Actual equal coordinates: [(0,), (1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,), (10,), (11,), (12,), (13,)]\u001b[0m\n",
      "  logger.warn(\n",
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\env_checker.py:221: UserWarning: \u001b[33mWARN: A Box observation space minimum value is -infinity. This is probably too low.\u001b[0m\n",
      "  logger.warn(\n",
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\env_checker.py:225: UserWarning: \u001b[33mWARN: A Box observation space maximum value is -infinity. This is probably too high.\u001b[0m\n",
      "  logger.warn(\n",
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\gymnasium\\utils\\env_checker.py:317: UserWarning: \u001b[33mWARN: Not able to test alternative render modes due to the environment not having a spec. Try instantialising the environment through gymnasium.make\u001b[0m\n",
      "  logger.warn(\n"
     ]
    }
   ],
   "source": [
    "check_env(gym_env, warn=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, `gym_env` is really an environment from gym. It also meets the gym requirements and pass the checks performed by gym.\n",
    "\n",
    "By default however, the action space and observation space are Dictionnaries, which is not convenient for most machine learning algorithm (they need to be transformed into vectors somehow)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict('change_bus': MultiBinary(57), 'change_line_status': MultiBinary(20), 'curtail': Box([-1. -1.  0.  0.  0. -1.], [-1. -1.  1.  1.  1. -1.], (6,), float32), 'redispatch': Box([ -5. -10.   0.   0.   0. -15.], [ 5. 10.  0.  0.  0. 15.], (6,), float32), 'set_bus': Box(-1, 2, (57,), int32), 'set_line_status': Box(-1, 1, (20,), int32))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gym_env.action_space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'gym_env' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[1], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[43mgym_env\u001b[49m\u001b[38;5;241m.\u001b[39mobservation_space\n",
      "\u001b[1;31mNameError\u001b[0m: name 'gym_env' is not defined"
     ]
    }
   ],
   "source": [
    "gym_env.observation_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is why it is often a good idea to customize the environment to have proper observations and actions. \n",
    "\n",
    "This is covered more in depth on the notebook [11_IntegrationWithExistingRLFrameworks](11_IntegrationWithExistingRLFrameworks.ipynb) (especially for the observation space). \n",
    "\n",
    "Here we use simple converters to focus on the training of the agent. You can set them with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Discrete(219)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from grid2op.gym_compat import DiscreteActSpace\n",
    "gym_env.action_space = DiscreteActSpace(training_env.action_space,\n",
    "                                        attr_to_keep=[\"set_bus\" , \"set_line_status_simple\"])\n",
    "gym_env.action_space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Box(0.0, inf, (20,), float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from grid2op.gym_compat import BoxGymObsSpace\n",
    "gym_env.observation_space = BoxGymObsSpace(training_env.observation_space,\n",
    "                                           attr_to_keep=[\"rho\"])\n",
    "gym_env.observation_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now your agent will receive a \"box\" as observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0.30036613, 0.28311425, 0.2861164 , 0.21761145, 0.61658865,\n",
       "        0.17046662, 0.40539283, 0.66137135, 0.5095609 , 0.7850837 ,\n",
       "        0.12672818, 0.2863957 , 0.38630578, 0.50913906, 0.48141804,\n",
       "        0.4284782 , 0.39317185, 0.41320527, 0.45101961, 0.3583579 ],\n",
       "       dtype=float32),\n",
       " {'time serie id': 'c:\\\\Users\\\\thoma\\\\anaconda3\\\\envs\\\\MA\\\\lib\\\\site-packages\\\\grid2op\\\\data\\\\l2rpn_case14_sandbox\\\\chronics\\\\0002'})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs = gym_env.reset()\n",
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And will be able to interact with the environment with standard \"int\" (integers that are the normal way to represent \"Discrete\" gym space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'disc_lines': array([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
       "        -1, -1, -1]),\n",
       " 'is_illegal': False,\n",
       " 'is_ambiguous': False,\n",
       " 'is_dispatching_illegal': False,\n",
       " 'is_illegal_reco': False,\n",
       " 'reason_alarm_illegal': None,\n",
       " 'reason_alert_illegal': None,\n",
       " 'opponent_attack_line': None,\n",
       " 'opponent_attack_sub': None,\n",
       " 'opponent_attack_duration': 0,\n",
       " 'exception': [],\n",
       " 'time_series_id': 'c:\\\\Users\\\\thoma\\\\anaconda3\\\\envs\\\\MA\\\\lib\\\\site-packages\\\\grid2op\\\\data\\\\l2rpn_case14_sandbox\\\\chronics\\\\0002',\n",
       " 'rewards': {}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs, reward, terminated, truncated, info = gym_env.step(0)  # perform action labeled 0\n",
    "info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.26116648, 0.29671553, 0.15601616, 0.24886668, 0.7067254 ,\n",
       "       0.01935879, 0.4648358 , 0.6743001 , 0.5137373 , 0.78809553,\n",
       "       0.11703431, 0.26651052, 0.39410654, 0.5062262 , 0.48689505,\n",
       "       0.4520237 , 0.4006156 , 0.46336463, 0.4904254 , 0.35797244],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs, reward, terminated, truncated, info = gym_env.step(53)  # perform action labeled 53\n",
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Even better gym environment\n",
    "The environment show here directly \"maps\" a grid2op environment to a gym environment: at each \"grid2op step\" the agent is asked to chose an action.\n",
    "\n",
    "\n",
    "After a few itérations of the \"l2rpn\" competitions it appears that the vast majority of top performers all rely on \"heuristics\" to handle the problem showed in grid2op.\n",
    "\n",
    "For example, most of the time the \"agent\" does not take any actions when all the flows are bellow a certain threshold (say 90%) and only do actions when at least one powerline sees a flow above this limit. \n",
    "\n",
    "In `l2rpn_baselines` we embeded directly the possibility to have a \"gym environment\" that does some of these heuristics. For example, the \"gym environment\" keeps performing \"steps\" (with the do nothing action) while all the flows are bellow a certain threshold. Otherwise it asks for an action at the agent. In this setting, the agent is directly trained with the heuristics used at inference time.\n",
    "\n",
    "This is available in:\n",
    "\n",
    "```python\n",
    "from l2rpn_baselines.utils import GymEnvWithReco, GymEnvWithRecoWithDN\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II) Train an Agent\n",
    "\n",
    "In this tutorial we will train an agent that uses only discrete actions thanks to the \"stable baselines 3\" RL package. \n",
    "\n",
    "More precisely, we will use the \"PPO\" algorithm. We do not cover the details of such algorithm here.\n",
    "\n",
    "**NB** here we show a minimal complete code to get started. We recommend however to use the l2rpn_baselines `PPO_SB3` model (that integrates all this, which much better customization) for such purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cpu device\n",
      "Wrapping the env with a `Monitor` wrapper\n",
      "Wrapping the env in a DummyVecEnv.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\stable_baselines3\\ppo\\ppo.py:155: UserWarning: You have specified a mini-batch size of 8, but because the `RolloutBuffer` is of size `n_steps * n_envs = 2`, after every 0 untruncated mini-batches, there will be a truncated mini-batch of size 2\n",
      "We recommend using a `batch_size` that is a factor of `n_steps * n_envs`.\n",
      "Info: (n_steps=2 and n_envs=1)\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from stable_baselines3 import PPO\n",
    "nn_model = PPO(env=gym_env,\n",
    "               learning_rate=1e-3,\n",
    "               policy=\"MlpPolicy\",\n",
    "               policy_kwargs={\"net_arch\": [100, 100, 100]},\n",
    "               n_steps=2,\n",
    "               batch_size=8,\n",
    "               verbose=True,\n",
    "               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------\n",
      "| time/              |    |\n",
      "|    fps             | 19 |\n",
      "|    iterations      | 1  |\n",
      "|    time_elapsed    | 0  |\n",
      "|    total_timesteps | 2  |\n",
      "---------------------------\n",
      "-----------------------------------------\n",
      "| rollout/                |             |\n",
      "|    ep_len_mean          | 4           |\n",
      "|    ep_rew_mean          | 182         |\n",
      "| time/                   |             |\n",
      "|    fps                  | 16          |\n",
      "|    iterations           | 2           |\n",
      "|    time_elapsed         | 0           |\n",
      "|    total_timesteps      | 4           |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.039701223 |\n",
      "|    clip_fraction        | 0.3         |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -5.39       |\n",
      "|    explained_variance   | -0.00227    |\n",
      "|    learning_rate        | 0.001       |\n",
      "|    loss                 | 4.46e+03    |\n",
      "|    n_updates            | 10          |\n",
      "|    policy_gradient_loss | -0.0893     |\n",
      "|    value_loss           | 9.29e+03    |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| rollout/                |             |\n",
      "|    ep_len_mean          | 4           |\n",
      "|    ep_rew_mean          | 182         |\n",
      "| time/                   |             |\n",
      "|    fps                  | 19          |\n",
      "|    iterations           | 3           |\n",
      "|    time_elapsed         | 0           |\n",
      "|    total_timesteps      | 6           |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.042268544 |\n",
      "|    clip_fraction        | 0.25        |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -5.39       |\n",
      "|    explained_variance   | -0.00192    |\n",
      "|    learning_rate        | 0.001       |\n",
      "|    loss                 | 642         |\n",
      "|    n_updates            | 20          |\n",
      "|    policy_gradient_loss | -0.0795     |\n",
      "|    value_loss           | 1.33e+03    |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| rollout/                |             |\n",
      "|    ep_len_mean          | 4           |\n",
      "|    ep_rew_mean          | 182         |\n",
      "| time/                   |             |\n",
      "|    fps                  | 21          |\n",
      "|    iterations           | 4           |\n",
      "|    time_elapsed         | 0           |\n",
      "|    total_timesteps      | 8           |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.042270362 |\n",
      "|    clip_fraction        | 0.3         |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -5.39       |\n",
      "|    explained_variance   | 0.0013      |\n",
      "|    learning_rate        | 0.001       |\n",
      "|    loss                 | 4.85e+03    |\n",
      "|    n_updates            | 30          |\n",
      "|    policy_gradient_loss | -0.08       |\n",
      "|    value_loss           | 9.85e+03    |\n",
      "-----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 4          |\n",
      "|    ep_rew_mean          | 182        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 23         |\n",
      "|    iterations           | 5          |\n",
      "|    time_elapsed         | 0          |\n",
      "|    total_timesteps      | 10         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.05120638 |\n",
      "|    clip_fraction        | 0.35       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.38      |\n",
      "|    explained_variance   | -7.75e-05  |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 4.78e+03   |\n",
      "|    n_updates            | 40         |\n",
      "|    policy_gradient_loss | -0.0844    |\n",
      "|    value_loss           | 9.66e+03   |\n",
      "----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 4          |\n",
      "|    ep_rew_mean          | 182        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 24         |\n",
      "|    iterations           | 6          |\n",
      "|    time_elapsed         | 0          |\n",
      "|    total_timesteps      | 12         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.05928427 |\n",
      "|    clip_fraction        | 0.35       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.38      |\n",
      "|    explained_variance   | 0.000136   |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 4.8e+03    |\n",
      "|    n_updates            | 50         |\n",
      "|    policy_gradient_loss | -0.087     |\n",
      "|    value_loss           | 9.7e+03    |\n",
      "----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 4          |\n",
      "|    ep_rew_mean          | 182        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 25         |\n",
      "|    iterations           | 7          |\n",
      "|    time_elapsed         | 0          |\n",
      "|    total_timesteps      | 14         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.06397128 |\n",
      "|    clip_fraction        | 0.4        |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.38      |\n",
      "|    explained_variance   | 0.000158   |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 5e+03      |\n",
      "|    n_updates            | 60         |\n",
      "|    policy_gradient_loss | -0.0893    |\n",
      "|    value_loss           | 1.01e+04   |\n",
      "----------------------------------------\n",
      "-----------------------------------------\n",
      "| rollout/                |             |\n",
      "|    ep_len_mean          | 7.5         |\n",
      "|    ep_rew_mean          | 410         |\n",
      "| time/                   |             |\n",
      "|    fps                  | 24          |\n",
      "|    iterations           | 8           |\n",
      "|    time_elapsed         | 0           |\n",
      "|    total_timesteps      | 16          |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.065179765 |\n",
      "|    clip_fraction        | 0.35        |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -5.38       |\n",
      "|    explained_variance   | -3.24e-05   |\n",
      "|    learning_rate        | 0.001       |\n",
      "|    loss                 | 5e+03       |\n",
      "|    n_updates            | 70          |\n",
      "|    policy_gradient_loss | -0.0887     |\n",
      "|    value_loss           | 1.01e+04    |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| rollout/                |             |\n",
      "|    ep_len_mean          | 5.67        |\n",
      "|    ep_rew_mean          | 292         |\n",
      "| time/                   |             |\n",
      "|    fps                  | 24          |\n",
      "|    iterations           | 9           |\n",
      "|    time_elapsed         | 0           |\n",
      "|    total_timesteps      | 18          |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.103364766 |\n",
      "|    clip_fraction        | 0.55        |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -5.38       |\n",
      "|    explained_variance   | -4.76e-05   |\n",
      "|    learning_rate        | 0.001       |\n",
      "|    loss                 | 1.2e+03     |\n",
      "|    n_updates            | 80          |\n",
      "|    policy_gradient_loss | -0.101      |\n",
      "|    value_loss           | 2.41e+03    |\n",
      "-----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 5          |\n",
      "|    ep_rew_mean          | 248        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 23         |\n",
      "|    iterations           | 10         |\n",
      "|    time_elapsed         | 0          |\n",
      "|    total_timesteps      | 20         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.13034236 |\n",
      "|    clip_fraction        | 0.55       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.37      |\n",
      "|    explained_variance   | 3.97e-05   |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 1.14e+03   |\n",
      "|    n_updates            | 90         |\n",
      "|    policy_gradient_loss | -0.104     |\n",
      "|    value_loss           | 2.3e+03    |\n",
      "----------------------------------------\n",
      "---------------------------------------\n",
      "| rollout/                |           |\n",
      "|    ep_len_mean          | 5         |\n",
      "|    ep_rew_mean          | 248       |\n",
      "| time/                   |           |\n",
      "|    fps                  | 23        |\n",
      "|    iterations           | 11        |\n",
      "|    time_elapsed         | 0         |\n",
      "|    total_timesteps      | 22        |\n",
      "| train/                  |           |\n",
      "|    approx_kl            | 0.1801331 |\n",
      "|    clip_fraction        | 0.6       |\n",
      "|    clip_range           | 0.2       |\n",
      "|    entropy_loss         | -5.37     |\n",
      "|    explained_variance   | 1.05e-05  |\n",
      "|    learning_rate        | 0.001     |\n",
      "|    loss                 | 550       |\n",
      "|    n_updates            | 100       |\n",
      "|    policy_gradient_loss | -0.108    |\n",
      "|    value_loss           | 1.11e+03  |\n",
      "---------------------------------------\n",
      "-----------------------------------------\n",
      "| rollout/                |             |\n",
      "|    ep_len_mean          | 4.8         |\n",
      "|    ep_rew_mean          | 235         |\n",
      "| time/                   |             |\n",
      "|    fps                  | 23          |\n",
      "|    iterations           | 12          |\n",
      "|    time_elapsed         | 1           |\n",
      "|    total_timesteps      | 24          |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.068656355 |\n",
      "|    clip_fraction        | 0.35        |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -5.35       |\n",
      "|    explained_variance   | -1.47e-05   |\n",
      "|    learning_rate        | 0.001       |\n",
      "|    loss                 | 4.9e+03     |\n",
      "|    n_updates            | 110         |\n",
      "|    policy_gradient_loss | -0.0826     |\n",
      "|    value_loss           | 9.89e+03    |\n",
      "-----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 4.8        |\n",
      "|    ep_rew_mean          | 235        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 24         |\n",
      "|    iterations           | 13         |\n",
      "|    time_elapsed         | 1          |\n",
      "|    total_timesteps      | 26         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.27749115 |\n",
      "|    clip_fraction        | 0.7        |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.33      |\n",
      "|    explained_variance   | -4.17e-06  |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 544        |\n",
      "|    n_updates            | 120        |\n",
      "|    policy_gradient_loss | -0.114     |\n",
      "|    value_loss           | 1.09e+03   |\n",
      "----------------------------------------\n",
      "---------------------------------------\n",
      "| rollout/                |           |\n",
      "|    ep_len_mean          | 4.8       |\n",
      "|    ep_rew_mean          | 235       |\n",
      "| time/                   |           |\n",
      "|    fps                  | 24        |\n",
      "|    iterations           | 14        |\n",
      "|    time_elapsed         | 1         |\n",
      "|    total_timesteps      | 28        |\n",
      "| train/                  |           |\n",
      "|    approx_kl            | 0.1430957 |\n",
      "|    clip_fraction        | 0.6       |\n",
      "|    clip_range           | 0.2       |\n",
      "|    entropy_loss         | -5.33     |\n",
      "|    explained_variance   | -7.99e-06 |\n",
      "|    learning_rate        | 0.001     |\n",
      "|    loss                 | 5.07e+03  |\n",
      "|    n_updates            | 130       |\n",
      "|    policy_gradient_loss | -0.104    |\n",
      "|    value_loss           | 1.02e+04  |\n",
      "---------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 4.8        |\n",
      "|    ep_rew_mean          | 235        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 25         |\n",
      "|    iterations           | 15         |\n",
      "|    time_elapsed         | 1          |\n",
      "|    total_timesteps      | 30         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.15314603 |\n",
      "|    clip_fraction        | 0.6        |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.33      |\n",
      "|    explained_variance   | 7.15e-07   |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 5.15e+03   |\n",
      "|    n_updates            | 140        |\n",
      "|    policy_gradient_loss | -0.105     |\n",
      "|    value_loss           | 1.04e+04   |\n",
      "----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 5.33       |\n",
      "|    ep_rew_mean          | 272        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 25         |\n",
      "|    iterations           | 16         |\n",
      "|    time_elapsed         | 1          |\n",
      "|    total_timesteps      | 32         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.16055435 |\n",
      "|    clip_fraction        | 0.6        |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.32      |\n",
      "|    explained_variance   | 1.66e-05   |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 5.31e+03   |\n",
      "|    n_updates            | 150        |\n",
      "|    policy_gradient_loss | -0.106     |\n",
      "|    value_loss           | 1.07e+04   |\n",
      "----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 5.33       |\n",
      "|    ep_rew_mean          | 272        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 25         |\n",
      "|    iterations           | 17         |\n",
      "|    time_elapsed         | 1          |\n",
      "|    total_timesteps      | 34         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.36924535 |\n",
      "|    clip_fraction        | 0.7        |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.31      |\n",
      "|    explained_variance   | 9.48e-06   |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 612        |\n",
      "|    n_updates            | 160        |\n",
      "|    policy_gradient_loss | -0.116     |\n",
      "|    value_loss           | 1.23e+03   |\n",
      "----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 5.14       |\n",
      "|    ep_rew_mean          | 260        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 25         |\n",
      "|    iterations           | 18         |\n",
      "|    time_elapsed         | 1          |\n",
      "|    total_timesteps      | 36         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.20377463 |\n",
      "|    clip_fraction        | 0.65       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.27      |\n",
      "|    explained_variance   | -0.000245  |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 4.72e+03   |\n",
      "|    n_updates            | 170        |\n",
      "|    policy_gradient_loss | -0.11      |\n",
      "|    value_loss           | 9.52e+03   |\n",
      "----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 5.14       |\n",
      "|    ep_rew_mean          | 260        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 25         |\n",
      "|    iterations           | 19         |\n",
      "|    time_elapsed         | 1          |\n",
      "|    total_timesteps      | 38         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.37223315 |\n",
      "|    clip_fraction        | 0.75       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.26      |\n",
      "|    explained_variance   | -1.73e-05  |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 546        |\n",
      "|    n_updates            | 180        |\n",
      "|    policy_gradient_loss | -0.119     |\n",
      "|    value_loss           | 1.09e+03   |\n",
      "----------------------------------------\n",
      "-----------------------------------------\n",
      "| rollout/                |             |\n",
      "|    ep_len_mean          | 5.14        |\n",
      "|    ep_rew_mean          | 260         |\n",
      "| time/                   |             |\n",
      "|    fps                  | 26          |\n",
      "|    iterations           | 20          |\n",
      "|    time_elapsed         | 1           |\n",
      "|    total_timesteps      | 40          |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.104675114 |\n",
      "|    clip_fraction        | 0.3         |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -5.24       |\n",
      "|    explained_variance   | 2.01e-05    |\n",
      "|    learning_rate        | 0.001       |\n",
      "|    loss                 | 4.96e+03    |\n",
      "|    n_updates            | 190         |\n",
      "|    policy_gradient_loss | -0.0743     |\n",
      "|    value_loss           | 9.95e+03    |\n",
      "-----------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 5.14       |\n",
      "|    ep_rew_mean          | 260        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 26         |\n",
      "|    iterations           | 21         |\n",
      "|    time_elapsed         | 1          |\n",
      "|    total_timesteps      | 42         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.25769198 |\n",
      "|    clip_fraction        | 0.7        |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.22      |\n",
      "|    explained_variance   | 8.52e-06   |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 4.8e+03    |\n",
      "|    n_updates            | 200        |\n",
      "|    policy_gradient_loss | -0.113     |\n",
      "|    value_loss           | 9.67e+03   |\n",
      "----------------------------------------\n",
      "---------------------------------------\n",
      "| rollout/                |           |\n",
      "|    ep_len_mean          | 5.14      |\n",
      "|    ep_rew_mean          | 260       |\n",
      "| time/                   |           |\n",
      "|    fps                  | 26        |\n",
      "|    iterations           | 22        |\n",
      "|    time_elapsed         | 1         |\n",
      "|    total_timesteps      | 44        |\n",
      "| train/                  |           |\n",
      "|    approx_kl            | 0.279845  |\n",
      "|    clip_fraction        | 0.7       |\n",
      "|    clip_range           | 0.2       |\n",
      "|    entropy_loss         | -5.2      |\n",
      "|    explained_variance   | -4.91e-05 |\n",
      "|    learning_rate        | 0.001     |\n",
      "|    loss                 | 4.79e+03  |\n",
      "|    n_updates            | 210       |\n",
      "|    policy_gradient_loss | -0.113    |\n",
      "|    value_loss           | 9.66e+03  |\n",
      "---------------------------------------\n",
      "----------------------------------------\n",
      "| rollout/                |            |\n",
      "|    ep_len_mean          | 5.75       |\n",
      "|    ep_rew_mean          | 299        |\n",
      "| time/                   |            |\n",
      "|    fps                  | 26         |\n",
      "|    iterations           | 23         |\n",
      "|    time_elapsed         | 1          |\n",
      "|    total_timesteps      | 46         |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.27931464 |\n",
      "|    clip_fraction        | 0.65       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -5.2       |\n",
      "|    explained_variance   | -2.46e-05  |\n",
      "|    learning_rate        | 0.001      |\n",
      "|    loss                 | 4.99e+03   |\n",
      "|    n_updates            | 220        |\n",
      "|    policy_gradient_loss | -0.111     |\n",
      "|    value_loss           | 1.01e+04   |\n",
      "----------------------------------------\n",
      "---------------------------------------\n",
      "| rollout/                |           |\n",
      "|    ep_len_mean          | 5.33      |\n",
      "|    ep_rew_mean          | 272       |\n",
      "| time/                   |           |\n",
      "|    fps                  | 26        |\n",
      "|    iterations           | 24        |\n",
      "|    time_elapsed         | 1         |\n",
      "|    total_timesteps      | 48        |\n",
      "| train/                  |           |\n",
      "|    approx_kl            | 0.4941588 |\n",
      "|    clip_fraction        | 0.8       |\n",
      "|    clip_range           | 0.2       |\n",
      "|    entropy_loss         | -5.2      |\n",
      "|    explained_variance   | -2.54e-05 |\n",
      "|    learning_rate        | 0.001     |\n",
      "|    loss                 | 579       |\n",
      "|    n_updates            | 230       |\n",
      "|    policy_gradient_loss | -0.121    |\n",
      "|    value_loss           | 1.16e+03  |\n",
      "---------------------------------------\n",
      "---------------------------------------\n",
      "| rollout/                |           |\n",
      "|    ep_len_mean          | 5         |\n",
      "|    ep_rew_mean          | 250       |\n",
      "| time/                   |           |\n",
      "|    fps                  | 26        |\n",
      "|    iterations           | 25        |\n",
      "|    time_elapsed         | 1         |\n",
      "|    total_timesteps      | 50        |\n",
      "| train/                  |           |\n",
      "|    approx_kl            | 0.453353  |\n",
      "|    clip_fraction        | 0.8       |\n",
      "|    clip_range           | 0.2       |\n",
      "|    entropy_loss         | -5.21     |\n",
      "|    explained_variance   | -7.27e-05 |\n",
      "|    learning_rate        | 0.001     |\n",
      "|    loss                 | 576       |\n",
      "|    n_updates            | 240       |\n",
      "|    policy_gradient_loss | -0.121    |\n",
      "|    value_loss           | 1.15e+03  |\n",
      "---------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<stable_baselines3.ppo.ppo.PPO at 0x199b0f1d040>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn_model.learn(total_timesteps=LEARNING_ITERATION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Important notes\n",
    "\n",
    "A few notes here:\n",
    "- the meta parameters (\"net_arch\", \"learning_rate\", \"n_steps\", \"gamma\", \"batch_size\", ...) are not fine tuned and are unlikely to lead to good results\n",
    "- the number of training steps is way too low (it should be much greater, probably in >= 1M)\n",
    "- there is no \"heuristics\" embeded here. Agents performs best when they act only when the grid is \"in danger\" (otherwise it takes a while for an agent to learn to \"do nothing\")\n",
    "- we use a \"test\" environment, this is why we specified \"test=True\" at the environment creation.\n",
    "- observations are not scaled\n",
    "- reward is not tuned for a particular goal: default reward is used and this might not be a good things to do for the particular agent we are trying to build.\n",
    "\n",
    "All of these probably lead to a quite terrible agent...\n",
    "\n",
    "### Other important notes\n",
    "\n",
    "We do not recommend to train \"real\" model with such simple features. In particular, you might need to save (at different stage of training) your agent, log the progress using tensorboard (for example) etc.\n",
    "\n",
    "This can be done with \"callbacks\" here that are not covered in this \"getting started\".\n",
    "\n",
    "\n",
    "### Yet another important note\n",
    "If you chose to use \"l2rpn baselines\" it will be as easy to learn an agent taking into account account all of the above if you us:\n",
    "\n",
    "```python\n",
    "from l2rpn_baselines.utils import GymEnvWithRecoWithDN\n",
    "from l2rpn_baselines.PP0_SB3 import train\n",
    "```\n",
    "\n",
    "You can have a look at the \"examples\" section of the l2rpn baselines repository (https://github.com/rte-france/l2rpn-baselines/tree/master/examples)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III) Evaluating the Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, it is time to test the agent that we trained on different scenarios (ideally) using another environment (to mimic a real process)\n",
    "\n",
    "To do that, we have two main choices:\n",
    "1) we use the trained model, convert the other environment to a gym env, run the environment and ask for the neural network what it should od\n",
    "2) embed the trained neural network in a \"grid2op agent\" class and use the built grid2op runner transparently\n",
    "\n",
    "Solution 1) is to be preferred for \"quick and dirty\" tests. In all other cases we strongly recommend to use solution 2 as it also allows you to not recode the \"gym loop\" to benefit from the saving functionality (as so to be able to inspect your agent in grid2viz for example) etc.\n",
    "\n",
    "We will code a minimal agent to leverage solution 2. As always, this is much easier to use with l2rpn_baselines package...\n",
    "\n",
    "### III A) Create the \"grid2op agent\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid2op.Agent import BaseAgent\n",
    "\n",
    "class MyPPOAgent(BaseAgent):\n",
    "    def __init__(self,\n",
    "                 action_space,  # grid2op action space\n",
    "                 used_trained_gym_env,  # gym env used for training\n",
    "                 trained_ppo_nnet,  # neural net after training\n",
    "                 ):\n",
    "        super().__init__(action_space)\n",
    "        self.nn_model = trained_ppo_nnet\n",
    "        self.gym_env = used_trained_gym_env\n",
    "    \n",
    "    def act(self, observation, reward, done=False):\n",
    "        gym_obs = self.gym_env.observation_space.to_gym(observation)\n",
    "        gym_act, _ = self.nn_model.predict(gym_obs, deterministic=True)\n",
    "        grid2op_act = self.gym_env.action_space.from_gym(gym_act)\n",
    "        return grid2op_act\n",
    "    \n",
    "my_agent = MyPPOAgent(training_env.action_space, gym_env, nn_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NB** Of course, if you use \"l2rpn_baselines\" you are not forced to recode it. Everything is properly set up for you to use directly the \"trained l2rpn_baselines agent\" in the grid2op runner !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### III.B) Perform the evaluation\n",
    "\n",
    "Now we create the test environment (in this case it's the same as the training one, but we really do recommend to have different environment with different data...)\n",
    "\n",
    "For a real experiment this could look like:\n",
    "\n",
    "```python\n",
    "training_env = grid2op.make(env_name+\"_test\", ...) \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\thoma\\anaconda3\\envs\\MA\\lib\\site-packages\\grid2op\\MakeEnv\\Make.py:438: UserWarning: You are using a development environment. This environment is not intended for training agents. It might not be up to date and its primary use if for tests (hence the \"test=True\" you passed as argument). Use at your own risk.\n",
      "  warnings.warn(_MAKE_DEV_ENV_WARN)\n"
     ]
    }
   ],
   "source": [
    "testing_env = grid2op.make(env_name, test=True, backend=bk_cls()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ebe924f4a584cd490f21d9caf2b40e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "episode:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8739142e8123454986e6165cae6b7b21",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "episode:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import shutil\n",
    "from tqdm.notebook import tqdm\n",
    "from grid2op.Runner import Runner\n",
    "from grid2op.Episode import EpisodeData\n",
    "\n",
    "save_path = \"saved_agent_PPO_{}\".format(LEARNING_ITERATION)\n",
    "path_save_results = \"{}_results\".format(save_path)\n",
    "shutil.rmtree(path_save_results, ignore_errors=True)\n",
    "\n",
    "test_runner = Runner(**testing_env.get_params_for_runner(),\n",
    "                     agentInstance=my_agent, agentClass=None)\n",
    "res = test_runner.run(nb_episode=EVAL_EPISODE,\n",
    "                      max_iter=MAX_EVAL_STEPS,\n",
    "                      pbar=tqdm,\n",
    "                      path_save=f\"./{path_save_results}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### III.B) Inspect the Agent "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please refer to the official documentation for more information about the contents of the directory where the data is saved. Note that saving the information is triggered by the \"path_save\" argument of the \"runner.run\" function.\n",
    "\n",
    "The information contained in this output will be saved in a structured way and includes :\n",
    "For each episode :\n",
    "  - \"episode_meta.json\": json file that represents some meta information about:\n",
    "\n",
    "    - \"backend_type\": the name of the `grid2op.Backend` class used\n",
    "    - \"chronics_max_timestep\": the **maximum** number of timesteps for the chronics used\n",
    "    - \"chronics_path\": the path where the temporal data (chronics) are located\n",
    "    - \"env_type\": the name of the `grid2op.Environment` class used.\n",
    "    - \"grid_path\": the path where the powergrid has been loaded from\n",
    "\n",
    "  - \"episode_times.json\": json file that gives some information about the total time spent in multiple parts of the runner, mainly the\n",
    "    `grid2op.Agent` (and especially its method `grid2op.Agent.act`) and the\n",
    "    `grid2op.Environment`\n",
    "\n",
    "  - \"_parameters.json\": json representation of the `grid2op.Parameters` used for this episode\n",
    "  - \"rewards.npy\": numpy 1d-array giving the rewards at each time step. We adopted the convention that the stored\n",
    "    reward at index `i` is the one observed by the agent at time `i` and **NOT** the reward sent by the\n",
    "    `grid2op.Environment` after the action has been taken.\n",
    "  - \"exec_times.npy\": numpy 1d-array giving the execution time for each time step in the episode\n",
    "  - \"actions.npy\": numpy 2d-array giving the actions that have been taken by the `grid2op.Agent`. At row `i` of \"actions.npy\" is a\n",
    "    vectorized representation of the action performed by the agent at timestep `i` *ie.* **after** having observed\n",
    "    the observation present at row `i` of \"observation.npy\" and the reward showed in row `i` of \"rewards.npy\".\n",
    "  - \"disc_lines.npy\": numpy 2d-array that tells which lines have been disconnected during the simulation of the cascading failure at each\n",
    "    time step. The same convention has been adopted for \"rewards.npy\". This means that the powerlines are\n",
    "    disconnected when the `grid2op.Agent` takes the `grid2op.Action` at time step `i`.\n",
    "  - \"observations.npy\": numpy 2d-array representing the `grid2op.Observation` at the disposal of the\n",
    "    `grid2op.Agent` when he took his action."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can first look at the repository were the data is stored:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0000',\n",
       " 'dict_action_space.json',\n",
       " 'dict_attack_space.json',\n",
       " 'dict_env_modification_space.json',\n",
       " 'dict_observation_space.json']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.listdir(path_save_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see there are 2 folders, each corresponding to a chronics.\n",
    "There are also additional json files.\n",
    "\n",
    "Now let's see what is inside one of these folders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('c:\\\\Users\\\\thoma\\\\MA\\\\Grid2Op\\\\getting_started\\\\saved_agent_PPO_50_results',\n",
       "  '0000')]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EpisodeData.list_episode(path_save_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we can load the actions chosen by the Agent, and have a look at them.\n",
    "\n",
    "To do that, we will load the action array and use the `action_space` function to convert it back to `Action` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_episodes = EpisodeData.list_episode(path_save_results)\n",
    "this_episode = EpisodeData.from_disk(*all_episodes[0])\n",
    "li_actions = this_episode.actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This allows us to have a deeper look at the actions, and their effects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will inspect the actions that has been taken by the agent :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reconnected lines : 0\n",
      "Total disconnected lines : 0\n"
     ]
    }
   ],
   "source": [
    "line_disc = 0\n",
    "line_reco = 0\n",
    "for act in li_actions:\n",
    "    dict_ = act.as_dict()\n",
    "    if \"set_line_status\" in dict_:\n",
    "        line_reco +=  dict_[\"set_line_status\"][\"nb_connected\"]\n",
    "        line_disc +=  dict_[\"set_line_status\"][\"nb_disconnected\"]\n",
    "print(f'Total reconnected lines : {line_reco}')\n",
    "print(f'Total disconnected lines : {line_disc}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, during this episode, our agent never tries to disconnect or reconnect a line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also analyse the observations of the recorded episode :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of disconnected powerlines cumulated over all the timesteps : 0\n"
     ]
    }
   ],
   "source": [
    "li_observations = this_episode.observations\n",
    "nb_real_disc = 0\n",
    "for obs_ in li_observations:\n",
    "    nb_real_disc += (obs_.line_status == False).sum()\n",
    "print(f'Total number of disconnected powerlines cumulated over all the timesteps : {nb_real_disc}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the kind of actions that the agent chose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The agent did 1 different valid actions.\n"
     ]
    }
   ],
   "source": [
    "actions_count = {}\n",
    "for act in li_actions:\n",
    "    act_as_vect = tuple(act.to_vect())\n",
    "    if not act_as_vect in actions_count:\n",
    "        actions_count[act_as_vect] = 0\n",
    "    actions_count[act_as_vect] += 1\n",
    "print(\"The agent did {} different valid actions.\".format(len(actions_count)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actions chosen by the agent were :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT modify any storage capacity\n",
      "\t - NOT perform any curtailment\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element(s):\n",
      "\t \t - Assign bus 1 to line (extremity) id 3 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (extremity) id 5 [on substation 3]\n",
      "\t \t - Assign bus 2 to line (origin) id 6 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 15 [on substation 3]\n",
      "\t \t - Assign bus 1 to line (origin) id 16 [on substation 3]\n",
      "\t \t - Assign bus 1 to load id 2 [on substation 3]\n"
     ]
    }
   ],
   "source": [
    "for act in li_actions:\n",
    "    print(act)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV) Improve your Agent \n",
    "\n",
    "As we stated above, the goal of this notebook was not to show a \"state of the art\" agent but rather to explain with a \"minimal example\" how you could get started into training an agent operating a powergrid.\n",
    "\n",
    "To improve your agent, we strongly recommend to use the l2rpn_baselines repository, to fine tune the hyper parameters of your agents, to train it for longer on more diverse data, to use a different RL algorithm (maybe PPO is not the best one for this environment ?) etc. etc.\n",
    "\n",
    "Using some pre training (for example training the policy in a supervised fashion using heuristic of human demonstration data) might be a good solution.\n",
    "\n",
    "Another promising tool would be to use some \"curiculum learning\" where, at the beginning, the agent would interact with a simplified version of the environment (easier rules, no disconnection due to overflow, no cooldown after an action has been made,  etc.) and after some time, when the agent is starting to perform well to increase this difficulty.\n",
    "\n",
    "Yet another possibility would be to access some kind of \"model based\" reinforcement learning, such as alpha-* models (*eg* alpha go)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}