{"id":7966,"date":"2020-10-30T16:01:20","date_gmt":"2020-10-30T16:01:20","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/10\/30\/how-to-make-sense-of-the-reinforcement-learning-agents\/"},"modified":"2020-10-30T16:01:20","modified_gmt":"2020-10-30T16:01:20","slug":"how-to-make-sense-of-the-reinforcement-learning-agents","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/10\/30\/how-to-make-sense-of-the-reinforcement-learning-agents\/","title":{"rendered":"How to Make Sense of the Reinforcement Learning Agents?"},"content":{"rendered":"
\n

By Piotr Januszewski<\/a>, Research Software Engineer and PhD Student<\/b><\/p>\n

\"\"<\/p>\n

Based on simply watching how an agent acts in the environment it is hard to tell anything about why it behaves this way and how it works internally. That\u2019s why it is crucial to establish metrics that tell WHY the agent performs in a certain way.<\/p>\n

This is challenging especially when the\u00a0agent doesn\u2019t behave the way we would like it to behave, \u2026 which is like always<\/strong>. Every AI practitioner knows that whatever we work on, most of the time it won\u2019t simply work out of the box (they wouldn\u2019t pay us so much for it otherwise).<\/p>\n

In this blog post, you\u2019ll learn\u00a0what to keep track of to inspect\/debug your agent learning trajectory<\/strong>. I\u2019ll assume you are already familiar with the Reinforcement Learning (RL) agent-environment setting (see Figure 1) and you\u2019ve heard about at least some of the most common RL\u00a0algorithms<\/a>\u00a0and\u00a0environments<\/a>.<\/p>\n

Nevertheless, don\u2019t worry if you are just beginning your journey with RL. I\u2019ve tried to not depend too much on readers\u2019 prior knowledge and where I couldn\u2019t omit some details, I\u2019ve put references to useful materials.<\/p>\n

\n\"Figure\"
<\/p>\n

Figure 1: The Reinforcement Learning framework (Sutton & Barto, 2018).<\/em><\/p>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

I\u2019ll start by\u00a0discussing<\/strong>\u00a0useful\u00a0metrics<\/strong>\u00a0that give us a glimpse into the training and decision processes of the agent.<\/p>\n

Then we will focus on the\u00a0aggregation statistics of these metrics<\/strong>, like average, that will help us analyze them for many episodes played by the agent throughout the training. These will help root cause any issues with the agent.<\/p>\n

At each step, I\u2019ll base my suggestions on my own experience in RL research. Let\u2019s jump right into it!<\/p>\n

\u00a0<\/p>\n

Metrics I use to inspect RL agent training<\/h3>\n

\u00a0
There are multiple types of metrics to follow and each of them gives you different information about the model\u2019s performance. So the researcher can get the information about…
\u00a0<\/p>\n

…how is the agent doing<\/strong><\/h3>\n

\u00a0
Here, we will take a closer look at three metrics that diagnose the overall performance of the agent.<\/p>\n

\n\"Figure\"
<\/p>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

Episode return<\/strong><\/b><\/p>\n

This is what we care about the most. The whole agent training is all about getting to the\u00a0highest expected return possible<\/strong>\u00a0(see Figure 2). If this metric goes up throughout the training, it\u2019s a good sign.<\/p>\n

\"\"
\u00a0<\/p>\n

\n\"Figure\"
<\/p>\n
Figure 2:\u00a0The RL Problem<\/a>. Find a policy \u03c0 that maximizes the objective J. The objective J is an expected return E[R] under the environment dynamics P. \u03c4 is the trajectory played by the agent (or its policy \u03c0).<\/em><\/div>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

However, it\u2019s much more useful to us when we know what return to expect, or what is a good score.<\/p>\n

That\u2019s why you should\u00a0always look for baselines<\/strong>, others result in an environment you work on, to compare your results with them.<\/p>\n

Random agent baseline is often a good start and allows you to recalibrate, feel what is true \u201czero\u201d score in the environment \u2013 the minimal return you can get simply from bunging into the controller (see Figure 3).<\/p>\n

\"\"<\/p>\n

\n\"Figure\"
<\/p>\n
Figure 3. Table 3 from the\u00a0SimPLe<\/a>\u00a0paper with their results on Atari environments compared to many baselines alongside the random agent and human scores.<\/em><\/div>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

Episode length<\/strong><\/b><\/p>\n

This is a useful metric to analyze in conjunction with the episode return. It tells us if our agent is able to live for some time before termination. In MuJoCo environments, where diverse creatures learn to walk (see Figure 4), it tells you e.g. if your agent does some moves before flipping and resetting to the beginning of the episode.<\/p>\n

\n\"Figure\"
<\/p>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

Solve rate<\/strong><\/b><\/p>\n

Yet another metric to analyze with episode return. If your environment has a\u00a0notion of being solved<\/strong>, then it\u2019s useful to check how many episodes it can solve. For instance, in Sokoban (see Figure 5) there are partial rewards for pushing a box onto a target. That being said, the room is only solved when all boxes are on targets.<\/p>\n

\n\"Figure\"
<\/p>\n

Figure 5. Sokoban is a transportation puzzle, where the player has to push all boxes in the room on the storage targets.<\/em><\/p>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

So, it is possible for the agent to have a positive episode return, but still don\u2019t finish the task it is required to solve.<\/p>\n

One more example can be Google Research Football (see Figure 6) with its academies. There are some partial rewards for moving towards the opponents\u2019 goal, but the academy episode (e.g. exercising counterattack situation in smaller groups) is only considered \u201csolved\u201d when the agent\u2019s team scores a goal.<\/p>\n

\n\"Figure\"
<\/p>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

\u2026progress of training<\/strong><\/h3>\n

\u00a0
There are multiple ways of representing the notion of \u201ctime\u201d and against what to measure progress in RL. Here are the top 4 picks.<\/p>\n

Total environment steps<\/strong><\/b><\/p>\n

This simple metric tells you\u00a0how much experience, in terms of environment steps or timesteps, the agent already gathered<\/strong>. This is often more informative on training advancement (steps) than wall-time, which highly depends on how fast your machine can simulate the environment and do calculations on a neural network (see Figure 6).<\/p>\n

\"\"<\/p>\n

\n\"Figure\"
<\/p>\n

Figure 6. DDPG training on the MuJoCo Ant environment. Both runs took 24h, but on different machines. One did ~5M steps and the other ~9.5M. For the latter, it was enough time to converge. For the former not and it scored worse.<\/em><\/p>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

Moreover, we report the final agent score together with how much environment steps (often called samples) it took to train it.\u00a0The higher the score with the fewer samples, the more sample efficient is the agent.<\/strong><\/p>\n

Training steps<\/strong><\/b><\/p>\n

We train neural networks with the Stochastic Gradient Descent (SGD) algorithm (see\u00a0Deep Learning Book<\/a>).<\/p>\n

The training steps metric tells us how many batch updates we did to the network<\/strong>. When training from the off-policy replay buffer, we can match it with total environment steps in order to better understand how many times, on average, each sample from the environment is shown to the network to learn from it:<\/p>\n

batch size * trainings steps \/ total environment steps = batch size \/ rollout length<\/em><\/p>\n

where\u00a0rollout length<\/em>\u00a0is the number of new timesteps we gather, on average, during the data collection phase in between training steps (when data collection and training are run sequentially).<\/p>\n

The above ratio, sometimes called training intensity,\u00a0shouldn\u2019t be below 1<\/strong>\u00a0as it would mean that some samples aren\u2019t shown even once to the network! In fact, it should be much higher than 1, e.g. 256 (as set in e.g. RLlib implementation of\u00a0DDPG<\/a>, look for \u201ctraining intensity\u201d).<\/p>\n

Wall time<\/strong><\/b><\/p>\n

This simply tells us\u00a0how much time an experiment is running<\/strong>.<\/p>\n

It can be useful when planning in the future how much time do we need for each experiment to simply finish:<\/p>\n