The Atari Data Scraper

Reinforcement learning allows an agent to learn from interacting with an environment iteratively, learning sequences of actions in order to perform tasks and reach goals [15]. As reinforcement learning algorithms have achieved new records on benchmarks, they have also become more complex; many of the top-performing reinforcement learning algorithms today use deep neural networks, which are considered black-box algorithms. It is unlikely that significantly less complex or more transparent reinforcement learning algorithms will be able to achieve the same performance, in part because reinforcement learning incorporates the temporal aspects of problem solving. Additionally, reinforcement learning algorithms perform best when they have access to more information about the environment. Consequently, top-performing algorithms to have the capability to represent complex features, often in non-linear ways.

As deep reinforcement learning algorithms approach and surpass human capabilities in some domains, a new approach to interpreting and explaining such agents may be needed. Suppose we treat deep reinforcement learning algorithms as if they were human-like subjects. In doing so, we could use select methodologies from studies of human subjects in sociology, marketing, psychology, environmental sciences and more. In biological field research, for example, animals under observation may be fitted with a device to record specific information. As fitness trackers have developed, some sociological studies have similarly asked participants to wear devices to track information like daily activity levels.

Current commonly-used reinforcement learning libraries like OpenAI’s Baselines¹¹1https://github.com/openai/baselines, Tensorforce²²2https://github.com/openai/baselines , keras-rl³³3https://github.com/keras-rl/keras-rl, TF-Agents⁴⁴4https://www.tensorflow.org/agents and more make it difficult and time-consuming to collect additional data about deep reinforcement learning agents beyond simply their scores. However, we argue, improving access to data regarding the agent’s actions, rewards, location, etc. could help to make deep reinforcement learning agents more interpretable. Consequently, users from novices to experts get more out of each time an agent interacts with an environment. In order to facilitate a movement towards more data-availability, we have created a function that creates an Atari Data Scraper. The Atari Data Scraper collects information about agents as they interact with various games from the Atari 2600 suite of OpenAI’s gym [1] and saves the data for later examination.

The remainder of this article is structured as follows: Section II covers relevant background on interpretable deep reinforcement learning. Section III details the creation of a data scraper for use with the popular Stable Baselines deep reinforcement learning library. Section IV features the outcome from running two agents with an Atari Data Scraper attached. Section V concludes with a discussion of the Atari Data Scraper’s known limitations and the resulting directions for future work.

Deep reinforcement learning’s use of deep neural networks as function approximators makes it inherently difficult to interpret. The first algorithm to successfully learn from pixel-only input was the Deep Q-Network algorithm proposed by Mnih et al. in 2015 [10]. The algorithm, which is today referred to as DQN, consists of the original DQN algorithm with three improvements, all of which were proposed within a year of the groundbreaking paper’s publication. In the paper advocating for one of those three improvements, Wang et al. used saliency maps to visualize how Dueling Q-Networks altered the functioning of the original DQN algorithm [16], showing that from the very start of deep reinforcement learning, visualizations were being used to interpret algorithms and their resulting agents. Saliency maps have played a large role in this, with some researchers examining in detail how well various existing saliency-creation methods work in deep reinforcement learning [7] [13]. Others are proposing entirely new methods tailored specifically to the challenges and capabilities of deep reinforcement learning algorithms [4] [13] [11] [12].

However, visual inspection of saliency maps is not the only way deep reinforcement learning algorithms have been interpreted. There has long been a desire for more quantitative ways to interpret deep reinforcement learning agents. In a 2013 preprint of a less-developed DQN algorithm, Mnih et al. included a discussion about whether averaged reward or the estimated action-value function should be graphed to examine agent improvements and compare different agents [9]. Since then, in many published landmark deep reinforcement learning papers, graphs are prominently featured as figures designed to provide a summary of the claims in the papers. Even saliency maps have been compared using distance measurements and other quantitative evaluation metrics rather than pure visual inspection [2].

As deep reinforcement learning has developed, the search for quantitative metrics has similarly evolved. In 2020, Sequeira et al. proposed a framework using records from an agent’s interaction with an environment to make a more interpretable agent [14]. They showed how the framework might be used on a traditional Q-learning agent, but plan to extend the framework so that it can be used for DRL agents, too. The framework collected data such as how often a state was visited, current estimates of the values of states and actions, and the agent’s uncertainty in each state. By collecting such data, the authors were able to do things like finding sequences from bad situations to good situations, or locate places in the environment where the agent is generally uncertain.

In a similar vein, our Atari Data Scraper project seeks to uncover what could be learned by collecting data on a deep reinforcement learning agent, just as one might collect data on an animal species of interest in the wild. As a proof-of-concept, the first iteration of a data scraper was created as an additional class inserted into a fork of the OpenAI Baselines repository [3]. We modified the implementation of the Deep Q-Network (DQN) algorithm within the original repository to use this additional class to collect a set of data. After each time step, the data scraper object was given the following:

• •

  The number associated with the action taken at that time step

• •

  The reward received from the environment

• •

  An array of the sum total of each action for the current life

• •

  An array of the sum total of each action for the current game

• •

  A boolean indicating if a life had been lost on this step

• •

  The number of lives remaining

• •

  An array of the $Q$ values for this step

• •

  The feature vector describing the current game describing the current game screen

The first iteration of the data scraper concept used this dataset to create a record for the current step. For all data passed in, besides the current $Q$ values and the feature vector, the data was added as-is to the step record. Additional data was also collected to aid in creating visualizations. These include a flag to denote if the step was the end of a game, the running total reward for the current life and the current game, and the running total of steps taken for the current life and the current game Additionally, the feature vector was used to extract the x- and y-coordinates for Ms. Pacman and each of the four ghosts, which we will refer to as the characters in the environment. Then this data was used to calculate the distance between Ms. Pacman and each ghost, and the distance between Ms. Pacman and each of the four Power Pills. This data was in turn used to determine if Ms. Pacman had just eaten a Power Pill and, if so, mark that Power Pill as consumed for this game.

The initial iteration of the Atari Data Scraper only worked with the DQN implementation and the Ms.Pacman environment. By using the callback method, the improved Atari Data Scraper could easily be passed into any reinforcement learning algorithm’s implementation in the Stable Baselines library. We have tested and verified that the callback works with the Stable Baselines implementations of DQN, Advantage Actor Critic (A2C), and Proximal Policy Optimization (PPO2). We adjusted the callback to ensure it would work, not only for the Ms. Pacman environment, but also for the Pong environment. In the Pong environment, the two paddles and the ball are treated as characters, and their locations recorded at each time step along with the score and game number. In the Ms. Pacman environment, Ms. Pacman and the four ghosts are treated as characters. For both the Ms.Pacman environment and the Pong environment, we used the fourth version available through OpenAI gym. For each of the two environments, all three algorithms were run: PPO2, A2C, and DQN. A notebook with the visualizations made using the data collected by the Atari Data Scraper can be found in the Atari Data Scraper repository.

The DQN algorithm runs an agent through an environment, and data is collected about the characters in that environment. A2C and PPO2 can use multiple agents, each in its own environment, to do batch training. In situations with multiple environments running concurrently, the Atari Data Scraper collects information for each environment. For an agent trained to play the game Ms.Pacman using DQN, the information collected can be used to create a summary of all the games played by the agent, as shown in figure 1. In addition, we used the Atari Data Scraper with agents trained to play the game Ms. Pacman with both the A2C and PPO2 algorithms. For the agent trained using the A2C algorithm, the collected data was used to examine the quality of the games played in each of 4 concurrent environments over time as seen in figure 2. In figure 3 and figure 4, we show how the collected data can be filtered to display the top 3 best and worst performing games, with additional information about each game.

In Pong, the resulting data allowed us to examine the change in the distance between the agent’s paddle and the ball each time the agent missed the ball, allowing the opponent to score a point, as seen in figure 5. A game of Pong ends whenever the two players’ scores differ by 20 points. Using the Atari Data Scraper, we were also able to visualize the agent’s learning process in a more traditional manner, via the agent’s score at the end of each game. The plot in figure 6 starts consistently negative, as the opponent continues to get 20 more points than the agent quickly and easily, but gets into positive numbers more often as the agent learns how to score against the opponent.

In an ideal world, we could find a data collection mechanism that could be used across all existing or forthcoming deep reinforcement learning algorithms. It would likewise be applicable to all environments, be it part of the OpenAI gym framework or a custom environment. In reality, any mechanism of collecting data from a deep reinforcement learning agent needs an attachment into either the agent or the environment by which it can access and collect the data. Thus, we had to impose limitations on the Atari Data Scraper. Any one of these imposed limitations is a potential avenue for future work.

For one, there are some existing algorithm implementations which learn faster and have additional bells and whistles. In this project, we wanted to create a data collection mechanism that could be used easily by someone who is just trying to get their first deep reinforcement learning agent running, and also by a researcher who has worked in the field of deep reinforcement learning for years. For this reason, we avoided single-algorithm deep reinforcement learning implementations. Instead, we chose from among deep reinforcement learning libraries which have many of the most common algorithms implemented, any of which can be trained with a generic function call. The Stable Baselines library was chosen in part because it has more documentation and fewer bugs than other options we explored, creating a shallower learning curve for those just getting started in deep reinforcement learning. The actual collection process is set in motion by a function call specific to the Stable Baselines library, the callback. A clear limitation of the existing Atari Data Scraper is that is that this specific function call would not work in other deep reinforcement libraries.

The Atari Data Scraper was also specifically designed around Atari environments within the OpenAI gym framework. We chose to focus on this set of environments because it was less limiting: the popular gym framework generalizes much of deep reinforcement learning, and so an Atari-focused Data Scraper could more easily be generalized to a multitude of other existing environments which are registered in the OpenAI gym. In comparison, a data scraper designed around a single custom environment, especially one not built around the gym framework, would need significant re-writing to be adapted to any other environment. To that end, the data collection performed by the Atari Data Scraper could also be achieved by an OpenAI gym wrapper. Such wrappers also allow access to a function called on each step. Future work which implements the Atari Data Scraper as a gym wrapper would allow the Data Scraper to be generalized across deep reinforcement learning libraries and potentially even to some of the most cutting-edge algorithms. A data collection mechanism written as a wrapper would still be limited to only environments registered in the OpenAI gym. Other future work could choose instead to write data collection mechanisms for some of the most popular environments for deep reinforcement learning outside of the gym framework.

However, this would still leave one of the most restrictive limitations on the current Atari Data Scraper in place. The biggest limitation is the method by which characters and items in the environment are located. In order to locate characters, the Atari Data Scraper saves a screenshot of each step, requiring a lot of storage space. The screenshot is then searched, using image processing to find pixels within hand-crafted color ranges. Thus, to be applied to a new environment, the Atari Data Scraper must be equipped with painstakingly calculated color ranges for individual objects in the environment. Logically, this also limits the Atari Data Scraper to environments where the color of important objects does not appear elsewhere in the environment. Future work could include developing a method of automatically tracking items in the environment, ideally in a way that also allows broader generalization across the gym framework and across various libraries and implementations.

Finally, the Atari Data Scraper uses the reward signal to infer other information. For example, in Ms. Pacman, eating a Power Pill earns the agent a reward of positive fifty. The Atari Data Scraper uses this fact, along with the agent’s coordinates, to record when each Power Pill is eaten in a game. Thus, the Atari Data Scraper will not work as well for environments with clipped rewards or with rewards that do not differentiate between events. For example, in Ms. Pacman, eating one type of fruit and one type of ghost result in the same reward, making it much more difficult to determine which event occurred just by looking at the current reward signal. Additional future work could seek other ways of accessing the information currently carried in rewards signals. Alternatively, it could focus on transforming any rewards signal so that all important events provide differing rewards.

Part of this work has taken place in the Intelligent Robot Learning Laboratory at the University of Alberta (IRLL), supported in part by research grants from the Alberta Machine Intelligence Institute (Amii), CIFAR, and NSERC. We would like to acknowledge the help provided in this project by the members of IRLL. We are honored to be researching alongside such hard-working people and grateful for all their help in developing these ideas, proof-reading our writing, and testing our code. We would especially like to thank Dr. Matthew Taylor for his assistance in putting this team together and his guidance throughout this project. In addition, we would like to thank Dr. Dustin Arendt at Pacific Northwest National Laboratory for his advice and guidance on this project.