Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks

Minigrid and Miniworld were originally created at Mila - Québec AI Institute to be primarily used by graduate students. Due to the variety in usages, customizability and simplicity were the highest priority to allow as many users to use and understand the codebase. To support this, Python and Gym’s RL environment API [3] (now updated to Gymnasium due to Gym no longer being maintained) was selected to implement the libraries due to their popularity within the machine learning and reinforcement learning communities. Example code for interacting with the environment is provided in Listing 2.

An additional core design point was to intentionally have as few external dependencies as possible, as fewer dependencies make these packages easier to install and less likely to break. As a result, Minigrid uses NumPy for the GridWorld backend along with the graphics to generate icons for each cell. Miniworld uses Pyglet for graphics with the environments being essentially 2.5D due to the use of a flat floorplan, which allows for a number of simplifications compared to a true 3D engine. This allows the libraries to run relatively fast but more importantly enables users to understand the whole environment implementations and customize them for their own needs.

Each Minigrid environment is a 2D GridWorld made up of $n\times m$ tiles where each tile is either empty or occupied by an object, e.g., a wall, key, or goal. Using different tile configurations, tasks of varying complexity can be constructed. By default, the environments are deterministic with no randomness in the transition function ($\mathcal{T}$).

By default, agent observations ($\mathcal{O}$) are a dictionary with three items: ‘‘image’’, ‘‘direction’’, and ‘‘mission’’. Example environments with their corresponding ‘‘image’’ and ‘‘mission’’ are provided in Figure 3. The ‘‘image’’ observation is a top-down render of the agent’s view which can be limited to a fixed distance or of the whole environment. The direction is an integer representing the direction the agent is facing. The ‘‘mission’’ is a text-based instruction specifying the task to solve. The ‘‘mission’’ can change on each environment’s reset as the goal might change, for example, “Minigrid-GoToObject” has a mission of ‘‘go to the {color} {obj_type}’’ where color can be one of [‘‘red’’, ‘‘green’’, ‘‘blue’’, ‘‘purple’’, ‘‘yellow’’, ‘‘grey’’] and obj_type can be one of [‘‘key’’, ‘‘ball’’, ‘‘box’’]. As a result, the instructions can be encoded as a one-hot vector. For more complex instructions, a language model is required to interpret for an RL agent.

The agents have a discrete action space ($\mathcal{A}$) of seven options representing [‘‘turn left’’, ‘‘turn right’’, ‘‘move forward’’, ‘‘pickup’’, ‘‘drop’’, ‘‘toggle’’, ‘‘done’’]. These actions are consistent across all environments however some actions might not produce any effect in certain states, e.g. ‘‘pickup’’ will not do anything if the agent is not next to an object that can be picked up.

The default reward function ($\mathcal{R}$) for the environment is sparse such that the reward is only non-zero when the mission is accomplished. Furthermore, the function can be easily customized for specific user needs through overwriting the environment’s MiniGridEnv._reward function.

Each Miniworld environment is a 3D world that consists of connected rooms with objects inside (e.g. box, ball, or key). Like Minigrid, the worlds can be configured for various tasks with different goals and complexity.

For the agent, the observation space ($\mathcal{O}$) is, by default, an RGB image of size $80\times 60$ from the agent’s perspective of the world. This image size can be modified by passing obs_width and obs_height arguments to the environment constructor. Five of the images in Figure 1 are example observations (with the final being a top-down view of the agent in an environment). To act in the world, agents are provided with a similar action space ($\mathcal{A}$) to Minigrid with an additional move-back action. Thus, there are in total eight discrete actions: [‘‘turn left’’, ‘‘turn right’’, ‘‘move forward’’, ‘‘move back’’, ‘‘pickup’’, ‘‘drop’’, ‘‘toggle’’, ‘‘done’’]. Like Minigrid, the default reward function is sparse with the agent only being rewarded when the agent completes the environment goal but can be modified in custom environments.

In both libraries, the environments can be created using a small set of functions. To demonstrate this feature, we showcase two sample scripts used to define the simulation environment in Listing 4, one for Minigrid and one for Miniworld.

The structure of the environment generation function is the same for more complex scenarios, with a few more helper functions. We have created tutorials for new environment creation: https://minigrid.farama.org/main/content/create_env_tutorial and https://miniworld.farama.org/main/content/create_env respectively. Both libraries can be directly integrated with existing RL libraries, e.g., Stable-Baselines3 (SB3). Additionally, to augment the libraries, we have created extra wrappers that customize the behavior of the libraries, such as adding stochastic actions and varying observation spaces, https://minigrid.farama.org/api/wrapper/ and https://github.com/Farama-Foundation/Miniworld/blob/master/miniworld/wrappers.py.

Since their creation, Minigrid and Miniworld have been widely adopted by the RL research community and used for various applications. Together, the two repositories have around 2400 stars and 620 forks on GitHub. We detail several instances where the two libraries have been utilized effectively.

Curriculum Learning: The two libraries provide a programmatic approach to creating new environments on-the-fly, this functionality can be utilized for automatic environment generation. For example, Dennis et al. [5] generated a natural curriculum of increasingly complex environments and Parker-Holder et al. [16] harnessed the power of evolution in a principled, regret-based curriculum.

Exploration: The reward function in the two libraries is, by default, a sparse reward making them ideal candidates for developing new exploration techniques. Using Minigrid and Miniworld, Seo et al. [21] developed an exploration approach using state entropy as the extrinsic reward and Zhang et al. [29] proposed a simple yet effective exploration criterion by equally weighting the novel areas.

Meta Learning & Transfer Learning: Given the ease of creating new simulation environments, the two libraries have also been used in developing meta-learning and transfer-learning algorithms. In Igl et al. [9], Minigrid has been used to develop regularization techniques to encourage agents to generalize to new environments. In Liu et al. [12], Miniworld is used to develop a new meta-learning approach that avoids local optima in end-to-end training, without sacrificing optimal exploration and Hutsebaut-Buysse et al. [8] explored the use of pre-trained task-independent word embedding for transfer learning.

In this case study, we showcase the ability to transfer policies learned on Minigrid to Miniworld. We created two similar simulation environments in Minigrid and in Miniworld, where the task is to follow an instruction to go to an object, we call the two environments minigrid-gotoobj-env and miniworld-gotoobj-env (screenshots of the two environments are shown in Figure 5). In both environments, the agent is given an instruction to “go to the {color} {object}" with the color and object being randomly selected from $[\texttt{``blue"},\texttt{``green"},\texttt{``grey"},\texttt{``purple"},\texttt{``red"},\texttt{``yellow"}]$ and $[\texttt{``ball"},\texttt{``box"},\texttt{``key"}]$, respectively.

When transferring the learned weights, one key question is which part of the agent’s weights should be transferred. We first trained a PPO agent [20] on minigrid-gotoobj-env and then we transferred the learned weights to the PPO agent for miniworld-gotoobj-env. The PPO policy consists of a mission instruction encoder, an image encoder, an actor network, and a critic network. The policy transfer is made easy due to the unified APIs for Minigrid and Miniworld. We tested with 12 different weight transfer options, the results are given in Table 1.

To measure the effectiveness of the transfer learning, we define the transfer improvement as

where Transfer_Learning_AUC represents the area under the curve (AUC) of the reward curve for the agent that is initialized with Minigrid learned weights, and Miniworld_Learning_AUC represents the AUC of the reward curve for the agent with randomly initialized weights. Both the transfer learning agent and the Miniworld learning agent are trained for 200k time steps. As Table 1 shows: (1) the transfer learning behavior was improved when the critic network and mission embedding weights were not frozen; (2) transferring only the critic network and mission embeddings produces better results compared with also transferring the actor network weights.

In this case study, we show how we can use the Minigrid and Miniworld libraries to collect and visualize human data. We utilized two similar environments in the Minigrid and Miniworld libraries, where there are four rooms and the goal is to reach a target position denoted with a green box, in the least amount of steps. In both cases, the human subject has only partial observation of the environment. We performed two sets of experiments. The first set of experiments lets the subject collect experience in the Minigrid environment for 10 episodes, then transfers to the Miniworld environment, and plays for another 10 episodes. The second set of experiments directly asks the subject to play on the Miniworld environment for 10 episodes. In both the Minigrid and Miniworld environments, the action space has dimension three with the actions: turn left, turn right, and go forward. To make the human experience more similar to the RL agent, we randomly assign the three actions to the 1-9 numbers keys on the keyboard. The average rewards over 10 episodes on the Miniworld environments are shown in Table 2 and a sample subject trajectory is shown in Figure 6. Similar to what is shown in Figure 6, we empirically observe an adaption phase to the random key assignment during the first episode for every human subject.

In this section, we discuss how the case studies were implemented using the Minigrid and Miniworld libraries. The RL agent transfer learning case study represents a custom experimental setting not natively supported by Minigrid, Miniworld, and SB3. Thus, on top of the two libraries, we implemented the following functionalities:

1. 1.

   created the minigrid-gotoobj-env (26 lines for _gen_grid()) and miniworld-gotoobj-env (19 lines for _gen_world()) environments;

2. 2.

   augmented miniworld-gotoobj-env with mission instructions (3 lines of code similar to the _gen_mission() from Minigrid);

3. 3.

   created a custom wrapper for minigrid-gotoobj-env to process the mission instructions (10 lines of code that are highly similar to the ImgObsWrapper in Minigrid);

4. 4.

   created a custom feature extractor in SB3 for minigrid-gotoobj-env (23 lines mostly copied from the SB3 NatureCNN class);

5. 5.

   created and trained a PPO agent on minigrid-gotoobj-env using SB3 (6 lines);

6. 6.

   transfer the learned policy from minigrid-gotoobj-env to miniworld-gotoobj-env (14-51 lines depending on the settings, can be reduced using a for loop).

In total, the implementation of this highly customized case study required 101 - 138 lines of code, highlighting the facility of use that our design provides. For the human transfer learning case study, we utilized two existing simulation environments, namely MiniGrid-FourRooms-v0 and MiniWorld-FourRooms-v0. The human action input is achieved using the already existing manual control feature. To render the human trajectories, we altered the existing manual control feature and included data recording and data plotting functionalities. Although additional effort was required, given the Pythonic design of the two libraries, it only required three hours of coding. A detailed list of the required implementations is provided in the supplementary materials. For research with similar purposes, we plan to open-source our case study implementations as a separate codebase.