A Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments

Learning through trial and error is a critical approach for humans to learn new concepts which has also been an inspiration for developing Reinforcement Learning (RL) methods [34]. In RL methods, the desired behavior is learned through interaction with the environment, where the agent receives feedback based on the decisions it makes. While RL methods have demonstrated remarkable performance across various domains, including video games [28, 21, 38], conventional recommender systems [44, 35, 19], and enhancing the responses of large language models [8, 26]. Their applications in many real-world situations such as autonomous driving [20, 3], medical usage [32, 31, 37] and recommendation systems in sensitive domains [31, 30] have been restricted due the potential to make dangerous and irreversible mistakes. This problem is more severe for model-free agents given that they don’t have any prior knowledge about the environment and they predominantly rely on a trial-and-error approach to learn the optimal policy.

In numerous real-world scenarios, the environment within which the agents interact demands careful consideration of safety. For instance, in the realm of autonomous cars, many safety features have been developed to identify possible collision situations and take preventive actions to avoid them. In the field of medical applications, providing suggestions for intricate and potentially dangerous situations often involves discussions led by professional experts, rather than relying solely on AI-based systems. Similarly, recommendation systems on movie platforms must consciously account for safety considerations, ensuring the appropriateness of suggestions for specific age ranges. In all the examples provided, diverse solutions for safety considerations have been implemented to identify hazardous states and prompt alternative responses beyond their initially planned actions. These responses may involve reducing the speed of the vehicle to avoid a collision, deferring diagnosis to experts instead of relying solely on AI, or executing predefined tasks such as excluding certain suggestions from the available options in a recommendation system.

Applying RL in real-world scenarios that demand safety considerations, similar to the examples mentioned earlier, introduces the risk of potentially dangerous mistakes due to the random exploration required during the training phase  [31, 12, 17]. On the other hand, exploration is an indispensable component of RL, allowing the agent to learn the optimal policy, especially in model-free settings where no information about the task or environment is provided to the agent. The inherent risk associated with the use of RL in these scenarios, arising from the free exploration strategy, makes employing RL an inefficient approach.

In this work, we propose a framework for training model-free RL agents to navigate grid environments where exploration requires consideration of safety constraints. These environments encompass hazardous states, the exploration of which might be dangerous or costly for the agent or the environment. Consequently, employing RL in such contexts amplifies the potential of encountering undesirable states due to unsupervised exploration strategies.

Our method empowers the RL agent to identify risky states, where the agent may navigate towards undesirable states through inappropriate actions selected by random exploration or a suboptimal policy. Subsequently, our framework enables the agent to recognize situations where free exploration becomes unsafe, prompting the selection of actions based on a safe policy rather than relying on random or suboptimal policies. Defining the safe policy is highly dependent on the environment and can be a set of predefined actions for the agent, relinquishing control to a human or supervisor, or employing any other secure approach as a substitute for the suboptimal RL policy that utilizes a free exploration strategy. The primary contribution of this paper lies in the design of a framework that can efficiently detect situations when free interaction (random exploration or using suboptimal policy) is unsafe for the model-free RL agent. The proposed method determines when free interaction should be replaced with a more reliable and secure strategy for selecting actions to learn the optimal policy while reducing the risk of visiting undesirable states in the training phase.

As shown in Figure 1 our proposed framework involves a pre-training phase conducted in a simulator or controlled segment of the environment, that has the same dynamics as the main environment but without hazardous consequences for entering undesirable states. During the pre-training phase, the agent interacts freely with the pre-training environment to train a model capable of identifying states that may not be initially undesirable but if the agent enters these states, inappropriate actions selected by random exploration or suboptimal policy may lead the agent into undesirable states. The model trained in the pre-training phase is then applied to the main environment, which the model-free agent has not encountered before. This enables the agent to recognize unsafe situations, where free interaction with the environment is risky. In such cases, future actions are selected using a safe policy which is defined particularly for each environment and task.

Given the inherent risk and uncertainty associated with applying RL methods, safety considerations have been extensively investigated within the realm of RL methods [6, 13, 12, 11, 37, 39, 40].

One of the most important safety considerations in RL revolves around safe exploration  [31, 12, 24, 16, 15, 27, 23] to avoid exposing the agent to dangerous states and consequently avoid harmful outcomes. Abeel et al. [1] proposed an apprenticeship framework to train an RL agent in safety-critical environments where the agent relies on a teacher that should act safely and near-optimally. Hans et al. [17] introduced a method to assess a state’s safety level, alongside a backup policy to navigate the system from critical to safe states. In [25] the need for a safe exploration strategy in Markov Decision Process (MDP) has been discussed, and a safe but potentially suboptimal exploration strategy for safety-constrained environments has been proposed. Turchetta et al. [39] proposed SAFEMDP algorithm which is more similar to our work and enables RL agents to explore the reachable portion of the MDP while adhering to safety constraints and gaining statistical confidence in unvisited state-action pairs using noisy observations.

Unlike many similar works on safe exploration techniques, our framework does not require prior knowledge about the distribution of unsafe states or initializing in a safe state. However, our framework needs pre-training in an environment with similar dynamics to the main environment in order to learn risky situations, given information provided in the state representation and/or agent observations.

Safe training for RL methods in safety-constrained environments has been extensively explored [41, 43, 36, 42, 2]. As a few examples, in [43] a model-free safe control strategy for deep reinforcement learning (DRL) agents has been suggested to prevent safety violation in the training phase. Thananjeyan et al. [36] proposed an algorithm called Recovery RL that leverages offline data in order to identify safety violation zones before policy learning and similar to our method, utilizes two different policies, one for learning the task and another for considering the safety constraints in the environment. Our framework proposes a more structured strategy to identify dangerous situations by computing reachable sets of undesirable states using high-level features that are observable for the agent. Another difference between our framework and methods similar to Recovery RL lies in defining the backup (safe) policy. Unlike these methods, our approach leverages a predefined safe policy instead of learning it from offline data. We propose defining a predefined safe policy and primarily focus on identifying dangerous situations in previously unseen environments with different state spaces and reward distributions, rather than learning the safe policy and applying it to a similar environment for a different task.

Another set of methods similar to ours addresses safety in reachability problems, aiming to enable RL agents to learn safe policies considering environmental safety constraints. Similar approaches [10, 9, 18, 22] have proposed enabling RL agents to attain a safe policy by computing reachable sets using Hamiltonian methods and defining a value function to assign negative values to reachable areas of undesirable states. Some key differences between these methods and our framework lie in applications, computational complexity, and the primary goal of the methods. Our method focuses on providing a safe learning framework, whereas other methods primarily analyze the safety levels of learned policies.

Similar to most RL problems, we formulate our problem of interest as a Markov Decision Process (MDP) which is defined as a tuple $(S,A,P,R,\gamma)$, where $S$ is the set of states; $A$ denotes the set of actions; $P(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)$ is the transition function that represents the dynamics of the environment and returns the probability of transitioning to a specific state $s_{t+1}\in S$ given the agent’s previous state $s_{t}\in S$ and the action $a_{t}\in A$ chosen at state $s_{t}$ state; $R:S\times A\rightarrow\Re$ is the reward function and $\gamma\in[0,1]$ is the discount factor.

We denote the set of failure states as $S_{f}\subset S$ which getting into them is dangerous or costly for the agent and/or the environment. The complementary set of $S_{f}$ is $S_{s}\subset S$ which contains all safe states that the agent can interact with to learn the optimal policy for the given task. Our objective is to equip the agent with the capability to identify states that upon transitioning to them, random exploration or utilizing a suboptimal policy may direct the agent into undesirable states $S_{f}$ within the next $t$ timesteps.

To determine these states, we utilize the computation of a set called Backward Reachable Set (BRS). BRS is widely used in various domains such as control and reachability problems  [9, 5], robot navigation [4], and reinforcement learning  [18, 10]. The BRS is the set of states $S_{BRS}\subset S$ in which the agent can end up within a set of states called target states $S_{target}\subset S$ within the next $t$ timesteps  [5]. In this study, we compute the BRS for our failure states. Thus our set of target states is $S_{f}$ and in the following sections, $S_{BRS}$ denotes the backward reachable set of failure states $S_{f}$.

We consider a trajectory of states $\xi_{s_{0}}^{\pi}=\{s_{0},s_{1},s_{2},...\}$, to be safe, when it starts from $s_{0}$ and follows a policy $\pi$ for a maximum of $T$ timesteps if and only if $\forall s_{j}\in\xi_{s_{0}}^{\pi};s_{j}\in S_{s}$ for $j\in[0,T]$. $S_{BRS}$ comprises of all states $s_{i}\in\xi_{s_{0}}^{\pi}$ leading to failure states within $t$ timesteps, i.e., $\exists k\in[0,t],$ s.t. $s_{i+k}\in S_{f}$. $S_{BRS}$ with respect to $S_{f}$ over a time horizon of maximum $t$ timesteps is defined as follows:

Our proposed framework includes two phases, namely pre-training and safe training. In the pre-training phase, a binary classification model is trained to detect dangerous states. In the safe training phase, this model is applied to identify dangerous states in a new environment. Upon detection, the agent switches from its current action selection strategy, which may involve random exploration or a suboptimal policy, to a more reliable policy known as the safe policy to prevent the agent from entering undesirable states.

We evaluated our framework through an autonomous navigation task designed with MiniGrid [7] platform. As shown in Figure 2 our designed environments contain a moving obstacle that the agent must navigate around and reach the goal state. The obstacles move vertically in all environments and change direction upon reaching the upper or lower boundary of the environments. Reaching the goal state or interacting with the environment for a duration exceeding a maximum threshold of timesteps terminates the episode with a reward of +1 and 0, respectively. Collision with the moving obstacles constitutes a safety constraint violation in this setting which terminates the episode with a reward of -1. The environment encompasses four actions: ”turn right”, ”turn left”, ”move forward”, and ”do nothing”, that turning actions alter the agent’s direction, while the ”move forward” action moves the agent by one state in its current direction. The ”do nothing” action is exclusively designated for our safe policy, as outlined later, and is only accessible when the agent employs the safe policy. The objective of each task in this setup is for the agent to learn a policy to reach the goal state without colliding with the moving obstacle. The state representation in this scenario includes the agent’s vertical and horizontal positions, as well as the vertical position of the obstacle, (given its exclusive vertical movement) and its direction. QLearning and SARSA, two instances of model-free RL algorithms with distinct characteristics, were examined to assess our framework. However, given that our framework primarily concerns exploration strategy rather than specific learning algorithm properties, any alternative model-free algorithm can be utilized in place of them.

We propose a framework to enhance the efficiency of model-free agents exploring new safety-constrained grid environments. In this section, we highlight a few limitations and considerations to be aware of when utilizing this framework for other domains.

Our framework offers insights into potentially unsafe states for the model-free agent. The primary contribution of our work lies in detecting instances where random exploration or suboptimal policies may pose risks in safety-constrained environments and should be substituted with more reliable approaches. However, defining a suitable safe policy depends on the specific environment and the task. In this work, to facilitate defining an appropriate safe policy, we narrowed the scope of the work to grid environments. While we demonstrate an instance of a safe policy as a set of predefined behaviors in our experiments, defining a reliable safe behavior can be challenging for certain environments, potentially diminishing the effectiveness of this framework.

One factor that may affect the efficacy of our method is the feasibility of designing an appropriate pre-training environment for the agent. In our problem of interest, navigating the grid environments, our pre-training environment comprised the similar dynamics of the main environments. However, in many real-world scenarios, designing an ideal pre-training environment may not be feasible due to the complexity and uncertainty of these environments. Therefore, it is crucial to consider this factor before deploying the proposed framework for a task.

Another limitation of our framework arises from the nature of training a machine learning model to detect unsafe situations and the inherent risk of false detections. Therefore, this framework does not guarantee error-free recognition. Consequently, in highly sensitive domains such as medical applications, relying solely on this framework may not be advisable.

Taking into account the mentioned limitations, our framework is most effective for tasks where we can design a pre-training zone with properties similar to the real environment and define a reliable safe policy. As a few examples of potential real-world applications of our framework, we can mention the task of indoor and outdoor autonomous navigation using RL [33, 29] and teaching new skills to humanoid robots [14].

In future works, we aim to explore alternative approaches for defining reliable safe policies in order to enable model-free agents to devise a set of actions for each unique situation, rather than solely relying on predefined behaviors. Additionally, we aim to work on more complex environments and evaluate the effectiveness of our framework in environments with high dimensions and complex dynamics.

In this study, we introduce an exploration framework to enable model-free RL agents to explore new environments and adapt to new tasks more safely compared to the traditional $\epsilon$-greedy strategy. The core component of our framework is the pre-training phase that enables the agent to detect BRS states given high-level features extracted from the agent’s observation by training a binary classification model. Then, this model is used in a new environment to detect whether employing random exploration or a suboptimal policy is safe. Our experimental results demonstrate that by defining an effective safe policy and pre-training the agent in an appropriate pre-training environment, the agent can learn the optimal policy in new environments with significantly fewer violations of safety constraints and higher cumulative discounted reward.