Evaluation of Safety Constraints in Autonomous Navigation with Deep Reinforcement Learning ^††thanks: This is a preprint version of the extended abstract accepted to TAROS’23.

Deep Reinforcement Learning has demonstrated tremendous success in many high-dimensional control problems, including autonomous navigation. Within RL, the interaction of the agent with the environment is modeled as a Markov decision process (MDP) [1], where the goal is to optimize the expected cumulative reward. The agent in MDP has a big freedom to explore any behavior which could improve its performance, including those that might cause damage. To this end, it is crucial to ensure safety constraints. A well-known approach to consider safety constraints in RL is a Constrained Markov Decision Process (CMDP) [2]. A survey of methods for solving CMDP can be found in [3]. In this short paper, we will apply safety constraints to ensure the safe behavior of the autonomous vehicle. Particularly, we will use the Lagrangian method [4] that is one of the most widely used approaches for solving CMDP.

We are interested in algorithms for autonomous navigation which provide certain safety guarantees. To this end, we model our problem as a CMDP, where the agent (autonomous vehicle) must generate a sequence of actions (trajectory) that drives it to a goal while avoiding obstacles and ensuring a tolerable safety cost limit. The latter in our work is interpreted as an upper limit of the vehicle’s velocity when it is moving near to the obstacles.

Trajectory generation We are interested in autonomous vehicles whose dynamics is described as [5]: $\dot{x}=vcos(\theta),\dot{y}=vsin(\theta),\dot{\theta}=\frac{v}{L}\tan(\gamma)$, where $x$,$y$ are the coordinates of the vehicle’s rear axle, $\theta$ is the orientation, $L$ is the wheel-base, $v$ is the linear velocity, $\gamma$ is the steering angle. The former three variables comprise the state vector: $\boldsymbol{x}(t)=(x,y,\theta)$. The latter two variables form the control vector: $\boldsymbol{u}(t)=(v,\gamma)$, which can also be re-written using the acceleration $a$ and the rotation rate $\omega$ as follows: $v=v_{0}+a\cdot t,\gamma=\gamma_{0}+\omega\cdot t$.

The robot is operating in the 2D workspace populated with static obstacles. Their shapes are rectangular (as the one of the robot). Let $Obs$ denote the set of obstacles. Denote by $\mathcal{X}_{free}$ all the configurations of the robot which are not in collision with the obstacles. The problem is to find the actions that move the vehicle from its start configuration $s_{start}$ to the goal one $s_{goal}$, s.t. that the kinodynamic constraints are met and the resultant trajectory lies in $\mathcal{X}_{free}$. These controls are generated using a sequential decision making based on CMDP.

Constrained Markov Decision Process Formally, CMDP can be represented as a tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\mathcal{C},d,\gamma)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{P}$ is the state-transition model, $\mathcal{R}$ is the reward function, $\mathcal{C}$ is a constraint cost function and $\gamma$ is the discounting factor. During learning at each time step the agent being in a state $s_{t}\in\mathcal{S}$ takes an action $a_{t}\in\mathcal{A}$ and receives a reward $r_{t}\in\mathcal{R}$ and a cost $c_{t}\in\mathcal{C}$. The goal is to learn a policy, i.e. the mapping from the states to the distributions of actions, $\pi:\mathcal{S}\rightarrow P(\mathcal{A})$. The policy should maximize the expected return $J(\pi)$ from the start state $s_{t}$ while satisfying the discounted cost with tolerable limit $d_{i}$ trough the discounted cost $C(\pi)$ under policy $\pi$:

where $\tau=(s_{0},a_{0},s_{1},a_{1},\ldots)$ denotes a trajectory. The objective of CMDP for the policy $\pi$ is to find: $\pi^{*}=\arg\max_{\pi\in\Pi}J(\pi),\;\;s.t.\;C_{i}(\pi)\leq d_{i}$.

To evaluate the influence of safety constraints, we will use a policy-gradient algorithm PPO [6] and its safe version – LagrangianPPO (LPPO) [4]. The learning and evaluation of these two algorithms are conducted in a gym environment from [7]. Next, we will briefly introduce some details of environment.

Environment Our environment provides an autonomous vehicle equipped with kinematic bicycle model and lidar-sensor in an environment which is populated with static obstacles as shown in Fig. 1. The vehicle state is described as a tuple $\boldsymbol{x}=(x,y,\theta,v,\gamma)$. We consider agent’s state $s_{t}=(\Delta x,\Delta y,\Delta\theta,\Delta v,\Delta\gamma,\theta,v,\gamma,\boldsymbol{l})$ where $\Delta x_{j}$ is the difference between the j-element of the tuple between goal and current vehicle state, $\theta,v,\gamma$ are elements of current vehicle state and $\boldsymbol{l}$ is a tuple of rays-measurements from lidar. In this paper, we consider the same actions and reward function from [7]. The actions $a_{t}=(a,\omega)$ are composed of the linear acceleration $a$ and rotation rate $\omega$. For details of the actions, kinematic model and reward function we refer the readers to  [7].

Safety Constraints The safety constraint cost is induced by a velocity of the autonomous vehicle at which it moves near to the obstacles due to the risk of collision like the method proposed in [8]. We impose an immediate constraint cost as $c_{i}(s_{i})=\|v_{i}\|\times 1\{\|\boldsymbol{l}\|\leq r_{safety}\}$, where $r_{safety}=0.5m$ is the safety radius – see Fig. 1, $\boldsymbol{l}$ is the current lidar signal, and $1$ is boolean function. This safety constraint allows to the agent to move near obstacles at tolerable velocity $v_{i}$ so as not to violate the tolerable limit $d_{i}$.

We trained two policies, i.e. PPO [6] and LPPO [4] on a dataset from [7]. The dataset is comprised of tasks which consist of start, goal and a set of static obstacles. The goal is to generate a trajectory from start to goal. In the Fig. 2 we show the learning process of these algorithms. We can see that PPO starts to converge earlier in comparison with LPPO. But at the same time it violates more safety constraints. On the other hand, the LPPO agent tries to find a trade-off.

After training we evaluate these two policies on a validation dataset through the following metrics: success rate (SR, %), collision rate (CR, %) and mean minimum clearance distance (MMC, m) of the successfully generated trajectories. The results for PPO are $SR=96.25\%$, $CR=3.75\%$, $MMC=0.61m$, and for LPPO are $SR=94.5\%$, $CR=3.5\%$, $MMC=0.67m$. We see that the LPPO agent tends to avoid the obstacles with more clearance in order to guarantee the safety constraints as shown in the Fig. 1. On the other hand, the PPO agent acquired more aggressive behavior that tries to reach the goal as soon as possible regardless of the risk of collision. Additionally, we can see that LPPO agent finishes some tasks without success or collision. In this cases, the LPPO agent prefers to make an abrupt stop instead of colliding with an obstacle. The latter is a safe behavior that agent acquired thanks to the safety constraints.

In this short paper, we have evaluated the consideration of safety constraints during the optimization process for policy gradient algorithm PPO. While PPO can converge earlier and has a slightly high success rate, the learned behavior of LPPO is more conservative due to taking the safety constraints into account. Overall, considering safety constraints in the optimization problem provides a more conservative and safe behavior without sacrificing of performance.