Efficient Off-Policy Safe Reinforcement Learning
Using Trust Region Conditional Value at Risk

Safe RL methods generally set safety constraints using the expectation of the cumulative costs. If expectation-based constraints are used, the Bellman operator for the traditional Markov decision process (MDP) can be applied to the safety critic, giving mathematical simplicity. However, since these constraints focus only on the average case, it is challenging to prevent failures which occurred in the worst case. For this reason, research on risk measure-based constraints is increasing to concentrate on the worst case. Kim and Oh [3] and Yang et al. [7] proposed methods constraining CVaR, which is a risk measure widely used in financial investment [15], and estimated the CVaR using distributional safety critics. Both methods resulted in fewer constraint violations than the expectation-constrained RL methods, and used the trust-region and Lagrangian methods to deal with constraints, respectively. Methods proposed by Ying et al. [16] and Chow et al. [8] also use the CVaR-based constraints but estimate CVaR using the sampling method proposed by Rockafellar et al. [15]. However, the estimation error can increase when the state space is high-dimensional because of the nature of sampling methods. Instead of CVaR, methods proposed by Thananjeyan et al. [17] and Bharadhwaj et al. [2] use chance constraints restricting the likelihood that the cumulative costs become above a specific value. These methods calculate the violation likelihood and correct the action to lower the likelihood at every environmental interaction, which can cause a longer action execution time.

The usage of off-policy data not only improves the sample efficiency, but also reduces the optimality gap to reach higher final performance.¹¹1Since the MuJoCo tasks [13] are dominated by off-policy algorithms as shown in the benchmarks of the Spinning-Up from OpenAI [18], using off-policy data can reduce the optimality gap. Therefore, it is crucial to use off-policy data in safe RL to rapidly satisfy the constraints and maximize the return. Wang et al. [19] and Ha et al. [20] proposed safe RL methods based on the SAC method. These methods use off-policy data to estimate the safety critic and utilize the Lagrangian approach to reflect the expectation-based constraints. There is another method based on the Q-learning approach proposed by Huang et al. [21], which deals with the safe RL problem with multiple objectives. Liu et al. [10] transform the safe RL problem into a variational inference problem and proposes a new safe RL approach based on the expectation-maximization (EM) algorithm. This method uses off-policy data in the E-step to fit a non-parametric distribution, and updates a policy in the M-step by minimizing the distance between the policy and the non-parametric distribution.

In addition to the methods mentioned above, there are various approaches to safe RL. First, there are Lyapunov-based methods to train a safe policy [22, 23]. They learn the Lyapunov function using auxiliary costs and update policies to satisfy the Lyapunov property. In [24, 25], control barrier functions are used to prevent agents from entering unsafe regions, and the control barrier functions can be constructed using trained transition models or prior knowledge of systems. Also, there are projection-based methods that handle constraints through a primal approach [6, 26]. They update policies to maximize the sum of reward and project the update direction to a safe policy set.

We use constrained Markov decision processes (CMDPs) to define a safe reinforcement learning (RL) problem. A CMDP is defined with a state space $\mathcal{S}$, an action space $\mathcal{A}$, a transition model $\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto\mathbb{R}$, an initial state distribution $\rho$, a discount factor $\gamma$, a reward function $R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto\mathbb{R}$, a cost function $C:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto\mathbb{R}_{\geq 0}$, and a safety measure $\mathbf{S}$. Given a policy $\pi$, value and advantage functions are defined as follows:

The cost value $V_{C}^{\pi}$ and advantage $A_{C}^{\pi}$ can be defined by replacing the reward $R$ with the cost $C$ in (1). Additionally, a discounted state distribution is defined as $d^{\pi}(s):=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}P(s_{t}=s|\rho,\pi)$. For safety constraints, we introduce a notion called the cost return, which is used in [3, 7]:

where $s_{0}\sim\rho$, $a_{t}\sim\pi(\cdot|s_{t})$, and $s_{t+1}\sim\mathcal{P}(\cdot|s_{t},a_{t})$. Then, we can define a safe RL problem as follows:

where $d$ is a limit value, and the objective function is denoted by $J(\pi):=\underset{s\sim\rho}{\mathbb{E}}\left[V^{\pi}(s)\right]$.

Kim and Oh [3] proposed a trust region-base safe RL method for conditional value at risk (CVaR) constraints, called TRC. CVaR is one of the commonly used risk measures in financial risk management and formulated as follows [15]:

where $(\cdot)_{+}$ is a clipping function that truncates values below zero. Because CVaR is calculated by conditional expectations above a certain level, it focuses on the worst case rather than the average case. Therefore, constraining CVaR is a compelling way to prevent a worst-case failure. By using CVaR for safety measures, TRC outperforms other expectation-constrained RL methods. To formulate CVaR constraints, the followings are defined in [3].

where $S_{C}^{\pi}$ and $A_{S}^{\pi}$ are called cost square value and advantage, and expectations of the cost value and cost square value are denoted as $J_{C}(\pi):=\underset{s\sim\rho}{\mathbb{E}}\left[V_{C}^{\pi}(s)\right]$ and $J_{S}(\pi):=\underset{s\sim\rho}{\mathbb{E}}\left[S_{C}^{\pi}(s)\right]$, respectively. Also, a doubly discounted state distribution is defined as $d_{2}^{\pi}:=(1-\gamma^{2})\sum_{t=0}^{\infty}\gamma^{2t}P(s_{t}=s|\pi)$. With state distributions, the expectations of the cost and cost square value can be expressed as follows [3]:

Assuming that the cost return follows the Gaussian distribution as in [3, 7], CVaR can be expressed as follows:

where $\alpha$ is a risk level for CVaR, and $\phi$ and $\Phi$ are probability and cumulative density functions of the standard normal distribution, respectively.

Meng et al. [12] improved trust region policy optimization (TRPO) [27] using replay buffers and shown that monotonic improvement is guaranteed. The method proposed in [12], called off-policy TRPO, estimates an objective of a new policy $\pi^{\prime}$ using a surrogate function which can utilize a large number of data from replay buffers. The surrogate function is expressed as follows:

where $\mu$ is a behavioral policy and $\pi$ is a policy before being updated. The objective function of $\pi^{\prime}$ has a lower bound, which is derived using the surrogate function $J^{\mu,\pi}(\pi^{\prime})$ as follows:

where $D_{\mathrm{TV}}^{\mathrm{max}}(\mu||\pi):=\underset{s}{\mathrm{max}}D_{\mathrm{TV}}(\mu(\cdot|s)||\pi(\cdot|s))$, $D_{\mathrm{TV}}$ is the total variation (TV) distance, and $\epsilon_{R}=\underset{s,a}{\mathrm{max}}\left|A^{\pi}(s,a)\right|$. By iteratively maximizing the lower bound of $J(\pi^{\prime})$, off-policy TRPO can guarantee that the objective is monotonically improved [12]. However, the surrogate function (8) is only valid for expectation-based objectives, so there is a need to develop new surrogate functions for objectives and constraints if a different safety measure is considered.

Calculating CVaR of a random policy $\pi^{\prime}$ using (7) requires trajectories sampled by $\pi^{\prime}$, which is computationally expensive. Therefore, it is desirable to estimate CVaR using trajectories sampled from a behavioral policy $\mu$ and the current policy $\pi$ rather than calculate it directly. To estimate the CVaR constraint, two surrogate functions are proposed as follows:

In (11), $J_{C}(\pi)$ and $J_{S}(\pi)$ can be calculated using (6) with trajectories sampled from $\pi$, and the other terms can be calculated using trajectories from $\mu$. Using the proposed surrogate functions and (7), CVaR of a random policy $\pi^{\prime}$ can be approximated as follows:

The next section shows that the difference between the above approximation and the true value of CVaR is bounded.

Before showing the upper bound of CVaR, the following notations are introduced for brevity:

Then, the upper bound of $\mathrm{CVaR}_{\alpha}(C_{\pi^{\prime}})$ can be derived as follows.

The proof is given in Appendix -B. Theorem 1 shows that $\overline{\mathrm{CVaR}}_{\alpha}(C_{\pi^{\prime}})$ approximates $\mathrm{CVaR}_{\alpha}(C_{\pi^{\prime}})$ well enough when $\pi$ and $\pi^{\prime}$ are close to each other since $D(\mu,\pi^{\prime})D(\pi,\pi^{\prime})$ becomes sufficiently small. In addition, as shown in the definition, $\epsilon_{\mathrm{CVaR}}$ is highly correlated to $\epsilon_{C}$ and $\epsilon_{S}$, and they become larger when the state change according to actions is huge. Therefore, it is possible to decrease $\epsilon_{\mathrm{CVaR}}$ by designing an RL environment with a shorter time interval.

Directly calculating the objective and constraints of a random policy through sampling is computationally expensive, so we utilize their bounds to get the policy gradient efficiently. Using the upper bound of CVaR derived in Theorem 1 and the lower bound of the objective in (9), we can build a subproblem to obtain a new policy $\pi^{\prime}$ given an old policy $\pi$ and a behavioral policy $\mu$ as follows:

However, the TV distances $D(\mu,\pi^{\prime})D(\pi,\pi^{\prime})$ in the subproblem hinder training policies with large update steps, which is an issue raised in several trust region-related literature [12, 5, 27]. Thus, we remove the TV distances from (15) and add a trust region constraint as done in [12, 3, 27] to enlarge update steps. For the trust region constraint, we convert the TV distances into KL divergences using the Pinsker’s inequality and formulate the constraint as follows:

where $\delta_{\mathrm{old}}=\sqrt{D_{\mathrm{KL}}(\mu||\pi)(\delta+D_{\mathrm{KL}}(\mu||\pi)/4)}-D_{\mathrm{KL}}(\mu||\pi)/2$, $D_{\mathrm{KL}}(\pi||\pi^{\prime})=\underset{s\sim d_{\mu}}{\mathbb{E}}\left[D_{\mathrm{KL}}(\pi(\cdot|s)||\pi^{\prime}(\cdot|s))\right]$, and $\delta$ is a constant for the trust region size. The derivation is presented in Appendix -C. When the KL divergence between $\pi$ and $\mu$ is large enough, $\delta-\delta_{\mathrm{old}}$ becomes nearly zero, so the trust region constraint ensures that the policy is updated not far from the replay buffer. Hence, the subproblem (15) can be reformulated as follows:

The subproblem (17) is nonconvex, so we approximate the objective and the CVaR constraint as linear and the trust region as quadratic, and obtain $\pi^{\prime}$ using linear and quadratic constrained linear programming (LQCLP) as in [3, 5]. If the feasibility set of (17) is empty, we solely minimize the approximated CVaR under the trust region constraint.

To update the value and cost value networks, we use the following retrace estimators as target values [12, 28].

where $r_{t}=R(s_{t},a_{t},s_{t+1})$, $c_{t}=C(s_{t},a_{t},s_{t+1})$, $\overline{\rho}_{t}=\mathrm{min}(1,\frac{\pi(a_{t}|s_{t})}{\mu(a_{t}|s_{t})})$, and $\lambda$ is a trace-decay value. The retrace estimator for the cost square value can also be defined as follows.

Then, the value, cost value, and cost square networks are updated by minimizing the mean squared error between the target values. The overall process of off-policy TRC is summarized in Algorithm 1.

To verify that off-policy TRC is applicable to real robots, we conduct a sim-to-real experiment on a UGV robot, Jackal from Clearpath [29], shown in Fig. 1a. Agents are first trained in a MuJoCo simulator, where the state space is a 32-dimensional space consisting of LIDAR values, linear and angular velocities of the robot, and the goal position, and the action space is a two-dimensional space consisting of the linear acceleration and angular velocity. Eight obstacles are spawned at random positions in the beginning of each episode, and the reward and cost function are the same as the Safety Gym. Once trained in simulation, agents are evaluated in the real world without additional training on a task shown in Fig. 1b, where goals are spawned sequentially. If the agent gets closer than $0.2\;m$ to obstacles, a failure occurs and the episode ends.

In all tasks, the number of constraint violations (CVs) is counted when $C(s_{t},a_{t},s_{t+1})\geq 0.5$, and the following score metric is used as in [3].²²2The score metric is defined by reward sum divided by the number of CVs. Since the reward sum increases as safety is ignored, this metric is used for fair comparisons.

The policy and all value functions are modeled by neural networks with two hidden layers of 512 nodes, and the activation functions are ReLU. The learning rate for all value networks is $0.0002$ and $\delta$ for the trust region is $0.001$. We update networks at every collect steps $S$ ($=1000$) using trajectories of length $B$ ($=5000$) sampled from the replay buffer of length $L$ ($=50000$). Exceptionally, we set $B=20000$ and $L=100000$ for the MuJoCo and doggo goal tasks. For constraints, we use $\alpha$ for CVaR as $0.125$ or $1.0$ and $d$ as $0.025$.³³3 To give a tip for how to set $\alpha$ and $d$, if you want the agent to violate safety less than $25$ times out of $1000$ interaction steps with $95\%$ probability, set $d$ as $25.0/1000.0$ and $\alpha$ as $f^{-1}(\Phi^{-1}(0.95))$, where $f(x)=\phi(\Phi^{-1}(x))/x$.

Throughout the experiments, we evaluate the proposed method, off-policy TRC, with $\alpha=0.125$, indicated as OffTRC, and with $\alpha=1.0$, indicated as OffCPO. If $\alpha=1.0$, off-policy TRC becomes the off-policy version of constrained policy optimization (CPO) [5], as the CVaR becomes an expectation if $\alpha=1.0$. In the MuJoCo experiment, to show the sample efficiency, the off-policy TRC is compared to other trust region-based methods: CPO [5] and TRC [3]. In addition, to check how the constraints defined in (20) and (21) have an effect on the locomotion tasks, the traditional RL method, off-policy trust region policy optimization (OffTRPO) [12] is also used. For the Safety Gym and the sim-to-real experiments, we use the following risk measure-constrained RL methods as baselines: CVaR-proximal policy optimization (CPPO) [16], WCSAC [7], and TRC [3]. CPPO estimates CVaR from sampling, while WCSAC and TRC estimate CVaR using distributional safety critics. To compare with expectation-based safe RL methods, CPO is also used as a baseline.

The MuJoCo and Safety Gym simulation results are shown in Fig. 2, and the Jackal simulation and real-world evaluation results are shown in Fig. 3. WCSAC and CPPO are excluded from the real-world evaluation because the agents trained by them do not move but freeze at initial positions in the Jackal simulation. In all tasks, including the real-world experiment, the proposed methods, which are OffTRC and OffCPO, show the highest scores and the lowest total number of CVs. OffTRC and OffCPO show the similar level in scores, but OffTRC shows lower total numbers of CVs than OffCPO in half-cheetah, point goal, and car goal tasks. In the walker and doggo tasks, it is difficult to satisfy the constraints due to the unstable dynamics, so the number of CVs of OffTRC and OffCPO are similar. However, OffCPO scores higher than OffTRC in the half-cheetah task, which means that the CVaR constraints make it difficult to learn tasks with stable dynamics. Observing that the scores increase the fastest and the total number of CVs are significantly low, the proposed methods show excellent sample efficiency by simultaneously increasing the return while satisfying the safety constraints. OffTRPO results in the third highest score in the half-cheetah task, showing that the number of CVs converges to zero. As the dynamics model of the half-cheetah is stable, OffTRPO can train policies well without any constraints. Nevertheless, OffTRPO shows the worst performance in the walker-2d task, inferring that restricting the height of CoM helps training. CPO shows the third-highest score in the walker-2d, but there is a significant score gap between off-policy TRC and CPO, which indicates that sample efficiency is essential to achieve outstanding performance. Additionally, CPO shows excessive numbers of CVs in all other tasks and records a failure in the real-world evaluation, as shown in Fig. 3b. It means that risk measure-constrained RL methods rather than expectation-constrained RL methods are required to prevent failures. TRC shows the third-best performance for all tasks except the MuJoCo tasks. Because TRC is an on-policy RL algorithm, it is difficult to train complex robots such as multi-joint robots, since these robotic tasks require large quantities of training data. Still, as TRC is a trust-region-based safe RL method, it shows monotonic performance improvement and low CVs. WCSAC uses the Lagrangian method to handle the CVaR constraint and is one of the off-policy RL methods, so it can be expected to show high sample efficiency. However, WCSAC shows low scores in all tasks and synthesizes immobile policies in the doggo and Jackal tasks because the Lagrangian method makes training unstable. CPPO is also one of the CVaR-constrained RL method and estimate CVaR using a sampling method. However, CPPO suffers performance degradation because it uses a sampling-based estimation for CVaR with large variances and the Lagrangian approach to handle the constraints.

To show that the proposed surrogate functions (11) are effective for off-policy data, we perform an experiment comparing a variant of TRC against off-policy TRC. The TRC variant treats off-policy data as on-policy data and updates the policy via the TRC method. The result is presented in Fig. 4. In the training curve, the CVs of the variant are high relative to the off-policy TRC, which can be considered estimation errors due to the distributional shift in the training data. Hence, we can conclude that the CVaR estimation through the proposed surrogate functions (11) is critical for utilizing off-policy data.

Since the off-policy TRC uses off-policy data from a replay buffer, it is required to analyze how the parameters related to the replay buffer affect learning. The replay buffer-related parameters are the batch size (the length of sampled trajectories for policy update), the length of the replay buffer, and the collect steps (the policy is updated for each collect step). We train the off-policy TRC on the point goal task for each parameter with three different values, and the results are shown in Table I. The parameter with the greatest influence on training is the batch size. The larger the batch size, the lower the total number of CVs. As the batch size increases, the search direction for policy update becomes accurate, so the agent can effectively lower the constraint value at the early training phase. Next, the collect steps affect the length of on-policy data and the number of policy updates. Since on-policy data is used to calculate $J_{C}(\pi)$ and $J_{S}(\pi)$, constituting the approximated CVaR $\overline{\mathrm{CVaR}}_{\alpha}(C_{\pi^{\prime}})$, the approximation error increases as the amount of on-policy data decreases. Thus, the larger the collect steps, the lower the total number of CVs, but also the lower the reward sum, as shown in Table I. Lastly, while the length of the replay buffer has a little effect on performance, if it is too short, the total number of CVs can increase slightly.

To give an intuition for the risk level $\alpha$, experiments with different risk levels for TRC and off-policy TRC are also conducted, and the result is shown in Fig. 5. Observing that policies with high-risk levels are located in the upper right corner of the figure, a high return but risky policy can be obtained with a high-risk level.

By Pinsker’s inequality, the following inequality holds:

From Appendix B in [12], the following inequality also holds:

Then, the trust region constraint can be expressed as:

As the maximum operation is difficult to implement in continuous state space settings, we replace it with expectation on trajectories sampled by $\mu$ as follows:

The square root operation on $\pi^{\prime}$ can lead to infinite value and increase nonlinearity while computing gradients. Thus, by rearranging (28), we remove the square root on $\pi^{\prime}$.