Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation

The main contribution is to propose a DRL algorithm to obtain an optimal policy for a given control performance index such as fuel costs under a constraint described by an STL formula. Our proposed algorithm has the following three advantages.

1. 1.

   We directly solve control problems with continuous state-action spaces. We apply DRL algorithms for problems with continuous spaces such as deep deterministic policy gradient (DDPG) [20] and soft actor critic (SAC) [21].

2. 2.

   We obtain a policy that not only satisfies a given STL formula but also is optimal with respect to a given control performance index. We consider the optimal control problem constrained by a given STL formula and formulate the problem as a constrained Markov decision process (CMDP) [22]. In the CMDP problem, we introduce two reward functions: one is the reward function for the given control performance index and the other is the reward function for the given STL constraint. To solve the CMDP problem, we apply a constrained DRL (CDRL) algorithm with the Lagrangian relaxation [23]. In this algorithm, we relax the CMDP problem into an unconstrained problem using a Lagrange multiplier to utilize standard DRL algorithms for problems with continuous spaces.

3. 3.

   We introduce a two-phase learning algorithm in order to make it easy to learn a policy satisfying the given STL formula. In a CMDP problem, it is important to satisfy the given constraint. The agent needs many experiences satisfying the given STL formula in order to learn how to satisfy the formula. However, it is difficult to collect the experiences considering both the control performance index and the STL constraint in the early learning stage since the agent may prioritize to optimize its policy with respect to the control performance index. Thus, in the first phase, the agent learns its policy without the control performance index in order to obtain experiences satisfying the STL constraint easily, which is called pre-training. After obtaining many experiences satisfying the STL formula, in the second phase, the agent learns its optimal policy for the control performance index under the STL constraint, which is called fine-tuning.

Through simulations, we demonstrate the learning performance of the proposed algorithm.

The remainder of this paper is organized as follows. In Section II, we review STL and the Q-learning algorithm to learn a policy satisfying STL formulae briefly. In Section III, we formulate an optimal control problem under a constraint described by an STL formula as a CMDP problem. In Section IV, we propose a CDRL algorithm with the Lagrangian relaxation to solve the CMDP problem. We relax the CMDP problem to an unconstrained problem using a Lagrange multiplier to utilize the DRL algorithm for unconstrained problems with continuous spaces. In Section V, by numerical simulations, we demonstrate the usefulness of the proposed method. In Section VI, we conclude the paper and discuss future works.

$\mathbb{N}_{\geq 0}$ is the set of the nonnegative integers. $\mathbb{R}$ is the set of the real numbers. $\mathbb{R}_{\geq 0}$ is the set of nonnegative real numbers. $\mathbb{R}^{n}$ is the $n$-dimensional Euclidean space. For a set $A\subset\mathbb{R}$, $\max A$ and $\min A$ are the maximum and minimum value in $A$ if they exist, respectively.

We consider the following discrete-time stochastic dynamical system.

where $x_{k}\in\mathcal{X}$, $a_{k}\in\mathcal{A}$, and $w_{k}\in\mathcal{W}$ are the system’s state, the agent’s control action, and the system noise at $k\in\{0,1,...\}$. $\mathcal{X}=\mathbb{R}^{n_{x}}$, $\mathcal{A}\subseteq\mathbb{R}^{n_{a}}$, and $\mathcal{W}=\mathbb{R}^{n_{x}}$ are the system’s state space, the control action space, and the system noise space, respectively. The system noise $w_{k}$ is an independent and identically distributed random variable with a probability density $p_{w}:\mathcal{W}\to\mathbb{R}_{\geq 0}$. $\Delta_{w}$ is a regular matrix that is a weighting factor of the system noise. $f:\mathcal{X}\times\mathcal{A}\to\mathcal{X}$ is a function that describes the system dynamics. Then, we have the transition probability density $p_{f}(x^{\prime}|x,a):=|\Delta_{w}^{-1}|p_{w}(\Delta_{w}^{-1}(x^{\prime}-f(x,a)))$. The initial state $x_{0}\in\mathcal{X}$ is sampled from a probability density $p_{0}:\mathcal{X}\to\mathbb{R}_{\geq 0}$. For a finite system trajectory whose length is $K+1$, $x_{k_{1}:k_{2}}$ denotes the partial trajectory for the time interval $[k_{1},k_{2}]$, where $0\leq k_{1}\leq k_{2}\leq K$.

STL is a specification formalism that allows us to express real-time properties of real-valued trajectories of systems [12]. We consider the following syntax of STL.

where $K_{e}$, $k_{s}$, and $k_{e}\in\mathbb{N}_{\geq 0}$ are nonnegative constants for the time bounds, $\Phi,\phi,\varphi,$ and $\psi$ are the STL formulae, $\psi$ is a predicate in the form of $h(x)\leq d$, $h:\mathcal{X}\to\mathbb{R}$ is a function of the system’s state, and $d\in\mathbb{R}$ is a constant. The Boolean operators $\lnot$, $\land$, and $\lor$ are negation, conjunction, and disjunction, respectively. The temporal operators $G_{\mathcal{T}}$ and $F_{\mathcal{T}}$ refer to Globally (always) and Finally (eventually), respectively, where $\mathcal{T}$ denotes the time bound of the temporal operator. $\phi_{i}=G_{[k_{s}^{i},k_{e}^{i}]}\varphi_{i}$, or $\phi_{i}=F_{[k_{s}^{i},k_{e}^{i}]}\varphi_{i}$, $i\in\{1,2,...,M\}$ are called STL sub-formulae. $\phi$ comprises multiple STL sub-formulae $\{\phi_{i}\}_{i=1}^{M}$.

The Boolean semantics of STL is recursively defined as follows:

The quantitative semantics of STL, which is called robustness, is recursively defined as follows:

which quantifies how well the trajectory satisfies the given STL formulae [13].

The horizon length of an STL formula is recursively defined as follows:

$\text{hrz}(\phi)$ is the required length of the state sequence to verify the satisfaction of the STL formula $\phi$.

In this section, we review the Q-learning algorithm to learn a policy satisfying a given STL formula [16]. Although we often regard the current state of the dynamical system (1) as the environment’s state for RL, the current system’s state is not enough to determine an action for satisfying a given STL formula. Thus, Aksaray et al. defined the following extended state using previous system’s states.

where $\tau=\text{hrz}(\phi)+1$ for the given STL formula $\Phi=G_{[0,K_{e}]}\phi$ (or $\Phi=F_{[0,K_{e}]}\phi$) and $\mathcal{Z}$ is an extended state space.We show a simple example in Fig. 1. We operate a one-dimensional dynamical system to satisfy the STL formula

At any time in the time interval $[0,10]$, the system should enter both the blue region and the green region before 3 time steps are elapsed, where there is no constraint for the order of the visits. Let the current system’s state be $x_{k}=1.5$. Note that the desired action for the STL formula is different depending on the past state sequence. For example, in the case where $x_{k-3:k}=-0.5,0.5,1.0,1.5$, we should operate the system to the blue region right away. On the other hand, in the case where $x_{k-3:k}=-1.5,-2.5,-0.5,1.5$, we do not need to move it. Thus, we regard not only the current system’s state but also previous system’s states as an environment’s state for RL.

Additionally, Aksaray et al. designed the reward function $R_{STL}:\mathcal{Z}\to\mathbb{R}$ using robustness and the log-sum-exp approximation. The robustness of a trajectory $x_{0:K}$ with respect to the given STL formula $\Phi$ is as follows:

We consider the following problem.

where $x_{0:K}^{\pi}$ is the system’s trajectory controlled by the policy $\pi$ and the function $\bm{1}:\mathbb{R}\to\{0,1\}$ is an indicator defined by

Since $\bm{1}(\min\{y_{1},\ ...,\ y_{n}\})=\min\{\bm{1}(y_{1}),\ ...,\ \bm{1}(y_{n})\}$ and $\bm{1}(\max\{y_{1},\ ...,\ y_{n}\})=\max\{\bm{1}(y_{1}),\ ...,\ \bm{1}(y_{n})\}$,

Then, we use the following log-sum-exp approximation.

where $\beta>0$ is an approximation parameter. We can approximate $\min\{\cdots\}$ or $\max\{\cdots\}$ with arbitrary accuracy by selecting a large $\beta$. Then, (5) can be approximated as follows:

Since the $\log$ function is a strictly monotonic function and $\beta>0$ is a constant, we have

Thus, we use the following reward function $R_{STL}:\mathcal{Z}\to\mathbb{R}$ to satisfy the given STL formula $\Phi$.

To design a controller satisfying an STL formula using the Q-learning algorithm, Aksaray et al. proposed a $\tau$-MDP as follows:

We parameterize a deterministic policy using a DNN as shown in Fig. 3, which is an actor DNN. Its parameter vector is denoted by $\theta_{\mu}$. In the DDPG-Lagrangian algorithm, the parameter vector $\theta_{\mu}$ is updated by maximizing (13). However, $J(\mu_{\theta_{\mu}})$ and $J_{STL}(\mu_{\theta_{\mu}})$ are unknown. Thus, as shown in Fig. 4, $J(\mu_{\theta_{\mu}})$ and $J_{STL}(\mu_{\theta_{\mu}})$ are approximated by two separate critic DNNs, which are called a reward critic DNN and an STL-reward critic DNN, respectively. The parameter vectors of the reward critic DNN and the STL-reward critic DNN are denoted by $\theta_{r}$ and $\theta_{s}$, respectively. $\theta_{r}$ and $\theta_{s}$ are updated by decreasing the following critic loss functions.

where $Q_{\theta_{r}}(\cdot,\cdot)$ and $Q_{\theta_{s}}(\cdot,\cdot)$ are the outputs of the reward critic DNN and the STL-reward critic DNN, respectively. The target values $t_{r}$ and $t_{s}$ are given by

$Q_{\theta_{r}^{-}}(\cdot,\cdot)$ and $Q_{\theta_{s}^{-}}(\cdot,\cdot)$ are the outputs of the target reward critic DNN and the target STL-reward critic DNN, respectively, and $\mu_{\theta_{\mu}^{-}}(\cdot)$ is the output of target actor DNN. $\theta_{r}^{-}$, $\theta_{s}^{-}$, and $\theta_{\mu}^{-}$ are parameter vectors of the target reward critic DNN, the target STL-reward critic DNN, and the target actor DNN, respectively. Their parameter vectors are slowly updated by the following soft update.

where $\xi>0$ is a sufficiently small positive constant. The agent stores experiences to the replay buffer $\mathcal{D}$ and selects some experiences from $\mathcal{D}$ randomly for updates of $\theta_{r}$ and $\theta_{s}$. $E_{(z,a,z^{\prime})\sim\mathcal{D}}[\cdot]$ is the expected value under the random sampling of the experiences from $\mathcal{D}$. In the standard DDPG algorithm [20], the parameter vector of the actor DNN is updated by decreasing

where $E_{z\sim\mathcal{D}}[\cdot]$ is the expected value with respect to $z$ sampled from $\mathcal{D}$ randomly. However, in the DDPG-Lagrangian algorithm, we consider (13) as an objective instead of $J(\mu_{\theta_{\mu}})$. Thus, the parameter vector of the actor DNN $\theta_{\mu}$ is updated by decreasing the following actor loss function.

The Lagrange multiplier $\kappa$ is updated by decreasing the following loss function.

where $E_{z_{0}\sim p_{0}^{z}}[\cdot]$ is the expected value with respect to $p_{0}^{z}$.

SAC is a maximum entropy DRL algorithm that obtains a policy to maximize both the expected sum of rewards and the expected entropy of the policy. It is known that a maximum entropy algorithm improves explorations by acquiring diverse behaviors and has the robustness for the estimation error [21]. In the SAC algorithm, we design a stochastic policy $\pi$. We use the following objective with an entropy term instead of $J(\pi)$.

where $\mathcal{H}(\pi(\cdot|z_{k}))=E_{a\sim\pi}[-\log\pi(a|z_{k})]$ is an entropy of the stochastic policy $\pi$ and $\alpha\geq 0$ is an entropy temperature. The entropy temperature determines the relative importance of the entropy term against the sum of rewards.

We use the Lagrangian relaxation for the SAC algorithm such as [28, 29]. Then, a Lagrangian function with the entropy term is given by

We model the stochastic policy $\pi_{\theta_{\pi}}$ using a Gaussian with the mean and the standard deviation outputted by a DNN with a reparameterization trick [30] as shown in Fig. 5, which is an actor DNN. The parameter vector is denoted by $\theta_{\pi}$. Additionally, we need to estimate $J_{ent}(\pi_{\theta_{\pi}})$ and $J_{STL}(\pi_{\theta_{\pi}})$ to update the parameter vector $\theta_{\pi}$ like the DDPG-Lagrangian algorithm. Thus, $J_{ent}(\pi_{\theta_{\pi}})$ and $J_{STL}(\pi_{\theta_{\pi}})$ are also approximated by two separate critic DNNs as shown in Fig. 4. Note that, in the SAC-Lagrangian algorithm, the reward critic DNN estimates not only $J(\pi_{\theta_{\pi}})$ but also the entropy term. The parameter vectors are also updated using the experience replay and the target network technique. $\theta_{r}$ and $\theta_{s}$ are updated by decreasing the following critic loss functions.

where $r=R_{z}(z,a)$, $s=R_{STL}(z)$, and $Q_{\theta_{r}}(\cdot,\cdot)$ and $Q_{\theta_{s}}(\cdot,\cdot)$ are the outputs of the reward critic DNN and the STL-reward critic DNN, respectively. The target values are computed by