Collaborative rover-copter path planning and exploration with temporal logic specifications based on Bayesian update under uncertain environments

Based on the above, the approach presented in this paper is related to the previous works of literature in terms of the following aspects:

1. (1)

   Motion planning/exploration employing the rover-copter collaboration;

2. (2)

   Temporal logic planning under environmental uncertainties;

In what follows, we discuss how our approach differs from the previous works and highlight our main contributions.

Several motion planning/exploration techniques employing the rover-copter collaboration have been provided, see, e.g., (nilsson2018, ; bharadwaj2018, ; sasaki2020, ). In particular, our approach is closely related to (nilsson2018, ), in the sense that the overall synthesis problem is decomposed into two sub-problems, i.e., the problem of synthesizing a copter’s exploration policy in the uncertain environment, and the problem of synthesizing the optimal policy for the rover so as to satisfy the scLTL formula. Our approach builds upon this previous work in terms of both synthesis for the rover’s optimal policy and the copter’s exploration policy in the following ways. Rather than capturing the environmental uncertainties by the belief MDPs (see Section II.A in (nilsson2018, )), in which the environmental belief states are given in a discrete space, this paper captures the environmental uncertainties by assigning beliefs in a continuous space (i.e., the beliefs can take continuous values in the interval $[0,1]$). This allows us to apply the Bayes rule to update the beliefs according to the sensor measurements provided by both the rover and the copter. Moreover, while the previous work solves a value iteration over the state-space in a product MDP that involves the set of environmental belief states, our approach defines a product MDP that does not involve such states, i.e., the set of states of product MDP combines only the set of states of the MDP motion model of the rover and the set of states of the FSA that accepts all trajectories satisfying the scLTL formula (not including the environmental belief states). Hence, we can alleviate the time complexity of synthesizing the optimal policies for the rover in comparison with the previous work. In addition, we provide a theoretical, convergence analysis of the proposed algorithm, in which the environmental beliefs of the atomic propositions are shown to converge to the appropriate values.

Besides the above, many temporal logic planning schemes under environmental uncertainties have been proposed. Most of the previous works assume that the environment has unknown properties (atomic propositions) (ayala2013, ; fu2016, ; maly2013, ; meng2013a, ; meng2015a, ; meng2018, ; livingston2012, ; wongpiromsarn2012, ), or that the motion model of the robot includes uncertainties (lahijanian2010, ; yoo2016, ; sadigh16, ; vasile2016, ; leahy2019, ; wolff2012, ; ulusoy2014, ; wongpiromsarn2012, ; kazumune2020, ), or that the motion model of the robot is completely unknown (sadigh2014, ; jing2015, ; li2018, ; hasanbeig2019, ). Since this paper assumes that it is unknown which atomic propositions are satisfied in the environment, our approach is particularly related to the first category, i.e., (ayala2013, ; fu2016, ; maly2013, ; meng2013a, ; meng2015a, ; meng2018, ; livingston2012, ; wongpiromsarn2012, ). For example, (ayala2013, ) proposed to combine an automata-based model checking and run-time verification for synthesizing a temporal logic motion planning under an incomplete knowledge about the workspace. (fu2016, ) proposed a temporal logic synthesis under probabilistic semantic maps obtained by simultaneous localization and mapping (SLAM). Moreover, (meng2013a, ) proposed a planning revision scheme under incomplete knowledge about the workspace. Our approach is essentially different from the above previous works, in the sense that we incorporate a sensor failure about observations on the atomic propositions. In particular, as previously mentioned, we employ a Bernoulli-type sensor model to describe erroneous observations, and update the beliefs based on the Bayes rule. Besides, the synthesis approach (e.g., construction of the product MDP) is also different from the above previous works, since we make use of the beliefs to synthesize control policies, see Section 4.3. Moreover, our approach is different from the above previous works, in the sense that we incorporate an explicit algorithm for exploration, so as to reduce environmental uncertainties. Other than the above previous works, a few approaches that take into account sensor failures/noise have been provided, see, e.g., (johnson2013, ; johnson2015, ; nuzzo, ; TIGER2020325, ). For example, in (johnson2015, ), the authors proposed a probabilistic model checking for a reactive synthesis under sensor failures and actuator failures. Moreover, (nuzzo, ) introduced the concept of stochastic signal temporal logic (StSTL), and provided both verification and synthesis techniques using assume/guarantee contracts. In contrast to the above previous works, we here propose a Bayesian approach, in which the beliefs that are assigned in the environment are introduced, and these are updated based on observations provided by the Bernoulli sensors.

In summary, the main novelties of this paper with respect to the related works are as follows: using the copter-rover collaboration, we develop a new approach to synthesizing an optimal policy for the rover so as to satisfy an scLTL formula, and an exploration policy for the copter so as to update the environmental beliefs of the atomic propositions and reduce the environmental uncertainties. In particular:

1. (1)

   Using the Bernoulli sensor model and the Bayesian update, we propose a novel exploration algorithm for the copter so as to update the environmental beliefs and effectively reduce the environmental uncertainties (for details, see Section 4.2);

2. (2)

   We propose a novel framework to synthesize the optimal policy for the rover. In particular, we solve a value iteration over a product MDP, whose state-space does not involve the set of the states of the environmental beliefs. This leads to the reduction of time complexity of the value iteration in comparisons with the previous work (for details, see Section 4.3);

3. (3)

   We provide a theoretical, convergence analysis of the proposed algorithm, where it is shown that the environmental beliefs of the atomic propositions converge to the appropriate values (for details, see Section 5).

The remainder of this paper is organized as follows. In Section 2, we provide some preliminaries of the Markov decision process and syntacticallly co-safe LTL formula. In Section 3, we formulate a problem that we seek to solve in this paper. In Section 4, we describe the main algorithm that aims to synthesize both an exploration policy to update the environmental beliefs of the atomic propositions and the optimal policy to satisfy an scLTL formula. In Section 5, we analyze the convergence property of the proposed algorithm. In Section 6, we illustrate the effectiveness of the proposed approach through a simulation example. We finally conclude in Section 7.

A Markov Decision Process (MDP) is defined as a tuple $\mathcal{M}=(X,x_{0},U,p)$, where $X$ is the finite set of states, $x_{0}\in X$ is the initial state, $U$ is the finite set of control inputs, and $p:X\times U\rightarrow\mathcal{D}(X)$ is the transition probability function that associates, for each state $x\in X$ and input $u\in U$, the corresponding probability distribution over $X$. For simplicity of presentation, we abbreviate $p(x,u)(x^{\prime})$ as $p(x^{\prime}|x,u)$. Given $\mathcal{M}=(X,x_{0},U,p)$, a policy sequence ${\mu}_{seq}=\mu_{1}\mu_{2}\ldots$ is defined as an infinite sequence of the mappings $\mu_{k}:X\rightarrow U$, $k\in\mathbb{N}$. Namely, each $\mu_{k}$, $k\in\mathbb{N}_{\geq 0}$ represents a policy as a mapping from each state in $X$ onto the corresponding control input in $U$. The policy sequence ${\mu}_{seq}$ is called stationary if the policy is the invariant for all times, i.e., $\mu_{k}=\mu_{k+1}$, $\forall k\in\mathbb{N}_{\geq 0}$. Given a policy sequence ${\mu}_{seq}=\mu_{1}\mu_{2}\ldots$, a trajectory induced by ${\mu}_{seq}$ is denoted by ${\bf x}_{{\mu}_{seq}}=x(0)x(1)\ldots\in X^{\omega}$, where $x(0)=x_{0}$ and $x(k+1)\sim p(\cdot|x(k),\mu_{k}(x(k)))$, $\forall k\in\mathbb{N}_{\geq 0}$.

Syntactically co-safe LTL (scLTL for short) is defined by using the set of atomic propositions $AP$, Boolean operators, and some temporal operators. Atomic propositions are the Boolean variables taking either true or false. Specifically, the syntax of the scLTL formulas are constructed according to the following grammar:

where $ap\in AP$ is the atomic proposition, $\neg$ (negation), $\wedge$ (conjunction), $\vee$ (disjunction) are the Boolean connectives, and $\bigcirc$ (next), $\mathit{U}$ (until) are the temporal operators. The semantics of LTL formula is inductively defined over an infinite sequence of sets of atomic propositions ${\bf w}=w_{0}w_{1}\cdots\in(2^{AP})^{\omega}$. Intuitively, an atomic proposition $ap\in AP$ is satisfied iff $ap$ is true at $w_{0}$ (i.e., $ap\in w_{0}$). Moreover, $\neg ap$ is satisfied iff $ap$ is not true at $w_{0}$ (i.e., $ap\notin w_{0}$). $\phi_{1}\wedge\phi_{2}$ is satisfied iff both $\phi_{1}$ and $\phi_{2}$ are satisfied. $\phi_{1}\vee\phi_{2}$ is satisfied iff $\phi_{1}$ or $\phi_{2}$ are satisfied. $\bigcirc\phi$ is satisfied iff $\phi$ is satisfied for the suffix of ${\bf w}$ that begins from the next position, i.e., i.e., $w_{1}w_{2}\cdots$. Finally, $\phi_{1}\mathit{U}\phi_{2}$ is satisfied iff $\phi_{1}$ is satisfied until $\phi_{2}$ is satisfied. Given ${\bf w}=w_{0}w_{1}\cdots\in(2^{AP})^{\omega}$ and an scLTL formula $\phi$, we denote by ${\bf w}\models\phi$ iff ${\bf w}$ satisfies $\phi$. It is known that every ${\bf w}=w_{0}w_{1}\cdots$ that satisfies the scLTL formula $\phi$ contains a finite good prefix $w_{0}w_{1}\cdots w_{n}$ for some $n\in\mathbb{N}_{\geq 0}$, such that $w_{0}w_{1}\cdots w_{n}{\bf w}^{\prime}\in(2^{AP})^{\omega}$ also satisfies $\phi$ for all ${\bf w}^{\prime}\in(2^{AP})^{\omega}$.

A finite state automaton (FSA) is defined as a tuple $\mathcal{A}=(Q,\Sigma,\delta,q_{0},Q_{f})$, where $Q$ is a set of states, $\Sigma$ is the input alphabet, $\delta:Q\times\Sigma\rightarrow Q$ is the transition function, $q_{0}\in Q$ is the initial state, and $Q_{f}\subseteq Q$ is the set of accepting states. Moreover, denote by $Post:Q\rightarrow 2^{Q}$ the successors for each state in $Q$, i.e., $Post(q)=\{q^{\prime}\in Q\ |\ \exists\sigma\in\Sigma,q^{\prime}\in\delta(q,\sigma)\}$. It is known that any scLTL formula $\phi$ can be translated into the FSA with $\Sigma=2^{AP}$, in the sense that all good prefix for $\phi$ can be accepted by $\mathcal{A}_{\phi}$. We denote by $\mathcal{A}_{\phi}$ the FSA corresponding to the scLTL formula $\phi$. The translation from scLTL formulas to the FSA can be automatically done using several off-the-shelf tools, such as SCHECK2 (latvala2003, ).

We capture an environment as a two-dimensional map consisting of $n$ cells. For example, this map is obtained by discretizing a given bounded search area into uniform grids with $n$ cells. Let $x_{i}\in\mathbb{R}^{2}$, $i\in\{1,\ldots,n\}$ be the position or the centroid of the cell $i$, and let $X=\{x_{1},\ldots,x_{n}\}$. Moreover, we denote by $AP$ the set of atomic propositions, which represents a set of labels or properties that can be satisfied in the states. In addition, we denote by ${L}:X\rightarrow 2^{AP}$ the labeling function, which represents a mapping from each state $x\in X$ onto the corresponding set of atomic propositions that are satisfied in $x$. For example, if $L(x)=\{\mathit{obstacle}\}$ with $AP=\{\mathit{obstacle}\}$, it intuitively means that there is an obstacle in the state $x$. In this paper, it is assumed that the labeling function $L$ is unknown due to the uncertainty of the environment, i.e., we do not have a complete knowledge about the properties of states in the environment. Thus, instead of $L$, we make use of the belief or the posterior probability (given the past observations) as follows:

for all $x\in X$ and $ap\in AP$, where $x\models ap$ denotes that $ap$ is satisfied in $x$ (i.e., $ap\in L(x)$). For example, $\mathcal{B}(x\models\mathit{obstacle})=1$ with $AP=\{\mathit{obstacle}\}$ intuitively means that, it is for sure that there exists an obstacle in $x$. In addition, $\mathcal{B}(x\models\mathit{obstacle})=0.5$ intuitively means that, it is completely unknown whether there exists an obstacle in $x$. The belief that $ap$ is not satisfied in $x$ is denoted as $\mathcal{B}(x\models\neg ap)$, and, from (2), it is computed as $\mathcal{B}(x\models\neg ap)=1-\mathcal{B}(x\models ap)$. In what follows, the beliefs in (2) are called the environmental beliefs of the atomic propositions. As we will see later, the environmental beliefs of the atomic propositions are updated based on the observations provided by sensors equipped with the copter and the rover.

The mission specification that the rover should satisfy is expressed by an scLTL, denoted by $\phi$, over the set of atomic propositions $AP$. The satisfaction relation of the scLTL formula $\phi$ is given over the word generated by the trajectory of the rover. That is, given a policy sequence $\mu_{r,seq}=\mu_{r,0}\mu_{r,1}\mu_{r,2}\ldots$, we say that the trajectory ${\bf x}_{\mu_{r,seq}}=x(0)x(1)\ldots\in X^{\omega}$ satisfies $\phi$, which we denote by ${\bf x}_{\mu_{r,seq}}\models\phi$, iff the corresponding word satisfies $\phi$, i.e., ${\bf w}=L(x(0))L(x(1))\ldots\models\phi$. Since the rover aims at achieving the satisfaction of $\phi$, we would like to derive an optimal policy, such that the probability of satisfying $\phi$, i.e., ${\rm Pr}[{\bf x}_{\mu_{r,seq}}\models\phi]$, is maximized. However, since the labeling function $L$ is unknown, the values of ${\rm Pr}[{\bf x}_{\mu_{r,seq}}\models\phi]$ are also unknown (i.e., we do not have direct access to this probability value). Hence, we will instead compute and maximize the belief that the trajectory of the rover satisfies $\phi$, which we denote by

That is, $\mathcal{B}({\bf x}_{\mu_{r,seq}}\models\phi)$ indicates the posterior that the trajectory of the rover satisfies $\phi$ given the past observations (sensor measurements) provided by the rover and the copter. As we will see later, (5) is computed and maximized based on the environmental beliefs of the atomic propositions in (2). Since the environmental beliefs of the atomic propositions will be updated based on the sensor measurements, the optimal policy that maximizes (5) will be also updated accordingly. Moreover, since we would like to reduce the environmental uncertainties as much as possible (i.e., we would like to make $\mathcal{B}(x\models ap)$ converge to $1$ or $0$ for all $x\in X,ap\in AP$), it is also necessary to explore the state space $X$ so as to collect the sensor measurements and update the environmental beliefs of the atomic propositions. In this paper, the copter has the main role to explore the uncertain environment, since, as previously mentioned in Section 1, it is able to move more quickly and freely than the rover. Therefore, we need to synthesize not only an optimal policy for the rover such that (5) is maximized so as to increase the possibility to satisfy $\phi$, but also an exploration policy for the copter so as to update the environmental beliefs of the atomic propositions and effectively reduce the environmental uncertainties.

Following (nilsson2018, ), we consider a sequential approach to solve Problem 1. The overview of the approach is shown in Algorithm 1.

With a slight abuse of notation, we denote by $\mathcal{B}$ in the algorithm the set of all environmental beliefs of the atomic propositions, i.e., $\mathcal{B}(x\models ap),x\in X,ap\in AP$. Specifically, the approach mainly consists of the two phases: the exploration phase and the mission execution phase. During the exploration phase, the copter explores the state-space $X$ so as to update $\mathcal{B}$ for a given time period $T_{c}\in\mathbb{N}_{>0}$. In (6) (as well as (7)), $b_{\max}:X\rightarrow[0,1]$ will denote a mapping from each state onto the corresponding maximum belief that the rover will reach within the time period $T_{r}$ according to the current optimal policy; for the detailed definition, see Section 4.3.3. As we will see later, the exploration policy is given by making use of the mapping $b_{\max}$ and the entropy that will be derived from the current environmental beliefs of the atomic propositions $\mathcal{B}$. Once the exploration is done, the copter transmits the updated environmental beliefs of the atomic propositions $\mathcal{B}$ to the rover and moves on to the mission execution phase. During the mission execution phase, the rover computes the optimal policy such that (5) is maximized, and executes the policy for a given time period $T_{r}\in\mathbb{N}_{>0}$. Moreover, during the execution, the rover provides sensor measurements to update $\mathcal{B}$. Once the execution is done, the rover transmits the updated $\mathcal{B}$ and $b_{\max}$ to the copter and moves back to the exploration phase. The sequential approach as above is motivated by the fact that, before allowing the rover to execute the optimal policy, we can let the copter in advance explore regions around the rover’s (future) path. For example, the copter checks if no obstacles are present along with the path that the rover intends to follow in the future. Then, if the copter finds some obstacles in the path, the rover can re-design the path for avoiding the obstacles and try to find another way to complete the mission. Such scheme is somewhat too careful, but may be necessary to be done especially for safety critical systems, such as the exploration on Mars.

As detailed below, the algorithms for both the exploration and the mission execution phases are significantly different from (nilsson2018, ) in the following three aspects. First, the environmental beliefs of the atomic propositions are updated based on the Bayes rule using the past sensor measurements. This allows us to provide novel copter’s explorations, in which the copter actively explores the environment by evaluating both the level of uncertainty and the relevancy to the mission completion (see Section 4.2). Second, we propose a novel framework to synthesize the optimal policy for the rover. In particular, we solve a value iteration over a product MDP, whose state-space does not involve the set of the states for the environmental beliefs. This leads to the reduction of the time complexity of solving the value iteration algorithm (for details, see Section 4.3). Finally, we provide a theoretical, convergence analysis of the proposed algorithm, where it is shown that the environmental beliefs of the atomic propositions converge to the appropriate values (for details, see Section 5).

In this subsection, we propose an algorithm of how the copter explores the environment so as to effectively update the environmental beliefs of the atomic propositions.

In this subsection, we propose a detailed algorithm of the rover’s mission execution phase.