$\textsc{LTL}_{f}$ Synthesis on Probabilistic Systems

Recall that the semantics of $\textsc{LTL}_{f}$ formulas is over finite traces whereas the interpretation of LTL formulas is over infinite traces. In order to reduce the $\textsc{LTL}_{f}$ synthesis problem to an LTL one, we need to be able to capture the finite paths (traces) of $\mathcal{M}$ that satisfy $\phi$ and extend them to infinite paths (traces). Specifically, we need those satisfying finite paths that contain no $\phi$-satisfying proper prefixes, i.e., satisfy $\phi$ for the first time. To this end, we allow the system (policy) to decide when to “terminate.” We view the system to be “alive” until termination, at which point it is no longer alive. Then, we define an LTL formula that requires the system to be alive while it has not satisfied $\phi$ and to terminate after satisfying $\phi$.

To this end, we augment MDP $\mathcal{M}$ with a terminal action and state and an atomic proposition $alive$. Formally, we construct MDP $\mathcal{M}^{\prime}=(S^{\prime},A^{\prime},P^{\prime},s^{\prime}_{\text{init}},AP^{\prime},L^{\prime})$, where $S^{\prime}=S\cup\{s_{\text{term}}\}$, $A^{\prime}=A\cup\{a_{\text{term}}\}$, $s^{\prime}_{\text{init}}=s_{\text{init}}$, $AP^{\prime}=AP\cup\{alive\}$,

In this MDP, the system can decide to terminate by taking action $a_{\text{term}}$, in which case it transitions to state $s_{\text{term}}$ with probability one and remains there forever. The labeling of the corresponding path includes the atomic proposition $alive$ at every time step until $s_{\text{term}}$ is visited and is empty thereafter.

Example 2. Figure 2 illustrates the augmented MDP $\mathcal{M}^{\prime}$ constructed from the example MDP $\mathcal{M}$ in Figure 1. State $s_{\text{term}}$ along with the dashed edges and atomic proposition $alive$ are added to $\mathcal{M}$. The dashed edges are enabled by action $a_{\text{term}}$ and have transition probability of one. The label of $s_{\text{term}}$ is the empty set, and the labels of the rest of the states contain $alive$.

With this augmentation, the system is able to terminate once a satisfying finite path is generated. We show that, for finite properties, the maximum probability of satisfaction in $\mathcal{M}$ equals the maximum probability of satisfaction in $\mathcal{M}^{\prime}$ under a policy that enforces a visit to $s_{\text{term}}$.

Let $Paths_{fin,\phi}\subseteq Paths_{fin}(s_{\text{init}})$ be a set of finite paths of interest in arbitrary MDP $\mathcal{M}$. Denote the probability of satisfaction of infinite paths with prefixes from $Paths_{fin,\phi}$ under policy $\pi\in\Pi$ by:

The following lemma states the equivalence of the maximum probability of satisfying this set of paths in $\mathcal{M}$ and $\mathcal{M}^{\prime}$.

To translate an $\textsc{LTL}_{f}$ formula on $\mathcal{M}$ to its equivalent LTL formula on $\mathcal{M}^{\prime}$, we follow [11]. Let $\Phi_{f}$ be the set of $\textsc{LTL}_{f}$ formulas $\phi$ defined over atomic propositions in $AP$ and $\Phi$ be the set of LTL formulas $\varphi$ defined over $AP^{\prime}$. Then $g:\Phi_{f}\rightarrow\Phi$ is defined as:

where $t:\Phi_{f}\rightarrow\Phi$ is inductively defined as:

• •

  $t(p)=(p\wedge alive)$, where $p\in AP$;

• •

  $t(\neg\phi)=\neg t(\phi)$;

• •

  $t(\phi_{1}\wedge\phi_{2})=t(\phi_{1})\wedge t(\phi_{2})$;

• •

  $t(X\>\phi)=X\>(alive\>\wedge\>t(\phi))$;

• •

  $t(\phi_{1}\>U\>\phi_{2})=t(\phi_{1})\>U\>(alive\>\wedge\>t(\phi_{2}))$.

A direct result of the above theorem is the reduction of $\textsc{LTL}_{f}$ policy synthesis to classical LTL policy synthesis on MDPs.

Therefore, to solve Problem 1, we can use LTL synthesis on augmented MDP $\mathcal{M}^{\prime}$ and property $\varphi=g(\phi)$. The general LTL synthesis algorithm is well-established [6] and follows the following steps: (1) translation of the LTL formula to a DRA, (2) composition of the DRA with the MDP, which results in another MDP called the product MDP, (3) identification of the maximal end-components on the product MDP that satisfy the accepting condition of DRA, and finally (4) solving the maximal reachability probability problem (stochastic shortest path problem) [10] on the product MDP with the accepting end-components as the target states. There exist many tools such as PRISM [19] that can solve the LTL synthesis problem. Specifically, PRISM has a symbolic implementation of this algorithm, enabling fast computations. In Section 4, however, we show that the native approach introduced below outperforms the LTL-reduction approach even by using PRISM’s symbolic engine.

A Deterministic Finite Automaton (DFA) is a tuple: $\mathcal{A}=(Q,\Sigma,\delta,q_{0},F)$, where $Q$ is the set of states, $\Sigma$ the alphabet, $\delta:Q\times\Sigma\mapsto Q$ the transition function, $q_{0}$ the initial state and $F$ the set of accept states. A finite run of a DFA on a trace $\rho=\rho_{0}\rho_{1}\ldots\rho_{n}$, where $\rho_{i}\in\Sigma$, is the sequence of states $q_{0}q_{1}\ldots q_{n+1}$ such that $q_{i+1}=\delta(q_{i},\rho_{i})$ for all $0\leq i\leq n$. This run is accepting if $q_{n+1}\in F$.

Following [28], we translate $\textsc{LTL}_{f}$ to a DFA using MONA [17]. $\textsc{LTL}_{f}$ is expressively equivalent to First-order Logic on finite words, which in turn is a fragment of Weak Second-order Theory of One Successor. We use the translation given in [11] to convert $\textsc{LTL}_{f}$ formula $\phi$ to a First-order Logic formula. MONA offers translations from Weak Second-order Theory of One or Two Successors (WS1S/WS2S) to a DFA that accepts precisely the language of our $\textsc{LTL}_{f}$ formula $\phi$. We denote this DFA by $\mathcal{A}_{\phi}$.

Given DFA $\mathcal{A}_{\phi}$, we can take the product with the MDP $\mathcal{M}$ to achieve a new MDP $M\times\mathcal{A}_{\phi}$ as follows. The product of MDP $\mathcal{M}=(S,A,P,s_{\text{init}},AP,L)$ and DFA $\mathcal{A}_{\phi}=(Q,\Sigma,q_{0},\delta,F)$ is an MDP

where $q_{\text{init}}=\delta(q_{0},L(s_{\text{init}}))$, and

The paths of this product MDP have one-to-one correspondence to the paths of MDP $\mathcal{M}$ as well as the runs of $\mathcal{A}_{\phi}$. Therefore, the projection of the paths of $\mathcal{M}\times\mathcal{A}_{\phi}$ that reach state $(s,q)$, where $q\in F$, on $\mathcal{A}_{\phi}$ are accepting runs and on $\mathcal{M}$ are $\phi$-satisfy paths. Thus, we solve Problem 1 by synthesizing an optimal policy on this product MDP, using standard tools for the maximal reachability probability problem as discussed in Sec. 3.1. The resulting policy is stationary on the product MDP but history-dependent on $\mathcal{M}$.

In theory, both the LTL approach and the native approach exhibit runtimes doubly-exponential in the size of the formula, due to the need to construct deterministic automata [9, 12]. In practice, the native approach’s translation to a DFA using specialized tools offers better runtime and memory usage compared to the LTL-based approach. Additionally, it produces a minimal DFA, while the LTL pipeline produces an $\omega$-automaton which cannot be minimized effectively by existing tools. We show the benefits of the native approach experimentally in Sec. 4.

We consider four types of MDPs: Gridworld, Dining Philosophers, Nim, and Double Counter. In Gridworld, an agent is given some goal as an $\textsc{LTL}_{f}$ formula and must maximize the probability of satisfaction. The agent has four actions: North, South, East, and West. Under each action, the probability of moving to the cell in the intended direction is 0.69, then 0.01 for its opposite cell, and finally 0.1 for each of the other directions and for remaining in the same cell. If the resulting movement would place the agent in collision with the boundary, the agent remains in its current cell. We model the motion of this agent as an MDP, where the states correspond to the cells of the grid and the set of actions is $\{a_{\text{north}},a_{\text{south}},a_{\text{east}},a_{\text{west}}\}$.

We base the Dining-Philosophers domain on the tutorial from the PRISM website. We consider a ring of five philosophers with two different specifications. Note that typical Dining-Philosophers specifications of interest are infinite-trace; our tests are merely meant to show that our approach works on MDPs other than Gridworld. Both Nim and Double Counter are probabilistic versions of games presented in [21].

Unfortunately, there is no standard set of $\textsc{LTL}_{f}$ formulas that we can use for benchmarking. In [28], random LTL formulas are used as the basis of $\textsc{LTL}_{f}$ benchmarks; however, because we also consider an MDP, random formulas are frequently tautologies or non-realizable with respect to the MDP. For instance, consider a randomly generated formula $\phi=p_{1}Up_{2}$, where $p_{1}$ and $p_{2}$ are randomly assigned to the labels of the states in the Gridworld MDP. There is likely no path of the MDP that can satisfy this formula. As a matter of fact, in a test of more than 70 randomly generated $\textsc{LTL}_{f}$ formulas and Gridworld MDPs, only one was “interesting” (yielding probability between zero and one) on the corresponding MDP. For benchmarking, the interest is in sets of formulas that make sense in a probabilistic setting.

Therefore, as the first set of test formulas for the Gridworld MDP, we considered a natural finite-trace specification where the agent has $n$ goals to accomplish in any order, as well as a “safety” property where it must globally avoid some states. This is typical of e.g., a robotics domain [15]. We use Fn to denote this formula for a given $n$. For the second set of test formulas, we keep regions to visit and avoid but introduce some ordering and repeat visits. OS is a short formula introducing order and OL is a longer formula that also contains nested temporal operators. All formulas are given in the appendix of the online version [27].

For the Dining Philosophers domains, typical examples focus on infinite-run properties. We test several finite-run properties on these domains. These properties are meant to illustrate our tool on an MDP other than Gridworld, but are not representative of typical finite-trace properties. One version (D5) asserts that all philosophers must eat at least once. The other (D5C) is a more complex property involving orders of eating. Both formulas are available in the appendix of the online version [27].

For Nim, the game of Nim is played against a stochastic opponent. To increase difficulty, the specifications require that randomly chosen stack heights are to be reached or avoided by the system player. For Double Counter, two four-bit binary counters are used. One counter is controlled by a stochastic environment, and the other is controlled by the system. The aim of the system is to make its counter match that of the environment.

First we measure how the length of the formula affects the synthesis computation time. For all of these experiments, we use a $10\times 10$ Gridworld as the original MDP. For small formulas, the time needed to read the HOA file from hard disk outweighs the shorter construction time and smaller automata of the native approach as shown in 3(a). For formulas longer than F3, the native approach offers better computation time (e.g., see 3(b) and 3(c)).

PRISM successfully builds automata for formulas up to F9. In 3(d), we highlight the superiority of the native approach in constructing the automaton for F10. For this formula, PRISM runs out of memory when constructing the automaton according to the LTL pipeline. SPOT completes for F10 but runs out of memory on F11. MONA works for formulas up to F17, which takes 8.586 seconds to compute, though writing the resulting file to disk is prohibitively expensive (the file is larger than 10GB). MONA runs out of memory constructing the automaton for F18.

We also considered a Gridworld with random obstacles (3(e)) and with hallways (3(f)), both with the formula F9. Finally, we consider two other formulas (OS and OL) in 3(g) and 3(h) to demonstrate that our improvement is not tied to the specific form of the specification.

For the Dining-Philosophers example, we consider five philosophers. Again, our results show that the native pipeline is faster and more memory efficient than the other approaches for automata construction (3(l)). However, the Dining-Philosophers example illustrates an issue where PRISM’s built-in automata translation sometimes constructs automata such that computing the maximal accepting end-component using BDDs within PRISM is significantly faster (3(k)). The automata construction is still slower than both SPOT and our native approach, and the automata returned have more states, but the BDD representation is more efficient. This issue affects some automata, not only for our tool, but also for other external tools we tested. This means not only automata construction and size, but also the BDD representation are important for overall runtime. However, PRISM’s built-in translator’s memory usage scales more poorly than SPOT or our native approach. On a more complex formula 3(l), PRISM’s built-in construction runs out of memory, even though SPOT and the native approach can finish construction in less than ten seconds. Thus, for sufficiently complex formulas, PRISM’s built-in translation is not viable. This suggests a need for automata construction methods that are not only more efficient but also produce automata that work well within PRISM.

In the Nim game, the native approach far outperforms both PRISM and PRISM with SPOT (4(a)). In the Double-Counter game, PRISM runs out of memory and PRISM with SPOT times out after more than 3 hours, whereas the native approach completes synthesis is less than 5 seconds.

Overall we observe that for large formulas the native pipeline offers significantly better scalability than the LTL pipeline (tested with PRISM’s built-in translator, SPOT, Rabinizer, and LTL3DRA automata translators). With an implementation that does not access the hard disk, we expect even better performance of the native pipeline.

The DFA generated by the native pipeline is minimal and typically much smaller than the LTL pipeline’s DRA. SPOT and PRISM typically produce similarly sized DRAs. Because we take the product of the input MDP with these automata, we expect the size of the resulting product to be smaller in the native pipeline. Table 1 shows the automaton sizes for the various formulas. Table 2 shows the sizes of the product MDPs for the LTL and native pipelines respectively.

Interestingly, while the size of the DFA obtained from MONA is roughly half the size of the DRA constructed by PRISM, the final sizes of the products of the MDP and the automata are comparable for both approaches. The number of states, transitions and nondeterministic choices are all measured after reachability analysis is performed. We see time savings in the reachability analysis and product construction phases (both about two times faster), but the final products are similar in size. However, the product from the native pipeline has fewer nondeterministic transitions than the product from the LTL pipeline. The improvements from this are small relative to the time it takes to construct a large gridworld MDP (3(j)).

Examples of this are shown in 3(i) and 3(j). Note that the majority of the computation time is spent constructing the MDPs. For the LTL pipeline, this construction takes slightly longer time because of the augmented MDP. It is important to note that the native approach allows us to use larger formulas with these models whereas the LTL pipeline is limited to F10 or smaller.

In summary, we observe that the native pipeline is the most efficient in terms of both runtime and memory. There are two possible drawbacks to the native pipeline. First, it requires reading from disk, which for small formulas negates the advantage in automata construction time. Second, PRISM’s built-in automata translation sometimes constructs automata whose BDD representations work better within PRISM than automata from any external tools we tested (3(k)).