Recursively-Constrained Partially Observable Markov Decision Processes

Partially Observable Markov Decision Processes (POMDPs) are powerful models for sequential decision making due to their ability to account for transition uncertainty and partial observability. Their applications range from autonomous driving Pendleton et al. [2017] and robotics to geology Lauri et al. [2023], Wang et al. [2022], asset maintenance Papakonstantinou and Shinozuka [2014], and human-computer interaction Chen et al. [2020]. Constrained POMDPs (C-POMDPs) are extensions of POMDPs that impose a bound on expected cumulative costs while seeking policies that maximize expected total reward. C-POMDPs address the need to consider multiple objectives in applications such as autonomous rover that may have a navigation task as well as an energy usage budget, or human-AI dialogue systems with constraints on the length of dialogues. However, we observe that optimal policies computed for C-POMDPs exhibit pathological behavior in some problems, which can be opposed to the C-POMDP’s intended purpose.

In this example, suppose the agent receives an observation that leads to an updated belief of $0.8$ probability that tunnel $A$ is rocky. Intuitively, the agent should avoid tunnel $A$ since the expected cost of navigating it is $8$, which violates the cost constraint of $5$. However, an optimal policy computed from a C-POMDP decides to go through the rocky region, violating the constraint and damaging the wheels. Such behavior is justified in the C-POMDP framework by declaring that, due to a low probability of observing that tunnel $A$ is rocky in the first place, the expected cost from the initial time step is still within the threshold, and so this policy is admissible. However, this pathological behavior is clearly unsuitable especially for some (e.g., safety-critical) applications.

In this paper, we first provide the key insight that the pathological behavior is caused by the violation of the optimal substructure property over successive decision steps, and hence violation of the standard form of Bellman’s Principle of Optimality (BPO). To mitigate the pathological behavior and preserve the optimal substructure property, we propose an extension of C-POMDPs through the addition of history-dependent cost constraints at each reachable belief, which we call Recursively-Constrained POMDPs (RC-POMDPs). We prove that deterministic policies are sufficient for optimality in RC-POMDPs and that RC-POMDPs satisfy BPO. These results suggest that RC-POMDPs are highly amenable to standard dynamic programming techniques, which is not true for C-POMDPs. RC-POMDPs provide a good balance between the BPO-violating expectation constraints of C-POMDPs and constraints on the worst-case outcome, which are overly conservative for POMDPs with inherent state uncertainty. Then, we present a point-based dynamic programming algorithm to approximately solve RC-POMDPs. Experimental evaluation shows that the pathological behavior is a prevalent phenomenon in C-POMDP policies, and that our algorithm for RC-POMDPs computes polices which obtain expected cumulative rewards competitive with C-POMDPs without exhibiting such behaviors.

In summary, this paper contributes (i) an analysis that C-POMDPs do not exhibit the optimal substructure property over successive decision steps and its consequences, (ii) the introduction of RC-POMDPs, a novel extension of C-POMDPs through the addition of history-dependent cost constraints, (iii) proofs that all RC-POMDPs have at least one deterministic optimal policy, satisfy BPO, and the Bellman operator has a unique fixed point under suitable initializations, (iv) a dynamic programming algorithm for RC-POMDPs, and (v) a series of illustrative benchmarks to demonstrate the advantages of RC-POMDPs.

POMDPs model sequential decision making problems under transition uncertainty and partial observability.

The evolution of an agent according to a POMDP model is as follows. At each $t\in\mathbb{N}_{0}$, the agent has a belief $b_{t}$ of its state $s_{t}$ as a probability distribution over $S$ and takes action $a_{t}\in A$. Its state evolves from $s_{t}\in S$ to $s_{t+1}\in S$ according to $T(s_{t},a_{t},s_{t+1})$, and it receives an immediate reward $R(s_{t},a_{t})$ and observation $o_{t}\in O$ according to observation probability $Z(s_{t+1},a_{t},o_{t})$. The agent then updates its belief using Bayes theorem; that is for $s_{t+1}=s^{\prime}$,

Then, the process repeats. Let $h_{t}=\{a_{0},o_{0},\cdots,a_{t-1},o_{t-1}\}$ denote the history of the actions and observations up to but not including time step $t$; thus, $h_{0}=\emptyset$. The belief at time step $t$ is therefore $b_{t}=P(s_{t}\mid b_{0},h_{t})$. For readability, we do not explicitly include $b_{0}$, as all variables are conditioned on $b_{0}$.

The agent chooses actions according to a policy $\pi:\Delta(S)\to\Delta(A)$, which maps a belief $b$ to a probability distribution over actions. $\pi$ is called deterministic if $\pi(b)$ is a unitary distribution for every $b\in\Delta(S)$. A policy is typically evaluated according to the expected rewards it accumulates over time. Let $R(b,a)=\mathbb{E}_{s\sim b}[R(s,a)]$ be the expected reward for the belief-action pair $(b,a)$. The expected discounted sum of rewards that the agent receives under policy $\pi$ starting from belief $b_{t}$ is

Additionally, the $Q$ reward-value is defined as

The objective of POMDP problems is often to find a policy that maximizes $V_{R}^{\pi}(b_{0})$.

As an extension of POMDPs, Constrained POMDPs add a constraint on the expected cumulative costs.

In C-POMDPs, the constraint $V^{\pi}_{C}(b_{0})\leq\hat{c}$, where $\leq$ refers to the component-wise inequality, is imposed on the POMDP optimization problem as formalized below.

Unlike POMDPs that have at least one deterministic optimal policy Sondik [1978], optimal policies of C-POMDPs may require randomization, and hence there may not exist an optimal deterministic policy Kim et al. [2011].

Next, we discuss why the solutions to Problem 1 may not be desirable and an alternate formulation is necessary.

To mitigate the pathological behaviors and obtain a (policy-independent) optimal substructure property, we aim to align optimal policies computed at a current belief with optimal policies computed at future (successor) beliefs. We propose a new problem formulation called Recursively-Constrained POMDP (RC-POMDP), which imposes additional recursively defined constraints on a policy.

An RC-POMDP has the same tuple as a C-POMDP, but with recursive constraints on beliefs at future time steps. These constraints enforce that a policy must satisfy a history dependent cumulative expected cost constraint at every future belief state. Intuitively, we bound the cost value at every belief such that the constraint in the initial node is respected.

The expected cumulative cost of the trajectories associated with history $h_{t}$ is given as:

We can direct the optimal policy at each time step $t$ by imposing that the total expected cumulative cost satisfies the initial cost constraint $\hat{c}$ . For a given $h_{t}$ and its corresponding $b_{t}$, the expected cumulative cost at $b_{0}$ is given by:

Therefore, the following constraint should be satisfied by a policy $\pi$ at each future belief state:

We define the admissibility of a policy $\pi$ accordingly.

Since RC-POMDP policies are constrained based on history, it is not sufficient to directly use belief-based policies. Thus, we consider history-based policies in this work. A history-based policy maps a history $h_{t}$ to a probability over actions $\Delta(A)$.

The RC-POMDP optimization problem is formalized below.

Note that Problem 2 is an infinite-horizon problem since the optimization objective (10) is infinite horizon. The admissibility constraint $k$ is a user-defined parameter. In this work, we focus on $k=\infty$, i.e., admissible policies.

With the theoretical foundation above, we devise a first attempt at an algorithm that approximately solves Problem 2 with scalar cost and admissibility constraint $k=\infty$. We leave the multi-dimensional and finite $k$ cases for future work. The algorithm is called Admissibility Recursively Constrained Search (ARCS). ARCS takes advantage of the Markovian property of the belief-admissible cost formulation in Proposition 2, and Theorems 1-3 to utilize point-based dynamic programming in the space of deterministic and admissible policies, building on unconstrained POMDP methods [Shani et al., 2013].

ARCS is outlined in Algorithm 1. It takes as input the RC-POMDP $\mathcal{M}$ and $\epsilon>0$, a target error between the computed policy and an optimal policy at $b_{0}$. ARCS explores the search space by incrementally sampling points in the history space. These points form nodes in a policy tree $T$. At each iteration, a SAMPLE step expands a sequence of points starting from the root. Then, a Bellman BACKUP step is performed for each sampled node. Finally, a PRUNE step removes sub-optimal nodes. These three steps are repeated until an admissible $\epsilon$-optimal policy is found. Pseudocode for SAMPLE, BACKUP and PRUNE are provided in the appendix.

Policy Tree Representation   We represent the policy with a policy tree $T$. A node in $T$ is a tuple $v=(b,d,k,\overline{V}_{R},\underline{V}_{R},\overline{V}_{C},\underline{V}_{C},\overline{Q}_{R},\underline{Q}_{R},\overline{Q}_{C},\underline{Q}_{C})$, where $b$ is a belief, $d$ is a history-dependent admissible cost bound, $k$ is a lower bound on admissible horizon, $\overline{V}_{R}$ and $\underline{V}_{R}$ are the two-sided bounds on reward-values, $\overline{V}_{C}$ and $\underline{V}_{C}$ are the two-sided cost-value bounds, $\overline{Q}_{R},\underline{Q}_{R}$ represent two-sided bound on $Q$ reward-value, and $\underline{Q}_{C},\overline{Q}_{C}$ represent the two-sided bounds on $Q$ cost-value. The root of $T$ is the node $v_{0}$ with $b=b_{0}$, $d=\hat{c}$, and admissible horizon lower bound $k_{0}$.

From Theorem 3, a key aspect of effective dynamic programming for RC-POMDPs is computing admissible policies. This can be approximated by minimizing $V_{C}(\hat{b}_{t})$. As a pre-processing step, we first approximate a minimum cost-value policy $\pi_{c}^{min}=\arg\inf_{\pi}V^{\pi}_{C}$. An arbitrarily tight under-approximation (upper bound) $\hat{\pi}_{c}^{min}$ as a set of $|S|$-dimensional hyperplanes, called $\alpha$-vectors, can be computed efficiently with a POMDP algorithm Hauskrecht [2000]. The reward-values obtained by $\hat{\pi}_{c}^{min}$ is also a lower bound on the optimal reward-value. Thus, $\hat{\pi}_{c}^{min}$ is represented by a set of $\alpha$-vector pairs $(\alpha_{r},\alpha_{c})\in\Gamma_{C}^{min}$. $\hat{\pi}_{c}^{min}$ is used to initialize our policy, and is used from leaf nodes of $T$.

To initialize a new node, belief $b^{\prime}$ is computed with Eq. (1), and $d^{\prime}$ is computed recursively with Eq. (13). We initialize $\underline{V}_{R}$ and $\overline{V}_{C}$ with $\hat{\pi}_{c}^{min}$, and initialize $\underline{V}_{C}$ and $\overline{V}_{R}$ independently using the Fast Informed Bound (FIB) Hauskrecht [2000], and $k^{\prime}$ is a lower bound of the admissible horizon.

Admissible Horizon Lower Bound   It is computationally intractable to exhaustively search the possibly infinite policy space. Thus, we maintain a lower bound on the admissible horizon of the policy for every node. It is used to compute admissibility beyond the current search depth of the tree, and to improve search efficiency via pruning. To initialize the admissible horizon guarantee of a leaf node, we compute a lower bound on the admissible horizon $k$ when using $\hat{\pi}_{c_{min}}$.

A proof is provided in the appendix. This lemma provides sufficient conditions for admissibility of computed policies. We compute an upper bound on the parameter $C_{max}$,

where $\alpha_{c}$ refers to a cost $\alpha$-vector. This can be solved efficiently with the maximin LP Williams [1990]:

Sampling   ARCS uses a mixture of random sampling Spaan and Vlassis [2005] and heuristic search (SARSOP) Kurniawati et al. [2008]. Our empirical evaluations suggest that this approach is an effective balance between finding policies with high cumulative reward and that are admissible. At each SAMPLE step, ARCS expands the search space from the root of $T$, with either heuristic sampling or random sampling. For heuristic sampling, we use the same sampling strategy and sampling termination condition as SARSOP. It works by choosing actions with the highest $\overline{Q}_{R}$, and observations that have the largest contribution to the gap at the root of $T$, and sampling terminates based on a combination of selective deep sampling and a gap termination criterion. With random sampling, actions and observations are chosen randomly while traversing the tree until a new node is reached and added to the tree. Sampled points are chosen for BACKUP.

Backup   The BACKUP operation at node $v$ updates the values in the node by back-propagating the information of the children of $v$ back to $v$. First, the values of $\overline{Q}_{R},\underline{Q}_{R}$ and $\overline{Q}_{C}$ are computed for each action using Eq. (3) and Eq. (5) for rewards and costs, respectively. Then, an RC-POMDP backup Eq. (15) is used to update $\overline{V}_{R},\underline{V}_{R},\underline{V}_{C},\overline{V}_{C}$. The action selected to update $\underline{V}_{R}$ is used to update $k$ by back-propagating the minimum $k$ of all children. If no actions are feasible, all current policies from that node are inadmissible, and we update the reward- and cost-values using the action with the minimum $Q$-cost value, and set $k=0$.

Pruning   To keep the size of $T$ small and improve tractability, we prune nodes and node-actions that are suboptimal, using the following criteria. First, for each node $v\in B_{sam}$, if $v.\underline{V}_{C}>v.d$, no admissible policies exist from $v$, so $v$ and its subtree are pruned. Next, we prune actions as follows. Let $k(v,a)$ be the admissible horizon guarantees of the successor nodes from taking action $a$ at node $v$. Between two actions $a$ and $a^{\prime}$, if $k(v,a^{\prime})=\infty\text{ and }v.\overline{Q}_{R}(a)<v.\underline{Q}_{R}(a^{\prime})$, we prune the node-action $(v,a)$ (disallow taking action $a$ at node $v$), since action $a$ can never be taken by the optimal admissible policy. Next, if all node-actions $(v,a)$ are pruned, $v$ is also pruned. Finally, the node-action $(v,a)$ is pruned if any successor node from taking $a$ at $v$ is pruned. Nodes and node-actions that are pruned are not chosen during action and observation selection during SAMPLE and BACKUP.

To the best of our knowledge, this work is the first to propose and solve RC-POMDPs, and thus there are no existing algorithms to compare to directly. The purpose of our evaluation is to (i) empirically compare the behavior of policies computed for RC-POMDPs with those computed for C-POMDPs, and (ii) evaluate the performance of our proposed algorithm for RC-POMDPs. To this end, we consider the following offline algorithms to compare against our ARCS³³3Our code is open sourced at https://github.com/CU-ADCL/RC-PBVI.jl:

• •

  CGCP Walraven and Spaan [2018]: Algorithm that computes near-optimal policies for C-POMDPs using a primal-dual approach.

• •

  CGCP-CL: Closed-loop CGCP with updates on belief and admissible cost at each time step.

• •

  Exp-Gradient Kalagarla et al. [2022]: Algorithm that computes mixed policies for C-POMDPs using a no-regret learning approach with a primal-dual approach using an Exponentiated Gradient method.

• •

  CPBVI Kim et al. [2011]: Approximate dynamic programming that uses admissible cost as a heuristic.

• •

  CPBVI-D: We modify CPBVI to compute deterministic policies to evaluate its efficacy for RC-POMDPs.

Since the purpose of our comparison between RC-POMDPs and C-POMDPs is mainly with regard to constraints, we do not compare to online C-POMDP algorithms such as CC-POMCP Lee et al. [2018] which can handle larger problems but do not have anytime guarantees on constraint satisfaction.

We consider the following environments: (i) CE: Counterexample in Figure 1, (ii) C-Tiger: A Constrained version of Tiger POMDP Kaelbling et al. [1998], (iii) CRS: Constrained RockSample Lee et al. [2018], and (iv) Tunnels: A scaled version of Example 1, shown in Figure 2. Details on each problem, experimental setup, and algorithm implementation are in the Appendix. For all algorithms except CGCP-CL, solve time is limited to $300$ seconds and online action selection to $0.05$ seconds. For CGCP-CL, $300$ seconds was given to re-compute each action. We report the mean discounted cumulative reward and cost, and constraint violation rate in Table 1. The constraint violation rate is the fraction of trials in which $d(h_{t})$ becomes negative, which means Eq. (11) is violated.

In all environments, ARCS found admissible policies ($k=\infty$). In contrast, CGCP, Exp-Gradient, CPBVI and CPBVI-D only guarantee an admissible horizon of $k=1$, since the C-POMDP constraint is only at the initial belief. CGCP-CL may have a closed-loop admissible horizon greater than 1, but does not provide guarantees, as indicated in the violation rate.

The benchmarking results show that the policies computed for ARCS generally achieve competitive cumulative reward to policies computed for C-POMDP, without any constraint violations and thus no pathological behavior. ARCS also generally performs better in all metrics than CPBVI and CPBVI-D, both of which could not search the problem space sufficiently to find good solutions in large RC-POMDPs.

Although the C-POMDP policies generally satisfy the C-POMDP expected cost constraints, the prevalence of high violation rates of C-POMDP policies across the environments strongly suggests that the manifestation of the stochastic self-destruction in C-POMDPs is not an exceptional phenomenon, but intrinsic to the C-POMDP problem formulation. This behavior is illustrated in the Tunnels problem, shown in Figure 2. CGCP (in blue) decides to traverse tunnel $A$ $51\%$ of the time even when it observes that $A$ is rocky, and traverses tunnel $B$ $49\%$ of the time. In contrast, ARCS never traverses tunnel $A$, since such a policy is inadmissible. Instead, it traverses $B$ or $C$ depending on observation of rocks in tunnel $B$, to maximize rewards while remaining admissible.

Finally, the closed-loop inconsistency of C-POMDP policies is evident when comparing open loop CGCP with closed loop CGCP-CL. In most cases (all except CRS(7,8)), the cumulative reward is decreased when going from CGCP to CGCP-CL, sometimes drastically. The violation rate also decreases, but not to $0$, suggesting that planning with C-POMDPs instead of RC-POMDPs can lead to myopic behavior that cannot be addressed by re-planning. As seen in CE and both CRS, CGCP-CL attains lower reward than ARCS while still having constraint violations. Therefore, even for closed-loop planning, RC-POMDP can be more advantageous than C-POMDP.

We introduce and analyze the stochastic self-destruction behavior of C-POMDP policies, and show C-POMDPs may not exhibit optimal substructure. We propose a new formulation, RC-POMDPs, and present an algorithm for RC-POMDPs. Results show that C-POMDP policies exhibit unintuitive behavior not present in RC-POMDP policies, and our algorithm effectively computes policies for RC-POMDPs. We believe RC-POMDPs are an alternate formulation that can be more desirable for some applications.

A future direction is to explore models that exhibit strong cases of stochastic self-destruction, and develop more metrics that signal stochastic self-destruction. Another future direction is to analyze classes (or conditions) of RC-POMDPs that are approximable and to design algorithms that converge for such cases.

Further, our offline policy tree search algorithm can benefit from better policy search heuristics and more efficient policy representations (e.g. finite state controllers). We plan to extend this work by exploring other approaches, such as searching for finite state controllers directly [Wray and Czuprynski, 2022] and online tree search approximations [Lee et al., 2018].

Finally, this work shows that RC-POMDPs can provide more desirable policies than C-POMDPs, but the cost constraints remain on expectation. For some applications, probabilistic or risk measure constraints may be more desirable than expectation constraints. These formulations also benefit from the recursive constraints that we propose for RC-POMDPs.