Risk-Averse Decision Making Under Uncertainty

Autonomous systems are being increasingly deployed in real-world settings. Hence, the associated risk that stems from unknown and unforeseen circumstances is correspondingly on the rise. This demands for autonomous systems that can make appropriately conservative decisions when faced with uncertainty in their environment and behavior. Mathematically speaking, risk can be quantified in numerous ways, such as chance constraints [50] and distributional robustness [51]. However, applications in autonomy and robotics require more “nuanced assessments of risk” [32]. Artzner et. al. [10] characterized a set of natural properties that are desirable for a risk measure, called a coherent risk measure, and have obtained widespread acceptance in finance and operations research, among other fields.

A popular model for representing sequential decision making under uncertainty is a Markov decision processes (MDP) [37]. MDPs with coherent risk objectives were studied in [47, 46], where the authors proposed a sampling-based algorithm for finding saddle point solutions using policy gradient methods. However, [47] requires the risk envelope appearing in the dual representation of the coherent risk measure to be known with an explicit canonical convex programming formulation. While this may be the case for CVaR, mean-semi-deviation, and spectral risk measures [42], such explicit form is not known for general coherent risk measures, such as EVaR. Furthermore, it is not clear whether the saddle point solutions are a lower bound or upper bound to the optimal value. Also, policy-gradient based methods require calculating the gradient of the coherent risk measure, which is not available in explicit form in general. For the CVaR measure, MDPs with risk constraints and total expected costs were studied in [36, 16] and locally optimal solutions were found via policy gradients, as well. However, this method also leads to saddle point solutions (which cannot be shown to be upper bounds or lower bounds of the optimal value) and cannot be applied to general coherent risk measures. In addition, because the objective and the constraints are in terms of different coherent risk measures, the authors assume there exists a policy that satisfies the CVaR constraint (feasibility assumption), which may not be the case in general. Following the footsteps of [35], a promising approach based on approximate value iteration was proposed for MDPs with CVaR objectives in [17]. A policy iteration algorithm for finding policies that minimize total coherent risk measures for MDPs was studied in [41] and a computational non-smooth Newton method was proposed in [41].

When the states of the agent and/or the environment are not directly observable, a partially observable MDP (POMDP) can be used to study decision making under uncertainty introduced by the partial state observability [26, 2]. POMDPs with coherent risk measure objectives were studied in [22, 23]. Despite the elegance of the theory, no computational method was proposed to design policies for general coherent risk measures. In [3], we proposed a method for finding finite-state controllers for POMDPs with objectives defined in terms of coherent risk measures, which takes advantage of convex optimization techniques. However, the method can only be used if the risk transition mapping is affine in the policy.

Summary of Contributions: In this paper, we consider MDPs and POMDPs with both objectives and constraints in terms of coherent risk measures. Our contributions are fourfold:

• (i)

  For MDPs, we use the Lagrangian framework and reformulate the problem into a inf-sup problem. For Markov risk transition mappings, we propose an optimization-based method to design Markovian policies that lower-bound the constrained risk-averse problem;

• (ii)

  For MDPs, we evince that the optimization problems are in the special form of DCPs and can be solved by the DCCP method. We also demonstrate that these results generalize linear programs for constrained MDPs with total discounted expected costs and constraints;

• (iii)

  For POMDPs, we demonstrate that, if the coherent risk measures can be defined as a Markov risk transition mapping, an infinite-dimensional optimization can be used to design Markovian belief-based policies, which in theory requires infinite memory to synthesize (in accordance with classical POMDP complexity results);

• (iv)

  For POMDPs with stochastic finite-state controllers (FSCs), we show that the latter optimization converts to a (finite-dimensional) DCP and can be solved by the DCCP framework. We incorporate these DCPs in a policy iteration algorithm to design risk-averse FSCs for POMDPs.

We assess the efficacy of the proposed method with numerical experiments involving conditional-value-at-risk (CVaR) and entropic-value-at-risk (EVaR) risk measures.

Preliminary results on risk-averse MDPs were presented in [4]. This paper, in addition to providing detailed proofs and new numerical analysis in the MDP case, generalizes [4] to partially observable systems (POMDPs) with dynamic coherent risk objectives and constraints.

The rest of the paper is organized as follows. In the next section, we briefly review some notions used in the paper. In Section III, we formulate the problem under study. In Section IV, we present the optimization-based method for designing risk-averse policies for MDPs. In Section V, we describe a policy iteration method for designing finite-memory controllers for risk-averse POMDPs. In Section VI, we illustrate the proposed methodology via numerical experiments. Finally, in Section VII, we conclude the paper and give directions for future research.

Notation: We denote by $\mathbb{R}^{n}$ the $n$-dimensional Euclidean space and $\mathbb{N}_{\geq 0}$ the set of non-negative integers. Throughout the paper, we use bold font to denote a vector and $(\cdot)^{\top}$ for its transpose, e.g., $\boldsymbol{a}=(a_{1},\ldots,a_{n})^{\top}$, with $n\in\{1,2,\ldots\}$. For a vector $\boldsymbol{a}$, we use $\boldsymbol{a}\succeq(\preceq)\boldsymbol{0}$ to denote element-wise non-negativity (non-positivity) and $\boldsymbol{a}\equiv\boldsymbol{0}$ to show all elements of $\boldsymbol{a}$ are zero. For two vectors $a,b\in\mathbb{R}^{n}$, we denote their inner product by $\langle\boldsymbol{a},\boldsymbol{b}\rangle$, i.e., $\langle\boldsymbol{a},\boldsymbol{b}\rangle=\boldsymbol{a}^{\top}\boldsymbol{b}$. For a finite set $\mathcal{A}$, we denote its power set by $2^{\mathcal{A}}$, i.e., the set of all subsets of $\mathcal{A}$. For a probability space $(\Omega,\mathcal{F},\mathbb{P})$ and a constant $p\in[1,\infty)$, $\mathcal{L}_{p}(\Omega,\mathcal{F},\mathbb{P})$ denotes the vector space of real valued random variables $c$ for which $\mathbb{E}|c|^{p}<\infty$.

In this section, we briefly review some notions and definitions used throughout the paper.

In the past two decades, coherent risk and dynamic risk measures have been developed and used in microeconomics and mathematical finance fields [49]. Generally speaking, risk-averse decision making is concerned with the behavior of agents, e.g. consumers and investors, who, when exposed to uncertainty, attempt to lower that uncertainty. The agents may avoid situations with unknown payoffs, in favor of situations with payoffs that are more predictable.

The core idea in risk-averse planning is to replace the conventional risk-neutral conditional expectation of the cumulative cost objectives with the more general coherent risk measures. In path planning scenarios, in particular, we will show in our numerical experiments that considering coherent risk measures will lead to significantly more robustness to environment uncertainty and collisions leading to mission failures.

In addition to total cost risk-aversity, an agent is often subject to constraints, e.g. fuel, communication, or energy budgets [27]. These constraints can also represent mission objectives, e.g. explore an area or reach a goal.

Consider a stationary controlled Markov process $\{q_{t}\}$, $t=0,1,\ldots$ (an MDP or a POMDP) with initial probability distribution $\kappa_{0}$, wherein policies, transition probabilities, and cost functions do not depend explicitly on time. Each policy $\pi=\{\pi_{t}\}_{t=0}^{\infty}$ leads to cost sequences $\boldsymbol{c}_{t}=c(q_{t},\alpha_{t})$, $t=0,1,\ldots$ and $\boldsymbol{d}_{t}^{i}=d^{i}(q_{t},\alpha_{t})$, $t=0,1,\ldots$, $i=1,2,\ldots,n_{c}$. We define the dynamic risk of evaluating the $\gamma$-discounted cost of a policy $\pi$ as

and the $\gamma$-discounted dynamic risk constraints of executing policy $\pi$ as

where $\rho^{\gamma}$ is defined in equation (4), $q_{0}\sim\kappa_{0}$, and $\beta^{i}>0$, $i=1,2,\ldots,n_{c}$, are given constants. We assume that $c(\cdot,\cdot)$ and $d^{i}(\cdot,\cdot)$, $i=1,2,\ldots,n_{c}$, are non-negative and upper-bounded. For a discount factor $\gamma\in(0,1)$, an initial condition $\kappa_{0}$, and a policy $\pi$, we infer from [41, Theorem 3] that both $J_{\gamma}(\kappa_{0},\pi)$ and $D_{\gamma}^{i}(\kappa_{0},\pi)$ are well-defined (bounded), if $c$ and $d$ are bounded.

In this work, we are interested in addressing the following problem:

We call a controlled Markov process with the “nested” objective (8) and constraints (9) a constrained risk-averse Markov process.

For MDPs, [17, 33] show that such coherent risk measure objectives can account for modeling errors and parametric uncertainties. We can also interpret Problem 1 as designing policies that minimize the accrued costs in a risk-averse sense and at the same time ensuring that the system constraints, e.g., fuel constraints, are not violated even in the rare but costly scenarios.

Note that in Problem 1 both the objective function and the constraints are in general non-differentiable and non-convex in policy $\pi$ (with the exception of total expected cost as the coherent risk measure $\rho^{\gamma}$ [9]). Therefore, finding optimal policies in general may be hopeless. Instead, we find sub-optimal polices by taking advantage of a Lagrangian formulation and then using an optimization form of Bellman’s equations.

Next, we show that the constrained risk-averse problem is equivalent to a non-constrained inf-sup risk-averse problem thanks to the Lagrangian method.

At any time $t$, the value of $\rho_{t}$ is $\mathcal{F}_{t}$-measurable and is allowed to depend on the entire history of the process $\{s_{0},s_{1},\ldots\}$ and we cannot expect to obtain a Markov optimal policy [34, 11]. In order to obtain Markov policies, we need the following property [41].

In fact, if $\rho_{t}$ is a coherent risk measure, $\sigma_{t}$ also satisfies the properties of a coherent risk measure (Definition 3). In this paper, since we are concerned with MDPs, the controlled kernel is simply the transition function $T$.

The simplest case of the risk transition mapping is in the conditional expectation case $\rho_{t}(v(s_{t+1}))=\mathbb{E}\{v(s_{t+1})\mid s_{t},\alpha_{t}\}$, i.e.,

Note that in the total discounted expectation case $\sigma$ is a linear function in $v$ rather than a convex function, which is the case for a general coherent risk measures. For example, for the CVaR risk measure, the Markov risk transition mapping is given by

where $(\cdot)_{+}=\max\{\cdot,0\}$ is a convex function in $v$.

If $\sigma$ is a coherent, Markov risk measure, then the Markov policies are sufficient to ensure optimality [41].

In the next result, we show that we can find a lower bound to the solution to Problem 1 via solving an optimization problem.

Once the values of $\boldsymbol{\lambda}^{*}$ and $\boldsymbol{V}^{*}_{\gamma}$ are found by solving optimization problem (1), we can find the policy as

One interesting observation is that if the coherent risk measure $\rho^{t}$ is the total discounted expectation, then Theorem 1 is consistent with the classical result by [9] on constrained MDPs.

In [4], we presented a method based on difference convex programs to solve (1), when $\rho^{\gamma}$ is an arbitrary coherent risk measure and we described the specific structure of the optimization problem for conditional expectation, CVaR, and EVaR. In fact, it was shown that (1) can be written in a standard DCP format as

Optimization problem (IV) is a standard DCP [25]. DCPs arise in many applications, such as feature selection in machine learning [29] and inverse covariance estimation in statistics [48]. Although DCPs can be solved globally [25], e.g. using branch and bound algorithms [28], a locally optimal solution can be obtained based on techniques of nonlinear optimization [13] more efficiently. In particular, in this work, we use a variant of the convex-concave procedure [30, 43], wherein the concave terms are replaced by a convex upper bound and solved. In fact, the disciplined convex-concave programming (DCCP) [43] technique linearizes DCP problems into a (disciplined) convex program (carried out automatically via the DCCP Python package [43]), which is then converted into an equivalent cone program by replacing each function with its graph implementation. Then, the cone program can be solved readily by available convex programming solvers, such as CVXPY [18].

We end this section by pointing out that solving (1) using the DCCP method, only finds the (local) saddle points to optimization problem  (1). Nevertheless, every saddle point to (1) satisfies (16) (from Theorem 1). In fact, every saddle point is a lower bound of the optimal value of Problem 1.

Next, we show that, in the case of POMDPs, we can find a lower bound to the solution to Problem 1 via solving an infinite-dimensional optimization problem. Note that a POMDP is equivalent to a belief MDP $\{b_{t}\}$, $t=1,2,\ldots$, where $b_{t}$ is defined in (2).

Unfortunately, since $b\in\Delta(\mathcal{S})$ and hence $V_{\gamma}:\Delta(\mathcal{S})\to\mathbb{R}$, optimization (2) is infinite-dimensional and we cannot solve it efficiently.

If the one-step coherent risk measure $\rho_{t}$ is the total discounted expectation, we can show that optimization problem (2) simplifies to an infinite-dimensional linear program and equality holds in (23). This can be proved following the same lines as the proof of Corollary 1 but for the belief MDP. Hence, Theorem 2 also provides an optimization based solution to the constrained POMDP problem.