Nonzero-sum Discrete-time Stochastic Games with Risk-sensitive Ergodic Cost Criterion

We consider infinite horizon ergodic-cost risk-sensitive two-person nonzero-sum stochastic games for discrete time Markov decision processes (MDPs) on a general state space with bounded cost function. We first establish the existence of unique solutions of the corresponding optimality equations. This is used to establish the existence of optimal randomized stationary strategies for a player fixing the strategy of the other player. We then define a suitable topology on the space of randomized stationary strategies. Then under certain separability assumptions we show that a suitably defined set-valued map has a fixed point. This gives the existence of a Nash equilibrium for the nonzero-sum game under consideration. The main step towards this end is to establish the upper semi-continuity of the defined set-valued map.

In a stochastic control problem, since the cost payable depends on the random evolution of the underlying stochastic process, the same is also random. So the most basic approach is to minimize the expected cost. This is the risk-neutral approach. But the obvious short coming of this approach is that it does not take care of the risk factor. In the risk-sensitive criterion, the expected value of the exponential of the cost is considered. Generally the risk associated with a random quantity is quantified by the standard deviation. Hence risk-sensitive criterion provides significantly better protection from the risk since it captures the effects of the higher order moments of the cost as well, see [24] for more details. The analysis of risk-sensitive control is technically more involved because of the exponential nature of the cost; accumulated costs are multiplicative over time as opposed to additive in the risk-neutral case. Risk-sensitive control problems also has deep connections to robust control theory, control theory literature which deals with model uncertainty, see [24] and references therein. Risk-sensitive criterion also arises naturally in portfolio optimization problems in mathematical finance, see [6] and associated references. Nonzero-sum stochastic games arise naturally in strategic multi-agent decision-making system like socioeconomic systems [16] , network security [1], routing and scheduling [19] and so on.

Following the seminal work of of Howard and Matheson [15] there has been a lot of work on risk-sensitive control problems. For an up to date survey on ergodic risk-sensitive control problems with underlying stochastic process being discrete-time Markov chains, see [7]. In the multi-controller set up, discrete-time risk-sensitive zero-sum games on countable state space have been studied in [2, 11]. In [4] the authors consider discrete-time zero-sum risk-sensitive stochastic games on Borel state space for both discounted and ergodic cost criterion. Their analysis involves transforming the original risk-sensitive problem to a risk-neutral problem. Nonzero-sum risk-sensitive stochastic games with both discounted and ergodic cost criterion for countable state space and bounded cost has been studied in [3, 22]. In [23] discrete-time nonzero-sum stochastic games under the risk-sensitive ergodic cost criterion with countable state space and unbounded cost function has been investigated. To the best of our knowledge, this is the first paper which considers risk-sensitive nonzero-sum games on a general state space. The analysis of nonzero-sum games on general state space is significantly more involved as compared to the case of countable state space. One of the reasons for this is that in case of countable state space the topology on the space of randomized stationary controls is fairly simple to work with. In case of general state space this becomes a substantial challenge. In the risk-neutral setup itself the analysis becomes substantially involved and requires additional assumptions. In the risk-neutral set up discrete-time nonzero-sum ergodic cost games with general state space has been studied in [10, 17, 21]. In [10], the authors prove the existence of a Nash-equilibrium under an additive reward additive transition(ARAT) assumption. In [17], the authors establish the existence of an $\epsilon$-Nash equilibrium. In [21], the authors assume that the transition law is a combination of finitely many probability measures. In this work we also assume ARAT condition.

In this paper we first consider ergodic optimality equations which correspond to control problems with the strategy of one player fixed. We show the existence of unique solutions to these equations. We follow a span contraction approach. For this analysis we make certain ergodicity assumption along with a few extra assumptions on the state transition kernel. Analogous assumptions appear in [8] where the authors consider risk-sensitive control problems with ergodic cost criterion. In order to establish the existence of a Nash equilibrium we first define an appropriate set valued map. In order to to show the existence of a Nash equilibrium for the nonzero-sum game we use Fan’s fixed point theorem [9]. This involves showing that the defined set valued map is upper semi-continuous under the appropriate topology.

The rest of the paper is organized as follows. Section 2 introduces the game model, some preliminaries and notations. In section 3 we show the existence of a unique solution to the optimality equation using a certain span-norm contraction. Existence of the Nash-equilibrium has been shown in section 4.

In this section, we present the discrete-time nonzero-sum stochastic game model and introduce the notations utilized throughout the paper. The following elements are needed to describe the discrete-time nonzero-sum stochastic game:

where each component is described below.

• •

  $X$ is the state space assumed to be Polish space endowed with the Borel $\sigma$-algebra $\mathcal{X}$.

• •

  $A$ and $B$ are action spaces of player $1$ and $2$ respectively, assumed to be compact metric spaces. Let $\mathcal{A}$ and $\mathcal{B}$ denote the Borel $\sigma$-algebras on $A$ and $B$ respectively.

• •

  $P$ is the transition kernel from $X\times A\times B\to\mathcal{P}(X)$, where for any metric space $D$, $\mathcal{P}(D)$ denotes the space of all probability measures on $D$ with the topology of weak convergence.

• •

  $c_{i}:X\times A\times B\to\mathbb{R}$, $i=1,2,$ is one-stage cost function for player $i$, assumed to be bounded and continuous on $A\times B$. Since, the cost is bounded, without loss of generality let, $0\leq c_{i}\leq\bar{c}$.

At each stage(time) the players observe the current state $x\in X$ of the system and then player $1$ and $2$ independently choose actions $a\in A$ and $b\in B$ respectively. As a consequence two things happen:

• (i)

  player $i,i=1,2$, pays an immediate cost $c_{i}(x,a,b).$

• (ii)

  the system moves to a new state $y$ with the distribution $P(\cdot|x,a,b).$

The whole process then repeats from the new state $y$. The cost accumulates throughout the course of the game. The planning horizon is taken to be infinite and each player wants to minimize his/her infinite-horizon risk-sensitive cost with respect to some cost criterion, which is in our case defined by (2.1) below.

At each stage the players choose their actions independently on the basis of the available information. The information available for decision making at time $t\in\mathbb{N}_{0}:=\{0,1,2,\ldots\}$ is given by the history of the process up to that time

where $H_{0}=X,H_{t}=H_{t-1}\times(A\times B\times X),\ldots,H_{\infty}=(X\times A\times B)^{\infty}$ are the history spaces. The history spaces are endowed with the corresponding Borel $\sigma-$algebra. A strategy for player 1 is a sequence $\pi^{1}=\left\{\pi_{t}^{1}\right\}_{t\in\mathbb{N}_{0}}$ of stochastic kernels $\pi_{t}^{1}:H_{t}\rightarrow\mathcal{P}(A)$. The set of all strategies for player 1 is denoted by $\Pi_{1}$. A strategy $\pi^{1}\in\Pi_{1}$ is called a Markov strategy if

for all $h_{t-1},h_{t-1}^{\prime}\in H_{t-1},a,a^{\prime}\in A,b,b^{\prime}\in B,x\in X,t\in\mathbb{N}_{0}$. Thus a Markov strategy for player 1 can be identified with a sequence of measurable maps $\left\{\varPhi_{t}^{1}\right\},\varPhi_{t}^{1}:X\rightarrow\mathcal{P}(A)$. A Markov strategy $\left\{\Phi_{t}^{1}\right\}$ is called a stationary strategy if $\varPhi_{t}^{1}=\Phi:X\rightarrow\mathcal{P}(A)$ for all $t$. Let $M_{1}$ and $S_{1}$ denote the set of Markov and stationary strategies for player 1 , respectively. The strategies for player 2 are defined similarly. Let $\Pi_{2},M_{2},$ and $S_{2}$ denote the set of arbitrary, Markov and stationary strategies for player 2, respectively.

Given an initial distribution $\tilde{\pi}_{0}\in\mathcal{P}(X)$ and a pair of strategies $(\pi^{1},\pi^{2})\in\Pi_{1}\times\Pi_{2}$, the corresponding state and action processes $\left\{X_{t}\right\},\left\{A_{t}\right\},\left\{B_{t}\right\}$ are stochastic processes defined on the canonical space $(H_{\infty},\mathfrak{B}(H_{\infty}),P_{\tilde{\pi}_{0}}^{\pi^{1},\pi^{2}})$(where $\mathfrak{B}(H_{\infty})$ is the Borel $\sigma$-field on $H_{\infty}$) via the projections $X_{t}\left(h_{\infty}\right)=x_{t},A_{t}\left(h_{\infty}\right)=a_{t},B_{t}\left(h_{\infty}\right)=b_{t}$, where $P_{\tilde{\pi}_{0}}^{\pi^{1},\pi^{2}}$ is uniquely determined by $\pi^{1},\pi^{2}$ and $\tilde{\pi}_{0}$ by Ionescu Tulcea’s theorem [5, Proposition 7.28]. When $\tilde{\pi}_{0}=\delta_{x_{0}}$ (the dirac measure at $x_{0}$),$~{}x_{0}\in X$, we simply write this probability measure as $P_{x_{0}}^{\pi_{1},\pi_{2}}$. For $h_{t}\in H_{t},a\in A,b\in B$, we have

Let the corresponding expectation operator be denoted by $E_{{\tilde{\pi}}_{0}}^{\pi^{1},\pi^{2}}(E_{x_{0}}^{\pi^{1},\pi^{2}})$ with respect to the probability measure $P_{{\tilde{\pi}}_{0}}^{\pi^{1},\pi^{2}}(P_{x_{0}}^{\pi^{1},\pi^{2}})$. Now from [13] we know that under any $(\Phi\times\Psi)\in S_{1}\times S_{2}$, the corresponding stochastic process $X_{t}$ is a Markov process.

For $i=1,2$, let us define transition measures $\tilde{P}_{i}$ from $X\times A\times B\to\mathcal{P}(X)$ by

Moreover, define for $i=1,2,~{}\varphi\in\mathcal{P}(A)$ and $\psi\in\mathcal{P}(B)$

and

Obviously $\tilde{P}_{i}$ for $i=1,2$ is in general is not a probability measure. The normalizing constant for $x\in X,\varphi\in\mathcal{P}(A),$ and $\psi\in\mathcal{P}(B)$ is given by

Since $0\leq c_{i}\leq\bar{c}$, the function $\tilde{c}_{i}$ is also bounded. More precisely $1\leq\tilde{c}_{i}(x,\varphi,\psi)\leq e^{\bar{c}}$ for $i=1,2$ and for each $x\in X,\varphi\in\mathcal{P}(A)$ and $\psi\in\mathcal{P}(B)$. Thus for $i=1,2$

defines a probability transition kernel and we also use the notation $\hat{c}_{i}(x,\varphi,\psi)$:= $\ln\tilde{c}_{i}(x,\varphi,\psi)$ for $i=1,2$.

We will use the above transformations, to convert our optimality equations (3.5) and (3.9) into a well-known equation. This process is beneficial as it helps us to prove the existence of the unique solution to the optimality equations as we will see in the next section.

Define $\mathbb{B}(X)$, the space of all real valued bounded measurable functions on $X$ endowed with the supremum norm $\|.\|$. For a fixed $\varphi\in\mathcal{P}(A),$ and $v\in\mathbb{B}(X)$ define the operator:

Due to the dual representation of the exponential certainty equivalent [12, Lemma 3.3] it is possible to write (2.5) as

where the supremum is over all probability measures $\mu\in\mathcal{P}(X)$ and $I(p,q)$ is the relative entropy of the two probability measures $p,q$ which is defined by

when $p<<q$ and $+\infty$ otherwise. Note that the supremum in (2.6) is attained at the probability measure given by

for measurable set $E\subset X$. Obviously $\mu^{\varphi}_{x\psi v}$ can be interpreted as a transition kernel.

Let $v\in\mathbb{B}(X)$. The span semi-norm of $v$ is defined as:

In the last part of this section we make a couple of assumptions that will be in force throughout the rest of the paper. First we will consider the following continuity assumption which is quite standard in literature, see [4, 8, 10, 18] for instance. This will allow us to show that the map $T$ defined in (2.5) is a contraction in the span semi-norm.

It follows from (2.2) that $\tilde{P}_{i}(dy|x,a,b)$ is also strongly continuous in $(a,b)$ for $i=1,2$ which lead us the following remark.

Next we have the following ergodicity assumption.

In this section, we demonstrate that the operator $T$ defined in (2.5) is a contraction. The fixed point of $T$ corresponds to the solution of the optimality equation for player 1. In the latter part of this section, we define another operator $U$ corresponding to player 2 and establish results analogous to those obtained for $T$.

We will now make additional assumptions to show that $T$ is a global contraction in $\mathbb{B}_{L}(X).$

Before proceeding to the main theorem of this section, we briefly outline some main points. Suppose player 2 announces that he/she is going to employ a strategy $\Psi\in S_{2}$. In such a scenario, player 1 attempts to minimize

over $\pi^{1}\in\Pi_{1}$. Thus for player 1 it is a discrete-time Markov decision problem with risk sensitive ergodic cost. Player 2 go through equivalent situations when player 1 announces his strategy to be $\Phi\in S_{1}$. This leads us to the following theorem.