Hardness of Independent Learning and
Sparse Equilibrium Computation in Markov Games

In spite of active research effort and many promising pieces of progress [JLWY21, SMB22, MB21, DGZ22, ELS⁺22], no-regret learning guarantees for Markov games have been elusive. A barrier faced by naive algorithms is that it is intractable to ensure no-regret against an arbitrary adversary, both computationally [BJY20, AYBK⁺13] and statistically [LWJ22, KECM21, FRSS22].

Fortunately, many of the implications of no-regret learning (in particular, convergence to equilibria) do not require the algorithm to have sublinear regret against an arbitrary adversary, but rather only against other agents who are running the same algorithm independently. This observation has been influential in normal-form games, where the well-known line of work on fast rates of convergence to equilibrium [SALS15, CP20, DFG21, AFK⁺22] holds only in this more restrictive setting. This motivates the following relaxation to our central question.

Two recent lines of research have made progress toward addressing Problem 1.1 and related questions. In one direction, several recent papers have provided algorithms, including V-learning [JLWY21, SMB22, MB21] and SPoCMAR [DGZ22], that do not achieve no-regret, but can nevertheless compute and then sample from a coarse correlated equilibrium in a Markov game in a (mostly) decentralized fashion, with the caveat that they require a shared source of random bits as a mechanism to coordinate. Notably, V-learning depends only mildly on the shared randomness: agents first play policies in a fully independent fashion (i.e., without shared randomness) according to a simple learning algorithm for $T$ episodes, and use shared random bits only once learning finishes as part of a post-processing procedure to extract a CCE policy. A question left open by these works, is whether the sequence of policies played by the V-learning algorithm in the initial independent phase can itself guarantee each agent sublinear regret; this would eliminate the need for a separate post-processing procedure and shared randomness.

Most closely related to our work, [ELS⁺22] recently showed that Problem 1.1 can be solved positively for a restricted setting in which regret for each agent is defined as the maximum gain in value they can achieve by deviating to a fixed Markov policy. Markov policies are those whose choice of action depends only on the current state as opposed to the entire history of interaction. This notion of deviation is restrictive because in general, even when the opponent plays a sequence of Markov policies, the best response will be non-Markov. In challenging settings that abound in practice, it is standard to consider non-Markov policies [LDGV⁺21, AVDG⁺22], since they often achieve higher value than Markov policies; we provide a simple example in Proposition 6.1. Thus, while a regret guarantee with respect to the class of Markov policies (as in [ELS⁺22]) is certainly interesting, it may be too weak in general, and it is of great interest to understand whether Problem 1.1 can be answered positively in the general setting.¹¹1We remark that the V-learning and SPoCMAR algorithms mentioned above do learn equilibria that are robust to deviations to non-Markov policies, though they do not address Problem 1.1 since they do not have sublinear regret.

We provide two computational lower bounds (Theorems 1.2 and 1.3) which show that under standard complexity-theoretic assumptions, there is no efficient algorithm that runs for a polynomial number of episodes and guarantees each agent non-trivial (“sublinear”) regret when used in tandem by all agents. Both results hold even if the Markov game is explicitly known to the algorithm designer; Theorem 1.3 is stronger and more general, but applies only to $3$-player games, while Theorem 1.2 applies to $2$-player games, but only for agents restricted to playing Markovian policies.

To state our first result, Theorem 1.2, we define a product Markov policy to be a joint policy in which players choose their actions independently according to Markov policies (see Sections 2 and 3 for formal definitions). Note that if all players use independent no-regret algorithms to choose Markov policies at each episode, then their joint play at each round is described by a product Markov policy, since any randomness in each player’s policy must be generated independently.

Theorem 1.2 provides a decisive negative resolution to Problem 1.1 under the assumption that $\textsf{PPAD}\neq\textsf{P}$,²²2Technically, the class we are denoting by P, namely of total search problems that have a deterministic polynomial-time algorithm, is sometimes denoted by $\mathsf{FP}$, as it is a search problem. We ignore this distinction. which is standard in the theory of computational complexity [Pap94].³³3PPAD is the most well-studied complexity class in algorithmic game theory, and is widely believed to not admit polynomial time algorithms. Notably, the problem of computing a Nash equilibrium for normal-form games with two or more players is PPAD-complete [DGP09, CDT06, Rub18]. Beyond simply ruling out the existence of fully decentralized no-regret algorithms, it rules out existence of centralized algorithms that compute a sequence of product policies for which each agent has sublinear regret, even if such a sequence does not arise naturally as the result of agents independently following some learning algorithm. Salient implications include:

• •

  Theorem 1.2 provides a separation between Markov games and normal-form games, since standard no-regret algorithms for normal-form games i) run in polynomial time and ii) produce sequences of joint product policies that guarantee each agent sublinear regret. Notably, no-regret learning for normal-form games is efficient whenever the number of agents is polynomial, whereas Theorem 1.2 rules out polynomial-time algorithms for as few as two agents.

• •

  A question left open by the work of [JLWY21, SMB22, MB21] was whether the sequence of policies played by the V-learning algorithm during its independent learning phase can guarantee each agent sublinear regret. Since V-learning plays product Markov policies during the independent phase and is computationally efficient, Theorem 1.2 implies that these policies do not enjoy sublinear regret (assuming $\textsf{PPAD}\neq\textsf{P}$).

Our second result, Theorem 1.3, extends the guarantee of Theorem 1.2 to the more general setting in which agents can select arbitrary, potentially non-Markovian policies at each episode. This comes at the cost of only providing hardness for 3-player games as opposed to 2-player games, as well as relying on the slightly stronger complexity-theoretic assumption that $\textsf{PPAD}\nsubseteq\textsf{RP}$.⁴⁴4We use RP to denote the class of total search problems for which there exists a polynomial-time randomized algorithm which outputs a solution with probability at least $2/3$, and otherwise outputs “fail”.

Theorems 1.2 and 1.3 rely on the widely-believed complexity theoretic assumption that PPAD-complete problems cannot be solved in (randomized) polynomial time. Such a restriction is inherent if we assume that the game is known to the algorithm designer. To avoid complexity-theoretic assumptions, we consider a setting in which the Markov game is unknown to the algorithm designer, and algorithms must learn about the game by executing policies (“querying”) and observing the resulting sequences of states, actions, and rewards. Our final result, Theorem 1.4, shows unconditionally that, for $m$-player Markov games whose parameters are unknown, any algorithm computing a no-regret sequence as in Theorem 1.3 requires a number of queries that is exponential in $m$.

Similar to our computational lower bounds, Theorem 1.4 goes far beyond decentralized algorithms, and rules out even centralized algorithms that compute a no-regret sequence by jointly controlling all players. The result provides another separation between Markov games and normal-form games, since standard no-regret algorithms for normal-form games can achieve sublinear regret using $\operatorname{poly}(m)$ queries for any $m$. The $2^{\Omega(m)}$ scaling in the lower bound, which does not rule out query-efficient algorithms when $m$ is constant, is to be expected for an unconditional result: If the game has only polynomially many parameters (which is the case for constant $m$), one can estimate all of the parameters using standard techniques [JKSY20], then directly find a no-regret sequence.

Rather than directly proving lower bounds for the problem of no-regret learning, we establish lower bounds for a simpler problem we refer to as SparseCCE. In the SparseCCE problem, the aim is to compute by any means—centralized, decentralized, or otherwise—a coarse correlated equilibrium that is “sparse” in the sense that it can be represented as the mixture of a small number of product policies. Any algorithm that computes a sequence of product policies with sublinear regret (in the sense of Theorem 1.3) immediately yields an algorithm for the SparseCCE problem, as—using the standard connection between CCE and no-regret—the uniform mixture of the policies in the no-regret sequence forms a sparse CCE. Thus, any lower bound for the sparse SparseCCE problem yields a lower bound for computation of no-regret sequences.

To provide lower bounds for the SparseCCE problem, we reduce from the problem of Nash equilibrium computation in normal-form games. We show that given any two-player normal-form game, it is possible to construct a Markov game (with two players in the case of Theorem 1.2 and three players in the case of Theorem 1.3) with the property that i) the description length is polynomial in the description length of the normal-form game, and ii) any (approximate) SparseCCE for the Markov game can be efficiently transformed into a approximate Nash equilibrium for the normal-form game. With this reduction established, our computational lower bounds follow from celebrated PPAD-hardness results for approximate Nash equilibrium computation in two-player normal-form games, and our statistical lower bounds follow from query complexity lower bounds for Nash. Proving the reduction from Nash to SparseCCE constitutes the bulk of our work, and makes novel use of aggregation techniques from online learning [Vov90, CBL06], as well as techniques from the literature on anti-folk theorems in game theory [BCI⁺08].

For $n\in\mathbb{N}$, we write $[n]:=\{1,2,\ldots,n\}$. For a finite set $\mathcal{T}$, $\Delta(\mathcal{T})$ denotes the space of distributions on $\mathcal{T}$. For an element $t\in\mathcal{T}$, $\mathbb{I}_{t}\in\Delta(\mathcal{T})$ denotes the delta distribution that places probability mass $1$ on $t$. We adopt standard big-oh notation, and write $f=\widetilde{O}(g)$ to denote that $f=O(g\cdot{}\max\left\{1,\mathrm{polylog}(g)\right\})$, with $\Omega(\cdot)$ and $\widetilde{\Omega}(\cdot)$ defined analogously.

We use the following notation: for some quantity $x$ (e.g., action, reward, etc.) indexed by agents, i.e., $x=(x_{1},\ldots,x_{m})$, and an agent $i\in[m]$, we write $x_{-i}=(x_{1},\ldots,x_{i-1},x_{i+1},\ldots,x_{m})$ to denote the tuple consisting of all $x_{i^{\prime}}$ for $i^{\prime}\neq i$.

A randomized Markov policy for agent $i$ is a sequence $\sigma_{i}=(\sigma_{i,1},\ldots,\sigma_{i,H})$, where $\sigma_{i,h}:\mathcal{S}\rightarrow\Delta(\mathcal{A}_{i})$. We denote the space of randomized Markov policies for agent $i$ by $\Pi^{\mathrm{markov}}_{i}$. We write $\Pi^{\mathrm{markov}}:=\Pi^{\mathrm{markov}}_{1}\times\cdots\times\Pi^{\mathrm{markov}}_{m}$ to denote the space of product Markov policies, which are joint policies in which each agent $i$ independently follows a policy in $\Pi^{\mathrm{markov}}_{i}$. In particular, a policy $\sigma\in\Pi^{\mathrm{markov}}$ is specified by a collection $\sigma=(\sigma_{1},\ldots,\sigma_{H})$, where $\sigma_{h}:\mathcal{S}\rightarrow\Delta(\mathcal{A}_{1})\times\cdots\times\Delta(\mathcal{A}_{m})$. We additionally define $\Pi^{\mathrm{markov}}_{-i}:=\prod_{i^{\prime}\neq i}\Pi^{\mathrm{markov}}_{i^{\prime}}$, and for a policy $\sigma\in\Pi^{\mathrm{markov}}$, write $\sigma_{-i}$ to denote the collection of mappings $\sigma_{-i}=(\sigma_{-i,1},\ldots,\sigma_{-i,H})$, where $\sigma_{-i,h}:\mathcal{S}\rightarrow\prod_{i^{\prime}\neq i}\Delta(\mathcal{A}_{i^{\prime}})$ denotes the tuple of all but player $i$’s policies.

When the Markov game $\mathcal{G}$ is clear from context, for a policy $\sigma\in\Pi^{\mathrm{markov}}$ we let $\mathbb{P}_{\sigma}\left[\cdot\right]$ denote the law of the trajectory $\tau$ when players select actions via $\mathbf{a}_{h}\sim{}\sigma(s_{h})$, and let $\operatorname{\mathbb{E}}_{\sigma}\left[\cdot\right]$ denote the corresponding expectation.

In addition to Markov policies, we will consider general history-dependent (or, non-Markov) policies, which select actions based on the entire sequence of states and actions observed up the current step. To streamline notation, for $i\in[m]$, let $\tau_{i,h}=(s_{1},a_{i,1},r_{i,1},\ldots,s_{h},a_{i,h},r_{i,h})$ denote the history of agent $i$’s states, actions, and reward up to step $h$. Let $\mathscr{H}_{i,h}=(\mathcal{S}\times\mathcal{A}_{i}\times[0,1])^{h}$ denote the space of all possible histories of agent $i$ up to step $h$. For $i\in[m]$, a randomized general (i.e., non-Markov) policy of agent $i$ is a collection of mappings $\sigma_{i}=(\sigma_{i,1},\ldots,\sigma_{i,H})$ where $\sigma_{i,h}:\mathscr{H}_{i,h-1}\times\mathcal{S}\rightarrow\Delta(\mathcal{A}_{i})$ is a mapping that takes the history observed by agent $i$ up to step $h-1$ and the current state and outputs a distribution over actions for agent $i$.

We denote by ${\Pi}^{\mathrm{gen,rnd}}_{i}$ the space of randomized general policies of agent $i$, and further write ${\Pi}^{\mathrm{gen,rnd}}:={\Pi}^{\mathrm{gen,rnd}}_{1}\times\cdots\times{\Pi}^{\mathrm{gen,rnd}}_{m}$ to denote the space of product general policies; note that $\Pi^{\mathrm{markov}}_{i}\subset{\Pi}^{\mathrm{gen,rnd}}_{i}$ and $\Pi^{\mathrm{markov}}\subset{\Pi}^{\mathrm{gen,rnd}}$. In particular, a policy $\sigma\in{\Pi}^{\mathrm{gen,rnd}}$ is specicfied by a collection $(\sigma_{i,h})_{i\in[m],h\in[H]}$, where $\sigma_{i,h}:\mathscr{H}_{i,h-1}\times\mathcal{S}\rightarrow\Delta(\mathcal{A}_{i})$. When agents play according to a general policy $\sigma\in{\Pi}^{\mathrm{gen,rnd}}$, at each step $h$, each agent, given the current state $s_{h}$ and their history $\tau_{i,h-1}\in\mathscr{H}_{i,h-1}$, chooses to play an action $a_{i,h}\sim\sigma_{i,h}(\tau_{i,h-1},s_{h})$, independently from all other agents. For a policy $\sigma\in{\Pi}^{\mathrm{gen,rnd}}$, we let $\mathbb{P}_{\sigma}\left[\cdot\right]$ and $\operatorname{\mathbb{E}}_{\sigma}\left[\cdot\right]$ denote the law and expectation operator for the trajectory $\tau$ when players select actions via $\mathbf{a}_{h}\sim{}\sigma(\tau_{h-1},s_{h})$, and write $\sigma_{-i}$ to denote the collection of policies of all agents but $i$, i.e., $\sigma_{-i}=(\sigma_{j,h})_{h\in[H],j\in[m]\backslash\{i\}}$.

We will also consider distributions over product randomized general policies, namely elements of $\Delta({\Pi}^{\mathrm{gen,rnd}})$.⁷⁷7When $\mathcal{T}$ is not a finite set, we take $\Delta(\mathcal{T})$ to be the set of Radon probability measures over $\mathcal{T}$ equipped with the Borel $\sigma$-algebra. We will refer to elements of $\Delta({\Pi}^{\mathrm{gen,rnd}})$ as distributional policies. To play according to some distributional policy $P\in\Delta({\Pi}^{\mathrm{gen,rnd}})$, agents draw a randomized policy $\sigma\sim P$ (so that $\sigma\in{\Pi}^{\mathrm{gen,rnd}}$) and then play according to $\sigma$.

It will be helpful to introduce notation for deterministic general (non-Markov) policies, which correspond to the special case of randomized policies where each policy $\sigma_{i,h}$ exclusively maps to singleton distributions. In particular, a deterministic general policy of agent $i$ is a collection of mappings $\pi_{i}=(\pi_{i,1},\ldots,\pi_{i,H})$, where $\pi_{i,h}:\mathscr{H}_{i,h-1}\times\mathcal{S}\rightarrow\mathcal{A}_{i}$. We denote by $\Pi^{\mathrm{gen,det}}_{i}$ the space of deterministic general policies of agent $i$, and further write $\Pi^{\mathrm{gen,det}}:=\Pi^{\mathrm{gen,det}}_{1}\times\cdots\times\Pi^{\mathrm{gen,det}}_{m}$ to denote the space of joint deterministic policies. We use the convention throughout that deterministic policies are denoted by the letter $\pi$, whereas randomized policies are denoted by $\sigma$.

For a general policy $\sigma\in{\Pi}^{\mathrm{gen,rnd}}$, we define the value function for agent $i\in[m]$ as

this represents the expected reward that agent $i$ receives when each agent chooses their actions via $a_{i,h}\sim\sigma_{h}(\tau_{i,h-1},s_{h})$. For a distributional policy $P\in\Delta({\Pi}^{\mathrm{gen,rnd}})$, we extend this notation by defining $V_{i}^{P}:=\mathbb{E}_{\sigma\sim P}[V_{i}^{\sigma}]$.

For a policy $\sigma\in{\Pi}^{\mathrm{gen,rnd}}$, we denote the distributional policy which puts all its mass on $\sigma$ by $\mathbb{I}_{\sigma}\in\Delta({\Pi}^{\mathrm{gen,rnd}})$. Thus $\frac{1}{T}\sum_{t=1}^{T}\mathbb{I}_{\sigma^{\scriptscriptstyle{(t)}}}\in\Delta({\Pi}^{\mathrm{gen,rnd}})$ denotes the distributional policy which randomizes uniformly over the $\sigma^{\scriptscriptstyle{(t)}}$. We define regret as follows.

The following standard result shows that the uniform average of any no-regret sequence forms an approximate coarse correlated equilibrium.

Fact 2.4 is an immediate consequence of Definitions 2.3 and 2.2. A standard approach to decentralized equilibrium computation, which exploits Fact 2.4, is to select $\sigma^{\scriptscriptstyle(1)},\ldots,\sigma^{\scriptscriptstyle(T)}\in{\Pi}^{\mathrm{gen,rnd}}$ using independent no-regret learning algorithms. A no-regret learning algorithm for player $i$ selects $\sigma_{i}^{\scriptscriptstyle(t)}\in{\Pi}^{\mathrm{gen,rnd}}_{i}$ based on the realized trajectories $\tau^{\scriptscriptstyle(1)}_{i,H},\ldots,\tau^{\scriptscriptstyle(t-1)}_{i,H}\in\mathscr{H}_{i,H}$ that player $i$ observes over the course of play,⁸⁸8An alternative model allows for player $i$ to have knowledge of the previous joint policies $\sigma^{\scriptscriptstyle{(1)}},\ldots,\sigma^{\scriptscriptstyle{(t-1)}}$, when selecting $\sigma_{i}^{\scriptscriptstyle{(t)}}$. but with no knowledge of $\sigma^{\scriptscriptstyle(t)}_{-i}$, so as to ensure that no-regret is achieved: $\mathrm{Reg}_{i,T}(\sigma^{\scriptscriptstyle{(1)}},\ldots,\sigma^{\scriptscriptstyle{(T)}})\leq\epsilon\cdot T$. If each player $i$ uses their own, independent no-regret learning algorithm, this approach yields product policies $\sigma^{\scriptscriptstyle(t)}=\sigma_{1}^{\scriptscriptstyle(t)}\times\cdots\times\sigma^{\scriptscriptstyle(t)}_{m}$, and the uniform average of the $\sigma^{\scriptscriptstyle{(t)}}$ yields a CCE as long as all of the players can keep their regret small.⁹⁹9In Section 6, we discuss the implications of relaxing the stipulation that $\sigma^{\scriptscriptstyle{(t)}}$ be product policies (for example, by allowing the use of shared randomness, as in V-learning). In short, allowing $\sigma^{\scriptscriptstyle{(t)}}$ to be non-product essentially trivializes the problem.

For the special case of normal-form games, the no-regret learning approach has been fruitful. There are several efficient algorithms, including regret matching [HMC00], Hedge (also known as exponential weights) [Vov90, LW94, CBFH⁺97], and generalizations of Hedge based on the follow-the-regularized-leader (FTRL) framework [SS12], which ensure that each player’s regret after $T$ episodes is bounded above by $O(\sqrt{T})$ (that is $\epsilon=O(1/\sqrt{T})$), even when the other players’ actions are chosen adversarially. All of these guarantees, which bound regret by a sublinear function in $T$, lead to efficient, decentralized computation of approximate coarse correlated equilibrium in normal-form games. The success of this motivates our central question, which is whether similar guarantees may be established for Markov games. In particular, a formal version of Problem 1.1 asks: Is there an efficient algorithm that, when adopted by all agents in a Markov game and run independently, ensures that for all $i$, $\mathrm{Reg}_{i,T}\leq\epsilon\cdot{}T$ for some $\epsilon=o(1)$?

Given a Markov game $\mathcal{G}$, we let $\beta(\mathcal{G})$ denote the maximum number of bits needed to describe any of the rewards $R_{i,h}(s,\mathbf{a})$ or transition probabilities $\mathbb{P}_{h}(s^{\prime}|s,\mathbf{a})$ in binary.¹⁰¹⁰10We emphasize that $\beta(\mathcal{G})$ is defined as the maximum number of bits required by any particular $(s,\mathbf{a})$ pair, not the total number of bits required for all $(s,\mathbf{a})$ pairs. We define $|\mathcal{G}|:=\max\{S,\max_{i\in[m]}A_{i},H,\beta(\mathcal{G})\}$. The interpretation of $|\mathcal{G}|$ depends on the number of players $m$: If $m$ is a constant (as will be the case in the current section and Section 4), then $|\mathcal{G}|$ should be interpreted as the description length of the game $\mathcal{G}$, up to polynomial factors. In particular, for constant $m$, the game $\mathcal{G}$ can be described using $|\mathcal{G}|^{O(1)}$ bits. In Section 5, we discuss the interpretation of $|\mathcal{G}|$ when $m$ is large.

From Fact 2.4, we know that the problem of computing a sequence $\sigma^{\scriptscriptstyle{(1)}},\ldots,\sigma^{\scriptscriptstyle{(T)}}$ of joint product Markov policies for which each player has at most $\epsilon\cdot{}T$ regret is equivalent to computing a sequence $\sigma^{\scriptscriptstyle{(1)}},\ldots,\sigma^{\scriptscriptstyle{(T)}}$ for which the uniform mixture forms an $\epsilon$-approximate CCE. We define $(T,\epsilon)$-SparseMarkovCCE as the computational problem of computing such a CCE directly.

Decentralized learning algorithms naturally lead to solutions to the SparseMarkovCCE problem. In particular, consider any decentralized protocol which runs for $T$ episodes, where at each timestep $t\in[T]$, each player $i\in[m]$ chooses a Markov policy $\sigma_{i}^{\scriptscriptstyle{(t)}}\in\Pi^{\mathrm{markov}}_{i}$ to play, without knowledge of the other players’ policies $\sigma_{-i}^{\scriptscriptstyle(t)}$ (but possibly using the history); any strategy in which players independently run online learning algorithms falls under this protocol. If each player experiences overall regret at most $\epsilon\cdot T$, then the sequence $\sigma^{\scriptscriptstyle(1)},\ldots,\sigma^{\scriptscriptstyle(T)}$ is a solution to the $(T,\epsilon)$-SparseMarkovCCE problem. However, one might expect the $(T,\epsilon)$-SparseMarkovCCE problem to be much easier than decentralized learning, since it allows for algorithms that produce $(\sigma^{\scriptscriptstyle{(1)}},\ldots,\sigma^{\scriptscriptstyle{(T)}})$ satisfying the constraints of Definition 3.1 in a centralized manner. The main result of this section, Theorem 3.2, rules out the existence of any efficient algorithms, including centralized ones, that solve the SparseMarkovCCE problem.

Before moving on, let us give a sense for what sort of scaling one should expect for the parameters $T$ and $\epsilon$ in the $(T,\epsilon)$-SparseMarkovCCE problem. First, we note that there always exists a solution to the $(1,0)$-SparseMarkovCCE problem in a Markov game, which is given by a (Markov) Nash equilibrium of the game; of course, Nash equilibria are intractable to compute in general.¹¹¹¹11Such a Nash equilibrium can be seen to exist by using backwards induction to specify the player’s joint distribution of play at each state at steps $H,H-1,\ldots,1$. For the special case of normal-form games (where there is only a single state, and $H=1$), no-regret learning (e.g., Hedge) yields a computationally efficient solution to the $(T,\widetilde{O}(1/\sqrt{T}))$-SparseMarkovCCE problem, where the $\widetilde{O}(\cdot)$ hides a $\max_{i}\log|A_{i}|$ factor. Refined convergence guarantees of [DFG21, AFK⁺22] improve upon this result, and yield an efficient solution to the $(T,\widetilde{O}(1/T))$-SparseMarkovCCE problem.

The proof of Theorem 3.2 is based on a reduction, which shows that any algorithm that efficiently solves the $(T,\epsilon)$-SparseMarkovCCE problem, for $T$ not too large, can be used to efficiently compute an approximate Nash equilibrium of any given normal-form game. In particular, fix $n_{0}\in\mathbb{N}$, and let a 2-player normal form game $G$ with $n_{0}$ actions be given. We construct a Markov game $\mathcal{G}=\mathcal{G}(G)$ with horizon $H=n_{0}$ and action sets identical to those of the game $G$, i.e., $\mathcal{A}_{1}=\mathcal{A}_{2}=[n_{0}]$. The state space of $\mathcal{G}$ consists $n_{0}^{2}$ states, which are indexed by joint action profiles; the transitions are defined so that the value of the state at step $h$ encodes the action profile taken by the agents at step $h-1$.¹³¹³13For technical reasons, this only is the case for even values of $h$; we discuss further details in the full proof in Section B.2. At each state of $\mathcal{G}$, the reward functions are given by the payoff matrices of $G$, scaled down by a factor of $1/H$ (which ensures that the rewards received at each step belong to $[0,1/H]$). In particular, the rewards and transitions out of a given state do not depend on the identity of the state, and so $\mathcal{G}$ can be thought of as a repeated game where $G$ is played $H$ times. The formal definition of $\mathcal{G}$ is given in Definition B.3.

Fix any algorithm for the SparseMarkovCCE problem, and recall that for each step $h$ and state $s$ for $\mathcal{G}$, $\sigma^{\scriptscriptstyle{(t)}}_{h}(s)\in\Delta(\mathcal{A}_{1})\times\Delta(\mathcal{A}_{2})$ denotes the joint action distribution taken in $s$ at step $h$ for the sequence of $\sigma^{\scriptscriptstyle(1)},\ldots,\sigma^{\scriptscriptstyle(T)}$ produced by the algorithm. The bulk of the proof of Theorem 3.2 consists of proving a key technical result, Lemma B.4, which states that if $\sigma^{\scriptscriptstyle(1)},\ldots,\sigma^{\scriptscriptstyle(T)}$ indeed solves $(T,\epsilon)$-SparseMarkovCCE, then there exists some tuple $(h,s,t)$ such that $\sigma_{h}^{\scriptscriptstyle{(t)}}(s)$ is an approximate Nash equilibrium for $G$. With this established, it follows that we can find a Nash equilibrium efficiently by simply trying all $HST$ choices for $(h,s,t)$.

To prove Lemma B.4, we reason as follows. Assume that $\overline{\sigma}:=\frac{1}{T}\sum_{t=1}^{T}\mathbb{I}_{\sigma^{\scriptscriptstyle{(t)}}}\in\Delta({\Pi}^{\mathrm{gen,rnd}})$ is an $\epsilon$-CCE. If, by contradiction, none of the distributions $\left\{\sigma_{h}^{\scriptscriptstyle{(t)}}(s)\right\}_{h\in[H],s\in\mathcal{S},t\in[T]}$ are approximate Nash equilibria for $G$, then it must be the case that for each $t$, one of the players has a profitable deviation in $G$ with respect to the product strategy $\sigma_{h}^{\scriptscriptstyle{(t)}}(s)$, at least for a constant fraction of the tuples $(s,h)$. We will argue that if this were to be the case, it would imply that there exists a non-Markov deviation policy for at least one player $i$ in Definition 2.2, meaning that $\overline{\sigma}$ is not in fact an $\epsilon$-CCE.

To sketch the idea, recall that to draw a trajectory from $\overline{\sigma}$, we first draw a random index $t^{\star}\sim[T]$ uniformly at random, and then execute $\sigma^{\scriptscriptstyle{(t^{\star})}}$ for an episode. We will show (roughly) that for each player $i$, it is possible to compute a non-Markov deviation policy $\pi_{i}^{\dagger}$ which, under the draw of a trajectory from $\overline{\sigma}$, can “infer” the value of the index $t^{\star}$ within the first few steps of the episode. Then policy $\pi_{i}^{\dagger}$ then, at each state $s$ and step $h$ after the first few steps, play a best response to their opponent’s portion of the strategy $\sigma_{h}^{\scriptscriptstyle{(t^{\star})}}(s)$. If, for each possible value of $t^{\star}$, none of the distributions $\sigma_{h}^{\scriptscriptstyle{(t^{\star})}}(s)$ are approximate Nash equilibria of $G$, this means that at least one of the players $i$ can significantly increase their value in $\mathcal{G}$ over that of $\overline{\sigma}$ by playing $\pi_{i}^{\dagger}$, which contradicts the assumption that $\overline{\sigma}$ is an $\epsilon$-CCE.

It remains to explain how we can construct a non-Markov policy $\pi_{i}^{\dagger}$ which “infers” the value of $t^{\star}$. Unfortunately, exactly inferring the value of $t^{\star}$ in the fashion described above is impossible: for instance, if there are $t_{1}\neq t_{2}$ so that $\sigma^{\scriptscriptstyle{(t_{1})}}=\sigma^{\scriptscriptstyle{(t_{2})}}$, then clearly it is impossible to distinguish between the cases $t^{\star}=t_{1}$ and $t^{\star}=t_{2}$. Nevertheless, by using the fact that each player observes the full joint action profile played at each step $h$, we can construct a non-Markov policy which employs Vovk’s aggregating algorithm for online density estimation [Vov90, CBL06] in order to compute a distribution which is close to $\sigma_{h}^{\scriptscriptstyle{(t^{\star})}}(s)$ for most $h\in[H]$.¹⁴¹⁴14Vovk’s aggregating algorithm is essentially the exponential weights algorithm with the logarithmic loss. A detailed background for the algorithm is provided in Section B.1. This guarantee is stated formally in an abstract setting in Proposition B.2, and is instantiated in the proof of Theorem 3.2 in (Equation 6). As we show in Section B.2, approximating $\sigma_{h}^{\scriptscriptstyle{(t^{\star})}}(s)$ as we have described is sufficient to carry out the reasoning from the previous paragraph.

SparseCCE is the problem of computing a sequence of non-Markov product policies $\sigma^{\scriptscriptstyle{(1)}},\ldots,\sigma^{\scriptscriptstyle{(T)}}$ such that the uniform mixture forms an $\epsilon$-approximate CCE. The problem generalizes SparseMarkovCCE (Definition 3.1) by relaxing the condition that the policies $\sigma^{\scriptscriptstyle{(t)}}$ be Markov.