Heterogeneous Beliefs and Multi-Population Learning in Network Games

Smooth Fictitious play (SFP) and variants thereof are arguably amongst the most well-studied learning models in AI and game theory [2, 3, 21, 22, 9, 19, 36, 37, 42, 18, 17]. SFP describes a belief-based learning process: agents form beliefs about the play of opponents and update their beliefs based on observations. Informally, an agent’s belief can be thought as reflecting how likely its opponents will play each strategy. During game plays, each agent plays smoothed best responses to its beliefs. Much of the literature of SFP is framed in the context of homogeneous beliefs models where all agents in a given role have the same beliefs. This includes models with one agent in each player role [3, 2, 39] as well as models with a single population but in which all agents have the same beliefs [21, 22]. SFP are known to converge in large classes of homogeneous beliefs models (e.g., most 2-player games [9, 19, 3]). However, in the context of heterogeneous beliefs, where agents in a population have different beliefs, SFP has been explored to a less extent.

The study of heterogeneous beliefs (or more broadly speaking, population heterogeneity) is important and practically relevant. From multi-agent system perspective, heterogeneous beliefs widely exist in many applications, such as traffic management, online trading and video game playing. For example, it is natural to expect that public opinions generally diverge on autonomous vehicles and that people have different beliefs about the behaviors of taxi drivers vs non-professional drivers. From machine learning perspective, recent empirical advances hint that injecting heterogeneity potentially accelerates population-based training of neural networks and improves learning performance [25, 29, 44]. From game theory perspective, considering heterogeneity of beliefs better explains results of some human experiments [10, 11].

Heterogeneous beliefs models of SFP are not entirely new. In the pioneering work [12], Fudenberg and Takahashi examine the heterogeneity issue in 2-population settings by appealing to techniques from the stochastic approximation theory. This approach, which is typical in the SFP literature, relates the limit behavior of each individual to an ordinary differential equation (ODE) and has yielded significant insights for many homogeneous beliefs models [3, 2, 19, 39]. However, this approach, as also noted by Fudenberg and Takahashi, “does not provide very precise estimates of the effect of the initial condition of the system.” Consider an example of a population of agents each can choose between two pure strategies $s_{1}$ and $s_{2}$. Let us imagine two cases: (i) every agents in the population share the same belief that their opponents play a mixed strategy choosing $s_{1}$ and $s_{2}$ with equal probability $0.5$, and (ii) half of the agents believe that their opponents determinedly play the pure strategy $s_{1}$ and the other half believe that their opponents determinedly play the pure strategy $s_{2}$. The stochastic approximation approach would generally treat these two cases equally, providing little information about the heterogeneity in beliefs as well as its consequential effects on the system evolution. This drives our motivating questions:

How does heterogeneous populations evolve under SFP? How much and under what conditions does the heterogeneity in beliefs affect their long-term behaviors?

Model and Solutions. In this paper, we study the dynamics of SFP in general classes of multi-population network games that allow for heterogeneous beliefs. In a multi-population network game, each vertex of the network represents a population (continuum) of agents, and each edge represents a series of 2-player subgames between two neighboring populations. Note that multi-population network games include all the 2-population games considered in [12] and are representation of subclasses of real-world systems where the graph structure is evident [czechowski2021poincar]. We consider that for a certain population, individual agents form separate beliefs about each neighbor population and observe the mean strategy play of that population. Taking a approach different from stochastic approximation, we define the system state as a probability measure over the space of beliefs, which allows us to precisely examine the impact of heterogeneous beliefs on system evolution. This probability measure changes over time in response to agents’ learning. Thus, the main challenge is to analyze the evolution of the measure, which in general requires the development of new techniques.

As a starting point, we establish a system of partial differential equations (PDEs) to track the evolution of the measure in continuous time limit (Proposition 1). The PDEs that we derive are akin to the continuity equations¹¹1The continuity equation is a PDE that describes the transport phenomena of some quantity (e.g., mass, energy, momentum and other conserved quantities) in a physical system. commonly encountered in physics and do not allow for a general solution. Appealing to moment closure approximation [13], we circumvent the need of solving the PDEs and directly analyze the dynamics of the mean and variance (Proposition 2 and Theorem 1). As one of our key results, we prove that the variance of beliefs always decays quadratically fast with time in all network games (Theorem 1). Put differently, eventually, beliefs will homogenize and the distribution of beliefs will collapse to a single point, regardless of initial distributions of beliefs, 2-player subgames that agents play, and the number of populations and strategies. This result is non-trivial and perhaps somewhat counterintuitive. Afterall, one may find it more natural to expect that the distribution of beliefs would converge to some distribution rather than a single point, as evidenced by recent studies on Q-learning and Cross learning [23, 24, 27].

Technically, the eventual belief homogenization has a significant implication — it informally hints that the asymptotic system state of initially heterogeneous systems are likely to be the same as in homogeneous systems. We show that the fixed point of SFP correspond to Quantual Response Equilibria (QRE)²²2QRE is a game theoretic solution concept under bounded rationality. By QRE, in this paper we refer to their canonical form also referred to as logit equilibria or logit QRE in the literature [14]. in network games for both homogeneous and initially heterogeneous systems (Theorem 2). As our main result, we establish the convergence of SFP to QRE in different classes of games capturing both network competition as well as network coordination, independent of belief initialization. Specifically, for competitive network games, we first prove via a Lyapunov argument that the SFP converges to a unique QRE in homogeneous systems, even when the underlying game has many distinct Nash equilibria (Theorem 3). Then, we show that this convergence result can be carried over to initially heterogeneous systems (Theorem 4), by leveraging that the mean belief dynamics of initially heterogeneous systems is asymptotically autonomous [31] with its limit dynamics being the belief dynamics of a homogeneous system (Lemma 7). For coordination network games, we also prove the convergence to QRE for homogeneous and initially heterogeneous systems, in which the underlying network has star structure (Theorem 5).

On the other hand, the eventual belief homogenization may lead to a misconception that belief heterogeneity has little effect on system evolution. Using an example of 2-population stag hunt games, we show that belief heterogeneity actually plays a crucial role in equilibrium selection, even though it eventually vanishes. As shown in Figure 1, changing the variance of initial beliefs results in different limit behaviors, even when the mean of initial beliefs remains unchanged; in particular, while a small variance leads to the less desirable equilibrium $(H,H)$, a large variance leads to the payoff dominant equilibrium $(S,S)$. Thus, in the case of network coordination, initial belief heterogeneity can help select the highly desirable equilibrium and provides interesting insights to the seminal thorny problem of equilibrium selection [26]. On the contrary, in the case of network competition, we prove (Theorems 3 and 4 on the convergence to a unique QRE in competitive network games) as well as showcase experimentally that the resulting limit behavior is independent of initialization of beliefs, even if the underlying game has many distinct Nash equilibria.

Observe that for an arbitrary agent $k$, its belief $\bm{\upmu}^{i}_{j}(k)$ is in the simplex $\Delta_{j}=\{\bm{\upmu}^{i}_{j}(k)\in\mathbb{R}^{|S_{j}|}|\sum_{s_{j}\in S_{j}}\mu_{js_{j}}^{i}(k)=1,\mu_{js_{j}}^{i}(k)\geq 0,\forall s_{j}\in S_{j}\}$. We assume that the system state is characterized by a Borel probability measure $P$ defined on the state space $\Delta=\prod_{i\in V}\Delta_{i}$. Given $\bm{\upmu}_{i}\in\Delta_{i}$, we write the marginal probability density function as $p(\bm{\upmu}_{i},t)$. Note that $p(\bm{\upmu}_{i},t)$ is the density of agents having the belief $\bm{\upmu}_{i}$ about population $i$ throughout the system. Define $\bm{\upmu}=\{\bm{\upmu}_{i}\}_{i\in V}\in\Delta$. Since agents maintain separate beliefs about different neighbor populations, the joint probability density function $p(\bm{\upmu},t)$ can be factorized, i.e., $p(\bm{\upmu},t)=\prod_{i\in V}p(\bm{\upmu}_{i},t)$. We make the following assumption for the initial marginal density functions.

This assumption is standard and common for a “nice” probability distribution. Under this mild condition, we determine the evolution of the system state $P$ with the following proposition, using the techniques similar to those in [27, 23].

For every marginal density function $p(\bm{\upmu}_{i},t)$, the total mass is always conserved (Corollary 1 of the supplementary); moreover, the mass at the boundary of the simplex $\Delta_{i}$ always remains zero, indicating that agents’ beliefs will never go to extremes (Corollary 2 of the supplementary).

Generalizing the notion of a system state to a distribution over beliefs allows us to address a very specific question — the impact of belief heterogeneity on system evolution. That said, partial differential equations (Equation 8) are notoriously difficult to solve. Here we resort to the evolution of moments based on the evolution of the distribution (Equation 8). In the following proposition, we show that the characterization of belief heterogeneity is important, as the dynamics of the mean system state (or the mean belief dynamics) is indeed affected by belief heterogeneity.

In general, the mean belief dynamics is under the joint effects of the mean, variance, and infinitely many higher moments of the belief distribution. To allow for more conclusive results, we apply the moment closure approximation⁴⁴4Moment closure is a typical approximation method used to estimate moments of population models [13, 15, 32]. To use moment closure, a level is chosen past which all cumulants are set to zero. The conventional choice of the level is $2$, i.e., setting the third and higher cumulants to be zero. and assume the effects of the third and higher moments to be negligible.

Now, just for a moment, suppose that the system beliefs are homogeneous —- the beliefs of every individuals are the same. Hence, the mean belief dynamics are effectively the belief dynamics of individuals. The following proposition follows from Equation 7.

Intuitively, the mean belief dynamics indicates the trend of beliefs in a system, and the variance of beliefs indicates belief heterogeneity. Contrasting Propositions 2 and 3, it is clear that the variance of belief (belief heterogeneity) plays a role in determining the mean belief dynamics (the trend of beliefs) for heterogeneous systems. It is then natural to ask: how does the belief heterogeneity evolve over time? How much does the belief heterogeneity affect the trend of beliefs? Our investigation to these questions reveals an interesting finding — the variance of beliefs asymptotically tends to zero.

Such quadratic decay of the variance stands no matter what 2-player subgames agents play and what initial conditions are. Put differently, the beliefs will eventually homogenize for all population network games. This fact immediately implies the system state in the limit.

Note that while the choice distributions will homogenize within each population, they are not necessarily the same across different populations. This is because the strategy choice of each population is in response to its own set of neighbor populations (which are generally different).

The finding on belief homogenization is non-trivial and also technically important. One implication is that the fixed points of systems with initially heterogeneous beliefs are the same as in systems with homogeneous beliefs. Thus, it follows from the belief dynamics for homogeneous systems (Proposition 3) that the fixed points of systems have the following property.

Note that the above theorem applies for all population network games.

We study the convergence of SFP to the QRE under the both cases of network competition and network coordination. Due to space limits, in the following, we mainly focus on network competition and present only the main result on network coordination.

In this section, we complement our theory and present an empirical study of SFP in a two-population coordination (stag hunt) game and a five-population zero-sum (asymmetric matching pennies) game. Importantly, these two games both have multiple Nash equilibria, which naturally raises the problem of equilibrium selection.

We study a heterogeneous beliefs model of SFP in network games. Representing the system state with a distribution over beliefs, we prove that beliefs eventually become homogeneous in all network games. We establish the convergence of SFP to Quantal Response Equilibria in general competitive network games as well as coordination network games with star structure. We experimentally show that although the initial belief heterogeneity vanishes in the limit, it plays a crucial role in equilibrium selection and helps select highly desirable equilibria.