Learning the Structure and Parameters of Large-Population Graphical Games from Behavioral Data

Game theory has become a central tool for modeling multi-agent systems in AI. Non-cooperative game theory has been considered as the appropriate mathematical framework in which to formally study strategic behavior in multi-agent scenarios.¹¹1See, e.g., the survey of Shoham [2008] and the books of Nisan et al. [2007] and Shoham and Leyton-Brown [2009] for more information. The core solution concept of Nash equilibrium (NE) [Nash, 1951] serves a descriptive role of the stable outcome of the overall behavior of systems involving self-interested individuals interacting strategically with each other in distributed settings for which no direct global control is possible. NE is also often used in predictive roles as the basis for what one might call causal strategic inference, i.e., inferring the results of causal interventions on stable actions/behavior/outcomes in strategic settings (See, e.g., Ballester et al. 2004, 2006, Heal and Kunreuther 2003, 2006, 2007, Kunreuther and Michel-Kerjan 2007, Ortiz and Kearns 2003, Kearns 2005, Irfan and Ortiz 2014, and the references therein). Needless to say, the computation and analysis of NE in games is of significant interest to the computational game-theory community within AI.

The introduction of compact representations to game theory over the last decade have extended computational/algorithmic game theory’s potential for large-scale, practical applications often encountered in the real-world. For the most part, such game model representations are analogous to probabilistic graphical models widely used in machine learning and AI.²²2The fundamental property such compact representation of games exploit is that of conditional independence: each player’s payoff function values are determined by the actions of the player and those of the player’s neighbors only, and thus are conditionally (payoff) independent of the actions of the non-neighboring players, given the action of the neighboring players. Introduced within the AI community about a decade ago, graphical games [Kearns et al., 2001] constitute an example of one of the first and arguably one of the most influential graphical models for game theory.³³3Other game-theoretic graphical models include game networks [La Mura, 2000], multi-agent influence diagrams (MAIDs) [Koller and Milch, 2003], and action-graph games [Jiang and Leyton-Brown, 2008].

There has been considerable progress on problems of computing classical equilibrium solution concepts such as NE and correlated equilibria (CE) [Aumann, 1974] in graphical games (see, e.g., Kearns et al. 2001, Vickrey and Koller 2002, Ortiz and Kearns 2003, Blum et al. 2006, Kakade et al. 2003, Papadimitriou and Roughgarden 2008, Jiang and Leyton-Brown 2011 and the references therein). Indeed, graphical games played a prominent role in establishing the computational complexity of computing NE in general normal-form games (see, e.g., Daskalakis et al. 2009 and the references therein).

An example of a recent computational application of non-cooperative game-theoretic graphical modeling and causal strategic inference (CSI) that motivates the current paper is the work of Irfan and Ortiz [2014]. They proposed a new approach to the study of influence and the identification of the “most influential” individuals (or nodes) in large (social) networks. Their approach is strictly game-theoretic in the sense that it relies on non-cooperative game theory and the central concept of pure-strategy Nash equilibria (PSNE)⁴⁴4In this paper, because we concern ourselves primarily with PSNE, whenever we use the term “equilibrium” or “equilibria” without qualification, we mean PSNE. as an approximate predictor of stable behavior in strategic settings, and, unlike other models of behavior in mathematical sociology,⁵⁵5Some of these models have recently gained interest and have been studied within computer science, specially those related to diffusion or contagion processes (see, e.g., Granovetter 1978, Morris 2000, Domingos and Richardson 2001, Domingos 2005, Even-Dar and Shapira 2007). it is not interested and thus avoids explicit modeling of the complex dynamics by which such stable outcomes could have arisen or could be achieved. Instead, it concerns itself with the “bottom-line” end-state stable outcomes (or steady state behavior). Hence, the proposed approach provides an alternative to models based on the diffusion of behavior through a social network (See Kleinberg 2007 for an introduction and discussion targeted to computer scientists, and further references).

The underlying assumption for most work in computational game theory that deals with algorithms for computing equilibrium concepts is that the games under consideration are already available, or have been “hand-designed” by the analyst. While this may be possible for systems involving a handful of players, it is in general impossible in systems with at least tens of agent entities, if not more, as we are interested in this paper.⁶⁶6Of course, modeling and hand-crafting games for systems with many agents may be possible if the system has particular structure one could exploit. To give an example, this would be analogous to how one can exploit the probabilistic structure of HMMs to deal with long stochastic processes in a representationally succinct and computationally tractable way. Yet, we believe it is fair to say that such systems are largely/likely the exception in real-world settings in practice. For instance, in their paper, Irfan and Ortiz [2014] propose a class of games, called influence games. In particular, they concentrate on linear influence games (LIGs), and, as briefly mentioned above, study a variety of computational problems resulting from their approach, assuming such games are given as input.

Research in computational game theory has paid relatively little attention to the problem of learning (both the structure and parameters of) graphical games from data. Addressing this problem is essential to the development, potential use and success of game-theoretic models in practical applications. Indeed, we are beginning to see an increase in the availability of data collected from processes that are the result of deliberate actions of agents in complex system. A lot of this data results from the interaction of a large number of individuals, being not only people (i.e., individual human decision-makers), but also companies, governments, groups or engineered autonomous systems (e.g., autonomous trading agents), for which any form of global control is usually weak. The Internet is currently a major source of such data, and the smart grid, with its trumpeted ability to allow individual customers to install autonomous control devices and systems for electricity demand, will likely be another one in the near future.

In this paper, we investigate in considerable technical depth the problem of learning LIGs from strictly behavioral data: We do not assume the availability of utility, payoff or cost information in the data; the problem is precisely to infer that information from just the joint behavior collected in the data, up to the degree needed to explain the joint behavior itself. We expect that, in most cases, the parameters quantifying a utility function or best-response condition are unavailable and hard to determine in real-world settings. The availability of data resulting from the observation of an individual public behavior is arguably a weaker assumption than the availability of individual utility observations, which are often private. In addition, we do not assume prior knowledge of the conditional payoff/utility independence structure as represented by the game graph.

Motivated by the work of Irfan and Ortiz [2014] on a strictly non-cooperative game-theoretic approach to influence and strategic behavior in networks, we present a formal framework and design algorithms for learning the structure and parameters of LIGs with a large number of players. We concentrate on data about what one might call “the bottom line:” i.e., data about“end-states”, “steady-states” or final behavior as represented by possibly noisy samples of joint actions/pure-strategies from stable outcomes, which we assume come from a hidden underlying game. Thus, we do not use, consider or assume available any temporal data about the detailed behavioral dynamics. In fact, the data we consider does not contain the dynamics that might have possibly led to the potentially stable joint-action outcome! Since scalability is one of our main goals, we aim to propose methods that are polynomial-time in the number of players.

Given that LIGs belong to the class of 2-action graphical games [Kearns et al., 2001] with parametric payoff functions, we first needed to deal with the relative dearth of work on the broader problem of learning general graphical games from purely behavioral data. Hence, in addressing this problem, while inspired by the computational approach of Irfan and Ortiz [2014], the learning problem formulation we propose is in principle applicable to arbitrary games (although, again, the emphasis is on the PSNE of such games). In particular, we introduce a simple statistical generative mixture model, built “on top of” the game-theoretic model, with the only objective being to capture noise in the data. Despite the simplicity of the generative model, we are able to learn games from U.S. congressional voting records, which we use as a source of real-world behavioral data, that, as we will illustrate, seem to capture interesting, non-trivial aspects of the U.S. congress. While such models learned from real-world data are impossible to validate, we argue that there exists a considerable amount of anecdotal evidence for such aspects as captured by the models we learned. Figure 1 provides a brief illustration. (Should there be further need for clarification as to the why we present this figure, please see Footnote 14.)

As a final remark, given that LIGs constitute a non-trivial sub-class of parametric graphical games, we view our work as a step in the direction of addressing the broader problem of learning general graphical games with a large number of players from strictly behavioral data. We also hope our work helps to continue to bring and increase attention from the machine-learning community to the problem of inferring games from behavioral data (in which we attempt to learn a game that would “rationalize” players’ observed behavior).⁷⁷7This is a type of problem arising from game theory and economics that is different from the problem of learning in games (in which the focus is the study of how individual players learn to play a game by a sequence of repeated interactions), a more matured and perhaps better known problem within machine learning (see, e.g., Fudenberg and Levine 1999).

We provide a brief summary overview of previous work on learning games here, and delay discussion of the work presented below until after we formally present our model; this will provide better context and make “comparing and contrasting” easier for those interested, without affecting expert readers who may want to get to the technical aspects of the paper without much delay.

Table 1 constitutes our best attempt at a simple visualization to fairly present the differences and similarities of previous approaches to modeling behavioral data within the computational game-theory community in AI.

The research interest of previous work varies in what they intend to capture in terms of different aspects of behavior (e.g., dynamics, probabilistic vs. strategic) or simply different settings/domains (i.e., modeling “real human behavior,” knowledge of achieved payoff or utility, etc.).

With the exception of Ziebart et al. [2010], Waugh et al. [2011], Kearns and Wortman [2008], previous methods assume that the actions as well as corresponding payoffs (or noisy samples from the true payoff function) are observed in the data. Our setting largely differs from Ziebart et al. [2010], Kearns and Wortman [2008] because of their focus on system dynamics, in which future behavior is predicted from a sequence of past behavior. Kearns and Wortman [2008] proposed a learning-theory framework to model collective behavior based on stochastic models.

Our problem is clearly different from methods in quantal response models [McKelvey and Palfrey, 1995, Wright and Leyton-Brown, 2010, 2012] and graphical multiagent models (GMMs) [Duong et al., 2008, 2010] that assume known structure and observed payoffs. Duong et al. [2012] learns the structure of games that are not graphical, i.e., the payoff depends on all other players. Their approach also assumes observed payoff and consider a dynamic consensus scenario, where agents on a network attempt to reach a unanimous vote. In analogy to voting, we do not assume the availability of the dynamics (i.e., the previous actions) that led to the final vote. They also use fixed information on the conditioning sets of neighbors during their search for graph structure. We also note that the work of Vorobeychik et al. [2007], Gao and Pfeffer [2010], Ziebart et al. [2010] present experimental validation mostly for 2 players only, 7 players in Waugh et al. [2011] and up to 13 players in Duong et al. [2009].

In several cases in previous work, researchers define probabilistic models using knowledge of the payoff functions explicitly (i.e., a Gibbs distribution with potentials that are functions of the players payoffs, regrets, etc.) to model joint behavior (i.e., joint pure-strategies); see, e.g., Duong et al. [2008, 2010, 2012], and to some degree also Wright and Leyton-Brown [2010, 2012]. It should be clear to the reader that this is not the same as our generative model, which is defined directly on the PSNE (or stable outcomes) of the game, which the players’ payoffs determine only indirectly.

In contrast, in this paper, we assume that the joint actions are the only observable information and that both the game graph structure and payoff functions are unknown, unobserved and unavailable. We present the first techniques for learning the structure and parameters of a non-trivial class of large-population graphical games from joint actions only. Furthermore, we present experimental validation in games of up to $100$ players. Our convex loss minimization approach could potentially be applied to larger problems since it has polynomial time complexity in the number of players.

In classical game-theory (see, e.g. Fudenberg and Tirole 1991 for a textbook introduction), a normal-form game is defined by a set of players $V$ (e.g., we can let $V=\{1,\dots,n\}$ if there are $n$ players), and for each player $i$, a set of actions, or pure-strategies $A_{i}$, and a payoff function $u_{i}:\times_{j\in V}A_{j}\to\mathbb{R}$ mapping the joint actions of all the players, given by the Cartesian product $\mathcal{A}\equiv\times_{j\in V}A_{j}$, to a real number. In non-cooperative game theory we assume players are greedy, rational and act independently, by which we mean that each player $i$ always want to maximize their own utility, subject to the actions selected by others, irrespective of how the optimal action chosen help or hurt others.

A core solution concept in non-cooperative game theory is that of an Nash equilibrium. A joint action $\mathbf{x}^{*}\in\mathcal{A}$ is a pure-strategy Nash equilibrium (PSNE) of a non-cooperative game if, for each player $i$, $x^{*}_{i}\in{\arg\max}_{x_{i}\in A_{i}}u_{i}(x_{i},\mathbf{x}^{*}_{-i})$; that is, $\mathbf{x}^{*}$ constitutes a mutual best-response, no player $i$ has any incentive to unilaterally deviate from the prescribed action $x^{*}_{i}$, given the joint action of the other players $\mathbf{x}^{*}_{-i}\in\times_{j\in V-\{i\}}A_{j}$ in the equilibrium. In what follows, we denote a game by ${\mathcal{G}}$, and the set of all pure-strategy Nash equilibria of ${\mathcal{G}}$ by¹²¹²12Because this paper concerns mostly PSNE, we denote the set of PSNE of game ${\mathcal{G}}$ as ${\mathcal{NE}}({\mathcal{G}})$ to simplify notation.

A (directed) graphical game is a game-theoretic graphical model [Kearns et al., 2001]. It provides a succinct representation of normal-form games. In a graphical game, we have a (directed) graph $G=(V,E)$ in which each node in $V$ corresponds to a player in the game. The interpretation of the edges/arcs $E$ of $G$ is that the payoff function of player $i$ is only a function of the set of parents/neighbors $\mathcal{N}_{i}\equiv\{j\mid(i,j)\in E\}$ in $G$ (i.e., the set of players corresponding to nodes that point to the node corresponding to player $i$ in the graph). In the context of a graphical game, we refer to the $u_{i}$’s as the local payoff functions/matrices.

Linear influence games (LIGs) [Irfan and Ortiz, 2014] are a sub-class of $2$-action graphical games with parametric payoff functions. For LIGs, we assume that we are given a matrix of influence weights $\mathbf{W}\in\mathbb{R}^{n\times n}$, with $\mathbf{diag}(\mathbf{W})=\mathbf{0}$, and a threshold vector $\mathbf{b}\in\mathbb{R}^{n}$. For each player $i$, we define the influence function $f_{i}(\mathbf{x}_{-i})\equiv\sum_{j\in\mathcal{N}_{i}}w_{ij}x_{j}-b_{i}={\mathbf{w}_{{i},{{-i}}}}^{\rm T}\mathbf{x}_{-i}-b_{i}$ and the payoff function $u_{i}(\mathbf{x})\equiv x_{i}f_{i}(\mathbf{x}_{-i})$. We further assume binary actions: $A_{i}\equiv\{-1,+1\}$ for all $i$. The best response $x_{i}^{*}$ of player $i$ to the joint action $\mathbf{x}_{-i}$ of the other players is defined as

Intuitively, for any other player $j$, we can think of $w_{ij}\in\mathbb{R}$ as a weight parameter quantifying the “influence factor” that $j$ has on $i$, and we can think of $b_{i}\in\mathbb{R}$ as a threshold parameter quantifying the level of “tolerance” that player $i$ has for playing $-1$.¹³¹³13As we formally/mathematically define here, LIGs are $2$-action graphical games with linear-quadratic payoff functions. Given our main interest in this paper on the PSNE solution concept, for the most part, we simply view LIGs as compact representations of the PSNE of graphical games that the algorithms of Irfan and Ortiz [2014] use for CSI. (This is in contrast to a perhaps more natural, “intuitive” but still informal description/interpretation one may provide for instructive/pedagogical purposes based on “direct influences,” as we do here.) This view of LIGs is analogous to the modern, predominant view of Bayesian networks as compact representations of joint probability distributions that are also very useful for modeling uncertainty in complex systems and practical for probabilistic inference [Koller and Friedman, 2009]. (And also analogous is the “intuitive” descriptions/interpretations of BN structures, used for instructive/pedagogical purposes, based on “causal” interactions between the random variables Koller and Friedman, 2009.)

As discussed in Irfan and Ortiz [2014], LIGs are also a sub-class of polymatrix games [Janovskaja, 1968]. Furthermore, in the special case of $\mathbf{b}=\mathbf{0}$ and symmetric $\mathbf{W}$, a LIG becomes a party-affiliation game [Fabrikant et al., 2004].

In this paper, the use of the verb “influence” strictly refers to influences defined by the model.

Figure 1 provides a preview illustration of the application of our approach to congressional voting.¹⁴¹⁴14We present this game graph because many people express interest in “seeing” the type of games we learn on this particular data set. The reader should please understand that by presenting this graph we are definitely not implying or arguing that we can identify the ground-truth graph of “direct influences.” (We say this even in the very unlikely event that the “ground-truth model” be an LIG that faithfully capture the “true direct influences” in this U.S. Congress, something arguably no model could ever do.) As we show later in Section 4.2, LIGs are not identifiable with respect to their local compact parametric representation encoding the game graph through their weights and biases, but only with respect to their PSNE, which are joint actions capturing a global property of a game that we really care about for CSI. Certainly, we could never validate the model parameters of an LIG at the local, microscopic level of “direct influences” using only the type of observational data we used to learn the model depicted by the graph in the figure. For that, we would need help from domain experts to design controlled experiments that would yield the right type of data for proper/rigorous scientific validation.

Our goal is to learn the structure and parameters of an LIG from observed joint actions only (i.e., without any payoff data/information).¹⁵¹⁵15In principle, the learning framework itself is technically immediately/easily applicable to any class of simultaneous/one-shot games. Generalizing the algorithms and other theoretical results (e.g., on generalization error) while maintaining the tractability in sample complexity and computation may require significant effort. Yet, for simplicity, most of the presentation in this section is actually in terms of general 2-action games. While we make sporadic references to LIGs throughout the section, it is not until we reach the end of the section that we present and discuss the particular instantiation of our proposed framework with LIGs.

Our main performance measure will be average log-likelihood (although later we will be considering misclassification-type error measures in the context of simultaneous-classification, as a result of an approximation of the average log-likelihood). Our emphasis on a PSNE-based statistical model for the behavioral data results from the approach to causal strategic inference taken by Irfan and Ortiz [2014], which is strongly founded on PSNE.¹⁶¹⁶16The possibility that PSNE may not exist in some LIGs does not present a significant problem in our case because we are learning the game, and can require that the LIG output has at least one PSNE. Indeed, in our approach, games with no PSNE achieve the lowest possible likelihood within our generative model of the data; said differently, games with PSNE have higher likelihoods than those that do not have any PSNE.

Note that our problem is unsupervised, i.e., we do not know a priori which joint actions are PSNE and which ones are not. If our only goal were to find a game ${\mathcal{G}}$ in which all the given observed data is an equilibrium, then any “dummy” game, such as the “dummy” LIG ${\mathcal{G}}=(\mathbf{W},\mathbf{b}),\mathbf{W}=\mathbf{0},\mathbf{b}=\mathbf{0}$, would be an optimal solution because $|{\mathcal{NE}}({\mathcal{G}})|=2^{n}$.¹⁷¹⁷17Ng and Russell [2000] made a similar observation in the context of single-agent inverse reinforcement learning (IRL). In this section, we present a probabilistic formulation that allows finding games that maximize the empirical proportion of equilibria in the data while keeping the true proportion of equilibria as low as possible. Furthermore, we show that trivial games such as LIGs with $\mathbf{W}=\mathbf{0},\mathbf{b}=\mathbf{0}$, obtain the lowest log-likelihood.

In this section, we show a generalization bound for the maximum likelihood problem as well as an upper bound of the VC-dimension of LIGs. Our objective is to establish that with probability at least $1-\delta$, for some confidence parameter $0<\delta<1$, the maximum likelihood estimate is within $\epsilon>0$ of the optimal parameters, in terms of achievable expected log-likelihood.

Note that our hypothesis space $\Upsilon$ as stated in Definition 7 includes a continuous parameter $q$ that could potentially have infinite VC-dimension. The following lemma will allow us later to prove that uniform convergence for the extreme values of $q$ implies uniform convergence for all $q$ in the domain.

In the remaining of this section, denote by $d(\mathcal{H})\equiv\left|\cup_{{\mathcal{G}}\in\mathcal{H}}\{{\mathcal{NE}}({\mathcal{G}})\}\right|$ the number of all possible PSNE sets induced by games in $\mathcal{H}$, the class of games of interest.

The following theorem shows that the expected log-likelihood of the maximum likelihood estimate $\widehat{\theta}$ converges in probability to that of the optimal $\bar{\theta}=(\bar{{\mathcal{G}}},\bar{q})$, as the data size $m$ increases.

The following theorem establishes the complexity of the class of LIGs, which implies that the term $\log d(\mathcal{H})$ of the generalization bound in Theorem 12 is only polynomial in the number of players $n$.