Can we infer player behavior tendencies from a player’s decision-making data? Integrating Theory of Mind to Player Modeling

We make the following assumptions to abstract a player’s decision making process. First, we assume that a player perfectly knows the game mechanism. Thus, there is no uncertainty stemming from a player’s lack of knowledge about the game. This is equivalent to assuming that a player is an expert. Second, we assume that a player has bounded rationality and limited cognitive resources. This implies that a player does not think multiple steps ahead neither can they realize the end state of the map. Thus, a player’s decision making is modeled myopically such that only the situational state of the game at any time step influences their subsequent decisions. Third, we assume a player can view the entire game map. This is equivalent to having a “mini-map” feature in a game that enables players to have a birds eye view of the map. Fourth, we do not model multiplayer interactions and consider individual player’s decision making and cognitive behaviors.

We abstract a player’s sequential decision making process as follows. We consider that a player makes two decisions $D=\left(X,Y\right)$ in a sequential manner. We assume that the decisions have discrete and finite outcomes such that the outcomes of decision $X$ influence the outcomes of decision $Y$. Such an abstraction enables us to consider sequential decision making in a parsimonious manner (minimum number of decisions required to create a sequence of decisions). We also assume to have the game state data $\mathcal{G}$ which holds relevant information about the state of the game when a player made the corresponding decisions. Thus, for a random number of samples $N$ of a player’s decision making data across several sessions of their gameplay, we assume to have a set of player’s sequential decisions $D_{1:N}=\left(X_{1:N},Y_{1:N}\right)$ and the corresponding game states $\mathcal{G}_{1:N}$ .

After abstracting a player’s decision-making process, we model how players make the specific decisions. To do so, we leverage decision theory to model decisions as functions of attributes or features of observed data within the game weighted differently by each individual player. Feature functions enable deterministic modeling that leverage game states to model decision attributes. Decision outcomes are modeled probabilistically using likelihood functions, with function parameters such as an individual’s behavior tendency $\boldsymbol{\theta}$, which adds stochasticity in the predictions. Our modeling approach acknowledges that players make decisions subjectively based on their behavior tendency or individual preferences. Moreover, the assumption of probabilistic decisions assumes the limited cognitive ability of a player to make accurate decisions even though their judgments may be aligned with rational judgments.

Formally, we refer to a mapping between the observed game data to some situational factor as a feature function. A feature function (or simply feature) incorporates the observed game state $\mathcal{G}_{n}$ for a data sample $n$ into the decision models. Given that multiple situational factors may influence decisions, a decision strategy is specified in terms of a weighted sum of multiple independent features. Moreover, a threshold value is associated with each feature to model an individual’s mental activation to the subjective strength of a particular feature.

Mathematically, we characterize a decision strategy for a decision $X_{n}$ in a data sample $n$ with $O$ discrete outcomes $\{x_{n,1},\ldots,x_{n,O}\}$ using $R$ independent attributes or features denoted by $g_{1,x}(\mathcal{G}_{n}),\ldots,g_{R,x}(\mathcal{G}_{n})$. The values of the features can be dependent on the decision alternative in consideration. Then, we model the stochastic decision process as follows:

$X_{n}=\begin{cases}x_{n,o},&\text{with probability}\;\operatorname{softmax}_{x_{o}}\left(\sum_{r=1}^{R}w_{r}\left(g_{r,x}(\mathcal{G}_{n})-\delta_{r}\right)\right)\end{cases}$

(1)

and, the outcome probability is given by,

$\operatorname{softmax}_{x_{o}}\left(\sum_{r=1}^{R}w_{r}\left(g_{r,x}(\mathcal{G}_{n})-\delta_{r}\right)\right)=\dfrac{\exp\left(\sum_{r=1}^{R}w_{r}\left(g_{r,x_{o}}(\mathcal{G}_{n})-\delta_{r}\right)\right)}{\sum\limits_{\forall X}\exp\left(\sum_{r=1}^{R}w_{r}\left(g_{r,x}(\mathcal{G}_{n})-\delta_{r}\right)\right)}.$

(2)

where, $X_{n}=x_{n,o}$ is the observation that an individual chose an alternative $x_{n,o}$ for a decision $X$ for a data sample $n$, $\boldsymbol{\theta}=\{w_{1:R},\delta_{1:R}\}$ are player-specific cognitive parameters modeled as the feature weight and feature threshold parameters. The weight parameter $w_{r}$ can be positive or negative depending on whether an increase in $g_{r,x}(\mathcal{G}_{n})$, respectively, increases or decreases the probability of an outcome. The threshold parameters imply that the weighted sum of the situational factors as activated by a player determines their decision strategy.

The decision strategy for a decision $Y_{n}$ can also be defined using a similar approach as discussed for the decision $X_{n}$ in Equation 1. The only difference in consideration of the outcome probabilities of $Y_{n}$ will be that they are conditioned on the outcomes of $X_{n}$.

Here, we mention a general strategy for inverse Bayesian inference. However, we note that it’s important to consider game-specific causal models that represent the influence of situational factors on a player’s decision making. Given decision data $D_{1:N}$, a prior over $\theta$, $p(\theta)$, a prior over game states $\mathcal{G}_{1:N}$, and the outcome probabilities for decision $X_{n}$, $p(X_{n}|\theta,\mathcal{G}_{1:N})$ and the outcome probabilities for decision $Y_{n}$, $p(Y_{n}|\theta,X_{n},\mathcal{G}_{1:N})$, inference over posterior of $\theta$ is given by Bayes’ rule:

$p(\boldsymbol{\theta}|D_{1:N},\mathcal{G}_{1:N})\propto p(X_{1:N}|\boldsymbol{\theta},\mathcal{G}_{1:N})p(Y_{1:N}|\boldsymbol{\theta},X_{1:N},\mathcal{G}_{1:N})p(\boldsymbol{\theta})p(\mathcal{G}_{1:N}).$

(3)

We illustrate this inference in further detail in our use case in Section 4.2.

For BoomTown, we consider players’ sequential moving decisions. These include the decision to move or not $M_{n}$ and which direction to move $D_{n}$. At any point in the game, a player decides whether they want to move in the map or not. If they decide to move, then they need to decide which direction to move. If they choose to stay, they decide to use items in the game to mine gold. We do not model the use items decision. We only model the moving decisions to illustrate the sequential nature of the decision making process.

We note that the first decision to move or not has two outcomes. We consider two significant situational features in the game for a player’s decision to move: the rock tile and the gold nugget tile. Thus, $R=2$ for our move decision model. For other games, model developers would need to investigate the situational factors that influence players’ gameplay. The first situational factor is Gold Around, $GA_{n}=g_{1}(\mathcal{G}_{n})$ which represents how many gold nuggets are around a player within some map region for a data sample $n$. The second situational factor is Rock Around, $RA_{n}=g_{2}(\mathcal{G}_{n})$ which represents how many rocks are around a player within some map region for a data sample $n$. The calculation of these features is independent of the alternative to move or not thus we drop the term $x$ in $g_{r,x}(.)$ as discussed in Section 3.2. We model the stochastic moving process $M_{n}$ for BoomTown as follows:

$M_{n}=\begin{cases}1,&\text{with probability}\;\operatorname{sigm}\left(\sum_{r=1}^{2}w_{r}\left(g_{r}(\mathcal{G}_{n})-\delta_{r}\right)\right)\\ 0,&\text{otherwise},\end{cases}$

(4)

and, the moving probability is given by,

$$\leavevmode\resizebox{433.62pt}{}{$p(M_{n}=1|\mathcal{G}_{n})=\dfrac{1}{1+\exp{\left(w_{1}(GA_{n}-\delta_{1})-w_{2}(RA_{n}-\delta_{2})\right)}}$},$$

(5)

where, without loss of generality, the weight parameter $w_{1:2}$ are considered to be positive while the negative or positive influence of each of the features on the moving probability is intuitively coded. Thus, increase in gold around the player decreases the probability to move while increase in rock around the player increases their probability to move. Moreover, the sigmoid function $\operatorname{sigm}()$ is a special case of the $\operatorname{softmax}()$ function discussed in Equation 1 for a decision with two outcomes.

The second decision of where to move $D_{n}$ is considered to have five alternatives, namely, North ($d_{n,1}$), South ($d_{n,2}$), East ($d_{n,3}$), West ($d_{n,4}$), and no direction ($d_{n,5}$) . Thus, $D_{n}=\{d_{n,1:5}\}$. Moreover, we consider five situational factors such that $R=5$. The situational factors are dependent on a direction alternative $d_{n,i}$ and include Gold Around, $GA_{n,i}=g_{1,i}(\mathcal{G}_{n})$, and Rock Around, $RA_{n,i}=g_{2,i}(\mathcal{G}_{n})$ which represents how many gold nuggets and rocks are around a player in a direction $d_{n,i}$ for a data sample $n$. There are three additional situational factors, namely, $GD_{n,i}=g_{3,i}(\mathcal{G}_{n})$ the average distance of the gold around, $RD_{n,i}=g_{4,i}(\mathcal{S}_{t})$ the average distance of the rock around the player, and $OA_{n,i}=g_{5,i}(\mathcal{G}_{n})$ the obstacle tiles around player position in direction $d_{n,i}$.

The player moves in no direction $D_{n}=d_{n,5}$ given the decision to move $M_{n}=0$. Thus, the stochastic decision of which direction $D_{n}$ to move, given the decision to move $M_{n}=1$, is modeled as follows:

$D_{n}=\begin{cases}d_{n,1},&\text{with probability}\;\operatorname{softmax}_{d_{1}}\left(w_{1:R},g_{r,d}(\mathcal{G}_{n})\right)\\ d_{n,2},&\text{with probability}\;\operatorname{softmax}_{d_{2}}\left(w_{1:R},g_{r,d}(\mathcal{G}_{n})\right)\\ d_{n,3},&\text{with probability}\;\operatorname{softmax}_{d_{3}}\left(w_{1:R},g_{r,d}(\mathcal{G}_{n})\right)\\ d_{n,4},&\text{with probability}\;\operatorname{softmax}_{d_{4}}\left(w_{1:R},g_{r,d}(\mathcal{G}_{n})\right)\\ \end{cases}$

(6)

where, the probability to move in a direction is given by,

$p(D_{n}=d|M_{n}=1,\mathcal{G}_{n},\boldsymbol{\theta})=\operatorname{softmax}_{d_{i}}\left(w_{1:R},g_{r,d}(\mathcal{G}_{n})\right),$

(7)

such that,

$\operatorname{softmax}_{d_{i}}\left(.\right)=\dfrac{\mathbbm{1}_{0}\left(g_{5,d}(\mathcal{G}_{n})w_{5}\right)\exp\left(\sum_{r=1}^{2}\dfrac{w_{r}}{w_{r+2}}\dfrac{g_{r,d_{i}}(\mathcal{G}_{n})}{g_{r+2,d_{i}}(\mathcal{G}_{n})}\right)}{\sum\limits_{d=1}^{d=4}\mathbbm{1}_{0}\left(g_{5,d}(\mathcal{G}_{n})w_{5}\right)\exp\left(\sum_{r=1}^{2}\dfrac{w_{r}}{w_{r+2}}\dfrac{g_{r,d}(\mathcal{G}_{n})}{g_{r+2,d}(\mathcal{G}_{n})}\right)}.$

(8)

where, $D_{n}=d$ is the observation that an individual moves in direction $d$ on the map for the $n^{\text{th}}$ sample, $\boldsymbol{\theta}=\{w_{1:R}\}$ are player-specific cognitive parameters modeled as feature weights for $R=5$. The weight parameter $w_{r}$ can be positive or negative depending on whether an increase in $g_{r,i}(\mathcal{G}_{n})$, respectively, increases or decreases the probability of moving. We consider the average distance of the gold and rock in a particular direction to be inversely proportional to an individual’s utility to move in that direction. Moreover, the ratio of the total gold amount to the average gold distance in a particular direction signifies that an individual player would balance exploration of large gold clusters further away with the exploitation of smaller clusters closer to them. Similarly for rock, a player would be averse to large rock clusters in vicinity and prefer less rock dense areas. The feature $g_{5,d}(\mathcal{G}_{n})$ indicates if there is an obstacle in the cell next to a player in direction $d$. Thus, the indicator function $\mathbbm{1}_{0}\left(g_{5,d}(\mathcal{G}_{n})w_{5}\right)$ only avails the directions where a player can move. Consequently, we drop $w_{5}$ as obstacles are a part of game mechanics that affects each player in the same way. Moreover, we set the threshold parameters $\delta_{r}=0$ to model where to move. This is because the player has already decided to move and the minimum threshold required to move in any direction is a positive value of the features. Also, it’s the relative evaluation of the gold and rock amount to their average distances which is of importance to a player’s direction decision. We note that we deliberately wrote Equation 8 in a different form than Equation 2 to highlight that game-specific considerations will influence how model developers choose to represent decision models towards better explainability. Figure 2 illustrates the behavior tendency parameters $\boldsymbol{\theta}$ as utilized in BoomTown.

In this section, we discuss how to infer an individual’s modeled cognitive parameters, that is, their $\boldsymbol{\theta}$ given $N$ samples of the game state $\textbf{g}_{1:N}$ and the decision data history $h_{1:N}=\{m_{1:N},d_{1:N}\}$ which includes the player’s data for their decisions to move $m_{n}$ and where to move $d_{n}$ for each sample $n$.

We proceeded in a Bayesian way which required the specification of a prior $p(\boldsymbol{\theta})$ for $\boldsymbol{\theta}$, a prior $p(\textbf{g}_{1:N})$ for game state, a likelihood $p(h_{1:N}|\boldsymbol{\theta})$ for decisions to move $m_{1:N}$ and where to move $d_{1:N}$ given $\boldsymbol{\theta}$. The posterior state of knowledge about $\boldsymbol{\theta}$ is simply given by Bayes’ rule:

$$p(\boldsymbol{\theta}|h_{1:N},\textbf{g}_{1:N})\propto p(h_{1:N}|\boldsymbol{\theta},\textbf{g}_{1:N})p(\boldsymbol{\theta})p(\textbf{g}_{1:N}),$$

(9)

and we characterized it approximately via sampling. We now describe each of these steps in detail.

We associate behavior tendency with the vector of parameters $\boldsymbol{\theta}=\{w_{1:R},\delta_{1:R}\}$ defined in Section 4.1. We describe our prior state of knowledge about $\boldsymbol{\theta}$ by assigning a probability density function such that it becomes a random vector modeling our epistemic uncertainty about the actual cognition of the individual. Having no reason to believe otherwise, we assume that all components of an individual’s behavior tendency are a priori independent, i.e., the prior probability density (PDF) factorizes as:

$$p(\boldsymbol{\theta})=\prod_{r=1}^{r=4}p(w_{r})\prod_{r=1}^{r=2}p(\delta_{r}),$$

(10)

where, $p(w_{r})$ is assigned an uninformative Jeffrey’s prior, i.e., $p(w_{r})\propto\frac{1}{w_{r}}$, and

$$\begin{array}[]{ccc}\delta_{1}&\sim&\mathcal{N}(50,25),\\ \delta_{2}&\sim&\mathcal{N}(50,25).\\ \end{array}$$

(11)

The mean and standard deviation of the normal distribution for the threshold priors were chosen based on the game ranges for rock and gold values.

The game state $\textbf{g}_{1:N}$ is a vector of features sampled randomly for a sample $N$ and is thus assumed to have a uniform distribution. Such an assumption enables us to circumvent the problem of modeling the game mechanics where player actions influence the game states. Thus, we note that $N$ samples of game data are equivalent to sampling decision data $m_{1:N}$ and $d_{1:N}$ of a player from $N$ randomly generated map scenarios. In general, derivation of game specific prior probabilities for game states will require understanding of the game mechanics that govern the initialization of game states for a game map.

The likelihood $p(h_{1:N}|\boldsymbol{\theta},\textbf{g}_{1:N})$ is calculated conditioned on $\boldsymbol{\theta}$ and $\textbf{g}_{1:N}$. We have:

$$p(h_{1:N}|\boldsymbol{\theta},\textbf{g}_{1:N})=\prod_{q=1}^{N}p(h_{q}|\boldsymbol{\theta},\textbf{g}_{q}),$$

(12)

given the independent sampling assumption of our model For each product term, we have:

$$p(h_{q}|\boldsymbol{\theta},\textbf{g}_{q})=p(m_{q}|\textbf{g}_{q},\boldsymbol{\theta})p(d_{q}|m_{q},\textbf{g}_{q},\boldsymbol{\theta}).$$

(13)

The first term in Equation 13 is:

$\begin{array}[]{cc}p(m_{q}|\textbf{g}_{q},\theta)=&\left[\operatorname{sigm}\left(\sum_{r=1}^{R}w_{r}\left(g_{r}(\textbf{g}_{q})-\delta_{r}\right)\right)\right]^{m_{q}}\\ &\left[1-\operatorname{sigm}\left(\sum_{r=1}^{R}w_{r}\left(g_{r}(\textbf{g}_{q})-\delta_{r}\right)\right)\right]^{1-m_{q}},\end{array}$

(14)

where, weights $w$ and threshold $\delta$ parameters are conditioned on $\theta$. This equation is derived from Equation 4.

The second term is:

$\begin{array}[]{cc}p(d_{q}|m_{q},\textbf{g}_{q},\theta)=&\left[\operatorname{softmax}_{d_{1}}(\theta,\textbf{g}_{q,r,d})^{d_{1,q}}\operatorname{softmax}_{d_{2}}(\theta,\textbf{g}_{q,r,d})^{d_{2,q}}\right]^{m_{q}}\\ &\left[\operatorname{softmax}_{d_{3}}(\theta,\textbf{g}_{q,r,d})^{d_{3,q}}\operatorname{softmax}_{d_{4}}(\theta,\textbf{g}_{q,r,d})^{d_{4,q}}\right]^{m_{q}}\end{array},$

(15)

where, $d_{q}=\{d_{1,q},d_{2,q},d_{3,q},d_{4,q}\}$ is $1$ or $0$ when a player moves in one of the directions or not for each sample $q$. We note that when a player chooses to stay then $m_{q}=0$ and the decision of where to move is not relevant. This equation is derived from Equation 6.

We generate game play data by simulating the model discussed in Section 4.1 with $N=5000$. We considered two behavior tendencies, (1) rock agnostic tendency, and (2) a rock aversion tendency. A player with rock agnostic tendency is one who attributes a lot of consideration to large gold clusters and is agnostic about the amount of rock structures. On the other hand, a player with rock averse tendency focuses more on the gold in the rock-free regions such that they would not have to mine through the rocks to reach to the gold.

Our model enables capturing such player tendencies through an initialization of the weight and threshold parameters. Table 1 tabulates the initialized parameters for the two behavior tendencies. We note that the differentiating parameter for the two tendencies is $w_{2}$. This parameter quantifies the tendency of a player to consider the rock around. We note that a rock agnostic tendency doesn’t focus much on the rock around $w_{2}=0.3$ whereas the rock averse tendency attributes high weight to the rock around $w_{2}=0.8$ implying an avoidance to rock clusters. We note that since the game objective is to collect gold, both behavioral tendencies have a high weight $w_{1}$ on gold. The threshold parameters $\delta_{1,2}$ focus on a player’s emphasis on the size of the rock and gold clusters but they do not explain the tendencies of interest.

We sampled from the posterior (Equation 9) using the No-U-Turn Sampler (NUTS) (?), a self-tuning variant of Hamiltonian Monte Carlo (?) from the PyMC3 (?) Python module. We ran two chains of the (Markov Chain Monte Carlo) MCMC simulations and for each chain we ran $10000$ iterations with a burn-in period of $2000$ samples that are discarded.

We infer the behavioral tendency parameters for both the simulated datasets and are able to differentiate the behavioral data using the inferred parameters. Specifically, the posterior of $w_{2}$ differentiates the two tendencies of interest as intended from the simulated data sets. Figure 3 shows the posteriors distributions over each modeled parameter. The blue and red vertical lines represents the setting of the rock agnostic and rock averse tendency used for data simulation, respectively.

For rock agnostic tendency, Table 2 shows the statistical summary data from MCMC simulation.The (highest density interval) hdi%3 and hdi97% show the range of points of distribution which is credible. We find that the weight estimations have less standard deviation than the threshold estimations. The narrow range between hdi3% and hdi97% also represents the certainty of belief on weight estimations. Figure 4 shows the sampling process. Figure 5 shows a representative autocorrelation of all six cognitive variables. The low autocorrelation at the end shows the convergence of $w_{1}$. The same is also true for other cognitive variables. We find similar results for rock averse players and we only show the summary statistics in Table 3.