Role of the effective payoff function in evolutionary game dynamics

In a Darwinian evolutionary process, it mainly incorporates three fundamental ingredients: inheritance, mutation, and selection. Due to the influence of perturbation or random drift, a genetic process frequently accompanies the occurrence of mutations, which creates genotypic or phenotypic variation, thereby finally leading to the differences in individual fitness that selection acts upon [1]. By integrating game theory with Darwinian evolution [2, 3], evolutionary game theory has become a powerful mathematical framework to model biological [4, 5] and social [6, 7, 8, 9] evolution in a population consisting of different types of interacting individuals under frequency-dependent selection.

Traditionally, a widely used system that focuses on the effects of frequency-dependent selection is the replicator equation [10, 11], where the population is infinitely large well-mixed and the stochastic effect is exclusively overlooked usually. However, if we relax this setting to a more realistic situation where the population is finite well-mixed and subject to fluctuations, this deterministic approach is augmented and disturbed by random drift [12, 13, 14]. In such a finite population with fluctuations, it needs to resort to the tool of stochastic evolutionary game dynamics for investigating the evolution of different traits [1, 15]. In addition to the classical Wright-Fisher process [12, 16], frequency-dependent Moran process [13, 15], and imitation-based pairwise comparison process [17, 18, 19] are two most common microscopic models of strategy spreading in finite populations. In contrast to the well-mixed population setting above, there are also lots of interest in studying evolutionary game dynamics in structured populations [17, 20, 21, 22]. Typically, the spatial geometry of population structure is modeled by regular lattices [23, 24, 25, 26, 27, 28] or more general complex networks [29, 30, 31, 32], where individual interactions merely occur among nearest neighbors.

For a game system, in general, the ingredients influencing the final evolutionary outcomes are nothing but the model, update rule, mutation rate, and population structure, etc. Models and update rules determine the way of strategy spreading. Mutation rates measure the intensity of randomness, while the underlying population structure describes the geometry of individual interactions. Depending on the game interactions, each individual obtains a payoff, and finally it needs to translate into reproductive success, termed effective payoff [33]. For example, the effective payoff is known as fitness in Moran process [13, 15] and imitation probability in pairwise comparison process [18, 19, 34]. Based on the usual assumption that the effective payoff is the form of $1+(\text{Selection intensity})\cdot(\text{Payoff}$), for standard two-strategy two-player games, Tarnita et al. demonstrate that if the selection intensity is weak the condition for one strategy to dominate the other is determined by a ‘$\sigma$-rule’ [33], which holds on any population structure [35]. The parameter $\sigma$, termed structure coefficient, is a quantity that only depends on the population structure, update rule, and mutation rate, but not on the payoff values. Later, this work attracts wide interest [36, 37, 38]. For two-player games with multiple strategies, it involves two structure coefficients [36]. To calculate them, investigating games with three strategies is enough. While for multi-player games with two strategies where $d$ individuals are selected to play a game, the $\sigma$-rule will depend on $d-1$ structure coefficients [37]. In particular, for a more general setting of multi-player games with many strategies, this rule turns out to be quite complicated and the number of structure coefficients required for a symmetric game with $d$-player and $n$-strategy grows in $d$ like $d^{n-1}$ [38]. Clearly, because the form of the effective payoff is a specific function in these works, the $\sigma$-rule does not reflect the influence of the effective payoff function on evolutionary outcomes.

In evolutionary biology, however, how to measure the genotype-fitness map (i.e., the fitness landscape) is always a challenging issue, and now it has been accepted that the shape of the genotype-fitness map has fundamental effects on the course of evolution [39]. In addition, based on a Markov chain model, it has been demonstrated that the heterogeneity of individual background fitness can act as a suppressor of selection [40]. In a way, therefore, it means that the form of the effective payoff function (which translates the payoff derived from game interactions into the ability of reproductive success) has a significant effect on the evolution of game dynamics.

To this end, in this letter we study the effect of different function forms of effective payoff on evolutionary dynamics and accordingly extend the results given by Tarnita et al. [33]. We find that if the first-order derivative of the effective payoff function can be written by a linear combination of payoff, then the condition for one strategy to dominate the other depends not only on the $\sigma$-rule, but also on an extra constant which is determined by the effective payoff function. This constant determines the direction of $\sigma$-rule (strategy selection). Additionally, taking the Moran process and pairwise comparison process as specific models in well-mixed populations, we demonstrate that different fitness or imitation mappings are equivalent under weak selection and the extra constant curbs the direction of one-third law and risk-dominance in the limit of large populations.

In a structured population with $N$ individuals, we consider stochastic evolutionary dynamics induced by a mutation-selection process. Each player can choose an arbitrary strategy from $A$ and $B$. Then, depending on the payoff matrix

players obtain an accumulative payoff by interacting with other individuals based on the underlying population structure. For example, when an $A$ player interacts with another $A$ player, it will obtain a payoff $a$, but $b$ when interacting with a $B$ player. Likewise, a $B$ player can obtain a payoff $c$ when interacting with an $A$ player, and payoff $d$ when interacting with another $B$ player. Therefore, the total payoff of each player is a linear function of $a$, $b$, $c$, and $d$ without including constant terms (evidently, if the payoff is calculated in an average way, the linear relation also holds). For an $A$ player, for instance, the total payoff is $a\cdot(\text{number of {A}-neighbors})+b\cdot(\text{number of {B}-neighbors})$. To study the effect of effective payoff functions on evolutionary dynamics, instead of a specific form, we assume that the effective payoff of a player is given by $\varphi(\beta,\text{Payoff})$. Parameter $\beta$ measures the intensity of selection, and $\beta\rightarrow 0$ corresponds to the case of weak selection [13, 19, 41].

The reproductive process of each player is dependent on the update rule and its effective payoff, and subject to mutations. With probability $\mu$, a mutation occurs and the offspring adopts a strategy ($A$ or $B$) at random. Otherwise, with probability $1-\mu$, the offspring inherits its parent’s strategy. For $\mu=1$, there are only mutations, no selection, and strategy choice is completely random. If $0<\mu<1$, however, there exists a mutation-selection equilibrium [42, 43].

For the game of two strategies, the frequency of $A$ players in the population defines a finite state space, $S$, and the evolutionary dynamics can be captured by a Markov process on this state space. We denote the transition probability from state $S_{i}$ to state $S_{j}$ by $P_{ij}$. Since the transition probability depends on the update rule and on the effective payoff of players, it can be given by $P_{ij}[\varphi(\beta,\text{Payoff})]$. Furthermore, we assume that $\varphi(\beta,\text{Payoff})$ is differentiable at $\beta=0$. In the limit of weak selection, then we can give $\varphi(\beta,\text{Payoff})$ in the form of first-order Taylor expansions, $\varphi(\beta,\text{Payoff})=\varphi_{0}+\varphi^{(1)}(0)\cdot\beta+o(\beta)$. Here $\varphi_{0}:=\varphi(0,\text{Payoff})$ represents the baseline effective payoff of each player, and $\varphi^{(1)}(0):=[\partial\varphi(\beta,\text{Payoff})/\partial\beta]_{\beta=0}$ represents the first-order coefficient of selection intensity. Particularly, if $\varphi^{(1)}(0)$ can be written by a linear combination of payoff, that is $\varphi^{(1)}(0)=[\partial\varphi(\beta,\text{Payoff})/\partial\beta]_{\beta=0}=k_{0}\cdot\text{Payoff}+c_{0}$, then the transition probability is given by $P_{ij}[\varphi_{0}+(k_{0}\cdot\text{Payoff}+c_{0})\beta+o(\beta)]$. Clearly, the constants, $k_{0}$ and $c_{0}$, are dependent on the choice of the function $\varphi(\beta,\text{Payoff})$ and may rely on the entries of the payoff matrix. In addition, actually a large body of functions meet the condition $\varphi^{(1)}(0)=k_{0}\cdot\text{Payoff}+c_{0}$, such as $1-\beta+\beta\cdot\text{Payoff}$ [13], $exp(\beta\cdot\text{Payoff})$ [37], and $\varphi(\beta\cdot\text{Payoff})$ [19]. Therefore, this condition is not a harsh requirement. Note that the payoff of players is linear in $a$, $b$, $c$, and $d$ without constant terms, it follows that the transition probability is the function $P_{ij}[(k_{0}a+c_{0})\beta,(k_{0}b+c_{0})\beta,(k_{0}c+c_{0})\beta,(k_{0}d+c_{0})\beta]$. Then, based on the notation $(a^{\prime},b^{\prime},c^{\prime},d^{\prime}):=(k_{0}a+c_{0},k_{0}b+c_{0},k_{0}c+c_{0},k_{0}d+c_{0})$ and following the proof given in Ref. [33], we know that the condition that strategy $A$ is favored over strategy $B$ (i.e., strategy $A$ is more abundant than $B$ in the mutation-selection equilibrium) is $\sigma a^{\prime}+b^{\prime}>c^{\prime}+\sigma d^{\prime}$, where $\sigma$ is a parameter that depends on the population structure, update rule, and mutation rate. Accordingly, we have the following theorem:
Theorem $1$. Consider a population structure and an update rule that satisfy the following three conditions: (i) the transition probabilities are differentiable at $\beta=0$; (ii) the update rule is symmetric for the two strategies $A$ and $B$; and (iii) in the game given by the matrix entries, $a=c=d=0$ and $b=1$, strategy $A$ is not disfavored. Then, in the limit of weak selection, when the effective payoff function $\varphi(\beta,\text{Payoff})$ satisfies $[\partial\varphi(\beta,\text{Payoff})/\partial\beta]_{\beta=0}=k_{0}\cdot\text{Payoff}+c_{0}$, strategy $A$ is favored over strategy $B$ if $k_{0}(\sigma a+b-c-\sigma d)>0,$ where $k_{0}$ is a constant that relies on the function of the effective payoff, and $\sigma$ is the structure coefficient which depends on the model and the dynamics (population structure, update rule, and mutation rate), but not on the entries of the payoff matrix.

This theorem implies that for determining the condition under which one strategy dominates the other, the classical $\sigma$-rule [33] is not enough. It also depends on an additional constant $k_{0}$ determined by the effective payoff function. Actually, the constant $k_{0}$ controls the direction of $\sigma$-rule (strategy selection). If the effective payoff function is given such that the constant $k_{0}$ is positive, the theorem recovers the classical $\sigma$-rule (selection favors $A$ to dominate $B$). Otherwise, if the effective payoff function is given such that $k_{0}$ is negative, the classical $\sigma$-rule will reverse the direction (selection favors $B$ to dominate $A$).

To check the validity of our theorem and to study how the effective payoff function influences the evolutionary outcomes in a specific dynamic process, here we consider the frequency-dependent Moran process and pairwise comparison process. These two processes represent two classes of typical evolutionary dynamics. The former describes how successful strategies spread in the population through genetic reproduction, whereas the latter describes such a process through cultural imitation.

In the Moran process, the effective payoff is known as the individual fitness, which measures the ability to survive and produce offspring. With a probability proportional to the fitness, an individual is selected randomly for reproduction. And then one identical offspring replaces another randomly chosen individual. Usually, the fitness is assumed to be a convex combination of a background fitness (which is set to one) and the payoff from the game [13, 15], or an exponential function of payoff [37]. Under these specific forms, the constant $k_{0}$ related to the fitness function actually turns to $1$ and it reduces to the previous results [13, 18]. Instead of using a specific fitness form, here we adopt a generalized mapping that the fitness of a player is any a non-negative function of two variables on selection intensity and payoff, $f(\beta,\text{Payoff})$. Since mutation occurs during the process of reproduction, it follows that the transition probabilities are given by

where $\pi_{A}(i):=[a(i-1)+b(N-i)]/(N-1)$ and $\pi_{B}(i):=[ci+d(N-i-1)]/(N-1)$ are the average payoffs of an $A$ player and a $B$ player, respectively.

While in the pairwise comparison process, the effective payoff is known as the imitation probability. Two individuals are sampled randomly and then a focal player imitates the strategy of the role model with a probability depending on the payoff comparison [43, 18]. As usual, the imitation probability is modeled by the Fermi function with considering the effect of noise [24, 17, 25]. Thus, this process is also called Fermi process. Under the situation that the effective payoff mapping is non-specified, however, the imitation probability function for the pairwise comparison process should be given by $g(\beta,\Delta\pi)$. Here, $\Delta\pi$ denotes the difference of average payoffs between strategy $A$ and $B$. In the presence of mutations, this imitation process occurs accurately with probability $1-\mu$. Otherwise, with probability $\mu$, the focal player adopts a random strategy, $A$ or $B$. Then, it leads to the transition probabilities,

where $\Delta\pi(i):=\pi_{A}(i)-\pi_{B}(i)=ui+v$. Herein, parameters $u$ and $v$ are defined by $u:=(a-b-c+d)/(N-1)$ and $v:=(Nb-Nd-a+d)/(N-1)$, respectively.

Moreover, for both processes, the probability to stay in the current state is $1-P_{i,i+1}-P_{i,i-1}$, and the probability to transform to other states is vanishing. In what follows, we first calculate the fixation probabilities and fixation times under weak selection when mutations are absent. Then, we derive the criterion that strategy $A$ is favored over strategy $B$ in this case, and finally extrapolate this criterion to small mutation rates.

Most previous studies exclusively assume that the effective payoff function is a specific form. One direct result of this setting is that the final evolutionary outcomes do not reflect the effect of the effective payoff function on evolutionary dynamics. Therefore, it still remains unclear how the effective payoff function influences the evolutionary dynamics in a game system. With introducing a generalized mapping, we investigate such effect in this letter.

For standard $2\times 2$ games where a specific form of the effective payoff is adopted, it has been demonstrated that under weak selection the condition for one strategy to dominate the other is determined by a $\sigma$-rule [33]. This rule almost captures all aspects of evolutionary dynamics, but ignores the effect of the effective payoff function. Particularly, if the effective payoff function is an any non-negative mapping of the product of the payoff and the selection intensity [19, 37], the rule still holds. But it does not change the basic fact that the role played by the effective payoff function in evolutionary game dynamics is exclusively overlooked. With introducing a more generalized mapping that the effective payoff of individuals is a function of payoff and selection intensity, however, we find that the condition determining a strategy to be selected relies not only on the $\sigma$-rule, but also on an extra constant which characterizes and depends on the effective payoff function. As an extension, a multi-strategy version is also given in the Supplementary Material.

Based on specific effective payoff forms, it has been found that the linear and exponential functions lead to identical evolutionary outcomes under weak selection [43, 37, 19]. Here we generalize this equivalence understanding to any two fitness functions in a Moran process, and imitation probability functions in a pairwise comparison process. In addition, except for the standard $2\times 2$ games and weak selection, there are lots of research interest in the games of multiple players or strategies [37, 50, 36] and strong selection [51], which are worth the effort in the future.