Mean extinction times in cyclic coevolutionary rock-paper-scissors dynamics

There are several examples of cyclic dominance in biology: Males of the californian lizards Uta stansburiana are known to inherit three different mating strategies which cyclically dominate each other [22, 24]. Another example is given by three strains of Escherichia coli [14], in tropical marine ecosystems [4] or in high arctic vertebrates [10]. Cyclic dominance is also important in some theoretical models like the susceptible-infected-recovered-susceptible (SIRS) model in epidemiology [2], in cyclic extensions of the Lotka-Volterra model [31, 32] or the Public Goods Game [13, 11]. Here, cyclic dominance is a way to preserve the coexistence of strategies (what is often called ‘biodiversity’ in a biological context) in the models.

A straightforward model system for cyclic dominance is the rock-paper-scissors game (RPS) well known from schoolyards. It contains the three strategies ‘rock’, ‘paper’, and ‘scissors’ with a simple domination rule: Paper wraps rock, scissors cut paper, rock crushes scissors. The payoff a dominating strategy gets from the dominated one is set to $1$ for normalization, the payoff for a tie is $0$, and a dominated strategy gets a payoff $-s$, where we assume $-s<0$. Hence the payoff matrix of this game reads

For the standard choice $s=1$ we have a zero-sum RPS, for all other values the game is non zero-sum. One can simply intuit the impact of $s$ [5]: For huge $s$ it is important for a player to avoid loosing, so it is successful playing the same strategy as the majority; hence an equilibrium that includes all three strategies is unstable. On the other hand, for small $s$ it is more important to win occasionally so that the mixed equilibrium becomes stable. Experiments indicate $s<1$ for the lizard example [36] and $s>1$ for E. coli [14].

For a long time, the resulting dynamics have been described by a meanfield approximation in the limit of infinite populations. The traditional way to describe such evolutionary games was the standard replicator equation, an equation of motion for the density of an arbitrary strategy $\alpha$ [12],

where $\psi$ is a constant prefactor which can be absorbed in the time scale by a constant rescaling of time. $\pi^{\alpha}(\vec{x})=x_{1}P_{\alpha,1}+x_{2}P_{\alpha,2}+x_{3}P_{\alpha,3}$ and $\langle\pi(\vec{x})\rangle$ are the payoff of strategy $\alpha$ and the average payoff of the whole population, respectively. The adjusted replicator equation [23]

has been used less frequently. The prefactor $1/(\Gamma+\langle\pi(\vec{x})\rangle)$ can be absorbed in the time scale by a dynamical rescaling of time for symmetric conflicts like the RPS, preserving stability properties. For asymmetric conflicts, like Dawkins Battle of the Sexes [7], this is not possible, as the average payoffs in each population do not coincide; hence the adjusted replicator equation can lead to qualitatively modified dynamics [26, 6]. The derivation of the replicator equations and Fokker-Planck-equations (comprising the first-order corrections) for the intrinsic noise are commonly taken from the master equation of the underlying discrete stochastic process [27, 28, 26, 31, 29].

Both replicator equations predict the existence of a fixed point at $\left(x_{R},x_{P},x_{S}\right)=\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$. The FP is neutrally stable for $s=1$, asymptotically stable for $s<1$, and unstable for $s>1$. In the case of finite populations of size $N$, the population can drive out of the fixed point because of stochastic fluctuations, and after some time one of the strategies will go extinct. Once this has happened, a second strategy will die out soon because of the lack of cyclic dominance, and the third strategy will therefore survive forever. Several efforts have been made to overbear this shortcoming especially of the zero-sum RPS, for example spatial discretization of populations (for a review see [25]), mobility [21], the introduction of intelligent update rules (best response [3]), the introduction of the parameter $s$ as mentioned above [5], or the computation of a critical system size above which coexistence of strategies is likely [26], but it is still an open question how long it takes on average until the first strategy has gone extinct. In this paper we investigate the scaling of the mean time to the extinction of the first strategy (mean extinction time, MET), depending on the system size $N$ and the parameter $s$, for well mixed populations and three evolutionary processes (with two of them having the standard and accordingly the adjusted replicator equation as limits $N\to\infty$), and present two analytical approaches which give theoretical insight in the scaling of the MET.

Contrary to replicator equations describing the dynamics of relative abundance densities, real populations are finite (and discrete), appropriate modeling thus bases on discrete stochastic processes (of birth and death). In the classical Moran process [16] an individual $a$ is chosen with probability proportional to its fitness. It reproduces, and the offspring replaces another randomly chosen individual $b$. In the frequency-dependent Moran process [18] which extends the classical Moran process [16] by considering coevolution, the fitness is not static but depends on the frequencies of the strategies. For better comparison with the processes mentioned beneath, in each time step we choose an individual $a$ at random which reproduces with probability

and the offspring replaces another randomly chosen individual $b$. Greek lettes $\alpha,\beta,\gamma$ can assume each of the strategies $R,S,P$, respectively. Here, $\omega$ is the selection strength, the $n_{i}$ are the number of agents playing strategy $i$, $\pi_{\alpha}=(n_{1}P_{\alpha,1}+n_{2}P_{\alpha,2}+n_{3}P_{\alpha,3})/(N-1)$ the payoff for an individual of strategy $\alpha$, and $\langle\pi\rangle$ the average payoff of the whole population. In the limit $N\to\infty$, the Moran process leads to the so-called adjusted replicator equation [26]. Note that a factor $\frac{1}{2}$ is introduced here for a better comparability with the two processes mentioned beneath.

The Moran process is a well-established stochastic process for evolutionary birth-death (for growing population sizes, see e.g. [33]) dynamics with overlapping generations and therefore (in its frequency-dependent extension) serves as a standard model of evolutionary game theory. However, the Moran process requires perfect global information about the whole population, an assumption that can be unrealistic and undesirable. Because of that, two local processes have been mentioned. Again we choose an individual $a$ at random for reproduction. Another randomly chosen individual $b$ changes its strategy to the strategy of $a$ with probability

for the local update [26] and

for the Fermi process [34], respectively, where $\Delta\pi_{max}$ is the maximum of the possible payoff difference. In the limit $N\to\infty$, these processes lead to the standard replicator equation (local update) [26], and a nonlinear replicator equation (Fermi process) [30] with similar properties, respectively. The common approach for the derivation of the equations of motion and the first-order corrections for demographic stochasticity are to derive a Fokker-Planck equation from the master equation for the respective stochastic process [27, 28, 26, 6, 29].

Here we focus on the mean extinction time for the Moran and Local Update processes. For each process, the probability of increasing the population strength of the strategy $\alpha$ by $1$ in a single time step and decreasing the population strength of strategy $\beta$ by $1$, is given by

This quantity is known as hopping rate. Note that the sum over all hopping rates is $\neq 1$ because reactions are possible that do not lead to changes in strategy frequencies. The time scale is chosen so that one reaction takes place every unit time step, including empty steps with no strategy change.

Compared to the time scale a mutant occurs – re-introducing a strategy – the time scale characterizing the survival properties of a genetic strain is defined by the mean extinction time. For two strategies, and fixed $N$, one has a onedimensional Markov process for which a closed expression allows to proceed analytically as has been demonstrated by Antal and Scheuring [1]. For higher-dimensional cases unfortunately exact general solutions of the mean extinction time (or a mean-first-passage time [19]) problems are not known; in this case the situation is furthermore hampered by the location-dependent dynamics within a simplex boundary. So in most cases one has to rely on systematic numerical investigations. We have carried out extensive simulations to quantify the mean times until one of the three strategies has gone extinct. We have analyzed the mean extinction times for all three described processes depending on the number of agents $N$ and the parameters $s$ and $\omega$. In general, we observe the following properties (see Fig. 1): For all three processes the MET becomes independent of the selection strength $\omega$ for $s=1.0$. Further on, the MET for $s=1.0$ is identical for all three processes and proportional to $N^{2}$,

As expected, for $s<1$ the MET is larger and for $s>1$ smaller than for $s=1$, but for weak selection the difference is small. In a logarithmic plot of the MET divided by $N^{2}$ one can easily see that $\langle t_{ext}(s=1.0)\rangle\propto N^{2}$, but also $\langle t_{ext}(s<1)\rangle\propto N^{2}e^{f(s,\omega)N}$ with $f(s,\omega)>0$ and independent of $N$. For the Moran process the stabilizing effect for $s<1$ is larger than for the other processes.

On the first view, for $s>1$ the MET seems to be $\langle t_{ext}(s>1)\rangle\propto N^{2}e^{-f(s,\omega)N}$. But such a proportionality would imply a disappearance of the MET for large $N$. This is unrealistic because even the shortest way would require $N/3$ steps. For that a dependence $\langle t_{ext}(s>1)\rangle=c_{1}N^{2}+c_{2}N^{2}e^{-f(s,\omega)N}$ is more likely (with $f(s,\omega)$ having the same properties as for the $s<1$ case), as it is predicted by one of our analytical approaches.

In summary, it can be said that the MET switches from polynomial to exponential scaling at $s=1$. This agrees with the drift reversal picture from [26, 5]. So the (global) Moran process and the (local) local update and Fermi process behave similar, but the impact of the parameters $s$ and $\omega$ is much stronger for the Moran process.

An exact analytical solution for two- or higher dimensional Markov chains is impossible, and a direct computation of the MET of such a three-strategy evolutionary game is not feasible. Although some efforts have been made, for example the work of Reichenbach et al. [20], wich have analyzed mean extinction properties in an urn model of a three-strategy game, employing the usual approach of applying van Kampen’s linear noise approximation and deriving a Fokker-Planck equation [35, 27, 28, 26].

Adapting this approach to the zero-sum RPS for the local update and the Moran process, we find a predicted MET proportional to $N^{2}$ and independent of $\omega$, which is in accordance with the simulation data. Unfornately this approach has some shortcomings: Although the overall scaling is correct, the numerical value (prefactor) of the predicted MET is too great by a factor of $\approx 5$, and it is not possible to use this approach for the non zero-sum RPS or in the Fermi process. For that we have developed two analytical approaches which do better. We will present both approaches for the local update. Following the same schemes for the other two processes is, although possible, a bit more sophisticated because not all integrals can be done analytically here in general. As we have seen from the numerical investigations, the $s\neq{}1$ payoffs override differences between the underlying processes, so that it is warranted to concentrate on one analytically more feasible process in the remainder.

Evolutionary games with three cyclically dominating strategies can stabilize the coexistence of strategies superior compared to games with only two strategies which include a dominating strategy. In finite populations, it is still likely that one of these strategies goes extinct, though on average it takes a time which is considerably longer than in two strategy games. We have analyzed the mean times to the extinction of one of the three strategies numerically in the RPS-game for three different evolutionary processes, the local update, the frequency-dependent Moran process and the Fermi process. These mean extinction times (MET) have the same fundamental dependencies of the number of agents $N$, the parameter $s$ and the selection strength $\omega$.

For the zero-sum RPS game ($s=1$), the MET is proportional to $N^{2}$ and independent of $\omega$ for all three processes; even the proportionality constants are the same. For $s<1$, we find a stabilization of biodiversity, as the MET grows exponentially with $N$. In contrast to that, for $s>1$, there is a destabilization of biodiversity, and the MET now grows smaller than in the zero-sum RPS (where it grows $\propto N^{2}$), namely, a part of the proportionality factor of the zero-sum RPS now decays exponentially with $N$. In both non zero-sum cases, the coefficient that is multiplied with the $N$ in the exponent, grows with both $\omega$ and $(s-1)$.

Solving such two-dimensional Markov chains is not possible, but we have developed two analytical approaches for the approximation of the dependencies of the MET of $N$ and the parameters $s$ and $\omega$. The first approach can be used for $s\geq 1$ only, but serves for all three processes. In contrast, the second approach gives a good approximation for all parameter values, but it can be used completely analytical only for the local update process, as for the other two processes not all integrals can be done analytical.

In summary, the fixation time – as a long-time property of the process – shows a crossover from exponential to polynomial scaling with the population size, being fully consistent with the critical population size deived from the “drift reversal” picture which is based on the short-time dependence of the average drift in phase space - repelling versus attracting towards the fixed point. Thus both approaches to describe stochastic stability in finite populations here lead to the same conclusions: coexistence is preserved (stabilized) for a positive-sum cyclic game, whereas negative-sum games as well as small populations destabilize coexistence.