A memory-based spatial evolutionary game with the dynamic interaction between learners and profiteers

In this study, we adopt the classical snowdrift game (SDG) for its generality, where the reward (R) for the interaction of two cooperative strategies is fixed to 1, the punishment (P) for the interaction of two defective strategies is set to 0, and for the interaction of cooperative and defective strategies, the cooperator receives a sucker’s payoff (S) of $1-r$ while the defector yields a temptation to defect (T) of $1+r$. Hence, the payoff matrix of SDG is represented as follows:

where $r$ indicates the cost-to-benefit when both individuals are cooperators, and it takes a value ranging from 0 to 1, which is a flexible parameter.

Next, we provide a detailed explanation of the memory mechanism proposed in this paper. Each individual in the system possesses a memory, i.e. they are capable of knowing their payoffs from the previous $M$ rounds of interactions with their neighbors. Hereby, $M$ denotes the length of the individual’s memory, and the payoff in the past $M$ rounds has an impact on the individual’s current payoff. However, it is acknowledged that the impact of past interactions diminishes over time. To account for this, we introduce a memory decay factor $\beta$, which characterizes the decreasing influence of past events on an individual’s current payoff. Therefore, the payoff of individual $i$ at time $t$ based on the memory mechanism can be expressed as

where $\Pi_{i}^{k}$ represents the actual payoff of individual $i$ obtained from playing the snowdrift game with all neighbors in round $k$, and it can be calculated as

where $\Omega_{i}$ means the set consisting of all neighbors of individual $i$. We emphasize that when the evolutionary time is smaller than the memory length, i.e., $t<M$, then the individual’s memory length is considered as $M=t$. Besides, $s_{x}$ is the strategy of individual $x$, where a unit vector $s_{x}=[0,1]^{T}$ indicates a defective strategy and $s_{x}=[1,0]^{T}$ denotes cooperation.

In reality, individuals exhibit diverse behaviors, with some driven primarily by profit-seeking motives (referred to as profiteers), while others rely on self-learning strategies (referred to as learners). Therefore, our proposed model incorporates the interaction between these two categories of individuals mentioned above. Furthermore, individuals are not fixed in their roles throughout the evolutionary game process, i.e., individuals cannot always be profiteers or learners, and they undergo mutual transitions between the two states. For this reason, we introduce a Markov process to capture this dynamic. Specifically, a profiteer (resp. learner) changes to be a learner (resp. profiteer) with the probability $q$ (resp. $p$) at every moment, and the state transition matrix representing these probabilities is given by

where $p$ denotes the probability of an individual transitioning from a learner to a profiteer and $1-p$ represents the probability of remaining a learner. Analogously, $q$ indicates the probability of an individual switching from a profiteer to a learner, while $1-q$ denotes the probability of staying a profiteer. We highlight that there is no relationship between the transition probabilities $p$ and $q$. The only requirement is that they must satisfy the definition of probability, meaning both $p$ and $q$ must fall between 0 and 1.

For learners and profiteers, the key difference between them lies in their strategy updating rules. Profiteers tend to imitate the strategy of the neighbor with the highest payoff, employing the Fermi rule for strategy updating, i.e., at each round of evolution, a profiteer $i$ randomly selects an individual $j$ from its neighbors and adopts neighbor’s strategy according to the following probability:

where $s_{i}$ and $U_{i}$ signify the strategy expressed as unit vectors and payoff based on the memory mechanism of individual $i$, respectively. The parameter $\kappa$ means the noise factor, which is utilized to describe the irrational choices of individuals in the game.

On the other hand, learners use a reinforcement learning algorithm known as Q-learning for strategy updates. Specifically, the Q-learning algorithm can be regarded as a Markov decision process, where the decision of an individual is only relevant to the current situation and is not influenced by past events, which can be represented by a tuple $(S,A,W_{l},r)$. Hereby, $S=\{0C,1C,2C,\cdots,|\Omega_{i}|C\}$ (the number of cooperators among neighbors) and $A=\{Cooperate,Defect\}$ denote the state space and action space of individual $i$. $W_{l}:S\times A\rightarrow p$ is the state transition probability after adopting action $a\in A$ in state $s\in S$, and $r:S\times A\rightarrow U$ represents the reward of individual for performing action $a\in A$ in state $s\in S$, which can be obtained by Eq. 2. Each learner owns a Q-table, which will be updated after each round of the game with all neighbors according to the following equation:

where $\alpha\in[0,1]$ and $\gamma\in[0,1]$ denote the learning rate and discount factor, respectively. A smaller $\gamma$ causes the individual to focus more on the immediate payoff, otherwise, the individual focuses more on past experiences. $Q_{S_{t}}^{A_{t}}(t)$ represents the utility obtained by the individual in state $S_{t}$ when taking action $A_{t}$ at time $t$. $r_{S_{t}}^{A_{t}}(t+1)$ indicates the immediate payoff gained by the individual in state $S_{t}$ at time $t+1$ when performing action $A_{t}$ at time $t$, which is determined by Eq. 2. The term $\max_{a\in A}Q_{S_{t+1}}^{a}(t)$ signifies the maximum Q-value received by the individual in the future state $S_{t+1}$. Therefore, the learner learns to update the Q-table by constantly playing games with its neighbors during the evolutionary process to guide the updating of its strategy. In particular, to portray the random choice of individuals in real situations, which is a compromise between exploration and exploitation, we introduce the $\epsilon$-greedy algorithm. Specifically, at each decision point, exploration occurs with probability $\epsilon$, i.e., a strategy is randomly selected from the action set $A$ with a uniform probability distribution, and exploitation is performed with probability $1-\epsilon$, i.e., the strategy with the highest Q-value in the current state is taken, and if there is more than one, then one is chosen randomly.

As we mentioned before, individuals are allowed to switch between profiteers and learners, with the corresponding transition matrix shown in Eq. 4. The transition probability is independent of the initial moment, which can be viewed as a two-state homogeneous Markov chain $\{X_{n},n=0,1,2,\cdots\}$ with state space $E=\{Profiteer,Learner\}$, where $X_{n}$ denotes the category of each individual at time $n$. Moreover, for any $i,j\in E$, there always exists a positive integer $n_{0}$ such that $B_{i,j}^{n_{0}}>0$, indicating the ergodicity of the Markov chain. By the ergodic theorem, we have:

where $(\pi_{1},\pi_{2})$ indicates the stationary distribution of the Markov chain $\{X_{n},n=0,1,2,\cdots\}$. Substituting Eq. 4 into Eq. 7, we obtain the solution $\pi_{1}=q/(p+q)$ and $\pi_{2}=p/(p+q)$, which implies that as evolutionary time extends indefinitely, the probability of an individual being a learner is $\pi_{1}$, while the probability of being a profiteer is $\pi_{2}$.

Furthermore, based on the fact that whether each individual is a profiteer or a learner is independent of the other individuals, i.e., the categories of individuals are independent of each other, we can yield the expected number of profiteers and learners in the network when the evolution stabilizes, as expressed in the following equation:

where $N$ represents the size of the network.

To provide a clearer understanding of our model, we present an illustrative example of the model in Fig. 1. Each individual possesses a probability of transitioning between being a profiteer and a learner, showcasing dynamic interactions between the two categories in the network. Profiteers and learners utilize Fermi rules and Q-learning to update their strategies, respectively. Notably, each learner maintains a Q-table. Taking individual $v_{5}$ highlighted within the red square as an example, it is surrounded by two cooperators and two defectors among its neighbors. According to the Q-table, we can get that the Q-value of adopting a cooperative strategy in the current state is 17.72, surpassing the Q-value (13.92) for choosing a defective strategy. Consequently, individual $v_{5}$ will opt for the cooperative strategy during exploitation at the next moment.

In order to verify the theory of dynamic interaction between learners and profiteers proposed in this paper, we first plot the evolutionary curves of the number of learners over time under different transition probabilities of $p$ and $q$, and the result is shown in Fig. 2(a), where the red straight line indicates the theoretical value, which is calculated according to Eq. 8. We do not plot the evolution of profiteers since it can be obtained by subtracting the number of learners from the size of the network. From Fig. 2(a), we can see that the number of learners in all three cases evolves over time and stabilizes around $t=100$, subsequently fluctuating around a certain value, which closely aligns with the one marked by the red theoretical straight line. Additionally, we present the theoretical and simulated values of the number of learners and profiteers, along with the relative error between them in Tab. 1 in the form of data for the three different scenarios. This provides a more intuitive insight into the gap between theory and simulation. The theoretical value is calculated using Eq. 8, while the simulated value is obtained by averaging the last 1000 steps of the evolution curve in Fig. 2(a). The relative error is determined by $e=|x-x^{*}|/x^{*}$, where $x^{*}$ and $x$ denote the theoretical and simulated values, respectively. We find that the theoretical and simulated results for all three situations are very close to each other, and the maximum relative error is only 0.096%, which confirms the correctness of our theoretical derivation. Furthermore, by comparing the number of learners and profiteers at the stable time under the three cases, we observe that increasing $p$ leads to an increase in the number of profiteers, while a larger $q$ results in more learners in the network, which is consistent with the results demonstrated in Fig. 2(a).

Subsequently, we record the number of learners in the last 4000 steps in each scenario and plot the probability distribution of the number of learners under three sets of transition probabilities according to the law of large numbers by considering the frequency as probability. The result is shown in Fig. 2(b), where the red straight line represents the theoretical values. Similarly, we present some numerical characteristics for each of the three distributions in Tab. 2, including standard deviation, skewness, and kurtosis, where the standard deviation is calculated using the $std()$ method in the $numpy$ library in Python, while the skewness and kurtosis are obtained employing the $skew()$ and $kurt()$ methods in the $pandas$ library, respectively. We observe that all three distributions approximately follow a normal distribution and that the span of each distribution is relatively small, explaining the small standard deviation of each distribution, which is consistent with the result in Tab. 2. Moreover, it can be clearly seen that the red theoretical straight line exactly passes through the tip of each distribution, which also verifies the correctness of our theory in another way. The skewness of all three statistical distributions of the number of learners exhibited in Tab. 2 is less than 0. It indicates that all three distributions have negative skewness, i.e., the left skewness, where there are fewer data located on the left side of the mean than on the right side of the mean, whereas profiteers are the reverse. The kurtosis of learners under both $p=0.8,q=0.5$ and $p=0.5,q=0.5$ is less than 0, which means that the overall data distribution is relatively flat compared to the normal distribution and is platykurtic, while the kurtosis under $p=0.5,q=0.8$ is greater than 0, which implies that the overall data distribution is relatively steep compared to the normal distribution and is leptokurtic. It is worth noting that the standard deviation and kurtosis are the same for both profiteers and learners, while the skewness is just the opposite. This discrepancy arises because the network contains only two categories of individuals, profiteers and learners, and the standard deviation and kurtosis are essentially even-ordered central moments, whereas the skewness is an odd-ordered central moment.

In this section, we investigate the effect of some model parameters on the emergence and evolution of cooperative behaviors. Primarily, we present the heat map of the cooperation ratio with respect to the transition probabilities $p$ and $q$ between profiteers and learners, as shown in Fig. 3. The $x$-axis is set as $q$ with the range [0, 1], which denotes the transition probability from profiteers to learners, and the $y$-axis is set as $p$ with the same range, which means the transition probability from learners to profiteers. Both networks exhibit that when $p$ is fixed, increasing the transition probability $q$ leads to a rise in the proportion of cooperators, i.e., the incorporation of learners promotes the emergence of cooperative behavior compared to a population of pure profiteers. This insight offers a fresh perspective to explain the widespread cooperative behavior in real-world scenarios. Although many individuals strive to behave as profiteers by imitating the strategies of individuals with high payoffs, there are still learners who are self-learning based on their past experiences, and the existence of learners is exactly the reason that further enhances the proportion of cooperators in the population.

Subsequently, we delve into the impact of the memory mechanisms proposed in this paper on the evolution of cooperative behavior. We set the transition probability between profiteers and learners as $p=q=0.5$, thus we can get that the profiteers and learners in the network are uniformly mixed according to Eq. 8. In this case, we plot the heat map of the cooperation ratio concerning the memory length and the memory decay factor, and the results of SL and WS networks are respectively shown in Figs. 4(a) and 4(b), from which we can obtain that increasing both the memory length $M$ and the memory decay factor $\beta$ causes a decrease in the number of cooperators for both SL and WS networks.

In addition, we demonstrate the changes in the proportion of cooperators concerning the memory mechanism for the pure profiteer scenario in Fig. 5, where the probability of converting from learner to profiteer is set to $p=1$. In contrast, the probability of changing from profiteer to learner is $q=0$. All other parameters are the same as in the numerical simulation in Fig. 4. It is obvious that the memory mechanism greatly facilitates the emergence of cooperative behavior in the case of pure profiteers. Notably, pure cooperators even appear on both networks when the memory length and memory decay factor are relatively large. Furthermore, upon comparing Figs. 5(a) and 5(b), we can observe that the area of the pure cooperators’ region of WS shown in Fig. 5(b) is significantly larger than that of SL presented in Fig. 5(a), which suggests that the WS network is more favorable to the survival of cooperators than the SL network. Therefore, we can conclude that while the memory mechanism facilitates the emergence of cooperative behavior in profiteers, it inhibits the evolution of cooperation in learners.

It is also crucial to study the effect of learners’ learning rate $\alpha$ and discount factor $\gamma$ in Eq. 6 on cooperative behavior. Thus, we exhibit the heat map depicting the percentage of cooperators on the SL and WS networks concerning $\alpha$ and $\gamma$ in Figs. 6(a) and 6(b), respectively. We set the payoff parameter, the transition probabilities between profiteers and learners, the memory decay factor, and the memory length to $r=0.5,p=q=0.5,\beta=0.5,$ and $M=5$. From Figs. 6(a) and 6(b), we can obtain that both SL and WS networks demonstrate a higher learning rate $\alpha$ leads to a larger percentage of cooperators. On the contrary, increasing the discount factor $\gamma$ results in a decrease in the cooperation frequency. This indicates that a larger learning rate and a greater tendency of learners toward immediate benefits promote the emergence and maintenance of cooperative behaviors in the network.

Next, we explore the influence of exploration rate $\epsilon$ and payoff parameter $r$ on the cooperative behavior, and the results are depicted in Fig. 7. With transition probabilities between profiteers and learners set to $p=0.4,q=0.8$, and memory mechanism parameters held constant at $\beta=0.5$ and $M=5$, both WS and SL networks reveal interesting insights. For relatively small values of the payoff parameter ($r<0.3$), an appropriate increase in the exploration rate fosters cooperation, but excessive exploration hampers cooperative behavior. However, when $r$ is relatively large ($r>0.3$), elevating the exploration rate consistently diminishes the prevalence of cooperators. Furthermore, we observe that an increase in the payoff parameter always reduces the fraction of cooperators, as higher $r$ values lead to greater payoffs for defectors based on Eq. 1, thereby incentivizing more individuals to adopt the defective strategy.

Then, we display the evolutionary curves of the cooperation ratio under three different parameter pairs ($\beta$, $r$) and illustrate the distribution of cooperators and defectors on the SL networks at different instants from a micro perspective in Fig. 8. From Fig. 8(a), it is evident that the frequency of cooperators $f_{c}$ stabilizes around $t=100$ in all three situations, and then stabilizes with slight fluctuations around a certain value. In addition, we observe that the cooperation ratio of $\beta=0.9,r=0.5$ marked by blue squares is higher than that of $\beta=0.1,r=0.5$ labeled by green triangles, and lower than that of $\beta=0.9,r=0.3$ marked by black circles, which can be visually confirmed from the snapshots depicted in Figs. 8(b)-(m), where red represents the defector and blue indicates the cooperator.

Specifically, Figs. 8(b), (f), and (j) all demonstrate that the network is almost evenly mixed with cooperators and defectors at $t=1$, while the blue region for $\beta=0.9,r=0.3$ (Figs. 8(b)-(e)) grows large as time progresses and eventually occupies the entire network, indicating that all the individuals become cooperators, and there are no defectors exist. For $\beta=0.9,r=0.5$ (Figs. 8(f)-(i)), the blue region gradually increases and eventually stabilizes, signifying that cooperators play a dominant role in it. In contrast, the blue region gradually diminishes over time and eventually stabilizes for $\beta=0.1,r=0.5$ (Figs. 8(j)-(m)), meaning that the defector holds a dominant role in it. Additionally, we can observe that the cooperators resist the invasion of the defectors mainly by forming clusters. These observations can be attributed to the fact that the transition probabilities between profiteers and learners are set to $p=1$ and $q=0$, i.e., the network is ultimately all occupied by profiteers. Moreover, we can get that increasing the memory decay factor $\beta$ promotes cooperative behaviors as indicated in Fig. 5. Besides, according to Eq. 1, when a cooperator meets a defector, the defector’s payoff increases but the cooperator’s payoff decreases as the payoff parameter $r$ grows, which promotes the emergence and maintenance of defectors in the network.

Note that in all of our previous simulations, the sizes of the SL and WS networks are fixed to $50\times 50$. In this subsection, we aim to examine the evolution of cooperative behaviors on SL and WS networks with different sizes to verify the robustness of the model. We depict the variation of the cooperation frequency $f_{c}$ on SL and WS networks as a function of network size $N$ for three different sets of parameters ($M$, $\beta$) and three distinct groups of parameters ($r$, $p$) in Figs. 9(a) and 9(b), respectively.

From Fig. 9, we obtain that both SL and WS networks exhibit almost no influence of network size on cooperative behavior under the same set of parameters, which is further demonstrated by the range and standard deviation of each curve, and the results are shown in Tabs. 3 and 4. The reason we choose the range and standard deviation as our statistical metrics is that range indicates the extent of fluctuation in the proportion of cooperators and standard deviation quantifies the magnitude of fluctuation in the frequency of cooperators. A small range and standard deviation signify that network size has a negligible impact on the frequency of cooperators, validating the robustness of our findings across different network sizes. It can be seen that the maximum range and standard deviation of SL do not exceed 0.0402 and 0.0113, respectively, and the maximum range and standard deviation of WS are within 0.0324 and 0.0089, respectively, which signifies that the fluctuation ranges and magnitudes of $f_{c}$ on both networks are minimal. Notably, the fluctuation ranges and magnitudes on the WS network are even smaller than those on the SL network, consistent with the trends presented in Fig. 9. Furthermore, by comparing Figs. 9(a) and 9(b), we notice that the WS network exhibits a higher percentage of cooperators for the parameter set ($M$, $\beta$) (denoted as dashed lines) compared to the SL network, while both networks show similar cooperation ratio for the parameter set ($r$, $p$) (denoted as solid lines). Therefore, through this simulation, we can conclude that the results are consistent across different network sizes for SL and WS networks, confirming the robustness of the proposed model.