Agent-Based Modelling Meets Generative AI in Social Network Simulations

In each iteration ($i$) of the simulation, every generative agent is immersed in an environment resembling a social media platform. In line with prior research [9], we design a prompt that introduces the social media environment, highlighting the presence of other agents and outlining the possible actions to take within the environment: (i) generating original content, (ii) resharing content from other agents, and (iii) remaining inactive. This setup, though simple, accurately mirrors common actions observed across various social media platforms, irrespective of platform-specific regulations. Moreover, we emphasize that these types of actions are the sole predefined aspects in the simulation. Instead, the agent behavior is not determined by deterministic or probabilistic processes as LLM-agents autonomously decide the actions and the content to generate based on their personality, interests, and other agents’ behavior. Subsequently, the output of the Reasoning Module — the response generated by each agent — is structured as a triplet denoted by Choice-Reason-Content. For instance, as illustrated in Figure 1, an agent may choose to publish original content to discuss persistent societal issues like climate change or immigration often targeted during electoral campaigns. Additionally, the agent provides the reasoning behind the post, indicating an intent to engage in political discourse and foster a healthy exchange of ideas.

To achieve a comprehensive simulation of a social network, enabling interaction among agents is imperative. This involves gathering all actions performed by agents and presenting to each agent the activities of others. However, including all information generated by other users within a single prompt presents practical challenges. First, the context window of any LLM-agent is limited, causing memory saturation if all information is included in one prompt. Second, longer prompts increase the risk of hallucinations [13], resulting in agents producing inaccurate, contextually inconsistent, or nonsensical content. To tackle these challenges, we introduce the innovative use of Retrieval-Augmented Generation (RAG). This technique enables LLM-agents to access additional contextually relevant data stored in an external vector database [18]. Figure 2 illustrates the integration of the RAG technique within the Interaction Module. Specifically, the vector database is continuously updated to record all agent actions and their corresponding published contents. This tracking mechanism facilitates monitoring the simulation’s progression for subsequent analysis. Furthermore, the Retriever step within RAG serves as a recommendation system, determining which content to present to each LLM-agent. For instance, as depicted in Figure 1, the agent may receive recommendations for two posts from the entire corpus of published content. These posts might discuss topics such as investing in immigrant education and addressing the challenges posed by the COVID-19 pandemic. Importantly, these recommendations are aligned with the user’s previous decisions, reflecting the salience of these topics during the run-up to the 2020 US election, and potentially foster further interactions between agents. Notably, this approach maintains a nearly constant prompt size throughout the simulation and enables the integration of various recommendation strategies. In our work, we focus on preference-based recommendation, i.e., recommending content aligned with agents’ preferences, and random recommendation, i.e., exposing agents to random content and thus more diverse viewpoints. We will empirically assess the impacts of these strategies in our experiments.

It is important to clarify that these recommendations are not actual recommendation methods like collaborative filtering but rather a simplified approximation of them. We will empirically assess the impacts of these strategies in our experiments. Finally, the Interaction Module provides the Reasoning Module with a series of contents published by other agents to inform the LLM about the environment’s state and other agents’ publications, ensuring continuous updating of the environment’s state regarding agents’ decisions. Upon querying all agents, the current iteration concludes, and the next begins, following a round-robin logic involving sequential querying of each agent. This iterative process continues until the simulation’s end, emulating social network evolution based on individual agent actions.

We employ YAKE [6] and KeyBERT [26] algorithms to extract keywords using statistical features and contextual embeddings, respectively. Specifically, we focus on original content and merge the vocabulary used by all agents during simulation as well as the vocabulary used by real users. This approach is motivated by the significant diversity in the scale of simulation activities compared to real users, where real users publish much more original tweets than their LLM-agents’ counterparts. Table 1 shows the top-10 keywords used by LLM-agents and real Twitter users, demonstrating that the simulation, and consequently the LLM-agents, align with the primary discussion topics of real Twitter conversations, particularly the debates between Trump and Biden during the 2020 US election. Additionally, Figure 3 shows the distributions of keywords usage in both real Twitter discussions (in blue) and the simulation (in red). The visibly right-skewed distributions, coupled with statistical validation through a Mann–Whitney test ($p$-value$<0.05$), affirm the simulation’s fidelity in reflecting natural language patterns. This observation underscores the simulation’s ability to accurately mimic the overarching themes of Twitter conversations, with few keywords being frequently utilized while others are less prevalent. Collectively, these results demonstrate that LLM-agents effectively capture the main topic of the conversation, indicating the robustness of the simulation in replicating real-world discourse.

We examine the political leaning of LLM-agents as a proxy for their (political) interests. Initially, we fine-tuned a BERT-base transformer using the annotated dataset of 100 Twitter users, achieving impressive performance metrics: 91% accuracy and 94% F1 score. Subsequently, we utilized this model to predict the political leaning scores of LLM-agents at the simulation’s conclusion. As previously mentioned, we investigate two exposure strategy: preference-based recommendation and random recommendation. To enhance robustness, we repeated the random recommendation simulation three times. Notably, the RAG-enhanced Interaction Module allows a seamless integration of any recommendation strategy. Table 2 shows a strong positive correlation between the political leaning scores of real users and LLM-agents⁶⁶6We employed the Spearman index with a significance level of $p$-value$<0.05$ to verify this correlation.. Specifically, the majority of agents (always $\geq 86\%$) has retained the political orientations of the real users they are instructed to impersonate. Interestingly, this is irrespective of the type of recommendation strategy. However, fewer agents alter their alignments during the simulations. After qualitative scrutiny, we have found that these shifting agents typically impersonate nonpartisan users. Therefore, exposure to diverse ideas may blur their political orientations. Altogether, this analysis indicates that LLM-agents accurately interpret real users’ political alignments, regardless of the content they are exposed to.

Lastly, we investigate the semantic similarity of the content published by LLM-agents. Specifically, we utilized a sentence-transformers model⁷⁷7https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 to extract contextual embeddings for each original content posted by LLM-agents. We then constructed a cosine similarity matrix $\mathcal{M}=\{m_{ij}=\text{sim}(c_{i},c_{j})\}$, where $c_{i}$ and $c_{j}$ represent the vector embeddings of two posts and $\mathcal{M}\in\mathcal{R}^{k\times k}$, $k$ being the total number of posts published by the agents. To evaluate the semantic similarity, we define two measures:

• •

  Self-Similarity: it assesses the similarity between posts from the same agent $A$. In other words, it represents the average cosine similarity measured on $\mathcal{M}$ considering only the agent’s original content, that is $\{m_{ij}\}|c_{i}\neq c_{j}\wedge c_{i},c_{j}$ posted by $A$;

• •

  Intra-Cluster Similarity: it assesses the similarity between posts from an agent $A$ and those from other agents within the same political group $\mathcal{A}_{p}=\{A_{1},A_{2},\cdots,A_{k}|A_{i}\neq A\wedge pol(A_{i})=pol(A)\}$, $pol$ being the political leaning scoring function. Concretely, the metric is determined by averaging the cosine similarity of $\mathcal{M}$’s elements, that is $\{m_{ij}\}|c_{i}$ and $c_{j}$ posted by $A$ and $A_{i}\in\mathcal{A}_{p}$, respectively.

Figure 5 shows the distributions of Self-Similarity for real users and LLM-agents across two simulation scenarios: one featuring preference-based recommendation and the other with random recommendation. Notably, for the real case, the cosine similarity matrix is built considering the original tweets of the users. We notice a clear disparity in self-similarity between LLM-agents and real users within the preference-based recommendation framework. Specifically, the median self-similarity for real users (0.274) contrasts with that of LLM-agents (0.531), with the latter exhibiting significantly higher self-similarity⁸⁸8This finding was validated using a Mann–Whitney test ($p$-value$<0.05$). This observation indicates a tendency for the same LLM-agent to converge more strongly around similar topics compared to the real user it is impersonating, suggesting that preference-based recommendation might penalise the ability of LLM-agents to engage with brand-new content. Conversely, random recommendations appear to mitigate this polarization trend and, surprisingly, achieve a better approximation of the real case. Additionally, Figure 5 replicates the same analysis but considers Intra-Cluster Similarity, revealing minimal distinctions among the three distributions. Surprisingly, the simulation employing preference-based recommendation aligns better with the real scenario compared to the previous analysis. This suggests that although the agents may not publish as diverse content as their real counterparts, they still preserve the overall semantic characteristics of their community. This outcome, combined with the previous discovery that the majority of LLM-agents retain the political leaning of their real counterparts, implies an accurate representation of the target users and an effective portrayal of their respective communities by the LLM-agents.

Table 2 shows the interaction patterns in terms of the percentages of actions performed by the agents. The influence of the recommendation system is evident: in simulations featuring random recommendations, agents exhibit a significant decrease in content resharing from others ($\leq 8.4\%$) compared to simulations employing preference-based recommendations (50.6%). This trend stems from the increased exposure to (random) content not aligned with the agent’s preferences. Indeed, the random recommendation strategy results in agents publishing original content approximately 90% of the times in an attempt to push their ideas in the environment. Surprisingly, while personalized recommendations usually boost user engagement on real social media platforms [19], our simulations reveal a greater proportion of non-interactions when recommendations are preference-based (10.5%) with respect to random recommendation ($\leq 2.5\%$). We attribute this to our empirical observation that the preference-based recommendation is not flawless and may sometimes suggest irrelevant posts to the agents. Overall, these findings suggest that the recommendation strategy strongly affects the agents’ activity, with preference-based recommendations fostering increased interactions among agents.

To investigate homophily, we examine the above-mentioned interaction graph among agents in each simulation. Homophily, defined as individuals’ inclination to associate with others sharing similar attributes, leads to the formation of homogeneous groups [22]. Given the political context of our data, we focus on agents’ political alignment to delineate group clusters. Subsequently, we analyze the inter-cluster edges of the interaction graph, representing connections between nodes from different clusters. The results of the homophily analysis presented in Table3 indicate that networks from simulations employing preference-based recommendations exhibit reduced inter-cluster connectivity, signifying a stronger homophily effect. This results in a 10% lower number of inter-cluster edges than the number needed to declare the network non-homophilic [14]. Statistical analyses confirm the significance of these homophily values (p-value $<$ 0.05). Conversely, simulations using random recommendations demonstrate relatively higher inter-cluster connectivity (+4% inter-cluster edges) with respect to preference-based recommendation, resulting in networks that are not statistically homophilic (p-value $>$ 0.05). These results affirm our previous findings regarding the determining effect of the recommendation strategy, but also complement them by illustrating that preference-based recommendation not only fosters agents’ engagement but also encourages the formation of distinct clusters with limited inter-cluster interaction.

In line with previous works [4, 1, 10], we utilize established metrics to examine the emergence of echo chambers in our simulations. These metrics analyse the interaction graph to evaluate the level of controversy within discussions. Specifically, we utilize Random Walk Controversy (RWC) to analyze transition probabilities between ideological clusters, Betweenness Centrality Controversy (BCC) to assess partition distances, and Boundary Connectivity (GMCK) to compute the structural arrangement of the interaction graph [10]. In all cases, the higher the metric, the more controversy is the interaction graph. The underlying premise is that contentious topics often involve individuals with contrasting viewpoints engaging in dialogue, while individuals sharing similar beliefs tend to reinforce each other’s arguments [1]. Results in Table 3 unveil a notable prevalence of controversy in preference-based simulations, indicating a stronger inclination towards echo chamber formation. Conversely, random recommendation mitigates echo chamber formation, promoting a broader diversity of opinions. These findings underscore that LLM-agents within a community are unlikely to interact with individuals holding opposing viewpoints, suggesting that the recommendation strategy not only influences community formation but also reinforces these communities by fostering closed networks, wherein users are exposed to limited diversity.