BotSCL: Heterophily-aware Social Bot Detection with Supervised Contrastive Learning

Previous approaches to social bot detection primarily involve manual analysis of collected data and extraction of distinctive characteristics for input into diverse classifiers(Yang et al., 2022a). Subsequently, deep neural networks with diverse architectures are developed to enhance classification performance(Kudugunta and Ferrara, 2018; Wu et al., 2021). However, the detectability of these characteristics is vulnerable to imitation and evasion by social bots, rendering them ineffective over time. To tackle the challenge of bot disguise, graph-based social bot detection methods have been extensively studied and shown great success in social bot detection with the advent of benchmark datasets that incorporate graph information(Feng et al., 2021c, 2022b).

(Ali Alhosseini et al., 2019) is the first attempt to introduce graph convolutional networks to take advantage of both the features of the accounts and the structure of the relation graph, which takes Twitter users as nodes. Satar (Feng et al., 2021b) utilizes graph convolutional networks in a feature engineering manner and employs self-supervision to detect social robots. Relational graph convolutional networks (R-GCNs) (Schlichtkrull et al., 2018) were used by (Feng et al., 2021a) to aggregate information from different relations and (Feng et al., 2022a) later improved it with additional relations and apply graph transformer to better aggregate information from neighbors adaptively. (Yang et al., 2022b) proposed RoSGAS, a framework that leverages heterogeneous information network to effectively model multiple entities and relationships within a social network and performs subgraph embedding with reinforcement learning for social bot detection.

All of these methods are based on the assumption that both humans and bots tend to interact more with the same class, and classification benefits from smoothing the representations of neighboring nodes. However, in reality, advanced bots can easily escape graph-based detection by actively interacting with humans because none of these approaches take heterophily into account. In this paper, we propose a bot detection framework that recognizes the negative impact of heterophilic relations between bots and humans for more effective user representations to enhance the performance of bot detection.

Due to the widespread existence of heterophily, graph neural networks for graphs with heterophily have received significant attention in recent years. Essentially, there are mainly two kinds of approach: 1) Aggregation of non-local neighbor information. For example, some techniques gather information from higher-order neighbors (Abu-El-Haija et al., 2019; Zhu et al., 2020b) and potential neighbors of the same class (Pei et al., 2020; Wang et al., 2022) to obtain more intra-class information. 2) Adaptive Message Passing. FAGCN (Bo et al., 2021) simultaneously aggregates high- and low-frequency information. GPRGNN (Chien et al., 2020) employs learnable weights for information aggregated from distinct hop neighbors. (Luan et al., 2021) argues that not all heterophily edges are harmful for classification and proposes Adaptive Channel Mix (ACM) to adaptively aggregate self-information, low-frequency information, and high-frequency information. Despite the emergence of numerous heterophilic GNNs, these methods primarily focus on single-relation and undirected simple graphs, particularly spectral-based GNNs. In social bot detection, bots exhibit a higher tendency towards heterophily, while humans display a higher inclination towards homophily, respectively. Therefore, directly applying heterophilic GNNs to social bot detection may not yield optimal results.

Contrastive learning aims to learn an encoder that can generate representations consistent with different views by attracting positive pairs and repelling negative pairs. Graph contrastive learning (GCL) extends the technique to graph domain and learns the representations of nodes in a self-supervised manner due to rich information implicitly existing in the connections between nodes. GRACE (Zhu et al., 2020a) applies edge removal and feature mask to generate augmented views and treat the same node in different views as a positive pair. DGI (Velickovic et al., 2019) learns node representations by maximizing the mutual information between local and global embeddings. However, self-supervised contrastive learning methods face class collision problem – the representations of similar samples may be far apart, while the representations of dissimilar samples may be close (Zheng et al., 2021). Supervised contrastive learning (Khosla et al., 2020) firstly applied in computer vision field treats intra-class images as positive pairs, while inter-class images as negative pairs. Thus, embeddings from the same class are pulled closer than embeddings from different classes. Because this property is opposite to the feature mixing caused by message propagation on heterophilic edges, we utilize supervised contrastive loss to train an encoder that can adapt to both homophilic and heterophilic edges simultaneously to aggregate information beneficial for further classification.

Graph contrastive learning employs multiple graph augmentation strategies, including edge addition, feature masking, and personalized pagerank diffusion, to generate various graph views (You et al., 2020). It should be noted that not all graph augmentation methods can be universally applicable to graphs with heterophily. (Yang and Mirzasoleiman, 2023) uses a low-pass filter and high-pass filter to generate graph views for self-supervised contrastive learning. However, this approach may result in the absence of directional information on relations and higher-order neighbor information. (Liu et al., 2022) first determines whether each edge belongs to within-class or between-class, and then generates a homophilic view and a heterophilic view for contrastive learning. However, error propagation will inevitably happen. This paper aims to use supervised contrastive learning loss to enable simultaneous perception of both heterophilic and heterophilic edges, instead of employing extra differentiation between the two.

As supervised contrastive learning leverages label information during the training process, we propose a class-aware node shuffling (CNS) graph augmentation method that randomly swaps nodes belonging to the same class. By using this approach, one can learn representations that are invariant to the neighborhood without excessively disrupting the graph structure. This graph augmentation method can be implemented through intra-class feature swapping:

Additionally, we also employed a traditional graph augmentation method, edge removing (ER), to augment the graph topology. The edge removal method can be formulated as follows:

where $\mathcal{E}$ is total edge set, each element of $M^{\mathcal{E}}$, $m_{ij}^{\mathcal{E}}$ stands for the mask value of edge $e_{ij}$ and $m^{\mathcal{E}}_{ij}\sim\mathcal{B}(1-pe)$, $pe$ is the probability to remove. Through the aforementioned two graph augmentation methods, both the features and topological structure of the original graph are augmented.

Following the traditional contrastive learning framework, we use a projection head consisting of two MLP layers to obtain $z_{i}$:

where $h_{i}^{L}$ is the output of last layer $L$. Thus, we can obtain projections $z_{i}^{\alpha}$ and $z_{i}^{\beta}$ of node $v_{i}$ in two graph views $\mathcal{G}^{\alpha}$ and $\mathcal{G}^{\beta}$ in respect.

Next, we use supervised contrastive learning as the objective loss function for training. However, due to the fact that supervised contrastive learning treats all nodes of the same class in different views as positive pairs, it is prone to overfitting, where the representations of all same-class training nodes become too similar and cannot generalize to test nodes. To avoid overfitting, we use supervised contrastive learning in a cross-view manner. For a set of N samples randomly sampled from the training nodes, we obtain the projection of each node through the graph augmentation, encoder, and projection head mentioned above. Taking node $v_{i}$ in graph view $\mathcal{G}^{\alpha}$ as an example, we treat its projection and the projections of same-class nodes in the other view as positive pairs, and those of different-class nodes as negative pairs, to calculate the contrastive loss between them:

where $N_{y_{i}}$ represents the number of samples in the same class as node $v_{i}$ among $N$ samples, $cos(\cdot,\cdot)$ function is used to calculate the cosine similarity, and $\tau$ is the temperature coefficient which can regulate the degree of distribution uniformity.

Finally, we calculate the loss for all nodes in the sampled set of two views in the same way, and take the average:

Before answering RQ1, it is necessary to analyze the degree of homophily and heterophily in two datasets. As shown in Table 2, bots exhibit a evident heterophily tendency in the follower relation compared to human, while they display a stronger homophily tendency in the following relation in TwiBot-20. For TwiBot-22, bots show a high degree of heterophily in both relations, which can be attributed to the fact that the number of bots in the graph is significantly smaller than humans and bots are often surrounded with humans. This indicates that there is indeed an active tendency for social bots to interact more with humans. To answer RQ1, we visualize the curve of changes in accuracy as heterophilic edges are removed. We conduct experiments using four different models: GCN, GAT, BotRGCN, and RGT. As shown in Figure 3, the accuracy of all models gradually increases as the proportion of heterophilic edges decreases with a step size of 0.1. Furthermore, we can observe that after all heterophilic edges are removed, the performance of different models is almost the same. We suspect that the core difference between these methods lies in their varying abilities to adapt to heterophily. This experiment validates that heterophilic edges are detrimental to social bot detection. Therefore, it is necessary to consider both homophilic and heterophilic edges in graph-based social bot detection.

Table 4 summarizes the detection results of all the baseline methods and our BotSCL on TwiBot-20 and TwiBot-22. It is evident that BotSCL outperforms all the other 14 baselines in both datasets. The superior performance also indicates the significance of modeling both homophily and heterophily in finding advanced bots with the capability of actively building heterophilic edges to evade detection.

Firstly, compared to homophilic GNNs and previous graph-based social bot detection methods which simply treat interactions in social bot detection as homophilic, BotSCL has better adaptability to distinct neighborhood distributions associated with homophily and heterophily. According to Table 4, heterophilic GNNs perform better than homophilic GNNs. In addition, heterophilic GNNs achieve performance similar to previous graph-based detection methods even without the leverage of multi-relation information. Taking heterophilic edges into consideration can help uncover highly sophisticated bot accounts that are closely connected with humans.

In contrast to heterophilic GNNs, BotSCL not only considers different types of relations, but also uses supervised contrastive loss as the optimization objective, leading to improvements in both accuracy and f1-score in the two datasets. Experiments conducted by (Feng et al., 2022a) have shown that information from different types of relations can enhance model classification ability. Furthermore, with supervised contrastive learning, BotSCL enforces the aggregation of homophilic neighbor information and the differentiation of heterophilic neighbor representations.

The self-supervised contrastive learning methods perform poorly on TwiBot-20 with a large number of unlabeled nodes, while performing better on TwiBot-22. We think that the reason is due to the issue of uneven training distribution, as the second training stage uses fewer training nodes on TwiBot-20. Our method is significantly superior to these self-supervised methods, indicating that supervised signals play a vital role in adapting to both homophilic and heterophilic edges in social bot detection.

To answer RQ3, we conduct an ablation study to investigate the effects of different modules and graph augmentation methods on social bot detection. Specifically, we build three ablation models: w/o Sup, w/o Neg, and CE, developed through self-supervised contrastive learning, removal of negative information aggregation, and cross-entropy loss. We also evaluate four graph augmentation methods: CNS, Edge Adding (EA), ER, and Feature Masking (FM). The results on TwiBot-20 and TwiBot-22 are shown in Table 5.

Compared to BotSCL, w/o Sup shows significant performance degradation, indicating that self-supervised methods struggle to aggregate similar or dissimilar information during message passing without labeled signals. The w/o Neg variant, using the softmax function for attention coefficients, aggregates similar information from neighbors but still performs well due to supervised contrastive learning, which limits aggregation from heterophilic neighbors. The CE variant suffers a notable performance drop, as cross-entropy loss focuses on classification, ignoring exceptional cases.

Additionally, different graph augmentation methods appear to have little impact on model performance, except for CNS and ER which are employed by BotSCL and show better performance. Under supervised conditions, CNS plays a crucial role in generating category-invariant representations. Among these different graph augmentation methods, CNS and ER, which BotSCL adopts, demonstrate better performance comparatively. FM and EA may potentially disrupt node feature information to some extent or introduce new noise.

To answer RQ4, we evaluate the performance of BotSCL with regards to hyperparameters $\lambda^{\{1\}}$ and $\lambda^{\{2\}}$, as we employ two layers of information aggregation. We keep the other parameters in the model unchanged and control the two parameters ranging from 0.1 to 1.0 with a step size of 0.1. The experimental results on TwiBot-20 and TwiBot-22 are shown in Figure 4a and Figure 4b, respectively.

From Figure 4, we can clearly observe that as the hyperparameters $\lambda^{\{1\}}$ and $\lambda^{\{2\}}$ increase, the accuracy of the model on both TwiBot-20 and TwiBot-22 gradually improves. Especially when both hyperparameters are set to 1, the model exhibits the strongest classification ability. This implies that in social bot detection, preserving the information of individual nodes is crucial for classification. Furthermore, we have found that BotSCL achieves similar performance when hyperparameters $\lambda^{\{1\}}$ and $\lambda^{\{2\}}$ have symmetric values. This could be because when they are set to symmetric values, their product remains the same, and the preservation of self-feature weights is equivalent after two layers of information aggregation. From Figure 4b, it can be observed that as the hyperparameter decreases to 0.8, there is a significant drop in accuracy on TwiBot-22. On the other hand, on TwiBot-20, this decrease is more gradual, and a steep drop is only observed when the hyperparameter decreases to nearly 0.4. Compared to TwiBot-20, TwiBot-22 includes labels for all nodes and suffers from the class imbalance issue, where the number of instances for each class is uneven. If the weights assigned to the original information are relatively small, the representation of the ego node is susceptible to being overwhelmed by neighbor information.

Overall, regardless of the variations in hyperparameters, the accuracy changes within the range of 0.04 for TwiBot-20 and within the range of 0.01 for TwiBot-22. This indicates that BotSCL is not highly sensitive to hyperparameters $\lambda^{\{1\}}$ and $\lambda^{\{2\}}$, although the hyperparameters do have some impact on the model’s performance.

To answer RQ5, we visualize node representations obtained from six different models, including GCN, FAGCN, BotRGCN, RGT, DGI, and our method BotSCL. Due to the larger scale of TwiBot-22, it is more representative of real-world social networks. Therefore, we take it as an example for visualization. First, we obtain node representations for each model using the same implementation method as in the comparison experiments. Then, we employ the t-SNE (Van der Maaten and Hinton, 2008) to reduce the dimensionality of the representations to 2D space for visualization purposes. To facilitate better observation, we randomly select 1000 human nodes and 1000 bot nodes from the test set for visualization. The visualization results of these 6 models are shown in Figure 5.

From Figure 5, we can see that node representations obtained by homophilic GNNs (GCN, BotRGCN, RGT) are more scattered compared to those from FAGCN. These methods based on the homophily assumption can only smooth the representations of neighboring nodes without differentiation. As a result, the generated representations are primarily composed of local information, leading to a more scattered distribution. Among homophilic GNNs, RGT produces the poorest node representations on TwiBot-22, which is contrary to its performance on TwiBot-20. This suggests that the adaptive mechanism can be greatly influenced by the training data, leading to poor classification performance.

Compared to FAGCN, the node representations generated by BotSCL exhibit stronger clustering characteristics and include fewer local clusters. Although FAGCN takes into account both homophilic and heterophilic edges, using cross-entropy directly may inevitably overlook samples from less frequent distributions. Furthermore, the lack of consideration for multi-relational information is also one of the reasons for the suboptimal performance of FAGCN.

The node representations generated by DGI exhibit poor discriminative properties, as no supervised signals are used during the training process. The issue of class collision is also evident here, where the representations of bots and humans are almost mixed. In social bot detection, the problem of class collision in self-supervised contrastive learning is particularly severe. A large number of test nodes from different classes exhibit similar representations.