Toward Model-centric Heterogeneous Federated Graph Learning: A Knowledge-driven Approach

Federated Graph Learning (FGL) applies the principles of federated learning to graph neural networks, enabling collaborative training on graph-structured data while preserving data privacy. FGL allows multiple clients to jointly learn graph representation models without exchanging original graph data. The predominant FGL research can be categorized into two settings: (i) Graph-FL includes GCFL [20] and FedStar [21], where each client collects multiple independent graphs to collaboratively address graph-level downstream tasks (e.g., graph classification). (ii) Subgraph-FL [11, 12, 22, 23], where each client holds a subgraph of an implicit global graph, aims to tackle node-level downstream tasks (e.g., node classification). Notably, previous studies have primarily focused on addressing challenges related to varying data distributions [11, 12] across different clients. However, in real-world scenarios, graph neural networks from different clients may also belong to different architectural models. We are the first to investigate this issue, aiming to design new paradigms for heterogeneous client collaboration.

Model-centric heterogeneous federated learning (MHtFL) accommodates the need for personalized model architectures while offering advantages in privacy protection and model customization. Based on the level of model heterogeneity, we classify the existing MHtFL methodologies into two categories: partial heterogeneity and complete heterogeneity. Methods based on partial heterogeneity, such as LG-fedAvg [13], FedGen [24], and FedGH [14], permit the primary components of client models to be heterogeneous, although they assume the remaining (smaller) components to be homogeneous. However, clients can acquire limited global knowledge through this small shared component. In contrast, completely heterogeneous MHtFL approaches impose no architectural restrictions on client models. Classic knowledge distillation-based methods [25, 26] share model outputs over a global dataset. However, acquiring such datasets in practice can be challenging. FML [17] and FedKD [18] do not rely on global datasets but instead leverage mutual extraction between small auxiliary global models and client models. Nevertheless, in early iterations, there is a risk of exchanging ineffective knowledge when both the auxiliary and client models perform poorly. Another strategy involves sharing class prototypes [15, 16, 27]. However, biased classifier phenomena have been widely observed in federated learning when handling heterogeneous data. This bias becomes more pronounced when the models exhibit heterogeneity, leading to biased prototypes, which challenges the aggregation of global knowledge for classification [28]. Notably, the aforementioned methods have not been tailored for the graph data domain. To address the challenges of model-centric heterogeneous federated graph learning (MHtFGL) scenarios, our approach innovatively employs knowledge distillation techniques to facilitate the transfer of graph information within heterogeneous federated contexts.

Knowledge distillation [29] was originally proposed to transfer knowledge from larger, more complex models to smaller, more efficient ones to conserve computational resources. In graph learning, knowledge distillation can facilitate knowledge transfer across different models, where the key lies in determining what knowledge to distill and how to do so. Three types of information can be regarded as transferable knowledge: output logits, graph structure, and embeddings. Logits represent the inputs to the final Softmax layer, and some methods aim to minimize the differences between probability distributions from different models after computing logits to extract knowledge [30, 31, 32]. The graph structure delineates the connectivity and relationships between graph elements (such as nodes and edges), with some approaches aligning nodes based on their neighboring relationships to extract structural information [33, 34]. In addition to the logits and graph structures, there are studies that utilize node embeddings learned from intermediate layers of models to guide the training of another model [35, 36, 37]. Regarding distillation methods, direct distillation minimizes differences between models, promoting intuitive knowledge transfer [31]; adaptive distillation offers flexibility by considering the importance of different knowledge components. Methods like RDD [38], FreeKD [39], and MulDE [30] adjust the distillation process based on the certainty or correctness of the knowledge, allowing models to focus more on reliable information while avoiding knowledge that is less informative or noisy. Furthermore, various distillation strategies tailored for specific tasks exist within machine learning [36]. Our study aims to ensure the preservation and transfer of effective graph information within heterogeneous federated scenarios, thereby facilitating collaborative training among multiple clients.

In this study, we propose a model-centric heterogeneous federated graph learning framework designed to leverage diverse graph data and GNN models from multiple clients for collaborative training, while ensuring the protection of data privacy.

Let there be $K$ clients, where the graph data owned by client $k$ is represented as $G_{k}=(V_{k},E_{k})$ with $|V_{k}|=N_{k}$ nodes and $|E_{k}|=M_{k}$ edges. The graph’s adjacency matrix, node feature matrix, and label matrix are denoted as $A_{k}\in\mathbb{R}^{N_{k}\times N_{k}}$, $X_{k}\in\mathbb{R}^{N_{k}\times f}$, and $Y_{k}\in\mathbb{R}^{N_{k}\times C}$, respectively, where $f$ represents the number of node features and $C$ denotes the number of classes. Note that the graph data and the GNN model architectures can vary among different clients. GNNs constitute a class of deep learning models tailored to perform feature embedding and inference tasks on graph-structured data, requiring an adjacency matrix $A_{k}$ and node features $X_{k}$ as inputs. Federated learning represents a collaborative machine learning paradigm that enables the training of a shared model across multiple data owners without the direct exchange of raw data. Each training round involves the selection of participating clients, the update of the local model via local training on each chosen client, the aggregation of local model parameters on a central server, and the subsequent distribution of the aggregated parameters back to the clients. The objective of this work is to enhance the performance of each client’s local model on graph node classification tasks in the heterogeneous federated environment.

Figure 2 illustrates an overview of the FedGKC architecture that we designed. At first, the parameters of all models are initialized. Then, in each round, the following steps are executed: (1) Client local models and their copilot models are trained using Self-Mutual Knowledge Distillation (SMKD); (2) At the server, Knowledge-Aware Model Aggregation (KAMA) is performed on the copilot models; (3) The aggregated model parameters are distributed back to all client copilot models. Steps (1) to (3) are repeated for multiple rounds until convergence, allowing each client’s local model to learn global knowledge without transmitting graph data, providing power for optimizing towards the global direction.

The model learning process on the client side consists of two main components. Firstly, there is a mutual knowledge learning process between the local model and the copilot model. This process not only allows the copilot model to acquire more localized knowledge but also enables the local model to obtain global knowledge that includes information from other clients. Secondly, to enhance the predictive capability of the local model, we generate two different views using data augmentation from the same graph data to implement self-distillation. This ensures that the local model generates consistent embedding features and prediction results for the same data across different views.

Mutual distillation addressing model heterogeneity: In traditional knowledge distillation methods [32], knowledge is typically transferred by aligning the prediction results of different models on the same data. However, merely aligning the prediction results may limit the knowledge learned by the models, resulting in a lack of comprehensiveness and representativeness. In the context of graph data processed by GNNs, node representations are primarily derived from capturing neighborhood information. Therefore, we propose propagating neighborhood knowledge from one model to another model to more effectively leverage the topological information captured by GNNs, formulated as follows:

where $z$ and $h$ represent the feature embedding outputs of the feature extractor of the two models, respectively, $\mathcal{V}_{i}$ represents the set of neighbor nodes of node $i$, $\mathcal{N}$ denotes the total number of nodes, $\sigma=softmax(\cdot)$ denotes an activation function, and $\mathcal{D}(\cdot)$ denotes a distance function, such as the Kullback Leibler (KL) divergence function [29]. For the copilot model, its optimization objective can be denoted as:

where $\alpha$ and $\beta$ are the weights to balance the influence of different distillation losses, and $\mathcal{L}_{CE}(y,p_{c})=-y\log(softmax(p_{c}))$ represents the cross-entropy loss between the copilot model prediction $p_{c}$ and the truth label $y$. Here, $\mathcal{L}_{KL}$ is the Kullback Leibler (KL) divergence function, and $p_{l}$ and $p_{c}$ are the predicts of local and copilot models, respectively.

Similarly, the local model needs to learn information from the copilot model while training on local data, and its optimization objective is

where $\alpha$ and $\beta$ are the weights to balance the influence of different distillation losses, and $p_{l}$ and $p_{c}$ are the predicts of local and copilot models, respectively.

Self-distillation mining model potential: To further stimulate the predictive performance of the local model, we introduce a self-distillation strategy. We first create two perspectives on the same graph data — a strongly augmented view and a weakly augmented view. This dual-perspective strategy effectively enhances the diversity of graph data by applying varying degrees of edge removal and feature masking [40]. The weakly augmented view retains more original information, while the strongly augmented view significantly obscures and removes more graph information. We expect the model’s output based on the strongly augmented view to closely align with the prediction results of the weakly augmented view for the same input. This aims to ensure that, even when faced with substantial information alteration, the local model maintains consistency in its prediction results, thereby deepening its understanding of the data’s intrinsic nature. The optimization objective for the self-distillation of the local model is denoted as

where $\mathcal{L}_{MSE}$ denotes the mean-square error loss, and $e$ and $p$ represent the embedding output of the model feature extractor and the prediction output of the classifier, respectively.

In summary, the overall optimization objective for the local model integrates the mutual distillation objective with the self-distillation objective:

The optimization goal for the copilot model is shown in Eq. (2). Through the coordinated optimization of these two components, we aim to significantly enhance the learning efficiency and global knowledge acquisition capability of the local model in heterogeneous environments.

Given the heterogeneity among clients, the learning efficiency of each client’s model varies accordingly. Traditional federated learning methods typically employ an averaging aggregation strategy [19], which does not take into account the differences among the models from various clients. This oversight can result in lower-performing models adversely affecting overall performance. To address this issue, we propose a Knowledge-Aware Model Aggregation mechanism that dynamically assigns weights based on the information content and learning knowledge levels of each client’s model. This dynamic weighting aggregation enhances collaborative efficiency by ensuring that the models from clients with rich knowledge do not suffer from the shortcomings of those with limited knowledge. Simultaneously, it allows clients with lesser knowledge to receive more guidance from the knowledgeable clients, thereby improving the model’s convergence and robustness.

Specifically, when aggregating the copilot models from each client, the server first considers the data volume from an overall perspective and calculates volume weight accordingly, denoted as:

where $\mathcal{N}_{k}$ indicates the amount of data on the $k$-th client.

Next, it assesses the knowledge levels learned by each node in the copilot model. Ideally, each node should focus on relevant knowledge associated with its respective category, implying that the knowledge should be strong and clear. We measure knowledge level from two dimensions: knowledge strength and knowledge clarity. The knowledge strength at a node can be represented by the maximum value of the output prediction probabilities, which intuitively reflects the model’s confidence in recognizing a particular category, denoted as:

where $\text{max}(\cdot)$ returns the max value of the predict vector $p$ for the $i$-th node. On the other hand, assessing knowledge clarity is more complex as it involves evaluating the distinction between the predicted results of this node and those of other categories, expressed as $\text{max}(p_{i})-\sum{p_{i}^{*}}$, where $p^{*}$ represents the set of values other than the maximum value. Additionally, we must be wary of the over-smoothing phenomenon that can occur in GNNs due to interactions with neighboring nodes, which may compromise the model’s discriminatory ability. We define the clarity evaluation formula as:

where $M$ represents the number of categories, $\mathcal{V}_{i}$ represents the set of neighbor nodes of node $i$, $\lambda$ is the weight of the over-smoothing effect, and $\text{simi}(\cdot)$ represents the cosine similarity calculation function. Then, the degree of knowledge mastery of each client is quantified by:

where $\mathcal{N}_{k}$ indicates the amount of data on the $k$-th client. The knowledge-related weights are expressed as:

By considering data volume, node knowledge strength, and node knowledge clarity, we quantify the learning knowledge level of each client’s copilot model. Consequently, the aggregation weight for the $k$-th client’s model can be determined as follows:

The model aggregation formula is:

where $\theta_{c,k}$ is the parameters of the $k$-th client copilot model.

Finally, the aggregated model parameters will be distributed back to each client’s copilot model in preparation for the next round of client training. Through this meticulous adjustment, our aggregation strategy effectively integrates multi-source information, achieving a more efficient and equitable model aggregation process. The complete algorithm of FedGKC is presented in Algorithm 1.

To answer Q1, we conduct comprehensive comparative experiments to demonstrate the superior performance of FedGKC. Specifically, we test two scenarios under heterogeneous settings and further comparisons under non-heterogeneous settings. 1) Clients with Different Model Architectures. In the first scenario, we evaluate the performance of FedGKC in a setting where client models have different architectures. The experimental results, detailed in Table II, show that FedGKC significantly outperforms multiple baseline methods in this scenario. Knowledge distillation-based methods, like FML and FedKD, have demonstrated performance advantages in heterogeneous environments. In comparison to these methods, FedGKC achieves greater performance enhancements, with an average improvement of 2.31% over FML and 2.34% over FedKD. This indicates that FedGKC not only effectively handles model architectural heterogeneity but also achieves better knowledge transfer and model fusion across different architectures. 2) Clients with Different Model Scales. In the second scenario, we consider the variations in model scales due to differences in the computational capabilities of various client devices. As shown in Table III, FedGKC demonstrates superior performance on the test set compared to other methods, with an average of 1.80% higher accuracy than FML and 2.17% higher accuracy than FedKD. It is confirmed that FedGKC can not only adapt to the heterogeneity of simple model architecture, but also support and perform well for models of different scales. 3) Comparative Experiments under Non-Heterogeneous Settings. In addition to the experiments under heterogeneous settings, we also conduct comparative experiments in non-heterogeneous settings to evaluate the performance of FedGKC in traditional federated learning environments. The results, presented in Table IV, show that despite being specifically designed for heterogeneous client scenarios, FedGKC performs exceptionally well even in non-heterogeneous settings, outperforming the baseline approach FedAvg by 10.5% on average. FedGKC even surpasses federated learning methods tailored for homogeneous clients by an average of 1.07% over the suboptimal method Fed-Pub, which further validates the generality and robustness of FedGKC.

To answer Q2, we conduct a series of ablation experiments focusing on two critical modules introduced in FedGKC: Self-Mutual Knowledge Distillation (SMKD) and Knowledge-Aware Model Aggregation (KAMA). Table V illustrates the performance enhancements achieved by each module when applied independently to the baseline method. In terms of implementation, when the SMKD module is added alone, we employ an aggregation strategy based on sample size on the server; when incorporating the KAMA module alone, a traditional dual-model architecture is utilized on the clients. Notably, the SMKD module demonstrates greater improvements on the Cora and CiteSeer datasets, while the KAMA module shows a larger contribution to the PubMed dataset, suggesting its advantageous performance with larger scale datasets. The combination of both modules further enhances overall performance.

To gain deeper insights into the contributions of the internal mechanisms of the SMKD and KAMA modules, we conduct a more detailed ablation study on the CiteSeer dataset with 10 clients, with the experimental results presented in Fig. 3(a). First, within the SMKD module, we test the effects of removing self-distillation and mutual distillation on model performance. The results indicate that the absence of the self-distillation mechanism limits the model’s self-optimization capabilities, while neglecting mutual distillation leads to the model relying solely on local data, thereby failing to effectively assimilate global information and diminishing overall performance. Next, within the KAMA module, we focus on the elements that constitute knowledge weights, including knowledge strength (as shown in Eq. (7)), the clarity of class-related knowledge (the first term of Eq. (7)), and the clarity of smooth associations (the second term of Eq. (7)). The experimental results reveal that knowledge strength significantly impacts model performance, whereas the effect of knowledge clarity is relatively minor; nevertheless, it still contributes positively to the model’s stability and accuracy.

To answer Q3, we investigate the impact of replacing different architectures of the copilot model on overall performance, as illustrated in Fig. 3(b). Our FedGKC is not restricted to specific copilot model architectures. In fact, our method demonstrates significant performance improvements across most widely adopted model architectures, further validating the flexibility and efficacy of FedGKC.

To answer Q4, we evaluate the performance of FedGKC using 20 clients from the CS dataset across various hyperparameter settings. Specifically, we explore different combinations of the trade-off parameters $\alpha$ and $\beta$ in Eq. (2) and Eq. (3). The experimental results are shown in Fig. 4(a), where we find that the method performs optimally with $\alpha$ set between 0.6 and 0.7, and $\beta$ maintained within the range of 0.1 to 0.2. Based on these findings, we finalize our hyperparameters with $\alpha=0.6$ and $\beta=0.2$. Additionally, we conduct a sensitivity analysis on $\lambda$ in Eq. (8), which represents the influence of over-smoothing on the clarity of node knowledge. From Fig. 4(b), we find that setting $\lambda$ to 0.1 effectively enhances model performance, whereas excessively high values lead to performance deterioration, underscoring the importance of balancing node similarity and the over-smoothing issue.

To address Q5, a computational complexity analysis of FedGKC is provided herein. Each client employs a GNN as the local model. For an $l$-layer GNN model with a batch size $b$, the propagated feature $X^{(l)}$ has a space complexity of $\mathcal{O}((b+l)f)$, and the overhead for linear regression operations is $\mathcal{O}(f^{2})$, where $f$ are the number of feature dimensions. The introduction of the copilot model necessitates double the parameter storage space, leading to an overall space complexity of $\mathcal{O}(2(b+l)f+2f^{2})$. Furthermore, the inclusion of additional alignment losses within the self-mutual knowledge distillation module introduces a time complexity of $\mathcal{O}(f^{2})$. For the server, the reception and aggregation of model parameters impose space and time complexities of $\mathcal{O}(Kf^{2})$ and $\mathcal{O}(K)$, respectively, where $K$ represents the number of clients involved.