CDFL: Efficient Federated Human Activity Recognition using Contrastive Learning and Deep Clustering

FL is a prominent distributed machine learning approach that has attracted attention in recent years. The main goal of FL is to preserve privacy through decentralized learning. The first attempt in this area is FedAvg [9] algorithm. Although the results obtained from FedAvg are promising compared to centralized learning, this scheme suffers from several challenges, including data heterogeneity, communication overhead, and naive aggregation.

To mitigate the data heterogeneity issue, Li et al. [27] presented a model-contrastive federated learning (MOON) to similarize the representation of local models to those of the global model. While MOON performs effectively when the global model is consistent with the global objective, it overlooks the fact that a deviated global model, particularly in strongly heterogeneous conditions, can lead to deviation of all local clients from the global objective. Another category of algorithms focuses on clustered FL to manage heterogeneous states such as cFedFN [18], IFCA [20], FedCluster [28], FedGroup [29], FlexCFL [23], CFL [19]. IFCA [20] is an iterative federated clustering algorithm that alternates between estimating the cluster identities and minimizing the loss function. This method does not need a centralized clustering algorithm, leading to a considerable reduction in server computational cost. It has been shown that the convergence rate of their proposed algorithm follows the exponential curve. CFL [19] is another cluster-based FL framework that does not require any prior knowledge about the number of clusters. In this framework, the geometric properties of the FL loss are leveraged for client clustering. Further, they discovered that by calculating the cosine similarity between client weights, the similarity of their data distribution can be perceived. In cFedFN [18], Wei et al. have argued that feature norm is a suitable metric to cluster clients in view of its capability to reflect the clients’ data distribution. By leveraging feature norms for clustering, a performance improvement is achieved without the need for any pre-clustering steps. The additional computation cost required for feature norm calculation of cFedFN is negligible, which makes it applicable in real-life situations. While clustered FL methods can mitigate data heterogeneity issues by identifying underlying client clusters, clustering becomes challenging as the number of clients increases.

A number of existing works [30] have focused on communication overhead reduction to handle bandwidth-limited situations. Chen et al. [31] have designed a probabilistic client selection such that clients that can potentially enhance the convergence rate are selected with higher probability in the following communication round. Besides, in this scheme, a quantization method has been developed to shrink the model parameters before transmission to the server. In [32], an asynchronous model update algorithm is proposed to reduce client-server communication. In this algorithm, the various layers of the model are divided into shallow and deep categories. The parameters in the deep category of layers are updated less often compared to the parameters in the shallow layers. With this strategy, the number of parameters to be exchanged between the server and clients is reduced per communication round. Although their proposed strategy is successful in reducing the total communication cost, it needs more communication rounds to achieve a target accuracy compared to FedAvg. The FetchSGD algorithm, presented in [33], is based on the Count Sketch data structure to overcome the communication bottleneck issue. The FetchSGD sends the sketches of each local client to the server for aggregation. In the server, the momentum and error accumulation are applied to the aggregated sketch. Then, top-k values are selected as sparse updates to broadcast to the connected clients in the following round.

On the server side, naive aggregation of local models suffers from global drift in heterogeneous situations [26], and therefore, deteriorates the FL performance. Chen et al. [32] have designed a temporally weighted aggregation strategy to resolve this issue. This scheme is based on timestamps, and local clients with recent updates get larger weights compared to other clients. FedMA [34] has used parameter averaging of the local clients by utilizing a permutation matrix. Specifically, this scheme ensures that similar elements in the model, like channels in convolution layers or neurons in fully connected layers, are matched and averaged appropriately. However, the computational cost of finding the optimal permutation matrix hinders its practical usage. EK et al. [35] introduced a new aggregation algorithm called FedDist, which follows a similar approach to FedMA but focuses on identifying differences between specific neurons among the clients to modify the global model’s architecture. It has been demonstrated that this algorithm considers the clients’ specificity without affecting the overall performance.

Many existing works in HAR applications still rely on central servers to train their models. This exposes the users’ data to substantial privacy threats, especially when dealing with sensitive user information. To maintain user privacy, federated learning has been increasingly adopted in the context of HAR [36, 35, 37].

Xiao et al. [38] paid attention to the structure of the model in the FL system. They designed an FL system with an advanced feature extractor for recognizing human activities. This feature extractor consists of two main components. The initial component utilizes convolutional layers to capture local features, while the second component combines LSTM and attention layers to uncover the global relationships within the data. Another line of studies tries to tackle data heterogeneity in HAR applications. Gad et al. [39] introduced Federated Learning via Augmented Knowledge Distillation (FedAKD) to manage the heterogeneous state. They have demonstrated that FedAKD not only surpasses FedAvg in terms of communication efficiency but also outperforms FedMD [40] due to the incorporation of augmentation techniques. FedDL, as presented in Tu et al.’s work [41], is another FL framework tailored for activity recognition purposes under non-IID circumstances. FedDL clusters clients based on the similarity of model weights and subsequently integrates models through an iterative layer-wise approach. With this strategy, the system dynamically acquires personalized models for distinct users. Although FedAKD [39] and FedDL [41] address the data heterogeneity issue, they do not resolve the communication overhead problem that can be challenging in large scale scenarios. ClusterFL [24] also directed its attention toward client clustering, in which, the cluster indicative matrix that captures the similarity between users, is utilized. By employing alternating optimization techniques, they iteratively update both local models and the cluster indicator matrix.

Some research endeavors focus on the FL system in a semi-supervised approach to address the limited availability of labeled samples in the HAR problem. Notably, Bettini et al. [42] introduced FedHAR which is a semi-supervised FL framework. Specifically, in this scheme, a combination of active learning and label propagation is employed to semi-automatically annotate data for each individual client, effectively addressing the scarcity of labeled samples. In [43], a semi-supervised learning-based HAR scheme is proposed, in which clients train autoencoders in an unsupervised approach with their unlabeled local data while the server trains an activity classifier through supervised learning. Their scheme can achieve competitive performance compared to the supervised approach in the HAR application.

In the proposed framework, a given dataset, $D$, containing $|C|$ distinct classes, is distributed across $N$ distinct clients, which $N$ represents the total number of clients. For any given client, designated as the $i$-th party, their local dataset is represented as $D_{i}=\{(x_{i}^{j},z_{i}^{j},y_{i}^{j})\}_{j=1}^{n_{i}}$, where $x_{i}^{j}$ refers to the original image of the $j$-th sample in the $i$-th client, $z_{i}^{j}$ is the pixelized version of $x^{i}_{j}$, where the human identity (face) is covered using image pixelization operation, $y_{i}^{j}$ is the human activity label corresponding to $x^{i}_{j}$. The term $n_{i}$ represents the total number of samples that are present within the $i$-th client’s dataset.

It’s imperative to mention that due to the non-independent and identically distributed (non-iid) nature of our setup, there might be clients, in which certain classes are absent. Therefore, while $|C|$ represents the total number of classes in dataset $D$, a client’s local dataset might contain fewer classes, represented as $|C_{i}|$. This relationship can be mathematically defined as:

Furthermore, for each client, two neural networks $g(.;\theta_{i})$ and ${g}(.;\tilde{\theta}_{i})$ are associated with the personalized local model and local model of the $i$-th client. The functionality of these two networks is explained in the following paragraphs.

Each of the two neural networks mentioned above consists of two principal segments; 1) a feature extractor that maps the input data (i.e., images in the context of our HAR application) into embedding vectors; 2) a classifier that maps the embedding vector into categorical vectors to make a classification decision.

Let $\theta_{i}$ and $\tilde{\theta}_{i}$ represent, respectively, the weights of personalized and local models for $i$-th client. We denote the penultimate layers of personalized and local models of $i$-th client with $g^{p}(.;\theta_{i})$ and ${g}^{p}(.;\tilde{\theta}_{i})$, respectively. In fact, $g^{p}(.;\theta_{i})$ and ${g}^{p}(.;\tilde{\theta}_{i})$ are feature extractors in personalized and local models that provide the representations of the input in the embedding space. For simplicity, in the following sections, the notion ${g}^{p}(.;\tilde{\theta_{i}})$ and ${g}^{p}(.;{\theta_{i}})$ is represented as $\tilde{g}_{i}^{p}(.)$ and ${g}_{i}^{p}(.)$, respectively.

Each client in our scheme partakes in the training of two distinct neural architectures: the personalized model and the local model.

• •

  Personalized Model: The objective of this model is to minimize the canonical cross-entropy loss using the original image data. For the $i$-th client’s data sample $(x_{i}^{j},z_{i}^{j},y_{i}^{j})$, the personalized model’s loss is given as:

  $$\mathcal{L}_{CE}^{personalized}=CE(g(x_{i}^{j};\theta_{i}),y_{i}^{j})$$

  (1)

  Utilizing original images poses privacy concerns since they could inadvertently disclose participant identities. To address this, we propose another model for every client, that is trained with the pixelized visual signals. By employing image pixelization, it is assured that the sensitive data in the images, in our case the faces, are masked. Hence, in addition to sending the parameters of each local model to the server, we can transmit their pixelized data (that preserves the human identities) as well. This strategy could have a positive impact on enhancing the FL performance.

  To generate the masked images, we employ the RetinaFace model [44], an efficient facial detection scheme, to detect faces within these images. Subsequently, the detected faces undergo pixelization through bilinear downsampling and nearest neighbor upsampling operations, both conducted with a factor of $a$, as follows:

  $$z_{i}^{j}=\text{Nearest Neighbor}_{\uparrow a}(\text{Bilinear}_{\downarrow a}(x_{i}^{j}))$$

  (2)

  A substantial value for $a$ ensures effective pixelization. Figure 1 demonstrates the face masking process for a sample of Stanford40 dataset [45].

  

  

• •

  Local Model: The local model is trained on pixelized images using a similar architecture as the personalized model. Its design addresses the aforementioned privacy concerns, aiming to achieve reasonable performance on pixelized images. The loss for this model is the standard cross-entropy based on pixelized images, which is represented as

  $$\mathcal{L}_{CE}^{local}=CE({g}(z_{i}^{j};\tilde{\theta}_{i}),y_{i}^{j})$$

  (3)

  The proposed local model incorporates both knowledge distillation and contrastive learning to improve the model’s capability to generate efficient representations for pixelized images, as shown in Figure 2. With the addition of contrastive learning, the local model aims for congruent representations between original and pixelized images. Similar to [46], we define the contrastive loss as:

  $$\mathcal{L}_{con}=-\log\frac{e^{\frac{cos(\tilde{g}_{i}^{p}(z_{i}^{j}),\tilde{g}_{i}^{p}(x_{i}^{j}))}{\tau}}}{e^{\frac{cos(\tilde{g}_{i}^{p}(z_{i}^{j}),\tilde{g}_{i}^{p}(x_{i}^{j}))}{\tau}}+\sum_{k\neq j}e^{\frac{cos(\tilde{g}_{i}^{p}(z_{i}^{j}),\tilde{g}_{i}^{p}(x_{k}^{j}))}{\tau}}}$$

  (4)

  Here, $\tilde{g}_{i}^{p}(z_{i}^{j})$ and $\tilde{g}_{i}^{p}(x_{i}^{j})$ represent the local model’s representations over pixelized and original images respectively. The term $\tau$ denotes the temperature parameter that controls the strength of penalties on hard negative samples, while $cos(.,.)$ is the cosine similarity metric.

  To further refine the local model’s performance on original images, knowledge distillation is employed, leveraging insights from the personalized model which enhances the similarity between the local model representation on pixelized images and the representations of the personalized model on the original counterparts:

  $$\mathcal{L}_{KD}=-\log\frac{e^{\frac{cos(\tilde{g}_{i}^{p}(z_{i}^{j}),{g}_{i}^{p}(x_{i}^{j}))}{\tau}}}{e^{\frac{cos(\tilde{g}_{i}^{p}(z_{i}^{j}),{g}_{i}^{p}(x_{i}^{j}))}{\tau}}+\sum_{k\neq j}e^{\frac{cos(\tilde{g}_{i}^{p}(z_{i}^{j}),{g}_{i}^{p}(x_{k}^{j}))}{\tau}}}$$

  (5)

  where ${g}_{i}^{p}(x_{i}^{j})$ denotes the personalized model’s representations over the original images.

  Finally, the total loss used for training the local model is as:

  $$\mathcal{L}^{local}=\mathcal{L}_{CE}^{local}+\lambda_{1}\mathcal{L}_{con}+\lambda_{2}\mathcal{L}_{KD}$$

  (6)

  here, $\lambda_{1}$ and $\lambda_{2}$ work as hyperparameters to calibrate the contributions from contrastive learning and knowledge distillation components, respectively.

Clustered Federated Learning (FL) [47, 48, 49, 50] categorizes clients with similar data distributions into groups, and consequently, trains a distinct global model for every cluster. Previous investigations have adopted diverse criteria for client clustering, which range from using contextual information [28], model parameters [19, 51, 29, 23], loss functions [20], and embedding vectors [18, 52]. While earlier efforts largely focused on client clustering to mediate data heterogeneity, our research harnesses image clustering to transmit a minimal number of representative pixelized images to the server in order to enhance the FL performance. It should be noted that local data remains unshared with servers or other clients in conventional FL systems. As delineated in Section III-B, pixelized images effectively obscure individual sensitive data and contain the necessary information for performing the task of HAR. Nonetheless, the indiscriminate transmission of all pixelized images is memory-consuming and inefficient, emphasizing the need for efficient image selection methodologies.

In order to perform image selection, we utilize the idea of deep clustering in the feature space. Mathematically, deep clustering for the $i$-th client is expressed as:

where $\mu_{ik}$ is the centroids corresponding to the $k$-th cluster in the $i$-th client. The centroids will be learned during training of the local model. $\mathbbm{1}_{jk}$ is an indicator that becomes one if $\tilde{g}_{i}^{p}(z_{i}^{j})$ belongs to the $k$-th cluster, and zero otherwise.

The training parameters in Equation (7) scale with the embedding space’s size. For obtaining high-quality centroids for each client, their sample count should be considerably large relative to this space. Given that FL systems rarely assign individual clients substantial samples, the deep clustering cannot be carried out in an efficient manner [53, 54, 55, 56]. In order to address this, we employ the pseudo-labeling [57, 58, 59] technique for carrying out the deep clustering operation. In this prevalent deep clustering strategy, first, cluster centroids are derived through methods like k-means clustering. These centroids serve as the reference points for assigning data samples to their corresponding clusters. These assigned cluster labels are then considered as the samples’ pseudo-labels. Subsequently, the model is re-trained with the objective of aligning the representations of the samples with their corresponding cluster centroids.

Our algorithm employs the global model in conjunction with K-means for pseudo-label prediction, leveraging the global model’s encompassing ability to derive representations from all clients. In the first stage, the global model is applied to pixelized images within each client, yielding individual representations in the embedding or penultimate layer for each client:

Subsequently, K-means extracts cluster centroids and pseudo-labels, given the representations learned earlier as follows:

Then the pseudo-labels $a_{jk}^{i}$ is defined as

Therefore, Equation (7) can be reformulated as

By introducing the deep clustering loss into Equation (6), the local model’s loss can be computed as:

where $\lambda_{3}$ adjusts the significance of the clustering term.

After the local model’s training using Equation (12), K-means determine updated centroids based on the representations generated by the local model:

Subsequently, for each centroid $\bar{\mu}_{ik}$, the Euclidean distance to all local samples is computed, and the $m$ nearest samples to each centroid are then selected as the optimal samples for transmission to the server. The collection of chosen images for the $k$-th cluster is denoted as:

By performing the selection process on all centroids for each client, a set containing the best images for transmission to the server is obtained as:

where $\mathcal{S}_{i}$ will be shared with the server for further processing. It is worth mentioning that $\mathcal{S}_{i}$ remains stored on the server until the next communication round when the respective client is selected to contribute to the server update process and update its corresponding set, $\mathcal{S}_{i}$. Consequently, in each communication round, privacy-preserved data samples from all clients are present on the server, regardless of their contributions.

The advantages of the proposed image selection method are two-fold. Firstly, it employs global model representations for obtaining the pseudo-labels, and hence, provides centroids with a global view from all clients. This prevents the deviation of the local model from the global model during local model training. Secondly, the global and local models are updated in each communication round. As a result, the selected images are dynamically refined to improve performance over time.

In traditional FL systems, during each communication round, the central server aggregates model weights from all connected clients to update the global model. There exist three issues with using this approach. First, due to the data heterogeneity among clients, the aggregated global model is far from the global objective. Second, it necessitates significant memory and bandwidth resources, which might be unattainable in many real-world scenarios. Third, some clients may demonstrate slower convergence compared to others. Including such clients during the global model update can detrimentally impact the convergence rate of the entire FL framework.

In order to address these challenges, we consolidate the global model after naive aggregation using the privacy-preserved selected images. Additionally, we propose a client selection algorithm that identifies and integrates contributions from the most useful clients in each round. This algorithm leverages the dataset of selected images from Section III-C to determine the optimal clients.

For each client, the received dataset, denoted as $\mathcal{S}_{i}$, is divided into two sets of training ($\mathcal{S}_{i}^{tr}$) and validation ($\mathcal{S}_{i}^{val}$). The aggregation of local models follows the fundamental aggregation:

Here, $\mathcal{C}$ represents the set of selected clients forwarding their model weights to the central server. The process of selecting clients will be explained later in this section.

A simple parameter aggregation in the server is not sufficient to approach the global objective, since the global model requires an efficient fine-tuning using the selected privacy-preserved data of all clients in order to align with the global objective and provide a superior FL performance. To mitigate this, we utilize images chosen through the methodology detailed in Section III-C to adjust the global weights, aligning them with the global objective.

After the naive aggregation of the global model, it undergoes training on the combined training set, $\mathcal{S}^{tr}=\bigcup\limits_{i=1}^{N}\mathcal{S}_{i}^{tr}$, and is fine-tuned using the privacy-preserved data samples stored on the server from all clients. The loss for the global model, for any data instance $(z_{i}^{j},y_{i}^{j})\in\mathcal{S}^{tr}$, is:

This loss undergoes updates via the Stochastic Gradient Descent (SGD) optimizer. Finally, the updated global model is broadcast to the clients in order to initialize the personalized and local models for the next communication round. This strategy will be helpful to overcome the slow convergence in the typical FL algorithms. The overall architecture of the global model update is shown in Figure 3.

To optimize the bandwidth, the trained global model is evaluated on the validation subsets and then clients are ranked based on their validation performance in a descending order. A certain percentage of clients with the highest validation accuracy are selected to contribute their weights in the next communication round. These selected clients are stored in $\mathcal{C}$ set. It should be noted that in the next round, the updated global weights are sent to all connected clients. However, after local training is done, only the chosen clients share their weights with the server and update their pixelized images on the central server.

At the end of each communication round, the local models are applied to the test samples to evaluate their individual performance. Then, the performance of all connected clients is averaged to obtain the performance of the scheme. The mean accuracy of all schemes, tested with varying numbers of clients (10, 20, and 30), is summarized in Table I. First, we have to say CDFL is the best. The SOLO approach consistently exhibits the poorest performance across all settings, representing the beneficial impact of the FL framework in decentralized learning. As the number of clients increases, there is a drop in performance across all schemes. In the case of FedAvg [9], this drop is particularly significant, ranging from 7% to 16% among different datasets. The relatively lower accuracy of FedAvg can be attributed to its limited ability to handle heterogeneous conditions. The performance improvement of CDFL in comparison to MOON [27] is up to 4.2%, 7.1%, and 9.8% for Stanford40, PPMI, and VOC2012, respectively. Among the baselines, the performance of FedDyn [61] is closer to CDFL. Compared to FedDyn [61], CDFL improves the performance up to 1.3%, 5.4%, and 5.7% in Stanford40, PPMI, and VOC2012, respectively.

As shown in Table I, CDFL consistently outperforms all other baselines across various settings, demonstrating the strength of CDFL in comparison to previous works. In general, CDFL has a smaller standard deviation compared to other baselines. This implies that it not only works well on the whole set of clients according to its higher mean accuracy but also performs well on all individual clients due to its low standard deviation. In contrast, the higher standard deviation in other baselines represents difficulties in achieving good performance on some clients due to non-IID data distribution. Furthermore, CDFL is still able to provide higher performance, even when the number of clients increases, which is crucial in realistic applications where the number of clients is often large.

To demonstrate the scalability of CDFL to a large number of clients, we evaluate the Stanford40 dataset in two scenarios with 50 and 100 clients. The data samples were distributed among clients using Dirichlet distribution with $\alpha=1.0$. It is worth noting that Stanford40 is selected for these large-scale scenarios due to the small size of other datasets, whose number of samples is approximately 50% of the samples available in Stanford40. Since the number of clients increases, the number of communication rounds is set to 30 to make sure that all schemes converge to optimal solutions.

The performance of various schemes on a large number of clients for the Stanford40 dataset is presented in Table II. As expected, the SOLO approach exhibits a significant drop in performance when the number of clients increases. Among the baseline methods we considered, FedProx [10], MOON [27], and FedHKD [62] perform similarly to FedAvg [9], struggling to handle a large number of clients effectively. In contrast, SCAFFOLD [60] and FedDyn [61] show improved performance compared to the other baselines in large-scale settings. As shown in Table II, CDFL consistently outperforms the other baselines in the case of large-scale FL setups. The higher mean performance of CDFL implies its ability to effectively manage a large number of clients. Furthermore, its lower standard deviation compared to other baselines demonstrates its consistent performance across all the individual clients.

To assess the performance of our CDFL under different levels of data heterogeneity, we conduct experiments using two different values of $\alpha$ across all datasets. Specifically, we tested $\alpha=0.1$ to represent a higher level of heterogeneity as well as $\alpha=10$ to consider less heterogeneous data. The results are presented in Table III.

CDFL consistently outperforms other baselines when data heterogeneity is high ($\alpha=0.1$), demonstrating its robustness in handling challenging, heterogeneous situations. In this scenario, CDFL achieves a performance improvement of up to $\sim 6\%$, 10%, and 6% compared to other baselines for Stanford40, PPMI, and VOC2012, respectively.

For $\alpha=10.0$, CDFL achieves the highest accuracy for the Stanford40 and PPMI datasets, while FedProx leads to the best accuracy for VOC2012. Generally, the performance of all schemes is quite similar under homogeneous conditions ($\alpha=10.0$), which demonstrates the ability of FL systems to obtain desired performance in homogeneous data settings.

we compare CDFL with other baselines in terms of the number of communication rounds required to achieve a specific target accuracy as well as communication overhead in each communication round. Table IV provides an overview of the communication rounds needed to reach the test target accuracy for various numbers of clients across all datasets. In all settings, we continue training until we reach the target test accuracy or reach a maximum of 50 communication rounds.

As illustrated in Table IV, CDFL requires the fewest communication rounds in all settings and demonstrates a remarkable increase in convergence speed up to 10 times faster compared to other baselines. For the VOC2012 dataset with 10 clients, both SCAFFOLD [60] and FedHKD [62] failed to reach the target accuracy after 50 communication rounds.

Each communication round involves two types of communication overhead: uplink and downlink. Uplink includes the transmission of local models to the server, while downlink corresponds to broadcasting the updated global model to the clients. As the downlink overhead is considerably lower than the uplink, our analysis focuses on the uplink overhead. In this study, we define the number of tansmitted parameters as communication overhead metric.

In Table V, we summarize the communication overhead of various FL schemes. Since the uplink communication overhead of FedProx [10], MOON [27], and FedDyn [61] matches that of FedAvg [9], we only report the uplink overhead of FedAvg [9] in Table V. In this table, $p$ represents the ratio of connected clients to the total number of clients, and $|\theta_{i}|$ is the number of parameters in each client’s model. In the case of SCAFFOLD [60], $c_{i}$ is the control variate whose size matches the client’s model ($|c_{i}|=|\theta_{i}|$). For the FedHKD [62] scheme, hyperknowledge vectors are transmitted to the server alongside the clients’ models, and $|\mathbf{v}|$ represents the size of the penultimate layer in the model. In most cases, the overhead arising from hyperknowledge vectors is negligible compared to the model transmission overhead. In CDFL, the pixelized images are transmitted beside the clients’ models but for a smaller number of clients ($|\mathcal{C}|$).

We evaluate Stanford40, PPMI, and VOC2012 datasets with 100, 30, and 30 clients, respectively. We select the largest number of clients for each dataset, as the communication overhead would be more challenging with a larger number of clients. The overhead of all schemes is calculated in Table V. For CDFL, we assumed the size of images to be $|I|=224\times 224\times 3$, and this is the same as the input size of ResNet50. However, there exists an opportunity to compress or downsample the images before transmission to the server, which can save more bandwidth. Since ResNet50 is utilized in our experiments, the total number of parameters is 24.1 M.

As shown in Table III, SCAFFOLD has the highest overhead among all schemes, while CDFL provides the lowest overhead due to its client selection. We achieved a reduction in overhead by up to $\sim$ 64% compared to other baselines for all datasets.

In this section, we study the effects of key modules and hyper-parameters in CDFL. The experiments are mainly conducted on the PPMI datasets with 10 clients and $\alpha=1.0$.

• •

  Imapact of Contrastive Learning & Deep Clustering Modules

  We investigate the effect of the contrastive learning and deep clustering modules by excluding them from CDFL, as shown in Table VI. The contrastive learning module is eliminated by setting $\lambda_{1}=\lambda_{2}=0$ in local model loss function. Therefore, this test only relies on the training of pixelized images. In the second test, the deep clustering term is omitted from the local model loss function by setting $\lambda_{3}$ to zero, and random pixelized images are chosen for transmission to the server. As presented in Table VI, removing the contrastive learning module from CDFL results in a 1.27% decrease in performance. In addition, eliminating the deep clustering modules and transmitting random pixelized images to the server leads to a 3.9% reduction in performance, highlighting the effectiveness of our image selection method.

  Removing contrastive learning	Removing deep clustering
  $89.47\pm 5.368$ ($1.27\%\downarrow$)	$86.84\pm 4.30$ ($3.9\%\downarrow$)

• •

  Impact of $\lambda_{1}$ and $\lambda_{2}$ of Model Contrastive Learning

  We investigate the effect of various values of $\lambda_{1}$ and $\lambda_{2}$ on the performance of CDFL. Test performance results for different values of $\lambda_{1}$ and $\lambda_{2}$ are presented in Table VII. An increase in $\lambda_{1}$ results in a more similar representation between the local model on both original and pixelized images. Further, larger values of $\lambda_{2}$ lead to a more similar representation between personalized and local models on original images.

  $\lambda_{1}=\lambda_{2}=0.01$	$\lambda_{1}=\lambda_{2}=0.1$	$\lambda_{1}=\lambda_{2}=1.0$
  $89.63\pm 4.31$	$90.74\pm 2.98$	$88.68\pm 3.23$

• •

  Impact of $\lambda_{3}$ in Deep Clustering

  Deep clustering is a crucial element in CDFL as it helps select representative images for the clients. The impact of $\lambda_{3}$ in deep clustering loss is investigated for three different values of $\{0.01,0.05,0.1\}$, as summarized in Table VIII. Smaller values of $\lambda_{3}$ result in poor clustering and image selection, while larger values of $\lambda_{3}$ fade the impact of contrastive learning and knowledge distillation terms in the local loss function, causing the local model to diverge from the personalized model.

  $\lambda_{3}=0.01$	$\lambda_{3}=0.0.5$	$\lambda_{3}=0.1$
  $90.11\pm 2.30$	$90.74\pm 2.98$	$90.19\pm 3.69$

• •

  Impact of Number of Selected Images

  $m=1$	$m=3$	$m=5$	$m=7$
  $88.13\pm 5.39$	$89.32\pm 4.39$	$88.77\pm 5.26$	$90.74\pm 2.98$

  In CDFL, we select the $m$ nearest neighbors of each centroid for transmission to the server. A very small $m$ selection leads to a non-representative image set, while a large $m$ increases communication overhead. The performance of CDFL for four different values of $m$ is shown in Table IX. As shown in Table IX, smaller $m$ values, while slightly sacrificing performance, reduce bandwidth usage and communication overhead. This shows that CDFL can be effective even with fewer selected images.

• •

  Impact of Number of Selected Clients

  We also study the impact of the number of selected clients in CDFL. We repeat the experiments when only 3, 5, and 7 clients share their weights and images with the central server, and the results are summarized in Table X. As expected, when fewer clients contribute to the server aggregation, the performance drops, however, the performance drop is negligible. This demonstrates that CDFL works well even if only 30% of clients are involved in server aggregation resulting in a significant decrease in communication overhead.

  $|\mathcal{C}|=3$	$|\mathcal{C}|=5$	$|\mathcal{C}|=7$
  $89.48\pm 4.44$	$90.74\pm 2.98$	$91.02\pm 2.65$