Federated Transfer-Ordered-Personalized Learning for Driver Monitoring Application

FL is a potentially feasible solution to the privacy problem in IoT, which is able to avoid the proliferation, distribution, and exchange of local client data by sharing model parameters after training the model on local client data. FL frameworks are widely used in mobile service [6], healthcare [7, 8], industrial [9, 10], IoV [11, 12], etc., due to their usages of large scale and personalized data in an efficient and privacy-preserving way. Although FL has significant contributions to massively parallel devices and computations, it still has a notable drawback in that it cannot efficiently handle non-independent and identically distributed (non-i.i.d.) data. It is required to customize the applicable FL framework according to the features, resources, and constraints possessed by users, data, clients, and servers.

Non-i.i.d. data and heterogeneity have always been a challenge, and a key to research in FL [13, 14, 15]. Non-i.i.d. data is a common phenomenon for real-world clients that are scattered and not interoperable: Taking IoV as an example, each driver is heterogeneous as a client. FedAvg [16], as one of the first proposed feasibility methods, has been the subject and center of research. FedAvg averages all local models to get the global model so that the local model may deviate far from the global optimum in the parameter space leading to some limitations in FedAvg. It is necessary to ensure that the local model does not deviate from the global model (prevent overfitting) and, simultaneously, that the local model can effectively learn the local client dataset (prevent underfitting). Based on FedAvg, FedProx [17] is proposed to limit the deviation of the local model from the global model by adding a proximal term.

Besides considering accuracy, the FL framework in IoT should not underestimate communication and training resource constraints, cybersecurity, and ubiquity. Some of the recent surveys summarized challenges, threats, and solutions of the FL decentralization paradigm for IoT, including limited computing power, unreliable and limited availability, local training, accuracy, communication overhead, etc. [18, 19, 20, 21, 22, 23].

Transfer and edge learning are popular solutions to reduce communication resource consumption in FL frameworks. Zhang et al. [24] performed a federated transfer learning framework to detect driver drowsiness, where transfer learning was employed to save the communication cost in the FL framework. Su et al. [25] introduced edge servers as a collaborative mechanism, where local models were aggregated in the edge server and then sent to the global server to aggregate the global model. The benefit of the additional edge server was that the communication between massively parallel clients and the edge server was consumed because the edge server was geographically close to the clients. High latency and intermittent connections could be mitigated. In addition, the edge server could also provide personalized aggregated local models due to the similarity of geographically adjacent clients.

Cyber attack is a problem that cannot be ignored for FL frameworks. Sun et al. [26] developed an attack method for FL framework in IoT, in which a bi-level optimization framework was proposed to compute optimal poisoning attacked FL framework, including direct, indirect, and hybrid attacks. Meanwhile, Zhang et al. [27] utilized a generative adversarial network (GAN)-based approach to attack the FL framework, especially since the attacker did not need any prior knowledge to carry out the attack.

Personalization is a common approach for FL frameworks to improve applicability for diverse users [28]. Fallah et al. [29] proposed a personalized variant of the FL, which allowed clients to perform several gradient descent iterations on an initial global model using local data to obtain a personalized local model. Wu et al. [30] explored a cloud edge-based personalized FL framework for in-home health monitoring, which addressed the problem that a single global model performed poorly on a specific client. Since the global model could only capture the common features of all clients, it lacked the ability to analyze fine-grained information of specific clients. Ma et al. [31] proposed a personalized FL framework of layer-wised weighted aggregation, which determines mutual contribution factors through a hypernetwork on the server to identify similarity between users at the layer granularity.

Driver monitoring application (DMA) in IoV is adopted as the research direction in this paper due to its real and visual image data, valuable application scenarios, and relatively blank research area. DMA also has challenges in terms of driver privacy issues, communication, and diverse and personalized driver state and behavior. Related DMA literature covers a wide variety of devices with algorithms to achieve different purposes, such as dangerous state detection [32], driver emotion recognition [33], driver lane change inference [34], etc. Compared to other methods [35, 36, 37], FL not only highlights efficient learning but also effectively protects the privacy of driver, passenger, and pedestrian biometric information, driving routes, and confidential driving areas such as military installations.

In this paper, we introduce and adapt FL to DMA. Although some FL frameworks exist for DMA, they all suffer from some critical problems. Doshi et al. [38] proposed a FL edge-device framework to obtain a global model by aggregation feature representations and obtained considerable accuracy in recognizing driver activities. For the i.i.d. setting, the dataset was partitioned for each edge node in a random way, while for the non-i.i.d. setting, the dataset was assigned selectively. Zhao et al. [39] proposed a FL framework to monitor fatigue driving, where the non-i.i.d. setting was simulated by controlling the number of images per client. The above FL frameworks for DMA did not really take into account the actual situation of the application but artificially created a simulation scenario. Therefore, there is an urgent need for realistic analysis and research for real-world DMA, considering that the user (driver) should exist independently and be non-interoperable with different clients (vehicles). Moreover, in addition to the necessity of test datasets, the test client is also a critical evaluation criterion, which can reflect the universality of the FL framework. We summarize the existing neglects and challenges in the current FL for DMA frameworks as follows.

• •

  Clients in FL for DMA frameworks are often defined in unreasonable and incomprehensible forms. A real and natural definition of a client should be a driver or a vehicle.

• •

  There is no paper proposing to test on a testing client (not involved in the training process), which lacks universal testing for the FL framework.

• •

  For the DMA scenario, there is a great diversity and individuality of driver behaviors, postures, and facial expressions, which call for more personalized studies than other general IoV scenarios.

• •

  Similarly, DMA also has diverse scenarios, including diverse vehicle models, interior colors, seat positions, etc., which will greatly increase the learning difficulty.

In this paper, we aim to propose a FL framework applicable and specific to practical applications in IoV, especially DMA, where an imaginary FL framework for IoV is illustrated in Fig. 1. Each local client, i.e., vehicle, includes a training module and a perception module. The training module uploads the model parameters to the server after learning and training the local data. After aggregation and optimizing the parameters of the local client models, the server downloads the global model parameters to the perception module in the local client. Moreover, transfer learning can be used to reduce the number of trainable parameters, resulting in reduced communication consumption. The server can save different global models for different scenarios, such as road types, weather types, and vehicle types, so that the model can have better applicability.

Therefore, a federated transfer-ordered-personalized learning (FedTOP) framework is proposed to address the problems of accuracy, cybersecurity, communication resources, and diversified scenarios. In addition to the transfer-extension shown in Fig. 1, the FedTOP framework also enhances robustness and cybersecurity by orderly dropout clients due to their possible overfitting and poisoning of the data. Furthermore, the FedTOP framework is able to remarkably improve accuracy by adapting all clients through personalized-extension. The contributions of this paper are:

• •

  For realistic problems and usage scenarios in DMA, we propose a feasible FL framework FedTOP, realizing privacy protection, high accuracy, low communication requirements, cybersecurity, and pervasiveness. To the best of our knowledge, this is one of the first papers to establish a feasible FL framework for DMA.

• •

  The proposed FedTOP framework is tested on two real-world driver monitoring datasets with and without system heterogeneity, systematically characterizing system heterogeneity in real-world datasets and achieving considerable accuracies with 92.32$\%$ and 95.96$\%$, respectively.

• •

  The experiments highlight a realistic and natural client setup, i.e., drivers and vehicles are naturally formed as clients. Moreover, we innovatively propose evaluation criteria for training and testing clients to test the generalization ability of the proposed FedTOP on different clients.

• •

  Through an ablation study, we demonstrate the performance and utility of the transfer, ordered, and personalized extensions. These detachable extensions can be selectively installed according to the task description, and the FL framework combined with different extensions can effectively adapt to different IoT application scenarios.

The presentation of this paper is as follows. The problem statement and proposed solution are described in Section II. The experimental setup, heterogeneity, and results have been demonstrated in Section III. Section IV discusses the performances of three extensions of the proposed framework, followed by Section V summarizing the paper and expounding on future work.

FL frameworks are able to protect privacy, increase training efficiency, and save communication resources by sharing only model parameters in IoT. In this paper, the FL framework is used to solve a driver activity classification task in DMA. Clients in real-world IoT are independent and heterogeneous due to the presence of only a minimal number of users per client. Considering the more general application scenarios, the global model $\omega$ for training clients $C$ aggregation needs to be compatible with non-training clients $C^{\prime}$ in addition to $C$. The data of each client $D_{c}$ is non-i.i.d. when the data is not interoperable. We can consider a nested model,

where $\omega_{c}$ is the classifier model corresponding to client $c\in C$. $D_{c}\in\mathbb{R}^{n_{c}\times i\times j\times d}$ is the image set with $n_{c}$ samples, $i$ rows, $j$ columns, and $d$ channels. $L_{c}\in\mathbb{Z}^{n_{c}}$ is the corresponding label set. The global model $\omega$ is obtained by aggregating, e.g., averaging the weights of the local models,

where $p_{c}\in[0,1]$ is a weight density function of clients, for which $\sum p_{c}=1$, $p_{c}$ will be assigned according to the number of samples. Therefore, the optimization problem of the FL algorithm can be formulated as minimizing the global loss, which is equivalent to minimizing the sum of the local losses,

where $\mathcal{L}$ is the loss function that will be assigned.

For real-world classification tasks, we assume that the distribution of the local model in the parameter space presents a multivariate Normal distribution $\omega_{c}\sim\mathcal{N}\left(\mu_{\omega},\sigma^{2}_{\omega}\right)$, where $\mu_{\omega}$ is mean of all local models, and $\sigma^{2}_{\omega}$ is the variance of all local models. Fig. 2 shows the process of the FL algorithm finding the optimal solution of the global model in the parameter space. After the initial model is trained locally, communicated, and aggregated globally, the final global model will be obtained by averaging and can be estimated as $\hat{\omega}=\mu_{\omega}$. Especially in the large-scale parallel application scenarios of IoT, according to the law of large numbers, $\hat{\omega}=\mu_{\omega}=\omega^{\ast}$ is an unbiased estimation.

However, there are still some defects in the method of obtaining the global model through average aggregation. Firstly, we can confirm that there is enormous system heterogeneity in IoT, and the global model cannot ensure high accuracy for all clients. Secondly, we inevitably need a measure to prevent system heterogeneity and potential attacks and poisoning. As shown in Fig. 2, the farther the optimal local model is from the global model, the lower the accuracy, and vice versa. Therefore, it is conceivable that in the FL problem with heterogeneity, the clients’ accuracy will also obey a Normal distribution.

According to the problem statement, we propose a FedTOP algorithm to address all of the following issues. First, the aggregation of global models needs to be more stable, which can be achieved by preventing the overfitting of local models. Second, considering the actual communication situation in IoT, we propose a transfer learning method to reduce the trainable parameters and hence reduce communication requirements. Third, the global model should have the ability to resist interference, attacks, and data poisoning, which can be achieved by orderly dropping out local models with a large loss. Fourth, a global model cannot take into account the situation of all clients, especially in the presence of data and system heterogeneity. Therefore, we recommend personalizing the global model to suit all the training and testing clients.

We refer to FedProx [17] using a proximal term to prevent local models $\omega_{c}$ from deviating from the global model $\omega$. In which the proximal item $\mathcal{L}_{p}$ that computes the distance between the local and global model is added to the loss function,

where $\mu$ is deviation coefficient, $\omega_{c}$ is local client model parameters, and $\omega$ is global model parameters. The overall loss function can be updated as

where $\mathcal{L}_{l}$ is the loss between the true labels and the predicted labels, such as the negative log-likelihood loss used in our experiments.

Transfer-extension is a common and popular solution in many learning frameworks. In particular, FL frameworks are favored because they can effectively reduce local client training resources and communication resources. In our experiments, the base model is ResNet34 [40] pre-trained on ImageNet, where only the last residual block and fully connected layer are trainable parameters. Although ImageNet is a large object classification dataset far from DMA images, the lower layers are similar to convolutional neural networks (CNN) and are used to extract image features. Therefore, the upper layers that are used to obtain high-level features and representations are given more attention. The ratio of reduced communication resource requirement in the network is approximately equal to the ratio of non-trainable parameters to total parameters,

where $\text{Commun}_{\downarrow}$ is the reduced communication resource requirement, $|\omega_{\text{non-trainable}}|$ is the number of non-trainable model parameters, and $|\omega|$ is the total number of the model parameters. Therefore, the transfer-extension reduces the communication requirement by 37.46$\%$ by decreasing the trainable parameters.

Ordered-extension is for orderly dropout clients with enormous variance, which may be subject to malicious attacks and poisoning, extensive data and system heterogeneity, and model underfitting. These local clients with large losses should be discarded to enhance the generalizability of the global model. Ordered-extension not only enhances accuracy and robustness but also secures the global model. After all of the clients upload the local model parameters and the final training loss to the server, the server only aggregates the $q\in\mathbb{N}\leq|C|$ local models with the lowest loss as the global model. The set of $q$ local models can be expressed as

Therefore, only the models of clients in $C_{q}$ with lower loss will be used to aggregate the global model.

Personalized-extension is to promote, popularize, and adapt the global model to the heterogeneity of all clients. As shown in Fig. 2, the global model cannot be applied to all clients due to the ubiquitous heterogeneity. The region of interest (ROI) of the model may vary depending on system heterogeneity, such as different camera angles, seat positions, and vehicle structures, resulting in differences in the relative position of the driver in the image. However, personalized-extension proposes to train the global model several times in each client to obtain a more personalized local model to improve accuracy. On the one hand, compared with the traditional FL algorithm, the personalized-extension can significantly and effectively improve accuracy and confidence. On the other hand, compared to the method that only trains locally, the personalized FL algorithm improves the training efficiency and avoids the overfitting of the local model. In particular, the personalized FL algorithm can help and generalize to other non-training clients $C^{\prime}$, which may have minimal training resources. After receiving the global model, the non-training clients $C^{\prime}$ can obtain a highly accurate and reliable local model with minimal training. The system diagram of the proposed FedTOP is shown in Fig. 3.

For the proposed FedTOP framework, the client communicates with the server $T$ rounds, and all clients $C$ train $E$ epochs in parallel between each communication. For our preliminary experiments, we set $T=10$ and $E=5$. For transfer-extension, the local model is the transfer learning model of ResNet34 pre-trained on ImageNet. Only the last residual block and fully connected layer are set as trainable parameters. In addition, we add an additional fully connected layer to match the number of our classification categories. Based on FedProx, the activation function of the last layer is LogSoftmax, and the setting of the loss function $\mathcal{L}_{l}$ is a negative log-likelihood loss. $\omega^{1}$ is the initial model parameter. The proposed FedTOP is described in Algorithm 1, and the personalization process is described in Algorithm 2.

To compare the impact of system heterogeneity on FL frameworks, the proposed FedTOP is tested on driver monitoring datasets with and without system heterogeneity. The SFDDD dataset includes 26 drivers and 10 activities, and the DriveAct dataset includes 15 drivers and 12 activities. The SFDDD dataset considers system heterogeneity, that is, different drivers have different vehicles, different seat positions, different camera angles, etc., as shown in Fig. 4a, 4b, 4c, and 4d. The DriveAct dataset does not take into account system heterogeneity, i.e., all subjects had their data collected in the same system. Recorded from the same camera angle, different drivers read the same magazine in the same vehicle, as shown in Fig. 4e, 4f, 4g, and 4h.

To show more clearly and visually the heterogeneity between different clients in the two datasets, Fig. 5 shows histograms of the sample images of the two datasets. It can be seen that the SFDDD dataset with system heterogeneity has a more considerable difference in the distribution of histograms than the DriveAct dataset without system heterogeneity, and the mean value of the SFDDD images is larger. The possible reason is that the vehicle interiors of the DriveAct dataset view are darker, resulting in most of the pixel values being lower. Therefore, FL frameworks may be more challenged by the scene information when training on the SFDDD dataset, such as different vehicle interiors.

Clients are naturally divided based on the drivers. In order to better demonstrate the role of personalized-extension, the datasets are first divided into training clients and testing clients at a ratio of about 0.8, 0.2, with $|C_{\text{SFDDD}}|=20$, $|C^{\prime}_{\text{SFDDD}}|=6$, $|C_{\text{DriveAct}}|=12$, and $|C^{\prime}_{\text{DriveAct}}|=3$. And then, the datasets for each client are divided into a training set, verification set, and testing set at a ratio of 0.7, 0.15, and 0.15, respectively. After the global model is trained by the training dataset of training clients, the final trained global model is shared with all clients for personalization. The personalization of the global model will only be processed on the training sets, while the personalized local model will be tested on the unseen testing sets. FL architectures are established on Pytorch and trained on an Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz, and a Nvidia GeForce RTX(TM) 3080 GPU.

We explore the role of each FedTOP extension on two real-world datasets through an ablation study. FedProx is used as a baseline for comparison. According to the experimental setup described in the previous subsection, the experimental results are shown in Table I.

The results and comparisons for two datasets and three extensions are shown in Fig. 6, which is equivalent to demonstrating Algorithm 1. By observing the accuracy and loss curves on the two datasets, it can be concluded that the SFDDD dataset with system heterogeneity is fundamentally different from the DriveAct dataset without system heterogeneity. It can be clearly seen that the SFDDD dataset with system heterogeneity requires more communication to converge, while the DriveAct dataset without system heterogeneity has a fast convergence speed, especially at the first communication. Therefore, for real-world datasets, system heterogeneity can be mitigated by more communication times.

By observing Fig. 6b, 6d, 6f, and 6h, it can be found that the ordered-extension diminishes the stability of the system. Although the anomalous large-loss local model is discarded to reduce the bias of the global model, it also increases the variance of the global model resulting in reduced generalizability. By observing Fig. 6c, 6d, 6g, and 6h, we can see that the effect of transfer-extension is different for datasets with and without system heterogeneity. On the one hand, transfer-extension increases the variance of the model on the SFDDD dataset and leads to a reduced and unstable model convergence. On the other hand, transfer-extension improves the speed of model convergence on DriveAct, and the convergence effect is more stable. The possible reason is that the transfer-extension retains only a small number of trainable parameters, resulting in the neural network model not being able to learn human behavioral features effectively in the SFDDD dataset with system heterogeneity. However, for the DriveAct dataset without system heterogeneity, the factors are constant except for the driver, and the local model does not need to focus on these exact same pixels, but only on the changing pixels, including objects such as drivers, computers, and magazines. Therefore, for the DriveAct dataset, transfer-extension can effectively increase convergence and stability. The proposed FedTOP framework is able to obtain 92.32$\%$ and 95.96$\%$ accuracy on the SFDDD and DriveAct datasets, respectively, when considering five times of personalization training. Compared to FedProx as a baseline, FedTOP can effectively improve the accuracy by 462$\%$ in addition to considering a 37.46$\%$ reduction in communication resources. The results demonstrate the feasibility of the proposed FedTOP in terms of communication resource-saving, accuracy improvement, robustness, and cybersecurity.

Personalized-extension needs to be further discussed and analyzed as the most effective approach to improve accuracy. Based on the division of training and testing clients in Section III-A, in this subsection, we further discuss how the trained and aggregated global model is adapted to both training and testing clients. The results of the personalized-extension on the two datasets are shown in Fig. 7 with different personalization epochs, which is equivalent to demonstrating Algorithm 2. It can be seen that the personalization process differs significantly on the datasets with and without system heterogeneity, which is similar to the results in Fig. 6. The clients in the DriveAct dataset have faster convergence, minor accuracy variance, and higher final accuracy. On the contrary, the clients in the SFDDD dataset not only converge slower but also have an anomalous client with relatively low accuracy. The possible reason is that the anomalous client has a huge data and system heterogeneity, causing the optimal model to deviate significantly far from the aggregated global model.

Fig. 8 further demonstrates that the trained global model repositions the ROI during the personalized training process via class activation map (CAM) [43]. The test client of the SFDDD dataset can be seen struggling with the personalization process. The trained global model focuses the ROI on the seat backrest, driver’s chest, hand, and knee, and vehicle door. Due to the system heterogeneity present in the SFDDD dataset, the positions of the driver, seat, and steering wheel, as shown in Fig. 8a is different from other clients, as shown in Fig. 4b, 4c, and 4d. Therefore, the initial ROI is likely to be a driver’s position among other clients. During the five personalization training processes, the local model is able to effectively reposition the ROI to the driver, which is what the personalized-extension is intended to show. Moreover, the personalization process also reduces the number of ROI while targeting more attention to a specific area.

On the contrary, for the test clients in the DriveAct dataset, the adjustment of the ROI is negligible. Note that the ROI does not necessarily have to cover the driver’s body or an object such as the magazine. The ROI should cover those pixels that can distinguish between different activities, such as static activities like reading the magazine, and dynamic activities like wearing a seatbelt in the DriveAct dataset activity setting. The ROI focuses on areas where large differences are likely to occur. The fact that the ROI in the DriveAct dataset covers almost the same pixels during the personalization process can also prove the negative impact of system heterogeneity on the FL framework.