Trusted AI in Multi-agent Systems: An Overview of Privacy and Security for Distributed Learning

FL is a collaborative ML technique [37, 38, 39] developed by Google, which allows decoupling of data provision at UEs, and machine learning model aggregation, such as network parameters of deep learning, at a centralized server. A structure of FL is plotted in Fig. 4. The purpose of FL is to cooperatively learn a global model without directly sharing data. In particular, FL has distinct privacy advantages compared to data center training on a dataset. At a server, holding even an anonymized dataset can put client privacy at risk via linkage to other datasets. In contrast, the information transmitted for FL consists of minimal updates to improve a particular ML model. The updates can be ephemeral, and will not contain more information than the raw training data (by the data processing inequality). Further, the source of the updates is not needed by the aggregation algorithm, and so updates can be transmitted without identifying metadata over a mixed network such as Tor [40] or via a trusted third party. General categories are distributed horizontal FL, where clients have different sample spaces with the same feature space, and share models during aggregation, distributed vertical FL with the same sample space and different feature spaces, sharing models or knowledge to the central server, and distributed transfer learning with various sample and feature spaces when uploading model or knowledge in aggregation [41].

However, although the data is not explicitly shared in the original format, it is still possible for adversaries to reconstruct the raw data approximately, especially when the architecture and parameters are not completely protected. In addition, FL can expose intermediate results such as parameter updates from an optimization algorithm like SGD, and the transmission of these gradients may actually leak private information when exposed together with a data structure such as image pixels. Furthermore, the well-designed attacks such as inference attack (stealing membership information) [42, 43, 44], and poisoning attack (polluting the quality of datasets or parameter models) [45] may induce further security issues.

Split learning, as a type of distributed deep learning [47, 17, 48, 49], has another name of split neural network (SplitNN). Similar to FL, split learning is effective when data uploading is not available because of privacy and legal restrictions. In the SplitNN, each participant first trains a NN until a predefined layer, called the cut layer, and then transmits the output of the cut layer to the server. Upon receiving the outputs, a central server will continue training the rest layers. Then, the loss function value is calculated and back-propagated to the participant. When receiving the feedback, the participant continues the back-propagation until the network finishes training. In Fig. 5, we show a combination of FL and split learning, where the logits are shared and aggregated at a centralized server.

The computational and communication costs on the client side are reduced in split learning because part of the network is processed locally. In addition, instead of transmitting the raw data, the activation function of the cut layer is uploaded to the server, which has a relatively smaller size. Some experimental results show that split learning has higher performances and fewer costs than FL over figure classification tasks, i.e., CIFAR-100 datasets, using Resnet-50 architectures for hundreds of clients-based setups [47]. However, it needs further explanations on how split learning works and makes decisions, which is linked to the trust of distributed networks, especially in the health area [47].

The difference between the large batch synchronous SGD-based collaborative learning and FL lies in that the updates in LBS-SGD are processed on each batch of training data, and multiple epochs of local training are required before uploading in FL. In LBS-SGD, model parallelism and data parallelism are two common ways to support updating, such as distributed large mini-batch SGD [50], distributed synchronous SGD with backups [51, 17], and selective SGD [52]. In [52], each participant is asked to choose a part of the models to update at each epoch and share them asynchronously with others. The work [50] considered synchronous SGDs by dividing local epochs into mini-batches over multiple clients and model aggregations. While the aggregated updates were performed synchronously in [50] that the aggregator will wait for all clients, the straggler may slow down the learning, and a synchronous optimization with backup participants has been provided in [51].

Reinforcement learning (RL) is trial-and-error learning by interacting directly with the environment, training according to the feedback, and finally achieving the designed goal. Specifically, RL defines a decision maker as an agent and the interaction with the environment, where three essential elements: the state, action, and reward, are used to describe the interaction. For each interaction, the client arrives at a certain state and processes a corresponding action, and then obtains feedback that is used to alter the current state to the next state. However, a single RL framework has no capability to address complex real-world problems, and thus, a multi-agent reinforcement learning system (MARL) has attracted increasing attention. Within a MARL, agents will cooperate with each other and observe the complex environment in a more comprehensive way. For example, as shown in Fig. 6, a three-agent reinforcement learning system, where actions and rewards are shared between different users, is provided. By absorbing the learning experiences from the user-self and other participants, a faster convergence rate with better performance is always achieved. However, compared to the single-agent setting, controlling multiple agents poses several additional challenges, such as the heterogeneity of participants, the design of achieved goals, and the more serious malicious client problem. Although a number of methods have been proposed to address these challenges, e.g., approximate actor-critic [53] and lenient-DQN, limitations like nonseasonal communication among agents and privacy leakage prevent the rapid development of MARL and existing methods cannot be extended to large-scale multi-agent scenarios.

Following the discussed background of distributed ML, we present the structure of this survey work in Fig. 7. The rest of the paper is structured as follows. In Section III, privacy and security issues are discussed and several robust protection methods are provided in Section IV. Then, in Section V, we survey the attacks and defenses in various paradigms in distributed ML. Several research challenges and future directions are shown in Section VI. Finally, conclusions are drawn in Section VII. In addition, a list of important abbreviations is provided in Table II.

Participants in distributed ML can be malicious or curious. For example, a car insurance company with limited user attributes might want to improve its risk evaluation model by incorporating more attributes from other businesses, e.g., a bank, a taxation office, etc. The role of the other participants is simply to provide additional feature information without directly disclosing their data to other participants, and in return, obtain financial and/or reputation rewards. However, the competitors may be disguised as collaborators, and then damage the training process or steal the ML model.

In terms of exchanged information, eavesdropping, modification or deletion can occur during communication in distributed ML as well. We can notice that the exchanged information usually contains the updated direction and extracted features from private data, and thus it is crucial to ensure its correctness, especially for the client-server framework. An external attacker may control the final output by modifying or deleting the exchanged information in the communication. In addition, via eavesdropping on the extracted features from private data, an external attacker can further infer sensitive information [54].

• •

  White-Box: The adversary is assumed to acknowledge certain information about the training data or the learning model, e.g., model parameters, network structures, or part of/the whole training dataset.

• •

  Black-Box: The adversary does not have any knowledge about the ML model, but the adversary can further explore the model by injecting some designed inputs and observing related outputs [55].

The second factor is the place where the attack happens:

• •

  Training Stage: The adversary attempts to learn the model by accessing a part or all of the training data, and creating a substitute model, i.e., a shadow model.

• •

  Inference Stage: The adversary observes the outputs from the learning and sums up the model characteristics [54].

A third factor is to distinguish between passive and active attacks.

• •

  Passive attack: The adversary can passively observe and obtain the updates but change nothing during the training process.

• •

  Active attack: The adversary actively performs and adjusts the learning operation. For example, the adversary can upload unreasonable parameters to degrade the aggregate model in FL [56].

The goal of a poisoning attack is to degrade the model quality, which misleads the learning to an incorrect direction by carefully crafting poisoning samples during training, also called adversarial examples [57]. In the black-box attack, the attacker can only inject a relatively small amount of crafted/poisoned data into the training model, where the amount and the undiscovered capability of these poisoning data are two basic metrics to estimate the attacking performance. For example, the authors in [58] have first investigated poisoning attacks against linear regression models and proposed a fast optimization algorithm with limited crafting samples to perturb outputs. Further, Suciu et al. have investigated the minimum information required by the attacker to achieve various attacking goals [59]. In the white-box attack, the adversaries have full knowledge of the training model and can take advantage of it to reconstruct a powerful poisoning attack. For example, Yuan et al. in [60] have proposed a white-box attack with perfect knowledge under different goals. Although the mentioned method might be unrealistic in practical settings, it can achieve almost five times than the black-box attack in success rate.

An evasion attack often happens in the prediction process, which aims to mislead the outputs. In detail, the evasion attack is to change real data from one category to a determined or random one and destroy the integrity of the original dataset. From a black-box attack angle, the adversary only knows the type of the training dataset and observes the outputs. Based on this assumption, the authors in [61] have realized it in the speech recognition system. The generated adversarial samples achieve a $91.67\%$ successful rate on moving one data from one category to another. While in the white-box attack, the adversary is able to acknowledge more useful information, such as the network structure and the type of training samples, rather than the predictive interface. For example, Kevin Eykholt et al. in [62] has shown the weakness for DNNs when random noises are added to the inputs and an advanced robust physical perturbations-based method has been proposed.

The model inversion attack proposed in [63] works in a black-box fashion, and the adversary only knows the input and can observe the corresponding outputs, which is used to detect correlations between uncertain inputs and respective outputs. A follow-up work has presented a combination with a black-and-white box attack [43]. The proposed attack aims to predict the highest probability of one input for a given label, which the adversary is able to reconstruct the input for a known label, i.e., a figure from a specific class. However, the proposed model inversion attack only works in linear models for most cases, and a major weakness is that the complexity grows exponentially with the input size since it relies on searching all linear combinations by brute force.

The membership inference attack (MIA) is mainly focused on privacy attacks. A previous attack targeting distributed recommend systems [64] intended to infer which input will lead to a change in the output by observing temporal patterns from the learning model. In [54], Shokri et al. have investigated the differences between the models to infer whether an input exists in the training dataset for the supervised model. In particular, a shadow model analogs as a similar structure to the targeted model in a black-box fashion. Following [54], Song et al. in [65] attempted to record the training data with black-box access. Then, the authors in [66] have exploited the knowledge of learning models to hide the Markov model and attack support vector machine in classification tasks. Also, related works [67, 44, 68] presented inference attacks against distributed deep learning [37, 52]. In particular, Aono et al. [67] aimed to attack the privacy-preserving learning framework proposed in [52], and revealed that partial data samples can be revealed by an honest-but-curious server. However, the operation that the single-point batch size limits its effectiveness. Also, a white-box attack against [52] has been proposed in [44], which used generative adversarial networks (GAN) to produce similar samples with a targeted training dataset, however, the proposed algorithm lost effectiveness in the black-box access. Finally, Truex et al. in [69] has shown that the MIA is usually data-driven, and Melis et al. in [68] have demonstrated the way that a malicious participant infers sensitive properties in distributed learning. Other MIAs focused on genomic research studies [70, 71], in which the attack is designed to infer the presence of specific information of individuals within an aggregated genomic dataset [71], locations [72], and noisy statistics in general [73].

• •

  Model Extraction. The aim of model extraction is first proposed in [74], in which they proposed to infer the parameters from a trained classifier with a black-box fashion; however, it only works when the adversary has access to the predictions, i.e., the probabilities for each class in a classification task. In follow-up works, other researchers went a step further to perform hyper-parameter stealing [75], which are external configurations. These values cannot be estimated by data samples, architecture extraction [76] that infers the deep model structures as well as the updating tools (e.g., SGD or alternating direction method of multipliers (ADMM)), etc.

• •

  Functionality Extraction. The concept of functionality extraction is, rather than stealing the model, to create knock-off models. Orekondy et al. [77] have processed this attack only based on design inputs and relative outputs to observe correlation from ML as a service (MLaaS) queries. In particular, the adversary uses the input-output pairs, e.g., image-prediction pairs in a figure classification task, to train a knock-off model, and compares it with one of the victims for the same task. In addition, the authors in [55] have trained a shadow model to replace a DNN which directly uses inputs generated by the attacker and labeled by the attacking DNN.

Although existing works have proposed a mount of mechanisms to hide identifiers of the raw data, it is also possible for attackers to steal privacy by analyzing hidden features [133]. Moreover, deep neural networks have been proven vulnerable to adversarial examples, which poses security concerns due to the potentially severe consequences [134]. This means that if some adversaries successfully make adversarial examples participate in system training, the training performance will be unacceptable.

Attacks on data publishing models can be mainly categorized as adversarial examples and feature identification based on their goals. As shown in Table III, we summarize possible attacks as follows.

• $\bullet$

  Adversarial examples (data poisoning). The work in [134] integrated the momentum term into the iterative process for attacks and generated more transferable adversarial examples by stabilizing update directions and escaping from poor local maxima during the generating iterations. The research on this area is faced with an “arms race” between attacks and defenses, i.e., a defense method proposed to prevent the existing attacks will be soon evaded by new attacks.

• $\bullet$

  Feature identification. Although many works have proposed efficient methods to process original data in order to preserve sensitive information. Many feature identification attacks are emerging to expose hidden information. As one of the feature identification attacks, structure-based de-anonymization attacks to graph data have been proposed, which aims to de-anonymize the private users in terms of their uniquely distinguishable structural characteristics [135].

Many defensive mechanisms have been designed against aforementioned attacks as shown in Table IV, and we will discuss various defenses as follows.

• $\bullet$

  Adversarial training. Adversarial training is among the most effective techniques to improve model robustness by augmenting training data with adversarial examples. The work in [140] has proposed an adversarial distributional training (ADT) framework, which is formulated as a mini-max optimization problem and improves the model robustness obviously. In this framework, the inner maximization aims to learn an adversarial distribution to characterize the potential adversarial examples around a natural one under an entropic regularizer, and the outer minimization aims to train robust models by minimizing the expected loss over the worst-case adversarial distributions.

• $\bullet$

  Anonymization. An anonymization operation comes in several flavors: generalization, suppression, anatomization, permutation, and perturbation [141, 142]. These techniques aim to remove or hide identifying characteristics from raw data while guaranteeing the data utility. An information-theoretic approach has been formulated and proposed a new multi-objective loss function for training deep auto-encoders [143], which helps to minimize user-identity information as well as data distortion to preserve the application-specific utility. The work in [144] has proposed the conditional identity anonymization generative adversarial networks (CIA-GAN) model, which can remove the identifying characteristics of faces and bodies while producing high-quality images and videos that can be used for various computer vision tasks, such as detection or tracking. Unlike previous methods, CIA-GAN has full control over the de-identification (anonymization) procedure, ensuring both anonymization as well as diversity. In summary, the choice of anonymization operations has an implication for the search space of anonymous tables and data distortion. The full-domain generalization has the smallest search space with the largest distortion, and the local recording scheme has the largest search space but the least distortion.

• $\bullet$

  Dummy. Existing research methods to protect data privacy mainly focus on the protection of the user’s identities through anonymity. User attributes can be classified into identity information, quasi-identifier, and sensitive information. Given an anonymity table, if the attributes in the table have not been properly treated, an adversary may deduce the relationship between the user’s identity and sensitive information according to the user’s quasi-identifier, such as age and gender. A popular approach for data anonymity is $k$-anonymity, and any record in a $k$-anonymized dataset has a maximum probability $1/k$ of being re-identified [145, 146, 147]. The privacy model $l$-diversity and $t$-closeness in [148] further refines the concept of diversity and requires that the distribution of the sensitive values of each equivalent class should be as close as to the overall distribution of the dataset. The common rules for these algorithms are basically to produce dummy records to hide the real ones. In addition, the dummy-based methods also work for location privacy protection. Dummy data along with the true one will be sent to the server from users, which may hide the client’s contribution during training [149]. Because the collection is processed on the server, the system performance can still be guaranteed. As an efficient method to generate realistic datasets, GANs provide an alternative to balance user privacy and training performance. The work in [150] has proposed a novel data augmentation technique based on the combination of real and synthetic heartbeats using GAN to improve the classification of electrocardiogram (ECG) heartbeats of 15 different classes from the MIT-BIH arrhythmia dataset²²2https://www.physionet.org/content/mitdb/1.0.0/.

• $\bullet$

  DP. As a promising solution, a mechanism is said to be differentially private [124] if the computation result of a dataset is robust to any change of the individual sample. Several differentially private machine learning algorithms [151] have been developed in the community, where a trusted data curator is introduced to gather data from individual owners and honestly runs the private algorithms. Compared to DP, Local DP (LDP) [131, 132] eliminates the need for a trusted data curator and is more suitable for distributed ML. Rappor [152], which applies LDP by Google, is designed to collect the perturbed data samples from multiple data owners. Besides simple counting, a follow-up paper [153] shows that Rappor can also compute other types of statistics such as joint-distribution estimation and association testing. Besides Rappor, an alternative way that achieves DP is to add random noise on the sample value before publishing [131, 154]. To process this method, a numerical sample is always normalized and a categorical one is transformed to the same range by one-hot coding. In addition, the authors in [155] adopted the DP algorithm to handle the privacy concern in a communication problem that each distributed client needs to transmit data to one aggregated center to learn a model. The work[156] has proposed a distributed edge computing which for image classification, where each edge will upload its raw data after coding to latent data to protect privacy.

• $\bullet$

  Encryption. The work in [157] has instantiated a scalable privacy-preserving distributed learning (SPINDLE), an operational distributed system that supports the privacy-preserving training and evaluation of generalized linear models on distributed datasets. Moreover, it relies on a multiparty HE scheme to execute high-depth computations on encrypted data without significant overhead. The work in [158] has proposed a distributed algorithm for distributed data, where privacy is achieved by the data locality property of the Apache Hadoop architecture and only a limited number of cryptographic operations are required.

• $\bullet$

  Others. The work in [159] has aimed to develop secure, resilient, and distributed ML algorithms under adversarial environments. This work has established a game-theoretic framework to capture the conflicting interests between the adversary and a set of distributed data processing units. The Nash equilibrium of the game has allowed for predicting the outcome of learning algorithms in adversarial environments and enhancing the resilience of the ML through dynamic distributed learning algorithms.

• •

  RAPPOR. Randomized aggregatable privacy-preserving ordinal response provides a privacy-preserving way to learn software statistics to better safeguard users security, find bugs, and improve the overall user experience. Building on the concept of randomized response, RAPPOR enables learning statistics about the behavior of users software while guaranteeing client privacy [152]. The guarantees of differential privacy, which are widely accepted as the strongest form of privacy, have almost never been used in practice despite intense academic research. RAPPOR introduces a practical method to achieve those guarantees. In detail, the core of RAPPOR is a randomized response mechanism [161] for a user to answer a yes/no query to the record aggregator. A classic example is to collect statistics about a sensitive group, in which the aggregator asks each individual: “Are you a doctor?” To answer this question, each individual tosses a coin, gives the true answer if it is a head, and a random yes/or answer otherwise. This randomized approach provides plausible deniability to the individuals. Meanwhile, it is shown to satisfy $\epsilon$-LDP, and the strength of privacy protection (i.e., $\epsilon$) can be controlled by using a biased coin. Based on the collected randomized answers, the aggregator estimates the percentage of users whose true answer is “yes” (resp. “no”). RAPPOR allows the software to send reports that are effectively indistinguishable and are free of any unique identifiers. RAPPOR is currently an available implementation in Chrome, which learns statistics about how unwanted software is hijacking users settings.

• •

  DP in the IOS system. Apple has adopted and further developed local DP to enable Apple to learn about the user community while avoiding learning about individuals [162]. DP perturbs the information shared with random noise before it ever leaves the user s device such that Apple can never reproduce the raw data. The power of additive noise that has been added can be reduced without exposing raw data from users by averaging out over large numbers of data points, and meaningful information emerges. DP is utilized as the first step of a system for data analysis that consists of robust privacy protections at every stage. The system is optional and developed to provide transparency to users. Device identifiers are removed from the data, and it is transmitted to Apple over an encrypted channel. The Apple analysis system ingests the differentially private contributions, dropping IP addresses and other metadata. The final stage is aggregation, where the private records are processed to compute the relevant statistics and the aggregate statistics are then shared with relevant Apple teams. Since both the ingestion and aggregation stages are performed in a restricted access environment, the raw data is not broadly accessible to the public.

The guarantee of privacy and security in terms of data sharing models relies on the pre-processing of the raw data, such as perturbation, dummy, anonymization, and encryption. As shown in Fig. 8, data pre-processing happens at the first stage of a ML task, and thus, these pre-processing techniques are usually harmful to the utility of systems or involved extra computations. Therefore, it is more practical to select a proper mechanism to hide sensitive information from shared data while alleviating the negative influences on the system’s utility.

Although data is not required to upload in model sharing systems, private information can still be divulged by analyzing uploaded model parameters, e.g., weights trained in deep neural networks. Moreover, adversarial participants may degrade or even destroy the training systems by uploading unreliable models. Attacks can be carried out by the following three aspects.

• $\bullet$

  Insiders vs. outsiders. Insider attacks include those launched by the server and the participants in the model sharing systems. Outsider attacks include those launched by the eavesdroppers in the wireless transmission environment between participants and the server, and by users of the final model when it is deployed as a service. Insider attacks are generally stronger than outsider attacks, as it strictly enhances the capability of the adversary.

• $\bullet$

  Semi-honest vs. malicious. Under the semi-honest setting, adversaries are considered passive or honest but curious. They try to learn the private states of other participants without deviating from the model sharing protocol. The passive adversaries are assumed to only observe the aggregated or averaged gradient, but not the training data or gradient from other honest participants. Under the malicious setting, an active, or malicious adversary tries to learn the private states of honest participants and deviates arbitrarily from the model sharing protocol by modifying, re-playing, or removing messages. This strong adversary model allows the adversary to conduct particularly devastating attacks.

• $\bullet$

  Poisoning vs. inference. Attacks at the poisoning phase attempt to learn, influence, or corrupt the model sharing itself. During the poisoning phase, the attacker can run data poisoning attacks to compromise the integrity of training dataset collection, or launch model poisoning attacks to compromise the integrity of the learning process. The attacker can also launch a range of inference attacks on an individual participant’s update or on the aggregation of updates from all participants.

Attacks to model sharing models can be categorized as poisoning attacks, inference attacks, and model inversion based on their various goals as shown in Table V. We also summarize them as follows.

• $\bullet$

  Poisoning attack. Compromised clients by attackers always have opportunities to poison the global model in model sharing systems, in which local models are continuously updated by clients throughout their deployments. Moreover, the existence of compromised clients may induce further security issues such as bugs in pre-processing pipelines, noisy training labels, as well as explicit attacks that target training and deployment pipelines [188]. In order to destroy ML models, poisoning attackers may control part of clients and manipulate their outputs sent to the server. For example, the compromised clients can upload noisy and reversed models to the server at each communication round [189, 178], which has the advantage of low complexity to mount attacks. Other attackers may manipulate the outputs of the compromised clients carefully to achieve the evasion of defenses and downgrade the performance of ML models. Furthermore, the authors in [165, 190] have formulated the local model poisoning attack as optimization problems, and then apply this attack against recent Byzantine-robust FL methods. In this way, attackers can improve the success rate of attacks, dominate the cluster and change the judgment boundary of the global model, or make the global model deviate from the right direction. Besides, attackers may hope to craft the ML model to minimize this specific objective function instead of destroying it. Via using multiple local triggers and model-dependent triggers (i.e., generated based on local models of attackers), the collusive attackers can conduct backdoor attacks successfully [191]. Bagdasaryan et al. in [163] have developed and evaluated a generic constrain-and-scale technique that incorporates the evasion of defenses into the attacker’s loss function during training. The work in [164] has explored the threat of model poisoning attacks on FL initiated by a single, non-colluding malicious client where the adversarial objective is to cause the model to misclassify a set of chosen inputs with high confidence.

• $\bullet$

  Inference attack. The work in [190] has presented a new attack paradigm, in which a malicious opponent may interfere with or backdoor the process of distributed learning by applying limited changes to the uploaded parameters. The work in [163] has proposed a new model-replacement method that demonstrated its efficacy on poisoning models of standard FL tasks. Inferring privacy information about clients for attackers is also possibly achievable in ML models. A generic attacking framework mGAN-AI that incorporates a multi-task GAN has been proposed in [192], which conducted novel discrimination on client identity, achieving attack to clients’ privacy, i.e., discriminating a participating party’s feature values, such as category, reality, and client identity.

• $\bullet$

  Model inversion. By casting the model inversion task as an optimization problem, which finds the input that maximizes the returned confidence, the work in  [43] has recovered recognizable images of people’s faces given only their names and accesses to the ML model. In order to identify the presence of an individual’s data, an attack model trained by the shadow training technique has been designed, and can successfully distinguish the target model’s outputs on members versus non-members of its training dataset [52].

Specifically, in distributed reinforcement learning (DRL) systems, there has been literature available on security vulnerabilities. We provide many characteristics of an adversary’s capabilities and goals that can be studied as follows. First, we divide attacks based on what components in an MDP the attacker chooses to attack: the agent’s observations, actions, and environment (transition) dynamics. Then, we discuss the practical scenarios where attacks happen on these components.

• $\bullet$

  Observations. Existing work on attacking DRL systems with adversarial perturbations focuses on perturbing an agent’s observations, i.e., states and rewards, that are communicated between the agent and the environment. This is the most appealing place to start, with seminal results already suggesting that recognition systems are vulnerable to adversarial examples [136, 193, 194, 195, 196, 197, 198, 199, 200, 201]. Sandy et al. [136] have first shown that adversarial attacks are also effective when targeting neural network policies in RL adversarial examples. Based on this technique, part of the works enhance adversarial examples to attack DRL. To improve the attack efficiency, the strategically-timed attack [193], consuming a small subset of time steps in an episode, has been explored. Via stamping a small percentage of inputs of the policy network with the Trojan trigger and manipulating the associated rewards, the work in [197] has proposed the TrojDRL attack, which can deteriorate drastically the policy network in both targeted and untargeted settings. Another fancy idea for a reward-poisoning attack is to design an adaptive disturbing strategy [198], where the infinity norm constraint is adjusted on the DRL agent’s learning process at different time steps. For the theoretical analysis, two standard victims with adversarial observations, i.e., tabular certainty equivalence learner in reinforcement learning and linear quadratic regulator in control have been analyzed in a convex optimization problem on which global optimality and the attack feasibility and attack cost have been provided [196]. In addition, the effectiveness of a universal adversarial attack against DRL interpretations (i.e., UADRLI) has been verified by the theoretical analysis [199], from which the attacker can add the crafted universal perturbation uniformly to the environment states in a maximum number of steps to incur minimal damage. In order to stealthily attack the DRL agents, the work in [200] has injected adversarial samples in a minimal set of critical moments while causing the most severe damage to the agent. Another work in [201] has formulated an optimization framework in a stealthy manner to find an optimal attack for different measures of attack cost and solved it with an offline and online setting.

• $\bullet$

  Actions. Attacks applied on the action space usually aim to minimize the expected return or lure the agent to a designated state, e.g., the action outputs can be modified by installing some hardware virus in the actuator executing process. This can be realistic in certain robotic control tasks where the control center sends some control signals to the actuator. A vulnerability in the implementation, i.e., the vulnerability in the blue-tooth signal transmission, may allow an attacker to modify that signal [202]. A training policy network to learn the attack has been developed, which treats the environment and the original policy together as a new environment, and views attacks as actions [137]. However, existing works only concentrate on the white-box scenario, i.e., knowing the victim’s learning process and observations, which is not practical and inaccessible to attackers.

• $\bullet$

  Environment Dynamics. The environment (transition) dynamics can be defined as a probability mapping from state-action pairs to states, which is governed by the environmental conditions. For attacks applied on the environment dynamics, an attacker may infer environment dynamics[166] or perturb a DRL system’s environment dynamics to make an agent fail in a specific way [137, 203, 138, 201]. In the autonomous driving case, the attacker can change the material surface characteristic of the road such that the policy trained in one environment will fail in the perturbed environment. In a robot control task, the attacker can change the robot’s mass distribution so that the robot may lose balance when executing its original policy because it has not been trained in that case.

Then, we categorize these attacks based on what knowledge the attacker needs. Broadly, this breaks attacks down into the already recognized white-box attacks, where the attacker has full knowledge of the DRL system, and black-box attacks, where the attacker has less or no knowledge.

• $\bullet$

  White-Box. If the adversary attacks the DRL system with the capability of accessing the architecture, weight parameters of the policy and Q networks, and querying the network, we can call it a white-box attack. Clearly, the attacker can formulate an optimization framework for the white-box setting and derive the optimal adversarial perturbation [136, 199]. Moreover, via the theoretical analysis of the attack feasibility and cost, the adversary can further decrease the efficiency and stealth of the learning [196, 137]. However, this setting is inaccessible for the adversary in most practical scenarios.

• $\bullet$

  Black-Box. In general, the trained RL models are kept private to avoid easy attacks by certain secure access control mechanisms. Therefore, the attacker cannot fully acknowledge the weight parameters of the policy network and Q networks, and may or may not have access to query the policy network. In this case, the attacker can train a surrogate policy to imitate the victim policy, and then use a white-box method on the surrogate policy to generate a perturbation and applies that perturbation to the victim policy [137]. The finite difference (FD) method [204] in attacking classification models can be utilized to estimate the gradient on the input observations, and then perform gradient descent to generate perturbations on the input observations [137]. In this black-box setting, the adversary becomes difficult to perturb a DRL system and needs to estimate the victim’s information with large computation costs, such as policies and observations.

Based on the adversary’s objective, adversarial attacks are divided into two types: poisoning attacks and snooping attacks.

• $\bullet$

  Poisoning Attack. In particular, for poisoning attacks, there are at least two dimensions to potential attacks against learning systems as untargeted attacks [136] and targeted (induction) attacks [194]. In untargeted attacks, attackers focus on the integrity and availability of the DRL system, i.e., minimizing the expected return (cumulative rewards). Specifically, the work  [136] has shown existing adversarial example crafting techniques can be used to significantly degrade the test-time performance of trained policies. However, in terms of defensive mechanisms, the attacker may control time steps [200] or solve an optimization framework in a stealthy manner [199]. Another attack of this category aims at maliciously luring an agent to a designated state more than decreasing the cumulative rewards [194]. Via combining a generative model and a planning algorithm, the generative model predicts the future states and the planning algorithm generates a preferred sequence of actions for luring the agent [193]. Similar to untargeted attacks, by solving an optimization framework in a stealthy manner [201], the attacker can easily succeed in teaching any target policy.

• $\bullet$

  Snooping Attack. Different from poisoning attacks, the attacker only aims to eavesdrop on environment dynamics, the action and reward signals being exchanged between the agent and the environment. If the adversary can train a surrogate DRL model that closely resembles the target agent [166, 167], the desired information can be estimated by this model. Furthermore, the adversary only needs to train a proxy model to maximize reward, and adversarial examples crafted to fool the proxy will also fool the agent [205]. We can note that the snooping attacks can still launch devastating attacks against the target agent by training proxy models on related tasks, and leveraging the transfer-ability of adversarial examples.

Defensive mechanisms found in multiple works of literature are grouped by their underlying defensive strategies as shown in Table VI. We will discuss various defenses in model sharing frameworks as follows.

• $\bullet$

  DP. DP tackles the privacy leakage about the single data change in a dataset when some information from the dataset is publicly available and is widely used due to its strong theoretical guarantees [206]. Common DP mechanisms will add an independent random noise component to access data, i.e., the shared models in this level, to provide privacy. DP preserving distributed learning systems have been studied from various paradigms, such as distributed principal component analysis (PCA) [169], distributed ADMM [170], distributed SGD [127], FL [171, 172] and multi-agent reinforcement learning [173, 174]. In order to provide fine-tuned control over the trade-off between the estimation accuracy and privacy preservation, a distributed privacy-preserving sparse PCA (DPS-PCA) algorithm that generates a min-max optimal sparse PCA estimator under DP constraints has been proposed in [169]. Similarly, for distributed ADMM, distributed SGD, FL, and multi-agent reinforcement learning systems, all related works focus on improving the utility-privacy trade-off via two aspects as follows: a) analyzing the learning performance with a DP constraint and then optimizing system parameters; b) enhancing the DP mechanism by obtaining tighter estimates of the overall privacy loss.

• $\bullet$

  Model compression. Model compression techniques for distributed SGD and FL systems, e.g., sketches, can achieve provable privacy benefits [115, 175]. Therefore, a novel sketch-based framework (DiffSketch) for distributed learning has been proposed, improving absolute test accuracy while offering a certain privacy guarantee and communication compression. Moreover, the work in [175] has presented a family of vector quantization schemes, termed Vector-Quantized Stochastic Gradient Descent (VQSGD), provides an asymptotic reduction in the communication cost and automatic privacy guarantees.

• $\bullet$

  Encryption. Encryption, e.g., HE [117] and MPC [78], is also adopted to protect user data privacy through parameter exchange under the well-designed mechanism during ML. A novel deep learning system [117], bridging asynchronous SGD and cryptography, has been proposed to protect gradients over the honest-but-curious cloud server, using additively homomorphic encryption, where all gradients are encrypted and stored on the cloud server. To verify whether the cloud server is operating correctly, VerifyNet [176] has been proposed to guarantee the confidentiality of users’ local gradients via a double-masking protocol in FL, where the cloud server is required to provide proof of the correctness of its aggregated results to each user.

• $\bullet$

  MPC. The work in [78] has outlined an approach to advancing privacy-preserving ML by leveraging MPC to compute sums of model parameter updates from individual users’ devices in a secure manner. The problem of computing a multiparty sum where no party reveals its updates to the aggregator is referred to as secure aggregation. Via encoding local models into multiple secret shares in the first round, and then splitting each share into a public share and a private share, the work in  [207] can provide stronger protections for the security and privacy of the training data. MPC integrates the encryption technology and interactive protocols, aiming to make the receiver keep away from sensitive information and obtain the necessary messages  [208, 209, 210, 211].

• $\bullet$

  Statistical analysis. The work in [177] has proposed a robust aggregation rule, called adaptive federated averaging, that detects and discards bad or malicious local model updates based on a hidden Markov model. To tackle adversarial attacks in the FL aggregation process, the work in [178] presented a novel aggregation algorithm with the residual-based re-weighting method, in which the weights for the average of all local models are estimated robustly. Via controlling the global model smoothness based on clipping and smoothing on model parameters, a sample-wise robustness certification FL framework has been proposed, which can train certifiably robust FL models against backdoors [212]. Most of the defenses for FL aim to explore the latent model exception, such as similarities between malicious and benign clients, and then lessen the influence of these exceptional models [213, 214, 215, 216].

• $\bullet$

  Pretest on auxiliary datasets For detecting poisoned updates in collaborative learning [179], the results of client-side cross-validation were applied for adjusting the weights of the updates when performing aggregation, where each update is evaluated over other clients’ local data. The work in [179] considered the existence of unreliable participants and used the auxiliary validation data to compute a utility score for each participant to reduce the impact of these participants. The work in [180] has proposed a novel poisoning defense method in FL, in which the participant whose accuracy is lower than a predefined threshold will be identified as an attacker, and the corresponding model parameters will be removed from the training procedure in this iteration.

• $\bullet$

  Authentication and access control. The key question in considering security in a MARL consists of increasing the confidence that all parties involved in the system (agents, platforms, and users) will behave correctly, and this can be achieved through the authentication of these parties. The identification of the parties can make up a system and possibly establish an agent-trust relationship. Thus, how to design efficient identity certification mechanisms to uniquely authenticate known and trusted users and agents in the system has drawn heated attention. A domain-independent and reusable MARL infrastructure has been developed in [217], in which the system uses a certification authority (CA) and ensures full cooperation of secured agents and already existing (unsecured) agents. The work in [181] has introduced a method called trust composition, which combines several trust values from different agents. We can note that the trust composition can play a critical role in determining the trust and reputation values of unknown agents since it is impractical for an agent to get complete knowledge about other agents. A work called PTF (Personalized Trust Framework) has been proposed to establish a trust/reputation model for each application with personalized requirements [218]. Naturally, the idea of using blockchain technology to solve security problems in multi-robot systems was discussed in [182]. The work in [182] stated that combining peer-to-peer networks with cryptographic algorithms allows reaching an agreement by a group of agents (with the following recording this agreement in a verifiable manner) without the need for a controlling authority. Thus, blockchain-based innovations can provide a breakthrough in MARL applications. The work in [183] has developed an approach to using decentralized programs based on smart contracts to create secure swarm coordination mechanisms, as well as for identifying and eliminating Byzantine swarm members through collective decision-making. The work in [184] has proposed an approach combining blockchain technology and explainability supporting the decision-making process of MARL, in which blockchain technology offers a decentralized authentication mechanism capable of ensuring trust and reputation management.

• $\bullet$

  Authorization and trust model. Combined with authentication, authorization is used to restrict the actions that an agent can perform in a system, and control the access to resources by these agents. Sensitive information about principals is transferred online even across the Internet and is stored in local and remote machines. Without appropriate protection mechanisms, a potential attacker can easily obtain information about principals without their consent. In the context of authorization mechanisms, the algorithm proposed in [185] is designed to solve the problem of systems that are constantly changing. The main goal is to build a flexible and adaptive security policy management capable to configure itself and reflect the actual needs of the system. According to the authors, a system is not safe if a security model is developed but never managed afterward. Security of the proposed system in [186] has been further explored in the form of authorization and encryption of the data by introducing an authorization layer between the user and the system that will be responsible for providing access to the legitimate users of the system only. The work in [187] has ensured agent authorization and platform security with capability-based access and different agent privilege levels, in which the agent behavior is modeled with an activity transition graph (ATG) and implemented entirely in JavaScript with a restricted and encapsulated access to the platform API (AgentJS).

• •

  Electronic Medical Records (EMR) [219]. The use of information and network technologies in the healthcare field inevitably produces EMR, which is a necessary trend for the modernization of medical records in hospitals. The initial adoption of EMR in clinical practice has vastly improved the efficiency and quality of health care provided by hospitals. Empowered by algorithm technologies and data reconstruction, BaseBit [219] has constructed a robust and comprehensive knowledge base system and has a series of intelligent models with excellent abilities of expression. In various applications centered around electronic medical records, the proposed models effectively improve the abilities such as automatic medical record writing, overall quality control, cost monitoring systems for single diseases, early warning for infectious diseases, prompt for critical illnesses, clinical decision-making assistance for rare diseases, enabling hierarchical diagnosis and treatments.

As shown in Fig. 9, although due to the local training process, the raw data of each participant will not be exposed to the curious server or external attackers, defensive mechanisms are also necessary because of the existing possibility of feature inference and data reconstruction from models sharing, in addition to the model poisoning paradigm. Traditional HE and DP are proven beneficial to privacy preservation but lead to low efficiency or damaged utility. Therefore, the quantitative analysis of the relationship between the sensitive feature and the published model is imperative.

In knowledge sharing paradigms, adversarial participants or eavesdroppers still possibly exist. The adversarial participants can be categorized into two kinds: a) honest-but-curious (semi-honest) participants, who do not deviate from the defined learning protocol, but attempt to infer private training data from the legitimately received information; b) malicious participants, who may deviate from the defined learning protocol, and destroy this training task or inject Trojans to the training model.

Existing attacks on knowledge sharing paradigms can be mainly categorized as label leakage, feature inference, and data reconstruction as shown in Table VII. Then, we discuss existing attacks as follows.

• $\bullet$

  Label leakage. The labels in distributed learning frameworks might be highly sensitive, e.g., whether a person has a certain kind of disease. However, the bottom model structure and the gradient update mechanism of VFL or split learning can be exploited by a malicious participant to gain the power to infer the privately owned labels [226]. Worse still, by abusing the bottom model, he/she can even infer labels beyond the training dataset [227]. The work in [220] first made an attempt at a norm attack that uses the norm of the communicated gradients between the parties, and it can largely reveal the ground-truth labels from participants. The adversary (either clients or servers) can accurately retrieve the private labels by collecting the exchanged gradients and smashed data [228]. Thus, it is necessary to make gradients from samples with different labels similar.

• $\bullet$

  Feature inference. Through analysis, the work in [229, 230] demonstrated that, unless the feature dimension is exceedingly large, it remains feasible, both theoretically and practically, to launch a reconstruction attack with an efficient search-based algorithm that prevails over current feature protection techniques. In this paper, the authors have performed the first systematic study on relation inference attacks to reveal VFL’s risk of leaking samples’ relations. Specifically, the adversary is assumed to be a semi-honest participant. Then, according to the adversary’s knowledge level, the work  [230] formulated three kinds of attacks based on different intermediate representations and revealed VFL’s risk of leaking samples’ relations. Luo et al. [192] considered the most stringent setting that the active party (i.e., the adversary) only controls the trained vertical FL model and the model predictions, and then observed that those model predictions will leak a lot of information about features by learning the correlations between the adversary’s and the attacking target’s features.

• $\bullet$

  Data reconstruction. The work in [221] has provided the leakage analysis framework via three empirical and numerical metrics (distance correlation and dynamic time warping) indicating that the activated outputs after two and more convolutional layers can be used to reconstruct the raw data, i.e., sharing the intermediate activation from these layers may result in severe privacy leakage. In vertical FL, two simple yet effective attacks, reverse multiplication attack and reverse sum attack, have been proposed to steal the raw training data of the target participant [222]. Though not completely equivalent to the raw data, these stolen partial orders can be further used to train an alternative model which is as effective as the one trained on the raw data [231].

Defensive mechanisms found in multiple works of literature are grouped by their underlying defensive strategy as shown in Table VIII. Hence, we will discuss various defenses in model sharing frameworks as follows.

• $\bullet$

  DP. The work in [232] has proposed a privacy-preserving protocol for composing a differentially private aggregate classifier using local classifiers from different parties. In order to overcome the effects of the proposed information inference attacks [221], DP has been proven helpful in reducing privacy leakage but leading to a significant drop in model accuracy.

• $\bullet$

  MPC. The work in [233] has proposed a novel solution for privacy-preserving vertical decision tree training and prediction, termed Pivot, ensuring that no intermediate information is disclosed other than necessary releases (i.e., the final tree model and the prediction output).

• $\bullet$

  Encryption. A novel privacy-preserving architecture has been proposed in [234], which can collaboratively train a deep learning model efficiently while preserving the privacy of each party’s data via the HE technique. The work in [234] has explored a lossless privacy-preserving tree-boosting system known as SecureBoost by using the additive HE scheme.

• $\bullet$

  Secure aggregation. The work in [235] has proposed the vertical FederBoost which runs the gradient boosting decision tree (GBDT) training algorithm in exactly the same way as the centralized learning. Via further utilizing packetization and DP, this algorithm can protect the order of samples: participants partition the sorted samples of a feature into buckets, which only reveals the order of the buckets and add differentially private noise to each bucket.

• $\bullet$

  Others. The work in [236] has presented TIPRDC to learn a feature extractor that can hide the private information from the intermediate representations using an adversarial training process while maximally retaining the original information embedded in the raw data to accomplish unknown learning tasks. In [221], adding more hidden layers to the client side was proven helpful in reducing privacy leakage, but increasing the number of layers seems ineffective with the most highly correlated channels. In order to relieve the negative impact of random perturbation preserving techniques on the learned model’s predictive performance, the work in [220] has introduced an improved way to add Gaussian noise by making the expected norm of the positive and negative gradients in a mini-batch equal (un-distinguishable).

• •

  FATE. An open-source project, named FATE, provides a secure computing framework to support the federated AI ecosystem [237], led by Webank s AI Department. It can enable big data collaboration without privacy leakage by implementing multiple secure computation protocols, such as DP, HE, and so on. FATE accesses out-of-box usability and excellent operational performance with a modular modeling pipeline, explicit visual interface, and flexible scheduling system [238]. eHi Car Services, a national chain car rental brand, and WeBank jointly announced a deep strategic partnership, announcing that the two sides will carry out multi-scene and multi-dimensional innovation cooperation in car travel, member services, finance and insurance, blockchain technology, and other fields. eHi Car Services uses federal transfer learning, AI face authentication technology, payment technology, and other fin-techs to deeply integrate into the car rental service process for the purpose of optimizing and improving user experience, and combines the car rental scene with the bank’s big data risk control system, so as to provide a new way of travel and life for the young and long-term rental customers.

As shown in Fig. 10, split learning, vertical FL, and distillation-Based FL are the classical knowledge sharing systems, in which the knowledge can be viewed as the partial processing result to meet the requirement of the system learning. It is also challenging for knowledge sharing systems to hide sensitive information from the shared knowledge.

For the result sharing models, malicious participants may exist and provide false advice or results to hinder the learning performance of other participants or the global model. In addition, curious participants can infer some confidential information from the shared results.

As stated by da Silva et al.[243], the existence of malicious participants is a key concern in agent advises. The work in [244] has proposed the attack model that some of these agents might become self-interested and try to maximize car owners’ utility by sending out false information. Based on [244], Hayes et al. [245] have investigated attacks in the setting where the adversary is only permitted to access the shared results (such as the generated samples set in GAN), by retraining a local copy of the victim model. In addition, Hilprecht et al. [246] have proposed to count the number of generated samples that are inside an $\epsilon$-ball of the query, based on an elaborate design of distance metric. The work in [247] has presented the first taxonomy of membership inference attacks and focused on membership inference attack against deep generative models that reveals information about the training data used for victim models. In spirit to Hilprecht et al. [246], this work scored each query by the reconstruction error directly, which does not introduce additional hyper-parameter while achieving superior performance. We further summarize these attacks in Table IX.

In results sharing paradigms, Tab. X summarizes the use case, key idea, and effectiveness for existing attacks. Moreover, we will discuss various defenses in model sharing frameworks as follows.

• $\bullet$

  DP. The work in [174] has proposed a novel differentially private agent advising approach, which employs the Laplace mechanism to add noise to the rewards used by student agents to select teacher agents. By using the advising approach and the DP technique, this approach can reduce the impact of malicious agents without identifying them and naturally control communication overhead. The work in [248] adopted DP and studied the regret upper and lower bounds for MAB algorithms with a given local DP guarantee. The differentially private PATE framework has been proposed to achieve individual privacy guarantees with provable privacy bounds [250, 251].

• $\bullet$

  MPC. Zhao [249] has proposed to use the teacher-student framework in a more general distributed learning setting. The goal of this work is to address distributed deep learning under DP using the teacher-student paradigm. In the setting, there are a number of distributed entities and one aggregator. Each distributed entity leverages deep learning to train a teacher network on sensitive and labeled training data. The knowledge of the teacher networks is transferred to the student network at the aggregator in a privacy-preserving manner that protects the sensitive data. This transfer results from training non-sensitive and unlabeled data, which also applies secure MPC to securely combine the outputs of local ML for updating.

• $\bullet$

  Others. If an ensemble contains enough models, and each model is trained with disjoint subsets of the training data in a distributed manner, then “any predictions made by most of the models should not be based on any particular part of the training data” [252]. The private aggregation of teacher ensembles (PATE) is based on this idea [10]. In more detail, the ensemble is seen as a set of “teachers” for a new “student” model. The student is linked to the teachers only by their prediction capabilities, and is trained by “querying the teachers about unlabelled examples”. The prediction result is disjointed from the training data through this process. Therefore data privacy can be protected. The privacy budget for PATE is much lower than traditional DP-based ML approaches. But it may not work in many practical scenarios as it relies on an unlabelled public dataset.

• •

  Large-scale online taxicab platforms, such as Uber and DiDi, have revolutionized the way people travel and socialize in cities worldwide and are increasingly becoming essential components of the modern transit infrastructure [253, 254]. The reinforcement learning-based dynamic bipartite graph matching approach has been adopted to assign each worker with one or more tasks to maximize the overall revenue of the platform, where the workers are dynamic while the tasks arrive sequentially. Specifically, for each worker-task pair, the platform can obtain a reward based on value-based reinforcement learning. Then, via some solutions to bipartite graph matching, such as greedy search, the platform can make near-optimal decisions. However, if the platform can obtain all workers information and its purpose is only aiming to maximize the overall revenue, workers may be out of control. Thus, using DP to achieve fairness may be a solution [255].

As shown in Fig. 11, although the results from ML systems are various from the raw data, they are also existing risks of privacy leakage, such as the generated samples from the generator in GAN. Hence, several defensive mechanisms are utilized for preventing privacy leakage and against malicious participants.

The concerns of security and privacy issues are different [256, 257, 258]. On the one hand, security issues refer to unauthorized/malicious access, change, or denial of data or learning models. Such attacks are usually launched by adversaries with expert/full knowledge of the target system. Hence, the fundamental three goals of security are confidentiality, integrity, and availability [165]. On the other hand, privacy issues generally refer to the unintentional disclosure of personal information. For example, from a side-by-side comparison of a vote registration dataset and an anonymous set of health-care sensor records (e.g., no individuals name and ID), an adversary may have the ability to identify particular individuals and the health conditions of these individuals leaks [259, 68, 260]. This is because attributes such as gender, birth date, and zip code are the same in both datasets.

Security and privacy go hand-in-hand. Privacy issues can further induce security issues in some scenarios. If an adversary steals the private information of individuals, substantial profit from the information can be easily obtained. For example, when the adversary extracts the health conditions of an important person, he/she can blackmail the victim person by threatening to reveal the information. We know that one can envision an environment that is secure but does not guarantee privacy. Similarly, one can imagine an environment that is private, but it does not guarantee security from outsiders. Security can be achieved without privacy, but privacy cannot be achieved without security. This is because whether the security is weak or vulnerable, it will automatically affect privacy.

Privacy leakage can be formalized as a Bayes optimization problem from the aspect of an adversary with different assumptions on the probability distributions of the input data and interactive messages (such as gradients and extracted features). For example, the work in [261] constructed a theoretical framework that can measure the expected risk that an adversary has in the process of reconstructing an input, given the joint probability distribution of inputs and their gradients. This framework can reveal the gradient leakage level by analyzing the Bayes optimal adversary, which minimizes this risk with a specific optimization problem involving the joint distribution. DP constitutes a strong standard for privacy guarantees for algorithms on aggregate databases [124, 127, 172]. It is defined in terms of the application-specific concept of adjacent databases and aims to hide whether one sample exists in the database. Thus, DP is defined as the detecting probability of outputs of any two adjacent databases.

Attack algorithms can evaluate the security and privacy levels directly. In order to evaluate the adversarial robustness of image classification tasks, large-scale experiments have been conducted and the performance of different defense methods can be evaluated [262]. In addition, we can apply adversaries to DP-SGD, which allows for evaluating the gap between the private information that an attacker leaks (a lower bound) and what the privacy analysis establishes as being the maximum leak (an upper bound) [130]. We can notice that attack methods constantly emerge to face advanced defense methods. Thus, the experiment-based methods need to consume a lot of computation resources, such as $3,000$ GPU hours with parallelized over $24$ GPUs as shown in [130].

Attack algorithms in the white-box scenario draw a lot of attention in the last few decades, but they seem to be impractical and can only be used as an upper bound. For example, model poisoning attacks in FL can be divided into three scenarios based on various levels of background knowledge, i.e., full knowledge, partial knowledge, and no knowledge. The attack performance decreases drastically as the background knowledge decreases [165, 263, 264]. In this context, practical attack algorithms with no knowledge should be studied to explore potential privacy and security risks. In addition, the organizer usually obtains more background knowledge than the rest of the participants. In order to mitigate the risk of the organizer being an adversary/eavesdropper, the decentralized framework can be adopted as a solution.

For the same attack purpose, different levels of distributed learning require different background knowledge, since the level of distributed learning determines interactive messages which usually contain the private information of participants, such as extracted features and neural network gradients of private data. Thus, various attack methods have emerged to infer private information or poison training process instead of unified attack schemes. For example, MIA in level 1 (sharing model) needs shadow datasets to train shadow models and then estimates the confidence of the training models [54]. We know that the shadow datasets and their distribution affect the attack performance obviously. However, how to obtain the shadow datasets becomes controversial, such as generative networks, stealing, and so on.

Although distributed learning can achieve privacy-enhanced and scalable data sharing, it also presents some security and privacy risks. Four-level distributed learning frameworks show various risk levels of privacy leakage, due to the different interactive messages [265, 266]. The interactive messages usually contain the private information of participant users, such as extracted features and neural network gradients of private data. This data process can protect private data to some degree. Thus, it is of interest to study the potential privacy protection levels owing to these data process functions, and then design effective protection schemes to achieve a better trade-off between training performance and privacy.

Privacy/confidential computing for distributed learning is a high requirement compared with conventional privacy protection. However, existing privacy computing techniques usually cannot provide systematic privacy preservation, which will degrade the learning performance or training efficiency [155, 267]. In addition, the protection effectiveness of different privacy computing techniques varies. For example, DP is seen as an effective method to prevent membership inference attacks by perturbing the impact on whether one instance exists in the training process. Thus, the sensitivity of interactive messages in distributed learning for DP should be carefully investigated when estimating the privacy budget. MPC is another widely used privacy computing technique. However, the transfer ability of MPC is limited and the MPC protocols for different paradigms of distributed learning need to be well-designed. Overall, it is crucial to combine these privacy computing techniques and design a general privacy-preserving framework for different paradigms of distributed learning [268, 269].

Three classic paradigms in FL, i.e., horizontal FL, vertical FL, and federated transfer learning, can be categorized as level 1, level 2 and level 3 of distributed learning, and have the capability to address most of the challenges of training ML models in distributed scenarios. FL is an efficient approach for federated data sharing among multiple clients, in which raw data are kept on the client side, which in turn protects data privacy for tensor mining. The primary purpose of FL is to train a satisfied ML model without exposing participants’ data privacy. Thus, when we select or design a training framework, both participants data characteristics and privacy requirements should be considered. In addition, an increasing number of advanced paradigms have emerged to handle various challenges in FL training, such as multi-modal FL [270, 271, 272], federated knowledge distillation [273, 274, 275], quantized FL [118] and so on, which help to construct a secure and efficient federated AI ecosystem.

Although FL can benefit data privacy, security and privacy risks induced by the interactive messages also exist. Particularly, FL can be combined with other privacy techniques, such as DP, MPC, HE, and so on, to improve the privacy of local updates, by integrating them into gradient descent training to enable privacy-enhancing FL. Moreover, the security of FL-based data sharing can be improved by combining it with blockchain technology [276, 277, 278, 279]. In this context, the information of trained parameters can be appended into immutable blocks on a blockchain during client-server communications. Further, the vast communication cost in vertical FL should be noticed [280, 281, 282]. Specifically, in vertical FL, the total computation and communication cost is proportional to the training dataset size. In other words, the widely adopted batch computation method in horizontal FL cannot be applied to vertical FL. When facing a massive amount of data, e.g., billions of advertising data, communication, and local computation may be in many orders of magnitude, and the system may lose vitality due to limited resources, such as hardware capacity, bandwidth, and power.

As mentioned above, DP has widely been adopted to train a distributed ML model, which will add random noise to gradients during the training process. However, a strict privacy guarantee usually requires a large noise variance injected, so the DP-based training will lead to significant performance degradation. Although existing works in [283, 172] have explored the training performance of the differentially private distributed learning systems and provided some theoretical results, these results can only bring out some intuitions and cannot enhance the learning performance directly. Therefore, an accurate estimation of convergence performance on the differentially private ML training is beneficial to find a proper balance between utility and privacy.

In addition to the accurate estimation of convergence performance, dynamic parameter optimization is also a promising direction to balance the trade-off between utility and privacy. Because of privacy protection, the training performance caused by the original parameters has been changed. Correspondingly, the conventional parameter optimization method for distributed ML also becomes inapplicable. For example, the work in [172] has developed the upper bound on the differential private FL and revealed that there exists an optimal number of communication rounds with a given privacy level. This discovery brings a new look at the communication round in FL and rethinks the choice of communication parameters. The dynamic parameter optimization for differentially private ML has also been considered, which implements a dynamic privacy budget allocator over the course of training to improve model accuracy [284]. Although existing dynamic optimization methods have already been proposed and proven to improve a number of distributed learning systems obviously, there is still a huge room for improvement.

The various requirements for different scenarios or different participants in distributed ML systems are also challenging, especially when the data distribution is non-independently identically distributed [285, 286]. Therefore, designing a specific/personal protection mechanism for the distributed ML system can bring out a better balance between utility and privacy. The work in [287] has considered a social network and achieved a proven DP requirement by perturbing each participant’s option with a designated probability in each round. Combining sketch and DP techniques, the work in [115] has proposed a novel sketch-based framework, which compresses the transmitted messages via sketches to simultaneously achieve communication efficiency and provable privacy benefits. These designs can obtain a satisfactory trade-off between utility and privacy, because of the deep combination of original scenarios and DP techniques. Therefore, how to balance utility and privacy in the amount of distributed learning scenarios has not been fully explored.

PSI is an important step in distributed learning because of the feature or individual differences among multiple users. For example, in horizontal FL/SGD systems, we need to ensure that each record has the same features. Classical PSI protocols are third party-based PSI [288, 289], public-key-based PSI [290, 291], circuit-based PSI [292] and OT-based PSI [293]. However, there is still a research gap that using PSI in distributed learning to investigate the tradeoff between the privacy level and the learning performance.

The key question in adding security to a decentralized diagram is to increase the confidence that all parties involved in the system (agents, platforms, and users) will behave correctly, and can be achieved by authentication. The identification of the parties can make up a system and possibly establish a trusting environment between clients. Cryptology is proven useful in a large number of authentication and access control scenarios, but it cannot address the problem of fully new participants. In addition, a trust/reputation model has been proposed to determine the participating values for unknown clients, since it is hard for an agent to obtain complete knowledge about other participants [217, 181, 218]. Consequently, how to design efficient identity certification mechanisms to uniquely authenticate known, and trusted users and agents in the system has drawn much attention.

Coordination and cooperative control of multi-client in distributed ML always attract lots of attention from various research communities, where a fundamental approach to achieving cooperative control is the consensus-based algorithm [294]. Traditional consensus designs are mostly based on single and finite-time domain [295, 296], where in reality, the dynamics of the system are usually complicated and non-linear. Therefore, a useful and effective consensus design with dynamic or unknown parameters is urgent in future research. For example, the time-varying resources and requirements for participating clients are key and un-trivial factors in design. In addition, the security of consensus also raises several issues recently [297]. How to protect the integrity of the consensus from inside or outside attackers and how to prevent private information leakage from the published consensus are other interesting research directions.

The reasons for implementing blockchain in a distributed learning system are to increase the interaction efficiency between participants by providing more trusted information exchange, reaching a consensus in trust conditions, assessing participant productivity or detecting performance problems, identifying intruders, allocating plans and tasks, and deploying distributed solutions and joint missions [298, 299]. However, the challenges consist of assessing feasibility and finding an architectural approach for combining blockchain-based consensus algorithms with real-time distributed learning systems, while assuring incentive information exchange and compatibility with the already existent local processing protocols [258]. In addition, the incentive mechanism is also vital for the consensus design [300, 301].

Fairness attracts increasing attention in recent years, especially in the scenario where multiple participants are evolved in one learning task [302]. A max-min fairness distributed learning system has been developed in [303], where multiple clients are matched with the bandits with the minimum regret. Furthermore, collaborative fairness in FL has been investigated in [304]. Although several works throw out the idea of fairness, there is a lack of a common definition of fairness in distributed learning. Whether attending the same rounds of training or allocating training trials according to the users’ capability represents fairness is still an unclear question. In addition, the relationship between fairness with security and privacy also requires further discussion.

One of the oldest and most popular techniques used in information security is cryptography, and its use to protect valuable information is usually relying on symmetric encryption and decryption algorithms such as elliptic curve cryptography (ECC), homomorphic hash function, and secret sharing technology. A secure lightweight ECC-Based protocol, i.e., Broadcast based Secure Mobile Agent Protocol (BROSMAP) [305], has been improved to fulfill the needs of Multi-agent based IoT Systems in general and obtained better performance than its predecessor with the same security requirements. HE assisted MPC framework [176], enabling a participant to compute functions on values while keeping the values hidden, can allow certain mathematical operations (such as aggregation) to be performed directly on ciphertexts, without prior decryption. However, cryptography algorithms usually require complicated computation protocols and may not be achieved efficiently.

Secure protocols are designed to enable computation over data distributed between different parties so that only the result of the computation is revealed to the participants, but no other private information. Secure protocols usually combine several efficient security and privacy techniques, e.g., MPC, DP, and HE, and need several interactions to exchange intermediate results. However, too many interactions may increase the information leakage risk, communication, and computing overhead. Besides, it is also challenging to explore generic secure protocols over remote parties, especially for complicated scenarios and various applications. To realize an efficient communication protocol in a trusted and secure environment, an alternative way is to increase the transmission rate using an intelligent reflecting surface (IRS) by smartly reconfiguring the wireless propagation environment, with the help of massive low-cost passive reflecting elements integrated on a planar surface and to enable cover communication [306].

High accuracy of large neural networks is often achieved by paying the cost of hungry memory consumption and complex computational capability, which greatly impedes the deployment and development in distributed systems [307]. To efficiently accelerate the learning process, privacy preservation-based methods, such as compact model [308, 309], tensor decomposition [310], data quantization [311] and network sparsification [312], are recent key advances.

Distributed ML can enhance the ability to identify and control robotics with remote and distributed control or wireless connections to clouds. This scenario requires high precision control, which raises increasing security issues and vulnerability to transmission errors [313, 314]. How to preserve the integrity of the control system and how to prevent information leakage during data transmission needs further investigation. In addition, ethical issues related to bionic robots are hotly debated concerns [315, 316].

ML and its distributed styles can improve the quality of generated images and videos, such as GAN and diffusion models. With the rapid development in VR and AR-based applications, private information from generated videos may lead to personal information leakage [317, 318]. Adversaries can take advantage of the fake videos to analyze the unique behaviors, personal interests, and background environments of participants [319].

Quantum ML operates based on quantum mechanics, taking advantage of superposition to store and process information [320, 321]. However, if information sources are from distributed clients, information leakage and inside or outside attacks may occur during data transmission. Thus, conducting the protection on distributed ML raises several challenging problems, such as identifying attackers, ensuring the integrity and availability of transmission data, and preserving privacy.

Metaverse seamlessly integrates the real world with the virtual one. It allows avatars to carry out rich activities, including creation, display, entertainment, social networking, and trading. Thus, it is promising to build an exciting digital world and transform a better physical scenario by exploring the Metaverse [322, 323]. Intuitively, the breakthroughs of AI in the real world motivate people to realize the Metaverse. For example, distributed ML via integrating distributed data from Metaverse users can provide technical support for Metaverse systems to reach or exceed the level of human learning. This can significantly affect the operational efficiency and the intelligence of the Metaverse. Intelligent voice services provide technical support, such as voice recognition and communication. However, several new security and privacy challenges that can compromise the systems or divulge users’ privacy raise attention in the interaction process, such as the communication between metaverse users and service providers.

The digital twin can fill the gap between physical systems and digital spaces. Leveraging FL to construct digital twin models of IoT devices based on their running data has been proposed in [324, 325]. The physical security of IoT devices is critical as they can be damaged, destroyed, or even stolen by attackers. Digital twin systems also have other priorities than the traditional network/system security requirements because of their interactions with the physical components. For instance, defects in a critical product may lead to death, injuries, or environmental damage. For this reason, safety could be ranked as the top security requirement. Safety can broadly be defined as the avoidance of harm or hazard to the physical environment and infrastructure that could occur from system faults [326]. Meanwhile, the possible privacy leakage from the interactions with the physical components must also be considered.

Web 3.0 has attracted considerable attention due to its unique decentralized characteristics [327]. In Web 3.0, data presents a distributed storage structure, so there will be no central node for data management, significantly reducing the service cost of managing data. Web 3.0 emphasizes the protection of users personal data, and therefore, as a key technology to solve the data privacy problem, privacy computing is becoming the immediate need of Web 3.0 existence. Privacy computing technology can analyze and calculate data under the premise of protecting data privacy and security, which provides a strong guarantee for the efficient and safe circulation of data across industries and organizations.

Generative design uses AI to come up with multiple design variations for products or parts. This leads to a faster generation of design options than would be developed through manual design, which leads to faster product development times and more creative choices to select from. For example, the meteoric rise of diffusion models has been one of the most significant developments in ML in the past several years [328]. Although generative design AI can improve the qualities of several tasks, it also relies on massive data and may induce several security and privacy issues, especially for fake digital assets, like photos or videos, that are indistinguishable from real things.

• •

  IEEE 1363 (Standard Specifications for Public-Key Cryptography): Encryption is vital for securing data in distributed learning. IEEE 1363 provides specifications for public-key cryptography algorithms, ensuring confidentiality and integrity of communication in distributed systems.

• •

  IEEE P2089 (Standard for Privacy Impact Assessment for Internet of Things): This standard provides a framework for assessing the privacy impact of IoT systems, which often play a crucial role in distributed learning scenarios. It guides the identification of potential privacy risks and suggests mitigation strategies.

• •

  IEEE 3652.1-2020 (Guide for Architectural Framework and Application of Federated Machine Learning)³³3https://standards.ieee.org/standard/3652$\_$1-2020.html: It provides a blueprint for data usage and model building across organizations and devices while meeting applicable privacy, security and regulatory requirements in FL. In detail, the description and definition; the categories and the application scenarios to which each category applies; the performance evaluation; and the associated regulatory requirements of FL are defined.

• •

  IEEE P7000 series (Model Process for Addressing Ethical Concerns During System Design): Distributed learning involves ethical considerations, and this series offers a comprehensive model process to address ethical concerns throughout system design and development. It emphasizes transparency, accountability, and user consent.

• •

  GDPR (General Data Protection Regulation)⁴⁴4https://ec.europa.eu/info/law/law-topic/data-protection$\_$en: Although not an IEEE standard, GDPR is a significant regulation that affects distributed learning. It emphasizes the protection of personal data and requires explicit user consent for data processing. Organizations handling data in distributed learning must adhere to GDPR’s principles to ensure user privacy.

• •

  HIPAA (Health Insurance Portability and Accountability Act)⁵⁵5https://www.hhs.gov/hipaa/index.html: In healthcare-related distributed learning applications, HIPAA plays a crucial role. It sets regulations for protecting the privacy and security of patient’s health information, including data used in distributed learning scenarios.

• •

  NIST (National Institute of Standards and Technology) Guidelines⁶⁶6https://csrc.nist.gov/: While not IEEE-specific, NIST provides guidelines on security and privacy, including those applicable to distributed systems. NIST’s cybersecurity framework and privacy framework offer valuable insights for building secure and privacy-preserving distributed learning systems.

• •

  IEEE Code of Ethics⁷⁷7https://www.ieee.org/about/ieee-code-of-ethics.html: While not a policy or regulation in the legal sense, the IEEE Code of Ethics guides professionals working in technical fields, including distributed learning. It encourages ethical behavior, respect for privacy, and responsible decision-making.