Watermarking in Secure Federated Learning: A Verification Framework Based on Client-Side Backdooring

Backdoor attack is a special technique which trains an ML model to make predesigned incorrect predictions when encountering some specific inputs [16, 26, 27, 28]. The set of these specific inputs is called the trigger set.

A successfully backdoored ML model will not only output incorrect answers for the trigger set but also still perform well for benign inputs. Compared with the ground truth function $f$, the backdoor-attacked function $f^{*}$ satisfies

where $negl(\kappa)$ is a negligible function, and $\kappa$ is a security parameter.

Strong backdoors are related to the robustness of backdoor-based watermarks, and will be further discussed in Section 5.2.

Black-box watermarking uses backdoor attacks to force an ML model to remember specific patterns or features [17, 29, 30]. The backdoor attack causes the ML model to misclassify when encountering samples in the trigger set. The model owner keeps the trigger set secret and can thus verify the ownership of the model by triggering a misclassification.

A general black-box watermarking scheme for ML models can be split into three stages: trigger set construction, embedding, and verification.

Homomorphic encryption (HE) [31, 24] is a cryptographic scheme that enables some specific mathematic operations, e.g., addition or multiplication, via encrypted data without decryption. HE thus allows processing of data without sharing the plaintext and HE is consequently employed to provide secure transmission and aggregation in secure FL [32, 33, 34].

As shown in Fig. 1, three different parties are involved in watermarking in secure FL: the initiator, the central server, and ordinary clients. All three parties need to perform Secure FL and jointly train a DL model.

The clients are the data owner in FL. Each client holds a bunch of personal data. When training an FL model, clients obtain the global gradients from the central server during each iteration. They then update their local model based on the global gradients and train it for several epochs. Finally, they send their local gradients to the central server for aggregation. To avoid privacy leakage, the gradients are encrypted by HE.

The initiator is a special client who is chosen to embed the watermark. Besides training a local model, the initiator is responsible for embedding the black-box watermark into the global model. In case the watermarked model is stolen by an adversary, the watermark can provide copyright verification for the initiator.

The central server is responsible for aggregating the local gradients. In each iteration, the central server collects the encrypted local gradients and aggregates them using an aggregation method. After aggregation, the central server sends the ciphertext of gradients to each client.

In the threat model, we follow the prescriptive semi-honest assumption [25], which means any clients, including adversaries, should follow the pre-designed secure FL procedure, but might copy and steal the global model. When an adversary steals the watermarked model, they try to remove the watermark or forge their own trigger set. A benign user might also do some processing to the model before deployment. The model may thus be exposed to a variety of attacks.

While designing a black-box watermarking scheme for FL, it is important to satisfy the following properties. In general, an outstanding watermarking scheme should provide effectiveness, function preservation, low false-positive rate, robustness, and non-ambiguity.

• •

  Effectiveness: Effectiveness signifies that if the tested model is actually the model embedded with the watermark, the verification algorithm will always output ‘$1$’ when the input is an element from the trigger set $D_{s}$. This can be formally defined as

  $$\mathrm{Pr}[Verify(\hat{M},D_{s})=1\mid\hat{M}=Embedding(M,D_{b},D_{s})]=1.$$

  (7)

• •

  Function Preservation: Function Preservation refers to that the watermarked model performs approximately as well as the primitive model. It indicates that the watermarking scheme has negligible impact on the functionality of the model. This can be reflected by the model’s accuracy on the validation set $D_{v}$ and can be defined as

  $$Acc(M,D_{v})-Acc(\hat{M},D_{v})\leq negl(\kappa),$$

  (8)

  where $Acc:\mathbb{R}^{d}\times\mathbb{D}^{*}\rightarrow\mathbb{R}$ denotes the accuracy on validation set.

• •

  Low False Positive Rate: It is crucial that the watermark should not be extracted from an unwatermarked model, i.e., the watermarking scheme should have a low false positive rate. Generally, for a $k$-class model, the false positive rate should be near or below $1/k$, which is the probability of guessing. Formally defined, for a given secret trigger set $D_{s}$ and a model $M$ without watermark,

  $$Acc(M,D_{s})\leq 1/k\enspace\vee\enspace Acc(M,D_{s})\approx 1/k.$$

  (9)

• •

  Robustness: Robustness means that when an attacker applies any attack algorithm $Att:\mathbb{R}^{d}\times S\rightarrow\mathbb{R}^{d}$ to the watermarked model, with $S$ the set of parameters used in the attack, the model should maintain the watermark. A watermark is robust if one of the following two cases is true after attacks:

  Case 1.

  Case 1 signifies that the attack does not significantly influence the functionality of the model in both the primitive task and the watermark. The model owner can thus still successfully verify its ownership. This can be formally defined as Eq. (10).

  		$\displaystyle Acc(\hat{M},D_{b})-Acc(Att(\hat{M}),D_{b})\leq negl(\kappa)\enspace$		(10)
  	$\displaystyle\wedge$	$\displaystyle Verify(Att(\hat{M}),D_{s})=1,$

  where $D_{b}$ is the benign dataset, $D_{s}$ is the trigger set, and $\hat{M}$ is the watermarked model.

  Case 2.

  Case 2 signifies that the functionality of the attacked model drops significantly compared to the primitive model and is no longer useful for its task, rendering the attack unsuccessful. This can be formally defined as Eq. (11).

  $$Acc(\hat{M},D_{b})-Acc(Att(\hat{M}),D_{b})>negl(\kappa)$$

  (11)

• •

  Non-ambiguity: A watermarked model is non-ambiguous if for any ambiguity attack algorithm $Forge:\mathbb{R}^{d}\rightarrow\mathbb{D}^{|D_{s}|}$ the probability of successful verification is negligible, i.e.

  $$\mathrm{P}\mathbf{r}[Verify(\hat{M},D_{s}^{\prime})|D_{s}^{\prime}=Forge(\hat{M})]\leq negl(\kappa),$$

  (12)

  where $D_{s}^{{}^{\prime}}$ is the fake trigger set created by the attacker. Thus, the watermark is considered unforgeable.

In our proposed scheme, a key generation algorithm is utilized to help construct the trigger set. A key generation algorithm $KeyGen:\varnothing\rightarrow\{0,1\}^{*}$ randomly generates a bit string $sk$ as the secret key, although $sk$ can also be carefully chosen by the model owner. Non-repudiation and unforgeability of $sk$ directly affect the watermark’s robustness against ambiguity attacks. A watermark without $sk$ can be easily forged and thus the watermarking scheme is unfeasible.

Our insight of key generation and trigger set construction algorithms is based on the following observation. The primary dilemma of key generation and trigger set construction is that, on the one hand, intuitively, making the model remember unlabeled data such as random noise does less harm to the model’s functionality than misclassifying meaningful samples. On the other hand, since constructing a trigger set with only one class leads to a security problem as it will be easy to forge, a trigger set with multiple classes is required, while simply forcing a model to classify samples with similar noise to different classes is difficult and may result in overfitting. We therefore first divide an image into several patches and add to the patches to construct a multi-classes trigger set so that the model will remember the location of noise instead of a specific noise pattern.

In the key generation phase, the initiator generates, using the algorithm defined in Algorithm 1, a secret key used to construct the trigger set. In our work, we design the secret key $sk=(lk,ck)$ to comprise two parts. The first part denotes the positions of noise, while the other part denotes the corresponding labels. Assume that we try to watermark a $k$-class FL model. The owner first needs to choose two patch parameters, $\mu$ and $\nu$, that satisfy $\mu\nu\geq k$. The owner then generates a random permutation of $k$ numbers from $\{x\mid x\in\mathbb{N}\wedge x<\mu\nu\}$, where $\mathbb{N}$ is the set of all non-negative integers. The permutation, called location key $lk$, is the first part of $sk$. Since the second part are the labels, the owner generates another permutation of $k$ numbers from $\{x\mid x\in\mathbb{N}\wedge x<k\}$ to yield the classification key $ck$.

After key generation, the initiator uses $sk=(lk,ck)$ to construct the trigger set following the algorithm defined in Algorithm 2. The input images with size $\varphi\times\xi$ are divided into $\mu\times\nu$ patches. In this way, each patch corresponds to an integer between 0 and $\mu\nu-1$. Then the owner randomly samples $t$ patterns of $\lfloor\varphi/\mu\rfloor\times\lfloor\xi/\nu\rfloor$ pixels. For this, we use Gaussian noise to generate patterns, although the patterns can be any images that depend on the generation algorithm. Each pattern needs to be filled into the specific patch represented by $lk$ and the corresponding label $ck$.

For example, the first element of $lk$ is $lk_{0}$ (we use subscript $n$ to denote the $n$-th element of permutation). We thus find the $lk_{0}$-th patch and fill it with $t$ sampled patterns. Pixel values of other patches which have not been filled should be set to zero. Therefore, we get $t$ different images with only one patch that is non-zero. These $t$ images are labeled $ck_{0}$. The second element corresponds to the $lk_{1}$-th patch. Based on the images with $lk_{0}$-th patch filled, we fill another patch, identified by $lk_{1}$, with the same pattern. After that, we get another $t$ images filled with two patches which we label $ck_{1}$. The rest can be done in the same manner. In $l-$th step, we should fill the patches corresponding to $\{lk_{0},lk_{1},\cdots,lk_{l-1}\}$ with the $t$ patterns and label them with class $ck_{l-1}$. Eventually, we will get $kt$ images with all $k$ classes, and the trigger set is composed of all these images. An example of our key generation and trigger set construction algorithms is illustrated in Fig. 3.

Since $\mu$ and $\nu$ are used to divide the input image space into patches so that the secret key $sk$ can be embedded into the trigger set images, there are some restrictions when choosing the parameters. First, since we need to replace $k$ patches with the generated patterns, there should be at least $k$ patches. Second, each pattern needs to have at least one pixel for embedding. Therefore, the number of patches should not exceed the number of pixels in the input image. For $\varphi\times\xi$ input images, this leads to

Since $\mu\nu$ is the upper bound for the location key, it directly determines the size of the location key space; the larger $\mu\nu$, the bigger the location key space and the more difficult for an attacker to forge the secret key. On the other hand, a larger value of $\mu\nu$ yields fewer pixels for one patch, making it more challenging for the FL model to learn the trigger set images and the model potentially overfitting the trigger set. Thus, the robustness of the watermark is not guaranteed. Consequently, there is a trade-off between robustness and security that needs to be considered when choosing the patch parameters.

As illustrated in Fig. 4, we propose a gradient-enhanced algorithm for the initiator to embed the watermark into the FL model. The initiator only needs to perform a subtle change to carry out the embedding when transmitting the gradients. After key generation and trigger set construction, the initiator adds the trigger set into its normal training data samples to calculate the local gradients and embed the backdoor watermark into the jointly-trained model. Since the initiator is not guaranteed to be selected at each iteration and the initiator is only a single node in the FL model, we define a scaling factor $\lambda$ to improve the embedding process. When the initiator is selected to participate in the model update, $\lambda$ is used to enhance the gradient so that the FL model will remember the trigger set. Assuming that the initiator is the $1$-st client in FL, i.e., the index of the initiator is $0$, the clients get the encrypted messages as

where $L_{j}^{i}$ represents the local gradients of the $i$-th client in the $j$-th iteration, $C_{j}^{i}$ is the corresponding ciphertext, and $Enc$ represents any homomorphic encryption algorithm.

The watermark-embedding secure transmission algorithm is defined in Algorithm 3.

The scaling factor $\lambda$ plays a significant role in the embedding and directly influences the watermarking scheme’s effectiveness and function preservation quality. A small $\lambda$ results in a small effect of the initiator which might leads to a failed embedding, while a large $\lambda$ might significantly influence the global model’s functionality. We design a suitable scaling factor setting as

where $n$ is the number of clients selected in each iteration, and $N$ is the total number of clients. For a small local learning rate which is common in DL, the model parameters change consistently in the training procedure. The gradients of each client during several iterations are approximately equal. Therefore, this is approximately equivalent to the case of the initiator node being selected in every iteration, corresponding to a well-applied technique in existing strong backdoor attacks [16, 17]. Our method thus has a consistent updating process to improve the stability of model convergence while performing well for watermarking.

When Alice needs to verify an unauthorised model deployment by Eva, Alice and Eva can recruit a fully-trustworthy third party Trent as the arbitrator who employs the watermark verification protocol define in Algorithm 4.

Alice first sends a subset of the secret trigger set $D_{s}$ to Trent. Trent then gets the API of Eva’s model so that he can input samples and receive classification results. Trent uses the API to process Alice’s samples and checks whether the results are in accord with the labels provided by Alice. If the resulting accuracy is above a threshold $\gamma$, the Alice’s ownership of the model is confirmed and Trent can sentence Eva for infringement.

The threshold $\gamma$ controls the probability of a false positive verification. If a $k$-class model does not have a backdoor embedded watermark, it will classify samples from the backdoor dataset to random labels with a probability of correct classification of $1/k$. We expect the probability of successful copyright verification to be negligible on the model without backdoor. Assuming that Alice provides $n$ backdoor samples, threshold $\gamma$ satisfies

Our protocol has some distinct advantages. First, Trent does not need to access the weights of Eva’s model, avoiding the possibility of Eva cheating Trent by providing fake model parameters. Since the model is deployed to all the clients via API access, it is impossible for Eva to avoid scrutiny. Second, Alice needs to provide only some samples of her trigger set, allowing to preserve sufficient data for future verification.

In this section, we introduce the details of watermarking procedure in Secure FL. Secure FL enhances the protection of privacy by protecting both data and gradients, while watermarking provides copyright protection of FL models. In general, the training procedure of FL proceeds in four phases: initialisation, local training, secure transmission, and aggregation.

Non-ambiguity defined in Section 3.3 means the watermark can hardly be forged by the attacker. For our proposed scheme, we have the following theorem.

The robustness of a backdoor-based watermark is related to the strong backdoor defined in Section 2.1. That the watermark is robust is equivalent to the backdoor being a strong backdoor. Thus, to guarantee robustness, we need to prove the trigger set embedded using our proposed scheme is the strong backdoor.

Additionally, the theorem can also be confirmed by our experiments in Section 6.

In our implementation, we use TensorFlow (version 2.3.4) and TensorFlow-Federated (version 0.17.0) as DL and FL framework, respectively. We encrypt the gradients using Tenseal (version 0.3.4) which implements CKKS. The learning rate of each client is set to $0.01$ while the learning rate of the central server is $1.0$. In the local training phase, each selected client trains the model for two local epochs before transmitting the gradients to central server. The number of clients is set to $100$ to simulate a real-world FL environment.

In our experiments, we adopt two different convolutional neural network (CNN) architectures to evaluate the performance of the proposed method. The first CNN is, as also in [1], the classical LeNet [39] which consists of two convolutional layers, each followed by a $2\times 2$ max-pooling layer, and two fully connected layers. The second is VGG [40] with 13 convolutional layers and 3 fully connected layers.

In our experiments, we use two datasets, MNIST [41] and CIFAR10 [38]. MNIST is a grayscale image dataset with 60,000 training data of the 10 digits and an additional 10,000 for testing, while CIFAR10 comprises 50,000 RGB images for training and 10,000 for testing. To keep consistency, the $28\times 28$-pixel images in MNIST are resized to $32\times 32$ pixels, the same size as CIFAR10 images.

For data distribution, we split the data using three different distributions. One is the independent identically distribution (IID) which means that the whole training dataset is uniformly allocated to each client. The other two are non-IID distributions. For one, denoted as dn-IID, we use a Dirichlet distribution to split the data accordingly, as used in [42]. For the other, we employ the method from [1] to construct a pathological non-IID distribution (pn-IID), which first sorts the dataset according to the labels and then splits them into $tN$ parts where $N$ is the number of clients, with each client randomly choosing $t$ parts as its training set. In this case, each client only has data with $t$ different labels at most. We implement the settings in [1] and set $t=2$ in our experiments.

For trigger set construction, we set the default patch parameters to $\mu=\nu=4$ for both tasks, making them large enough for security, while in the experiments, for comparing different patch parameters, $\mu=\nu=4,6,16$ are used. Before embedding the watermark, we generate 100 images as the trigger set, with each class comprising 10 examples, while we use 1,000 images constructed with the same secret key for verification to test the generalisation ability of our watermarking scheme.

Effectiveness indicates whether the watermark is successfully embedded into the model, i.e., by the accuracy on the trigger set $D_{s}$. The effectiveness results are shown in Table II.

Table II demonstrates the success of embedding the watermark into the FL models. All accuracies on the secret trigger set exceed $98\%$. This also indicates that our trigger set construction algorithm has appropriate generalisation ability for both datasets and both architectures.

We compare the watermarked FL models with normal models without watermarks in terms of the primitive functionality. The results of this are shown in Fig. 5, from which we can see that the watermarked models do quite well, with a drop in accuracy of below $1\%$ in most cases. This demonstrates that our watermarking scheme has a negligible impact on the functionality of FL models.

We take the normal FL models to measure whether the watermark can be detected. In the black-box watermark scheme, this means calculating the accuracy of the secret trigger set with normally-trained models. As demonstrated in Fig. 6, our watermarking scheme satisfies the requirement defined in Section 3.3, with a maximum false positive rate of $11\%$, which means the rates are near or below the expected $10\%$ for a 10-class problem.

We conduct four different watermark removal attacks to evaluate the robustness of our watermark, including fine-tuning, pruning, quantisation, and PST attacks as defined in Section 3.2.

As discussed in Section 4.2, the patch parameters can affect the security and effectiveness of our watermarking scheme. We conduct some experiments where we investigate three different settings, $\mu=\nu=4$, $\mu=\nu=6$, and $\mu=\nu=16$, giving corresponding patch sizes of $8\times 8$, $5\times 5$, and $2\times 2$. The $2\times 2$ is the minimum patch size to generate sufficient trigger set images. For simplicity, we use $(x,x)$ to represents models with patch parameters $\mu=\nu=x$. Since data distribution is not considered in this section’s experiments, the models are trained following the IID setting.