Refiner: Data Refining against Gradient Leakage Attacks in Federated Learning

Training DNNs while maintaining client privacy has become a crucial necessity due to data protection regulations¹¹1https://www.dlapiperdataprotection.com/, such as General Data Protection Regulation of the European Union. FL serves as a promising privacy-preserving solution through a gradient exchange mechanism between clients and a server [23]. We focus on a typical FL scenario, where a server collaborates with a set of $m$ clients to train a model $F_{\theta}(\cdot)$ parameterized by $\theta$, utilizing loss function $\mathcal{L(\cdot,\cdot)}$. The $i$-th client’s local dataset is denoted by $D_{i}$. The server and clients interact with each other for a total of $N$ rounds by alternately executing the two steps:

• •

  Model Distribution and Local Training. The server samples a subset of $K$ clients from the entire client pool to participate in the current round while the others await their turn. The server broadcasts the global model $F_{\theta}(\cdot)$ to selected clients. The $k$-th selected client ($k=1,\cdots,K$) samples raw data $x,y$ from his local dataset and computes gradients $g_{k}=\nabla_{\theta}\mathcal{L}(F_{\theta}(x),y)$.

• •

  Global Aggregation. The server performs global aggregation by averaging the gradients received from selected clients, which are subsequently used to update the global model: $\theta=\theta-\frac{\eta}{K}\sum_{i=1}^{K}g_{i}$ where $\eta$ is the learning rate.

In the literature [27, 34], selected clients may update the local model over multiple steps and then upload the final model parameters. In this paper, we employ a default local step of 1, where clients directly upload the gradients without performing multiple local updates. The rationale behind this decision stems from the fact that the server typically derives gradients by comparing the global model parameters with the uploaded local model parameters. Multiple local steps could potentially lead to a discrepancy between the derived gradients and the ground-truth gradients associated with clients’ data, thereby reducing the effectiveness of attacks [52]. Thus, we adopt the setting that considers the most powerful attack scenario to better examine the effectiveness of defenses.

We assume that the server is honest-but-curious [44], which adheres to the pre-defined training rules but with a strong intention of recovering clients’ raw data by exploiting uploaded gradients. We allow the server to access public data sources and sufficient computational resources, thereby enabling the server to launch stronger attacks to evaluate our defense more thoroughly. We suppose that clients are aware of the privacy risks, and, as a remedy, clients proactively adopt defenses. To simplify the notations, subscripts, and superscripts representing specific clients are omitted throughout the paper.

The seminal work of GLAs was proposed by Zhu et al. [54]. Given randomly initialized dummy data $x^{\prime}$ with dummy label $y^{\prime}$ and uploaded gradients $g$, the server tackles a gradient matching problem as follows:

Empirical evaluations illustrated the remarkable similarity between resulting $x^{\prime}$ and clients’ data $x$. The Follow-up work [43, 53] showed how to infer the ground-truth label $y$ from the gradients of the fully-connected layer, reducing the complexity associated with solving Equation 1. However, several studies [46, 12, 48] empirically demonstrated that bad initialization causes convergence towards local optima, making recovered data noisy. In response, Geiping et al. [12] and Jeon et al. [20] dedicated efforts to explore various regularization techniques and heuristic tricks, including total variation, restart, and improved initialization, aiming to eliminate invalid reconstructions. Recently, since the output of generative models is not noisy, Li et al. [25] and Yue et al. [49] trained a generative network to recover clients’ data. The potential assumption is the availability of an abundance of data that is similar to the clients’ data. They optimized latent vectors of the generative network to produce images whose gradients are similar to uploaded ones.

While cryptographic methods have emerged as potential solutions in defending GLAs, the substantial overhead limits their applicability [54, 44, 25]. Similar to [54, 44, 25], this paper does not discuss cryptographic methods, but instead mainly investigates alternative perturbation-based methods [54, 3, 47]. One such perturbation-based method [54, 3, 47] is differential privacy (DP), which involves adding random noises to ground-truth gradients to generate perturbed gradients. Moreover, some work [39, 44] explored gradient pruning or gradient quantization (GQ) to obfuscate the server. Soteria, proposed by Sun et al. [39], employs a theoretically-derived metric to evaluate the privacy information contained in each gradient element and selectively prunes gradients in fully-connected layers. However, Balunović et al. [3], Wang et al. [44] discovered that Soteria could be compromised by muting gradients in fully-connected layers during the reconstruction process (Equation 1). To address this vulnerability, they developed a gradient mixing mechanism to solve the problem.

We here formulate the problem that Refiner solves. The primary objective is to construct robust data $x^{*}$ that possess sufficient utility while minimizing the disclosure of privacy information from clients’ raw data $x$. Then, Refiner upload gradients associated with $x^{*}$. Refiner formulates the following optimization problem to craft robust data:

where $UM$ and $PM$ denote the utility and privacy metrics. $UM$ evaluates the degree to which $x^{*}$ diverges from $x$ in terms of utility, while $PM$ quantifies the reduction in privacy associated with $x^{*}$ compared to $x$. $\beta$ serves as a balance factor to regulate the trade-off between utility and privacy. Increasing $\beta$ places a higher priority on privacy protection. Intuitively, with well-defined metrics, the optimal solution of Equation 2 aligns with our definition of robust data. Section 3.2 and Section 3.3 will elaborate on our $UM$ and $PM$, respectively.

Upon defining the concrete forms of $UM$ and $PM$, the remaining challenge is figuring out how to solve Equation 2. A straightforward solution is to harness gradient descent algorithm [6], which stands as one of the most commonly employed optimization algorithms. While gradient descent algorithm is capable of tackling optimization problems featuring solely convex terms like UM, it encounters obstacles when confronted with highly non-linear and non-convex neural networks like PM. Specifically, we initialize $x^{*}$ as $x$, subsequently aiming to maximize PM($x$,$x^{*}$), i.e., making $x^{*}$ noisy. As shown in Figure 6(b), the resulting $x^{*}$ deviate only slightly from $x$ (Figure 6(a)), yet the model confidently identifies it as a noisy sample. This phenomenon and $x^{*}$ are known as adversarial vulnerability and adversarial examples [13, 7].

To address the above problem, we propose noise-blended initialization, which by blending $x^{*}$ with random noises, i.e., $x^{*}=(1-\alpha)x+\alpha v,\ v\sim q(x),\alpha\in[0,1]$. $\alpha$ is a blend factor. A big value of $\alpha$ enables $x^{*}$ to be initialized around the area flooding with noises. Subsequently, during the optimization process, the evaluation network will maintain $x^{*}$ as a noisy sample. Figure 6(c) exemplify the noise-blended initialization with $\alpha=0.5$. The resultant $x^{*}$ does not contain any semantic information regarding $x$, showing the effectiveness of the noise-blended initialization.

This subsection examines how Refiner contributes to the convergence of the global model. By establishing Assumption 1, 2, 3, and 4 and demonstrating Theorem 1 (proof available in the Appendix B), we shed light on the subsequent discussion. Theorem 1 provides an upper bound on the gap between the optimal solution and the model trained with Refiner after sufficient iterations. As the gradient distance between robust and client data decreases, this upper bound also decreases, implying better performance of the global model. This highlights the effectiveness behind minimizing the gradient distance between robust data and clients’ data in maintaining performance. The inclusion of the gradient distance term ($UM$) in Equation 2 guarantees the boundedness of $||\nabla_{\theta}\mathcal{L}(F_{\theta}(x),y)-\nabla_{\theta}\mathcal{L}(F_{\theta}(x^{*}),y)||$. In practice, we modify the gradients of robust data slightly. If the difference between the gradients of robust data and clients’ data exceeds $\epsilon$, we search for the closest gradients within the $\epsilon$-constraint and use them as the final uploaded gradients (see Appendix C for more details). Doing so enables us to evaluate the utility-privacy trade-off of Refiner by varying values of $\epsilon$⁴⁴4Adjusting the value of $\beta$ is also an alternative way, but we find that tuning $\epsilon$ is more convenient..

It is worth noting that FL commonly involves two scenarios: when the client data distributions are identical (IID) and non-identical (Non-IID). Our theoretical analysis does not require uniformity in the client data distributions. Thus, Theorem 1 is applicable to both scenarios. Moreover, these assumptions are consistent with [24] and aim to ensure a unique global extreme point, or that any local optimal point is also a global optimal point. For neural networks with multiple extreme points, Theorem 1 can be understood as guaranteeing convergence to at least one local extreme point.

There exist various types of GLAs, each with unique features. Analyzing the effectiveness of Refiner against each specific attack instance is cumbersome. Therefore, we begin by distilling GLAs into a generalized form by identifying patterns shared across attack instances. Then we analyze the effectiveness of Refiner against the generalized attack. In this way, our privacy analysis can be extended to encompass a wide spectrum of attack variations and potential evolving attacks.

GLAs exploit the mapping relationship between data and gradients to reconstruct high-fidelity data. Specifically, we represent the mapping relationship from data to gradients by $f(x)=g$. The objective of GLAs is to construct an inverse map $\hat{f}^{-1}(\cdot)$ from gradients to data that yields $\hat{f}^{-1}(g)\approx x$, thus enabling the recovery of clients’ data. Different GLAs can be derived by specifying the concrete form of $\hat{f}^{-1}(\cdot)$⁵⁵5In fact, $f^{-1}(\cdot)$ and $\hat{f}^{-1}(\cdot)$ may not be one-to-one. But, after fixing random factors like random seeds, learning rate, etc., they have a unique solution and can be considered one-to-one. The term ”specified” here includes the meaning of fixing these factors.. Example 1, 2, and 3 are provided to demonstrate this.

In practice, these attacks demonstrate superiority in recovering high-quality data. Therefore, their $\hat{f}^{-1}(\cdot)$ is quite similar to ground-truth one $f^{-1}(\cdot)$. Inspired by this, we make $||\hat{f}^{-1}(\cdot)-f^{-1}(\cdot)||\leq\sigma$, where $\sigma$ is a small positive number.

Refiner uploads the gradients associated with $x^{*}$, which is denoted by $g^{*}=f(x^{*})$. The server uses $g^{*}$ to recover data, i.e.$\hat{f}^{-1}(g^{*})$. We now consider the distance of $x$ and $\hat{f}^{-1}(g^{*})$:

Equation 6 establishes a lower bound of $||\hat{f}^{-1}(g^{*})-x||$, which is determined by $x^{*}$, $x$, and $\sigma$. In practice, $x^{*}$ obtained by Equation 2 are often noisy and significantly differ from $x$, i.e., $||x^{*}-x||>\sigma$ (see Appendix F for the empirical validation). When $\sigma$ is small, the lower bound can be simplified as follows:

Equation 7 suggests that a smaller $\sigma$ results in higher privacy lower bound, i.e., stronger privacy protection capabilities. On the other hand, considering larger values of $\sigma$ may be a little trivial. In detail, a larger $\sigma$ leads to a significant discrepancy between the recovered data and clients’ data. Consequently, clients can directly upload ground-truth gradients since the attacker is unable to reconstruct high-fidelity data. In summary, the above analysis demonstrates the effectiveness and resilience of Refiner against GLAs, particularly those characterized by smaller $\sigma$ values.

Before delving into the analysis of time complexity, it is essential to acknowledge that achieving excellence in all aspects of utility, privacy, and time is almost impossible. Because striking a better utility-privacy trade-off often necessitates additional time investment for searching. For instance, DP equally perturbs all gradient elements, without considering the inherent privacy information contained in each gradient element. In contrast, Soteria addresses this limitation by estimating the privacy information of each gradient element and selectively pruning those with the highest privacy information, resulting in an improved utility-privacy trade-off. However, this estimation process incurs additional time costs.

DP, GQ, and gradient pruning, none of which necessitate extra forward-backward propagations, are discussed together. DP generates noises for each gradient element, GQ quantizes each gradient element, and gradient pruning sorts and then removes part of gradient elements. By assuming the model of interest owns a total of $\mathcal{N}$ parameters, the time complexity of both DP and GQ scales linearly with the number of parameters, $\mathcal{O}(\mathcal{N})$. For gradient pruning, the dominant factor is the sorting time complexity, i.e., $\mathcal{O}(\mathcal{N}\log(\mathcal{N}))$.

Soteria focuses on pruning gradients in the final fully connected layer. This entails a meticulous evaluation of the privacy information associated with each individual gradient element residing in the layer. Given that the total count of parameters contained within the layer to be denoted as $\mathcal{N}_{last}$, and the time complexity of once forward-backward propagation of the global model is $\mathcal{O}(S_{1})$. Soteria necessitates executing forward-backward propagation a specific number of times, corresponding to the number of neurons in the layer. Consequently, the overall time complexity of Soteria can be expressed as $\mathcal{O}(\mathcal{N}_{last}\cdot S_{1})$. The number of elements in a typical fully connected layer is usually $1024$.

The time complexity of Refiner primarily centers around solving Equation 2. Each iteration of Equation 2 requires twice forward-backward propagations of the global model, along with one forward-backward propagation of the evaluation network. Hence, the overall time complexity is the sum of these operations. Assuming the time complexity of running single forward-backward propagation for the evaluation network is $\mathcal{O}(S_{2})$ and the total number of iterations for solving Equation 2 is $\iota$, the time complexity of Refiner is $\mathcal{O}(\iota\cdot(2\cdot S_{1}+S_{2}))$. Table 3 summarizes the time complexities associated with these defenses. Section 6.2 conducts an empirical comparison of the time costs for these defenses.

Four state-of-the-art attacks are considered to examine the performance of Refiner, including iGLA [53], GradInversion [48], InvertingGrad [12], GGA [25, 49]. These attacks cover the primary types of GLAs currently explored. The first three attacks address variants of Equation 1 by employing varying optimizers, loss functions, and so on. GGA, on the other hand, focuses on optimizing the latent vectors of a GAN to align the gradients. We implement these attacks at the start of training, as this stage is most vulnerable to GLAs [3]. For GGA, we use public codes ⁶⁶6https://raw.githubusercontent.com/pytorch/examples/master/dcgan/main.py to train a GAN. Table 4 provides an overview of the attack distinctions and the hyperparameters employed.

We compare Refiner to the following defenses: DP [1], GQ [34, 49], gradient pruning [2, 54], and Soteria [44]⁷⁷7We use Soteria proposed in [44] to compare, which is an improved version compared with the original Soteria [39].. These defenses involve utility-privacy trade-offs that are regulated by hyperparameters such as noise magnitude for DP, discretization level for GQ, and pruning rate for pruning and Soteria. Stronger privacy protection can be achieved by increasing these hyperparameters, but this comes at the cost of performance degradation. We vary these hyperparameters to obtain the utility-privacy trade-off curves for each defense. Specifically, for DP, we employ two commonly used kinds of noises, namely Gaussian (DP-Gaussian) and Laplace (DP-Laplace), with magnitude ranging from $10^{-6}$ to $10^{2}$ and $C$ of 1. As for GQ, we consider discretization levels, including 1, 2, 4, 8, 12, 16, 20, 24, and 28 bits. For pruning and Soteria, we set pruning ratio in $\{0.1,0.2,\cdots,0.9\}$. Besides, we alter $\epsilon$ of Refiner over $\{0.01,0.03,0.05,0.07,0.1,0.3,0.5,0.7,0.9\}$ (see Section 4.1). Unless specified otherwise, for Refiner, the default values of $\alpha$, $\beta$, refinement iterations $\iota$ and $\tau$ are set to 0.5, 1, 10, and 0.95.

We assess defenses from three fundamental dimensions: performance maintenance, privacy protection, and time cost.

Performance maintenance. We define the performance maintenance metric (PMM), which calculates the accuracy ratio achieved with defenses compared to the original accuracy without any defenses:

where $original\_acc$ and $defense\_acc$ denote the accuracy of the model with and without defense⁸⁸8The original accuracies of LeNet are 84.12%, 54.01%, and 21.45% on SVHN, CIFAR10, and CIFAR100. The original accuracies of ResNet10 and ResNet18 are 83.25% and 84.95% on CIFAR10..

Privacy protection. Privacy protection measures the amount of privacy information recovered from the reconstructed images. Given the lack of a universally perfect privacy assessment metric, we employ a diverse range of metrics including PSNR [16], SSIM [45], LPIPS [51], and the evaluation network to achieve a more comprehensive assessment:

• •

  PSNR is the logarithmic average distance between the original and reconstructed images. A higher PSNR indicates high-quality reconstructed images.

• •

  SSIM captures human perception of image quality better by considering factors like brightness, contrast, and structure. SSIM ranges from 0 to 1, with values closer to 1 indicating greater similarity between two images.

• •

  LPIPS assesses the similarity of two images based on the comparison of features extracted by DNNs. A value close to 0 indicates a higher similarity between two images. LPIPS often better captures semantic and perceptual differences compared to PSNR and SSIM.

• •

  The evaluation network estimates the proportion of noise in the data, producing noise ratio between 0 and 1. As mentioned earlier, when the noise ratio exceeds 0.6, the human eye probably recognizes it as noisy data.

More details can be found in Appendix E.

Time cost. Time cost is also an important dimension for evaluating a defense. We record the time required for a single iteration of the model when employing defenses.

We select two widely-used model architectures, LeNet [22] and ResNet [14], on three benchmark datasets SVHN [30], CIFAR10 [21], and CIFAR100 [21]. We include ResNet10 and ResNet18 to study the impact of different scales. Our focus is on a typical FL scenario involving 10 clients collaborating to train a global model, with the assistance of a server.

Our evaluation also examines the effect of data distribution on FL by considering both IID and non-IID settings [19]. In the IID setting, the training datasets are randomly and equally divided among the 10 clients, ensuring identical distribution for each client’s local dataset. In the Non-IID setting, we allocate the training datasets unevenly among the 10 clients in a label imbalance fashion [26, 19].

The training process takes place over 8000 iterations, during which each client computes gradients locally using a batch size of 128. These gradients are then uploaded to the server, which updates the global model using FedAvg with a learning rate of 0.01. We scale the data to the range of 0 to 1 without employing additional data augmentation techniques.

Numerical results. Figure 7 illustrates the trade-off curves between PMM (utility) and PSNR (privacy) for various defenses against four state-of-the-art attacks on CIFAR10. Overall, Refiner consistently outperforms baselines by achieving lower PSNR while maintaining comparable PMM. For instance, when employing InvertingGrad attack, Refiner obtains an improvement of approximately 3 in PSNR compared to state-of-the-art defense Soteria, while Refiner maintains slightly higher utility than Soteria.

Moreover, we observe that the impact of Refiner, Soteria, and Pruning on model performance is relatively mild compared to other defenses, except Refiner. Particularly, excessively aggressive defense strengths, such as employing a noise magnitude of DP greater than or equal to $10^{-2}$ or employing a quantization precision less than or equal to 8 bits with GQ, can significantly deteriorate the model’s performance.

Semantic results. We see that the reconstruction quality of GGA, as measured by PSNR, is significantly inferior to the other three attacks. One possible explanation is that GGA focuses more on semantic rather than pixel-level reconstruction. To substantiate this conjecture, Figure 8 showcases the reconstruction images of GGA. Based on Figure 8, GGA indeed recovers the privacy information embedded in the original image (i.e., airplane), highlighting its attack effectiveness. Furthermore, compared to the PSNR metric, LPIPS places greater emphasis on the similarity of features extracted using DNNs, thereby underscoring the importance of semantic-level similarity over pixel-level similarity. Therefore, we deploy LPIPS metric and the evaluation network for further assessment of GGA’s effectiveness. From Figure 9(a), we observe that the attack performance of GGA is competitive with InvertingGrad against Soteria and Refiner. Remarkably, in Figure 9(b), the evaluation network consistently rates GGA’s attack effectiveness highly regardless of defense strength. This is intuitive since the generated images by GGA are noise-free, as displayed in Figure 8. In short, the above analysis stresses the necessity of employing diverse privacy metrics to assess defense performance.

Evaluating defenses with varied assessment metrics. Figure 10 shows the performance of different defenses under various privacy metrics. As can be seen, Refiner consistently achieves a superior trade-off between privacy and utility when evaluated using different privacy metrics. For instance, Refiner enables the inclusion of approximately 30% noise information in constructed data with negligible impact on the model’s performance, whereas Soteria achieves only around 20% noise inclusion capability.

Safeguarding different model architectures. We here examine the performance of different defenses when applied to ResNet. The corresponding results are presented in Figure 11. Remarkably, Refiner exhibits a superior trade-off between utility and privacy, especially when applied to larger models (ResNet18) compared to baselines. We hypothesize that this can be attributed to the increased non-linearity capacity of larger models, rendering the assumptions of other defenses less valid and consequently leading to degraded effectiveness.

Safeguarding different datasets. We also evaluate the performance of Refiner on different datasets, SVHN and CIFAR100. The results not only confirm the consistent superior performance of Refiner, but also reveal two noteworthy observations. Firstly, we observe a better reconstruction quality for attacks carried out on SVHN, with the highest achieved PSNR approaching 40. This is probably because SVHN is a less sophisticated dataset, rendering the construction easier. Secondly, we observe the remarkable effectiveness of Refiner compared to baselines when applied to the CIFAR100 dataset, suggesting that Refiner exhibits greater resilience in handling more complex datasets.

Defense with Non-IID setting. In the Non-IID setting, the datasets of clients are heterogeneous, posing greater challenges to maintaining performance. We evaluate the effectiveness of our approach in non-IID scenarios, as shown in Figure 13. Wherein, Refiner outperforms baselines. Moreover, while we observe little change in the quality of attacker-reconstructed images (measured by PSNR) under non-IID conditions, the difficulty of maintaining performance increases significantly compared to IID conditions.

The additional time cost they imposed by defenses is a crucial dimension of assessing their effectiveness. In this regard, Table 5 reports the runtime required for a single iteration of models when employing different defenses. For the small-scale model LeNet, Refiner and Soteria exhibit a time cost within the same order of magnitude. However, this time cost is one order of magnitude higher compared to Pruning, Gaussian, Laplace, and GQ. This is due to the fact that Refiner and Soteria require multiple forward-backward propagations to search for better utility-privacy trade-offs.

For larger models ResNet10 and ResNet18, The runtime required by Pruning and GQ increases in an approximate linear fashion. Refiner, Gaussian, and Laplace own comparable runtime overhead. Notably, Soteria incurs a significantly higher time cost of the 40s per iteration on ResNet34, which is 8.4x higher than the time required for Refiner. The heavy overhead of Soteria stems from the repeated execution of forward-backward propagation, which needs to be performed as many times as there are neurons in the fully connected layer. Consequently, as the number of fully connected neurons increases in larger models, the time cost of Soteria escalates significantly. In contrast, Refiner offers a scalable time cost for large models since the number of forward-backward propagations depends mainly on the user-specified number of iterations, granting flexibility based on factors such as clients’ privacy preferences and computing resources. Empirical results indicate that good performance can be achieved with only 10 iterations, far lower than the runtime cost associated with Soteria. In summary, Refiner achieves a better utility-privacy trade-off compared to DP, GQ, and gradient pruning, albeit at the expense of higher time cost. But, Refiner’s time cost is still lower than that of Soteria.

In this section, we investigate the impact of four sub-components of Refiner on its performance. We fix $\epsilon=0.1$.

Impact of balance factor $\beta$. Table 6 reports the performance of Refiner as $\beta$ varies against InvertingGrad. Overall, the higher $\beta$ is, the lower PMM is, and the higher MSE is. Moreover, the performance of Refiner is more robust to tuning $\beta$ compared with $\alpha$, i.e., there is not a huge performance shift w.r.t. different $\beta$. Thus, we believe setting $\alpha=1$ by default is a good option in practice.

Impact of refinement iterations $\iota$. Refinement iterations can influence the quality of resultant robust data and Table 7 reports the performance of Refiner over different refinement iterations against InvertingGrad. An observation is that, the performance of Refiner intensively increases by raising refinement iterations from 1 to 100, but further enhancing refinement iterations from 100 to 200 only produces a slight improvement, suggesting that refinement iterations of 100 are sufficient for optimization convergence in practice.

Impact of $\alpha$. Table 8 reports the performance of Refiner when configured with varying refinement iteration settings against InvertingGrad. It is observed that increasing the refinement iterations from 5 to 10 leads to a significant improvement in the performance of Refiner, enabling the acquisition of more PMM with similar PSNR (similar privacy protection ability). Intuitively, this is because during the early iterations, Refiner can maintain robust data to be noisy while effectively aligning gradients associated with clients’ and robust data. Further increasing the refinement iterations to 15 or 20 only yields marginal enhancements, suggesting that a refinement iteration value of approximately 10 would be a preferable choice for practical applications if considering the time cost.

Impact of $\tau$. When considering the promotion of the alignment between client data and robust data gradients, $\tau$ plays a significant role. It directs more attention towards aligning the gradients of early layers, serving a dual purpose. Firstly, it aids in maintaining the performance of the model since the early layers contribute more significantly to its performance. Secondly, according to [39], the issue of privacy leakage is more closely associated with the gradients of later layers. The experimental results are summarized in Table 9. By including $\tau$ in Refiner, i.e., a transition from 1 to 0.97, we see only a marginal 1% decrease in PMM, yet achieve a substantial 2-point reduction in PSNR. Further increasing the value of $\tau$ does not yield significant changes in PSNR and PMM, suggesting that the observed improvements reach a plateau.