Boosting Adversarial Transferability via
Commonality-Oriented Gradient Optimization

ViTs [dosovitskiy2020image], inspired by the self-attention mechanism in NLP, have transformed deep learning and demonstrated strong performance across a wide range of domains [liu2025aligning, wan2025srpo, wan2025d2o, zhang2025enhancing]. Since their introduction, numerous variants have emerged, including Visformer [chen2021visformer], DeiT [touvron2021training], PiT [heo2021rethinking], CaiT [touvron2021going], TNT [han2021transformer] and MG-ViT [zhang2024mg].

In this work, we categorize ViT variants based on their dominant design priorities. One group focuses on computational efficiency by simplifying attention operations or integrating convolutions to reduce inference cost, as seen in Visformer [chen2021visformer] and PiT [heo2021rethinking]. The other group emphasizes data efficiency by improving representation capacity and generalization, often through mechanisms such as knowledge distillation or deeper attention stacks, as in DeiT [touvron2021training] and CaiT [touvron2021going]. It is important to note that this categorization is not mutually exclusive—many models exhibit traits from both categories. However, we analyze and group them based on their most salient characteristics to guide the design of our method. Although these models are typically trained on the same datasets and tasks and tend to form shared decision patterns, their architectural differences lead to diverse gradient behaviors. This heterogeneity can reduce adversarial transferability, motivating the need to craft perturbations that emphasize shared patterns while suppressing surrogate-specific biases.

In transfer-based black-box attacks, adversarial examples generated on a surrogate model are designed to exploit cross-model vulnerabilities and transfer to attack unknown target models [liu2016delving]. Numerous methods have been developed to improve adversarial transferability.

Methods targeting CNNs as surrogate models have a long history. Early methods like BIM [kurakin2018adversarial], PGD [madry2017towards], and MIM [dong2018boosting] focused on stabilizing gradient directions, while DIM [xie2019improving] and TIM [dong2019evading] improved transferability via input transformations. Recently, SSA [long2022frequency] explored frequency-domain vulnerabilities comprehensively, but did not design frequency-aware perturbations specifically to enhance transferability. Additionally, HFA [zhuenhancing], inspired by SSA, only adapts high-frequency components, neglecting mid-to-low frequencies. These limitations lead to suboptimal results on ViTs, motivating our method to leverage a broader frequency spectrum tailored for ViTs to improve transferability.

Recent methods such as TGR [zhang2023transferable] and GNS-HFA [zhuenhancing] have advanced adversarial transferability on ViTs by mitigating gradient overfitting. TGR reduces the variance of gradients within the surrogate model by suppressing extreme gradient values, hypothesizing that these outliers drive overfitting. This effectively decreases internal gradient variance. Conversely, GNS-HFA posits that moderate gradients are more crucial for transferability, regularizing gradients towards a Gaussian distribution and amplifying those below a threshold to promote uniformity. It also employs high-frequency adaptation to enhance input diversity and further improve transferability.

Nevertheless, both approaches focus solely on homogenizing the gradient distribution within the surrogate model, without explicitly perturbing the shared decision patterns across different architectures. Consequently, their adversarial examples achieve only suboptimal transferability, as these methods inevitably incorporate some ineffective gradient updates. Furthermore, experiments show a notable performance drop when transferring attacks to CNN-based targets, reflecting the intrinsic gap between ViTs and CNNs in adversarial robustness.

We provide an overview of the COGO strategy in Figure 2, illustrating the process of a single iteration.

Starting with a zero-initialized perturbation added to the input, we apply a Discrete Cosine Transform (DCT) to project it into the frequency domain. There, the Commonality Enhancement (CE) module selectively amplifies mid-to-low frequency components based on an energy map, guiding the perturbation toward shared decision patterns across architectures. The modified frequency representation is then converted back to the spatial domain via Inverse DCT (IDCT) and passed through the surrogate model.

During backpropagation, the Individuality Suppression (IS) module adaptively suppresses surrogate-specific gradients using a thresholding mechanism, reducing overfitting and promoting transferable directions. This process iterates until convergence, producing the final adversarial example.

In the forward process of adversarial example generation, most existing methods focus on high-frequency perturbations [zhuenhancing]. However, recent studies [zhang2024mlip] show that ViTs rely more on mid-to-low frequency components, which encode essential structural and semantic information. As illustrated in Figure 3, these components dominate the energy spectrum, which are important in feature representation. Enhancing them not only reinforces ViT decision signals but also promotes gradients aligned with shared patterns, thereby improving adversarial transferability.

To leverage this property, our CE module adaptively enhances mid-to-low frequency components using the energy distribution of each input sample. Starting from a clean input $X_{\mathsf{clean}}\in\mathbb{R}^{N}$, we add the current perturbation $\delta\in\mathbb{R}^{N}$ (initialized as zero) to obtain:

Gaussian noise $\epsilon\sim\mathcal{N}(0,I_{N})$ is added to increase frequency diversity, and we apply the Discrete Cosine Transform (DCT) to map the perturbed sample to the frequency domain:

where $E(X_{\mathsf{DCT}})$ represents the normalized energy map. We then enhance the frequency representation in energy-dominant regions:

with $\gamma$ controlling the enhancement strength. Finally, the enhanced frequency sample is converted back to the spatial domain using IDCT. We apply a spatial mask $M$, inherited from HFA [zhuenhancing], to introduce additional perturbation diversity:

and feed $X_{\mathsf{IDCT}}$ into the surrogate model.

To validate the effectiveness of COGO, we conducted extensive experiments in various models. In this section, we describe the details of our experimental setup, including datasets, models, adversarial attack methods, evaluation metrics, and hyperparameter settings.

In this section, we evaluate the performance of our proposed methods against three categories of models: undefended ViTs, undefended CNNs, and adversarially trained CNNs. Specifically, we generate adversarial examples using a given surrogate model and attack other models in a black-box scenario, as well as the surrogate model itself in a white-box scenario. The results are shown in Table 1, where the best values in each column are highlighted in bold, and the second-best are underlined.

First, we evaluate transferability across different ViT models. In black-box settings, our strategy achieves an average improvement of 7.2% in attack success rates over GNS-HFA and an 10.1% enhancement over the ATT method.

Next, we evaluate the effectiveness of COGO against both undefended and adversarially trained CNNs to assess cross-architecture transferability. Despite the significant architectural differences between ViTs and CNNs, COGO achieves an average improvement of 2.3% over GNS-HFA and 10.5% over ATT.

To better understand the impact of COGO, we analyzed the gradient flow in ViT models during adversarial example generation. Without COGO, gradients in the $qkv$ module were highly concentrated on specific tokens and channels tied to dominant surrogate model features, causing perturbations to be overly model-specific and less transferable.

Applying COGO resulted in a more diverse gradient distribution across model blocks. To quantify this, we computed a dispersion-related indicator by flattening and normalizing gradient magnitudes into a probability-like vector and summing its log-weighted elements. This value reflects how broadly the gradients are spread across different dimensions. A larger value indicates a more balanced distribution and reduced dependence on surrogate-specific features. As shown in Figure 4, most blocks show increased values after applying COGO, suggesting improved gradient diversity. A few blocks, such as blocks[0] and blocks[10], show slightly reduced values due to the CE component, which enhances shared mid-to-low frequency features and causes gradient concentration in those regions. This complementarity between IS and CE promotes more transferable and model-common features.

Overall, the more balanced and diverse gradient landscape induced by COGO helps generate adversarial perturbations that better capture transferable features, thus improving attack success rates across different models.

To investigate the effect of the number of selected channel pairs ($n$) on gradient adjustments, we carried out experiments by varying $n$ for mutual information and correlation calculations. The default setting in our paper uses 5 channel pairs. However, we first tested additional settings with $n=1,3,7,9$. The results, summerized in Table 5 and visualized in Figure 6, show that the attack success rates remain largely consistent across different values of $n$. While minor fluctuations are observed, increasing the number of selected channel pairs does not significantly improve the attack success rates.

To ensure this consistency was not due to selecting too few channel pairs, we further conducted an experiment with a significantly larger value, $n=50$, and summarized the results in Table 6.Interestingly, the results for $n=50$ are comparable to, or even slightly worse than, the results for $n=9$, suggesting that selecting a larger number of channel pairs does not provide additional benefits. Furthermore, even with $n=1$, where only a single randomly selected channel pair is used for adjustments, the attack success rates are nearly on par with those achieved using larger values of $n$.

This phenomenon can be attributed to the nature of high-dimensional feature spaces, where most critical gradient information is often concentrated in a small subset of channels. Channels with higher mutual information or strong correlations tend to dominate the gradient flow, as they represent key relationships in the feature representation learned by the model. By adjusting even a small number of these channel pairs, it becomes possible to effectively disrupt redundant and overrepresented features in the gradient signal, thereby optimizing the perturbation’s impact.

Moreover, increasing $n$ beyond a certain point introduces diminishing returns because additional channel pairs often contribute redundant or low-variance information. In high-dimensional spaces, features are inherently structured such that a few dominant channels encode the majority of the relevant information, while others serve auxiliary roles or capture noise. Randomly selected channel pairs, even in small numbers, are likely to intersect with these dominant channels due to the distributional sparsity of critical features. This explains why the observed success rates remain consistent even with randomly chosen channel pairs, as the adjustments tend to influence significant regions of the feature space regardless of the specific pairs selected.

These findings show two important conclusions. First, the effectiveness of our gradient adjustment method depends more on its ability to mitigate redundancy in a targeted manner than on the exact number of selected channel pairs. Second, the method’s primary impact stems from its application itself rather than the specific channels being adjusted, with the critical distinction being between using this method and not using it at all. This robustness to the number of selected pairs underscores the generalizability and efficiency of our approach in optimizing perturbation transferability.

Although we adopted an adaptive threshold setting method, we evaluated different configurations of these thresholds to obtain more intuitive information.

The results, presented in Table 7, indicate that the selected threshold values of 0.5 for mutual information and 0.7 for correlation achieve the highest success rate across ViTs, CNNs, and adversarially trained CNNs. Lower or higher threshold values lead to decreased effectiveness. These findings provide valuable insights for our subsequent, more fine-grained research.

To further investigate the impact of suppressing the gradient of additional knowledge features in ViT variants, we conducted experiments by varying the scaling factor $c$ applied to the additional knowledge gradient. The scaling factors were set to $c=\{0.0,0.1,0.3,0.5,0.7,1.0\}$, and the resulting attack success rates were evaluated on ViTs, CNNs, and adversarially trained CNNs.

Table 8 presents the attack success rates across different scaling factors. The results demonstrate that $c=0.1$ achieves the best average attack success rate (61.20%) among all tested settings. This finding demonstrates that $c=0.1$ not only enhances the attack success rate across different model types but also achieves a balanced suppression of the additional knowledge gradient. In contrast, completely nullifying the additional knowledge gradient ($c=0.0$) or leaving it unadjusted ($c=1.0$) results in lower success rates, as these extremes either disregard the auxiliary knowledge provided by the additional features or overemphasize it, reducing the transferability of the perturbations. These results prove the importance of carefully tuning the additional knowledge gradient’s contribution to optimize adversarial attack performance.

To evaluate the effect of the step size constant $\lambda$ on attack success rates, we conducted experiments with varying $\lambda$ values: 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, and 2.0. The objective was to identify the optimal balance between perturbation strength and transferability for adversarial attacks.

The results in Table 9 and Figure 7 show a clear trend where smaller step sizes ($\lambda<1.0$) result in weaker perturbations, leading to lower attack success rates. As $\lambda$ increases to moderate values ($1.0\leq\lambda\leq 1.5$), the success rates improve significantly across all models, indicating that the perturbations are becoming more effective. This improvement can be attributed to the fact that appropriately increasing the step size introduces a degree of randomness to the generated perturbations, enhancing their generalization and reducing excessive reliance on precise gradient information.

However, for larger step sizes ($\lambda>1.5$), the performance plateaus or slightly declines, particularly for CNNs and adversarially trained CNNs. This suggests that over-amplifying the step size causes the gradient information to lose its guiding role in determining the perturbation direction, ultimately reducing the transferability of the perturbations. This observation underscores the importance of using a step size that is neither too conservative nor overly aggressive.

These findings prove that moderate step sizes, such as $\lambda=1.5$, are optimal to achieve a balance between perturbation strength and transferability. By ensuring sufficient perturbation strength without over-amplification, this step size allows for effective attacks across different model architectures while maintaining robustness and generalization.