YOLO-PRO: Enhancing Instance-Specific Object Detection with Full-Channel Global Self-Attention

The bottleneck structure is a crucial component of both the backbone and neck networks in the YOLO series algorithms. Its primary forms include the residual block structure (Fig.3 (a)) in YOLOv5, the ELAN structure (Fig.3 (b)) in YOLOv7, and the improved ELAN structure (Fig.3 (c)) in YOLOv8. Even YOLOv10 and YOLOv11 have adopted the bottleneck structure from YOLOv8. These structures not only enhance feature extraction capabilities through multi-level feature fusion but also improve the stability of gradient propagation, preventing gradient vanishing. Furthermore, they excel in computational efficiency and model lightweighting, significantly boosting inference speed, increasing the model’s AP, and enhancing real-time inference capabilities. However, during our investigation of these bottleneck structures, we observed that convolutional layers in all these structures utilize BN to normalize the features of batch samples and adjust the feature distribution through trainable parameters. While this approach effectively enhances the model’s generalization ability, it neglects the differences between individual sample channel features. Overemphasizing the statistical features of the batch may cause misclassification of different object categories with similar or ambiguous features (e.g., airplane and bird in the distance). This can lead to a reduction in AP during inference with new samples and cause gradient conflicts when optimizing the model’s weights during training. Furthermore, these bottleneck structures are primarily inherited or minimally adapted from image classification networks(He et al. 2016). It should be emphasized that image classification tasks focus on identifying global semantic information of entire images, where object localization and quantity estimation are irrelevant. In contrast, object detection requires precise recognition of multiple instances with accurate spatial localization and categorical identification. Consequently, the direct transplantation or minor adaptation of classification-oriented bottleneck(He et al. 2016) structures exhibits suboptimal alignment with detection objectives. The critical limitation lies in their exclusive focus on feature extraction while lacking explicit spatial positional modeling capabilities, thereby failing to establish pixel-wise global contextual correlations. This architectural deficiency directly induces positional insensitivity in detection networks. To address these issues, we propose a novel bottleneck structure, named Instance-Specific Bottleneck with Full-Channel Global Self-Attention (ISB) (Fig.4).

The ISB structure introduces an additional branch to the bottleneck of YOLOv8. This branch primarily aims to enhance the model’s ability to learn representations of individual instance features, as well as to improve the model’s attention to global contextual relationships while simultaneously increasing its sensitivity to spatial positional information. Based on these two objectives, the ISB branch is designed to incorporate a structure consisting of four key components : the instance-specific module, the feature patch-channel reconstructor, the self-attention module, and the feature patch-channel reassembler.

The construction of the instance-specific module (Fig.4 (a)) is relatively straightforward. The first layer consists of a $1\times 1$ convolutional layer combined with an IN layer and a SiLU(Ramachandran, Zoph, and Le 2017) activation, primarily used to compress the input feature map. Assuming the number of input channels is $c\in\mathbb{R}$, this layer reduces the channel count from $c$ to $c_{1}$, where $c_{1}=\lfloor\frac{c}{s}\rfloor\times 3$ (the factor of 3 is included for the calculation of the Q, K, and V in the self-attention module(Dosovitskiy et al. 2020)). Here, $s$ is set to 8 by default, representing the compression ratio. This channel compression not only improves model efficiency but also refines the features. The second layer is located at the end of the branch and consists of a $3\times 3$ convolutional layer combined with an IN layer and a SiLU activation. This layer restores the compressed channel count from $c_{2}=\lfloor\frac{c}{s}\rfloor$ (without multiplying by 3, as the results from the self-attention module’s Q, K, and V have already been computed) to $c_{3}$, where $c_{3}=c$. The primary objective of this module is to normalize the internal instance (channel) features of each sample independently, without relying on batch statistics. This enhances the distinct features of individual instances while avoiding potential issues related to inaccurate statistical estimates that may arise from batch normalization. Additionally, through the use of channel compression and expansion, the module effectively reduces the parameters and GFLOPs, thereby improving the inference efficiency of the model.

The feature patch-channel reconstructor is a pivotal module designed to restructure input feature maps through a three-stage transformation. Given an input $x\in\mathbb{R}^{B\times C\times H\times W}$, it first decomposes the spatial structure by unfolding non-overlapping $K\times K$ (default is 4) patches (stride K) into a flattened tensor $x_{1}\in\mathbb{R}^{B\times(CK^{2})\times L}$, where $L=\lfloor\frac{H-1}{K}+1\rfloor\times\lfloor\frac{W-1}{K}+1\rfloor$. Subsequently, the tensor is reshaped into $x_{2}\in\mathbb{R}^{B\times C\times K^{2}\times L}$ to decouple channel and patch dimensions, followed by permuting the $K^{2}$ and $C$ axes to produce the final representation $x_{3}\in\mathbb{R}^{B\times K^{2}\times C\times L}$. This hierarchical reorganization explicitly isolates spatial-local ($K^{2}$) and channel-wise ($C$) interactions, facilitating efficient feature reconstruction for downstream tasks. This module aims to reconstruct features for the self-attention module, fundamentally diverging from the feature processing paradigms of Vision Transformer (ViT)(Dosovitskiy et al. 2020) and Swin-Transformer(Liu et al. 2021).

In comparison with ViT and Swin-Transformer, our approach fundamentally redesigns the construction of input self-attention feature maps by replacing the head division method in the multi-head mechanism(Dosovitskiy et al. 2020) with feature patch size adaptation (Fig.6 (a)). The processed feature maps ($x_{3}$) are fed into the self-attention module to derive low-resolution full-channel global weighted features based on fixed positional pixels within feature patchs (Fig.6 (b)). These partial features are subsequently aggregated to form comprehensive full-channel global representations (Fig.6 (c)). To address the lack of intra-patch pixel information interaction, we implement a $3\times 3$ convolutional layer (CIS) to facilitate cross-pixel feature sharing within each block, thereby achieving complete full-channel global attention weighting. This methodology not only effectively captures cross-channel global dependencies at the feature block level but also significantly enhances computational efficiency of the self-attention mechanism. Furthermore, considering the heightened positional sensitivity in object detection models compared to classification tasks, our proposed full-channel attention mechanism demonstrates superior capability in modeling global spatial relationships between feature blocks and their constituent pixels, ultimately improving AP in object detection.

The computational procedure of the self-attention module has been improved compared to ViT. Given input feature maps $Q,K,V\in\mathbb{R}^{B\times K^{2}\times C\times L}$ and output feature maps $F\in\mathbb{R}^{B\times K^{2}\times C\times L}$, the specific computation process is implemented as follows:

Where the Softmax function serves as the activation function, and $L$ represents the number of feature patchs. The improvement in this equation, compared to the ViT, lies in the early division of $Q$ by the scaling factor $\sqrt{L}$. This adjustment aims to prevent the issue of excessively large values in the matrix multiplication of $Q$ and $K^{T}$, which can lead to NaN values during training.

The feature patch-channel reassembler operates through three key computational stages: First, the feature tensor $x_{f}\in\mathbb{R}^{B\times K^{2}\times C\times L}$ processed by the self-attention module undergoes dimension permutation, where the $K^{2}$ and C dimensions are transposed to produce intermediate feature $x_{f1}\in\mathbb{R}^{B\times C\times K^{2}\times L}$. Subsequently, a tensor reshaping operation is applied to transform $x_{f1}$ into a 2D feature matrix $x_{f2}\in\mathbb{R}^{B\times(CK^{2})\times L}$. Finally, the fold operation reassembles $x_{f2}$ into feature maps $x_{f3}\in\mathbb{R}^{B\times C\times H\times W}$, which are then fed into convolutional layers for subsequent processing.

In summary, the ISB module is fundamentally designed to enhance instance feature discriminability by fusing instance-level features into batch-level feature representations. Meanwhile, departing from the conventional multi-head attention paradigm with subspace partitioning, ISB innovatively reconstructs feature maps into feature patchs and establishes global attention correlations across full-channel dimensions and spatially fixed positions within each patch. A convolutional layer is subsequently employed to enable cross-patch feature interaction, thereby constructing an efficient full-channel global self-attention mechanism. This structure not only strengthens contextual dependency modeling and spatial position-aware representation, but also achieves significant improvements in AP.

The optimization of the decoupled head has been a central focus, as its need to decouple classification and regression tasks into separate network branches inherently results in a substantial increase in both model parameters and computational cost (GFLOPs), thus degrading the model’s overall inference efficiency. Furthermore, since the decoupled head directly generates final detection outputs, its structural design critically impacts the AP. To address these dual objectives, we optimize the decoupled head through architectural refinement and computational efficiency enhancements, proposing two key design innovations.

The first key design (The region enclosed by the orange dashed box in Fig.7 (b)) involves reducing the kernel size of the bounding box prediction branch in the YOLOv8 decoupled head (Fig.7 (a)) from $3\times 3$ to $1\times 1$. This modification significantly reduces the parameters and GFLOPs of the decoupled head while preserving AP performance. Our design rationale draws from YOLOCS’s analysis of the logical relationship between detection heads and loss functions(Huang et al. 2025). In object detection, the confidence branch (merged with the classification branch in YOLOv8) primarily handles global grid predictions, making bounding box regression subordinate to classification in task priority. Consequently, strategically minimizing parameters allocated to regression tasks not only maintains detection accuracy but also enhances decoupled head efficiency.

The second key design (The region enclosed by the green dashed box in Fig.7 (b)) involves introducing parallel instance-specific branches to both classification and regression task branches. Specifically, we construct a dedicated feature processing path comprising a $1\times 1$ convolutional layer, instance normalization layer, and activation function within each task branch. This structure aims to extract instance-specific channel-normalized features from individual samples and adaptively fuse them with batch-level statistical features, thereby enhancing the decoupled detection head’s capacity to characterize sample-unique discriminative features. The proposed dual-stream feature fusion mechanism not only incorporates global statistical information from batch normalization but also preserves instance-specific features. Through synergistic optimization of these complementary features, this approach achieves significant improvement in AP for the detection head.

In summary, these two key designs collectively constitute our proposed Instance-Specific Asymmetric Decoupled Head (ISADH) module. The structure aims to achieve dual objectives: 1) substantially improving inference efficiency by optimizing parameters and GFLOPs, and 2) significantly enhancing detection accuracy through synergistic integration of instance-specific features with batch-statistical features, as evidenced by the notable improvement in AP.

The visualization analysis employed Eigen-CAM(Muhammad and Yeasin 2020) to generate attention heatmaps for representative samples selected from the val2017 dataset. The selected samples were divided into five categories: single-class single-object, single-class multi-object, single-class overlapping objects, multi-class overlapping objects, and multi-class multi-object. Attention heatmaps were generated for these samples using the baseline YOLOv8 model, the state-of-the-art YOLO11 model, and our proposed YOLO-PRO model, followed by comparative analysis.

In the single-class single-object category group (Fig.8 (a)), our proposed YOLO-PRO model demonstrated more precise coverage of object feature regions in attention heatmaps compared to other models, with minimal background noise interference. In the single-class multi-object category (Fig.8 (b)), the YOLOv8 model exhibited uneven attention heatmap distributions across objects, showing a bias toward centrally located objects. The YOLO11 model generated broader attention heatmap regions for each object but suffered from excessive background noise and inconsistent attention allocation to peripheral objects. In contrast, our YOLO-PRO model achieved uniform attention distribution across all objects while maintaining minimal background interference.

For the single-class overlapping objects group (Fig.8 (c)), both YOLOv8 and YOLO11 displayed varying degrees of attention bias and failed to distinguish different instances within overlapping regions. The YOLO-PRO model, however, produced evenly distributed attention heatmaps that clearly differentiated overlapping instances. In the multi-class overlapping objects group (Fig.8 (d)), the YOLO11 model exhibited severe attention bias toward specific classes (e.g., prioritizing ”dog” detection), while YOLOv8 generated uneven heatmaps for coexisting classes. The YOLO-PRO model outperformed both, achieving balanced attention distribution without class-specific bias.

In the multi-class multi-object group (Fig.8 (e)), the YOLO11 model continued to show significant attention bias. The YOLOv8 model suffered from feature confusion between adjacent objects (e.g., the leftmost and second-left human instances) and prioritized high-frequency categories (e.g., the ”person” class in MS-COCO), resulting in suboptimal global attention allocation. The YOLO-PRO model avoided over-prioritizing dominant categories and mitigated suppression of small objects by large ones. While all three models exhibited uneven global attention distribution, YOLO-PRO demonstrated superior performance in suppressing excessive large-object dominance and class prioritization. The visualization analysis conclusively demonstrates the outstanding performance of YOLO-PRO.

To validate the effectiveness of the proposed modules, we conducted ablation experiments on the MS-COCO dataset. Based on the baseline YOLOv8 architecture, we successively incorporated the innovative ISB and ISADH modules individually and in combination, comparing the variations in AP metrics while evaluating changes in parameters (Params) and computational cost (GFLOPs). The experimental results shown in Tab.1 demonstrate the AP improvements over the baseline model, with green highlights indicating the enhancement magnitude.

Upon integrating the ISB module into the baseline architecture, we observed a 0.9% enhancement in the key detection metric AP, accompanied by a 2M increase in parameters and an 8.3 GFLOPs rise in computational cost. The achieved performance gain with minimal resource overhead confirms the structural efficacy of this module. Remarkably, the incorporation of the ISADH module into the baseline model not only elevated AP by 0.2% but also reduced both parameters and computational cost by 0.4M and 1.5 GFLOPs respectively, demonstrating its optimization capabilities. The combined integration of ISB and ISADH modules ultimately yielded a 1.0% AP improvement with 1.6M additional parameters and 6.7 GFLOPs computational increment, substantiating that the synergistic operation of dual modules achieves optimal performance enhancement within acceptable resource constraints.

This study conducts a multi-scale model comparison experiment by simultaneously adjusting network depth factors and width factors while constraining the maximum number of channels, thereby constructing five computational-scale model variants (N/S/M/L/X) as detailed in Tab.2. To ensure experimental comparability, the depth scaling ratios, width scaling ratios, and maximum channel configurations of all scale models strictly align with the YOLOv8 baseline model specifications (parameter configurations see Tab.2). All models were trained and evaluated on the MS-COCO dataset, utilizing AP as the primary evaluation metric while comparatively analyzing key indicators including parameters (Params), GFLOPs, and inference latency. Latency measurements were conducted under standardized hardware conditions: using an NVIDIA T4 GPU with TensorRT acceleration framework and half-precision floating-point (FP16) mode. To mitigate measurement deviations caused by hardware environmental fluctuations, the final latency data was determined based on the minimum inference time per single image.

As shown in Tab.3, the comparative analysis between the proposed YOLO-PRO and YOLOv8 baseline models across multiple computational scales demonstrates that YOLO-PRO variants achieve a 1.0%-1.6% improvement in AP over their YOLOv8 counterparts, while maintaining comparable inference latency, parameters (Params), and GFLOPs. All YOLO-PRO variants consistently outperform the baseline model in AP metrics. Remarkably, the L-scale variant of YOLO-PRO attains comparable AP to the X-scale YOLOv8 model while demonstrating superior computational efficiency. These results confirm that YOLO-PRO achieves comprehensive performance breakthroughs with preserved resource efficiency, establishing leading performance in the field.

To further validate the model’s advancement, this study conducts a multidimensional performance comparison between YOLO-PRO and the state-of-the-art (SOTA) YOLO11(baseline) model (Tab.4, where red highlighting indicates AP disadvantages and green highlighting denotes advantages). Experimental results show that YOLO-PRO variants (N/M/L/X) outperform their YOLO11 counterparts by 0.1%-0.5% in AP. Crucially, YOLO-PRO maintains persistent superiority in the core detection accuracy metric, which substantiates its SOTA effectiveness in object detection tasks.

As presented in Tab.5, this study conducts a systematic comparative analysis between the proposed YOLO-PRO and contemporary SOTA object detectors within a unified benchmarking framework. The evaluation encompasses five critical dimensions: primary detection accuracy metric (AP), input resolution, parameters (Params), GFLOPs, and inference latency. Notably, boldface numerical values in the table denote detectors achieving optimal AP performance within their respective computational scale groups (N/S/M/L/X).

As demonstrated by the experimental data in Tab.5, the proposed YOLO-PRO achieves optimal AP in medium (M) and large (L) computational scale groups while maintaining comparable parameters (Params), GFLOPs, and inference latency relative to peer models. This systematic comparison of core metrics confirms the architectural advancement of YOLO-PRO in object detection tasks, substantiating its SOTA status.