UniFormer: Unifying Convolution and Self-attention for Visual Recognition

In the past few years, the development of computer vision has been mainly driven by convolutional neural networks (CNNs). Beginning with the classical AlexNet [49], many powerful CNN networks have been proposed [80, 84, 37, 107, 41, 40, 120, 85] and achieved remarkable performance in various tasks of image understanding [22, 60, 121, 36, 6, 47, 104, 95]. Recently, due to the fact that video has gradually become one main data resource in many realistic applications, researchers have attempted to apply CNNs in the video domain. Naturally, one can adapt 2D convolution as 3D one, by temporal dimension extension [89]. However, 3D CNNs often suffer from difficult optimization problem and large computation cost. To resolve these issues, the prior works try to inflate the pre-trained 2D convolution kernels for better optimization [11] and factorize 3D convolution kernels in different dimensions to reduce complexity [91, 72, 90, 29, 30, 52]. Besides, many recent studies of video understanding [44, 59, 53, 56] focus on adapting vanilla 2D CNNs with elaborated temporal modeling modules, such as temporal shift [59, 67], motion enhancement [44, 56, 63], and spatiotemporal excitation [53, 52], etc. Unfortunately, due to the limited reception field, the traditional convolution struggles to capture long-range dependency even if they are stacked deeper.

To capture long-term dependencies, Vision Transformer (ViT) has been proposed [24]. With the inspiration of Transformer architectures in NLP [92], ViT treats image as a number of visual tokens and leverages attention to encode token relations for representation learning. However, vanilla ViT depends on sufficient training data and careful data augmentation. To tackle these problems, several approaches have been developed by improved patch embedding [55], data-efficient training [87], efficient self-attention [61, 23, 109], and multi-scale architectures [99, 28, 100]. These works successfully boost the performance of ViT on various image tasks [8, 123, 106, 15, 46, 55, 110, 114, 38, 12, 57]. Recently, researchers have attempted to extend image ViTs for video modeling. The classical work is TimeSformer [3] by spatial-temporal attention. Starting from this, many works propose different variants for spatiotemporal representation learning [3, 1, 28, 71, 5, 62], and subsequently they are adapted to various video understanding tasks [98, 25, 26, 31, 7]. Although these works demonstrate the outstanding ability of ViTs to learn long-term token relations, the self-attention mechanism requires costly token-to-token comparisons [24]. Hence, it is often inefficient to encode low-level features, as shown in Figure 2. Though Video Swin [62] advocates an inductive bias of locality with shift window, window-based self-attention is still less efficient than local convolution [37] when encoding low-level features. Moreover, the shifted window should be carefully configured.

To bridge the gap between CNNs and ViTs, researchers have tried to take advantage of them to build stronger vision backbones for image understanding, by adding convolutional patch stem for fast convergence [111, 105], introducing convolutional position embedding [17, 23], inserting depthwise convolution into feed-forward network [111, 114], utilizing convolutional projection in self-attention [102], and combining MBConv [77] with Transformer [21]. As for video understanding, the combination is also straightforward, i.e., one can insert self-attention as global attention [101], and/or use convolution as patch stem [64]. However, all these approaches ignore inherent relations between convolution and self-attention, leading to inferior local and/or global token relation learning. Several recent works have demonstrated that self-attention operates similarly to convolution [75, 20]. But they suggest replacing convolution instead of combining them together. Differently, our UniFormer unifies convolution and self-attention in the transformer style, which can effectively learn local and global token relation, and achieve better accuracy-computation trade-offs on all the vision tasks from image to the video domain.

In many practical applications, the running platforms usually lack enough computility. Hence, a series of lightweight CNNs are proposed to satisfy such on-device requirements. For example, the classical MobileNets [40, 77, 39] adopt depthwise separable convolution in well-organized efficient ResNet. ShuffleNets [120, 68] leverage channel shuffle for computation reduction. EfficientNets [85, 86] further scale model with neural architecture search at depth, width, and resolution. However, such lightweight design has not been fully investigated in ViTs. Two recent works are proposed by designing transformers as convolutions (i.e., MobielViT[69]), and introducing a parallel architecture of MobileNet and ViT (i.e., MobileFormer[14]). But both works ignore the inference speed (e.g., throughout). Hence, the prior efficient CNNs are still the better choice. To bridge this gap, we build a lightweight UniFormer by token shrinking and recovering in Section 5.

Figure 3 shows our Unified transFormer (UniFormer). For simple description, we take a video with $T$ frames as an example and an image input can be seen as a video with a single frame. Hence, the dimensions highlighted in red only exit for the video input, while all of them are equal to one for image input. Our UniFormer is a basic transformer format, while we elaborately design it to tackle computational redundancy and capture complex dependency.

Specifically, our UniFormer block consists of three key modules: Dynamic Position Embedding (DPE), Multi-Head Relation Aggregator (MHRA) and Feed-Forward Network (FFN):

Considering the input token tensor $\mathbf{X}_{in}\in\mathbb{R}^{C\times T\times H\times W}$ ($T$$=$$1$ for an image input), we first introduce DPE to dynamically integrate position information into all the tokens (Eq. 1). It is friendly to arbitrary input resolution and makes good use of token order for better visual recognition. Then, we use MHRA to enhance each token by exploiting its contextual tokens with relation learning (Eq. 2). Via flexibly designing the token affinity in the shallow and deep layers, our MHRA can smartly unify convolution and self-attention to reduce local redundancy and learn global dependency. Finally, we add FFN like traditional ViTs [24], which consists of two linear layers and one non-linear function, i.e., GELU (Eq. 3). The channel number is first expanded by the ratio of 4 and then recovered, thus each token will be enhanced individually.

As analyzed before, the traditional CNNs and ViTs focus on addressing either local redundancy or global dependency, leading to unsatisfactory accuracy or/and unnecessary computation. To overcome these difficulties, we introduce a generic Relation Aggregator (RA), which elegantly unifies convolution and self-attention for token relation learning. It can achieve efficient and effective representation learning by designing local and global token affinity in the shallow and deep layers respectively. Specifically, MHRA exploits token relationships in a multi-head style:

Given the input tensor $\mathbf{X}\in\mathbb{R}^{C\times T\times H\times W}$, we first reshape it to a sequence of tokens $\mathbf{X}\in\mathbb{R}^{L\times C}$ with the length of $L$$=$$T$$\times$$H$$\times$$W$. Moreover, ${\rm R}_{n}(\cdot)$ refers to RA in the $n$-th head and $\mathbf{U}\in\mathbb{R}^{C\times C}$ is a learnable parameter matrix to integrate $N$ heads. Each RA consists of token context encoding and token affinity learning. We apply a linear transformation to encode the original tokens into contextual tokens ${\rm V}_{n}(\mathbf{X})\in\mathbb{R}^{L\times\frac{C}{N}}$. Subsequently, RA can summarize context with the guidance of token affinity ${\rm A}_{n}\in\mathbb{R}^{L\times L}$. We will describe how to learn the specific ${\rm A}_{n}$ in the following.

The position information is an important clue to describe visual representation. Previously, most ViTs encode such information by absolute or relative position embedding [24, 99, 61]. However, absolute position embedding has to be interpolated for various input sizes with fine-tuning [87, 88], while relative position embedding does not work well due to the modification of self-attention [17]. To improve flexibility, convolutional position embedding has been recently proposed [16, 23]. In particular, conditional position encoding (CPE) [17] can implicitly encode position information via convolution operators, which unlocks Transformer to process arbitrary input size and promotes recognition performance. Due to its plug-and-play property, we flexibly adopt it as our Dynamical Position Embedding (DPE) in the UniFormer:

where ${\rm DWConv}$ refers to depthwise convolution with zero paddings. We choose such a design as our DPE based on the following reasons. First, depthwise convolution is friendly to arbitrary input shapes, e.g., it is straightforward to use its spatiotemporal version to encode 3D position information in videos. Second, depthwise convolution is light-weight, which is an important factor for computation-accuracy balance. Finally, we add extra zero paddings, since it can help tokens be aware of their absolute positions by querying their neighbors progressively [17].

It is important to progressively learn visual representation for capturing semantics in the image. Hence, we build up our backbone with four stages, as illustrated in Figure 3.

More specifically, we use the local UniFormer blocks in the first two stages to reduce computation redundancy, while the global UniFormer blocks are utilized in the last two stages to learn long-range token dependency. For the local UniFormer block, MHRA is instantiated as PWConv-DWConv-PWConv with local token affinity (Eq. 6), where the spatial size of DWConv is set to 5$\times$5 for image classification. For the global UniFormer block, MHRA is instantiated as multi-head self-attention with global token affinity (Eq. 7), where the number of attention heads is set to 64. For both local and global UniFormer blocks, DPE is instantiated as DWConv with a spatial size of 3$\times$3, and the expand ratio of FFN is 4.

Additionally, as suggested in the CNN and ViT literatures [37, 24], we utilize BN [43] for convolution and LN [2] for self-attention. For feature downsampling, we use the 4$\times$4 convolution with stride 4$\times$4 before the first stage and the 2$\times$2 convolution with stride 2$\times$2 before other stages. Besides, an extra LN is added after each downsampling convolution. Finally, the global average pooling and fully connected layer are applied to output the predictions. When training models with Token Labeling [45], we add another fully connected layer for auxiliary loss. For various computation requirements, we design three model variants as shown in Table I.

Given our image-based 2D backbones, one can easily adapt them as 3D backbones for video classification. Without loss of generality, we adjust Small and Base models for spatiotemporal modeling. Specifically, the model architectures keep the same with four stages, where we use the local UniFormer blocks in the first two stages and the global UniFormer blocks in the last two stages. But differently, all the 2D convolution filters are changed as 3D ones via filter inflation [11]. Concretely, the kernel size of DWConv in DPE and local MHRA are 3$\times$3$\times$3 and 5$\times$5$\times$5 respectively. Moreover, we downsample both spatial and temporal dimensions before the first stage. Hence, the convolution filter before this stage becomes 3$\times$4$\times$4 with the stride of 2$\times$4$\times$4. For the other stages, we just downsample the spatial dimension to decrease the computation cost and maintain high performance. Hence, the convolution filters before these stages are 1$\times$2$\times$2 with stride of 1$\times$2$\times$2 .

Note that we use spatiotemporal attention in the global UniFormer blocks for learning token relation jointly in the 3D view. It is worth mentioning that, due to the large model sizes, the previous video transformers [3, 1] divide spatial and temporal attention to reduce computation and alleviate overfitting, but such factorization operation inevitably tears spatiotemporal token relations. In contrast, our joint spatiotemporal attention can avoid the issue. Besides, our local UniFormer blocks largely save computation via 3D DWconv. Hence, our model can achieve effective and efficient video representation learning.

Dense prediction tasks are necessary to verify the generality of our recognition backbones. Hence, we adopt our UniFormer backbones for a number of popular dense tasks such as object detection, instance segmentation, semantic segmentation, and human pose estimation. However, direct usage of our backbone is not suitable because of the high input resolution of most dense prediction tasks, e.g., the size of the image is 1333$\times$800 in the COCO object detection dataset. Naturally, feeding such images into our classification backbones would inevitably lead to large computation, especially when operating self-attention of global UniFormer block in the last two stages. Taking $h$$\times$$w$ visual tokens as an example, the MatMul operation in token similarity comparison (Eq. 7) causes ${\mathcal{O}}(w^{2}h^{2})$ complexity, which is prohibitive for most dense tasks.

We propose to adjust the global UniFormer block for different downstream tasks. First, we analyze the FLOPs of our UniFormer-S under different input resolutions. Figure 4 clearly indicates that Relation Aggregator (RA) in Stage3 occupies large computation. For example, for a 1008$\times$1008 image, the MatMul operation of RA in Stage3 even occupies over 50% of the total FLOPs, while the FLOPs in Stage4 is only 1$/$28 of that in Stage3. Thus we focus on modifying RA in Stage3 for computation reduction.

Inspired by [75, 61], we propose to apply our global MHRA in a predefined window (e.g., 14$\times$14), instead of using it in the entire image with high resolution. Such operation can effectively cut down computation with the complexity of ${\mathcal{O}}(whp^{2})$, where $p$ is the window size. However, it undoubtedly drops model performance, due to insufficient token interaction. To bridge this gap, we integrate window and global UniFormer blocks together in Stage3, where a hybrid group consists of three window blocks and one global block. In this case, there are 2/5 hybrid groups in Stage3 of our UniFormer-Small/Base backbones.

Based on this design, we next introduce the specific backbone settings of various dense tasks, depending on the input resolution of training and testing images. For object detection and instance segmentation, the input images are usually large (e.g., 1333$\times$800), thus we adopt the hybrid block style in Stage3. In contrast, the inputs are relatively small for pose estimation, such as 384$\times$288, hence global blocks are still applied in Stage3 for both training and testing. Specially, for semantic segmentation, the testing images are often larger than the training ones. Therefore, we utilize the global blocks in Stage3 for training, while adapting the hybrid blocks in Stage3 for testing. We use the following simple design. The window-based block in testing has the same receptive field as the global block in training, e.g., 32$\times$32. Such design can maintain training efficiency, and boost testing performance by keeping consistency with training as much as possible.

Since a large computation load lies in token similarity comparison in the global UniFormer block, we propose an Hourglass UniFormer (H-UniFormer) block to reduce the number of visual tokens involved in global MHRA. Note that, the existing token pruning methods [76, 58, 124] are infeasible for our UniFormer and those ViTs with convolution [100, 111, 21]. The main reason is that, after pruning, the rest tokens often maintain a broken spatiotemporal structure, which makes convolution inapplicable. To overcome such difficulty, we propose a concise integration of token shrinking and recovering in our H-UniFormer block.

Token Shrinking. We introduce a score token $\mathbf{s}\in\mathbb{R}^{C}$ to measure the importance of the visual tokens $\mathbf{X}\in\mathbb{R}^{C\times THW}=\{\mathbf{X}_{1},...,\mathbf{X}_{THW}\}$ after DPE. Specifically, we compare similarity (using Eq. 7) between the score token $\mathbf{s}$ and a visual token $\mathbf{X}_{j}$, and average the similarity values over $N$ attention heads,

where $\mathbf{A}_{j}$ represents the importance of visual token $\mathbf{X}_{j}$, and we use $\mathbf{A}\in\mathbb{R}^{THW}$ to represent the importance vector of all the visual tokens. For tokens with high values in $\mathbf{A}$, we consider them to be crucial tokens and keep them. For tokens with low values, we consider them to be unimportant tokens. Hence, we fuse them as one representative token, where their score values are used as the fusing weights. By such unimportant token reduction, we can shrink visual tokens from $\mathbf{X}\in\mathbb{R}^{C\times THW}$ to $\mathbf{X}^{S}\in\mathbb{R}^{C\times M}$, where the number of tokens $M\ll THW$. Subsequently, the reduced visual tokens are fed into global MHRA and FFN, for computation saving. Additionally, we pass this score token through all the H-UniFormer blocks in Stage3 and Stage4, and use it to calculate the classification loss when training. In this case, the score token is effectively guided by the ground-truth label, and it thus becomes discriminative to weight token importance.

Token Recovering. After learning token interactions in global MHRA and FFN, we replicate the representative token to recover the unimportant tokens. Thus, we can maintain the spatiotemporal structure of all the visual tokens, for effective dynamic position encoding (i.e., convolution) in the next H-UniFormer block.

To build our light-weight UniFormer, We follow most of the architectures in Section 4.1, except that we change to use the H-UniFormer block in Stage3 and Stage4, and adopt smaller depth, width or resolution (e.g., 128). Specifically, for UniFomrer-XS, the block number, channel number and head dimension are [3, 5, 9, 3], [64, 128, 256, 512] and 32. For UniFomrer-XXS, the block number, channel number and head dimension are [2, 5, 8, 2], [56, 112, 224, 448] and 28.

Beginning from the second layer in Stage3, we utilize the similarity score in the previous layer $\mathbf{A}^{pre}$ to guide the token shrinking. Based on the phenomenon that the locations of the crucial tokens are basically the same among different layers [108], we update the similarity scores via mean, i.e., $\mathbf{A}=(\mathbf{A}+\mathbf{A}^{pre})/2$. Thus it can focus on the significant tokens consistently. By default, we keep half of the tokens and fuse the rest (i.e., the shrinking ratio is 0.5) in our light-weight UniFormer.

Settings. We train our models from scratch on the ImageNet-1K dataset [22]. For a fair comparison, we follow the same training strategy proposed in DeiT [87] by default, including strong data augmentation and regularization. Additionally, we set the stochastic depth rate as 0.1/0.3/0.4 respectively for our UniFormer-S/B/L in Table I. We train all models via AdamW [65] optimizer with cosine learning rate schedule [66] for 300 epochs, while the first 5 epochs are utilized for linear warm-up [33]. The weight decay, learning rate and batch size are set to 0.05, 1e-3 and 1024 respectively. For UniFormer-S$\dagger$, we follow state-of-the-art ViTs [23, 109] to apply overlapped patch embedding and blocks (3/5/9/3 blocks in each stage) for fair comparisons. As for UniFormer-B, we use the learning rate of 8e-4 for better convergence.

For training high-performance ViTs, hard distillation [87] and Token Labeling [45] are proposed, both of which are complementary to our backbones. Since Token Labeling is more efficient, we apply it with an extra fully connected layer and auxiliary loss, following the settings in LV-ViT [45]. Different from the training settings in DeiT, MixUp [118] and CutMix [115] are not used since they conflict with MixToken [45]. The base learning rate is 1.6e-3 for the batch size of 1024 by default. Specially, we adopt the base learning rate of 1.2e-3 and layer scale [88] for UniFormer-L to avoid NaN loss. When fine-tuning our models on larger resolution, i.e., 384$\times$384, the weight decay, learning rate, batch size, warm-up epoch and total epoch are set to 1e-8, 5e-6, 512, 5 and 30.

Results. In Table II, we compare our UniFormer with the state-of-the-art CNNs, ViTs and their combinations. It clearly shows that our UniFormer outperforms previous models under different computation restrictions. For example, our UniFormer-S$\dagger$ achieves 83.4% top-1 accuracy with only 4.2G FLOPs, surpassing RegNetY-4G [74], Swin-T [61], CSwin-T [23] and CoAtNet [21] by 3.4%, 2.1%, 0.7% and 1.8% respectively. Though EfficientNet [85] comes from extensive neural architecture search, our UniFormer outperforms it (83.9% vs. 83.6%) with less computation cost (8.3G vs. 9.9G). Furthermore, we enhance our models with Token Labeling [45], which is denoted by ‘^⋆’. Compared with the models training with the same settings, our UniFormer-L achieves higher accuracy but only 21% FLOPs of LV-ViT-M[45] and 61% FLOPs of VOLO-D3[113]. Moreover, when fine-tuned on 384$\times$384 images, our UniFormer-L obtains 86.3% top-1 accuracy. It is even better than EfficientNetV2-L [86] with larger input, demonstrating the powerful learning capacity of our UniFormer.

Settings. We evaluate our UniFormer on the popular Kinetics-400 [10], Kinetics-600 [9], UCF101 [81] and HMDB51 [93], and we verify the transfer learning performance on temporal-related datasets Something-Something (SthSth) V1&V2 [34]. Our codes mainly rely on PySlowFast [27]. For training, we adopt the same training strategy as MViT [28] by default, but we do not apply random horizontal flip for SthSth. We utilize AdamW [65] optimizer with cosine learning rate schedule [66] to train our video backbones.. The first 5 or 10 epochs are used for warm-up [33] to overcome early optimization difficulty. For UniFormer-S, the warmup epoch, total epoch, stochastic depth rate, weight decay are set to 10, 110, 0.1 and 0.05 respectively for Kinetics, 5, 50, 0.3 and 0.05 respectively for SthSth, and 5, 20, 0.2 and 0.05 for UCF101 and HMDB51. For UniFormer-B, all the hyper-parameters are the same unless the stochastic depth rates are doubled. Moreover, We linearly scale the base learning rates according to the batch size, which are 1e-4$\cdot\frac{batchsize}{32}$ for Kinetics, 2e-4$\cdot\frac{batchsize}{32}$ for SthSth, and 1e-5$\cdot\frac{batchsize}{32}$ for UCF101 and HMDB51.

We utilize the dense sampling strategy [101] for Kinetics, UCF101 and HMDB51, and uniform sampling strategy [96] for Something-Something. To reduce the total training cost, we inflate the 2D convolution kernels pre-trained on ImageNet for Kinetics [11]. To obtain a better FLOPs-accuracy balance, Besides, we adopt multi-clip testing for Kinetics and multi-crop testing for Something-Something. All scores are averaged for the final prediction.

Results on Kinetics. In Table III, we compare our UniFormer with the state-of-the-art methods on Kinetics-400 and Kinetics-600. The first part shows the prior works using CNN. Compared with SlowFast [30] equipped with non-local blocks [101], our UniFormer-S_16f requires $\mathbf{42}\times$ fewer GFLOPs but obtains 1.0% performance gain on both datasets (80.8% vs. 79.8% and 82.8% vs. 81.8%). Even compared with MoViNet [48], which is a strong CNN-based models via extensive neural architecture search, our model achieves slightly better results (82.0% vs. 81.5%) with fewer input frames ($16f$$\times$$4$ vs. $120f$). The second part lists the recent methods based on vision transformers. With only ImageNet-1K pre-training, UniFormer-B_16f surpasses most existing backbones with large dataset pre-training. For example, compared with ViViT-L [1] pre-trained from JFT-300M [83] and Swin-B [62] pre-trained from ImageNet-21K, UniFormer-B_32f obtains comparable performance (82.9% vs. 82.8% and 82.7%) with $\mathbf{16.7\times}$ and $\mathbf{3.3\times}$ fewer computation on both Kinetics-400 and Kinetics-600. These results demonstrate the effectiveness of our UniFormer for video.

Results on Something-Something. Table IV presents the results on Something-Something (SthSth) V1&V2. Since these datasets require robust temporal relation modeling, it is difficult for the CNN-based methods to capture long-term dependencies, which leads to their worse results. On the contrary, video transformers are good at processing long sequential data and demonstrate better transfer learning capabilities [122], thus they achieve higher accuracy but with large computation costs. In contrast, our UniFormer-S_16f combines the advantages of both convolution and self-attention, obtaining 54.4%/65.0% in SthSth V1/V2 with only 42 GFLOPs. It also demonstrates that small model UniFormer-S benefit from larger dataset pre-training (Kinetics-400, 53.8% vs. Kinetics-600, 54.4%), but large model UniFormer-B do not (Kinetics-400, 59.1% vs. Kinetics-600, 58.8%). We argue that the large model is easy to converge better. Besides, it is worth noting that our UniFormer pre-trained from Kinetis-600 outperforms all the current methods under the same settings. In fact, our best model achieves the new state-of-the-art results: 61.0% top-1 accuracy on SthSth V1 (4.2% higher than TDN_EN) [97] and 71.2% top-1 accuracy on SthSth V2 (1.6% higher than Swin-B [62]). Such results verify its high capability of spatiotemporal learning.

Results on UCF101 and HMDB51. We further verify the generalization ability on UCF101 and HMDB51. Since these datasets are relatively small, the performances have already saturated. As shown in Table VI, our UniFormer significantly outperforms the previous SOTA methods, revealing its strong generalization ability to transfer to small datasets.

Settings. We benchmark our models on object detection and instance segmentation with COCO2017 [60]. The ImageNet-1K pre-trained models are utilized as backbones and then armed with two representative frameworks: Mask R-CNN [6] and Cascade Mask R-CNN [36]. Our codes are mainly based on mmdetection [13], and the training strategies are the same as Swin Transformer [61]. We adopt two training schedules: 1$\times$ schedule with 12 epochs and 3$\times$ schedule with 36 epochs. For the 1$\times$ schedule, the shorter side of the image is resized to 800 while keeping the longer side no more than 1333. As for the 3$\times$ schedule, we apply the multi-scale training strategy to randomly resize the shorter side between 480 to 800. Besides, we use AdamW optimizer with the initial learning rate of 1e-4 and weight decay of 0.05. To regularize the training, we set the stochastic depth drop rates to 0.1/0.3 and 0.2/0.4 for our small/base models with Mask R-CNN and Cascade Mask R-CNN.

Results. Table V reports box mAP ($AP^{b}$) and mask mAP ($AP^{m}$) of the Mask R-CNN framework. It shows that our UniFormer variants outperform all the CNN and Transformer backbones. To reduce the training cost for object detection, we utilize a hybrid UniFomer style with a window size of 14 in Stage3 (denoted by $h14$). Specifically, with 1$\times$ schedule, our UniFormer brings 7.0-7.6 points of box mAP and 6.7-7.2 mask mAP against ResNet [37] at comparable settings. Compared with the popular Swin Transformer [61], our UniFormer achieves 2.6-3.4 points of box mAP and 2.2-2.5 mask mAP improvement. Moreover, with 3$\times$ schedule and multi-scale training, our models still consistently surpass CNN and Transformer counterparts. For example, our UniFormer-B outperforms the powerful CSwin-S [23] by +0.3 box mAP and +0.3 mask mAP, and even better than larger backbones such as Swin-B and Focal-B [109]. Table VII reports the results with the Cascade Mask R-CNN framework. The consistent improvement demonstrates our stronger context modeling capacity.

Settings. Our semantic segmentation experiments are conducted on the ADE20k [121] dataset and our codes are based on mmseg [19]. We adopt the popular Semantic FPN [47] and Upernet [104] as the basic framework. For a fair comparison, we follow the same setting of PVT [99] to train Semantic FPN for 80k iterations with cosine learning rate schedule [66]. As for Upernet, we apply the settings of Swin Transformer [61] with 160k iteration training. The stochastic depth drop rates are set to 0.1/0.2 and 0.25/0.4 for small/base variants with Semantic FPN and Upernet respectively.

Results. Table VIII and Table IX report the results of different frameworks. It shows that with the Semantic FPN framework, our UniFormer-S_h32/B_h32 achieve +4.7/+2.5 higher mIoU than the Swin Transformer [61] with similar model sizes. When equipped with the UperNet framework, they achieve +2.5/+1.9 mIoU and +2.7/+1.2 MS mIoU improvement. Furthermore, when utilizing the global MHRA, the results are consistently improved but with a larger computation cost. More results can be found in Table XXII.

Settings. We evaluate the performance of UniFormer on the COCO2017[60] human pose estimation benchmark. For a fair comparison with previous SOTA methods, we employ a single Top-Down head after our backbones. We follow the same training and evaluation setting of mmpose [18] as HRFormer [114]. In addition, the batch size and stochastic depth drop rates are set to 1024/256 and 0.2/0.5 for small/base variants during training.

Results. Table X reports results of different input resolutions on COCO validation set. Compared with previous SOTA CNN models, our UniFormer-B surpasses HRNet-W48 [95] by 0.4% AP with fewer parameters (53.5M vs. 63.6M) and FLOPs (22.1G vs. 32.9G). Moreover, our UniFormer-B can outperform the current best approach HRFormer [114] by 0.2% AP with smaller FLOPs (29.6G vs. 30.7G). It is worth noting that HRFormer [114], PRTR [51], TokenPose [55] and TransPose [110] are sophisticatedly designed for pose estimation task. On the contrary, our UniFormer can outperform all of them as a simple yet effective backbone.

Settings. For the light-weight UniFormer, we follow most of the previous settings. Differently, we train UniFormer-XSS and UniFormer-XS for 600 epochs on ImageNet, since the lightweight models are difficult to converge [35, 14]

Results of classification. Table XI represents the results on ImageNet. We roughly divide the models according to the FLOPs: $<$1G and 1G$-$2G. It clearly reveals that our efficient UniFormer achieves the best accuracy-throughput trade-off under similar FLOPs. For example, compared with the strong CNN method EfficientNet-B3[85], our UniFormer-XS₁₉₂ obtains 1.7$\times$ higher throughput with similar performance. Compared with SOTA MobileFormer[14] that combines CNN and ViT, our UniFormer-XXS₁₉₂ obtains 0.6% higher accuracy with 16% higher throughput. We further fine-tune the above models with different frames on Kinetics-400. Results in Table XII show that our efficient backbone surpasses the SOTA lightweight video backbones by a large margin. Compared with MoViNet-A0[48], our UniFormer-XXS_150×16f achieves 9.3% higher performance with 16% higher throughput. While compared with X3D-S[29], our UniFormer-XS_192×32f runs 4.2$\times$ faster with 5.4% higher accuracy. Note that we do not apply complicated designs as in the recent lightweight methods[14, 69, 48]. Our concise extension already shows powerful performance, which further demonstrates the great potential of UniFormer.

Results of dense prediction. We also verify the efficient UniFormer for COCO object detection and instance segmentation in Table XIII, and ADE20K semantic segmentation in Table XIV. Our models obviously beat ResNet[37] and PVTv2[100] on these dense prediction tasks. For example, our UniFormer-XS brings 2.7 points of box mAP and 2.1 mask mAP against PVTv2-B1 on COCO, and achieves +1.9 mIoU improvement on ADE20K.

To inspect the effectiveness of UniFormer as the backbone, we ablate each key structure design and evaluate the performance on image and video classification datasets. Furthermore, for video backbones, we explore the vital designs of pre-training, training and testing. Finally, we demonstrate the efficiency of our adaption on downstream tasks, and the effectiveness of H-UniFormer.