ScanFormer: Referring Expression Comprehension by Iteratively Scanning

Most conventional methods [50, 15, 25, 2, 51] explore REC through a two-stage framework. Concretely, in the first stage, numerous candidate proposals for the input image are pre-generated using a pre-trained object detector [36]. In the second stage, the proposal that best matches the given referring expression is considered the referred target box. However, two-stage methods are constrained by the accuracy and speed of the object detector. To this end, one-stage methods [54, 29, 47, 46] based on dense anchors [35] are proposed, which can achieve faster speed and comparable performance to the two-stage methods. In recent years, the success of the transformer [42] in vision-language fields has attracted researchers, leading to the emergence of REC methods [48, 53, 55, 6, 7, 40] based on the transformer. Due to the multi-head attention mechanism [42], transformer-based REC methods can effectively capture cross-modal relationships. While achieving impressive performance, these methods incur additional computational overhead due to their dense perception of images. To this end, this paper proposes a coarse-to-fine iterative perception framework to enhance the efficiency of the model.

The self-attention mechanism [42] is the primary reason for the inefficiency of ViT [9], as its computational complexity grows quadratically with the number of visual tokens. Recently, several methods [45, 3, 52, 44, 1] have have emerged to improve the efficiency of ViT by reducing the number of computed visual tokens. These methods can be broadly categorized into bottom-up token merging methods [44, 52, 1] and top-down coarse-to-fine methods [45, 3]. To be specific, bottom-up token merging methods [44, 52, 1] initially divide the high-resolution image into fine-grained patches, and gradually merge these patches in multiple stages to reduce the number of visual tokens. In addition, top-down coarse-to-fine methods [45, 3] start with coarse-grained partitioning, i.e. large patch size but a small number of tokens, and progressively reduce the patch size while performing fine-grained partitioning. However, existing relevant methods focus on efficient vision Transformers, with a limited exploration into efficient vision-language Transformers. In this paper, we explore an efficient vision-language Transformer framework for REC.

ScanFormer utilizes a unified Transformer-like structure for linguistic and visual modalities, as illustrated in Fig. 2. Concretely, the framework consists of word embedding, patch embedding, position-scale embedding, and encoders. Word embedding and patch embedding extract features from texts and images, respectively. Position-scale embedding is used to encode the spatial position and scale size of each image patch. The encoder consists of $N$ layers, each comprising a Multi-Head Attention (MHA) layer and a Feed-Forward Network (FFN). In addition, each encoder layer is equipped with a cache to store the output features. The query for MHA comes from the input features, while the key and value are composed of features from the input features and the previous cached features, as illustrated in Fig. 3. The causality in scale not only reduces the amount of calculations but also leverages previous linguistic and multi-scale visual information to update features.

The input of linguistic modality is initially encoded by the framework, and the extracted linguistic features are stored in the cache. Subsequently, for the visual modality, an image scale pyramid with $S$ scales is constructed based on the input image $I$. From top to bottom, for each iteration, selected patches are extracted and processed through the framework, where intermediate features are used to generate the selection of sub-patches in the next pyramid layer. In addition, the cache at each layer of the encoder stores the visual features obtained after each iteration. The features corresponding to the [REG] token in each iteration are used to predict the coordinates of the referred object at the corresponding scale. In particular, for the image at the top of the pyramid, all the patches are selected to ensure that the model captures global information. As the scale increases, ScanFormer incorporates finer-grained features to achieve accurate predictions, while discarding irrelevant patches to save substantial computing resources.

Specifically, for the linguistic modality, the referring text $t\in\mathbb{R}^{L\times|V|}$ is embedded with the word embedding matrix $T\in\mathbb{R}^{|V|\times d}$, prepended the [CLS] embedding $T^{cls}\in\mathbb{R}^{d}$, and then added with the text position embedding matrix $T^{pos}\in\mathbb{R}^{(L+1)\times d}$ and the type embedding $T^{type}\in\mathbb{R}^{d}$. The embedded linguistic features are first fed into the framework, and the updated linguistic features are stored in the cache at each layer of the encoder. For the visual modality, from top to bottom of the image scale pyramid, taking level-$i$ as an example, the $N_{i}$ selected patches with $(P,P)$ resolution and $C$ channels are first flattened to $v\in\mathbb{R}^{N_{i}\times(P^{2}\cdot C)}$ and then projected to $E\in\mathbb{R}^{N_{i}\times d}$ with a linear projection layer. After that, the patch features are added with the spatial embedding $E^{spatial}\in\mathbb{R}^{N_{i}\times d}$ and the type embedding $E^{type}\in\mathbb{R}^{d}$. $E^{spatial}$ is produced by the position-scale embedding $PSE:[0,1]^{3}\rightarrow\mathbb{R}^{d}$ with the normalized patch coordinates and scales $[cx,cy,s]$ as inputs. After that, The embedding of the [REG] token $E^{reg}\in\mathbb{R}^{d}$ is appended, which is used to regress the bounding box $[cx_{i},cy_{i},w_{i},h_{i}]$ of the object at level $i$.

To facilitate learning to select informative patches through back-propagation, a selection factor $s_{i}$ is generated for the $i$-th patch. There are two options for using $s_{i}$: (1) Apply $s_{i}$ to every head of the MHA on every Transformer layer. This is achieved by weighting the key and value, and gradually decaying $s_{i}$ to $0.0$ to minimize its impact on the remaining tokens. However, for a Transformer with $N$ layers and $H$ heads, obtaining clear gradient signals to optimize $s_{i}$ is challenging, making it difficult to achieve ideal learning choices. (2) Apply $s_{i}$ directly to the inputs of the Transformer, i.e. patch embedding, in a weighted manner. Since $s_{i}$ is only used at this location, it is easier to train. Therefore, this paper adopts the second candidate.

Furthermore, it is worth noting that even if the input patch embedding is set to zero, it still becomes non-zero in subsequent layers due to the bias terms of FFN and MHA, and the dot product attention. Fortunately, when the token sequence contains many identical tokens, the calculation of MHA can be simplified, leading to practical inference acceleration. To improve the model’s adaptability, the paper suggests replacing the patch embedding with a learnable constant token rather than directly setting it to zero. Therefore, the patch selection problem is transformed into a patch replacement problem. Next, the constant patch replacement and merging for acceleration will be introduced.

The referred object may exist at various scales. Similar to the object detection methods[36, 35], where multi-scale predictions are conducted at different feature levels, for each scale level in ScanFormer, we apply direct coordinate regression [6] to predict the bounding box of the referred object. The regression token [REG] is introduced to gather features of image patches across the Transformer. The output features corresponding to the [REG] token is fed to a shared multi-layer perception (MLP), followed by the Sigmoid function to predict the normalized bounding box $\hat{b}=(\hat{x},\hat{y},\hat{w},\hat{h})$ of the referred objects.

We optimize the proposed coarse-to-fine iterative perception framework end-to-end. For the $l$-th image scale, we can obtain predicted bounding box $\hat{b}_{l}=(\hat{x}_{l},\hat{y}_{l},\hat{w}_{l},\hat{h}_{l})$. Given the ground truth $b=(x,y,w,h)$, the detection loss function is defined as follows:

where $\mathcal{L}_{L1}(\cdot,\cdot)$ and $\mathcal{L}_{giou}(\cdot,\cdot)$ represent L1 loss and Generalized IoU loss [37], respectively, and $\lambda_{L1}^{l}$ and $\lambda_{giou}^{l}$ are the relative weights to control the detection loss penalty for the $l$-th image scale.

In addition, to control the sparsity of selected patches, we add a regularization loss function as follows:

where $\lambda_{sparse}$ represents the relative weights to control the sparsification penalty, and $s_{i}^{l}$ represents the selection factor for the $i$-th patch in Eq. 2 in the $l$-th image scale. $\beta^{l}$ is the hyperparameter to control the ratio of selected tokens from the $l$-th image scale. The total loss function of our ScanFormer is defined as follows:

The trained ScanFormer can strike a balance between accuracy and efficiency. The experimental analysis of the ScanFormer will be elaborated in Sec. 4.

To verify the effectiveness of the ScanFormer proposed in this paper, we compare with state-of-the-art methods on widely used datasets, i.e., RefCOCO [49], RefCOCO+ [49], RefCOCOg [30], and ReferItGame [19].

Concretely, we compare the performance of our ScanFormer with state-of-the-art REC methods, including two-stage methods [50, 15, 25, 2], one-stage methods [46, 47, 29, 54, 23], and transformer-based methods [6, 22, 41, 53, 55, 48, 4, 40]. The comparison results are shown in Tab. 1. It can be observed that our ScanFormer achieves a significant performance improvement compared to state-of-the-art one-stage method PLV-FPN [23] and two-stage method Ref-NMS [2]. Compared to state-of-the-art transformer-based methods, such as LADS [40], SeqTR [55], and Word2Pix [53], it can be found that the proposed ScanFormer also achieved good performance. In particular, compared to QRNet [48], our ScanFormer achieves comparable performance. In addition, unlike previous methods that use additional visual and linguistic backbones [12, 26, 14, 8], ScanFormer only utilizes a unified Transformer to achieve accurate language-to-vision localization.

We also compare ScanFormer with state-of-the-art large-scale pre-training methods, i.e. MDETR [18] and OFA [43], as shown in Tab. 2. Compared with training directly on specific datasets, large-scale pre-training greatly improves the performance of the model. In addition, ScanFormer achieves comparable performance to MDETR and OFA, but the unified Transformer structure is simpler.

Furthermore, we compare the performance and inference speed of state-of-the-art methods with our ScanFormer on the RefCOCO+ val set, as shown in Fig. 4. The inference speed is tested on 1080 Ti, and the proposed ScanFormer achieves a real-time inference speed of 34.9 FPS. Compared with state-of-the-art methods [48, 6, 55, 29, 54], we achieve high accuracy and fast inference speed, benefiting from the unified Transformer. In particular, compared to QRNet [48] with comparable performance, Scanformer achieves an inference speed of more than twice as fast.

Considering that our ScanFormer iteratively conducts visual perception from coarse granularity to fine granularity, we show the prediction results of the model at different scales, as shown in Figure Fig. 5. It can be observed that as the scale level increases, the performance of the model improves significantly, and better bounding box location can be predicted, as shown by the increasing IoU (Intersection over Union) values. The model can achieve acceptable performance even at the smallest scale. We also presented the distribution of IoU between predicted boxes and ground-truth boxes at different scales. It can be found that as the number of iterations increases, the upper and lower bounds of the iou distribution are significantly improved.

Although the gradually improved performance, it is not trivial to select a proper metric to decide when to stop iteration across the scale pyramids like [45]. We made a preliminary attempt by adding an extra branch sibling to the regression head to predict the exit metrics, e.g. IoU, GIoU, or the variance of the predicted bounding box [13]. The experimental results are not ideal, where the predicted exit metrics have a poor correlation with localization accuracy. So we left the early exit metric for further exploration.

We propose to reduce computational overhead by replacing not-selected tokens with constant tokens and then merging them. According to Fig. 6, massive tokens are merged in Scale 1 and Scale 2. The distributions in Scale 0 are not visualized as all the tokens are selected. There are 40 and 220 tokens replaced with constant tokens on average in Scale 1 and Scale 2, respectively. Token merging can remove 39 and 219 tokens by merging them into one. Therefore, token merging can increase speed and significantly reduce FLOPs. We also visualize the distributions of the number of selected tokens per sample in Fig. 7. For some samples in RefCOCOg [32], only 100 tokens are selected for predicting the localization. Relative to all 400 tokens, each sample has an average of 270 selected tokens participating in the calculation. In addition, causal attention across scales can further reduce computational overhead.

The qualitative results are shown in Fig. 8. It can be observed that our model can successfully locate the referred objects, and the regressed bounding boxes are gradually refined with the increasing scales. Furthermore, based on images reconstructed from selected patches, the model can leave regions with low texture or no relation to the reference representation at coarse scales.