LCTR: On Awakening the Local Continuity of Transformer for
Weakly Supervised Object Localization

In accordance with the long-range info-preserving ability of the transformer architecture, LCTR is designed to offer precise localization maps for WSOL. We denote the input images as $I=\{(I_{i},y_{i})\}_{i=0}^{M-1}$, where $y_{i}\in\{0,1,\dots,C-1\}$ indicates the label of the image $I_{i}$, $M$ and $C$ are the number of images and classes, respectively. We first split $I_{i}$ into $N$ same-sized patches $x_{p}\in\mathbb{R}^{1\times D}$, where $D$ denotes the dimension of each patch. We set $N=w\times h$, $w=W/P$ and $h=H/P$, where $P$ is the width/height of a patch, $H$ and $W$ denote image height and width. For simplicity, we omit the mini-batch dimension. A learnable class token $x_{cls}\in\mathbb{R}^{1\times D}$ is embedded into the patches. These patches are flattened and linearly projected before being fed to $L$ sequential transformer blocks, which can be formulated as:

$$X_{1}=[x_{cls};\mathcal{F}(x_{p}^{1});\mathcal{F}(x_{p}^{2});\cdots;\mathcal{F}(x_{p}^{N})]+\mathcal{P},$$

(1)

where $X_{1}$ denotes the input of the first transformer block, $\mathcal{P}\in\mathbb{R}^{(N+1)\times D}$ is the position embedding and $\mathcal{F}$ is a linear projection. In particular, the proposed RPAM is employed in each transformer block to obtain a patch relation map $M^{r}_{*}$, which aggregates cross-patch information on a global basis.

Denote ${X_{L}}\in\mathbb{R}^{N\times D}$ as the output feature of the last transformer block. We reshape ${X_{L}}\in\mathbb{R}^{D\times w\times h}$ and apply the proposed CDM for further highlighting weak local responses. After that we obtain the feature map $\mathrm{X}_{CDM}\in\mathbb{R}^{C\times w\times h}$. Finally, the $\mathrm{X}_{CDM}$ are fed to a global average pooling (GAP) layer (Lin, Chen, and Yan 2013) followed by a softmax layer to predict the classification probability $p\in\mathbb{R}^{1\times C}$. The loss function is defined as

$$\mathcal{L}=-\log p.$$

(2)

During testing, we extract the object map ${M_{CDM}}\in\mathbb{R}^{w\times h}$ from $\mathrm{X}_{CDM}$ according to the predicted class and obtain the final localization map by element-wise multiplication, given as

$$M^{fuse}=M_{CDM}\otimes M^{r}_{*}.$$

(3)

The $M^{fuse}$ is then resized to the same size as the original images by linear interpolation. For a fair comparison, we apply the same strategy detailed in CAM (Zhou et al. 2016) to produce the object bounding boxes.

The proposed relational patch-attention module (RPAM) (Figure 3) strengthens the global feature representation from two stages: First, we utilize the attention vectors of the class token in the transformer block to generate a global class-token attention map. To fully exploit the feature dependencies of the transformer structure, we then use all the attention vectors of the patches containing the correlation between local features to generate a patch relation map under the guidance of the class-token attention map.

In the $l$-th transformer block, we hypothesize that the output feature map is ${X_{l}}\in\mathbb{R}^{(N+1)\times D}$. The attention matrix ${A_{l}}\in\mathbb{R}^{S\times(N+1)\times(N+1)}$ of multi-head self-attention module in the block is formulated as:

$${A_{l}}=\mathrm{Softmax}\left(\frac{\bm{Q}_{l}\cdot\bm{K^{\top}}_{l}}{\sqrt{D/S}}\right),$$

(4)

where $\bm{Q}_{l}$ and $\bm{K}_{l}$ denote the queries and keys projected by ${X_{l}}$ of self-attention operation in $(l-1)$-th transformer block, respectively. $S$ represents the number of head and $\top$ is a transpose operator.

At this point, we first take the average operator to ${A}_{l}$ based on $S$ heads to obtain ${A^{\prime}_{l}}\in\mathbb{R}^{(N+1)\times(N+1)}$. Then, the class-token attention vector $M^{c}_{l}\in\mathbb{R}^{1\times(N+1)}$ is extracted from ${A^{\prime}_{l}}$. The $M^{c}_{l}$ reveals how much each patch contributes to the object regions for image classification. Unfortunately, this map simply captures the global interactions of the class token to all patches, while ignoring the cross-patch correlations, which affects the modeling of local features. To remedy it, we take advantage of the patch attention map $M^{p}_{l}\in\mathbb{R}^{(N+1)\times N}$ in ${A^{\prime}_{l}}$ to structure a patch relation vector $M^{r}_{l}$ under the guidance of $M^{c}_{l}$. The $M^{p}_{l}$ learns the correlation between each patch but couldn’t tell which one is more important. Therefore, we weight each patch attention map by multiplying $M^{p}_{l}$ by $M^{c}_{l}$ to obtain a new map $M^{h}_{l}\in\mathbb{R}^{(N+1)\times N}$. Note that $M^{c}_{l}$ is reshaped ($\mathbb{R}^{(N+1)\times 1}$) before the multiplication. After that, we squeeze the first dimension of $M^{h}_{l}$ to a vector ($M^{r}_{l}\in\mathbb{R}^{1\times N}$) by an average operation. The final patch relation map $M^{r}_{\ast}$ is calculated by

$$M^{r}_{\ast}=\Gamma^{w\times h}(\frac{1}{L}\sum_{l}M_{l}^{r}),$$

(5)

where $\Gamma^{w\times h}(\cdot)$ indicates the reshape operator which coverts the vector ($\mathbb{R}^{1\times N}$) to the map ($\mathbb{R}^{w\times h}$).

The patch relation map $M^{r}_{\ast}$ obtains the long-range dependencies that depends on the class-token attention vector $M^{c}_{l}$. Aggregating cross-patch information from the patch attention maps, $M^{r}_{\ast}$ facilitates better global representations of the object without extra parameters in a simple way.

RPAM considers the cross-patch information by using the class-token attention map from self-attention mechanism block of the transformer structure, but it is vulnerable if the transformer gets a poor class-token attention map. We thus further propose a cue digging module (CDM) to supply the long-range features based on a hide-and-seek manner.

Inspired by erasing-based methods that remove the most discriminative parts of the target object to induce the model to cover the integral extent of the object, we erase the object regions based on the global class-token attention map, leaving the weak response ones and the background. Then by weighting the learnable convolution kernels in the CDM on the basis of them, we shift part of the attention to object regions with weak responses. With the help of weighted kernels, we can highlight the local details as a supplement to the global representations.

Specifically, we convert the patch parts of class-token attention vectors to the map $\tilde{M}^{c}_{*}\in\mathbb{R}^{w\times h}$, and apply it to the feature map ${X_{L}}$ by spatial-wise multiplication after being reversed. Note that $\tilde{M}^{c}_{*}$ is calculated by $\tilde{M}^{c}_{*}=\frac{1}{L}{\textstyle\sum_{l}}M^{c}_{l}$. The feature map then passes through a convolutional layer to generate a new feature map ${\tilde{X}_{L}}\in\mathbb{R}^{D\times w\times h}$. Next, we score the features into $G$ parts corresponding to $G$ learnable convolution kernels. In particular, we apply two separate operators, the global average pooling and max pooling, to ${\tilde{X}_{L}}$. Then the feature maps are vectorized and sent to a fully connected layer, respectively. Besides, we add them together and apply a sigmoid function. In this light, we obtain the scores as $\{S_{g}\mid g=1,2,\dots,G\}$. Finally, ${X_{L}}$ passes through a convolutional layer with weighted convolution kernels by the scores $S_{g}$ for encouraging the model to learn the object regions with weak responses, which can be formulated as

$${X_{SGR}}={X_{L}}\sum_{g=1}^{G}S_{g}W^{e}_{g},$$

(6)

where $W^{e}_{g}\in\mathbb{R}^{D\times C\times k_{w}\times k_{h}}$ denotes the kernel weights of the convolutional layer, which are initialized with kaiming uniform initialization (He et al. 2015). $k_{w}$, $k_{h}$ represent the width and height of the kernels, respectively.

The convolution layer with the weighted kernels is applied to the global feature map $\mathrm{X}_{L}$ drawn from the transformer structure. Once the loss $\mathcal{L}$ in Eq. 2 is optimized, the weighted convolution kernels become more sensitive to the features (i.e., the weak response features of object regions) that favor image classification. In this way, the model pays more attention to local cues and forms better global representations for the target object.

Datasets. We evaluate the proposed methods on two challenging datasets, including CUB-200-2011 (Wah et al. 2011) and ILSVRC (Russakovsky et al. 2015). We only use image-level labels for training. CUB-200-2011 is a fine-grained bird dataset of $200$ categories, which contains $5,994$ images for training and $5,794$ for testing. ILSVRC has about $1.2$ million images in the training set and $50,000$ images in the validation set, with a total of $1,000$ different categories.

Evaluation Metrics. Following previous methods (Zhou et al. 2016; Russakovsky et al. 2015), we adopt the Top-1/Top-5 classification accuracy (Top-1/Top-5 Cls.), Top-1/Top-5 localization accuracy (Top-1/Top-5 Loc.) and localization accuracy with known ground-truth class (Gt-k.) as our evaluation metrics. Specifically, Top-1/Top-5 Cls. is correct if the Top-1/Top-5 predicted category contains the correct label. Gt-k. is correct when the intersection over union (IoU) between the ground-truth and the prediction is larger than $0.5$, and does not consider whether the predicted category is correct. Top-1/Top-5 Loc. is correct when Top-1/Top-5 Cls. and Gt-k. are both correct.

Implementation Details. We adopt the Deit (Touvron et al. 2021) as the backbone network, which is pre-trained on ILSVRC (Russakovsky et al. 2015). Particularly, we replace the MLP head with our proposed CDM. Finally, a GAP layer and a softmax layer are added on the top of the convolutional layers. The input images are randomly cropped to $224\times 224$ pixels after being resized to $256\times 256$ pixels. We adopt AdamW (Loshchilov and Hutter 2017) with $\epsilon$=$1e$-$8$, $\beta_{1}$=$0.9$, $\beta_{2}$=$0.99$ and weight decay of $5e$-$4$. On CUB-200-2011, we use a batch size of $128$ with a learning rate of $5e$-$5$ to train the model for 80 epochs. For ILSVRC, the training process lasts $14$ epochs with a batch size of $256$ and a learning rate of $5e$-$4$. After meticulous experiments, we set $G$=4 in the CDM. All the experiments are performed with four Nvidia Tesla V100 GPUs using the PyTorch toolbox.

Localization. We first compare the proposed LCTR with the SOTAs on the localization accuracy on the CUB-200-2011 test set, as illustrated in Table 1. We observe that LCTR outperforms the baseline (i.e., TS-CAM (Gao et al. 2021)) by 7.9% in terms of Top-1 Loc., and is obviously superior to these CNN-based methods. Besides, Table 2 illustrates the localization accuracy on the ILSVRC validation set. It reports 0.4% performance improvement over the state-of-the-art SLT-Net (Guo et al. 2021).

Classification. Table 3 and Table 3 show the Top-1 and Top-5 classification accuracy on the CUB-200-2011 test set and ILSVRC validation set, respectively. For the fine-grained recognition dataset CUB-200-2011, LCTR achieves remarkable performance of 85.0%/97.1% on Top1/Top-5 Acc.. In addition, LCTR obtains comparable results with SLT-Net (Guo et al. 2021) on Top-1 Acc. and surpasses other methods significantly on the ILSVRC validation set. Note that SLT-Net used a separated localization-classification framework, it cannot retain the global information for the objects in the individual classification network. To sum up, the proposed LCTR can greatly improve the quality of object localization while keeping high classification performance.

Visualization. For qualitative evaluation, Figure 4 visualizes the final localization results of CAM (Zhou et al. 2016) based on the CNNs, TS-CAM (Gao et al. 2021) based on the transformer and our method on CUB200-2011 and ILSVRC datasets. From the results, compared with the CAM, we consistently observe that our method can cover a more complete range of object regions instead of focusing only on the most discriminative ones. In addition, we capture more localized cues than the TS-CAM method, resulting in more accurate localization. For example, the tail regions of the Bank Swallow and the Colobus are ignored by CAM and TS-CAM methods, while our LCTR is able to aggregate more detailed features of the target object, which enhances the local perception capability of global features among long-range feature dependencies. Please refer to the supplementary materials for more visualized localization results of our method.

First, we visualize the localization maps with different settings in Figure 5. We observe that the RPAM strengthens the global representations of the baseline (Gao et al. 2021), e.g., the tail-feature response of the African chameleon is enhanced, as it considers more cross-patch information. When only using CDM, we find that the local feature details are mined. For example, the abdominal features of the Black grouse are further activated compared to the baseline. By applying both RPAM and CDM, the final localization map (Figure 5 (e)) highlights the full object extent.

Next, we explore the concise design of the CDM. From the results on Table 5, we can observe that the mode of using separate fcs with GAP and GMP reports the best performance. These results also verify that GAP and GMP work differently in the CDM. Then, we evaluate the accuracy under different parameters $G$ in the CDM, as shown in Table 6. From the experimental results, we observe that the best performance is achieved when $G=4$. Setting a larger $G$ leads to a larger number of parameters and degrades accuracy, which we believe is caused by overfitting. Besides, we examine the impact of the weighted kernel size (i.e., $k_{w}\times k_{h}$). Results shown in Table 7 indicate that a kernel size of $3\times 3$ yields better performance.

Lastly, we investigate the effect with different configurations on the accuracy, as reported in Table 8. On the CUB-200-2011 test set, we can see that RPAM increases the Top-1 Loc. by 0.9% compared with the baseline TS-CAM method. Note that the lightweight RPAM is directly applied in the test phase, so the classification performance remains unchanged. When applying the CDM to the network, we observe an improvement in both classification and localization performance. From this, we believe that the local cues captured by the CDM are important for both two tasks. The best localization/classification accuracy can be achieved when employing both RPAM and CDM. Meanwhile, we conduct the similar experiments on the ILSVRC validation set, which also validate the effectiveness of two modules, as shown in the lower part of Table 8.