Grounding Commands for Autonomous Vehicles via Layer Fusion
with Region-specific Dynamic Layer Attention

Language grounding for autonomous vehicles [3] is an important task for human-vehicle interactions. To tackle this task, Vandenhende et al. [12] propose the C4AV-Base model that utilizes the bi-directional GRU [13] and ResNet models [14] to encode the command and the regions respectively and then compute their feature correlation. Later, the ASSMR method [15] applies an attention mechanism to strengthen the features extracted by bi-directional GRU and ResNet. To enhance the reasoning ability, the MSRR method [16] introduces a spatial memory module and a multi-step reasoning module to iteratively score the regions. Other methods are built on the Transformer [6]. The CMTR method [17] applies the Transformer encoder-decoder model to model the command and regions separately. Dai et al. [5] use the VL-BERT pre-trained model [18] to jointly learn cross-modal representations for the input. They propose an iterative stacking algorithm, Stack-VL-BERT, to train a deeper VL-BERT model. This method only uses the top layer representations of a pre-trained model to compute matching scores, while our approach fuses all encoder layers in a pre-trained model.

The referring expression comprehension task aims to predict a region in an image according to a direct description of an object [11], e.g., “girl on the left”. In contrast, the command expressions in the language grounding for autonomous vehicles task are more complex and involve both action and object descriptions. Earlier methods [19, 20, 21, 22, 23] are built on CNN [24] and LSTM [25] models to encode the input regions and referring expression. Recent methods [4, 10, 18] use V&L pre-trained models based on the Transformer architecture and they achieve state-of-the-art performance. These methods only use the representations in the last layer of a pre-trained model to predict the matching scores, while our method fuses the representations across all encoder layers.

Several methods [26, 27, 28] allow robots to ask clarification questions and they rely on a semantic parser [27, 28] or a probabilistic model [26] to decompose a command and then ground the resulting constituents to visual regions. Shridhar and Hsu [29] propose a robot system that picks up an object according to a command. Their method generates a caption for each region and then clusters the generated captions with the input command. Kim et al. [30] introduce a vehicle controller that accepts advice from humans. Their model uses LSTM, CNN, and visual attention [31] to ground an advice to an image. In contrast, our approach is built on V&L pre-trained models to ground a command to a visual scene.

Encoder layer fusion techniques can be categorized into layer aggregation [32, 33, 34], layer-wise coordination [35], and layer attention [7, 9]. Layer-wise coordination modifies the structure of Transformer and cannot be easily applied to pre-trained models. Some layer aggregation methods [33] dynamically fuse the representations for each input word, but they do not provide normalized weights to interpret the importance of each layer and they introduce millions of new parameters. Layer attention utilizes normalized attention weights to fuse encoder layers and allows us to directly interpret the contribution of each layer. Existing layer attention methods assign static attention weights to layers independent of the input, whereas our RSD layer attention dynamically predicts attention weights for every input region.

As a preprocessing step, we obtain the region proposals $\{v_{i}\}_{i=1}^{n}$ of the input image from the Centernet model [36]. We use the WordPieces tokenizer of BERT [37] to tokenize the input command into a sequence of tokens $w_{1},\ldots,w_{m}$. A V&L pre-trained model takes the region proposals and command tokens as input. Inside the pre-trained model, an embedding layer first converts the input tokens and regions into a sequence of word embeddings, $\mathbf{h}_{w_{1}}^{0},\ldots,\mathbf{h}_{w_{m}}^{0}$, and region embeddings, $\mathbf{h}_{v_{1}}^{0},\ldots,\mathbf{h}_{v_{n}}^{0}$, respectively. We refer to the embedding layer as the 0-th encoder layer of the pre-trained model.

Then, the model feeds the embeddings to a Transformer encoder with $L$ layers. At each layer from $1$ to $L$, the encoder uses the multi-head attention mechanism [6] to model intra-modal and/or cross-modal interactions among the input to produce a $d$-dimensional contextualized representation vector for each region and token. We use $\mathbf{h}_{v_{i}}^{l}$ to denote the contextualized representation of region $v_{i}$ produced by the $l$-th encoder layer. Several studies [7, 8] reveal that lower encoder layers extract more concrete features (e.g, position) while higher encoder layers extract more abstract features (e.g., coreference relation). The model is pre-trained on a massive dataset to learn visual and linguistic knowledge.

There are two major classes of V&L pre-trained models: (1) single-stream architecture, which models both intra-modal and cross-modal interactions in every encoder layer; and (2) dual-steam architecture, which only allows intra-modal interactions in early encoder layers and then encodes both intra-modal and cross-modal interactions in latter layers. We apply the UNITER [4] model from single-stream architecture and LXMERT [10] from dual-stream architecture because they achieve excellent performance in many V&L tasks [38]. On the other hand, SOTA methods [5] in this task uses another single-stream model, VL-BERT [18]. Their model obtains lower scores than UNITER and LXMERT (see Table I for the results).

Prior methods feed the top layer representation of a region $\mathbf{h}_{v_{i}}^{L}$ to a linear layer to predict its matching score. To effectively exploit the concrete and abstract features embedded in different layers, we propose a novel RSD layer attention mechanism to fuse the encoder representations across layers before the prediction of matching scores. The overall idea is to dynamically assign attention weights to all encoder layers based on a region’s representation vectors in these layers. The weights are then used to aggregate the representation vectors across layers. Our intuition is that every region needs distinct types of features to decide whether it is referred to by the command.

More concretely, for every region proposal $v_{i}$, our RSD layer attention method passes its representation $\mathbf{h}_{v_{i}}^{l}$ at each encoder layer $l$ into a linear layer to compute a relevance score $\alpha^{l}_{i}$. Then we use the softmax function to normalize the relevance scores over all encoder layers and obtain the layer attention weights for region $v_{i}$, as shown in the following equations:

$\displaystyle\alpha_{i}^{l}=\mathbf{W}_{\alpha}\mathbf{h}_{v_{i}}^{l}+b_{\alpha}\text{,}\quad a_{i}^{l}=\frac{\text{exp}(\alpha_{i}^{l})}{\sum_{l^{\prime}=0}^{L}\text{exp}(\alpha_{i}^{l^{\prime}})}\text{,}$

(1)

where $a_{i}^{l}$ denotes the layer attention weight at layer $l$ by region $v_{i}$ and indicates the importance of the $l$-th layer to region $v_{i}$. In contrast, existing layer attention methods [7, 9, 39] only allocate fixed attention weights to encoder layers regardless of the model input. We then fuse the representations at different layers by weighted sum: $\tilde{\mathbf{h}}_{v_{i}}=\sum_{l=0}^{L}a_{i}^{l}\mathbf{h}_{v_{i}}^{l}$. Our RSD layer attention is parameter-efficient since it only introduces $d+1$ new parameters, where $d=768$ for the UNITER and LXMERT models.

Finally, we feed the fused encoder representation $\tilde{\mathbf{h}}_{v_{i}}$ of each region $v_{i}$ to a linear output layer to predict a matching score: $s_{i}=\sigma(\mathbf{W}_{s}{\tilde{\mathbf{h}}}_{v_{i}}^{L}+b_{s})$, where $\mathbf{W}_{s}\in\mathbb{R}^{d\times 1}$ and $b_{s}\in\mathbb{R}$, $\sigma$ is the sigmoid function that normalizes the output to the range of $[0,1]$. Following previous work [40], we use the intersection-over-union (IoU) score between a region $v_{i}$ and the ground-truth region as the ground-truth matching score $s^{*}_{i}$. The IoU score of a predicted region is the intersection between that region and the ground-truth region divided by the union of them. The training objective is the binary cross-entropy loss: $s^{*}_{i}\log\sigma(s_{i})+(1-s^{*}_{i})\log(1-\sigma(s_{i}))$. During inference, we use the region proposal that has the highest predicted matching score as the model output.

To verify the importance of region-specific attention weights in our method, we make an ablation in our RSD layer attention to construct a baseline called sample-specific layer attention. Instead of predicting layer attention weights for each individual region, we compute layer attention weights for the entire input sample using the mean-pooled representation of the regions. Then, each region in the sample uses the same set of attention weights to fuse encoder layers.

Specifically, at each encoder layer $l$, we perform mean-pooling over the representations of all regions: $\mathbf{h}^{l}=1/n\sum_{i=1}^{n}\mathbf{h}_{v_{i}}^{l}$. Next, we feed the mean-pooled representation $\mathbf{h}^{l}$ to a linear layer to learn a relevance score. All the relevance scores are then normalized by softmax to yield the layer attention weights shared by all the regions.

We use the Talk2Car [3] dataset to conduct our experiments. Talk2Car is the standard benchmark for the language grounding for autonomous vehicles task. The images are urban scenes captured by cameras on a car. Each input text is a natural language command referring to a particular object in the input image, e.g., “get a parking spot near the second car on the left side”. A command expression contains both an action instruction and an object description. The averaged length of expressions is 11.0. The training, validation, and test splits contain 8,349/1,163/2,447 samples.

We further evaluate our method on the RefCOCO+ [11] and RefCOCOg [19] datasets, which are popular benchmarks for referring expression comprehension. The input text in these two datasets is a description of an object in the image (e.g., “giraffe with lowered head”) rather than a command. The average expression length of RefCOCO+ and RefCOCOg datasets are 3.5 and 8.4 respectively. The images in these datasets are collected from the MSCOCO benchmark [41]. For RefCOCO+, we divide the data according to Bugliarello et al. [40] and obtain the split of 287,113/13,368/11,490 for training, validation, and test. For RefCOCOg, we use the UMD split of 42,226/2,573/5,023.

During evaluation, we compute the IoU score of the predicted region. If the IoU score is larger than 0.5, we consider the prediction correct. Following [3], we report the averaged number of correct predictions and refer to it as the IoU_0.5 score.

We adopt recent methods of language grounding for autonomous vehicles as baselines, including C4AV-Base [12], ASSMR [15], MSRR [16], CMTR [17], and Stack-VL-BERT [5]. Moreover, we consider the UNITER [4] and LXMERT [10] models as baselines. For the UNITER model, we adopt the base model with 12 layers. Furthermore, we compare with previous layer fusion methods: coarse-grained layer attention [7, 9], which assigns static attention weights to encoder layers, fine-grained layer attention [39], which assigns static attention weights to the elements of the representations in all layers, dynamic combination [33], which uses $L$ feedforward networks to aggregate the representations in different layers, dynamic routing [33], which iteratively refines the fused representation based on the agreement between each layer representation and the fused representation. We use RSD-UNITER and RSD-LXMERT to denote the UNITER and LXMERT models after being enhanced by our layer fusion approach.

We report the main comparison results on the Talk2Car dataset in Table I. It is observed that our layer fusion approach significantly and consistently improves the IOU_0.5 scores of both LXMERT and UNITER models in validation and test sets, which show that our RSD layer attention can increase the accuracy of both single-stream and dual-stream V&L pre-trained models. The reason is that our approach is agnostic to the architecture of the underlying pre-trained model. Moreover, both of our methods (RSD-LXMERT and RSD-UNITER) outperform the state-of-the-art models in this benchmark. The above results indicate that it is important to effectively utilize the features embedded in different layers of a V&L pre-trained model to locate the referred region.

We compare the performance of different encoder layer fusion methods on UNITER since it is the most accurate pre-trained model as shown in the previous section. From Table II, we observe that our RSD layer attention mechanism achieves better performance than existing layer fusion methods. Moreover, we can see that our constructed baseline, sample-specific layer attention, obtains lower IoU_0.5 scores than our RSD layer attention, which indicate that it is crucial to learn a distinct set of layer attention weights for each visual region. Surprisingly, our RSD layer attention substantially outperforms previous dynamic layer fusion methods (rows 4&5) that introduce millions of new parameters. The results suggest that the combination of region specific layer attention weights and weighted sum fusion operation is effective in this task. Previous dynamic aggregation methods may introduce unnecessary parameters which make the model more difficult to train.

We analyze the distribution of layer attention weights learned by our RSD layer attention for different regions. We partition the input regions into the following two groups according to their IoU with the ground-truth region. $\text{IoU}>0$: contains all the regions that have IoU greater than 0 (i.e., the regions that overlap the ground-truth region). $\text{IoU}=0$: includes all the regions with IoU equals to 0. Figure 3 shows the distribution of layer attention weights in two groups of regions. We draw the following two observations.

• •

  Both models allocate a larger proportion of attention weights to higher encoder layers. It is because the representations in higher encoder layers are exposed to more cross-modal interactions. Thus, they usually capture more cross-modal semantic features which are essential to command and scene understanding.

• •

  Compared with the regions with $\text{IoU}=0$, the regions with $\text{IoU}>0$ assign more attention weights to lower encoder layers. It is because most of the regions with $\text{IoU}=0$ are obviously irrelevant to the command. Hence, the abstract features embedded in higher encoder layers can provide sufficient information for the model to predict a matching score of zero. On the other hand, the regions with $\text{IoU}>0$ overlap the target region and they are more difficult to be disambiguated. Hence, the model requires more surface-level features (e.g., position) of the regions in lower encoder layers to decide their matching scores.

To illustrate how our layer fusion method results in more accurate predictions, we qualitatively analyze the representations of visual regions. We first collect the regions’ representations learned by our RSD-UNITER model. We then utilize the Principal Component Analysis (PCA) [42] technique to project the representation vectors into a 2-dimensional vector space for the sake of visualization.

Figure 4 visualizes the regions’ representations learned by our RSD-UNITER model. The ground-truth region’s representation is denoted by a red star. From Figure 4(a), we observe that in the top layer, the ground-truth region’s representation is far away from the representations of regions that do not overlap the ground-truth ($\text{IoU}=0$). Previous methods feed the top layer representation of a region to a linear layer to predict its matching score. However, the ground-truth region is extremely close to a region that has intersection with the ground-truth ($0<\text{IoU}\leq 0.5$). It is difficult for a linear layer to separate the target region from its closest neighbor. Then in Figure 4(b), we observe that after applying our RSD layer attention to fuse the representations across layers, the ground-truth region’s representation is pushed slightly further from its closest neighbor. Thus, a linear layer can separate out the ground-truth region more easily, which demonstrates the advantage of our approach.

We further evaluate the performance of our approach on the referring expression comprehension task. Table III presents the results in the RefCOCO+ and RefCOCOg datasets. We observe that our approach increases the performance of UNITER and LXMERT models in most of the cases but the improvements are less significant than that in the Talk2Car dataset. We analyze the reason as follow. The command expressions in the autonomous driving setting are complicated and contain both action and object descriptions. The model requires features from multiple layers of cross-modal representations to learn the alignment between text and image. Thus, our encoder layer fusion approach significantly improves the accuracy of predicted regions. On the other hand, the referring expressions in RefCOCO+ and RefCOCOg datasets are straight-forward descriptions of a target object. Hence, the top layer representations often capture enough information to find the target object and encoder layer fusion provides a smaller benefit to the model.