Context-Aware Multi-Task Learning for
Traffic Scene Recognition in Autonomous Vehicles

Scene recognition has been a topic of considerable research in the fields of the vehicle surrounding environment perception due to its important role in a wide variety of ADAS and ADS applications. Early works employed hand-crafted feature-based methods [20, 21, 22, 23, 24, 25] to extract spatiotemporal information of the scene. However, most of the scene recognition systems based on hand-crafted features rely heavily on heuristic priors and low-level features, thus resulting in not robust to complex environments. Moreover, relatively insufficient traffic-scene datasets compared to general-scene datasets (e.g. Maryland [23], YUPENN [24] and SUN [26]) restricted the performance of the methods, especially in unconstrained settings.

Recently, with increasing research interests in traffic scene understanding, some large-scale datasets [27, 28, 29, 30, 19] have been published in the past few years. Besides, several methods [19, 30, 31, 32, 33] focus on the deep neural architectures, inspired by the great success of image classification. [33] developed a multi-label classification framework that is suitable for many mobile agents through short image descriptors. [19, 32] proposed large-scale datasets for the dynamic driving scene and designs multi-label architecture including event proposal network and resolution-adaptive mechanism so that showed better performance than the previous methods. Nevertheless, most of such methods have not been concentrated on explicitly extracting contextual information. This can lead to problems of catastrophic forgetting, where learned feature representations can gradually end up worse recognition performance.

In this work, our proposed model adopts a context-aware learning approach. Although numerous methods are using the deeper and wider CNN module for better performance, our approach differs in the view of explicitly defining the degree of contextual information, which has not fully addressed in existing methods.

Multi-task learning (MTL) addresses the problem of conducting parameter sharing across the tasks. For surveys of this field, we recommend reviews by [34, 35] on the basis of the proposed network structure that is most closely our work. As we add the tasks of numerous traffic situations for the human-like visual understanding of autonomous vehicles, the number of task-specific networks to optimize is proportional. Therefore, the importance of MTL multiple tasks at the same time is obviously growing because MTL reduces the computation needed for inference as well as convergence time for training. Existing deep learning-based MTL methods in computer vision [36, 37, 38, 39, 10, 40, 41] demonstrated the efficiency, highlighting the extensibility for future developments and applications. A few works [42, 43, 11] considered the multi-modal inputs where multi-task network commonly handles multiple types of input. None of these methods attempts to explicitly define contextual information from the relationship between latent space and input. Our work focused on calculating neural mutual information for calculating contextual information. It is worth noting that, to our best knowledge, our work may be the first work to apply contextual information estimation for MTL.

We focus on the problem of context-aware MTL using deep learning in traffic scene recognition. Then we have the multi-task training set $\mathbb{X}=\{x_{1},x_{2},...,x_{N}\}$ of $N$ frames where the task spaces $\mathbb{Y}=\{y^{1},y^{2},...y^{T}\}$ with the number of tasks $T$. We denote $x_{i}\sim p(x)$ as the random variables of input image $x_{i}$ on the $i-th$ frame. We consider each image sample to project into a latent representation $z\in\mathbb{Z}$ that is dynamically learned for a variety of tasks. To resolve this goal, we utilize a deep feature extraction function $E(\cdot;\theta_{e})$ : $\mathbb{X}\rightarrow\mathbb{Z}$, with encoder network parameters $\theta_{e}$. Our encoders are learned and run with weight sharing for all tasks, while decoders are defined separately for each task. We acquire the representation $z$ from input image $x$ by sampling through conditional probability $p_{\theta_{e}}(z|x)$ with set of parameters $\theta_{e}$.

Regarding multi-task decoders, let $D^{t}$ denotes the task-specific network for neural hypothesis class per task that takes latent space $z$ as input. In this paper, we create decoders for a total of 23 tasks, including 12 in Road Place, 4 in Weather, 2 in Road Surface, and 5 in Road Environment. The decoder $D^{t}$ is parameterized as follows: $D(\cdot|\theta_{t})$. For $i={1,...,N}$, the $i$th image has MTL training labels: $x_{i}=\{(y^{(t)})\}^{T}_{t=1}$. Specifically, we employ multi-objective optimization based on Pareto optimality for eliminating the task-wise weights, which is a computationally inexpensive way. The multi-objective optimization for MTL using a vector-valued cross-entropy loss $\hat{L}$ is

$$\begin{split}&\underset{\theta_{e},\theta_{1},...,\theta_{T}}{\min}L(\theta_{e},\theta_{1},...,\theta_{T})=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{T}\sum_{t=1}^{T}\hat{{L}}^{t}(D((E(x_{i};\theta_{e});\theta_{j}))).\end{split}$$

(1)

Although this objective is intuitively appealing, it often reveals the performance of MTL is not better than single-task learning on the multi-task dataset. To handle the issue, we propose to regularization for capturing contextual information in the region of latent representations, thus assisting MTL optimization.

To create the context-aware latent representation, we adopt an information-theoretically constraint using mutual information. As previously discussed, the maintenance of contextual information for each task is essential. We encourage our encoder to extract context-aware features from the input image. Hence, the resulting latent representations of encoder would contain abundant information of input while preserving task-specific information. We now propose a useful term that retains task-specific information and captures contextual information of input image. Generally $z_{t}\sim p_{t,\theta_{e}}$ defines as the random variables of representations from $t$-th task. Let rewrite the MTL objective formulation (1) limiting of mutual information $I(z_{t},x)$ value for all of the task:

$$\begin{split}\underset{\theta_{e},\theta_{1},...,\theta_{T}}{\min}&L(\theta_{e},\theta_{1},...,\theta_{T})=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{T}\sum_{t=1}^{T}\hat{L}^{t}(D((E(x_{i};\theta_{e});\theta_{t})))\\ &s.t.\quad I(z_{t},x)>\epsilon_{m},\forall{t}\in\{1,...,T\}.\end{split}$$

(2)

where the hyper-parameter $\epsilon_{m}$ controls the amount of mutual information. The equation (2) can be equivalently expressed with Lagrangian multipliers $\lambda_{l}$ as follows:

$$\begin{split}&\underset{\theta_{e},\theta_{1},...,\theta_{T}}{\min}L(\theta_{e},\theta_{1},...,\theta_{T})\\ &=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{T}\sum_{t=1}^{T}\hat{L}^{t}(D((E(x_{i};\theta_{e});\theta_{t})))-\lambda_{l}\sum_{t=1}^{T}I(z_{t},x).\end{split}$$

(3)

However, objective (3) is not tractable in practice. Hence, we give a tractable approximation by using the lower bound of mutual information.

Although task-specific patterns are ignored by equation (1), forcing the similarity in the marginal distribution does not directly affect the capture of valid information in each task. In particular, traffic scenes require task-specific information to be maintained, as the geometric and semantic context-aware representation of the image. Therefore, maintaining contextual information that is mapped by input images can contribute to the abundant preservation of the essential characteristics of the image. To solve this issue, we limit the minimum value of MI $I(z_{t},X_{i})$ for all tasks and this maintains the representation specified by all tasks.

Mutual information is lower bounded by Noise Contrastive Estimation [44]:

$$\begin{split}I(z_{t},x)&\geq\hat{I}^{(NCE)}(z_{t},x)\\ &:=\mathbb{E}_{p_{(x)}}[D(E(x;\theta_{e});\theta_{t})-\mathbb{E}_{\tilde{p}_{(x)}}[log\sum_{x^{{}^{\prime}}}e^{D(E(x),x^{{}^{\prime}})}]],\end{split}$$

(4)

where $x^{{}^{\prime}}$ is sampled from the distribution $\tilde{p}(x)$ = $p(x)$. By providing stable approximation results [45], mutual information can be maximized by replacing maximization in Jensen-Shannon divergence (JSD):

$$\begin{split}\hat{I}^{JSD}(z_{t},x):=&\mathbb{E}_{p_{(x)}}[-sp(-D(E(x),x))]-\\ &\mathbb{E}_{p_{(x)}\times p_{(x^{{}^{\prime}})}}[sp(D(E(x),x^{{}^{\prime}}))],\end{split}$$

(5)

where $sp(\cdot)$ is a softplus function. As discussed in [45], the decoder $D$ can share backbone encoder $E$ so that maximizing equation. (5) will maximize the MI $I(z_{t},x)$. Finally equation. (3) can be rewritten as follows:

$$\begin{split}&\underset{\theta_{e},\theta_{1},...,\theta_{T}}{\min}L({\theta_{e}},{\theta_{1}},...,{\theta_{T}})\\ &=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{T}\sum_{t=1}^{T}\hat{L}^{t}(D((E(x_{i};{\theta_{e})};{\theta_{t}})))+{\lambda_{l}}\sum_{t=1}^{T}\hat{I}^{JSD}(z_{t},x).\end{split}$$

(6)

All the reported implementations are based on the PyTorch frameworks, and our method has done on one NVIDIA TITAN X GPU and one Intel Core i7-6700K CPU. In all the experiments, we use a gradient clipping trick and the Adam optimizer [46] with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.98, and weight decay to 0.0005. The proposed share encoder $E$ is implemented as a standard ResNet-50 architecture following the setting [19]. For the task decoders, we use a 2-layer CNN with global average pooling. In this paper, we deal with 23 tasks from HSD and optimize with only one objective function equation (6). The last convolutional layer is pooled to a size of class categories, respectively. All models adopt HSD dataset as the training set and are trained for the first 10 epochs with a learning rate of $0.0001$, and then for the remaining epochs at the learning rate of $0.00001$. Leaky-ReLU [47] and Batch normalization [48] are used in all layers of our networks with the mini-batch size set to 16 samples.

The Honda Scenes dataset (HSD) [19] is a large scale annotated dataset and created to enable practical scene classification. The dataset contains 80 hours of diverse high quality driving video samples collected in the San Francisco Bay, USA. The dataset contains temporal annotations for road places, weather, road environments, and road surface conditions.

Road Places The dataset contains 12 classes of road places - branch with gore on left, branch with gore on right, construction zone, merge with gore on left, merge with gore on right, 3-way intersection, 4-way intersection, 5-way intersection, overhead bridge, railway crossing, tunnel, and zebra crossing. Most classes have 3 temporal sub-classes, including approaching, entering, and passing. The merge and branch classes have approaching and passing sub-classes, while the zebra-crossing class has just approaching.

Road Environments and Weather The dataset spans 4 classes of road environments - rural, urban, highway and ramp and 4 weather conditions - rainy, sunny, cloudy, and foggy.

Road Surfaces The dataset contains 2 classes of road surfaces - dry and wet.

The dataset contains a total of about 20,000 instances spanned over these 12 classes. Moreover, the collected video images with camera are converted to a 1280 X 720 at 25 fps. The figure 3 outlines key statistics in the dataset.

As our method can properly be extended to deal with multi-task, we compare our proposed method with the baseline traffic scene recognition researches to prove the effectiveness. Results on HSD are reported for the following three types of models where each approach is gradually added: a) Single-Task Network; b) Multi-Task Network (Baseline); c) adding MI constraint to (b) (Ours).

We present the multi-task traffic scene recognition accuracy for each type on HSD in Table 1, and the evaluation results are shown in Figure 8. From Table 1, we can identify that expanding simply MTL network does not improve recognition performance. As shown in Figure 8., ’Place’ category is difficult to recognize due to unbalanced dataset distribution. Therefore, when mutual information constraint is added to the proposed MTL, recognition results can be better (Figure 8. (last column)) and we observe some enhancements on traffic scene recognition performance (Table 1. e)). This confirms that applying MI constraint at the same time is more helpful to capture contextual information. Accordingly, the results in Figure 8 are the most accurate of all results.

We demonstrate the effectiveness of our MI-based MTL approach for the application of traffic scene recognition [19]. Table 2 shows the task-wise F-Scores and the recognition performance generally outperforms the existing methods across most of tasks. This mainly due to the fact that traffic scene samples are processed with context-aware feature representations. Note that what we want to highlight in HSD benchmark (especially notice the difference between Single-Task [19] and ours) is that the proposed method can benefit from MI based MTL framework which captures the context-aware properties of images despite only one optimization formula and less memory volume.

Figure 5,6, and 7 indicate out method is significantly better than existing MTL method. Our visualization results based on t-SNE [50] indicate that our approach proves the ability of discriminative representation and the shared encoder can have a better context-awareness ability.

Here, we discuss two challenging problems, for which our method fails to recognize prediction. First, our method may fail to worst performance about classes with an extremely small number of data samples, e.g., Intersection-5 and Merge Left in the road place and ramp and rural in the road environment. In fact, the performance imbalance caused by this data imbalance can be found not only in our method but also in all existing methods. Second, our approach may wrongly recognize temporal classes in road place category. To address this issue, we believe that an online action proposal technique is needed for the MTL network to learn temporal-aware representations. Besides, we may carefully examine other possible mechanisms to avoid manual annotations for collecting more data.