C-SENN: Contrastive Self-Explaining
Neural Network

When people are given an input signal, they use concepts in their brains to understand its meaning. Concepts are abstract and verbalizable information arrangements [2], and the construction of models, which incorporate them, is essential for model utilization in a real environment where explainability and interpretability are required. To make deep learning models more accessible to the world outside the laboratory, we build a model in which individual neurons correspond to individual concepts on a one-to-one basis.

To develop models that incorporate concepts, this study focuses on the self-explaining neural network (SENN) [3, 1], which is a generalized linear model that acquires unsupervised concepts. Alvarez-Melis and Jaakkola [3] confirmed the SENN’s ability to acquire concepts for the final output [3]. However, their validation work was performed only on relatively simple image datasets, such as MNIST [4] and CIFAR-10 [5]; thus, no validation was performed on datasets that capture more complex and near-real-world environments, such as Berkeley DeepDrive (BDD) [6]. Sawada and Nakamura [1] tackled this problem by using the discriminator and shared intermediate network (They called modified SENN (M-SENN)). However, the interpretability of the concepts output by M-SENN is still low in such environments, as shown in Fig. 1 (A) (see Sec. 4 for details).

In this study, we adopt contrastive learning [7] to acquire concepts that are highly interpretable, even for images captured in complex environments. Contrastive learning is a self-supervised learning method that has been actively studied in recent years [7, 8, 9, 10]. We utilize supervised contrastive learning (SCL) [8] and the Barlow Twins (BT) [9]. SCL utilizes target labels for the final output, whereas BT ensures the independence of each feature (neuron) acquired via self-supervised learning. By incorporating these methods, we successfully acquire different concepts related to the final output without duplication. Furthermore, by augmenting contrastive learning data with the results of the Faster RCNN [11], which we use as a feature extractor, we generate concepts from verbalizable object regions. We refer to our proposed method as the Contrastive SENN(C-SENN, See Fig. 2). We apply C-SENN to the BDD-OIA dataset [12], finding that it improves the interpretability of the generated concepts (see Fig. 1 (B)). We also successfully achieved a higher accuracy than other methods, including conventional SENN.

As noted in Section 1, the interpretability of concepts generated by M-SENN is low in complex environments. One of the reasons is that it is challenging for the true/false discriminator to generate concepts while maintaining the similarity in the input space. In this study, we focus on contrastive learning to better maintain the similarity in the input space.

Contrastive learning is a kind of self-supervised learning that has been actively studied in recent years [7, 16]. Many contrastive learning methods combine multiple data augmentation techniques and learn to place transformations from the same image closer together in the feature space and those from different images farther apart. Then, the top layer (projection head) is removed, and the rest is used as a feature extractor. In contrast, as in [17], we minimize the contrastive loss simultaneously with other losses.

This study utilizes supervised contrastive learning (SCL) [8] and the Barlow Twins (BT) [9] for generating concepts.

where $\mathcal{L}^{{\rm SCL}}$ is the SCL loss function and $\mathcal{L}^{{\rm BT}}$ is the BT loss function. Furthermore,

where $\lambda$ is the hyperparameter, $\mathcal{A}_{\backslash i}$ is the set of concepts of all data except the $i$-th data, and $\mathcal{A}=\{\mbox{\boldmath$c$}_{i},\mathcal{A}_{\backslash i}\}$. $\mathcal{P}_{i}$ is the set of concepts in the data having the same label as the $i$-th target label, and $|\mathcal{P}_{i}|$ is the total number of such concepts. $c_{j,i}^{{\rm mask}}$ represents the $j$-th concept of $\mbox{\boldmath$x$}_{i}^{{\rm mask}}$, which is a result of data augmentation of $\mbox{\boldmath$x$}_{i}$. We utilize the detection results of Faster RCNN [11] for data augmentation. Note that the intermediate extractor $\mbox{\boldmath$h$}(.)$ is part of this Faster RCNN pretrained by the COCO [18] and fine-tuned with BDD100k [6]. We detect bounding boxes with Faster RCNN, and replace the regions other than the bounding box$+\epsilon$ ($\epsilon$ is a constant) with white noise. Then, we treat the resulting image as $\mbox{\boldmath$x$}_{i}^{{\rm mask}}$ (Fig. 3). As a result, the model learns to acquire concepts from the periphery of the object region. Notably, $\mbox{\boldmath$c$}^{\rm{mask}}=[c_{1}^{\rm{mask}},c_{2}^{\rm{mask}},\cdots,c_{D_{c}}^{\rm{mask}}]^{\top}$ is included in $\mathcal{P}$.

By combining Equations (9) and (10) with Equation (4), the final loss function of C-SENN is as follows:

where $\lambda_{{\rm SCL}}$ and $\lambda_{{\rm BT}}$ represent hyperparameters that control the loss function’s value. Notably, $\mathcal{L}_{{\rm DIS}}$ (the distance measure loss) is not used here. The purpose of $\mathcal{L}_{{\rm DIS}}$ is to preserve distance relations in the input image and concept space [1]. However, the same result is obtained using the SCL loss $\mathcal{L}_{{\rm SCL}}$ clearly from Equation (9). Thus, because we consider $\mathcal{L}_{{\rm DIS}}$ to be substitutable by $\mathcal{L}_{{\rm SCL}}$, we do not use $\mathcal{L}_{{\rm DIS}}$ to reduce redundancy. In this study, the resultant SENN is termed “C-SENN”.

This section describes our evaluation of C-SENN using the BDD-OIA [12] for autonomous driving, consisting of 16,802 training mages, 2,270 evaluation images, and 4,572 test images. Each image is assigned target labels (“forward ($y_{i,1}$)”, “stop ($y_{i,2}$)”, “right turn ($y_{i,3}$)”, and “left turn ($y_{i,4}$)”), representing possible driving actions (namely, $k=4$). Notably, BDD-OIA has 21 concept labels (e.g., “green light” or “‘good visibility”) for each image, but these are not used for C-SENN and M-SENN training (original SENN could not run).

In this article, we set $\beta=0.01$, $\lambda=0.1$, $\lambda_{\rm SCL}=1.0$, $\lambda_{\rm BT}=0.001$, and $D_{c}=21$ to equal the number of BDD-OIA concepts. The network structure and other hyperparameters are the same as those in [1], and all experiments were performed using one NVIDIA Tesla v100 GPU with 32-GB memory.

Figure 4 shows examples of concept saliency maps obtained by each method. These maps were generated by the grad-CAM [19]. Notably, the SC-SENN model replaced the $\mathcal{L}_{{\rm DIS}}$ of the SENN with $\mathcal{L}_{{\rm SCL}}$ and did not use the BT loss function. With $\mathcal{L}_{{\rm SCL}}$, each concept was focused on a more local region in the image. For example, the SC-SENN image on the right side of Fig. 4 focuses on traffic signals and signs. The addition of $\mathcal{L}_{{\rm BT}}$ improved the focus to even finer regions. For example, the C-SENN image on the left side of Fig. 4 focuses on the right side of each object, whereas the image on the right focuses on the foot of each. The former is a useful concept for the “right turn” action, whereas the latter is a useful concept for “forward” and “stop” actions. The results of the C-SENN saliency map for other images are shown in the appendix.

Figure B (A) shows the correlation matrix between concepts acquired by C-SENN, and Fig. B (B) shows the correlation matrix of C-SENN concepts and concept labels in the BDD-OIA dataset. Fig. B (A) shows that the correlation between concepts was small; thus, multiple concepts can be acquired by C-SENN without duplication. This is the effect of SCL. In contrast, Fig. B (B) shows that the concepts acquired by C-SENN had little correlation with the concept labels. Thus, the C-SENN makes predictions based on concepts that differ from those deemed necessary by human annotators. For example, concepts corresponding to the foot and right sides of each object, as shown in the C-SENN image in Fig. 4, do not exist in the teacher concepts. This suggests that it is possible to discover new insights by checking the concepts generated by C-SENN. Additional analyses are shown in the appendix.

Next, Fig. 6 shows examples of the regions in each image where each concept was focused. In the bottom part of the figure, the X-axis represents image ID, and the Y-axis represents the object detected by Faster RCNN. The higher the average gradCAM score in each object region, the brighter the display. This confirms that C-SENN generates concepts that span multiple object regions. Concerning interpretability, it is desirable that concepts be generated from a smaller number of objects. This is a topic for future research.

Finally, we show the driving behavior recognition results using each method. Table 1 shows result. Note that vanilla is the model that does not explicitly generate concepts between input and output, and the concept bottleneck model (CBM) [14] is the model that trains by utilizing concept labels assigned in the BDD-OIA dataset. The evaluation indices used were the F1-score for each driving behavior and $mF1$, which is the average F1-score for each action [12]. Comparing the results to vanilla confirms that recognition accuracy improved by generating concepts between input and output. Additionally, the results show that C-SENN was more accurate than M-SENN. C-SENN also showed higher values than CBM in many indices. This suggests that C-SENN can generate more appropriate recognition concepts than human-annotator preassigned concept labels. Although CBM offers the potential to improve accuracy by refining the concept labels related to driving behaviors in advance, it makes the annotation costs enormous.

In this study, we proposed C-SENN, which combines the M-SENN with two types of contrastive learning, BT and SCL. We conducted experiments on the BDD-OIA dataset, confirming that C-SENN acquired concepts that were more highly interpretable than those of other methods, and it recognizes driving actions with high accuracy. In the future, we plan to conduct a detailed analysis of what constitutes a highly interpretable concept through functional evaluations. However, collaboration with various researchers, such as psychologists, will be essential to know the ideal form of the concepts we are acquiring. This is not low-hanging fruit, but it is necessary to work on over the long term.

Figure A shows the C-SENN saliency maps for other images. Bottom right-most images are input, and other images are saliency maps obtained by the grad-CAM. As shown in these figures, some neurons focus on small areas, while others focus on the whole image with few obstacles and good visibility. When driving good visibility environments, humans also focus on the entire field of view. Therefore, we consider that neurons concentrating over the whole image corresponds to this human action.

Here, we show some results when changing conditions.

Figure B (A) and (B) show the correlation matrix using Barlow Twins [9] (same as Fig. 6(A)) and Variance-Invariance-Covariance Regularization (VICReg) [20], respectively. VICReg is also a method that ensures the independence of each concept, as in BT. Comparing these matrices, BT has smaller values of the non-diagonal components than VICReg. Therefore, we used the BT for C-SENN.

Figure B (C) and (D) show the correlation matrix when changing the number of concepts $D_{c}$. These figures show that C-SENN has smaller values of the non-diagonal components irrelevant to $D_{c}$. Figure C shows the saliency maps of $D_{c}=10$ and $D_{c}=21$. When increasing $D_{c}$, C-SENN focuses on small areas and vice-vasa. Namely, generated concepts change by changing $D_{c}$. Automatic acquisition of the appropriate number of concepts, taking interpretability and recognition rate into account, is an issue for future work.