Compositional Convolutional Neural Networks:
A Robust and Interpretable Model for Object Recognition under Occlusion

Many recent studies hongru ; kortylewski2019compositional have shown that, current deep neural networks are much less robust to partial occlusion compared with humans at object classification. Fawzi et al. fawzi2016measuring show that DCNNs are vulnerable to partial occlusions simulated by masking small patches of the input image. Several following works devries2017improved ; yun2019cutmix propose to augment the training data by masking out patches from the image. Our experimental results in Section 6.2 show that such data augmentation approaches only have limited beneficial effects. Moreover, data augmentation increases the amount of training data and thus the training time and cost. Therefore, it is desirable to develop novel neural network architectures that are inherently robust to partial occlusion. Xiao et al. xiao2019tdapnet propose TDAPNet, a deep network with an attention mechanism that masks out occluded features in lower layers to increase the robustness of the model against occlusion. Though it can work reliably on artificial occlusion, our results show that this model does not perform well on images with real occlusion.

Compared to image classification, object detection additionally involves the estimation of the object location and bounding box. While a search over the image can be implemented efficiently, e.g. using a scanning window lampert2008beyond , the number of potential bounding boxes is combinatorial in the number of pixels. Currently, the most widely used approach for solving this problem is to use region proposal networks (RPNs) girshick2014rich which enable the training of fast approaches for object detection girshick2015fast ; ren2015faster ; cai2018cascade . However, our experiments in Section 6.3 demonstrate that RPNs do not estimate the location and bounding box of an object correctly under occlusion, which consequentially deteriorates the performance of these approaches.

To resolve this problem, a boosted cascade-based method for detecting partially visible objects has been proposed by InferenOccDetect . However, this approach is based on hand-crafted features and can only be applied to images which are artificially occluded by cutting out patches. A number of deep learning based approaches have also been proposed for detecting occluded objects OR-CNN ; OccNet , but they require detailed part level annotation to reconstruct the occluded objects. The work of 3DAspectlets proposes to use 3D models and treat occlusion as a multi-label classification problem. However, in real-world scenarios, the classes of the occluders can be difficult to model and are often not known a-priori. Other approaches focus on videos or stereo images SymmNet ; Symmetricstereo . In this work, we consider the problem of partial occlusion in still images.

In contrast to deep learning approaches, generative compositional models jin2006context ; zhu2008 ; fidler2014 ; dai2014unsupervised ; kortylewski2017greedy have been shown to be inherently robust to partial occlusion. Such models have been successfully applied for detecting object parts wang2017detecting ; zhang2018deepvoting and recognizing 2D patterns george2017generative ; kortylewski2016probabilistic under partial occlusion. Such part-based voting approaches wang2017detecting ; zhang2018deepvoting perform reliably for semantic part detection under occlusion, but they assume a fixed size bounding box and are viewpoint specific, which limits their applicability in the context of object detection.

And-Or graphs (AOG) nilsson1980principles ; jin2006context ; dechter2007and have been investigated e.g. to build hierarchical part-based models for human parsing and for object detection. Intuitively, the or-nodes allow the model to learn different object/part configurations, while the and-node decomposes them into smaller components. Early works in this direction chen2008rapid ; zhu2008max ; li2013vehicle relied on pre-defined parts and pre-defined graph structures. To learn the AOG model with less supervision, Zhu et al. zhu2008 use recursive compositional clustering. However, this method may lead to unexplainable parts and structures. Song et al. song2013discriminatively used an over-complete set of shape primitives to quantize the image lattices and then organized them into an AOG by exploiting their compositional relations through iterative cutting. Xia et al. xia2016pose explicitly defined parts and part compositions, which correspond to the leaf node and non-leaf node in the AOG respectively. Then they used a score function with pre-defined adjacent part pairs to learn the structure of the AOG, which still required considerable amount of human input. Wu et al. wu2015learning made use of a large number of synthetic images generated by CAD simulations, on which 17 semantic parts were manually labeled. They enumerated all configurations observed from the synthetic data and then used a graph compression algorithm to get the refined AOG structure. All these models learned the AOG in two steps: after the graph structure was decided, variants of latent structural SVM was used to learn model parameters. In a different approach, Lin et al. lin2014discriminatively learned AOG by joint optimization of both model structures and parameters. The resulting model worked on object shape detection with moderate performance and may not be easily applied to general object detection.

Most of the works on AOGs used low-level features (e.g., HOG features) to model the part appearance, which may limit their capacity and discriminative power. Furthermore, none of these works modeled occluders explicitly or tested their method on images with different level of occlusions, therefore it is unclear how those models can be made robust to occlusion. CompositionalNets can be considered to be complementary to and-or-graphs. In fact, our mixture model can be interpreted as simple two-layered and-or-graph, where each mixture components combines parts (vMF kernels) for a certain object pose, and the final class score is an “or”-combination over the different object poses (mixture components). While our model could be generalized to have multiple layers with and-or-nodes to introduce more flexibility in the representation, the focus of this work is robustness to partial occlusion. Furthermore, our experiments show that a two-layered and-or graph seems to be good enough to achieve high performance at image classification and detection on popular datasets such as PASCAL3D+, MS-COCO and ImageNet. Moreover, and-or-graphs often require considerable amounts of supervision for the graph structure chen2008rapid ; zhu2008max ; li2013vehicle or are not well interpretable zhu2008 ; song2013discriminatively , whereas our graph is learned from class supervision only and still learns a meaningful human-interpretable representation.

An early work from Liao et al. liao2016learning proposes to integrate compositionality into DCNNs by regularizing the feature representations of DCNNs to cluster during learning. Their qualitative results show that the resulting feature clusters resemble part-like detectors. Zhang et al. zhang2018interpretable also demonstrate that part detectors emerge in DCNNs by restricting the activations in feature maps to have a localized distribution. However, these approaches have not been shown to enhance the robustness of deep models to partial occlusion. Other related works propose to regularize the convolution kernels to be sparse tabernik2016towards , or to force feature activations to be disentangled for different objects stone2017teaching . As the compositional model is not explicitly incorporated but rather implicitly encoded within the parameters of the DCNNs, the resulting models remain black-box and not expedcted to be robust to partial occlusion. A number of works li2019aognets ; tang2018deeply ; tang2017towards use differentiable graphical models to integrate part-whole compositions into DCNNs. However, these models are purely discriminative and thus are also deep networks with no internal mechanism to account for partial occlusion. Girshick et al. girshick2015deformable discussed that compositional deformable part models can be formulated as neural networks. However, they do not evaluate their models’ robustness to partial occlusion nor its explainability. Kortylewski et al. kortylewski2019compositional propose to learn a generative dictionary-based compositional model using the features of a DCNN. Instead of merging the compositional model into the DCNN, they use it as a “backup” for an independently trained DCNN. Only when the DCNN classification score falls below a certain threshold, the prediction will be substituted by the output of the compositional model.

Many post-hoc methods have been proposed to explain what has been encoded in the intermediate layers of DCNNs. Several works le2013building ; zhou2015object visualize a real or generated input that activates a filter most to study the roles of individual units inside neural networks. Similarly, Nguyen et al. nguyen2016synthesizing synthesize prototypical images for individual units by learning a feature inversion mapping, while Bau et al. bau2017network visualize segmentation masks extracted from filter activations and assign concept labels automatically. On the other hand, the works of mahendran2015understanding ; simonyan2013deep ; zeiler2014visualizing use variants of back-propagation to identify or generate salient image features. Moving beyond studying individual hidden units, Wang et al. wang2015unsupervised use clusters of activations from all units in a layer and shows that the cluster centers yield better part detectors. Alain and Bengio alain2016understanding probe mid-layer filters by training linear classifiers on the intermediate activations. They also analyze the information dynamics among layers and its effect on the final prediction. The work of Fong et al. fong2018net2vec shows that filter embeddings better characterize the meaning of a representation and its relationship to other concepts. Most of these works evaluate their results using human judgments.

Unlike the post-hoc methods that focus on visualizing/analyzing pre-trained DCNNs, other approaches aim to learn more meaningful representations during the network training stage. The work of ross2017right explains and regularizes differentiable models by examining and selectively penalizing their input gradients, but this method requires extra annotation from human experts. Hu et al. hu2016harnessing regularize the learning process by introducing an iterative distillation method that transfers the structured information of logic rules into the weights of neural networks. Stone et al. stone2017teaching encourage networks to form representations that disentangle objects from their surroundings and from each other, but they do not obtain part-level semantics explicitly. Sabour et al. sabour2017dynamic propose a capsule model, which used a dynamic routing mechanism to parse the entire object into a parsing tree of capsules, and each capsule may encode a specific meaning. The work of Zhang et al. zhang2018interpretable invents a generic loss to regularize the representation of a filter to improve its interpretability.

In this work, we unify generative compositional models and deep convolutional neural networks into a joint architecture with innate robustness to partial occlusion. The generative nature of the model enables it to localize occluders and to recognize objects based on the spatial configuration of visible object parts. CompositionalNets are naturally interpretable as their predictions can be understood in terms of part detection, occluder localization and viewpoint estimation.

We denote a feature map $F^{l}\in\mathbb{R}^{H\times W\times D}$ as the output of a layer $l$ in a DCNN, with $D$ being the number of channels. $f^{l}_{i}\in\mathbb{R}^{D}$ is a feature vector in $F^{l}$ at position $i$ on the 2D lattice $\mathcal{P}$ of the feature map. In the remainder of this section we omit the superscript $l$ for notational simplicity because this is fixed a-priori in our experiments.

Our goal is to learn a generative model $p(F|y)$ of the real-valued feature activations $F$ for an object class $y$. In the following, we assume the viewpoint of the object to be known. Later, we generalize the model to 3D objects with varying viewpoints. We define the probabilistic model $p(F|y)$ to be a mixture of von-Mises-Fisher (vMF) distributions:

where $\theta_{y}=\{\mathcal{A}_{y},\Lambda\}$ are the model parameters and $\mathcal{A}_{y}=\{\mathcal{A}_{i,y}\}$ are the parameters of the mixture models at every position $i\in\mathcal{P}$ on the 2D lattice of the feature map $F$. Note that the probabilistic model defined in Equation 1 has a tree-like independence structure and therefore enables efficient inference kortylewski2017greedy . Moreover,

are the mixture coefficients, $K$ is the number of mixture components and $\Lambda=\{\lambda_{k}=\{\sigma_{k},\mu_{k}\}|k=1,\dots,K\}$ are the variance and mean of the vMF distribution:

where $Z(\sigma_{k})$ is the normalization constant. As $Z(\sigma_{k})$ is difficult to compute for high-dimensional data banerjee2005clustering , we assume $\sigma_{k}$ to be fixed a-priori. Hence, the normalization constant is the same for each mixture component and does not need to be computed explicitly when optimizing the parameters. After learning the vMF cluster centers $\{\mu_{k}\}$ with maximum likelihood optimization, they resemble feature activation patterns that frequently occur in the training data. Interestingly, feature vectors that are similar to one of the vMF cluster centers, are often induced by image patches that are similar in appearance and often even share semantic meanings (see Figure 2). This property was also observed in a number of related works that used clustering in the neural feature space wang2015unsupervised ; liao2016learning ; wang2017detecting . Subsequently, we learn the mixture coefficients $\alpha_{i,k,y}$ with maximum likelihood estimation from the training images. Intuitively, $\alpha_{i,k,y}$ describes the expected activation of a cluster center $\mu_{k}$ at a position $i$ in a feature map $F$ for a class $y$.

An important property of convolutional networks is that the spatial information from the image is preserved in the feature maps. Hence, the set of mixture coefficients $\mathcal{A}_{y}$ intuitively can be thought of as being a 2D template that describes the expected spatial activation pattern of parts in an image for a given class $y$ - e.g. where the tires of a car are expected to be located in an image. Therefore, our proposed vMF model intuitively accumulates the part detections spatially into a hypothesis about the objects’ presence. Note that this implements a part-based voting mechanism.

As the spatial pattern of parts varies significantly with the 3D pose of the object, we represent 3D objects as a mixture of compositional models:

with $\mathcal{V}\texttt{=}\{\nu^{m}\in\{0,1\},\sum_{m}\nu^{m}\texttt{=}1\}$ and $\Theta_{y}=\{\theta^{m}_{y},m=1,\dots,M\}$. Here $M$ is the number of compositional models in the mixture distribution and $\nu_{m}$ is a binary assignment variable that indicates which mixture component is active. Intuitively, each mixture component $m$ will represent a different viewpoint of an object (see Experiments in Section 6.4.3).

The parameters of the mixture components $\{\mathcal{A}^{m}_{y}\}$ need to be learned in an EM-type manner by iterating between estimating the assignment variables $\mathcal{V}$ and maximum likelihood estimation of $\{\mathcal{A}^{m}_{y}\}$. We discuss how this process can be performed in a neural network in Section 5.2.

Following the approach presented in kortylewski2017model , compositional models can be augmented with an occlusion model. The intuition behind an occlusion model is that at each position $i$ in the image either the object model $p(f_{i}|\mathcal{A}^{m}_{i,y},\Lambda)$ or an occluder model $p(f_{i}|\beta,\Lambda)$ is active (note that this is closely related to robust statistics huber2011robust ):

The binary variables $\mathcal{Z}^{m}=\{z^{m}_{i}\in\{0,1\}|i\in\mathcal{P}\}$ indicate if the object is occluded at position $i$ for mixture component $m$. The occlusion prior $p(z^{m}_{i}\texttt{=}1)$ is fixed a-priori. We use a mixture of several occluder models that are learned in an unsupervised manner:

where {$\tau_{n}\in\{0,1\},\sum_{n}\tau_{n}=1\}$ indicates which occluder model explains the data best. Note that the model parameters $\beta$ are independent of the position $i$ in the feature map and thus the model has no spatial structure.

The parameters of the occluder models $\beta_{n}$ are learned from clustered features of random natural images that do not contain any object of interest (see Figure 4(a)). Hence, the mixture coefficients $\beta_{n,k}$ intuitively describe the expected activation of $\mu_{k}$ in regions of natural images. While it is possible to use just a single occluder model kortylewski2019compositional , we found that having a mixture model slightly increases the classification performance and the occluder localization performance. The reason is that different occluder models can focus at explaining part distributions for different typical regions in images e.g. uniform colored image patches or textured regions. We illustrate this in Figure 4(b) by visualizing patches from the training data in Figure 4(a) that have the highest likelihood for five different occluder models. Note that the purpose of the occluder models is not to be purely specific to one particular texture type, they also need to be general enough to explain a variety of local image patterns which the object model (Eq. 1) is not able to explain well. Therefore, there is a trade-off between specificity and generality of the occluder models. We found that using five models balances this trade-off well.

A natural way of generalizing CompositionalNets to object detection is to combine them with region proposal networks (RPNs). However, RPNs are typically trained end-to-end and therefore cannot predict the bounding box of strongly occluded objects reliably (see our experiments in Section 6.3). Figure 5 illustrates this limitation by depicting the detection results of Faster R-CNN trained with CutOut devries2017improved (red box) and a combination of RPN and CompositionalNet (yellow box). We address this limitation by generalizing CompositionalNets to object localization. In particular, we introduce a detection layer that accumulates votes for the object center for all mixtures $m$ over the positions $i$ in the feature map $F$. In order to achieve this, we compute the object likelihood in a scanning window manner. Thus, we shift the feature map, w.r.t. the object model along all points $i$ from the 2D lattice of the feature map. This process will generate a spatial likelihood map:

where $F_{i}$ denotes the feature map centered at the position $i$. Using this simple generalization we can perform object localization by selecting all maxima in $R$ above a threshold $t$ after non-maximum suppression. Our proposed detection layer can be implemented efficiently on modern hardware using convolution-like operations (see Section 6.3 for more details).

While CompositionalNets can be generalized to localize partially occluded objects using our proposed detection layer, estimating the bounding box of an object under occlusion is more difficult because a significant amount of the object might not be visible (Figure 5). We solve this problem by generalizing the part-based voting mechanism in CompositionalNets to vote for the bounding box corners in addition to voting for the object center. In this way, we can leverage the viewpoint-specific spatial distribution of part activations for bounding-box estimation. This makes the process highly robust to missing parts. In particular, we learn additional mixture components that model the expected feature activations around bounding box corners $p(F_{i}|\Theta_{y}^{c})$, where $c=\{ct,br,tl\}$ are the object center $ct$ and two opposite bounding box corners $\{br,tl\}$. Figure 5 illustrates the spatial likelihood maps $R^{c}$ of all three models. We generate a bounding box using the two points that have maximal likelihood. Note how in Figure 5 the bounding box can be localized accurately, despite the partial occlusion of the object. We discuss how the model can be learned in an end-to-end manner in Section 5.3.

While in image classification, the object of interest often dominates a large part of the image, in object detection the object is embedded in a larger context that is often biased in the training data (e.g. airplanes often have blue background). This gives rise to a critical problem when aiming to detect partially occluded objects. Intuitively, the objects’ context can be useful for recognizing objects due to biases in the data. However, relying too strongly on context can be misleading, because if objects are strongly occluded (Figure 6) the detection thresholds must be lowered. This, in turn, increases the influence of the objects context and leads to false-positive detections in regions with no object. Hence, it is important to have control over the influence of contextual cues on the detection result.

We overcome this issue by explicitly separating the representation of the context from that of the object, to control the influence of the contextual information on the detection result. In particular, we represent the feature map $F$ as a mixture of two models:

Here $\{{A}^{m}_{i,y},\chi^{m}_{i,y}\}$ are the parameters of the context-aware model that is defined to be a mixture of vMF likelihoods (Equation 2). The parameter $\omega$ controls the trade-off between context and object, and is fixed a-priori at test time. Note that setting $\omega=0.5$ retains the CompositionalNet as proposed in Section 3 as the context and object would be weighted equally. Figure 6 illustrates the benefits of reducing the influence of the context on the detection result under partial occlusion. The context parameters $\chi^{m}_{i,y}$ and object parameters $\mathcal{A}^{m}_{i,y}$ can be learned from the training data using maximum likelihood estimation, assuming a binary assignment $\pi_{i}$ of the feature vectors $f_{i}$ in the training data to either the context or the object. To obtain this binary assignment, we need to segment the training images into context and object based on the available bounding box annotation. We do so, by assuming that any feature vector with a receptive field outside of the scope of the bounding boxes can be considered to be context. Based on this assumption, we cluster the context features using K-means++ k-means++ to generate a dictionary of context feature centers $\mathcal{C}=\{\mathcal{C}_{q}\in\mathbb{R}^{D}|q=1,\dots,Q\}$. We apply a threshold on the cosine similarity $s(\mathcal{C},f_{i})=\max_{q}[(\mathcal{C}_{q}^{T}f_{i})/(\left\lVert\mathcal{C}_{q}\right\rVert\left\lVert f_{i}\right\rVert)]$ to segment the context and the object in any given training image (Figure 7).

The computational graph of our proposed fully generative compositional model is acyclic. Hence, we can perform inference in a single forward pass as illustrated in Figure 8. We use a standard DCNN backbone to extract a feature representation $F=\psi(I,\Omega)\in\mathbb{R}^{H\times W\times D}$ from the input image $I$, where $\Omega$ are the parameters of the feature extractor (purple tensor in Figure 8). The pipeline after the feature extractor illustrates the computation of the model likelihood $p(F_{i}|\Theta)$ when the model is centered at position $i$ in the feature map (illustrated by the dotted black square on the feature tensor $F$). For image classification, the model will always be positioned at the image center, whereas for object detection, the model will be evaluated at every position in $F$ (as defined in Equation 11). We omit the subscript in $F_{i}$ in the following for notational clarity.

After feature extraction the model computes the vMF likelihood function $p(f_{i}|\lambda_{k})$ (Equation 4). The function is composed of two operations: An inner product $b_{i,k}=\mu_{k}^{T}f_{i}$ and a non-linear transformation $\mathcal{N}=\exp(\sigma_{k}b_{i,k})/Z(\sigma_{k})$. Since $\mu_{k}$ is independent of the position $i$, computing $b_{i,k}$ is equivalent to a $1\times 1$ convolution of $F$ with $\mu_{k}$. Hence, the vMF likelihood (Figure 8 yellow tensor) can be computed by :

The mixture likelihoods $p(f_{i}|\mathcal{A}^{m}_{i,y},\chi^{m}_{i,y},\Lambda)$ (Equation 4.3) are computed for every position $i$ as a dot-product between the mixture coefficients $\{\mathcal{A}^{m}_{i,y},\chi^{m}_{i,y}\}$ and the corresponding vector $l_{i}\in\mathbb{R}^{K}$ from the vMF likelihood tensor $L$:

(Figure 8 blue planes). Similarly, the occlusion likelihood can be computed as $O=\{\max_{n}l_{i}^{T}\beta_{n}|\forall i\in\mathcal{P}\}\in\mathbb{R}^{H\times W}$ (Figure 8 red plane). Together, the occlusion likelihood $O$ and the mixture likelihoods $\{E^{m}_{y}\}$ are used to estimate the overall likelihood of the individual mixtures as $s^{m}_{y}=\log p(F|\theta_{y}^{m},\beta)=\sum_{i}\max(E^{m}_{i,y},O_{i})$. The final model likelihood is computed as $s_{y}=\log p(F|\Theta_{y})=\max_{m}s^{m}_{y}$ and the final occlusion map is selected accordingly as $\mathcal{Z}_{y}=\mathcal{Z}^{\bar{m}}_{y}\in\mathbb{R}^{H\times W}$ where $\bar{m}=\operatorname*{argmax}_{m}s^{m}_{y}$.

Our model is fully differentiable and can be trained end-to-end using backpropagation. In our image classification experiments, context-awareness does not have a significant influence as the background is largely cropped out. Therefore, we use CompositionalNets as defined in Section 3 for image classification. Hence, the trainable parameters of a CompositionalNet are $T=\{\Lambda,\mathcal{A}_{y}\}$. We optimize those parameters jointly using stochastic gradient descent. The loss function is composed of three terms:

$\mathcal{L}_{class}(y,y^{\prime})$ is the cross-entropy loss between the network output $y^{\prime}$ and the true class label $y$. We use a temperature $\mathcal{T}$ in the softmax classifier: $f(y)_{i}=\frac{e^{y_{i}\cdot\mathcal{T}}}{\Sigma_{i}e^{y_{i}\cdot\mathcal{T}}}$, with $\mathcal{T}=2$. $\mathcal{L}_{vmf}$ and $\mathcal{L}_{mix}$ regularize the parameters of the compositional model to have maximal likelihood for the features in $F$. The parameters $\{\gamma_{1},\gamma_{2}\}$ control the trade-off between the loss terms.

The vMF cluster centers $\mu_{k}$ are learned by maximizing the vMF-likelihoods (Equation 4) for the feature vectors $f_{i}$ in the training images. We keep the vMF variance $\sigma_{k}$ constant, which also reduces the normalization term $Z(\sigma_{k})$ to a constant. We assume a hard assignment of the feature vectors $f_{i}$ to the vMF clusters during training. Hence, the free energy to be minimized for maximizing the vMF likelihood wang2017visual is:

where $C$ is a constant. Intuitively, this loss encourages the cluster centers $\mu_{k}$ to be similar to the feature vectors $f_{i}$.

In order to learn the mixture coefficients $\mathcal{A}^{m}_{y}$ we need to maximize the model likelihood (Equation 5). We can avoid an iterative EM-type learning procedure by making use of the fact that the the mixture assignment $\nu_{m}$ and the occlusion variables $z_{i}$ have been inferred in the forward inference process. Furthermore, the parameters of the occluder model are learned a-priori and then fixed. Hence the energy to be minimized for learning the mixture coefficients is:

Here, $z^{\uparrow}_{i}$ and $m^{\uparrow}$ denote the variables that were inferred in the forward process (Figure 8).

For object detection, we train context-aware CompositionalNets as proposed in Section 4. The trainable parameters of the model are $T^{c}=\{\Lambda,\{\mathcal{A}^{c}_{y}\},\{\chi^{c}_{y}\}\}$ where $c\in\{ct,br,tl\}$. The loss function has three main objectives. The model should explain data with maximal likelihood ($\mathcal{L}_{g}$), while also localizing ($\mathcal{L}_{detect}$) and classifying ($\mathcal{L}_{class}$) the object accurately in the training images. :

where $\epsilon_{1},\epsilon_{2}$ control the trade-off between the loss terms. While $\mathcal{L}_{g}$ is learned from images $\hat{I^{c}}$ with feature maps $F^{c}$ that are centered at $c\in\{c,bl,tr\}$, the other losses are learned from unaligned training images $I$ with feature maps $F$.

Generative Regularization. The model is regularized to be generative in terms of the neural feature activations by optimizing:

where $\mathcal{L}_{vmf}(F^{c},\Lambda)$ is defined in Equation 17 and the parameters of the context-aware model $\mathcal{A}^{c}_{y}$ and $\chi^{c}_{y}$ are learned by optimizing the context loss:

Here, $\pi_{i}\in\{0,1\}$ is a context assignment variable that indicates if a feature vector $f_{i}$ belongs to the context or to the object model and $\mathcal{L}_{mix}$ is defined in Equation 19. We estimate the context assignments a-priori using a simple binary segmentation as described in Section 4.3.

Localization and Bounding Box Estimation. We denote the normalized response map of the ground truth class as $\hat{S}^{c}\in\mathbb{R}^{H\times W}$ and the ground truth annotation as $\bar{S}^{c}\in\mathbb{R}^{H\times W}$. The elements of the response map are computed as:

The ground truth map $\bar{S}^{c}$ is a binary map where the ground truth position $\bar{i}$ is set to $\bar{S}^{c}(\bar{i})=1$ and all other entries are set to zero. The detection loss is then defined as:

Datasets for image classification. We evaluate our model on the Occluded-P3D+-Vehicles dataset as proposed in wang2017detecting and extended in kortylewski2019compositional . The dataset is based on images of vehicles from the PASCAL3D+ dataset xiang2014beyond that were synthetically occluded with four different types of occluders: segmented objects as well as patches with constant white color, random noise and textures (see Figure 10(a) for examples). The amount of partial occlusion of the object varies in four different levels: $0\%$ (L0), $20$-$40\%$ (L1), $40$-$60\%$ (L2), $60$-$80\%$ (L3).

Furthermore, we introduce a dataset with images of real occlusions which we term Occluded-COCO-Vehicles. It contains the same classes as the Occluded-P3D+-Vehicles dataset. The dataset was generated by categorizing objects from the MS-COCO lin2014microsoft into the four occlusion levels as defined in the Occluded-P3D+-Vehicles dataset. To achieve this, we relate the amount of object that is visible in the MS-COCO images to the one from the Occluded-P3D+-Vehicles based on the segmentation masks that are available in both datasets. The numbers of test images for each occlusion level are: $2036$ (L0), $768$ (L1), $306$ (L2), $73$ (L3). For training purpose, we define a separate training dataset of $2036$ images from level L0. Figure 10(b) illustrates some example images from this dataset.

While the current generation of CompositionalNets has been primarily developed for recognizing a smaller number of vehicle-type objects, we also want to study if CompositionalNets retain their robustness to partial occlusion even when tested at a larger scale with non-vehicle type objects. For this, we introduce the Occluded-ImageNet dataset. In particular, we randomly select different numbers of classes $\{25,50,100\}$ from the ImageNet dataset deng2009imagenet to study the influence of the dataset scale on the performance. We crop the objects from the training images with available annotation to make our results comparable to the Occluded-P3D+-Vehicles and Occluded-COCO-Vehicles data. We randomly split the data into $50$ test images per class and assign the remaining images to the training data. This results in $\{12241,25120,49263\}$ training and $\{1250,2500,$ $5000\}$ test images for the different sized subsets. We artificially occlude the test images by using segmented objects from the MS-COCO dataset. Note that to simulate occlusion, we only use those object classes from MS-COCO that do not overlap with the ImageNet classes. The amount of partial occlusion of the object varies in four different levels (L0, L1, L2, L3) as in the Occluded-P3D+-Vehicles dataset.

Datasets for Object Detection. The object detection datasets are defined in a similar way as the classification data. We synthesize an Occluded-P3D+-Vehicles-Detection dataset, which contains vehicles at a certain scale and various levels of occlusion. The occluders, which include humans, animals and plants, are cropped from the MS-COCO dataset lin2014microsoft . In an effort to accurately depict real-world occlusions, we superimpose the occluders onto the object, such that the occluders are placed not only inside the bounding box of the objects but also on the background. We generate the dataset in a total of 9 occlusion levels along two occlusion dimensions: We define three levels of occlusion of the object (FG-L1: 20-40%, FG-L2:40-60% and FG-L3:60-80% of the object area is occluded). Furthermore, we define three levels of occlusion of the context around the object (BG-L1: 0-20%, BG-L2:20-40% and BG-L3:40-60% of the context area is occluded). Example images are shown in Figure 10(c). In order to evaluate the tested models on real-world occlusions, we test them on the subset of the MS-COCO dataset as defined for classification (Figure 10(d)).

Model initialization. We initialize the vMF kernels $\{\mu_{k}\}$ and the mixture components $\{\mathcal{A}_{y},\chi_{y}\}$ by maximum likelihood estimation after clustering the training data. We compute the mixture assignments using spectral clustering with the hamming distance between vMF kernel activations in the pool4 layer in VGG-16 of all training images. The intuition is that objects with a similar viewpoint and 3D structure will have similar vMF activation patterns, and thus should be assigned to the same mixture component.

Training setup for classification. We train CompositionalNets from the feature activations of the layer before the classifier for several popular DCNNs: VGG-16 simonyan2014very , ResNet50 he2016deep and ResNext xie2017aggregated . All models were pretrained on ImageNetdeng2009imagenet . We set the vMF variance to $\sigma_{k}=30,\forall k\in\{1,\dots,K\}$. We train the parameters of the generative model $\{\{\mu_{k}\},\{\mathcal{A}_{y},\chi_{y}\}\}$ using backpropagation and keep the parameters of the backbone $\Omega$ fixed (training $\Omega$ did not have significant effects on the results). We learn the parameters of $n=5$ occluder models $\{\beta_{1},\dots,\beta_{n}\}$ in an unsupervised manner as described in Section 3.1 and keep them fixed throughout the experiments. We set the number of mixture components to $M=4$. The mixing weights of the loss are set to: $\gamma_{1}=3.0$, $\gamma_{2}=3.0$. We train for $50$ epochs using stochastic gradient descent with momentum $r=0.9$ and a learning rate of $lr=0.01$. Our reported parameter settings are very general as they are fixed in all our experiments even for different datasets.

Training setup for object detection. We optimize the loss described in Equation 20, with $\epsilon_{1}=0.2$ and $\epsilon_{2}=0.4$. We apply the Adam Optimizer kingma2014adam with different learning rates on different parts of CompositionalNets: $lr_{vc}=2\cdot 10^{-5}$, $lr_{mixture\ model}=lr_{corner\ model}=5\cdot 10^{-5}$. The model is trained for a total of 2 epochs with 10600 iteration per epoch. The training takes three hours in total on a machine with 4 NVIDIA TITAN Xp GPUs.

Occluded-P3D+. Table 1 reports the results of VGG-16, ResNet50 and ResNext that were fine-tuned with the respective training data. Furthermore, we show the results of dictionary-based compositional models (CoD) kortylewski2019compositional and TDAPNet xiao2019tdapnet . We also report CompositionalNets learned from the pool4 and pool5 layer of the VGG-16 network respectively (CompNet-VGG-[p4 & p5]) and CompositionalNets learned from features of the last residual block (RB4) of ResNet50 (CompNet-Res50) and both the last (RB4) and second last (RB3) residual block of ResNext (CompNet-RXT). In this setup, all models are trained with non-occluded images ($L0$), while at test time the models are exposed to images with different amount of partial occlusion ($L0$-$L3$).

Overall, we observe that DCNNs do not generalize well to out-of-distribution examples in terms of partial occlusion. They perform well on the type of data that they were exposed to during training (L0) and generalize to a limited extent away from it (L1). However, their performance collapses when objects are strongly occluded (L3).

In contrast, we observe that CompositionalNets generalize very well even under strong occlusion. Moreover, using the higher layers of more powerful residual backbones also significantly increases the performance of CompositionalNets. We also observe that CompNets learned from the RB3 layer of ResNext perform significantly worse, compared to those learned from RB4 ($-5.3\%$ on average). In contrast, CompNets learned from pool4 of VGG still give comparable results pool5. We conjecture from this observation that the additional convolutional and residual layers in RB4 are of critical importance for the ResNext backbone.

Occluded MS-COCO. Table 2 shows classification results under a realistic occlusion scenario by testing on the Occluded-COCO-Vehicles dataset. The models in the first part of the Table (PASCAL3D+) are trained solely on non-occluded images of the PASCAL3D+ data. We can observe that from all standard DCNNs VGG-16 transfers the worst to the MS-COCO data, while ResNet50 and ResNext generalize better, in particular to strongly occluded objects. ResNext even outperforms the recently proposed TDAPNet that was specifically designed for image classification under occlusion. CompositionalNets consistently outperform non- compositional DCNNs by a large margin, highlighting the generality of our approach. In particular, CompNet-ResNext-RB4 can generalize very strongly from training on non-occluded PASCAL3D+ data to strongly occluded objects from MS-COCO.

The second part of the table (MS-COCO) shows the classification performance after fine-tuning on the non-occluded training set of the Occluded-COCO-Vehicles dataset. A common pattern of all three standard DCNNs is that their performance increases for low occlusion ($L0-2$) while it decreases for stronger occlusion ($L3$). In contrast, the performance of the CompositionalNets increases substantially after fine-tuning for all occlusion levels.

The third part of Table 2 (MS-COCO-CutPaste) shows classification results after training with strong data augmentation in terms of partial occlusion. In particular, we artificially occlude the non-occluded training images used in the previous experiment with all four types of artificial occluders used in the Occluded-P3D+-Vehicles dataset. This data augmentation increases the dataset by a factor of $5$. From the classification results, we can observe that data augmentation increases the performance of the DCNNs significantly. VGG-16 still suffers from strong occlusions. However, ResNet50 and particularly ResNext become significantly more robust to occlusion. The performance of the CompositionalNets learned from VGG-16 increases further when trained with augmented data, whereas the ones learned from ResNet50 and ResNext only benefit slightly, while still outperforming their non-compositional counterparts. Similar to the results in Table 1, CompNet-ResNext-RB3 performs significantly worse compared to CompNet-ResNext-RB4, highlighting the importance of the last layers in the backbone of ResNext. Interestingly, CompNet-ResNext-RB3 performs on par to CompNet-ResNext-p4 under real occlusion, whereas it is less robust to the artificial occluders (Table 1). This suggests that the ResNext features have developed an invariance to partial occlusion from the ImageNet pre-training that does not transfer as well to artificial occlusion. We discuss this in more detail in Section 6.4.1.

Occluded ImageNet. Table 3 shows classification results for larger scale experiments with non-vehicle objects on the Occluded-ImageNet dataset. We compare a standard ResNext model trained with strong data augmentation using CutOut devries2017improved with CompositionalNets learned from a ResNext backbone that were trained with non-augmented images only. We also test CompositionalNets with $512$ and $1024$ vMF kernels. We observe that CompositionalNets are more robust under strong occlusion in all experiments and stay competitive in low occlusion scenarios. The CompositionalNet performance decreases slightly on non-occluded images in the 100 class experiment (ResNext+CutOut: 97.1; CompNet-ResNext-1024: 94.2). However, CompositionalNets perform better under any amount of partial occlusion even for this large set of non-vehicle objects.

Furthermore, we observe that the number of vMF kernels does not have a critical influence on the performance. This highlights the advantage of the compositional representation which enables an efficient sharing of the vMF kernels among the different classes. Nevertheless, we can observe that increasing the number of vMF kernels has a positive effect on the performance for higher number classes, because the model can represent the objects more accurately when more parts are available.

Overall, we can observe that CompositionalNets are already capable to generalize to large amounts of non-vehicle classes, while retaining their robustness to partial occlusion. This is a very promising result considering that our model was mainly tested at the robust analysis of a smaller set of vehicle objects. Therefore, we expect improvements by investigating how CompositionalNets could better model articulated objects such as animals.

Summary of classification experiments. All classification experiments clearly highlight the robustness of CompositionalNets at classifying partially occluded objects, while also being highly discriminative when objects are not occluded. Overall, CompositionalNets learned from several popular DCNNs show a significantly improved generalization performance compared to non-compositional DCNNs under artificial as well as real occlusion. Notably, CompNet-ResNext trained from non-occluded PASCAL3D+ data performs essentially on par with the best DCNN (ResNext) trained from strongly augmented MS-COCO images. This highlights the data efficiency and strong generalization ability of CompositionalNets. Furthermore, we observe that CompositionalNets have a lot of potential for large-scale classification tasks with non-vehicle type objects.

In this Section, we evaluate CompositionalNets at the task of object detection. In the experiments, we use a context-aware CompNet-VGG16-pool4 and a CompNet-ResNext-RB3 because of their higher resolution compared to the other types of backbones tested in image classification.

Occluded-P3D+. In Table 5, we report the results of Faster-RCNN and CompositionalNets on the Occluded-P3D+-Vehicles-Detection dataset. The models are trained on the images from the original PASCAL3D+ dataset with non-occluded objects. Qualitative detection results are illustrated in Figure 11.

We observe that Faster R-CNN fails to detect strongly occluded objects reliably, while it performs well under low levels of occlusion. When trained with strong data augmentation in terms of partial occlusion using CutOut devries2017improved , the detection performance increases under strong occlusion. However, the model still suffers from a $59.9\%$ drop in performance on strong occlusion, compared to the non-occlusion setup. We suspect that the inaccurate prediction is because of two major factors. 1) The Region Proposal Network (RPN) in the Faster R-CNN is not able to predict accurate proposals of objects that are heavily occluded. 2) The VGG-16 classifier cannot successfully classify valid object regions under heavy occlusion.

We proceed to investigate the performance of the region proposals on occluded images. We conduct this experiment by replacing the VGG-16 classifier in the Faster R-CNN with a CompositionalNet classifier that is learned from the pool4 layer of VGG, which is expected to be more robust to occlusion. Based on the results in Table 1, we observe two phenomena. 1) In high levels of occlusion, the performance is better than Faster R-CNN. Thus, the CompositionalNet generalizes to heavy occlusions better than the VGG-16 classifier. 2) In low levels of occlusion, the performance is worse than Faster R-CNN.

The importance of spatial alignment in CompositionalNets. Recall that in the classification experiments, we observed that CompositionalNets are robust to occlusion because they can roughly localize occluders and subsequently focus on the non-occluded object parts for classification. They can localize occluders because the different components in the mixture models explicitly represent the spatial distribution of features of an object in a certain pose. To be successful, the localization process requires the features of the object in the image to be roughly aligned with the mixture models. Therefore, CompositionalNets require bounding box proposals in which the center of the object is roughly aligned to the center of the bounding box, independent of whether the object is occluded or not. In this sense, CompositionalNets are rather high precision models which require spatial alignment between image and model. This is in contrast to standard deep networks, which are observed to not use spatial information extensively and behave more like bag-of-words type models brendel2019approximating . The results in Section 6.2 and Section 6.4.1 show that the spatial distribution of object parts in an image is very important for computer vision models, because it enables the rough localization of occluders and thus a robust classification and detection under occlusion. One problem with the RPN proposals is that they mostly cover the visible part of the object only and often do not align the object center to the center of the bounding box.

Effect of robust bounding box voting. Our approach of accurately estimating the corners of the bounding box substantially improves the performance of the CompositionalNet, in comparison with the RPN. This further validates our conclusion that the CompositionalNet classifier requires precise proposals to classify objects correctly with partial occlusions.

Effect of context-aware representation. We observe that models with a reduced influence of the context ($\omega=0.2,\omega=0.0$) outperform models with equal weight on context and object representation ($\omega=0.5$). Hence, context-aware CompositionalNets are more robust to partial occlusion than unaware models. Furthermore, the performance of all models follows a similar trend over all three levels of foreground occlusions: the performance decreases as the level of background occlusion increases from BG-L1 to BG-L3. This further confirms our understanding of the effects of the context as a valuable source of information in object detection.

Occluded-MS-COCO. As show in Table 6 and Figure 11, the context-aware CompositionalNet with robust bounding box voting outperforms Faster R-CNN and CompNet+RPN significantly. Furthermore, the quantitative results clearly show the benefit of the context-awareness ($\omega=0.2$) over unaware CompositionalNets ($\omega=0.5$). While fully deactivating the context ($\omega=0$) slightly decreases the performance, controlling the prior of the context model to $\omega=0.2$ reaches a sweet spot where the context is helpful but does not have an overwhelming influence as the in the original CompositionalNet. Similar as observed in the classification experiments, CompNet-RXT-RB3 performs significantly worse compared to its pool4 variant for artificial occluders (Table 5), whereas it performs similarly under real occlusion (Table 6). We will discuss this phenomenon in more detail in Section 6.4.1.

While it is important that computer vision systems can robustly generalize to out-of-distribution examples in terms of partial occlusion, in real-world applications it is equally important that their prediction result is human interpretable. In this section, we show that CompositionalNets are highly interpretable models. We demonstrate that they can localize occluders accurately in image classification and object detection (Section 6.4.1), while being trained with class-level supervision only. Furthermore, we show that the predictions of CompositionalNets can be understood in terms of detecting object parts (Section 6.4.2) and estimating the objects’ viewpoint (Section 6.4.3).