Discriminative feature encoding for
intrinsic image decomposition

Our work is inspired by Land et al. [2]: the retinex approach assumes that albedo and shading layers possess different properties in the gradient domain. By utilizing such discriminative properties, the intrinsic decomposition results can be improved. In this work, we study and exploit the discriminative properties in a more general convolutional feature space. We next describe the proposed discriminative feature encoding in detail.

As Figure 1 shows, the encoding phase consists of multiple (convolution, relu, maxpooling) blocks, through which the input signal is encoded into several different abstraction levels. The multi-scale features are denoted $\{\bm{f^{E_{1}}},\dots,\bm{f^{E_{n}}}\}$, in which $\bm{f^{E_{i}}}$ represents the output feature of the $i^{th}$ block. We define the feature distance function as $d:\mathbb{R}^{m\times n\times c}\times\mathbb{R}^{m\times n\times c}\mapsto\mathbb{R}$, where $c$ denotes the feature channel number and the input signal has spatial size $m\times n$:

Feature distance measurement is based on the cosine distance between two vectors and $L_{1}$ norm. In Eq. (4), $\bm{f_{a}}$ and $\bm{f_{s}}$ represent features from the albedo encoder and the shading encoder respectively. $<\cdot,\cdot>$ is the inner product in Euclidean space, $N_{i}=m_{i}\times n_{i}$;, and $(x,y)$ represents a spatial location in a feature map. $h(\cdot)$ is a distance rescaling function in the form of a modified sigmoid function to ensure $d_{L_{1}}\in(0,1)$. We use

The feature distribution divergence $\mathcal{L}_{\mathrm{fdd}}$ is now formulated as:

where $\omega_{i}\geq 0$ is the weight for the feature distance for abstraction level $i$. Empirically, we use five different levels of abstraction in our experiments ($n=5$). We set $[\omega_{1},\dots,\omega_{5}]=[0.01,0.1,0.5,0.7,1.0]$ and $\alpha=0.3$, $\beta=0.1$.

The feature distribution divergence aims to increase the distance between the feature vectors embedded by different encoders. However, this is not sufficient for discriminative feature encoding. The core idea of Fisher’s linear discriminant is to maximize the distance between classes and minimize the distance within classes simultaneously. As an analogue of that, along with the feature distribution divergence described above, we use the feature perceptual loss [55] between the predicted and ground truth intrinsic images to constrain the encoding process, encouraging the embedded features to fit the real distribution.

We use the same distance measurement as for the feature distribution divergence. $d(\bm{f_{\mathrm{pred}}^{E_{i}}},\bm{f_{\mathrm{real}}^{E_{i}}})$ denotes the feature distance in the $i_{th}$ abstraction level.

The feature distribution consistency $\mathcal{L}_{\mathrm{fdc}}$ is formulated as:

where $\gamma_{i}\geq 0$ is a weight. Note that $(1-d(\bm{f_{\mathrm{pred}}},\bm{f_{\mathrm{real}}}))\in(0,1)$ represents the feature similarity between $\bm{f_{\mathrm{pred}}}$ and $\bm{f_{\mathrm{real}}}$. Minimizing Eq. (6) encourages the predicted and ground truth intrinsic images to have similar perceptual features. In practice, the encoders are reused to extract features from the predicted and target results in our framework, so the embedded feature distribution can be optimized directly during training. Empirically, we set $[\gamma_{1},\dots,\gamma_{5}]=[1.0,1.0,1.0,1.0,1.0]$ and $\alpha=0.1$, $\beta=0.9$.

Besides the above constraints for discriminative feature encoding, several basic supervision losses are adopted to train the intrinsic image decomposition network.

As described in Eq. (3), given an image $I$, the albedo image $A$ and the shading image $S$ are predicted through trained $g_{a}\circ f_{a}$ and $g_{s}\circ f_{s}$. With densely-labelled intrinsic images $\widehat{A}$ and $\widehat{S}$ as ground truth data, we constrain the pixel-wise predictions using the reconstruction loss $\mathcal{L}_{\mathrm{rec}}$ and the gradient loss $\mathcal{L}_{\mathrm{grad}}$.

We use the $L_{1}$ loss $\mathcal{L}_{L_{1}}$ combined with the SSIM (structural similarity index [56]) loss $\mathcal{L}_{\mathrm{SSIM}}$ as the reconstruction loss:

where $\mathrm{SSIM}(\bm{x},\bm{y})$ measures the structural similarity of images $\bm{x}$ and $\bm{y}$. The SSIM loss is $(1-\mathrm{SSIM}(\bm{x},\bm{y}))$, indicating structural dissimilarity. Empirically, we set $\lambda_{L_{1}}=30.0$ and $\lambda_{\mathrm{SSIM}}=0.5$. $A\times S$ is computed pixel-wise.

The cycle loss is used to encourage the product of the predicted $A$ and $S$ to be similar to that of the input image $I$.

We also use image gradients as supervision to help preserve details in the intrinsic images:

where $\nabla_{x},\nabla_{y}$ are the image gradients in the $x$ and $y$ directions.

For datasets with ground truth decomposition like the MIT intrinsic dataset and MPI Sintel, the total loss is constructed as:

Empirically, we set $[\lambda_{1},\dots,\lambda_{4}]=[1.0,1.5,0.1,1.0]$.

As well as the densely-labelled datasets mentioned above, recently, the sparsely-labelled dataset IIW has become available, with a larger number of real-world images. To apply our core idea to this situation, we must slightly adjust the training framework, as we first explain, and then give the loss functions measuring the intrinsic image reconstruction quality.

In order to train on the sparsely-labelled IIW dataset, there are a few barriers to overcome. Figure 3 shows the adjustments made to our framework.

One problem is the lack of dense supervision during the albedo reconstruction procedure. The reconstruction loss in Eq. (7) and the gradient loss in Eq. (8) both need pixel-wise dense supervision. However, the IIW dataset only has sparse annotations of reflectance comparisons at selected points of the images. Therefore, we have to apply alternative constraints utilizing sparse labelling. Specifically, we use the ordinal loss to measure the difference between the output albedo image and the annotated input. Furthermore, smoothness constraints are applied to model the smoothness prior of the albedo component.

A further problem is the lack of shading ground truth, which is necessary in our two-stream training framework. The core idea of the feature distribution divergence is to extract distinctive features corresponding to different target intrinsic layers from the same input image. However, there is no annotation for the shading layers in this dataset. To circumvent the lack of ground truth data, we directly synthesize the target shading image from the input $I$ and the reconstructed albedo $A$, using Eq. (1). Therefore, the synthesized shading image can be used as dense supervision.

Last but not least, the lack of reference intrinsic images causes problems for computing the feature distribution consistency. The FDC is designed to constrain the intrinsic image to have a similar distribution to the corresponding real reference. In detail, this constraint is achieved by minimizing the feature perceptual loss between the reconstructed and reference images. However, dense ground truth images are not provided in the IIW dataset, making ground truth features unavailable. To solve this problem, we maintain an image pool for each training stream, in which batches of reconstructed intrinsic images are gathered as the guidance for feature perceptual loss.

As noted, we thus modify the constraints to suit the sparse annotations in the IIW dataset. We now describe the ordinal loss and the smoothness constraints used.

We first consider the ordinal loss. Since dense ground truth labels are not available in [12], it introduced the weighted human disagreement rate (WHDR) as an error metric. Following [17], we use an ordinal loss based on WHDR as the sparse supervision term.

The ordinal loss $\mathcal{L}_{\mathrm{ord}}$ is obtained by accumulating all the annotated pairs in the albedo image:

where $e_{i,j}(A)$ represents the error for a pair of annotated pixels $(i,j)$ in the predicted albedo image $A$. A detailed definition is provided in Appendix A.

We next consider smoothness priors: we adopt the same ones as [17]. The smoothness constraints comprise the albedo smoothness constraint $\mathcal{L}_{\mathrm{asmooth}}$ and the shading smoothness constraint $\mathcal{L}_{\mathrm{ssmooth}}$. The albedo component is constrained using a multi-scale $L_{1}$ smoothness term, through which the albedo layer reconstruction is encouraged to be piecewise constant. Shading smoothness is constrained using a densely-connected $L_{2}$ term. Detailed definitions are provided in Appendix A.

The total loss for sparsely-labelled data is:

where $\mathcal{L}_{\mathrm{total}}^{\backslash\hat{A}}$ means the total loss function $\mathcal{L}_{\mathrm{total}}$ for densely-labelled data in Eq. (9), removing the terms containing dense albedo ground truth $\hat{A}$. $\lambda_{\mathrm{As}}$ and $\lambda_{\mathrm{Ss}}$ weight the smoothness priors. A study considering different values of $\lambda_{\mathrm{As}}$ and $\lambda_{\mathrm{Ss}}$ is provided in V-B3.

Sintel is an open source 3D animated short film, which has been published in many formats for various research purposes. For intrinsic image decomposition, the clean pass images and the corresponding albedo and shading layers have been published as the MPI Sintel dataset, containing 18 sequences with a total of 890 frames. As discussed in IV, there is severe illumination inconsistency between the input frames and the shading layers in this dataset. Therefore, we provide the refined MPI Sintel dataset as a more suitable dataset for intrinsic image decomposition.

Figure 4(bottom) compares our refined MPI dataset (MPI_RD) to the original MPI dataset (MPI). In the shading layer of the first column, we can see that in our refined shading image, the specularity on the shoulder of the girl is removed, making the shading illumination consistent with the original input image. In the second and third columns, the shading layers from the MPI_RD dataset contain more geometric details than those from the MPI dataset. For instance, the wooden cart’s coarse surface is depicted in the refined shading in the third column, while the original shading from the MPI dataset has a smooth surface. These examples demonstrate that our refined MPI_RD improves consistency between the intrinsic decomposition and the input image. In Figure 4(bottom, right), the mean squared error (MSE) between the input image $I$ and the resynthesized image $A\times S$ is computed. The MSE for the MPI_RD dataset is significantly smaller than for the MPI dataset, showing that the intrinsic decomposition model Eq. (1) is well respected in the refined dataset.

For data augmentation, we randomly resize the input image by a scale factor in $[0.8,~{}1.3]$, and randomly crop a $288\times 288$ patch from the resized image per iteration. We also use horizontal flipping in the training phase. When comparing methods, following [16], we evaluate our results on both a scene split and an image split. For a scene split, half of the scenes are used for training and the other half for testing. For an image split, all 890 images are randomly separated into two sets. Evaluation on a scene split is considered more challenging as it requires more generalization capacity.

For video data, the refined version MPI_VRD (video refined data) alleviate image illumination jitter and local region inconsistencies. Quantitatively, taking the shading images of video sleeping_2 for example, temporal consistency refinement increases TCM from 0.74 to 0.78.

Intrinsic Images in the Wild (IIW) [12] is a large scale, public dataset of real-world scenes intended for intrinsic image decomposition. It contains 5,230 real images of mostly indoor scenes, combined with a total of 872,161 crowd-sourced annotations of reflectance comparisons between pairs of points sparsely selected throughout the images (on average 100 judgements per image). Following many prior works [13, 41, 39, 16], we split the IIW dataset by placing the first of every five consecutive images sorted by image ID into the test set, and the others into the training set. WHDR from [12] is employed to measure the quality of the reconstructed albedo images.

For the IIW dataset, our proposed network structure cannot be directly used due to the lack of dense labelling of albedo and shading layers. Actually, only sparse and relative reflectance annotations are provided. In order to take advantage of the proposed feature distribution divergence and feature distribution consistency, we modify the network. In detail, the predicted dense albedo is collected into an image pool to describe the distribution of albedo. The reconstructed shading using the original image and predicted albedo is used as dense supervision for shading prediction, and is also collected in an image pool to describe the shading distribution. We set the weights in Eq. (6) to $[\gamma_{1},\dots,\gamma_{5}]=[0,~{}0,~{}0,~{}1.0,~{}1.0]$.

As Table I shows, our method achieves the best results on the MPI Sintel dataset using the image split. On the more challenging scene split, our method is competitive with the state of the art, and achieves the best results in 5 out of the 9 cases in the table. We show a group of qualitative results evaluated on the scene split in Figure 5. While the MSCR [14] results are relatively blurred due to the large kernel convolutions and down-sampling, our method provides sharper results comparable to Revisiting [16]. Moreover, our shading layer depicts better shadow area than [16].

As explained in IV, the MPI Sintel dataset has issues of data consistency between the original input images and the corresponding shading images. Because of the proposed feature distribution divergence, feature distribution consistency and the use of the cycle loss, our method is sensitive to such data inconsistency. Therefore, we compare our method to the state-of-the-art methods on the more challenging scene split of the refined MPI Sintel dataset. As Table II shows, our method achieves the best result, demonstrating the effectiveness of our method and data refinement process.

To further validate the effectiveness of the proposed method, we also conducted an ablation study on the training loses as well as the network architecture. Results are given in the bottom part of Table II. ‘Plain’ represents the baseline two-stream network structure shown in Figure 2(b). In the ablation study, the gradient loss and SSIM loss are progressively added to train the plain network. The experimental results show that using the gradient loss and SSIM loss simultaneously achieves better intrinsic image decomposition results. The proposed network architecture is denoted by ‘Ours’, as defined in Figure 2(c). It can be observed that using only the feature distribution divergence (w/o FDC) or the feature distribution consistency (w/o FDD) does not improve results much, while using both of them results in considerable improvement.

Figure 6 displays a side-by-side comparison with two other methods using the refined dataset MPI_RD. As can be seen, our method better separates shading from albedo information. For example, our method outputs consistent shadow around the girl’s neck.

We further experimented on the MIT intrinsic dataset [10], which consists of object-level real images. In this experiment, classical methods [7, 35, 33] as well as learning based methods [16, 15, 14, 39] were compared to our proposed method. We also conduct an ablation study on the training strategies of our proposed network, including training from scratch (Ours scratch), pre-training on the original MPI Sintel dataset (Ours MPI), and pre-training on the refined MPI Sintel dataset (Ours RD). The experimental results are reported in Table III, and representative instances are selected for visual comparison in Figure 7.

As in [16], we used the 220 images in the dataset. In comparisons to previous methods, the split from [7] was used. Our refined MPI Sintel dataset has grayscale shading images, so we first pre-train the model on MPI_RD and then fine-tuned it on the MIT training set.

Numerical results are shown in Table III. Our method (Ours RD) achieves the best results in most cases in the table. Moreover, Ours RD performs better than Ours MPI in terms of LMSE, and both pre-training methods perform better than training from scratch, demonstrating that pre-training on the refined MPI Sintel dataset helps intrinsic image decomposition on the MIT dataset.

Qualitative results are illustrated in Figure 7. We can observe that our method (Ours RD) predicts sharp and accurate intrinsic layers. Classical methods may produce meaningful layer separation results, but good results depend on parameter tuning, which requires expert knowledge. Compared to (Ours scratch) and (Ours MPI), (Ours RD) achieves better region consistency with the ground truth.

In Table IV, we report results obtained using the test set of the IIW dataset. Our proposed method achieves the best performance with a mean WHDR value of $13.60\%$, a considerable improvement over the second best [16] with a mean WHDR value of $14.45\%$. To better show the quality of our results, we give a qualitative comparison to state-of-the-art methods in Figure 8. Detailed intrinsic decomposition results are shown in the close-up windows. It can be observed that our method successfully preserves the texture of the floor tiles in the albedo layer, while the other approaches treat such texture as shading.

To further investigate the influence of the smoothness priors applied in the sparsely-labelled case, a comparison was conducted. Figure 9 shows the effects of changing the weights of the smoothness terms in Eq. (11): the albedo smoothness weight $\lambda_{\mathrm{As}}$ and shading smoothness weight $\lambda_{\mathrm{Ss}}$ were varied from 0 to 4 respectively. Results show that using the shading smoothness term increases the reconstruction loss, while using the albedo smoothness term decreases the reconstruction loss. Using both simultaneously provides the best result. In Figure 9(c, d, e), three representative parameter settings are used to show the resulting intrinsic decomposition. The albedo image in Figure 9(e) contains more precise contours and consistent region colors.

In this experiment, we evaluated the proposed intrinsic decomposition method on video data with respect to reconstruction quality of the intrinsic images, and to temporal consistency, through comparisons to alternative methods.

Framewise methods used for comparison include MSCR [14] and Revisiting [16]. The blind video temporal consistency method [52] is an unsupervised video smoothing method. In the experiment (Ours+DVP), it is applied as post-processing to increase the temporal consistency of the results produced by our framewise method (Ours). (Ours+Flow) is the proposed intrinsic decomposition method for video data. The temporal consistency constraint is applied by taking optical flow as input to construct correspondences between adjacent frames. We also extend the temporal consistency constraint to MSCR in the same way as for (Ours+Flow) in (MSCR+Flow). All methods were trained and tested on the MPI_VRD dataset.

The intrinsic decomposition accuracy scores of framewise methods and flow methods are shown in Table V, while the temporal consistency scores for 8 test short videos are shown in Table VI. Using adjacent frame temporal consistency as a constraint, (Ours+Flow) and (MSCR+Flow) achieve much better temporal consistency metric scores than their framewise counterparts, while the intrinsic decomposition accuracy scores remain almost unchanged. This demonstrates the effectiveness of our proposed extension method for video data. As a result, (Ours+Flow) achieves the best average accuracy and temporal consistency for intrinsic decomposition of video data.

We also visualize temporal consistency of a specific frame clipped from a video, using a temporal inconsistency metric (TICM) map to highlight temporally inconsistent areas. First, the TCM map is computed using Eq. (14). Then, the TCM map is smoothed by a Gaussian kernel of size 65. Finally, the Jet colormap is inverted to highlight inconsistent areas with warm colors (the colder the color, the better the consistency). A qualitative comparison of different intrinsic decomposition methods on video data is shown in Figure 10. It can be observed that, using the temporal consistency constraint, both (MSCR+Flow) and (Ours+Flow) achieve better temporal consistency compared to their framewise counterparts (MSCR and Ours).

In Figure 11, temporal consistency loss is added to framewise methods (MSCR) and (Ours). In the (Ours+Flow) result, temporally inconsistent areas are suppressed compared to (Ours). However, in the (MSCR+Flow) result, temporal inconsistency is unexpectedly amplified in the highlighted area in the red box. In addition, the (MSCR+Flow) result is more blurred than the one without temporal consistency loss. In MSCR, multiple scales of feature maps are merged in the encoding phase. While features from deeper layers contain high-level knowledge, they may lack texture details. The temporal consistency constraint may encourage MSCR to pay more attention to high-level features, resulting in more blurred outputs. The case of (MSCR+Flow) indicates that while temporal consistency loss could be easily extended to other deep neural networks, the outcome will largely depend on the characteristics of specific methods.