Shadow Generation with Decomposed Mask Prediction and Attentive Shadow Filling

Image composition refers to cutting out a foreground object from one image and pasting it on another background image to get a composite image. The issues that make the obtained composite images unrealistic can be summarized as appearance, geometry, and semantic inconsistencies (Niu et al. 2021). Lots of works are devoted to solving one or more types of inconsistencies. For instance, object placement methods (Zhang et al. 2020b; Liu et al. 2021; Lee et al. 2018) solved the geometric inconsistency and semantic inconsistency by adjusting the scale, location, and shape of the foreground object. To address the appearance inconsistency, image harmonization methods (Cong et al. 2022; Guo et al. 2021; Ling et al. 2021) adjusted the illumination statistics of foreground, while image blending methods (Zhang et al. 2021; Zhang, Wen, and Shi 2020; Wu et al. 2019) attempted to blend the foreground and background seamlessly. In this work, we target at the missing foreground shadow, which also belongs to appearance inconsistency.

Although there are lots of works on shadow detection (Ding et al. 2019; Chen et al. 2020; Wang et al. 2021) and removal (Zhang et al. 2020a; Le and Samaras 2020; Zhu et al. 2022; Guo et al. 2023), there are few works on shadow generation. Existing works on shadow generation can be divided into two groups according to their adopted technical routes: rendering based methods and image translation methods.

In this work, we propose novel decompose mask prediction and attentive shadow filling to achieve better transferability and better visual effects.

We use Unity-3D to construct 3D scenes and render images. There are abundant freely available 3D models and 3D scenes online. We collect 800 standard 3D objects from different categories (e.g., people, vehicles, plants, animals) from CG websites, and 30 representative 3D scenes (e.g., schools, streets, grass) from Unity Asset Store and CG websites. The collected 3D objects and 3D scenes cover a wide range of 3D geometry and object/scene categories, which lays the foundation for generating diverse rendered images.

For each 3D scene, we select 20 open areas to place 3D objects, then choose 10 suitable camera settings (e.g., camera position and viewpoint) for each area. After fixing the camera, we place a group of 3D objects (randomly select 1 to 5 objects from collected 800 objects) in the field of view of camera. For each camera setting, we place 10 groups of 3D objects. With the camera and objects ready, we choose 5 different illumination conditions (e.g., illumination direction, intensity) and render a set of 2D images. Therefore, we can render $30\times 20\times 10\times 10\times 5=300,000$ sets of images.

Once we set up an open area, camera settings, 3D object group, and lighting in a 3D scene, we generate a series of images. First, we render an image $I_{empty}$ before placing objects. After placing a group of $K$ 3D objects, we switch off the visibility of all objects to the camera and switch on the visibility of each object one by one. When switching on the visibility of the $k$-th object, we can choose whether to render shadow for this object in Unity-3D. Without the shadow, we render an image $I_{o,k}$ with the $k$-th object. With the shadow, we render an image $I_{os,k}$ with the $k$-th object and its shadow. Based on $I_{empty}$ and $I_{o,k}$, we can obtain the $k$-th object mask $M_{o,k}$ by calculating their difference. Similarly, based on $I_{o,k}$ and $I_{os,k}$, we can obtain the $k$-th shadow mask $M_{s,k}$. At last, we switch on the visibility of all objects and render an image $I_{g}$ with the shadows of all objects.

Among $K$ 3D objects, when we choose the $k$-th object as the foreground object, we can use $M_{o,k}$ (resp., $M_{s,k}$) as the foreground object (resp., shadow) mask $M_{fo}$ (resp., $M_{fs}$). We can merge $\{M_{o,1},\ldots,M_{o,k-1},M_{o,k+1},\ldots,M_{o,K}\}$ as the background object mask $M_{bo}$. Similarly, we can get the background shadow mask $M_{bs}$. Then, we calculate $I_{c}$ by $I_{c}=I_{o,k}*(1-M_{bs}-M_{bo})+I_{g}*(M_{bs}+M_{bo})$, in which $*$ denotes element-wise multiplication. Up to now, we have obtained data tuples whose form is identical to DESOBA’s. However, not all the tuples are of high quality because some shadow masks are incomplete or erroneous. After manually filtering out low-quality tuples, we have 280,000 tuples left. We have shown some examples in Figure 1 and more examples can be found in Supplementary.

Although our dataset construction requires certain manual efforts, it is still much more efficient than manually removing the shadows when constructing DESOBA dataset.

In the first stage, our goal is to predict the foreground shadow mask $\hat{M}^{\prime}_{fs}$. We use a CNN encoder $E_{c}$ to extract the bottleneck feature map $F_{e}$ with size $16\times 16$ from the concatenation of $\{I_{c},M_{bs},M_{bo},M_{fo}\}$ with size $256\times 256$.

Based on $F_{e}$, we predict the mask shape and bounding box separately, similar to instance segmentation method Mask-RCNN (He et al. 2017). Unlike instance segmentation where the instance already exists in the input image, we need to imagine the shadow’s shape and bounding box, which is much more challenging than instance segmentation.

Intuitively, by observing background object-shadow pairs, we can roughly estimate the relative scale and location offset of shadow compared with the inserted object. Therefore, we predict the regression from the bounding box of foreground object to that of foreground shadow. We use a quadruplet $B=(x,y,w,h)$ to describe a bounding box, where $(x,y)$ are the center coordinates of the bounding box and $w$ (resp., $h$) is the width (resp., height) of the bounding box. The bounding box of foreground object is denoted as $B_{o}=(x_{o},y_{o},w_{o},h_{o})$ and the ground-truth bounding box of foreground shadow is denoted as $B_{s}=(x_{s},y_{s},w_{s},h_{s})$. Then, we use another quadruplet $r=(r_{x},r_{y},r_{w},r_{h})$ to characterize the regression from $B_{o}$ to $B_{s}$:

$\displaystyle\left\{\begin{array}[]{l}r_{x}=(x_{s}-x_{o})/w_{o},\\ r_{y}=(y_{s}-y_{o})/h_{o},\\ r_{w}=ln(w_{s}/w_{o}),\\ r_{h}=ln(h_{s}/h_{o}).\\ \end{array}\right.$

(5)

We use box head $H_{b}$ to predict quadruplet $\hat{r}$, then calculate the predicted $\hat{B}_{s}$ by Eqn. 5. The aspect ratio and position may change significantly when mapping the object bounding box to the shadow bounding box. To consider the coordinate correlation and enhance the robustness, we employ CIoU loss (Zheng et al. 2020) to supervise $\hat{r}$:

$\displaystyle L_{reg}(\hat{r},r)=CIoU(\hat{B}_{s},B_{s}).$

(6)

For shape prediction, we do not consider position and aspect ratio handled in $H_{b}$, so we generate shape mask within a standardized box for each shadow. For images with size $256\times 256$, the mean size of shadow bounding boxes is close to $32\times 32$, so we use the shape head $H_{s}$ to predict a mask $\hat{M}_{s}$ with size $32\times 32$. We use $\xi_{B}(I)$ to denote the operation of cropping the bounding box $B$ from image $I$ and resizing it to $32\times 32$. Inversely, $\xi_{B}^{-1}(I)$ denotes the operation of placing the resized $I$ within the bounding box $B$ and padding zeros outside the bounding box. In the training phase, the ground-truth shape mask can be obtained by $M_{s}=\xi_{B_{s}}(M_{fs})$. We calculate $L_{1}$ loss between $M_{s}$ and $\hat{M}_{s}$:

$\displaystyle L_{shape}(\hat{M}_{s},M_{s})=L_{1}(\hat{M}_{s},M_{s}).$

(7)

After obtaining $\hat{B}_{s}$ and $\hat{M}_{s}$, we can get a rough shadow mask by $\hat{M}_{fs}=\xi_{\hat{B}_{s}}^{-1}(\hat{M}_{s})$. $\hat{M}_{fs}$ has roughly correct location, scale, and shape, but lacks fine-grained details. To compensate for detailed information, we employ a decoder $D_{t}$, which takes in the bottleneck feature map $F_{e}$ to produce the up-sampled feature map $F_{t}$. Then, $F_{t}$ is concatenated with $\{\hat{M}_{fs},M_{fo}\}$ and passed through several convolutions layers to yield the refined mask $\hat{M}^{\prime}_{fs}$. Again, we use L1 loss to supervise $\hat{M}^{\prime}_{fs}$:

$\displaystyle L_{mask}(\hat{M}^{\prime}_{fs},M_{fs})=L_{1}(\hat{M}^{\prime}_{fs},M_{fs}).$

(8)

In the second stage, we aim to fill the predicted foreground shadow region. To guarantee that the filled foreground shadow looks compatible with background shadows, one naive approach is calculating the mean value of background shadow pixels as the target mean value of foreground shadow pixels. However, the values of background shadow pixels vary widely and the provided background shadow masks are not perfectly accurate. Hence, we attempt to select reference background shadow pixels by learning different weights for different background shadow pixels.

We apply another encoder $E_{s}$ to the concatenation of $\{I_{c},M_{bs},M_{bo},M_{fo}\}$, and concatenate multi-scale encoder features as $F_{s}$. Then we average the pixel-wise feature vectors within $\hat{M}^{\prime}_{fs}$ as $f_{fs}$. Besides, we denote the pixel-wise feature vector for the $i$-th background shadow pixel as $f_{bs,i}$. After that, we project $f_{fs}$ and $f_{bs,i}$ to a common space via a fully-connected layer $\phi(\cdot)$ and calculate their similarities to produce the attention map $A=\{a_{1},a_{2},\ldots,a_{N_{bs}}\}$, in which $N_{bs}$ is the number of background shadow pixels and $a_{i}=\frac{\exp\left(\phi(f_{fs})^{T}\phi(f_{bs,i})\right)}{\sum_{i=1}^{N_{bs}}\exp\left(\phi(f_{fs})^{T}\phi(f_{bs,i})\right)}$. Finally, we calculate the weighted average of background shadow pixels as $\bar{p}_{bs}=\frac{1}{N_{bs}}\sum_{i=1}^{N_{bs}}a_{i}p_{bs,i}$, in which $p_{bs,i}$ is the color value of the $i$-th background shadow pixel.

Then, we calculate the average of pixel values within the predicted foreground shadow region $\hat{M}^{\prime}_{fs}$ as $\bar{p}_{fs}$. To match the target mean value $\bar{p}_{bs}$, we can get the scale $\bar{p}_{bs}/\bar{p}_{fs}$ for the foreground shadow region. In implementation, we scale the whole composite image $I_{c}$ by $I_{dark}=\bar{p}_{bs}/\bar{p}_{fs}*I_{c}$ and obtain the target image $\hat{I}_{g}$ by

$\displaystyle\hat{I}_{g}=\hat{M}^{\prime}_{fs}*I_{dark}+(1-\hat{M}^{\prime}_{fs})*I_{c}.$

(9)

We employ shadow-MSE loss to supervise the shadow color for the predicted target image:

$\displaystyle L_{rec}(\hat{I}_{g},{I}_{g})=MSE(\hat{I}_{g}*M_{fs},{I}_{g}*M_{fs}).$

(10)

The whole network can be trained in an end-to-end manner, and the overall loss function can be written as

$\displaystyle L_{total}=L_{reg}+L_{shape}+L_{mask}+L_{rec}.$

(11)

Our RdSOBA dataset has 280,000 training tuples. DESOBA dataset has 2792 training tuples and 580 testing tuples. All images are resized to $256\times 256$. We implement our model using PyTorch and train our model on 4*RTX 3090 with batch size being 16. We use the Adam optimizer with the learning rate being 0.0001 and $\beta$ set to (0.5,0.999). We train RENOS for 50 epochs and DESOBA for 1000 epochs without using data augmentation. Further details of our network can be found in the Supplementary.

Following (Hong, Niu, and Zhang 2022), we pick Pix2Pix (Isola et al. 2017), Pix2Pix-Res, Mask-ShadowGAN (Hu et al. 2019), ShadowGAN (Zhang, Liang, and Wang 2019), ARShadowGAN (Liu et al. 2020), and SGRNet (Hong, Niu, and Zhang 2022) as baselines. Pix2Pix is a typical image-to-image translation method. Pix2Pix-Res is a simple variant of Pix2Pix. Mask-ShadowGAN performs shadow removal and mask-guided shadow generation simultaneously, in which the shadow generation network can be adapted to our task. ShadowGAN, ARShadowGAN, and SGRNet work on the same task as ours and thus can be directly applied.

For comprehensive comparison, we adopt three groups of evaluation metrics.

The first group of metrics are calculated between the ground-truth and the predicted images. We adopt RMSE (resp., shadow-RMSE) and PSNR (resp., shadow-PSNR), that computed on the whole image (resp., ground-truth foreground shadow region). These metrics are valuable, but may not conform to human perception of visual quality.

The second group of metrics are calculated between the ground-truth masks and the predicted masks. We adopt balanced error rate (BER) (resp., shadow-BER), which is computed based on the whole image (resp., ground-truth foreground shadow region). We observe that BER is relatively more consistent with human perception of visual quality.

In the third group, considering that undesired holes and isolated fragments seriously affect human perception without significantly affecting the abovementioned metrics , we design two heuristic metrics to measure the quality of predicted masks. In particular, we calculate the difference between the original mask and that after filling the holes (resp., removing isolated fragments) using the functions in skimage.morphology¹¹1remove_small_objects and remove_small_holes. We set connectivity=2 and area_threshold=50, which is referred to d_hole (resp., d_frag). To judge the quality of predicted masks, we should refer to the values of d_hole and d_frag of ground-truth masks.

In this section, we only train and evaluate different methods on DESOBA dataset. We show the qualitative results in in Figure 3 and report the quantitative results in Table 1. Since some baselines do not predict the foreground shadow mask, we only calculate the metrics in the first group.

As shown in Figure 3, only our DMASNet and SGRNet can generate plausible shadows, while other methods usually generate terrible shadows. For the quantitative results in Table 1, although our DMASNet is worse than SGRNet on these metrics, the quality of generated images by our DMASNet is comparable with or better than SGRNet. For example, our DMASNet can generate shadows with more compact shapes (e.g., row 4) or more proper intensity values (e.g., row 1). The mismatch between quantitative evaluation and qualitative evaluation motivates us to include more metrics for more comprehensive comparison.

In this section, we only compare with SGRNet because the performances of other baselines are poor as demonstrated in Section 5.4. We conduct experiments in three settings: 1) only using RdSOBA training set; 2) only using DESOBA training set; 3) using both RdSOBA training set and DESOBA training set (first training on RdSOBA, then finetune on DESOBA). For all settings, the evaluation is performed on DESOBA test set. The full comparisons between DMASNet and SGRNet in three settings are summarized in Table 2. We show the advantages of our method from three aspects.

Our ultimate goal is generating realistic foreground shadow for real composite images, so we compare different methods on the 100 real composite images provided by (Hong, Niu, and Zhang 2022). We compare our DMASNet with SGRNet as well as relatively strong baselines Pix2Pix-Res and ShadowGAN (see Table 1). For DMASNet and SGRNet, we provide two versions depending on whether using RdSOBA dataset or not. As there are no ground-truth images, we report the metrics d_hole, d_frag and conduct user study. We only report d_hole, d_frag for DMASNet and SGRNet, because the other two methods do not predict mask.

In our user study, we invite 20 people to observe a pair of generated images by two methods at a time and ask them to choose the one with more realistic foreground shadow. Based on pairwise scores, we use Bradley-Terry(B-T) model (Bradley and Terry 1952) to calculate the global ranking score for each method. We list the results in Table 3 and show the generated images by different methods in Figure 5. We can see that our DMASNet can greatly benefit from RdSOBA and consistently produce more realistic results, while SGRNet fails to produce shadow in many cases.