A Generative Adversarial Framework for Optimizing Image Matting and Harmonization Simultaneously

Prepare Human Portrait Matting datasets: The original dataset we use is Matting Human Datasets [11] from Kaggle. The dataset has the real image $I$ and its corresponding alpha matte $A$. To make this dataset available to our framework, we use $A$ to segment out the foreground $F$ of $I$ and calculate the background $B$ through the inpaint function in OpenCv. Finally, we have four sub-sets: $I$, $A$, $F$, $B$.

Background Adjustment: In the background adjustment procedure, we randomly choose a transfer function to transfer the color, change the illumination degree, or enhance the color of the background image. In the procedure of the background adjustment, the color transfer method we use is Reinhard color transfer method [13] $R\left(\cdot\right)$. The color transfer function inputs are the background $B$ and a target image $I_{t}$ randomly selected from our training dataset, which is a different image that its background are different from the original background. The program extracts the target color from $I_{t}$ and transfers $B$ and generates $B_{f}$. This process is shown below:

Then $B_{f}$ will be recombined with the new foreground $F_{f}=A_{r}I$ as the disharmonious image $I_{d}$ as shown in equation 1. The process of the background adjustment are shown in Figure 3 in the supplementary materials.

IndexNet Generator. We employ IndexNet as the image matting network. The IndexNet can dynamically predict indices for individual local regions, conditional on the input local feature map itself. More details can be referred in [6]. We use alpha matte to generate trimap and mask, then we concatenate the real image and trimap and put them into the whole model. During training, we use the alpha prediction loss. Only loss from the unknown region of the trimap can be calculated. The output should be multiplied with mask $M$, and the trimap should be multiplied with $\bar{M}=1-M$. Then they added to generate result alpha $A_{r}$. We want the $A_{r}$ to be close to real alpha image $A$ through $\ell_{G_{m_{mse}}}=\|A_{r}-A\|_{1}$.

Attention Enhanced Generator. We employ a U-Net with attention blocks following [10] as our harmonization network. Different from [10], we enforce the generated image $I_{h}=G_{h}(I_{d},A_{r})$ to be close to real image $I$ by $\ell_{G_{h_{mse}}}=\|I_{h}-I\|_{1}$.

Self-attention Discriminator To generate a more natural composite image and optimize these two tasks together, we propose to use a self-attention discriminator [12] to evaluate the difference between synthesis images and real images. And the generators are required to produce more natural results to mimic the distribution of real images.

Dataset and Evaluation Metric. We perform our experiments on the conducted HMH dataset, which is build on Matting Human Datasets [11]. It is currently the largest portrait matting dataset, containing 34,427 images and corresponding alpha mattes. We split it into a training dataset with 30982 images and a testing dataset with 3444 images. During the test phase, the background dataset we use is the Scene UNderstanding (SUN) dataset [14], which contains 130,519 images. All the image for training and testing are resized into ${\rm{256}}\times{\rm{256}}$.

Implemention Details. For the matting network, we follow the training configurations used in [6] and pretrianed the matting network on Adobe Image Dataset. For the harmonization network, we also use the configuration and structure in [10] and pretrained the harmonization network on iHarmony4 dataset. Then we train IndexNet [6] on the conducted HMH dataset with the learning rate of $1e^{-7}$ for dconv, index and pred layers, and $1e^{-5}$ for other layers in fine-tuning stage. Finally, we jointly train the whole framework with the learning rate of 0.0002 for attention U-Net and self-attention discriminator, with the same learning rate in fine-tuning for IndexNet.

Compared with Single Methods. Previous models often work independently on image matting and image harmonization, leading to a sub-optimal. In Table 1, we show how the two tasks benefit from training in a GAN network. After IndexNet fine-tuned [6] on our dataset, the MSE error has reduced $78.8\%$, which indicate there exits a domain gap between Adobe Image Dataset and our human matting dataset. We trained our GAN framework on our dataset with the parameters of $\lambda_{1}=0.02,\ \lambda_{2}=0.01$. Compare with the baseline IndexNet, jointly training can improve the results of matting with 11.69% in MSE, 9.65% in SAD, and 10.1% in Conn. We also offer the three image matting qualitative results in Figure 2. We can see that the place circled by blue circles have holes on the matting result.

Compared with Existing Image Harmonization Methods. We also compare our model with exsiting image harmonization methods [16, 9, 10] qualitatively, and show the results in Figure 4 in supplemental material. From Figure 4, we can see that our method can produce more natural composite images compared with other methods.

Image Matting Result with Different $\lambda_{1}$. This section tests our image matting effect with different $\lambda_{1}$ parameters and with the same parameter $\lambda_{2}=0.01$. Then we compared our results alpha mattes with the ground truth alpha mattes pixel by pixel and shown the MSE results in Table 2. We can see that increasing the $\lambda_{1}$ value can improve the matting precision in a specific range. When $\lambda_{1}=0.02$, it has the minimum amount of MSE at present.

Image Harmonization Result with Different $\lambda_{2}$. We also test image harmonious effect with different $\lambda_{2}$, and with the same parameter $\lambda_{1}=0.01$. the qualitative image can be seen in Figure 5 in supplemental material. We can see as the parameter $\lambda_{2}$ increases, the background tends to change to a color closer to the foreground color and tends to become darker.

Input Foreground Alpha or Background Alpha Matte. We also do some experiments to adjust the color of foreground to meet with the background using the same methods. Here Figure 6 in supplemental material shows our algorithm’s effect sample. In some pictures, the faces of the portrait are very red and do not match the background. Our algorithm can produce more natural results when changing the foreground.