Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects

Deep learning-based colorization methods [1, 3, 2, 16, 23, 24, 4, 5] have proposed fully automatic colorization approaches without any additional conditions. These unconditional models predict the most plausible colors for the given input image, even without any laborious color annotations provided by a user. By leveraging conditions given by the user, the recent colorization methods have accomplished multi-modal colorization. One of the widely used conditions is a reference image [11, 8, 9]. However, a reference image containing visually different contents from the gray-scale often induces implausible results. On the other hand, a color palette or a scribble hint given by a user directly designates the user’s preference on both color and region [25, 6, 7]. However, as we observed in Fig. 2, both unconditional and conditional colorization often fail to preserve the color edge, resulting in generating color-bleeding artifacts along the boundary regions.

Classical approaches [14, 15] address the bleeding artifacts in an optimization problem. Huang et al. [14] develop an edge detection algorithm for improving edges information during colorization. Yin et al. [15] propose a sharpening filter applied over a colorized image, alleviating the bleeding artifacts via optimization. Similarly, Zhao et al. [16] propose a joint bilateral filter which considers the adjacent color values for sharpening the edges. However, as these approaches mainly rely on the edges of input image, they still fail on the boundaries between the objects that share the similar gray-scale intensities. In contrast, our approach involves a direct interaction, which refines the boundary representation in any region regardless of its edge pixel values in the input image.

Recently, Su et al. [5] and Zhao et al. [13] leverage the semantic segmentation and object detection, respectively, when training a colorization model. These tasks enforce the network to learn the semantic objects, which may help to recognize the boundary of such objects to colorize. However, since both semantic segmentation and object detection recognize the objects defined by particular classes, the methods may still suffer from the color bleeding along the objects that are not classified by any categories. For example, in Fig. 5, the color-bleeding artifacts across the patterns inside the balloon would not be fully addressed by these methods, as such patterns are not classified into a certain class. In our approach, we leverage the scribbles that are independent of object types, allowing more general edge enhancement against these methods.

This section provides a detailed description of our proposed method, as described in Fig. 4. As an interactive approach, we design an edge-enhancing network $E$ to take scribbles, which annotates the color-bleeding edges, as additional inputs. These scribbles, which we term pseudo-scribbles, are automatically generated to approximate the real-world user hints (Section 3.2). The network $E$ also takes the intermediate activation maps of the colorization network, and refines them along the edges annotated by the pseudo-scribbles. Afterward, these refined representations pass through the following layers to obtain an edge-enhanced colorized output in the end (Section 3.3). To train our model, we introduce an edge-enhancing loss, which enforces the model to recover the clear edges close to those of the ground-truth image. Also, we propose both feature-regularization loss and consistency loss to prevent undesirable color distortion which debases the overall quality of edge-enhanced outputs (Section 3.4).

Our approach requires a user-driven hint to be trained in an interactive manner, but collecting real-world user annotations needs a lot of time and human resources, which is prohibitive. Instead, we automatically generate the pseudo-scribbles $S_{\text{pseudo}}$, which emulate the real user scribbles, for each training image. The overall procedure of generating the $S_{\text{pseudo}}$ is presented in Fig. 3. First, we obtain the color-bleeding outputs $I_{\text{init}}$ from a pre-trained colorization model. Afterwards, we apply the Canny edge detector [18] $C(\cdot)$, a widely used edge detecting algorithm, onto the ab color channels of a ground-truth image $I_{\text{gt}}$ and a $I_{\text{init}}$, respectively (Fig. 3 (a)). Then, we can obtain the binary maps $C(I_{\text{gt},ab})$ and $C(I_{\text{init},ab})$ which represent the edges.

By selecting one of the edges that appears in $C(I_{\text{gt},ab})$, but not in $C(I_{\text{init},ab})$, we can have a single edge where the baseline fails to preserve boundary as clearly as $I_{\text{gt}}$ does (Fig. 3 (b)). Note that while a gecko’s tail is shown to be selected in (b), other scribbles can be chosen in the training as well, such as its paw. Afterward, to better approximate a real user’s hint, the $S_{\text{pseudo}}$ are formed to be thick and coarse enough to $1$) be easy to draw and $2$) contain the bleeding boundary. To this end, we apply a width transformation $w(\cdot)$ that randomly modifies the width of the selected edge between 1 and 11 pixels (Fig. 3 (c)).

We apply an edge-enhancing network $E$ to refine the intermediate representations of a colorization network, correcting the erroneously spread colors across the boundaries. This network encodes the corrective representations from both scribbles and the intermediate features as inputs and adds them to the original features with the residual connection. Suppose that we want to modify an activation map $A_{i}$, where $A=(A_{1},A_{2},...A_{l})$ is the set of intermediate activation maps from $l$ different encoder layers of the colorization model (Fig. 4 (a)). To obtain the scribbles, we generate a $S_{\text{pseudo}}$ by the procedure described in Section 3.2 and downscale it to match the spatial resolution of the activation map $A_{i}$ from the $i$-th layer (Fig. 4 (b)). Then, $S_{\text{pseudo}}$ and $A_{i}$ are concatenated to provide an input for edge-enhancing network $E$. Given the concatenated tensor, we can obtain an activation map representing the correction of $A_{i}$ to alleviate color-bleeding artifacts. Therefore, by applying a residual connection with $A_{i}$, (Fig. 4 (c)) refined activation map $\tilde{A}_{i}$ is calculated as

$$\tilde{A}_{i}=A_{i}+E\left(\left[S_{\text{pseudo}},A_{i}\right]\right),$$

(1)

where $E$ is the proposed edge-enhancing network and $[\cdot,\cdot]$ a concatenation.

We apply the edge-enhancing networks to the encoder because the edge-enhancing performance is empirically better when applying our network E to the encoder than the decoder. Detailed comparisons on the qualitative results of the network E applied in the encoder and decoder layer are provided in Section D. To encourage edge refinement in both low- and high-level representations, we apply multiple edge-enhancing networks in both shallow and deep layers of the encoder (Fig. 4 (d)).

The resulting color gradient is formally written as

$$\begin{split}&\mathcal{S}(I)=\sqrt{(G_{x}*I)^{2}+(G_{y}*I)^{2}},\\ &G_{x}=\begin{pmatrix}1&0&\leavevmode\hphantom{0}\hbox to0.0pt{\hss\leavevmode\resizebox{5.50005pt}{0.0pt}{$-$}}1\\ 2&0&\leavevmode\hphantom{0}\hbox to0.0pt{\hss\leavevmode\resizebox{5.50005pt}{0.0pt}{$-$}}2\\ 1&0&\leavevmode\hphantom{0}\hbox to0.0pt{\hss\leavevmode\resizebox{5.50005pt}{0.0pt}{$-$}}1\end{pmatrix},\hskip 2.84544ptG_{y}=\begin{pmatrix}1&2&1\\ 0&0&0\\ \leavevmode\hphantom{0}\hbox to0.0pt{\hss\leavevmode\resizebox{5.50005pt}{0.0pt}{$-$}}1&\leavevmode\hphantom{0}\hbox to0.0pt{\hss\leavevmode\resizebox{5.50005pt}{0.0pt}{$-$}}2&\leavevmode\hphantom{0}\hbox to0.0pt{\hss\leavevmode\resizebox{5.50005pt}{0.0pt}{$-$}}1\end{pmatrix},\end{split}$$

(2)

where $G_{x}$ and $G_{y}$ are horizontal and vertical Sobel filters which convolve with the given image, and $\mathcal{S}$ returns the gradient magnitude of them. Our proposed edge-enhancing loss can be written as

$$\begin{split}&\mathcal{L}_{\text{edge}}=\mathbb{E}_{x,y\in\mathbb{P}}\left[||S(x,y)-S_{\text{gt}}(x,y)||^{2}_{2}\right],\\ &S=\mathcal{S}(I_{ab}),\hskip 2.84544ptS_{\text{gt}}=\mathcal{S}(I_{\text{gt},ab}),\end{split}$$

(3)

where $\mathbb{P}$ denotes a set of coordinates $\left(x,y\right)$ whose values are non-zero in a binary mask $M$. $M$ only activates a set of pixels within certain distance from the target edge, i.e., $S_{\text{pseudo}}$.

In summary, the overall objective function for training the edge-enhancing network is defined as

$$\mathcal{L}_{\text{total}}=\lambda_{\text{edge}}\mathcal{L}_{\text{edge}}+\sum_{i=1}^{l}\lambda_{\text{reg}_{i}}\mathcal{L}_{\text{reg}_{i}}+\lambda_{\text{con}}\mathcal{L}_{\text{con}},$$

(6)

where $\lambda_{\text{edge}}$, $\lambda_{\text{reg}}$ and $\lambda_{\text{con}}$ are hyperparameters, and $l$ is the number of layers with the edge-enhancing network.

In our experiments, we use the colorization networks introduced in Zhang et al. [6] and Su et al. [5] as our backbone colorization models. Zhang et al. first proposes an interactive colorization approach that takes color hints, achieving state-of-the-art performance over the existing conditional methods. Su et al. achieves superior performance in an unconditional setting by leveraging an object detection module for an instance-level colorization. We empirically confirm that applying our framework to the baselines taking explicit color hints results in better optimized edge-enhancing networks, compared to training on unconditional ones. Therefore, similar to Zhang et al., we re-implement Su et al. to take local color hints as additional inputs, which is not available in the original paper. Further details are provided in Section I.

Although PSNR and LPIPS are generally used for evaluating the colorization performance in a rich literature [3, 6, 5, 4, 2, 9, 11, 16], they are essentially based on the color difference between a generated image and a ground-truth one. Therefore, a colorized image that contains well-preserved edges but different colors from the ground-truth may be underrated by these two metrics. Note that this image would appear even more realistic compared to a bleeding image with similar colors. This specific failure case is described with an example and its scores in Section C.

To compensate for this concern, we propose a novel evaluation metric that measures the discrepancy of color clusters grouped by the super-pixels defined by a simple linear iterative clustering method [28]. The super-pixels have their cluster assignments $C$ based on color similarity. Inspired by this, we can perform binary classification on whether two adjacent pixels with different cluster assignments in the ground-truth still have different color values in the colorized outputs, especially along the edges. Specifically, for the pixel $x_{\text{gt}}^{i}$ in a set of coordinates along the boundary $E$ of the ground-truth $I_{\text{gt},ab}$, we identify whether the adjacent pixel $x_{\text{gt}}^{j}$ within kernel size have different cluster assignments from that of $x_{\text{gt}}^{i}$. For those who have different cluster assignments from $C_{x_{\text{gt}}^{i}}$, the cluster index of $x_{\text{gt}}^{i}$, we define a set $\Omega(i)$ that consists of their coordinates. Then, in the generated outputs $I_{ab}$, we count the number of adjacent pixels $x^{j}$ that $1)$ belong to $\Omega(i)$ and $2)$ have the same cluster assignment as $C_{x^{i}}$. The number indicates how many pixels are from different clusters in the $I_{\text{gt},ab}$, but share the same colors in the $I_{ab}$, which corresponds to the color-bleeding artifacts. Therefore, the super-pixel-based cluster discrepancy ratio can be written as

$$\begin{split}&R\left(I_{0},I_{\text{gt}}\right)=\frac{1}{\lvert E\rvert}\sum_{i\in E}\left(1-\frac{1}{\lvert\Omega(i)\rvert}\sum_{k\in\Omega(i)}\mathbbm{1}_{C_{x^{i}}=C_{x^{i+k}}}\right),\\ &\Omega(i)=\left\{j:C_{x_{\text{gt}}^{i}}\neq C_{x_{\text{gt}}^{i+j}},j\in S\right\},\end{split}$$

(7)

where $E$ denotes a set of coordinates for the pixels of edges, $C_{x_{\text{gt}}}$ and $C_{x}$ a cluster assignment given to the super-pixels of the $I_{\text{gt},ab}$ and $I_{ab}$. All possible shifts $S$ within the kernel size $K$ are described as

$$S\coloneqq\left[-\left\lfloor\frac{K}{2}\right\rfloor,...,\left\lfloor\frac{K}{2}\right\rfloor\right]\times\left[-\left\lfloor\frac{K}{2}\right\rfloor,...,\left\lfloor\frac{K}{2}\right\rfloor\right].$$

(8)

In Fig. Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects, we visualize the images having color-bleeding artifacts from the conditional colorization model of Zhang et al. [6] and their enhanced results using our proposed method. This demonstrates that our approach robustly corrects the bleeding boundaries even when roughly drawn scribbles are given. In addition, multiple edges can be enhanced in a single feed-forward when their corresponding scribbles are given at once. Fig. 5 provides additional qualitative examples of edge enhancement in our approach applied to both the conditional and unconditional model of Zhang et al., and conditional of Su et al. [5] In Fig. 6, colorized images have the color-bleeding region for the baseline models. Comparing with other approaches, our method refines the coarse region. We present additional qualitative results of our approach with these two baselines in Figs. 15 and 16.

As the improvement in the colorization outputs induced by our approach arises in a local region along the edges, we present the results of PSNR and LPIPS measured in these particular locations, as well as in a global region. To conduct a local evaluation, we randomly sample $1,000$ $S_{\text{pseudo}}$ from the test samples of each dataset, which contain the regions of color-bleeding, as explained in Section 3.2. Afterward, we report the local scores measured within the kernel size of $7$ along the edge pixels of those $S_{\text{pseudo}}$. For global evaluation, we report the scores calculated on the entire region, specifying a kernel size as Full. To accommodate the scribbles similar to the real hints in our evaluation, we provide $S_{\text{pseudo}}$ with random widths ranging from 1 to 5 pixels for edge enhancement. Table 1 shows that our model effectively advances the colorization performance of two baselines, outperforming the existing methods. As bleeding effects are mostly mitigated in the local regions, large improvements are observed when the evaluation regions are localized along the edges. In addition, Table 2 presents the CDR measured for both our method and the baselines. Our method achieves a higher score against other baselines, demonstrating its superiority in colorizing the adjacent instances with different colors.

In Table 3, we ablate $\mathcal{L}_{\text{con}}$, $\mathcal{L}_{\text{reg}}$, and width augmentation technique to analyze their effectiveness quantitatively. When we ablate $\mathcal{L}_{\text{con}}$, overall performance on PSNR, LPIPS, and CDR is slightly degraded, which is mainly due to the color distortion in the wrong regions. Our method without $\mathcal{L}_{\text{reg}}$ results in degraded scores of PSNR and LPIPS while obtaining the best score in CDR. Since $\mathcal{L}_{\text{reg}}$ suppresses the excessive perturbations in the refined feature maps of our edge-enhancing network, removing this loss causes an excessive edge enhancement (e.g., saturated colors along the edge) as well as undesirable color distortions in the entire region. As we ablate the width augmentation for the $S_{\text{pseudo}}$ in the training, our approach achieves the best score in CDR, while PSNR and LPIPS score become even worse than Zhang et al. This implies that our edge-enhancing network tends to excessively enhance the color boundaries when unseen thick scribble is provided in the test time, ruining the overall colorization quality. Therefore, width augmentation plays a critical role in learning a robust color enhancing, invariant to the width of a given scribble. We support this claim with the qualitative results of the models with these objective functions ablated in Section B. In summary, our proposed method with every proposed objective function and augmented $S_{\text{pseudo}}$ achieves an optimal performance of edge enhancement.

We believe that our approach provides fast and reliable interactions, which help users to correct the color bleeding in an intuitive manner. To verify its usefulness with regard to labor efficiency, we conduct a user study with 13 novice participants using a user interface that we provide. Each participant is given five randomly selected colorized images that contain the color-bleeding artifacts. They are instructed to enhance the images by identifying color-bleeding areas and drawing scribbles until they obtain satisfactory results. On average, finding color-bleeding areas and drawing scribbles take $13.20\pm 8.46$ and $4.41\pm 3.08$ seconds per image, and each participant draws $2.9\pm 1.2$ scribbles to enhance an image. The resultant improvement of PSNR is from $23.7$ to $26.2$ and LPIPS from $0.14$ to $0.07$, indicating that participants provide meaningful edges for enhancement. These results demonstrate that our method have potentials to be applied in practical applications. We provide additional analysis on the robustness of our method across different users in Section A.