Manga Rescreening with Interpretable Screentone Representation

We observed that the diversity of screentones with different intensities has the following properties. Firstly, the domain conditioned on intensity should be a symmetric space, as each pattern can transform into a new pattern with opposite intensity by inverting the black and white pixels. Second, the screentone diversity is correlated with its intensity. The diversity gradually decreases when annihilated into a pure black or pure white pattern from 50% intensity, as shown in Fig. 5(b). In particular, when the intensity is 100% or 0%, there is no variation of screentones. Considering the above properties, we propose to model the domain as a hypersphere with intensity as one independent axis, as illustrated in Fig. 5(a).

Intensity-Aware Hypersphere Mapping (IAHM) is proposed to achieve the modeling. Considering the diversity of the screen types is conditioned by the intensity information, we hereby model the types of screentones with the same intensity as a normal distribution $\mathcal{N}(\textbf{0},r^{2}\cdot\textbf{I})$ instead of projecting all screentones to a standard normal distribution in high dimensions. Meanwhile, we force the embedding domain to be hyperspherical with constraints of $r=f(S_{\rm itn})\sim\sin(\pi\times S_{\rm itn})$. As we can have $\frac{1}{r}\cdot S_{\rm scr}^{\prime}\sim\mathcal{N}(\textbf{0, I})$ with $S_{\rm scr}^{\prime}\sim\mathcal{N}(\textbf{0},r^{2}\cdot\textbf{I})$, our Intensity-Aware Hypersphere Mapping (IAHM) is then defined as

where $\odot$ indicates point-wise multiplication. In particular, it is deterministic with $r=0$ at black and white intensity.

The IAHM encoding substantially improves the usage efficiency of latent space, which helps to avoid the model bias due to imbalanced varieties of different intensities. As shown in Fig. 6, the model without IAHM will not be able to generate screentones with darker or lighter tone intensity. Specifically, for some screentones with darker or lighter intensity, although they may be quite similar at the pixel level, they may have a large distance in the latent space without the proposed IAHM, which will make the model fail to learn these screentones.

Our model is trained with the loss function defined in Equ.2, consisting of reconstruction loss $\mathcal{L}_{\rm rec}$, adversarial loss $\mathcal{L}_{\rm adv}$, intensity loss $\mathcal{L}_{\rm itn}$, KL divergence loss $\mathcal{L}_{\rm kl}$, feature consistency loss $\mathcal{L}_{\rm fcons}$, and feature reconstruction loss $\mathcal{L}_{\rm frec}$. The objective is formulated as

where the hyper-parameters $\lambda_{\rm rec}$ = 10, $\lambda_{\rm adv}$ = 1, $\lambda_{\rm itn}$ = 5, $\lambda_{\rm kl}$ = 1, $\lambda_{\rm fcons}$ = 20, and $\lambda_{\rm frec}$ = 1 are set empirically.

Reconstruction loss. The reconstruction loss $\mathcal{L}_{\rm rec}$ ensures that the reconstructed manga $\tilde{X}$ is as similar to the input $X$ as possible, formulated in pixel level. The reconstruction loss is defined as

where $\|\cdot\|_{2}$ denotes the $L_{2}$ norm.

Adversarial loss. The adversarial loss $\mathcal{L}_{\rm adv}$ encourages the decoder to generate manga with clear and visually pleasant screentones. We treat our model as a generator and define an extra discriminator $D_{\rm m}$ with 4 strided downscaling blocks to judge whether the input image is generated or not [8]. Given the input image $X$ and the reconstructed image $\tilde{X}$ and $\tilde{X_{r}}$, we formulate the adversarial loss as

Intensity loss. The intensity loss $\mathcal{L}_{\rm itn}$ ensures the generated intensity map $S_{\rm itn}$ visually conforms to the intensity $I_{\rm itn}$ of the manga image. We formulate it as

KL divergence loss. The KL divergence loss $\mathcal{L}_{\rm kl}$ ensures that the statistics of the type feature $S_{\rm scr}$ are normally distributed. Given an input manga $X$ and the encoded representation $\mu$ and $\sigma$, we compute the KL divergence loss as the summed Kullback-Leibler divergence [19] of $\mu$ and $\sigma$ over the standard normal distribution $\mathcal{N}(\mathbf{0,I})$.

Here, $KL(\cdot,\cdot)$ denotes the KL divergence between two probability distributions.

Feature consistency loss. The feature consistency loss $\mathcal{L}_{\rm fcons}$ ensures that the type feature $S_{\rm scr}$ can summarize the local texture characteristics in the input $X$ [35]. Given the type feature $S_{\rm scr}$ and its label map $I_{\rm lb}$ which labels the screentone of each pixel, we encourage a uniform region representation within the region filled with the same screentone [20]. The feature consistency loss is formulated as

where $w_{l}$ is the binary mask to filter out structure lines (0 for structural lines). ${\rm Avg}(S_{\rm scr},I_{\rm lb})$ is a map in which each pixel is replaced by the average value of the corresponding region indexed by $I_{\rm lb}$ in the representation $S_{\rm scr}$.

Feature reconstruction loss. We find that intensity information may still entangle with type feature when only the above losses are imposed. To disentangle intensity feature from type feature, we encourage the type feature can still be reconstructed through the extra path with random intensity map. The feature reconstruction loss $\mathcal{L}_{\rm frec}$ is measured as the pixel-wise difference between the random representation $S_{r}$ and the reconstructed representation $\tilde{S_{r}}$.

Data Preparation. We use two types of data to train our model, including synthetic manga data and real manga data. For the synthetic manga data, we manually collected 100 types of screentones and generated their tone inverse by swapping the black and white pixels. We then collected 500 line arts and generated synthetic manga following Li et al. [21] which randomly choose and place screentones in each closed region. We synthesized 5,000 manga images of resolution 2,048$\times$ 1,536, together with the intensity and the screentone type labels of each pixel. Intensity maps and label maps are used to calculate intensity loss $\mathcal{L}_{\rm itn}$ and feature consistency loss $\mathcal{L}_{\rm fcons}$, respectively. For the real manga data, we manually collected 5,000 screened manga of resolution 2,048$\times$ 1,536 to train our model. For each screened manga, we extract the structural lines using the line extraction model [21] and the intensity maps using total-variation-based smoothing [36]. Note that we do not label their ground truth screen type, so the feature consistency loss $\mathcal{L}_{\rm fcons}$ is not calculated for this portion of data.

Training. We trained the model with PyTorch [27]. The network weights are initialized with [10]. During training, we empirically set parameters as $\lambda_{\rm rec}$ = 10, $\lambda_{\rm adv}$ = 1, $\lambda_{\rm itn}$ = 5, $\lambda_{\rm kl}$ = 1, $\lambda_{\rm fcons}$ = 20 and $\lambda_{\rm frec}$ = 1. Adam solver [17] is used with a batch size of 8 and an initial learning rate of 0.0001. All image pairs are cropped to $256\times 256$ before feeding to the networks. Considering the bias problem of real data, the whole model is first trained with synthetic data to obtain a stable latent space. Then, both synthetic and real data are imposed for training to improve generalization.

With the learned representation of screentones, users can easily edit the screentones in manga by segmenting the screentone region and generating new screentones.

Manga Segmentation. To perform automatic manga segmentation, we adopt the Gaussian mixture model (GMM) analysis [9] on the proposed representation. The optimal number of clusters is selected by the silhouette coefficient [14], a clustering validation metric. As the screentone types in a single page of manga are usually limited for visual comfort, we find the optimal number of clusters ranges from 1 to 10. We illustrate the consistency of the encoded screen type feature for regular patterns (dot pattern in Fig. 1) and noisy patterns (first row in Fig. 8). All features are visualized after principal component analysis (PCA) [23].

Controllable Manga Generation. After segmentation, each segmented region can be rescreened by modifying its latent and decoding the latent back to screentones. Users can either modify the screentone type while preserving the original intensity variation (Fig. 7(e)), or the intensity feature to change the intensity (Fig. 7(f)). Interestingly, the editing of the intensity is very flexible. The artist can either replace an intensity-varying region with an intensity-constant one, or preserve the visual effects by increasing/decreasing the overall tone intensity proportionally, as shown in the hair region in Fig.7(f)).

As there is no method tailored for manga rescreening, we compare with approaches on manga segmentation and controllable manga generation respectively.

Comparison on Manga Segmentation. In Fig. 8, we visually compare our method with the classic Gabor wavelet texture analysis [24] and the learning-based ScreenVAE model [35]. All results are visualized by considering the three major components as color values. We also apply the same segmentation to each feature by measuring texture similarity. As observed, all features have the capability of summarizing the texture characteristics in a local region and can distinguish different types of screentones. However, Gabor wavelet feature exhibits severe artifacts near region boundaries with blurry double edges due to its window-based analysis. ScreenVAE map can have tight boundaries towards structural lines, but it failed to segment regions with varying intensity (second row in Fig. 8). In contrast, our representation explicitly encodes the intensity information and can disentangle the types. We can generate a consistent type feature for varying intensity regions.

Comparison on Controllable Manga Generation. Manga artists commonly use smoothly changing screentones to express shading or atmosphere, such as the hair of the third example in Fig. 8 and the background in Fig. 1, which requires the intensity feature to be interpolated in tone. With our design, our representation can provide controllable generation and manipulation on both the intensity and types of screentones. Fig. 9 (a) shows image synthesis by replacing the screentone type features of [35] and our method. Note that although [35] can also generate various screentones, it is not providing stable controls of screentones, as the screentone intensity and type are non-linearly coupled in their model. As observed, the screentone types are disentangled with intensity in our model, and we can generate screentones with expected types while preserving the original intensity during rescreening.

Besides visual comparison, we also quantitatively evaluate the performance of our interpretable representation. In this evaluation, we mainly compare with Gabor Wavelet [24] and ScreenVAE [35]. The evaluation is on four aspects: i) screentone summarization to measure the standard deviation within each screentone region; ii) screentone distinguishability to measure the standard deviation among different screentone regions; iii) intensity accuracy to measure the difference (Mean Absolute Error) between the generated intensity map and the target intensity map; iv) reconstruction accuracy to measure the difference (LPIPS [40]) between the input image and the reconstructed image. Table 1 lists the evaluation results for all methods. Although Gabor Wavelet feature [24] obtains the lowest value of summarization score, it got a low score on distinguishability, which means it is difficult to distinguish different screentones. ScreenVAE [35] and ours achieve better performance for both screentone summarization and distinguishability. Furthermore, Gabor Wavelet feature [24] cannot reconstruct the original screentones, while the other two methods can be used to synthesize the screentones. More importantly, our method explicitly extracts the intensity of manga, which is critical for practical manga parsing and editing.

To study the effectiveness of individual loss terms, we performed ablation studies for each module by visually comparing the generated latent representation and the reconstructed image, as shown in Fig. 10. Note that for better visualization, we did not normalize the type feature here. In addition, there is no loss configuration without reconstruction loss $\mathcal{L}_{\rm rec}$ or intensity loss $\mathcal{L}_{\rm itn}$. The reconstruction loss is necessary to preserve information in latent space, while our model will degrade to the original ScreenVAE [35] without the intensity loss. Without adversarial loss $\mathcal{L}_{\rm adv}$, blurry and noisy screentones may be generated (background in the bottom row of Fig. 10(b)). Without KL convergence loss $\mathcal{L}_{\rm kl}$, the latent space is not normally distributed and balanced representations may not be generated, which will fail the segmentation (top row of Fig. 10(c)). Without feature consistency loss $\mathcal{L}_{\rm fcons}$, the network may not recognize multiscale screentones, and may generate inconsistent representation for the same screentone (top row of Fig. 10(d)). Without feature reconstruction loss $\mathcal{L}_{\rm frec}$, inconsistent screentones may be generated among local regions (background in the bottom row of Fig. 10(e)). In comparison, the combined loss can help the network recognize the type and intensity of screentones and generate a consistent appearance (Fig. 10(f)).

While our method can manipulate manga by segmenting the screentone region and generating screentones with desired screen type or intensity, our method may not extract some tiny screentones and some extra user hint may be required to generate good results (hair in Fig. 11(b)). In addition, some structural lines that are not extracted may be recognized as strip screentones (background in Fig. 11(b)).