Scene Depth Estimation from Traditional Oriental Landscape Paintings

Various depth estimation methods were proposed and showed satisfactory performances in data domains such as synthetic data[17, 64], multi-view cameras[12], and LiDAR[22]. Especially, MiDaS[48] showed excellent generalization performance, allowing their network to estimate depth in images completely unrelated to the training dataset such as paintings. Ranftl et al.[47] changed the architecture of MiDaS[48] to transformer-based encoder-to-decoder architecture to achieve better performance. Birkl et al.[9] presented variety of new models of MiDaS[48] based on different pre-trained vision transformers. Bhat et al.[7] and Miangoleh et al.[41] produced more detailed depth maps based on the approach of MiDaS[48].

On the other hand, there have been several attempts to leverage deep learning models to measure the depth of painting images. Jiang et al.[25] utilized a depth estimation model[62] to gauge the depth in western painting images. Lin et al. [37] introduced a user-interactive framework to generate depth maps of painting images and photo images manually. Bhattacharjee et al.[8] proposed a novel framework that estimates the depth of comics by transforming them into real scenes. Pauls et al. used depth maps of painting images to enhance the experience of museum visits[45]. Ehret et al.[14] reported that although DPT [47] and MiDaS [48] were not explicitly trained for measuring the depth of painting images, they produced reasonable depth maps based on the high generalization performance.

Previous depth estimation works in painting images only attempted to measure the depth of western landscape painting images that are somewhat similar to landscape photo images. We have observed SOTA depth estimation model such as MiDaS [48] cannot generate satisfactory depth maps for oriental landscape painting images due to their characteristics mentioned earlier. We take the strategy that first converts oriental landscape painting images into real scene images and then measures depth of real scene images using MiDaS[48].

I2I translation is a popularly used solution to transformation of an image in domain A into an image in domain B. Tomei et al.[53] proposed a method to convert painting images into photo images. This method employed a segmentation model[21] to divide painting images and photo images into patches and then match them, showing an outstanding translation performance. Despite its good performance, it cannot be applied to convert oriental landscape painting images to real scene images due to the huge domain gap between them.

Pix2Pix [23] was the first to apply conditional GAN to the I2I translation task and showed excellent performances. On the other hand, CycleGAN[63] was proposed as a way for unsupervised learning to the I2I translation. Park et al.[43] presented a method, dubbed CUT, that maximizes mutual information based on contrastive learning. Liu et al. [38] showed a shared-latent space assumptions and proposed a framework, called UNIT. Lee et al.[33] introduced a disentangled representation framework with cross-cycle consistency loss. Recently, Chen et al.[11] conducted I2I translation based on VQGAN[15] which gives an outstanding performance in image generation.

Although GAN-based models performed well in I2I translation tasks, Ho et al.[20] successfully applied diffusion process to neural network, yielding many diffusion-based I2I models that translate images with high-quality. Kwon et al.[32] proposed diffusion-based I2I and Text-to-Image (T2I) translation methods using disentangled style and content representation. Saharia et al. [49] implemented diffusion models for various tasks, including I2I translation, without task-specific hyperparameter tuning. Su et al.[52] proposed diffusion-based I2I translation in latent encoding and defined it via ordinary differential equations. Li et al.[36] successfully applied stochastic brownian bridge process to diffusion for translation between two domains directly through the bidirectional diffusion process.

GAN-based I2I translation models cannot realistically transform oriental landscape painting images into landscape photo images as diffusion-based models do. However, diffusion-based models are not directly applicable to our problem because they drastically modify structural information of image during translation when input images are structurally different. Therefore, to overcome these limitations, we employ GAN-based model in the first I2I step and diffusion-based model in the second I2I step.

Radford et al.[46] proposed a contrastive learning-based Image-to-Text (I2T) matching model which allows zero-shot prediction called CLIP. When CLIP[46] gets an input image and a set of text, it finds the text that best matches the image in multi-modal embedding space. Diverse CLIP-based methods are proposed, for image retrieval[50, 57], style transfer[39, 31], image generation[31, 44, 10], classification[13, 59], anomaly detection[60], and image aesthetic assessment[18, 56]. Vinker et al.[54] introduced a technique for converting images into sketches by employing diverse types and multiple levels of abstraction. Hessel et al.[19] proposed a novel metric for robust automatic evaluation of image captioning. Especially, Materzyńska et al.[40] found that CLIP[46] had a strong ability of matching words with natural images of scenes.

To generate more plausible real scene images, we build a CLIP-matched dataset consisting of pairs of oriental landscape painting images and landscape photo images since there are no actual location photo images that correspond to oriental landscape painting images. We opt to match semantically similar images to a given oriental landscape painting images. For this, we utilized CLIP[46] for building the CLIP-matched dataset because CLIP[46] may allow matching tasks in unseen image domains such as oriental landscape painting images to landscape photo images.

Since oriental landscape painting images, $I_{ori}$, and landscape photo images, $I_{photo}$, are in different domains, direct I2I matching in the embedding space does not perform well. Furthermore, actual location photo images that matches with oriental landscape painting images are not available. Thus, we utilize CLIP[46] to semantically match oriental landscape painting images with landscape photo images for building a CLIP-matched dataset.

To make the CLIP-matched dataset, we built a pre-defined dictionary. As the pre-defined dictionary is important for matching, we have selected frequently appearing objects in $I_{ori}$ by referencing papers [30, 42] and museum collections [3, 5, 4, 1, 2]. Park et al. [42] reported that the analysis of 629 oriental landscape paintings showed that 98.4% of them depicted mountains, rivers, and the sea, underlining that many oriental landscape paintings are related to nature, while those depicting villages are scarce. Therefore, when constructing a pre-defined dictionary, it is reasonable to include more words related to nature. According to Kim et al. [30], when expressing water in oriental landscape paintings, it is subdivided into details such as waterfall, stream, and pond based on the terrain.

Hence, when creating the pre-defined dictionary, we not only included keywords related to nature, but also categorized those keywords. To semantically match $I_{ori}$ with $I_{photo}$ using CLIP [46], we build the pre-defined dictionary as shown in Fig. 3.

Each set of images has $n$ and $m$ samples of $I_{ori}$ or $I_{photo}$ as in  (1).

To avoid limitation that CLIP-based image matching method incorrectly matched $I_{ori}$ with $I_{photo}$, we construct a CLIP-matched dataset using a 1-to-$K$ matching method instead of a 1-to-1 matching manner. Thus, for each $I_{ori}$, we search top-$K$ best matching results in a set of $I_{photo}$. The result of CLIP-based matching will be $I^{\prime}_{photo}$, a subset of $I_{photo}$, as expressed in (2).

In summary, since there are no actual location photo images corresponding to oriental landscape painting images, we employed CLIP[46] to match landscape photo images that are semantically similar to oriental landscape painting images. The CLIP-matched dataset that consists of matched images is used to train CycleGAN [63] in an unsupervised manner. The performance evaluation for different values of $K$ in 1-to-$K$ matching is presented in the Section 4.2.

As in CycleGAN[63], we employ two auto-encoders, $G_{photo\rightarrow ori}$ and $G_{ori\rightarrow photo}$, for mapping. $G_{photo\rightarrow ori}$ transforms $I_{photo}$ into $I_{ori}$, while $G_{ori\rightarrow photo}$ converts $I_{ori}$ into $I_{photo}$. Additionally, we use two discriminators, $D_{photo}$, and $D_{ori}$. $D_{photo}$ learns to differentiate $G_{photo\rightarrow ori}(I_{ori})$ and $I_{photo}$ as fake and real, while $D_{ori}$ distinguishes between $G_{ori\rightarrow photo}(I_{photo})$ and $I_{ori}$, aiming to discern fake and real images.

The utilized adversarial and consistency loss can be summarized as follows: The adversarial loss ensures that the auto-encoder $G_{photo\rightarrow ori}$ generates images as similar as possible to $I_{ori}$ and simultaneously trains the discriminator $D_{ori}$ to classify $I_{ori}$ as real and $G_{photo\rightarrow ori}(I_{ori})$ as fake. This process facilitates the transformation of $I_{photo}$ into $I_{ori}$ as closely as possible.

Adversarial loss is also employed during the transformation from $I_{ori}$ to $I_{photo}$, is expressed as:

The cycle consistency loss is utilized under the premise that when $I_{photo}$ is transformed using $G_{photo\rightarrow ori}$ and the resulting $G_{photo\rightarrow ori}(I_{photo})$ is further transformed using $G_{ori\rightarrow photo}$, the output $G_{ori\rightarrow photo}(G_{photo\rightarrow ori}(I_{photo}))$ should resemble $I_{photo}$ when case of $I_{ori}$ should be same. The equation can be represented as:

The final objective loss term can be expressed as follows, with each term adjusted using $\lambda$s.

In summary, in the first step, we use CycleGAN[63] to transform $I_{ori}$ into $I_{photo}$. We denote the translated $I_{photo}$ as $I_{pseudoRS}$, i.e., $\Phi_{CycleGAN}(I_{ori})=I_{pseudoRS}$. The generated $I_{pseudoRS}$ and $I_{ori}$ are fed into a pre-trained DiffuseIT[32] to predict a real scene image, $I_{RS}$, i.e., $\Phi_{DiffuseIT}(I_{ori},I_{pseudoRS})=I_{RS}$. The generated $I_{RS}$ enables depth prediction via a pre-trained depth estimation model such as MiDaS [48].

To evaluate qualitative results, we have compared our method with other I2I translation models. For our methods, CycleGAN[63] in the first step has been trained on the CLIP-matched dataset. ImageNet $256\times{}256$ pre-trained model has been employed for DIffuseIT[32] in the second step. Other I2I translation models have been trained on pairs with oriental landscape painting images from Chinese landscape painting dataset[58] and randomly sampled landscape photo images from LHQ dataset[51].

As illustrated in the first column of Fig. 4, direct application of pre-trained depth estimation model, MiDaS [48], fails to estimate depth of oriental landscape painting images although MiDaS[48] has excellent generalization ability. As depicted in the second, third, and fourth columns of Fig. 4, the GAN-based I2I translation methods[63, 43, 33] well preserve the structural information of the oriental landscape painting images, but cannot convert them realistically. Surprisingly, the GAN-based I2I translation models such as CAST[61] and VQ-I2I[11] not only fail to realistically translate oriental landscape painting images to real scene images, but also distort structural information as shown in the fifth and sixth columns of Fig. 4. In the seventh and eighth columns of Fig. 4, DiffuseIT [32] and BBDM [36], diffusion-based I2I models realistically transform oriental landscape painting images into real scene images. However, predicted real scene images drastically change structural information during translation. This means that even if depth can be estimated using pre-trained MiDaS[48], obtained depth map does not represent the scene in the given oriental landscape painting images. Our method predicts real scene images corresponding to oriental landscape painting images, preserving structural information such as CycleGAN [63], CUT[43] or DRIT++ [33] and realistically translating into real scene images, like BBDM [36] or DiffuseIT [32].

As depicted in Fig. 5, regardless of the use of CLIP-based image matching, absence of CycleGAN[63] leads to significant structural deformation during translation. Depth of structurally modified real scene images is meaningless even if depth can be measured using a pre-trained depth estimation model[48]. In cases that DiffuseIT[32] is missing, although the predicted real scene images maintain structural similarity, it fails to realistically convert oriental landscape painting images into real scene images that lead to unsatisfactory depth map outcomes. Without CLIP-based image matching, it creates low-quality pseudo-real scene images which affect the final result. To achieve plausible real scene images corresponding to given oriental landscape painting images, CLIP-based matching, CycleGAN[63] and DiffuseIT[32] should be employed.

As illustrated in Fig. 6, we have trained CycleGAN [63] using the CLIP-matched dataset. When building this dataset, a 1-to-$K$ manner is employed. To prove effectiveness of the 1-to-$K$ manner, we have compared real scene images with $K$ as 1, 3, 5, and 10. Setting $K$ to either 1 or 10 results in either too few or too many images treated as correct data, significantly influencing CycleGAN [63] and ultimately yielding poor translated outcomes. When $K$ is set to either 3 or 5, CycleGAN [63] produces the most plausible pseudo-real scene images, which also affect the final results through DiffuseIT [32].

To evaluate structural preservation and translation quality of predicted real scene image, we conducted a user study. We asked forty anonymous participants five question sets that are subdivided to $Q^{s}$ and $Q^{q}$.

First, for $Q^{s}$, we provided one real scene image and five oriental landscape painting images that include the one oriental landscape painting image corresponding to the given real scene image. We asked participants to select a single oriental landscape painting image that is most structurally similar to the real scene image. This question aims to assess how well the real scene image preserves the structure of the given oriental landscape painting image during translation.

After the first question was done, we asked another question, $Q^{q}$. We gave participants a pair of an oriental landscape painting image and its real scene image and asked them to rate how well the produced real scene image is realistically translated, scaling from min 1 to max 5.

As summarized in Table 1, regarding the assessment of structural preservation in questions, $Q^{s}$, an average of 93.6% of the participants chose oriental landscape painting images that corresponded to predicted real scene images. For the questions that evaluate to quality of the predicted real scene images, $Q^{q}$, an average score of 4.22 point was obtained. These results serve as evidence demonstrating the capability of our method to not only preserve the structural similarity of oriental landscape paintings during prediction, but also realistically translate oriental landscape painting image to real scene image.