Identity-Aware CycleGAN for Face Photo-Sketch Synthesis and Recognition

Existing FPSS approaches can be classified into 4 categories [2]: subspace learning, sparse representation, Bayesian inference and deep-learning-based methods, where the former three categories belong to data-driven method, and the last one belongs to model-driven method [14].

Subspace-learning-based methods mainly include linear subspace-based methods, such as principal component analysis (PCA) [15], and nonlinear subspace methods such as local linear embedding (LLE) [16]. Tang and Wang [17] assume that the target sketch can be synthesized by a linear combination of training sketches, similar to the relationship within photos. Later, inspired by LLE [16], Liu et al. [3] proposed a patch-based synthesis method that aims to synthesize the target sketch patch from several training sketch patches according to the photos’. In recent years, sparse-representation-based methods have been widely used in FPSS tasks. Chang et al. [18] introduce a sparse coding technique for feature extraction, assuming that the face photo and corresponding sketch have the same sparse representation. To further consider high-frequency information, Gao et al. [4] propose a two-step process to improve the quality of synthesized sketches, during which the mapping between the high-frequency information of the photo-sketch patches is learned using the sparse coding technique. Bayesian inference-based methods explore such an area in a different way by using embedded hidden Markov model (E-HMM) or Markov random field (MRF). Wang and Tang [1] employ a multi-scale MRF model for FPSS problem to learn the relations among neighboring image patches. Zhou et al. [5] improve the basic MRF method in [1] by proposing a weighted MRF algorithm, called Markov weight fields (MWF). Wang et al. [2] propose a novel face sketch synthesis method (TFSPS) to improve the robustness of MWF based on transductive learning. Those Bayesian inference-based methods pay special attention to the relations among target image patches in the corresponding domain but may overlook the domain-to-domain transformations, which may lead to high computation costs in practical use.

The deep-learning-based method actually focuses on model-driven strategy, where the mapping function is learned in advance and then applied to execute the transformation. Zhang et al. [19] propose an end-to-end photo-sketch generation model based on a 7-layer fully convolutional network (FCN) with a joint generative-discriminative optimization process. However, the synthesized images still produce blurring, with less facial texture information obtained. With GAN attracting increased attention, the development of FPSS was spawned. Wang et al. [20] perform photo-to-sketch synthesis with a GAN-based method and demonstrate high effectiveness; however, sketch-to-photo synthesis is not considered. Recently, Wang et al. [21] applied the multi-adversarial idea to CycleGAN to generate high-resolution images. For sketch-to-photo synthesis, Kazemi et al. [22] employ an additional geometry-discriminator to distinguish based on high-level facial features. However, these approaches have only focused on adapting the architecture of GAN for an improvement on synthesizing images. Instead, our framework additionally adopts a mutual optimization process by exploring identity information of images and utilizing the relation between FPSS and FPSR. Zhang et al. [9] propose a dual-transfer FPSS framework composed of an inter-domain transfer process and an intra-domain transfer process. They get the transferred image from the combination of two transfer results, while we transform images between sketch and photo directly through a synthesis model.

Many traditional algorithms have been used to address FPSS problem and achieved effectiveness results. However, these methods still have limitations in image conversion, since they may not be able to learn high-level features of target images and only address certain tasks that are generally single purpose. With the success of CNNs, many studies address image translation problems effectively. However, in some respects, traditional CNN methods may lead to blurry and conservative results because the only goal for a CNN model is to minimize the loss function, which is defined and constructed artificially such as in [19]. Under this training strategy, all reasonable outputs are averaged; this may cause indistinct outputs when only minimizing the predefined loss.

Considering these consequences, a more automated mechanism for training is needed. Generative networks, such as GANs [6] and VAEs [23], have become popular and have achieved great results. Their fundamental objective is to train a model in which output images have the same distributions as the ground truth but are separated from other domain. For image translation tasks, GANs have produced more impressive results due to their strong generative and discriminative ability compared to other generative networks. However, the selection, design and training process of the GANs are also highly uncertain, directly producing many difficulties. Specifically, the simple training requirements of GANs result in large degree of freedom with no pre-modelling necessary; thus, it may suffer performance losses when there are more pixels. Based on this concept, some explicit external information, such as extra inputs by users and category information, can then be added to the original GAN model to enhance its stability.

Isola et al. [7] propose conditional GAN (cGAN), whose generative process is constrained by extra information such as images, labels and tags. In other words, hints are given during training and play a guiding role in the data generation process for the generator. In addition, traditional GANs lack guidance, which can also be seen as a mode distortion problem since the only goal is to optimize an asymmetric KL divergence. Zhu et al. later proposed CycleGAN [8], which showed promising results on unsupervised image-to-image translation tasks based on two generators and two discriminators using unpaired image data. In addition, CycleGAN adds a new cycle consistency loss to the original loss function, which provides extra supervision between the input and cyclic output. However, facial content from CycleGAN cannot be well preserved because of the weak content constraint. Inspired by dual learning, Yi et al. [24] propose Dual-GAN with a similar unpaired training mechanism based on unsupervised performance. More recently, Mo et al. [25] design an instance-aware GAN (InstaGAN) to incorporate the instance information and improve multi-instance transfiguration, which shows no advantages in handling our problem of single face image translation.

Our goal is to learn the mapping functions between the face photo and the sketch, which correspond to two sample spaces $X$ and $Y$. A dataset $\mathcal{D}$ consisting of a number of photo-sketch pairs is needed. Given face photos $x$ and corresponding face sketches $y$, we want to obtain two generators $G_{X}$ and $G_{Y}$. $G_{X}$ transforms the face photo into a sketch, and $G_{Y}$ transforms the face sketch into a photo. The whole network framework is shown in Figure 2. The generated fake sketches from $G_{X}$ are expressed as $F_{y}$, and the generated fake photos from $G_{Y}$ are $F_{x}$. The relationship is expressed as follows:

To realize the cyclic process of the full image transformation, the generated fake images are input into the other generator:

And the two discriminators $D_{X}$ and $D_{Y}$ for two domains learn to distinguish between the real input images and the fake generated samples.

Since we have paired data in the target datasets, unsupervised training in [8, 22] is not necessary for our work. To learn the identity information of each person, high-level features of facial images are extracted from a pre-trained recognition model that is fine-tuned based on the VGGFace model [26]. With these two basic recognition networks $\Phi_{p}$ and $\Phi_{s}$, respectively for photo and sketch, the most prominent facial characteristic and facial structure for identity discrimination can be captured to directly regularize the distance between the real image and the fake image.

Our IACycleGAN is adapted from the CycleGAN architecture [8]. The generators come from the style transfer network [11], which contains two stride-2 convolutions, several residual blocks [16], and two stride-$\frac{1}{2}$ convolutions. For the discriminators, a PatchGAN architecture is adopted that attempts to classify an image at the patch-level [7, 27]. Tiling artifacts can be alleviated by using a 70$\times$70 Patch-GAN, and fewer parameters are necessary, from which better results are obtained in terms of both visual quality and efficiency compared to other PatchGAN models of different sizes [7].

As summarized in Figure 3, the loss function we defined contains four terms: the adversarial loss that matches the distribution of the generated images to the data distribution of the target domain; the cycle consistency loss that prevents the mapping functions $G_{X}$ and $G_{Y}$ from contradicting each other; the identity perception loss that aims to preserve identity information based on two recognition networks; and the identity mapping loss that favors to preserve the person texture identity between the input and output.

Adversarial Loss. The adversarial loss is applied in both mapping directions. For the mapping function $G_{X}:X\rightarrow Y$ and its discriminator $D_{Y}$, the adversarial loss is expressed as

where $G_{X}$ aims to generate a fake image $G_{X}(x)$ that is completely indistinguishable from the images in the target domain, while $D_{Y}$ tries to distinguish between the real and fake images. And for the mapping function $G_{Y}:Y\rightarrow X$ and the discriminator $D_{X}$, we define a similar adversarial loss: ${\mathcal{L}}_{GAN_{Y}}$. The generators try to minimize this loss, while discriminators try to maximize it.

Cycle Consistency Loss. The key to the success of CycleGAN is using the supervision of the cycle consistency loss, which assumes that the generated images can be transformed back to the source domain. Thus, for the images in domain $X$, we expect $x\rightarrow G_{X}(x)\rightarrow G_{Y}(G_{X}(x))\approx x$, and for the images in domain $Y$, we expect $y\rightarrow G_{Y}(y)\rightarrow G_{X}(G_{Y}(y))\approx y$, from which the objective can be expressed as:

Note that we still adopt a pix-level loss to satisfy backward cycle consistency, different from Kazemi et al. [22] who utilize a high-level perceptual loss for the cycle-consistency definition.

Identity Perception Loss. As illustrated in [28], only using the adversarial loss may lead to artifacts and training instabilities while synthesizing high-resolution images. Specifically, to add a strong supervision between the synthesized image and ground truth, we utilize a perception loss constructed by the face features relating to the activations of the fc7-layer of the VGG-16 [29]. The identity perception loss ${\mathcal{L}}_{ip}$ is defined as

where $\Phi_{p}$ and $\Phi_{s}$ are the recognition networks for photo and sketch.

Unlike Wang et al. [21], who minimize the pixel-level difference between synthesized images and corresponding real samples, we employ a perceptual supervision for facial features for two reasons. On the one hand, real sketches drawn by artists have distortions in the facial texture information and exaggerated facial characteristics in a certain style. Therefore, synthesizing images completely based on such hand-drawn sketches does not make sense. Taking photo-to-sketch synthesis as an example, generated sketches have different texture distributions from hand-drawn sketches; using a pix-to-pix supervision of these two types of sketches is thus unreasonable. On the other hand, the training of CycleGAN requires common data augmentation techniques during data preprocessing, such as random cropping, flipping and resizing, which may cause pix-level supervision to be difficult to implement.

Identity Mapping Loss. To constraint the generator to be near an identity mapping when real samples from the target domain are provided as the input, we introduce a pixel-wise consistency between the input images and the generated images. We express this loss as:

Full Objective. Thus, the total objective function can be formulated as

where $\lambda_{cyc}$, $\lambda_{ip}$ and $\lambda_{im}$ are parameters controlling the relative importance of every loss. The final objective can be expressed as a minmax problem:

There are only limited datasets for pairs of human-drawn sketches and face photos since it requires a great effort to collect face sketches. In this paper, the source of photo-sketch pairs is collected from two popular datasets: the Chinese University of Hong Kong (CUHK) Face Sketch Database (CUFS) [1] and the CUHK Face Sketch FERET Database (CUFSF) [32].

The CUFS dataset [1] consists of 606 faces in total: 188 faces from the CUHK student database [17], in which 88 pairs are taken as the training set and the remaining 100 pairs are used as the testing set; 123 faces from the AR database [33], in which 80 pairs are used for the training and the remaining 43 pairs are used for testing; and 295 faces from the XM2VTS database [34], in which 100 pairs are chosen as the training dataset and the remaining 195 pairs are used as the testing set. Each face has one photo-sketch pair, and the sketch is drawn based on a photograph by an artist under normal lighting conditions as well as with neutral expressions. The CUFSF dataset [32] includes 1,194 faces from the FERET database [35], in which 250 faces are taken as the training set and the remaining 944 faces are taken as the testing set. Each person has a face photo with illumination changes and a face sketch drawn by an artist. This database is quite challenging since the photos are taken under different illumination conditions, and the sketches have numerous exaggerations compared to the photos. We employ the MTCNN method [36] to perform face detection and alignment. Before the training process, all images are cropped to the standard size of 256$\times$256.

Implementation Details: For the training of CycleGAN and IACycleGAN, all these synthesis networks are trained from scratch, instance normalization is used to achieve better stability and lower noise, and Adam with $\beta_{1}=0.5$ and $\beta_{2}=0.999$ is used. We flip the images horizontally with a probability of 0.5 for data augmentation. The learning rate is set to 0.0002 for the first 100 epochs and linearly decayed down to 0 for the next 100 epochs. Training takes about 10 hours with a single NVIDIA Titan Xp GPU. We empirically set the hyperparameters of the loss functions in Eq. (7) as follows: $\lambda_{cyc}=10$, $\lambda_{ip}=30000000$, $\lambda_{im}=5$. As illustrated in [8], to reduce network oscillation, an image buffer storing several generated images is adopted to update the discriminator rather than using the last image generated.

Our final results of FPSS are obtained from the IACycleGAN synthesis in the optimization process. We compare three different networks (traditional CNN, conditional GAN and CycleGAN) with our proposed model from both qualitative and quantitative perspectives, therein focusing on the quality of the synthesized images. For the qualitative research, we employ visual observations. For the quantitative aspect, two measurements, the structural similarity index metric (SSIM) [37] and the feature similarity index metric (FSIM) [38], are used to evaluate structural and feature similarity between images. In addition, we test our trained synthesis network on other unseen datasets to prove the effectiveness of our framework.

Implementation Details: The fine-tuning process is implemented on Caffe. All networks are trained using $momentum=0.9,weight_{decay}=0.0002$. For first fine-tuning on the basic VGGFace, we adopt $step$ learning policy with $stepsize=100,gamma=0.96,base_{lr}=0.3$, and 600 iterations are used for photo recognition network and 1600 iterations for sketch recognition network. For second fine-tuning on the last fine-tuned VGG, we adopt $step$ training policy with $stepsize=200,gamma=0.96,base_{lr}=0.01$, and 2000 iterations are used for both photo and sketch recognition networks. We set $K=4$ and $\alpha=0.1$ for the parameters of triplet loss layer. Training takes about 30 hours with a single NVIDIA Titan Xp GPU.

As seen from Figure 10, there are two approaches to retrieving real face photos based on a query sketch: photo matching method and sketch matching method. In the photo matching method, sketch-to-photo synthesis (S2P) is first implemented on the query sketch; then, the generated photo is used for matching on the real photo dataset, where the photo matching rate needs to be computed. In the sketch matching method, all photos in the dataset will first be transformed into sketches, and then, the query sketch is matched to the generated fake samples; at last, the sketch matching rate is computed. In our experiment, we implement both two matching methods and compare the results with other synthesis-based recognition techniques.