FakePolisher: Making DeepFakes More Detection-Evasive by Shallow Reconstruction

Over the past several years, a lot of GAN-based image generation methods have been proposed, largely following two categories: full image synthesis and partial image manipulation.

Tolosana et al. and Verdoliva et al. (Tolosana et al., 2020; Verdoliva, 2020) recently make comprehensive surveys on the DeepFake detection methods (Wang et al., 2020b; Qi et al., 2020; Wang et al., 2019; Afchar et al., 2018; Nguyen et al., 2019b, a; Yu et al., 2019; Zhang et al., 2019b; Frank et al., 2020; Wang et al., 2020a). Overall, they surveyed thirty-seven papers in total, all of which are CNN-based and can be classified into three categories depending on their feature inputs: image-based methods, fingerprint-based methods, and spectrum-based methods. Image-based methods directly use images as inputs with various networks to solve the problem. Fingerprint-based methods detect DeepFake with the features of GAN fingerprints. Spectrum-based methods consider that DeepFake artifacts are manifested as replications of spectra in the frequency domain. Thus they propose classifiers based on the spectrum of fake images.

For both partial image manipulation (Choi et al., 2018; He et al., 2019; Liu et al., 2019) and full image synthesis (Karras et al., 2017, 2019b; Karras et al., 2019a), the generator of GAN-based image generation method has a decoder amplifies the random vectors or feature maps to images. Upsampling is a significant and indispensable design in the decoder. However, it is the upsampling design that makes the GAN-based image generation methods limited. In general, there are three types of upsampling methods: unpooling, transpose convolution and interpolation. It has been studied that transpose convolution results in checkerboard texture (Odena et al., 2016). In the amplification procedure, unpooling operation assigns zero values to the new pixels. This regular magnification produces special textures that do not exist in real images. For interpolation operation, at an intuitive level, the new pixel values are calculated based on existing pixels, which is regularity. Interpolation brings periodicity into the second derivative signal of images (Gallagher, 2005). Other papers (Zhang et al., 2019b; Frank et al., 2020; Huang et al., 2020) also introduced the imperfection of upsampling methods.

In this paper, we propose FakePolisher, a post-processing method that performs a shallow modification of fake images. Our method is composed of three steps. First, we train a dictionary model with a real image dataset. The learned dictionary forms a subspace that is intrinsically low dimensional, which compactly captures the essential structures and representations of the real images. Second, we seek the representation of a DeepFake image using the aforementioned subspace either by linear projection or sparse coding depending on the over-completeness of the learned dictionary. Third, once such a representation is obtained, we reconstruct the ‘fake-free’ version of the DeepFake image by using the said dictionary.¹¹1Throughout the paper, we use ‘clean’ to describe an image that is free of fake patterns. ‘Clean’ and ‘fake pattern-free’ are used interchangeably.

Intuitively, we are forcing the DeepFake image to find its ‘closest’ representation, on the subspace that subsequently leads to the reconstructed version of itself free of any fake patterns. By doing so, a shallow reconstruction can already effectively remove the fake patterns while preserving image fidelity to the greatest extent. In this context, deep reconstruction methods, on the contrary, can become futile because they potentially leave traces of upsampling artifacts as discussed above.

Geometrically, as shown in Figure 3, the learned dictionary forms an embedding space $\mathcal{L}$ that could be of lower or higher dimensionality. When a DeepFake image $\mathbf{x}$ comes along, we seek the ‘closest’ counterpart $\hat{\mathbf{x}}$ on the clean image manifold $\mathcal{M}$ by first obtaining the embedded representation $\mathbf{y}$, and reconstruct back to the clean image manifold $\mathcal{M}$.

When learning the dictionary on the real images that are free of fake patterns, one can choose to learn a local patch-based dictionary, leading to a patch-based image reconstruction; or, a global dictionary that spans the entire image, i.e., dictionary atom is of the same size as the image to be reconstructed. Since we can view the global case as a local case with a large patch size, we will mainly discuss patch-based local reconstruction in detail since it already covers both cases. The choice between various patch sizes (local vs. global) is largely dictated by the actual application and the type of images that are being processed. For example, if the images are aligned faces, it is advisable to use a global dictionary since it is more efficient, without the need of patch-by-patch reconstruction. On the other hand, if the images are of ImageNet type, it is more reasonable to use a patch-based dictionary. One note is that in this case, it is still possible to use a global dictionary, it is just we are to foresee a drop in the reconstruction fidelity.

Next, we formulate the patch-based dictionary learning procedure. The training data (patch) matrix $\mathbf{Y}\in\mathds{R}^{d\times n}$ is assumed with dimension $d$. All matrices have their elements arranged column-wise.

Dictionary learning methods have gained much popularity in tackling low-level computer vision problems. One widely adopted such an algorithm is the K-SVD (Aharon et al., 2006). K-SVD aims to be a natural extension of K-means clustering method with the analogy that the cluster centroids are the elements of the learned dictionary and the cluster memberships are defined by the sparse approximations ($\ell_{1}$ or $\ell_{0}$) of the signals in that dictionary. Formally, it provides a solution to the problem:

where $\mathbf{Y}$, $\mathbf{D}$ and $\mathbf{X}$ are the data, the learned dictionary, and the sparse approximation matrix, respectively. Here $\|.\|_{0}$ is the pseudo-norm measuring sparsity. The sparse approximations of the data elements are allowed to have some maximum sparsity $\|\mathbf{x}\|_{0}\leq K$.

In addition to the K-SVD method, we also explore a ubiquitously popular dictionary learning method: principal component analysis (PCA) (Wold et al., 1987). In the original formulation of PCA, it tries to minimize the following objective function in an $\ell_{2}$ sense. Of course, variants of PCA such as sparse PCA or $\ell_{1}$-PCA, etc., can also fit in with ease. Formally, PCA finds a solution to the problem:

where $\mathbf{Y}$, $\mathbf{D}$ and $\mathbf{X}$ are the data, the learned dictionary (principal components), and the dense coefficient matrix, respectively. The regularizer in the PCA optimization ensures that the learned dictionary atoms are orthogonal, which provides maximal reconstruction capability. The learned PCA dictionary is usually overdetermined (or undercomplete), which provides a good complement to the overcomplete K-SVD dictionary.

Once the said dictionaries are learned from real images that are fake-pattern-free, we can project a DeepFake image patch onto the learned subspace and obtain a new representation on the learned manifold. The dimensionality of such a representation can be lower or higher than the original image-domain representation depending on the overcompleteness of the learned dictionary.

More specifically, for example, when we want to reconstruct a single image patch $\mathbf{y}\in\mathds{R}^{d}$ using learned K-SVD dictionary, since it is overcomplete, we resort to pursuit algorithms such as the orthogonal matching pursuit (OMP) (Tropp and Gilbert, 2007) to obtain the sparse coefficient vector $\mathbf{x}$ according to the following optimization:

Note that there is a trade-off in choosing the sparsity $\tau$ while using OMP for obtaining the sparse representation. To determine the optimal reconstruction sparsity $\tau$ for the down-stream task, we conduct a pilot experiment that aims at selecting a $\tau$ value that is both relatively small (more efficient for the greedy OMP algorithm) and provides high-quality reconstruction. Also, the sparsity $\tau$ during the OMP step is independent and different from the sparsity $K$ during K-SVD dictionary learning.

The shallow reconstruction is straight-forward with the learned K-SVD dictionary $\mathbf{D}$ and the obtained sparse representation $\mathbf{x}$. The reconstructed image patch $\hat{\mathbf{y}}=\mathbf{Dx}$. The aforementioned dictionary learning and reconstruction were previously used in various domain-domain mapping problems such as (Juefei-Xu et al., 2014, 2015b; Juefei-Xu and Savvides, 2015b, a, 2016c, 2016a, 2016b; Abiantun et al., 2019).

As a comparison, shallow reconstruction from learned PCA dictionary requires first obtaining a dense representation for the image patch $\mathbf{y}$. As discussed above, the learned PCA dictionary $\mathbf{D}$ is usually overdetermined, and therefore, the resulting representation vector $\mathbf{x}$ on the manifold will be of lower dimensionality and dense. The representation vector $\mathbf{x}$ can be obtained through a least-square error solution in closed form which is extremely efficient:

To further make the shallow reconstruction using PCA more versatile, one can control what dimensions contribute more during the reconstruction by involving a selector vector $\mathbf{s}\in\mathds{R}^{d}$ that embeds e.g., prior knowledge such as confidence or importance of each dimension. In this case, the representation vector $\mathbf{x}$ can be obtained by incorporating a diagonal selector matrix $\mathbf{S}$, where $\mathbf{S}=\mathrm{diag}(\mathbf{s})$. The solution becomes:

It can be observed that when $\mathbf{S}^{\top}\mathbf{S}$ is close to the identity matrix $\mathbf{I}$, the $\hat{\mathbf{x}}$ is close to the original $\mathbf{x}$. The same principal can be applied to the aforementioned K-SVD sparse reconstruction. Also, the selector vector $\mathbf{s}$ can be both real-numbered (dimension re-weighting) or binary (dimension selection).

The shallow reconstruction is also straight-forward with the learned PCA dictionary $\mathbf{D}$ and the obtained dense representation $\mathbf{x}$, with the reconstructed image patch $\hat{\mathbf{y}}=\mathbf{Dx}$.

Both K-SVD and PCA reconstructions are shallow in the sense that they can bring back the manifold representations to the image domain with a single-step projection. More importantly, with the reconstruction being shallow and single-step, it does not induce unnecessary fake patterns, as commonly found in those DeepFake images produced or manipulated by deep generative models.

Subject Detection Methods and Dataset: We choose three state-of-the-art fake detection methods to verify the validity of our method, i.e., GANFingerprint (fingerprint-based method) (Yu et al., 2019), CNNDetector (image-based method) (Wang et al., 2019), and DCTA (spectrum-based method) (Frank et al., 2020). We use CelebA (Liu et al., 2018), LSUN (Yu et al., 2015), and FFHQ (Karras et al., 2019b) as the real image dataset. CelebA and FFHQ are the human face dataset while LSUN includes the images of different rooms such as classroom, bedroom, etc., which are widely used in previous work. Then, we leverage a total of 16 GAN-based methods for fake image generation on these datasets. In particular, ProGAN (Karras et al., 2017), SNGAN (Miyato et al., 2018), CramerGAN (Bellemare et al., 2017) and MMDGAN (Li et al., 2017) are the GAN-based image generation methods used by GANFingerprint and DCTA. For each GAN-based image generation method, the size of the testing dataset is 10,000. In CNNDetector, they choose 13 GAN-based image generation methods as the testing dataset. The methods include ProGAN (Karras et al., 2017), StyleGAN (Karras et al., 2019b) , BigGAN (Brock et al., 2018), CycleGAN (Zhu et al., 2017), StarGAN (Choi et al., 2018), GauGAN (Park et al., 2019), CRN (Shi et al., 2016), IMLE (Li and Malik, 2018), SITD (Chen et al., 2018), SAN (Dai et al., 2019), DeepFakes (Rössler et al., 2019), StyleGAN2 (Karras et al., 2019a), and Whichfaceisreal (Jevin West, 2019). The size of the testing dataset of these GAN-based image generation methods range from hundreds to thousands. The objects in the datasets of CycleGAN, ProGAN, StyleGAN and StyleGAN2 have two or more categories. For example, in StyleGAN, it has three different categories: bedroom, car, cat. The datasets of other GANs have only one category.

Evaluation Settings: For PCA reconstruction, we use 50,000 real human images of CelebA to train the PCA dictionary model. The component number of PCA dictionary model is 10,000. For K-SVD reconstruction, we use 100,000 patches to train a K-SVD model of 5,000 components. Each patch is of size 8*8, clipped from real images of CelebA. The number of nonzero coefficients in the training procedure is 15. In K-SVD reconstruction, we drop 10% pixels of the fake image before reconstruction. This is an important procedure in K-SVD reconstruction for that it can destroy the fake textures of the fake images. What’s more, it needs a lot of time to produce one K-SVD image. Therefore, we choose 200 of the 5,000 components of the K-SVD dictionary to reconstruct fake images. The reconstructed images of using 200 or 5,000 components are similar while the reconstruction time is significantly reduced. In the reconstructing procedure, the number of nonzero coefficients is 20. The graphical representation of the PCA dictionary and K-SVD dictionary are shown in Figure 4 and Figure 5.

Metrics: The main metric is the detection accuracy of the methods. We compare the detection accuracy of fake images and reconstructed images for each method. In addition, we also use cosine similarity (COSS), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) for measuring the similarity between fake image and its corresponding reconstructed image. COSS is a common similarity metric that measures the cosine of the angle. We transform the RGB images to vectors before calculating COSS. PSNR is the most commonly used measurement for the reconstruction quality of lossy compression. SSIM is one of the most popular and useful metrics for measuring the similarity between two images. COSS, PSNR and SSIM metrics are better if a higher value is provided. The value ranges of COSS and SSIM are both in [0,1].

All the experiments were run on a Ubuntu 16.04 system with an Intel(R) Xeon(R) CPU E5-2699 with 196 GB of RAM, equipped with four Tesla V100 GPU of 32G RAM.

Figure 6 gives the reconstructed image examples of our method, on a cat and a human, respectively. In the second row, PCA and K-SVD reconstruct the fake image successfully. In the first row, we can see that the fidelity of PCA reconstructed image is not as good as K-SVD reconstructed image. The reason is that the PCA dictionary we used is trained by real images of humans instead of cats. Thus, this suggests using K-SVD to reconstruct fake images if PCA dictionary with the same category is not available.

To analyze whether the reconstructed images contain artifacts (i.e., manipulation footprints), we analyze the spectrums of the real image, fake image, PCA reconstructed image, K-SVD reconstructed image (see Figure 7). We can observe that only the spectrum of the fake image has bright blobs at 1/4 and 3/4 of the width/height. The blobs correspond to the manipulation footprints in the fake image. We also verified that the PCA and K-SVD reconstruction methods can reduce artifacts on other types of images such as cat, bedroom, etc.

In the original experiment (Yu et al., 2019) of GANFingerprint, their method can successfully detect whether an input image is real or fake with high accuracy. It can even judge which GAN-based image generation method is used to produce the fake image. In our experiment, we randomly choose 10,000 real images from CelebA. Then, for each GAN-based image generation method (i.e., ProGAN, SNGAN, CramerGAN, MMDGAN), we produce 10,000 fake images, resulting in a total of 40,000. For each of PCA and K-SVD reconstruction method, we produce 40,000 images from these 40,000 fake images.

Table 1 summarizes the detailed detection accuracy of GANFingerprint. We use ProGAN as an example to explain the data in Table 1 (i.e., columns 2-6, rows 2-4). For ProGAN (Pro), it has five sub-items (columns): CelebA, ProGAN (Pro), SNGAN (SN), CramerGAN (Cramer), MMDGAN (MMD). They represent the possibility that the input ProGAN fake images be considered as one of them. Fake, PCA-reconstructed and K-SVD-reconstructed represent the type of input images. The row of Fake shows the results with fake images as inputs. As we can see, GANFingerprint can accurately classify the 10,000 images generated by ProGAN into subitem ProGAN with a detection accuracy of 99.91%. In the table, we highlight the difference in detection accuracy between reconstructed images and fake images (e.g., by color and number). For example, in the row PCA-reconstructed, when we put the 10,000 PCA-reconstructed images into GANFingerprint, it misclassifies most of the images into CelebA (i.e., real images). The ratio of images classified into CelebA raises from 0.03% to 88.90%. We use blue color and (+88.87) to highlight the difference. Similarly, the ratio of images classified into Pro decreases from 99.91% to 6.99%. We use red color and (-92.92) to show the difference. We can see that most of the fake images generated by ProGAN are misclassifies to be real images after our shallow reconstruction.

Similar conclusions could also be reached for the other three GAN-based image generation methods. PCA reconstruction reduces all of their classification accuracy effectively. K-SVD reconstruction also reduces the accuracy. The attack by PCA reconstruction shows slightly higher results compared with K-SVD reconstruction.

Table 2 shows the similarity between the fake images and reconstructed images. For both PCA and K-SVD, we use COSS, PSNR and SSIM as the metrics for similarity measurement. In the column of ProGAN, we can see that the values of COSS and SSIM are near 1.0, and the value of PSNR is more than 30, indicating high similarity. Likewise, for the other three GAN-based image generation methods, the reconstructed images are also very similar to the fake image counterparts. Compared with PCA reconstruction, images by K-SVD show higher similarity to original fake images.

As mentioned above, DCTA has a same testing dataset as GANFingerprint. In the original experiment of DCTA, it transformed the images into spectrum images before classification. We follow the exact same evaluation setting, except that we use the reconstructed images to replace the fake images. Overall, the number of testing images in our experiment is 48,000. Each category of CelebA, ProGAN, SNGAN, CramerGAN, MMDGAN, has 9,600 images. The PCA-reconstructed and K-SVD-reconstructed images used are the same as that in GANFingerprint. The number of reconstructed images is also 48,000. Table 3 summarizes detection accuracy decrease. When using PCA-reconstruction, the accuracy decreases from 88.99% to 16.42%. Similarly, 20.44% accuracy decreases when using K-SVD. The experimental results demonstrate the effectiveness of reconstructed images in misleading the fake detectors.

In CNNDetector, it is evaluated on a large number of 13 GAN-based image generation methods. In their original experiments, the objects in the images are very different, containing animals, human faces, road, etc. For each GAN-based image generation method, the size of the testing dataset ranges from hundreds to thousands. They use detection accuracy and average precision (AP) as the metrics, on the same number of fake images and real images as the testing dataset. We follow the same evaluation setting, except replacing fake images with reconstructed images by our methods.

As we can see in Table 4, the two models used by CNNDetector: blur_jpg_prob0.1 (prob0.1) and blur_jpg_prob0.5 (prob0.5) achieve high accuracy. For convenience, we only introduce the data of using blur_jpg_prob0.1. In the first row, Real & Fake means using real images and fake images as the testing dataset (i.e., the same testing dataset as used in CNNDetector).

To show the detection accuracy of real images and fake images, we conduct extra experiments on real images and fake images respectively. As shown in the second and third row, the testing datasets are Real images only and Fake images only. In the second row, we can observe that the model performs well. However, in the third row, the performance of the model of CNNDetector is different in various GAN-based image generation methods. It has various performance drops on SAN and DeepFakes although it achieves high accuracy on GANs, e.g., ProGAN, StarGAN, CRN.

For PCA reconstruction of our method, we produce a corresponding reconstructed image for each fake image in the testing dataset. For K-SVD reconstruction, it needs a lot of time to produce one K-SVD image. Thus for each category of each GAN-based image generation method, we choose 100 fake images and produce 100 corresponding K-SVD reconstructed images. No matter the detection accuracy of fake images, the reconstructed images generated by us can reduce its performance.

As we can see in the fourth and fifth row, the testing datasets are PCA-reconstructed images and K-SVD-reconstructed images respectively. Compared to the detection accuracy of that on fake images, most of them are both decreased. Only the accuracy of SAN increases slightly. In the experiment of blur_jpg_prob0.5, all the detection accuracy decrease. The similarity of PCA/K-SVD reconstructed images and fake images is shown in Table 5. The images of CycleGAN, StyleGAN, StyleGAN2 and ProGAN have quite a few different categories. For these four multi-category GANs, they use different folders to store different categories of images. In the other nine GANs, some of them involve only one category (DeepFakes, IMLE, StarGAN, Whichfaceisreal, CRN). The others combine images of different categories into one folder. We call these GANs (BigGAN, GauGAN, SAN, SITD) uncertain-category and use ‘-’ to represent the category.

Sometimes, fake images are produced by only modifying parts of the real images. For example, DeepFake methods may change the hair color of a person or the color of a chair in the bedroom. For these situations, we propose partial reconstruction.

In the GANs of GANFingerprint, DCTA and CNNDetection, only StarGAN have partially modified fake images. Therefore, we use StarGAN to show the comparison between partial reconstruction and full reconstruction. The numbers of real images and fake images of StarGAN are 1,999. We produce 1,999 reconstructed images for partial and full reconstruction of PCA. We also produce 100 reconstructed images for K-SVD. Since it needs a lot of time to produce K-SVD images, the number of K-SVD reconstructed images is not as large as that of PCA reconstructed images.

For each of GANFingerprint and DCTA, we train a binary classification model with real images from CelebA and partially modified fake images from StarGAN.

Table 6 summarizes the detection accuracy of fully reconstructed images and partially reconstructed images on three different types of fake detectors. Compared with full reconstruction, the detection accuracy of all the three fake detectors decrease similarly in partial reconstruction for both PCA and K-SVD. Table 7, shows the similarity metrics between reconstructed images and the original fake images. We can observe that the similarity of fake images and partial reconstruction is higher than that of fake images and full reconstruction.

To sum up, compared with fully reconstructed images, the partially reconstructed images are more similar to their original fake counterparts. Meanwhile, in terms of degrading the performance of fake detectors, their abilities are close, indicating the advantage of partial reconstruction for partially modified fake images.