A universal detector of CNN-generated images using properties of checkerboard artifacts in the frequency domain

A variety of generator models have been proposed for image generation and image-to-image translation. Models based on variational autoencoder (VAE) or GAN are typical image generator models[8, 9]. Autoencoders including VAE translate an image into latent variable $z$ by using an encoder, and generate an image from $z$ by using a decoder. Since $z$ has the standard normal distribution, VAE can generate images from a noise having the standard normal distribution by the decoder. Deepfakes[10] with VAE become a major threat to the international community.

GAN models estimate a generative model and a discriminative model via an adversarial process. PGGAN[11], BigGAN[12], StyleGAN[13] and StyleGAN2[14] generate high resolution images from a random noise vector. In addition, CycleGAN[15] and StarGAN[16] translate an image from a source domain to a target domain, e.g. changing apples to oranges. GauGAN[17] were proposed to generate an image from an input semantic layout.

The first approach for detecting CNN-generated images was inspired by photo-response non-uniformity noise (PRNU) that was used for discriminating camera devices[18, 19]. This approach enables us to discriminate GAN-generated images from fingerprints caused by GAN, and assume that the same GAN models are used for training and testing images.

To universally detect CNN-generated images even when images are not ones generated from a model used for training the detector, a universal detector was proposed, where it was trained by using images generated ony from AutoGAN[20]. In this work[20], the use of the frequency domain was demonstrated to improve the performance of the detector. In contrast, a universal detector by training only one specific GAN (PGGAN) was proposed[7], where RGB images are directly used for training a detector. This method was shown to detect not only PGGAN but also other generator models that were not used for training the detector.

Figure 1 shows an overview of the proposed detector with enhanced spectrums. For training a classifier, a novel enhancement method for clearly showing checkerboard artifacts included in CNN-generated images is applied to training image $I_{i}$. Enhanced spectrum $E_{i}$ calculated from $I_{i}$ are used for training the classifier. For testing, a query image $I_{Q}$ is enhanced as well as for training, the spectrum $F_{Q}$ calculated from $I_{Q}$ is inputted to the trained classifier to judge it to be a CNN-generated one or not in accordance with an outputted probability score $r_{F}\in[0,1]$.

Enhanced spectrum $F_{i}$ is calculated as below.

1. 1.

   A median filter with a size of $5\times 5$ is appliled to $I_{i}$, and the difference between $I_{i}$ and $I^{\prime}_{i}$, $I^{D}_{i}=I_{i}-I^{\prime}_{i}$ is calculated, where $I^{\prime}_{i}$ is an output image from the median filter.

2. 2.

   Cropping $I^{D}_{i}$ into $L$ rectangles with a size of $N\times N$ at random positions to generate $L$ images with a size of $N\times N$, $I^{1}_{i},\dots,I^{L}_{i}$.

3. 3.

   Applying $N\times N$-DFT to $I^{1}_{i},\dots,I^{L}_{i}$ to obtain their spectrums $F^{1}_{i},\dots,F^{L}_{i}$.

4. 4.

   Computing enhanced spectrum $E_{i}$ as.

Similarly, enhanced spectrum $E_{q}$ is calculated by using query $I_{q}$ (see Fig.2).

From properties of checkerboard artifacts, CNN-generated images are confirmed to have the influence of checkerboard artifacts at the same positions in the frequency domain for all cropped images. Accordingly, averaging spectrums as shown in the procedure can enhance the features that CNN-generated images have.

A state-of-the-art method for detecting CNN-generated images [7] is directly carried out with RGB images. In this paper, we also propose an ensemble of this conventional detector and the detector with enhanced spectrums.

An overview of the ensemble is shown in Fig. 3. Final probability score $r$ is calculated from $r_{I}$ and $F_{F}$ in accordance with Algorithm 1, where $r_{I}$ and $r_{F}$ are scores from the detector with enhanced spectrums and the detector with RGB images, respectively. Each detector has their own strengths and weaknesses, so this ensemble is expected to improve the performance of each detector.

In this experiment, the dataset prepared by Wang et al. [7] was used, where the training dataset consist of 720K images generated by using PGGAN [11], and test datasets consisted of 4K images in which 2K images are generated by using 11 models as CNN-generated images, and the others are images captured by cameras.

The performance of detectors was evaluated by using F-score and average precision (AP). F-score is given by,

where $P$ and $R$ are the precision and recall at a selected threshold $th$, and $th=0.5$ was selected for this evaluation. AP is also computed by summarizing a precision-recall curve as the weighted mean of precisions achieved at each threshold:

where $P_{j}$ and $R_{j}$ are the precision and recall at the $j_{th}$ threshold, and an input image is judged to be a CNN-generated image, when the probability score $r=[0,1]$ of the input image is higher than $th$.

Table I and II show AP and F-score values under the use of test images from each model. Table I shows the proposed method (single) outperformed Wang’s method under a number of models that had similar network structures to PGGAN. In addition, the proposed method (ensemble) outperformed Wang’s method for almost all models. Wang’s method and the proposed method (single) have their own strengths and weaknesses for detecting CNN-generated images, respectively. In contrast, the ensemble enables us to adopt only strong points of each method. Table II also shows mean values calculated from AP values of 11 models. From the table, the proposed detector with enhanced spectrums was confirmed to improve the accuracy of Wang’s method, although the models were trained only by using images from PGGAN. The difference between Table I and Table II was caused due to the difference in the selection of threshold values.