Synthesizing Bidirectional Temporal States of Knee Osteoarthritis Radiographs with Cycle-Consistent Generative Adversarial Neural Networks

After merging the KL levels, the subsequent phase involved a series of image processing steps. Initially, we performed a lateral flip on each right knee joint orientation, aligning it to the left. Subsequently, we identified and inverted negative channel images, finding 189 instances. Following the channel inversion process, we employed contrast equalization on the histograms of the images. The technique is detailed in Equation 1. Given a gray-scale image I of dimensions $m\times n$, we utilized its cumulative distribution function (cdf) and pixel value $v$ to attain an equalized value $h(v)$ within the range $[0,255]$:

Here, $\mathrm{cdf}_{\mathrm{min}}$ represents a non-zero minimum value of the image’s cumulative distribution, and $mn$ indicates the total pixel count.

We employed a Cycle-Consistent Generative Adversarial Neural Network[35] (CycleGAN) for training, using the ’None - Doubtful Stage’ and ’Moderate - Severe Stage’ images as target domains. Images classified as ’Mild Stage’ were utilized as a follow-up validation set after training. CycleGANs, are a specialized type of Generative Adversarial Network (GAN)[42] designed for mapping translation between two unpaired data domains. The unique architecture of a CycleGAN includes two generator networks ($G$ and $F$), and two discriminator networks ($D_{Y}$ and $D_{X}$). The generators ($G$ and $F$) translate data between one domain and the other and vice versa. In contrast, the discriminators ($D_{Y}$ and $D_{X}$) distinguish real images from those generated in their respective domains. The CycleGAN model utilizes a combined loss function comprising both adversarial and cycle consistency losses. The adversarial loss ensures that the generated images appear realistic, while the cycle consistency loss ensures that an image translated to the other domain and subsequently reversed yields the original image. This is represented mathematically as follows:

For the generators $G$ and $F$, $L_{GAN}(G,D_{Y},X,Y)$ and $L_{GAN}(F,D_{X},Y,X)$, provide the adversarial losses. The cycle consistency loss, $L_{cyc}(G,F)$, is a separate term in the loss function. Here, $\lambda$ represents a hyperparameter that controls the weight of the cycle consistency loss. The cycle consistency loss can be further detailed as:

where the first term represents the consistency loss for domain $X$ and the second term for domain $Y$. CycleGANs facilitate unpaired image-to-image translation tasks, which are practical for various applications in computer vision. A graphical representation of our basic architectural blocks and processes can be seen in Figure 3. Additionally, the specifics of our CycleGAN architecture are detailed in tables 3, 4, and 5. Each generator in our model was composed of 11,370,881 parameters, while the discriminators were slightly leaner, comprising 11,032,193 parameters each.

To validate our model, we employed a Convolutional Neural Network (CNN)[46] designed to differentiate between all classes of the Knee Osteoarthritis (KOA) problem (detailed bellow). Our ultimate aim was to fool this CNN into categorizing images into a class of our choice. We applied this strategy by using unseen ’Mild Stage’ test images, which were not used in either the GAN or the CNN training. Firstly, we used the CNN to rank 100 of these test images according to their class confidence level for belonging to the ’Mild Stage.’ Next, we fed these top 100 images to both generators within our CycleGAN. The generators produced two new sets of images, reflecting transformations to both ends of the KOA spectrum: the ’None - Doubtful Stage’ and the ’Moderate - Severe Stage.’ To assess the efficacy of our CycleGAN, we used the CNN to predict the classes of these transformed images and subsequently analyzed the results. We repeated this procedure for GAN and CNN shared test images from the ’None - Doubtful Stage’ and ’Moderate - Severe Stage’ classes. We considered this a ’one-shot’ adversarial attack because we aimed to drastically alter the CNN’s classification with a single application of our CycleGAN.

EfficientNet[47] is a novel model in the domain of deep learning that has earned significant acclaim for its effectiveness. The EfficientNet framework is engineered to uniformly scale all dimensions, including depth, width, and resolution. At the heart of EfficientNet lies the principle of compound scaling. This approach strives to strike a balance between the network’s depth (number of layers), width (the size of the layers), and resolution (size of the input image). A set of fixed scaling coefficients governs this balance. Specifically, for a baseline EfficientNet-B0 (4), if $\alpha,\beta,\gamma$ are constants to maintain the equilibrium, and $\phi$ is the coefficient defined by the user, the depth $d$, width $w$, and resolution $r$ of the network can be scaled as follows:

In this equation, $d_{0},w_{0},r_{0}$ signify the base model’s depth, width, and resolution, respectively.

An integral part of EfficientNet is the MBConv block, which takes inspiration from the MobileNetV2 architecture[48]. This block initiates a sequence of transformations: a $1\times 1$ convolution (expansion), followed by a depth-wise convolution (indicated by a depth-wise separable convolution kernel D), a Squeeze-and-Excitation (SE) operation[49] , and another $1\times 1$ convolution (projection). For an input image I, the transformation of the MBConv block, $T_{MB}$, can be depicted as:

In this formula $\ast$ denotes the convolution operation, $\textbf{{K}}_{1}$ and $\textbf{{K}}_{2}$ are the $1\times 1$ convolutional filters, D illustrates the depth-wise convolutional filter, and $SE(\cdot)$ stands for the Squeeze-and-Excitation operation. An activation function succeeds each convolution. The efficient employment of computational resources within the MBConv block, particularly via the depth-wise convolution and the SE block, significantly contributes to EfficientNet’s superior performance.

EfficientNetV2[50] brings several pivotal changes to the original EfficientNet blueprint. The depth, width, and resolution scaling remain identical to the original EfficientNet. However, the introduction of the Fused-MBConv block, which merges the initial $1\times 1$ convolution and the depth-wise convolution into a single $13\times 3$ convolution operation, followed by the Squeeze-and-Excitation (SE) operation and a final $1\times 1$ convolution, marks a significant adjustment.

The transformation of the Fused-MBConv block, $T_{FMB}$, can be represented as:

In this formula, $\textbf{{K}}_{f}$ is the $3\times 3$ convolutional filter that combines the initial $1\times 1$ convolution and the depth-wise convolution, $\textbf{{K}}_{2}$ is the concluding $1\times 1$ convolutional filter, and $SE(\cdot)$ denotes the Squeeze-and-Excitation operation. Each convolution, followed by an activation, may be in a block with a skip connection.

In our study, we utilized the EfficientNetV2-M model and augmented it by adding a flattening layer followed by a dense layer with 256 neurons. This dense layer employed an Exponential Linear Unit[51] (ELU) as its activation function, placed before the output layer. During the training phase, we used Categorical Cross-Entropy Loss and conducted training over 15 epochs. We divided the dataset into Training, Validation, and Testing sets, allocating 70%, 15%, and 15% respectively (patient aware). This CNN was trained across all three KOA stages. Early stopping was dictated by the minimum validation loss with a tolerance set at 4. The entire model training procedure was carried out using the Deep Fast Vision repository[52]. We used the Adam[53] optimizer for the task, with a learning rate of $2\times 10^{-5}$ and a batch size of 32. Furthermore, we enabled the advanced automatic augmentation capabilities from the library.

While the complexity of neural networks increases their capabilities, it also complicates the interpretation of their predictions[54]. Due to this complexity, these systems are often deemed as ’black boxes.’ However, the Grad-CAM[55] algorithm (based on the CAM[56] framework) helps demystify this ’black box’ effect. At a high level, Grad-CAM is an algorithm that visualizes how a convolutional neural network makes its decisions. It creates what are known as ”heat maps” or ”activation maps” that highlight the areas in an input image that the model considers important for making its prediction.

In this study, we developed a CycleGAN model that can synthesize osteoarthritis’s progression or regression on genuine radiographs. We successfully validated our model by using a Convolutional Neural Network (CNN) called EfficientNetV2M. Our primary objective was to effectively transform ’Mild Stage’ osteoarthritis images using our CycleGAN model, causing the CNN to misclassify the stage of the disease on demand. This confirmed our CycleGAN model’s effectiveness in transforming images to represent either an earlier or later stage of the disease.

We divided our results into three sections for clarity. The first section demonstrates all model training results. The second section highlights top examples from the test set that were used to fool the CNN’s osteoarthritis model. Finally, we globally visualize test images via the t-distributed stochastic neighbor embedding method[57].

As illustrated in Figure 5, there was a substantial enhancement in the quality of the synthetically generated states as the training of the CycleGAN progressed. Notably, once the generators mastered the initial representation of the input images, we observed the emergence of more significant transformations. This suggests an effective learning result within the CycleGAN, as it appears to have first developed an understanding of the input before executing more complex and impactful modifications.

Before we delve into the outcomes produced by our CycleGAN and its capacity to manipulate osteoarthritic radiographs, it is crucial to firstunderstand the performance of our base model, the Convolutional Neural Network (CNN) for Knee Osteoarthritis (KOA) classification. This is the model we attempted to ’fool’ through synthetic images produced by the CycleGAN. Figure 6 presents the confusion matrix and AUCs of the model, while table 1 presents some key metrics of its performance.

The classification report table 1 encapsulates our model’s effectiveness in identifying different stages of osteoarthritis. High precision and recall values for ’None - Doubtful Stage’ and ’Moderate - Severe Stage’ (approximately 0.88) indicate accurate predictions and minimal false negatives for these categories.

Table 2 in the study represents an empirical demonstration of the effectiveness of our method in misleading the CNN classifier, EfficientNetV2. This was achieved by applying the adversarial attack through our CycleGAN model, which altered the test images to misrepresent the original state of knee osteoarthritis.The table displays the original class of the radiographs (None-Early Stage, Moderate - Severe Stage, and Mild Stage), the direction of the adversarial attack (either towards the future, simulating progression, or towards the past, simulating regression of the disease), and the consequent CNN prediction across the three categories (None-Early Stage, Mild Stage, and Moderate - Severe Stage).

For the ’None - Doubtful Stage’ radiographs, when the adversarial attack was directed towards the future, the CNN prediction was largely shifted to the ’Moderate - Severe Stage’ (83.76%). Only a minor fraction was identified as ’None - Doubtful Stage’ (5.13%) or ’Mild Stage’ (11.11%), indicating a substantial alteration in classification due to the attack. The ’Moderate - Severe Stage’ radiographs displayed a similar trend. When manipulated towards the past, a considerable percentage (75.61%) was misclassified as ’None - Doubtful Stage’, diverging significantly from the original classification. When the adversarial attack was applied to the ’Mild Stage’ images towards the future, a majority (76.00%) was misclassified as ’Moderate - Severe Stage’. Conversely, when the attack was directed towards the past, the ’Mild Stage’ images were partly misclassified as ’None - Doubtful Stage’ (31.00%).

Figures 7 and 8 present the top six test images transformed by our CycleGAN. Initially, the CNN classifier was correct and over 90% confident about the osteoarthritis stage represented in these images (Mild stage). However, the classifier was tricked into assigning them an altered stage of the disease, once again with more than 90% confidence in the altered stage.

Figure 7 synthetic images presented the occurrence of joint space narrowing, a key indicator of progressing osteoarthritis. Notably, the CycleGAN model also emphasized or created osteophytes. These bony growths are characteristic of advanced osteoarthritis.

In contrast to Figure 7, figure 8 presents a marked expansion of the knee joint, an attribute commonly associated with no or early-stage osteoarthritic knees. In addition, the possible osteophytes - bony outgrowths typically found in osteoarthritic joints - seemed to be systematically removed by the CycleGAN model. The presence of osteophytes and joint space reduction are hallmark indications of advancing osteoarthritis. Thus, their removal and the expansion of the joint space suggested a transition towards an earlier state of the disease.

Upon examining Figure 9, it was evident that the Convolutional Neural Network (CNN) model focused on key areas in the images that were modified by the Cycle Consistent Generative Adversarial Network (CyGAN). This demonstrated that the CNN based its decisions on the essential features that the CyGAN altered. This showed not only the effectiveness of the CyGAN in generating meaningful features, but also highlighted the CNN’s focus on the relevant regions of interest.

By leveraging the features identified by the CNN, we’ve employed t-SNE[57] to craft a visualization that projected both the original and transformed test images onto a shared space. Our aim with this method was to discern possible overlaps between synthetic and authentic radiographs.This mapping, depicted in Figure 10, provided a compelling visualization of the connections between different image states. A rasterized rendition[58, 59] of this t-SNE plot further refined our understanding, offering an enhanced view of these intricate relationships. Looking at the figure, a notable observation was the appearance of a pseudo-linear diagonal, marking an intriguing transition from early to middle, and eventually to late stages of the image states. This observation was particularly striking as it suggested a structured, semi-ordered transition. As anticipated, overlap was observed in the representation, indicating shared features or similarities among different image states. This overlap was not surprising and reaffirmed the inherent interconnectedness of these transformations.