CAS-GAN for Contrast-free Angiography Synthesis

Generative adversarial networks (GANs), introduced by Goodfellow et al. [17] in 2014, revolutionized machine learning by establishing a minimax optimization game between a generator $\bm{G}$ and a discriminator $\bm{D}$. The generator aims to mimic the distribution of real data samples, while the discriminator evaluates their authenticity. Up till now, GANs have been applied to numerous computer vision tasks including medical image generation [18], image super-resolution [19], and data augmentation [20]. Despite their effectiveness, training GANs remains challenging due to issues such as gradient vanishing and mode collapse [21], [22], [23]. Many variants have been developed to address these challenges with more stable training objectives [24], [25], [26]. In our study, we employ GANs to generate X-ray angiographies from non-contrast X-ray images.

Image-to-image translation involves transforming images from one domain to another while maintaining their underlying structures or contents [27], [28], [29], [30]. Isola  et al. [10] pioneered this approach with a conditional GAN framework that utilizes both adversarial [17] and L1 reconstruction losses for training on paired data. However, obtaining a large number of paired data is often impractical, especially in clinical settings. Recent advancements have focused on unpaired data translation, introducing additional constraints like cycle-consistency [11], [31], [12] to preserve semantic consistency during domain transfer. This constraint ensures that an image converted into another domain and back again should remain largely unchanged, thereby maintaining the original content. Additionally, some studies have incorporated self-supervised contrast loss to enhance semantic relationships between different domains [32], [33], [34].

In the medical imaging field, most works focused on 3D medical imaging modalities, including CT $\rightarrow$ CTA [35], MR $\rightarrow$ CT [36], image denoising [37], etc. Notably, the most relevant studies to our work are [29], [38]. These works involve generating realistic X-ray angiographies from simulated images derived from projections of 3D vessel models. Our research extends these applications by focusing on the generation of vessels in non-contrast X-ray images, presenting a more complex challenge.

Disentanglement representation learning is gaining traction for its potential to increase the controllability and interpretability of generated images in image-to-image translation tasks [39], [40]. Many works attempt to disentangle images into content and style representations to achieve image translation through swapping style representations [41], [42], [43]. However, these representations are abstract concepts, as they cannot be directly visualized within the image space. Some other works have opted to disentangle images into components with physical significance [44], [45], such as separating scenes from elements like raindrops, occlusions, and fog [45]. Inspired by these efforts, our work seeks to disentangle X-ray angiographies into background and vessel components and construct the one-to-one mappings of these two components via neural networks.

In a specific X-ray angiography $y\in\mathcal{Y}$, it can be conceptualized as a composite of a background (i.e., non-contrast X-ray image $x\in\mathcal{X}$) and vessel components, consistent with its imaging principles [47]. Physicians can infer vessel morphology from prior anatomical details observed in the backgrounds before injecting contrast agents. Based on this observation, our approach aims to disentangle X-ray angiographies into background and vessel components, subsequently learning a predictor to establish interrelationships between these components. Specifically, we execute this disentanglement in the latent space using two specialized encoders: a background encoder $\bm{E}_{\rm BG}$ and a vessel encoder $\bm{E}_{\rm Vess}$. Note that we assume the background components of $\mathcal{X}$ and $\mathcal{Y}$ share the same latent space according to the above analysis. The encodings are formulated as follows:

where $z_{x}^{\rm BG}$, $z_{y}^{\rm BG}$, and $z_{y}^{\rm Vess}\in\mathbb{R}^{\frac{C}{r}\times\frac{H}{r}\times\frac{W}{r}}$ are latent space representations. $C$, $H$, and $W$ denote the channel number, height, and width of X-ray images ($x$ and $y$), and $r$ indicates the downsampling ratio of the encoders ($r=4$ in default). Given the absence of $z_{x}^{\rm Vess}$ in the forward cycle depicted in Fig. 2, we utilize the paired $\left\{z_{y}^{\rm BG},z_{y}^{\rm Vess}\right\}$ to train the predictor $\bm{M}\left(\cdot\right)$. With the powerful generalization ability of the predictor, the vessel representations $z_{x}^{\rm Vess}$ of $x$ can be easily derived:

For the backward cycle, we employ a single generator $\bm{G}_{\rm BG}$ to generate non-contrast X-ray images directly, because this task is comparatively simpler than the forward cycle. The mathematical formulation for the backward cycle $g:\mathcal{Y}\rightarrow\mathcal{X}$ is presented below:

The single generator approach is insufficient for the forward cycle, as it cannot discern the most discriminative features of the vessels. Drawing inspiration from attention mechanisms, we introduce a vessel semantic-guided generation to address this limitation. Rather than direct image generation, we deploy two specialized generators: $\bm{G}_{\rm Vess}^{\rm Attn}$ for attention masks $A_{g}\in\mathbb{R}^{1\times H\times W}$ and $\bm{G}_{\rm Vess}^{\rm Ctx}$ for context masks $C_{g}\in\mathbb{R}^{C\times H\times W}$. We employ Sigmoid to replace Tanh as the activation function in $\bm{G}_{\rm Vess}^{\rm Attn}$.

Finally, the corresponding attention masks $A_{\rm g}$, context masks $C_{\rm g}$, and non-contrast X-ray images $x$ are fused to synthesize X-ray angiographies.

The adversarial loss is fundamental in training generative adversarial networks (GANs). However, when two image domains are visually similar, traditional adversarial losses applied directly in the image space may not be sufficiently sensitive. For instance, the primary distinction between non-contrast X-ray images and X-ray angiographies often lies in vascular details, which may not always be distinctly opacified, rendering these images visually akin to one another. To address this, we introduce a vessel semantic-guided adversarial loss, focusing the adversarial training on semantic differences highlighted by vascular information.

Specifically, we employ a pre-trained U-Net [48] to extract vessel semantic images from both the original $y$ and the generated $y_{\rm g}$ angiographies:

The vessel semantic-guided adversarial loss is then defined as follows:

This loss function specifically targets the semantic representations of vessels, thereby enhancing the GAN’s ability to discriminate between real and generated images based on vascular features, rather than general imaging characteristics.

Due to the highly under-constrained nature of the mappings between the two image domains, incorporating a variety of effective loss functions is crucial for training CAS-GAN.

We adopt the same generator and discriminator architectures in CycleGAN [10] for fair comparisons. The predictor within our model is implemented using a multi-layer perceptron (MLP) with a configuration of $C^{\prime}\rightarrow C^{\prime}/2\rightarrow C^{\prime}/4\rightarrow C^{\prime}/2\rightarrow C^{\prime}$, where $C^{\prime}$ is set to $256$ in default. The whole framework is implemented based on PyTorch 2.1.1 [49], Python 3.8.18, and Ubuntu 20.04.6 with an NVIDIA GeForce RTX 4090 GPU. We set hyperparameters of CAS-GAN consistent with CycleGAN [11]. The input image size is set to $256\times 256$. Adam optimizer [50] is adopted to train our model for 1,000 epochs. The initial learning rate is set to $2e^{-4}$ and is linearly reduced to zero after 700 epochs. Hyperparameters $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$, and $\lambda_{5}$ are set to $0.5$, $10.0$, $1.0$, $1.0$, and $0.5$.

To evaluate the performance of CAS-GAN and other state-of-the-art methods, two popular utilized metrics in medical image translation are selected:

To demonstrate the effectiveness of CAS-GAN in X-ray angiography synthesis task, we compared it against several leading unpaired image-to-image translation methods.

Quantitative results on the XCAD dataset are presented in Table I. As seen, our CAS-GAN outperforms all baselines on FID and MMD. Compared with other methods, CAS-GAN and AttentionGAN which utilize attention-guided generation can produce more realistic results, reflecting higher FID. However, due to the integration of disentanglement representations and the vessel semantic-guided loss, CAS-GAN achieves superior performance compared to AttentionGAN on both FID and MMD.

Fig. 3 showcases qualitative results generated by several baselines and our CAS-GAN. Reference images are real X-ray angiographies obtained by injecting contrast agents¹¹1Since reference and non-contrast X-ray images are captured at different moments in the cardiac cycle, quantitative metrics are not applicable for measuring their similarities.. As seen, baselines frequently exhibit significant entanglement of backgrounds and vessels, leading to highly unrealistic outputs. This issue is particularly evident in the first and third rows of Figure 3, where there is a total loss of structural consistency in vessels. Moreover, baselines fail to infer vessel bifurcations based on background prior. For example, in the second row of Figure 3, all baseline models cannot generate a crucial bifurcation at the position indicated by the catheter. In contrast, CAS-GAN can effectively disentangle and understand the complex interrelationships between backgrounds and vessels. This capability allows it to maintain structural integrity and precisely generate vessel bifurcations, markedly outperforming the baselines.

Extensive ablation experiments are conducted to verify the efficacy of several designs in CAS-GAN. Default configurations are highlighted in gray. Quantitative results are detailed in Table II. Our main observations are as follows: i) Learning disentanglement representations facilitates the model’s ability to synthesize accurate and realistic vessels based on the given backgrounds, thus leading to $0.11\sim 1.89$ FID improvements; ii) Vessel semantic-guided generator is crucial as it directs the model’s focus toward target vessel regions while preserving backgrounds of the original images; iii) Vessel semantic-guided loss can further refine the realism of the generated X-ray angiographies. However, it is observed that integrating VSGL into a baseline model (i.e., Index 1) deteriorates performance. This suggests that VSGL requires the foundational enhancements provided by DRL and VSGG to be effective. Without them, the model struggles to generate high-quality vessels, and VSGL may exacerbate this by trying to force realism in poorly generated images.

We also conduct external validations to verify the generalization performance of different models. The data is derived from a collaborating hospital, consisting of 11 X-ray sequences from different patients. From each sequence, we extract two frames (before and after injecting contrast agents) to conduct experiments. Note that we directly evaluate models trained on the XCAD dataset without fine-tuning. Fig. 4 presents several qualitative results.

Despite all methods struggling with the significant variance between the XCAD and external datasets, CAS-GAN showcased relatively superior performance. The specialized design of our model, particularly its disentanglement approach, contributed to its better handling of the dataset’s challenges. In contrast, models like CycleGAN [11], MUNIT [41], and CUT [32] altered the image appearances significantly, while AttentionGAN [54] tended to generate unrealistic vessel structures, entangling with backgrounds. We anticipate that enhancing the dataset size and diversity will significantly improve model performance on external datasets and we will develop a more comprehensive dataset in future works.