The University of Tokyo

Convolutional Neural Networks (CNNs) have revolutionized Medical Image Analysis by extracting valuable insights for better clinical examination and medical intervention; the CNNs occasionally outperformed even expert physicians in diagnostic accuracy when large-scale annotated datasets were available [1, 2]. However, obtaining such massive datasets often involves the following intrinsic challenges [3, 4]: (i) it is costly and laborious to collect medical images, such as Magnetic Resonance (MR) and Computed Tomography (CT) images, especially for rare disease; (ii) it is time-consuming and observer-dependent, even for expert physicians, to annotate them due to the low pathological-to-healthy ratio. To tackle these issues, researchers have mainly focused on extracting as much information as possible from the available limited data [5, 6]. Instead, Generative Adversarial Networks (GANs) [7] can generate realistic but completely new samples via many-to-many mappings, and thus effectively cover the real image distribution; they showed great promise in Data Augmentation (DA) using natural images, such as 21% performance improvement in eye-gaze estimation [8].

Interpolation refers to new data point construction within a discretely-sampled data distribution. In terms of the interpolation, the GAN-based image augmentation is reliable on the medical images because medical modalites (e.g., X-ray, CT, MRI) can display the human body’s strong anatomical consistency at fixed position while clearly reflecting inter-subject variability [9, 10]—this is different from the natural images, where various objects can appear at any position; accordingly, to tackle large inter-subject, inter-pathology, and cross-modality variability [3, 4], we propose to use noise-to-image GANs (e.g., random noise samples to diverse pathological images) for (i) medical DA and (ii) physician training [11]. The noise-to-image GAN training is much more difficult than training image-to-image GANs (e.g., a benign image to a malignant one); but, it can perform more global regularization (i.e., adding constraints when fitting a loss function on a training set to prevent overfitting) and increase image diversity for further performance boost.

Regarding the DA, the GAN-generated images can improve Computer-Aided Diagnosis (CAD) based on supervised learning [12]. For the physician training, the GANs can display novel desired pathological images and help train medical trainees despite infrastructural and legal constraints [13]. However, we cannot directly use conventional GANs for realistic/diverse high-resolution medical image augmentation. Moreover, we have to find effective loss functions and training schemes for each of those applications [14]; the diversity matters more for the DA to sufficiently fill the real image distribution whereas the realism matters more for the physician training not to confuse the medical students and radiology trainees.

So, how can we perform clinically relevant GAN-based DA/physician training using only limited annotated training images? Always in collaboration with physicians, for improving 2D classification, we combine the noise-to-image [15, 16] (i.e., Progressive Growing of GANs, PGGANs [17]) and image-to-image GANs (i.e., Multimodal UNsupervised Image-to-image Translation, MUNIT [18]); the two-step GAN can generate and refine realistic/diverse original-sized $256\times 256$ brain MR images with/without tumors separately. Nevertheless, further DA applications require pathology localization for detection (i.e., identifying target pathology positions in medical images) and advanced physician training needs atypical image generation, respectively. To meet both clinical demands, we propose novel 2D/3D bounding box-based GANs conditioned on pathology position/size/appearance; the bounding box-based detection requires much less physicians’ annotation effort than rigorous segmentation.

Extrapolation refers to new data point estimation beyond a discretely-sampled data distribution. While it is not mutually-exclusive with the interpolation and both rely on a model’s restoring force, it is more subject to uncertainty and thus a risk of meaningless data generation. In terms of the extrapolation, the pathology-aware GANs (i.e., the conditional GANs controlling pathology, such as tumors and nodules, based on position/size/appearance) are promising because common and/or desired medical priors can play a key role in the conditioning—theoretically, infinite conditioning instances, external to the training data, exist and enforcing such constraints have an extrapolation effect via model reduction [19]; inevitable errors, not limited between two data points, caused by the model reduction forces a generator to synthesize images that the generator has never synthesized before.

For improving 2D detection, we propose Conditional PGGANs (CPGGANs) that incorporates highly-rough bounding box conditions incrementally into the noise-to-image GAN (i.e., the PGGANs) to place realistic/diverse brain metastases at desired positions/sizes on $256\times 256$ MR images [20]. As its pathology-aware conditioning, we use 2D tumor position/size on MR images. Since lesions vary in 3D position/appearance, for improving 3D detection, we propose 3D Multi-Conditional GAN (MCGAN) that translates noise boxes into realistic/diverse $32\times 32\times 32$ lung nodules placed naturally at desired position/size/attenuation on CT scans [21]; inputting the noise box with the surrounding tissues has the effect of combining the noise-to-image and image-to-image GANs. As its pathology-aware conditioning, we use 3D nodule position/size/attenuation on CT scans.

Lastly, we confirm our pathology-aware GANs’ clinical relevance for diagnosis as a clinical decision support system and non-expert physician training tool via two discussions: (i) Conducting a questionnaire survey about our GAN projects for 9 physicians; (ii) Holding a workshop about how to develop medical Artificial Intelligence (AI) fitting into a clinical environment in five years for 7 professionals with various AI and/or Healthcare background.

This Ph.D. thesis aims to present the clinical relevance of our novel pathology-aware GAN applications, medical DA and physician training, always in collaboration with physicians.

The thesis is organized as follows (Fig. 1.1). Chapter 2 covers the background of Medical Image Analysis and Deep Learning, as well as methods to address data paucity to bridge them. Chapter 3 describes related work on the GAN-based medical DA and physician training, which emerged after our proposal to use noise-to-image GANs for those applications in Chapter 4. Chapter 5 presents a two-step GAN for 2D classification that combines both noise-to-image and image-to-image GANs. Chapter 6 proposes CPGGANs for 2D detection that incorporates highly-rough bounding box conditions incrementally into the noise-to-image GAN. Finally, we propose 3D MCGAN for 3D detection that translates noise boxes into desired pathology in Chapter 7. Chapter 8 discusses both our pathology-aware GANs’ clinical relevance via a questionnaire survey and how to develop medical AI fitting into a clinical environment in five years via a workshop. Lastly, Chapter 9 provides the conclusive remarks and future directions for further GAN-based extrapolation.

Medical Image Analysis refers to the process of increasing clinical examination/medical intervention efficiency, based on several imaging modalities and digital image processing techniques [22, 23]; to effectively visualize the human body’s anatomical and physiological features, it covers various modalities including X-ray, CT, MRI, positron emission tomography, endoscopy, optical coherence tomography, pathology, ultrasound imaging, and fundus imaging. Its tasks are mainly classified into three groups: (i) Early detection/diagnosis/prognosis of disease often based on pathology classification/detection/segmentation and survival prediction [24, 25]; (ii) Clinical workflow enhancement often based on body part segmentation, inter-modality registration, 3D reconstruction, flow measurement, and surgery simulation [26, 27]; (iii) Clinically impossible image analysis, such as radiogenomics that identifies the correlation between cancer imaging features and gene expression [28].

Among the various modalities, this thesis focuses on the most common 3D modalities for non-invasive diagnosis, CT and MRI. To get a detailed picture inside the body, the CT merges multiple X-rays at different angles using computational tomographic reconstruction [9, 29]. Since X-ray intensity is associated with the mass attenuation coefficient, higher-density tissues show higher attenuation and vice versa. Accordingly, each voxel possesses its attenuation value following the Hounsfield scale from $-1,000$ to $+1,000$ (e.g., Hounsfield units $-1,000$ for air, 0 for water, and $+1,000$ for dense bone). The CT can provide a outstanding contrast within soft-tissue, bone, and lung while the soft-tissue contrast is poor—accordingly, it is especially performed for comprehensive lung assessment.

The MRI uses magnetization properties of atomic nuclei [10, 30]. Since different tissues show various relaxation processes when the nuclei return to their resting alignment, the tissues’ proton density maps serve as both anatomical and functional images. Since the tissues possess two different relaxation times, T1 (i.e., longitudinal relaxation time) and T2 (i.e., transverse relaxation time), as MRI sequences, we can obtain both T1-weighted (T1) and T2-weighted (T2) images. Moreover, using very long repetition time and time to echo, we can obtain FLuid Attenuation Inversion Recovery (FLAIR) images. The MRI can provide a superior soft-tissue contrast to the CT—accordingly, it is especially performed for comprehensive brain assessment.

Deep Learning is a kind of Machine Learning algorithms, based on Artificial Neural Networks [31]. The Deep Neural Networks consist of many linearly connected non-linear units whose parameters are optimized by gradient descent [32]; accordingly, their multiple layers can gradually grasp more-detailed features as training progresses (i.e., learning which features to place is automatic). Thanks to the good generalization ability, under large-scale data, the Deep Learning significantly outperforms classical Machine Learning algorithms relying on feature engineering. A visual cortex includes arrangements of simple and complex cells activated by a receptive field (i.e., subregions of a visual field); inspired by this biological structure [33], CNNs adopt a mathematical operation called convolution to achieve translation invariance [34]. Since the CNNs are excellent at image/video recognition, their diverse medical applications include pathology classification/detection/segmentation and survival prediction [24, 25].

Variational AutoEncoders (VAEs) often suffer from blurred samples despite easier training, due to the imperfect reconstruction using a single objective function [35]; meanwhile, GANs have revolutionized image generation in terms of realism and diversity [36], including denoising [37] and MRI-to-CT translation [38], based on a two-player objective function using two CNNs [7]: a generator $G$ tries to generate realistic images to fool a discriminator $D$ while maintaining diversity; $D$ attempts to distinguish between the real and synthetic images. However, difficult GAN training from the two-player objective function accompanies artifacts and mode collapse [39], when generating high-resolution images (e.g., $256\times 256$ pixels) [40]; to tackle this, multi-stage noise-to-image GANs have been proposed: AttnGAN generated images from text using attention-based multi-stage refinement [41]; PGGANs generated realistic images using low-to-high resolution multi-stage training [17].

Contrarily, to obtain images with desired texture and shape, some researchers have proposed image-to-image GANs: MUNIT translated images using both GANs and VAEs [18]; Simulated and unsupervised learning GAN (SimGAN) translated images for DA using the self-regularization term and local adversarial loss [8]; Isola et al. proposed Pix2Pix GAN to produce robust images using paired training samples [42]. Others have proposed conditional GANs: Reed et al. proposed bounding box-based conditional GAN to control generated images’ local properties [43]; Park et al. proposed multi-conditional GAN to refine base images based on texts describing desired position [44].

In Healthcare, medical images have generated the largest volume of data and this trend will no doubt increase due to equipment improvement [45, 46]. Accordingly, as the Deep Learning dominates Computer Vision, Medical Image Analysis is not an exception; their combination can analyze the large-scale medical images and extract valuable insights for better clinical examination and medical intervention. However, the biggest challenge to bridge them lies in the difficulty of obtaining desired pathological images, especially for rare disease [3, 4]. Moreover, it is time-consuming and observer-dependent, even for expert physicians, to annotate them.

So, how can we tackle the data paucity? We can either attempt to (a) overcome the lack of generalization or (b) overcome difficulties in optimization. The most straightforward and effective way to address the generalization is DA [47, 48]; because the best model when given data is uncertain, we commonly increase training set size. Human perception is invariant to size, shape, brightness, and color [49]. Accordingly, we recognize the same objects while their such features change, and thus intentionally changing the features is plausible to obtain more data. Such classical DA include (i) x/y/z-axis flipping and rotating, (ii) zooming and scaling, (iii) cropping, (iv) translating, (v) elastic deformation, (vi) adding Gaussian noise (i.e, the distortion of high frequency features), and (vii) brightness and contrast fluctuation.

Recent DA techniques focus on regularization: Mixup [50] and Between-class learning [51] mixed two images during training, such as a dog image and a cat one, for regularization; Cutout randomly masked out square regions during training for regularization [52]; CutMix combined the Mixup and Cutout [53]. As a recent impressive DA approach, AutoAugment automatically searched for improved DA policies [54]. Moreover, similarly to the Mixup among all images within the same class, GAN-based DA can fill the uncovered real image distribution by generating realistic and diverse images via many-to-many mapping [55].

Along with the DA, researchers proposed many other techniques to improve the generalization: semi-supervised learning can considerably increase accuracy under limited labeled data by using pseudo labels for unlabeled data [5]; unsupervised anomaly detection allows to detect out-of-distribution images from normal ones, such as disease, without any labeled data [56]; regularization techniques, such as dropout [57], Lasso [58], and elastic net [59], are commonly used for reducing overfitting; similarly, ensembling multiple models via bagging [60] and boosting [61] can effectively increase the robustness; Lastly, in Medical Image Analysis, we can fuse multiple image modalities and/or sequences, such as MRI $+$ CT [62] and T1 MRI $+$ T2 MRI [63].

Moreover, many techniques exist for overcoming the difficulties in optimization: transfer learning can achieve better parameter initialization [64]; problem reduction, such as inputting 2D/3D image patches instead of a whole image, can eliminate unnecessary parameters [65]; learning methods with less data, such as zero-shot learning [66], one-shot learning [6], and neural Turing machine [67], are also promising; meta-learning promotes a versatile model applicable to various tasks without requiring multiple training from scratch [68].

Because the lack of annotated pathological images is the greatest challenge in CAD [3, 4], to handle various types of small/fragmented datasets from multiple scanners, researchers have actively conducted GAN-based DA studies especially in Medical Image Analysis. For better classification, some researchers adopted image-to-image GANs similarly to their conventional medical applications, such as denoising [37] and MRI-to-CT translation [38]: Wu et al. translated $256\times 256$ normal mammograms into lesion ones [70], Gupta et al. translated $1024\times 512$ normal leg X-ray images into bone lesion ones [71], and Malygina et al. translated $256\times 256$/$512\times 512$ normal chest X-ray images into pneumonia/pleural-thickening ones [72]. Meanwhile, others adopted the noise-to-image GANs as we proposed, to increase image diversity for further performance boost—the diversity matters more for the DA to sufficiently fill the real image distribution: Frid-Adar et al. augmented $64\times 64$ liver lesion CT images [12], Madani et al. augmented $128\times 128$ chest X-ray images with cardiovascular abnormality [73], and Konidaris et al. augmented $192\times 160$ brain MR images with Alzheimer’s disease [74].

To facilitate pathology detection and segmentation, researchers conditioned the image-to-image GANs, not the noise-to-image GANs like our work in Chapter 6, with pathology features (e.g., position, size, and appearance) and generated realistic/diverse pathology at desired positions in medical images. In terms of extrapolation, the pathology-aware GANs are promising because common and/or desired medical priors can play a key role in the conditioning—theoretically, infinite conditioning instances, external to the training data, exist and enforcing such constraints have an extrapolation effect via model reduction [19]. To the best of our knowledge, only Kanayama et al. tackled bounding box-based pathology detection using the image-to-image GAN [75]; they translated normal endoscopic images with various image sizes ($458\times 405$ on average) into gastric cancer ones by inputting both a benign image and a black image (i.e., pixel value: 0) with a specific lesion Region Of Interest (ROI) at desired position. Without conditioning the noise-to-image GAN with nodule position, Gao et al. generated $40\times 40\times 18$ 3D nodule subvolumes only applicable to their subvolume-based detector using binary classification [76].

Since 3D imaging is spreading in radiology (e.g., CT and MRI), most GAN-based DA works for segmentation exploited 3D conditional image-to-image GANs. However, 3D medical image generation is more challenging than 2D one due to expensive computational cost and strong anatomical consistency; so, instead of generating a whole image including pathology, researchers only focused on a malignant Voxel Of Interest (VOI): Shin et al. translated $128\times 128\times 54$ normal brain MR images into tumor ones by inputting both a benign image and a tumor-conditioning image [77], similarly to the Kanayama et al.’s work [75]; Jin et al.  generated $64\times 64\times 64$ CT images of lung nodules including the surrounding tissues by only inputting a VOI centered at a lung nodule, but with a central sphere region erased [78]. Recently, instead of generating realistic images and training classifiers on them separately, Chaitanya et al. directly optimized segmentation results on cardiac MR images [26]; however, it segmented body parts, instead of pathology. Since effective GAN-based medical DA generally requires much engineering effort, we also published a tutorial journal paper [14] about tricks to boost classification/detection/segmentation performance using the GANs, based on our experience and related work.

While medical students and radiology trainees must view thousands of images to become competent [79], accessing such abundant medical images is often challenging due to infrastructural and legal constraints [80]. Because pathology-aware GANs can generate novel medical images with desired abnormalities (e.g., position, size, and appearance)—while maintaining enough realism not to confuse the medical trainees—GAN-based physician training concept is drawing attention: Chuquicusma et al. appreciated the GAN potential to train radiologists for educational purpose after successfully generating $56\times 56$ CT images of lung nodules that even deceived experts [81]; thanks to their anonymization ability, Shin et al. proposed to share pathology-aware GAN-generated images outside institutions after achieving considerable tumor segmentation results with only synthetic $128\times 128\times 54$ MR images for training [77]; more importantly, Finlayson et al. from Harvard Medical School are currently validating a class-conditional GANs’ radiology educational efficacy after succeeding in learning features that distinguish fractures from non-fractures on $1024\times 1024$ pelvic X-ray images [13].

Along with classic methods [85], CNNs have recently revolutionized medical image analysis [86], including brain MRI segmentation [87]. However, CNN training demands extensive medical data that are laborious to obtain [88]. To overcome this issue, DA techniques via reconstructing original images are common for better performance, such as geometry and intensity transformations [89, 90].

However, those reconstructed images intrinsically resemble the original ones, leading to limited performance improvement in terms of generalization abilities; thus, generating realistic (similar to the real image distribution) but completely new images is essential. In this context, GAN-based DA has excellently performed in general computer vision tasks. It attributes to GAN’s good generalization ability from matching the noise-generated distribution to the real one with a sharp value function. Especially, Shrivastava et al. (SimGAN) outperformed the state-of-the-art with a relative 21% improvement in eye-gaze estimation [8].

So, how can we generate realistic medical images completely different from the original samples? Our aim is to generate synthetic multi-sequence brain MR images using GANs, which is essential in medical imaging to increase diagnostic reliability, such as via DA in CAD as well as physician training (Fig. 4.1) [91]. However, this is extremely challenging—MR images are characterized by low contrast, strong visual consistency in brain anatomy, and intra-sequence variability. Our novel GAN-based approach for medical DA adopts Deep Convolutional Generative Adversarial Network (DCGAN) [40] and WGAN [83] to generate realistic images, and an expert physician validates them via Visual Turing Test [92].

Towards clinical applications utilizing realistic brain MR images, we generate synthetic brain MR images from the original samples using GANs. Here, we compare the most used two GANs, DCGAN and WGAN, to find a well-suited GAN between them for medical image generation—it must avoid mode collapse and generate realistic MR images with high resolution.

This section shows how DCGAN and WGAN generate synthetic brain MR images. The results include instances of synthetic images and their quantitative evaluation of the realism by an expert physician.

Our preliminary results show that GANs, especially WGAN, can generate $128\times 128$ realistic multi-sequence brain MR images that even an expert physician is unable to accurately distinguish from the real, leading to valuable clinical applications, such as DA and physician training. This attributes to WGAN’s good generalization ability with a sharp value function. In this context, DCGAN might be unsuitable due to both inferior realism and mode collapse in terms of intensity. We only use slices of interest in training to obtain desired MR images and generate both original/Concat sequence images for DA in medical imaging.

This study confirms the synthetic image quality by the human expert evaluation, but a more objective computational evaluation for GANs should also follow, such as Classifier Two-Sample Tests (C2ST) [94], which assesses whether two samples are drawn from the same distribution. Currently this work uses sagittal MR images alone, so we plan to generate coronal and transverse images. As this research uniformly selects middle slices in pre-processing, better data generation demands developing a classifier to only select brain MRI slices with/without tumors.

Towards DA, whereas realistic images give more insights on geometry/intensity transformations in classification, more realistic images do not always assure better DA, so we have to find suitable image resolutions and sequences; that is why we generate both high-resolution images and Concat images, yet they looked more unrealistic for the physician. For physician training, generating desired realistic tumors by adding conditioning requires exploring latent spaces of GANs extensively.

Overall, our novel GAN-based realistic brain MR image generation approach sheds light on diagnostic and prognostic medical applications; future studies on these applications are needed to confirm our encouraging results.

CNNs are playing a key role in Medical Image Analysis, updating the state-of-the-art in many tasks [87, 97, 98] when large-scale annotated training data are available. However, preparing such massive medical data is demanding; thus, for better diagnosis, researchers generally adopt classic DA techniques, such as geometric/intensity transformations of original images [89, 90]. Those augmented images, however, intrinsically have a similar distribution to the original ones, resulting in limited performance improvement. In this sense, GAN-based DA can considerably increase the performance [7]; since the generated images are realistic but completely novel samples, they can relieve the sampling biases and fill the real image distribution uncovered by the original dataset [99].

The main problem in CAD lies in small/fragmented medical imaging datasets from multiple scanners; thus, researchers have improved classification by augmenting images with noise-to-image GANs [11] or image-to-image GANs [70]. However, no research has achieved further performance boost by combining noise-to-image and image-to-image GANs.

So, how can we maximize the DA effect under limited training images using the GAN combinations? To generate and refine brain MR images with/without tumors separately (Fig. 5.1), we propose a two-step GAN-based DA approach: (i) PGGANs [17], low-to-high resolution noise-to-image GAN, first generates realistic/diverse $256\times 256$ images—the PGGANs helps DA since most CNN architectures adopt around $256\times 256$ input sizes (e.g., InceptionResNetV2 [95]: $299\times 299$, ResNet-50 [96]: $224\times 224$); (ii) MUNIT [18] that combines GANs/VAEs [35] or SimGAN [8] that uses a DA-focused GAN loss, further refines the texture and shape of the PGGAN-generated images to fit them into the real image distribution. Since training a single sophisticated GAN system is already difficult, instead of end-to-end training, we adopt a two-step approach for performance boost via an ensemble generation process from those state-of-the-art GANs’ different algorithms.

We thoroughly investigate CNN-based tumor classification results, also considering the influence of pre-training on ImageNet [34] and discarding weird-looking GAN-generated images. Moreover, we evaluate the synthetic images’ appearance via Visual Turing Test [92] by an expert physician, and visualize the data distribution of real/synthetic images via t-Distributed Stochastic Neighbor Embedding (t-SNE) [100]. When combined with classic DA, our two-step GAN-based DA approach significantly outperforms the classic DA alone, boosting sensitivity $93.67\%$ to $97.48\%$.

This section shows how PGGANs generates synthetic brain MR images and how MUNIT and SimGAN refine them. The results include instances of synthetic images, their quantitative evaluation by a physician, their t-SNE visualization, and their influence on tumor classification.