Brain PET-MR attenuation correction with deep learning: method validation in adult and clinical paediatric data

The training dataset consisted of data from research studies at the KCL & GSTT PET Centre (“PET Centre dataset”). Research studies were reviewed and approved by regional research ethics committees: West Midlands – Coventry & Warwickshire Research Ethics Committee (16/WM/0364), and West London & GTAC Research Ethics Committee (16/LO/0130). All participants provided written informed consent.

This dataset consisted of 110 individuals who were scanned as part of three PET-MR research studies at the KCL & GSTT PET Centre between 2016 and 2019 - the [¹⁸F]fluorodeoxyglucose ([¹⁸F]FDG) study, the [¹⁸F]GE-179 study and the [¹⁸F]Fallypride study. All low-dose reference CT data were acquired on a whole-body GE Discovery 710 PET/CT system on the same day as the MR scan. All MR data were acquired on a 3T Siemens Biograph mMR PET-MR system. Participant demographics and MR and CT acquisition details for each sub-group of data are given in Table 1. This dataset consisted primarily of adult data, however, there were two children aged 9 and nine teenagers between the ages of 13 to 17 years.

The “clinical paediatric dataset” consisted of anonymised clinical scans acquired at Guy’s and St Thomas’ Hospital available through an NHS clinical service evaluation of patients with dystonia from 2013 to 2017 (Service Evaluation GSTT Reference 10319). Scans were performed as part of standard clinical assessment for possible deep brain stimulation and included MR imaging, [¹⁸F]FDG-PET, and head CT under general anaesthesia. We included data from patients who had a complete set of [¹⁸F]FDG-PET, 3D T1-weighted MR and CT scans, resulting in a dataset consisting of 49 patients between 2 and 17 years old (mean age ± SD = 9.3 ± 4.3 years). Patients underwent [¹⁸F]FDG-PET-CT imaging on a GE (General Electric Medical Systems, Waukesha, WI) Discovery ST and a Discovery VCT scanner prior to October 2013, and thereafter on a GE Discovery 710 scanner, at the King’s College London & Guy’s and St Thomas’ PET Centre. CT images had voxel sizes of 0.5 × 0.5 × 2.5 mm³ (n = 32) or 0.977 × 0.977 × 3 mm³ (n = 17). 3D T1-weighted images were acquired in sagittal orientation with voxel sizes of 0.694 × 0.694 × 0.68 mm³ (n = 15), 0.977 × 0.977 × 1 mm³ (n = 28), 0.446 × 0.446 × 1 mm³ (n = 1), 0.898 × 0.898 × 1 mm³ (n = 1), and approximately 1 × 1 × 1 mm³ (n = 2). One patient had their T1-weighted MR image acquired in transverse orientation (voxel size = 0.55 × 0.55 × 1.1 mm³).

T1-weighted MR data were first bias corrected using N4ITK bias field correction [20]. Each individual’s CT image was rigidly realigned and resampled to their T1-weighted image using NiftyReg [17] with cubic spline interpolation, after the scanner bed was edited out of the CT images using custom scripts. In the PET Centre dataset, where voxel sizes were $\leq$ 1.1 mm, we did not apply any resampling but instead allowed the network treat this as a variation in head size, similar to a scaling augmentation (± 10%). In the clinical paediatric dataset, we resampled all data to 1 mm³ isotropic resolution to make it compatible with the resolution of the PET Centre dataset.

The convolutional neural network (CNN) we used for image translation is based on a residual U-Net architecture [18, 7, 10]. The network consists of blocks of 3D convolutions (kernel size = 3), each followed by instance normalisation [8] and a Parametric Rectified Linear Unit (PReLU) [6] activation. Our U-Net has five layers of these convolutional blocks with 32, 64, 128, 256 and 512 channels in each consecutive layer. In the encoding path, instead of max pooling, downsampling is performed using convolutions with a stride of 2 and a residual connection is added. In the decoding path, we use a modified 3D version of the pixel shuffle method [19] to upsample data from lower layers by a factor of 2. The upsampled input is then concatenated with the output from the encoding path in the same layer. Figure 1 illustrates the overall topology of our U-Net and the network units in the encoding and decoding paths.

We implemented the above network architecture in PyTorch https://pytorch.org using the MONAI framework [2] https://monai.io on an NVIDIA Quadro M4000 GPU with 8GB of RAM. Our network took a 3D patch from the T1-weighted MR as input and produced the corresponding CT patch.

In order to normalise intensities across each dataset, histogram normalisation was applied to the input T1-weighted MR images using the HistogramNormalize transform in MONAI. MR images were then rescaled between -1 and 1 (ScaleIntensity) and CT images were thresholded at -1000 Hounsfield units (HU) and clipped above 2400 HU (ThresholdIntensity).

The network was trained on batches of eight patches each of size 256 × 256 × 32 voxels sampled randomly from the pairs of images. We used the Adam optimiser with weight decay regularisation [15] with an initial learning rate of 3 × 10^-3, $\beta$₁ = 0.95, $\beta$₂ = 0.99, and weight decay = 0.03 (hyperparameters chosen based on [5]). We defined an epoch as one iteration of a randomly sampled patch over all images in the training dataset.

We performed a five-fold cross-validation of the network, dividing the data into 80% training and 20% testing data for each fold, with one image set from the testing data used for validation. We used the mean squared error loss and tracked the mean absolute error of the validation dataset every five epochs. The network was trained until the validation loss did not improve by more than 1 HU over 100 epochs (approximately 900–1000 epochs in total for all folds). At this point, the learning rate was reduced to 3 × 10^-4 and the network further trained until the validation loss did not improve by more than 1 HU over 100 epochs (approximately 100 to 450 epochs). We took the network weights at the epoch showing the best validation loss as the final weights for each fold of the cross-validation. The full 3D pseudo-CT volumes were produced from T1-w MR images of each test dataset by predicting overlapping patches (sliding_window_inference, overlap = 0.9, blend mode = gaussian).

We evaluated the accuracy of pseudo-CTs produced using our deep learning method against reference CTs using the mean absolute error (MAE) metric [1] defined as MAE=$\Sigma$($|pCT-CT|$/V), where V is the number of non-zero voxels within a head mask. The head mask was obtained by cropping the T1-weighted MR image to have a 17cm cranio-caudal extent so as to exclude the neck using robustfov (https://fsl.fmrib.ox.ac.uk). We also evaluated the Jaccard coefficient (JC; [9]) of overlap within a skull mask, which was obtained by thresholding the CT image within the head mask at 300 HU to obtain intensities relating to the skull.

The MaxProb method [16] was implemented on a Linux machine at the KCL & GSTT PET Centre to produce linear attenuation maps (µ-maps) for all 110 individuals. For the comparison with µ-maps produced by the MaxProb method, we first converted the reference CTs and pseudo-CTs produced by our method to 511 keV µ-map according to the bilinear equation below [3]:
Below Break Point: $\mu=9.6\times 10^{-5}(HU+1000)$ cm^-1,
Above Break Point: $\mu=a(HU+1000)+b$ cm^-1,
with the Break Point at 1030 HU, a = 5.64 × 10^-5 cm^-1 and b = 4.08 × 10^-2 cm^-1, based on a 140 kVp CT tube voltage.

We evaluated the accuracy of the µ-maps obtained from our pseudo-CTs (µ_pCT) to µ-maps from the reference CTs (µ_CT) using the relative MAE (rMAE) metric [1] defined as $rMAE_{pCT}=100\times\Sigma(|\mu_{pCT}-\mu_{CT}|)/\Sigma\mu_{CT}$. We also evaluated rMAE on the µ-maps obtained from the MaxProb method. We compared rMAE from our deep learning method against those obtained from the MaxProb method with a paired t-test.

We combined our deep learning model by redoing the training on the full set of images in the PET Centre dataset, using the same training scheme as in the cross-validation. We first implemented our fully trained network on the clinical paediatric dataset without refinement. Next, we investigated whether we could improve the performance of our network in paediatric data through transfer learning. Using the fully trained network as a starting point, we further trained the network on the clinical paediatric dataset only using the same training scheme as described above. We performed another 4-fold cross-validation in the clinical paediatric dataset and evaluated the resulting pseudo-CT against reference CT in each fold as above.

Figure 2 shows the reference CT and pseudo-CT produced by our deep learning method for a representative individual, together with the difference image (reference CT–pseudo-CT) and T1-weighted MR used to generate the pseudo-CT. Qualitatively, our pseudo-CT looks similar to the ground truth CT and our method was able to faithfully reproduce fine bone structures, for example in the tissue and air-space near the frontal sinuses (red arrow in Figure 2). The area with the largest error coincided with signal dropout on the MR image due to dental fillings (circled in Figure 2). On average, our method produced pseudo-CT with MAE of 100.4 ± 21.3 HU compared to reference CT within the head mask. The Jaccard coefficient (JC) for the amount of overlap between the skull masks from our pseudo-CT and the reference CT was 0.73 ± 0.07. Table 2 shows the results of each fold of the cross-validation.

Figure 3 shows the µ-maps derived from reference CT, the MaxProb method and our pseudo-CT along with difference images, for the same individual as in Figure 2. Our method was able to learn to generate some intensities in the area where there was signal dropout on the MR image due to dental fillings, whereas the MaxProb method was not able to. The fine bone structures in the air space near the frontal sinuses were not reproduced with the MaxProb method. Compared with reference CT µ-maps, our network produced µ-maps with smaller errors than the MaxProb method: mean rMAE was 8.4 ± 2.1% for our pseudo-CT µ-maps and 13.1 ± 1.5% for the MaxProb µ-maps.

Figure 4 shows the T1-weighted MR, corresponding reference CT, pseudo-CTs generated using the network as trained on adult data (i.e. without refinement), and pseudo-CTs generated using the network refined on paediatric data, together with difference images (reference CT–pseudo-CT) for three randomly selected individuals aged 3, 6 and 9 years. Qualitative inspection shows that the pseudo-CT (without refinement) generated in the 3-year-old shows the highest errors due to thinner skull structures not being reproduced. This was slightly better in the 6- and 9-year-old patients. Pseudo-CTs generated using the refined network were qualitatively better in all three cases. MAE improved from 174.7 ± 33.6 HU when using the existing network to 127.3 ± 39.9 HU after transfer learning in the paediatric dataset.