Towards A Device-Independent Deep Learning Approach for the Automated Segmentation of Sonographic Fetal Brain Structures: A Multi-Center and Multi-Device Validation

A total of 4370 two-dimensional (2D) standard plane USG images of the TV and TC planes were obtained from 3 centers (2 tertiary referral centers [TRC 1,2] + 1 routine imaging center [RIC]) using 6 USG devices (General Electric [GE] Voluson: P8, P6, E8, E10, S10; Samsung: HERA W10). The images were acquired from a cohort of 1349 pregnant women who received their targeted mid-trimester USG examination (transabdominal). All USG examinations were reviewed and approved by fetal medicine specialists for quality and correctness. Only live singleton fetuses that did not exhibit any growth anomalies were included in this study. Due to the study’s retrospective nature, all informed consent was waived. All data was anonymized as per the tenets of the Declaration of Helsinki after approval from the ethics committee of the respective centers. The USG examination was performed based on internationally recommended practice guidelines [1].

From the 4370 images, a total of 2744 images (TC/TV images from GE E8: 641/1307; TC/TV images from GE S10: 221/575) from TRC 1 were used for training and validation of the DL model. Data balancing (up-sampling) was done to ensure equal representation from each USG device and care was taken to ensure there were no duplicate images, or images from the same subjects between the training and validation datasets (training and validation splits: 89.7%:10.3%).

The remaining 1626 images were used as a part of 4 test sets. Test set 1 (TRC 1; trained device: GE E8 [82.3 %], GE S10 [17.7 %]) consisting of of 719 images (TC/TV: 131/588) was used to test the performance of best DL model. The robustness and generalizability of the DL model was assessed on independent test sets 2,3 and 4. Test set 2 (TRC 1, unseen devices: Samsung HERA W10 [79.2 %], GE P6 [6.8 %], GE E10 [14 %]) consisted of 192 images (TC/TV: 90/102), test set 3 (TRC 2, trained devices: GE E8 [69.5 %], GE S10 [30.5 %]) consisted of 557 images (TC/TV: 223/334); and test set 4 (RIC; unseen device: GE P8) consisted of 158 images (TC/TV: 100/58)

All images were resized to 160 (height) x 288 (width). To preserve the inherent variability (i.e., gain, zoom, contrast, speckle noise, patient-specific probe settings, pixel resolution, etc.) in the dataset, no additional pre-processing was done.

Manual segmentations were prepared for all the 4370 images by well-trained medical image annotators and reviewed by fetal medicine specialists. A total of 10 unique fetal brain structures/classes from the TV (cranium, brain parenchyma, cavum septum pellucidum [CSP], midline falx, choroid plexus, lateral ventricles [LV]) and TC planes (cranium, brain parenchyma, CSP, midline falx, cerebellum, cisterna magna, nuchal fold, skin) were segmented. Note that each class common to both the planes (i.e., cranium, brain parenchyma, midline falx and CSP) were assigned the same unique index while preparing the segmentation masks.

We propose a fully-convolutional and end-to-end DL network inspired (Figure 1)by U-Net [13] to simultaneously segment 10 fetal brain structures across 2 axial views. Inception v4A [14] blocks were used as feature extractors for the 4-levels of the (feature maps: 64, 128, 256, 512) encoder and latent space, while standard blocks (Figure 1) were used in the decoder for feature extraction. Segmentation of fetal brain structures can often be difficult especially in the cases of structural anomalies due to common co-existence of multiple fetal anomalies. Specifically for finer structures like LV, CSP, and nuchal fold where subtle changes in structure are associated with critical anomalies like ventriculomegaly, agenesis of corpus callosum, and Down syndrome. The inceptionv4A block with its multiple kernel sizes can take into account the global features, like structures in vicinity e.g. choroid plexus inside LV and local features, like boundaries and integrity of structures e.g. case of choroid plexus with a cyst inside it. The input image (160 [height] x 288 [width]) was downsampled by a factor of 2 at each level of the encoder (maxpooling, stride = 2), and sequentially upsampled to original scale (bi-linear interpolation) in the decoder. The output from the decoder was passed through a 1x1 convolutional layer (number of feature maps= [11; 10 fetal structures + background], stride = 1) followed by softmax activation to obtain the class-wise probability distributions. To get the final segmentation mask (herein referred to as DL segmentation), each pixel was assigned the class with maximum probability.

The entire network was trained end-to-end with Adam [15] optimizer, Dice loss, fixed learning rate of 0.0001, and a mini batch of 2 images. The model with the best validation loss was finally chosen. The proposed network consisted of 38 million trainable parameters and was trained and tested on an NVIDIA Tesla T4 with CUDA v11.0 and cuDNN v7.6.5 using PyTorch 1.7.0.

Fetal USG images from different USG devices and centers may appear visually different in terms of image quality (e.g., noise, resolution, acoustic shadowing), perception (e.g., brightness, contrast, intensity homogeneity) and presentation (e.g., zoom, orientation; operator dependent factors). To counter these effects, we used extensive domain-specific online data augmentation (DA; implemented using Albumentations [16]; Figure 2 A) to improve the DL model’s performance and generalizability across images from different USG devices/centers.

Image quality and perceptual factors were augmented using color-jitter (brightness: 0.8-1.2 and contrast: 0.8-1.2) - representative of various image enhancing settings, quality reduction (downsampling and upsampling the images; range: 0 to 0.5) - representative of poor quality hardware or image compression loss, speckle noise (Gaussian multiplicative noise; mean: 0, standard deviation: 0.1) - represents artifact introduced by proprietary post-processing, and acoustic shadowing - characteristic artifact formed due to acoustic property of cranium.

In this study, we propose a novel approach to accurately represent fetal acoustic shadows occurring due to attenuation of sound beam at the edges of cranium due to incorrect focal depth. As shown in Figure 2 B, the acoustic shadows are created using the cranium segmentation mask which is used to determine horizontal extremities (red dots) of the cranium. Then 2 patches of width 75-100 pixels are extended from the extreme points at angle 15-20 degrees. These patches are further processed with blur to achieve more realistic murky shadows. Then it is superimposed on original image resulting in image with artificial acoustic cranium shadows as shown in Figure 2 B.

We used affine transformations (zoom [range: 0.6 to 1.2], rotation [range: +20 to -20 degrees], translations [range: 40 to 60 pixels along height or width], shear transformations [range: 0 to 0.2]) and motion blur (kernel size: 50x50 pixels) to simulate operator variability. Finally, variation in fetal anatomy due to anomalies/gestation age were accounted by using elastic deformations (standard deviation: 50, affine: 50).

Qualitatively, the DL segmentations were comparable to the manual segmentations across all test sets for both the TV and TC plane structures. As it is evident from Figure 6, there is substantial difference between DL segmentations with DA and w/o DA. While DL segmentations with DA (3rd column) are comparable to manual segmentations, the DL segmentations w/o DA (4th column) even fail to correctly segment brain parenchyma, which covers more than half of segmentable area inside the brain region (Figure 6: light blue) and thus is essentially unable to predict finer structures such as CSP and LV.

Quantitatively (Table 1; mean of 5-fold cross validation was used), when benchmarked against widely used segmentation models such as U-Net [13], LinkNet [17], MNet [18], Deeplabv3+, [19] and HRNet [20], we obtained a performance improvement (mean DC) of 0.5% to 2% across 4 test sets. Specifically, compared to the benchmarking models, our model localized the the finer structures (LV, CSP, nuchal fold) that are crucial in the screening of CNS anomalies and overall resulted in better performance, i.e. average improvement over benchmarking models for CSP: 2.4 %, LV: 9.8 % and nuchal fold: 8.3 % when averaged across test sets.

The effect of domain-specific DA on the DL segmentations were qualitatively (Figure 6, 3rd and 4th columns) and quantitatively observed (Figure 4). As it is evident from (Figure 6) columns 3rd and 4th, data augmentation can have a profound impact on the segmentation performance especially for USG images from different centers. Across all test sets (especially unseen devices/unseen centers), in general, finer structures and structures of critical importance like LV, CSP and cerebellum were affected the most (Figure 6, 4th column) in the absence of domain-specific DA. Specifically, we observed performance gains (percentage change in mean DC; Figure 4 and Table 1) of 2.1%, 7.11%, 18.4%, and 22.7% across test sets 1, 2, 3, and 4 respectively. Besides, even when the domain-specific DA were used with the benchmarking models, on an average, we observed a minimum of 0.486% and a maximum of 30% performance gain for all cases.

When trained (with domain-specific DA) on increasing fractions of dataset (0.2, 0.4, 0.6, 0.8, 1.0), across all test sets, the DL segmentation performance continued to improve (Figure 5). When repeated the same without domain-specific DA, for a given fraction of training data, the segmentation performance was overall (mean of all fractions; vs. trained with domain-specific DA) reduced by 1%, 4%, 11.4%, and 14.8%, across test sets 1, 2, 3 and 4 respectively.

Further structure-wise analysis of model performance is provided in Figure 7. The results of each structure are averaged across all the 4 test sets and presented in the figure. As it is evident from Table 1 and Figure 7, our proposed approach generally performs better than benchmarking models. While the overall performance of U-Net model is similar (1.4 % lower than our approach when averaged across all test sets), our approach outperforms U-Net specifically for finer structures like CSP, LV and nuchal fold which are crucial for differential diagnosis (DD) of CNS anomalies.

In fetal brain examination certain structures are more important. Changes in structures like deviation from normal CSP ratio and dilated lateral ventricles are small but crucial for the differential diagnosis of several CNS anomalies. Our proposed approach always outperforms the U-Net for these structures in all the test sets. Numerical values for CSP: Test set 1 (Proposed-0.863, U-Net-0.857), Test set 2 (Proposed-0.767, U-Net-0.739), Test set 3 (Proposed-0.836, U-Net-0.790), Test set 4 (Proposed-0.710, U-Net-0.698). Numerical values for LV: Test set 1 (Proposed-0.720, U-Net-0.706), Test set 2 (Proposed-0.612, U-Net-0.604), Test set 3 (Proposed-0.603, U-Net-0.573), Test set 4 (Proposed-0.693, U-Net-0.630).