Mixed Supervision of Histopathology Improves Prostate Cancer Classification from MRI

Thus our contributions are as follows:

• •

  We introduce 3D UCNet, which features regional and global prediction modules that enable gland-level classification of cancer from a semantic segmentation backbone.

• •

  We introduce a histogram-based representation that enables a new type of indirect (weak) supervision consistent with how biopsy cores are annotated by pathologists.

• •

  We demonstrate that mixed supervision techniques expand the type of histopathology ground truth used in training and improve CNN-based detection of clinically significant prostate cancer from prostate MRI.

• •

  We open source our models, algorithms, and analysis: https://gitlab.com/abhe/prostate-mpmri

Multi-parametric prostate MRI (mp-MRI) is composed of T2-weighted imaging (T2WI), diffusion-weighted imaging (DWI) with an associated apparent diffusion coefficient (ADC) map, and dynamic contrast-enhanced (DCE) MRI. DWI, which typically includes both low and high b-value images, and the associated ADC map represent the restriction of water movement in tissue, which is thought to complement the anatomic T2WI series for detecting prostate cancer [13]. DCE provides data on the contrast enhancement kinetics of the imaged soft tissues and may help to delineate prostate tumors. All together mp-MRI provides a rich set of information, although variations due to differences in choice of MR pulse sequence, scanner hardware, and individual patient are commonplace.

Herein, we used bi-parametric (bp-MRI) consisting of just the T2WI and DWI data [14, 15]. The T2WI, high B-value DWI and ADC map were registered and resampled uniformly across the dataset to the same spatial resolution of [1.0, 1.0, 2.24]mm for x, y, and z axis respectively and normalized. For analysis, these imaging volumes are cropped with centering of the prostate gland and final pixel dimensions of [64, 64, 32].

Prostate MRI exams are typically performed after the prostate-specific antigen (PSA) is found to be elevated along with the clinical assessment of a urologist. For our dataset, fellowship-trained abdominal imaging radiologists at UCSF provided routine clinical interpretation of prostate MRI exams with identification of MRI targets, each with an associated PI-RADS scores. These MRI targets were subsequently underwent MRI/ultrasound fusion guided prostate biopsy by a urologist, whereby an axial T2WI is fused by software with live ultrasound to sample MRI targets under ultrasound guidance. In addition to these targeted biopsies, urologists routinely performed systematic biopsies, sampling the prostate in 6 sextants (left apex, left mid gland, left base, right apex, right mid gland, right base). As the exact coordinates of these systematic biopsies are not recorded, these biopsies are only associated with the sextant regions of the prostate gland. Thus, for our dataset, we assumed the location of the systematic biopsies corresponded to a region defined by the geometric division of the prostate into sextants in the registered MR-image space. For the targeted biopsies, we assumed the location of the targeted biopsy corresponded to a radiologist annotated bounding box around each MRI-identified lesion on each applicable slice 1. The ground truth histopathology for each biopsy sample was determined by a genitourinary pathologist with more than 20 years of experience who graded all specimens using Gleason score [16]. Gleason scores were converted to ISUP grade group (1-5). We used 0 to represent benign pathology results. ISUP grade group $\geq 2$ indicates clinically significant disease.

As T2 and DWI have arbitrary non-quantitative image amplitudes, we applied interquartile range (IQR)-based intra-image normalization to control to the MR image intensity values across research sites and eliminate outlying values created by imaging artifacts. Specifically, each image was normalized to the image-level IQR computed within the 3D prostate gland (annotated by a radiologist or defined by a previously developed neural network segmentation model) according to [17]:

We subsequently applied Z-score image normalization to each exam to overcome the problem of high variability of intensity distribution between different patients by transforming the intensities to have zero mean and unit variance.

Table 1 summarizes the distribution of ISUP grade groups in the training, validation, and testing cohorts used in this study. Clinical exam data from 2015-2018 was collected at the University of California, San Francisco (UCSF) under IRB approval. As evident, the distribution of ISUP grade group is imbalanced in our dataset.

All 973 patient exams used in this study included systematic biopsy results from all 6 sextants and at least one targeted-biopsy result. This results in 7278 biopsy results that were used in training, validation or testing. Each exam also includes, for each targeted biopsy, a PI-RADS score assigned by a board-certified radiologist when the case was interpreted clinically.

We used strong supervision for lesion segmentation and lesion grading, when we had relatively localized information.

We used a combination of dice and weighted binary cross-entropy losses to supervise the radiological lesion segmentation output, $\hat{y}_{\text{seg}}$, given the ground truth segmentations $y_{\text{seg}}$, since this has been shown to achieve higher accuracy [19]:

where $\epsilon$ is chosen arbitrarily small to prevent overflow, BCE represents the binary cross-entropy function, $n_{z}$ represents the number of voxels in class $z\in\{0,1\}$ and $|z|$ is the number of classes. Here, $y_{\text{seg}}$ is determined from as the voxel-wise max of the region masks $y\in\mathbb{R}^{R\times X\times Y\times Z}$, over regions where the supervision signal in the first column of the histopathology matrix $z\in\mathbb{R}^{R\times 2}$ is 1 to represent MRI-annotated lesions.

Similarly, we applied categorical binary cross entropy loss to each voxel of the predicted grade membership maps, $\hat{y}_{\text{gg}}\in\mathbb{R}^{K\times X\times Y\times Z}$, given one-hot encoded ground truth histopathology data $y_{\text{gg}}\in\mathbb{R}^{R\times K}$ from regions $R^{+}$ where $z$ holds a supervision signal of 1 and a regression grade group that is not NaN. This is an important subtlety that allows supervision of segmentation outputs from patient exams even when the pathological grade group is not known for individual lesions. This strong voxel-wise grading or classification loss is then defined as:

where $\mathbb{1}_{r}$ represents an indicator (Kronecker delta) function selecting the voxels corresponding spatially to region $r$.

Unfortunately, strong voxel-wise supervision signals are not available for all histopathology data types, such as biopsy data. This is because the biopsy core only targets a small portion of the prostate gland, either a lesion or a normal-appearing tissue. For targeted lesion biopsies, we assume that the biopsy sample is representative of a homogeneous cancer profile within a tumor that is reasonably localized in the MRI coordinate system, enabling the use of the aforementioned strong classification objective, $\mathcal{L}_{\text{strong}}$. Systematic biopsies, however, are neither localized nor representative of the cancer profile in heterogeneous tissue throughout the gland, preventing the use of such strong voxel-wise objectives.

To this end, we utilized two weak supervision objectives in regions $R^{*}$ where the supervision signal in $z$ is $\geq 1$:

The overall multi-task objective is:

where we empirically choose $\bar{\lambda}=[1.0,0.5,1.0,1.0]$, and $\bar{\alpha}$ is chosen based on the availability of different annotation ground truth available for each exam in a minibatch, as well as the type of supervision signals desired in the experiment. Herein, we choose $\bar{\alpha}=[1.0,1.0,1.0,1.0]$ to use all described objectives whenever possible. For training, we used a batch size of 6 MRI exams and the AdamW optimizer with initial learning rate of $0.0001$, determined empirically by repeatedly sweeping the initial learning rate.

Inference is performed in a similar fashion to training, except with all augmentation turned off. Note that, due to the nature of weak histopathology ground truth, it is not feasible to train UCNet using volumetric patches. Instead, we feed full (registered and resampled) multi-contrast prostate MRI exams as input to UCNet, producing a set of predictions corresponding to all volumetric regions of the prostate gland.

$\alpha_{4}=1$, $\alpha_{1,2,3}=0$ represents a baseline lesion segmentation experiment, where the combined class-balanced BCE-loss and dice loss (Eq. 2) is the sole loss term in the objective. The network model employed is simply a 3D residual UNet, representing the backbone of the UCNet architecture.

$\alpha_{3,4}=1$, $\alpha_{1,2}=0$ represents a baseline lesion segmentation and classification experiment, utilizing the same 3D residual UNet backbone of UCNet. The loss terms in the training objective are the segmentation loss (Eq. 2) as well as the strong voxel-wise cross entropy objective (Eq. 5). Here, classification decisions are based on the most frequent or mean value in each region based on thresholded voxels of $\hat{y}_{\text{GGmap}}$. Note, for binary classification the mean and mode inference methods are equivalent. This experiment serves as an important baseline for subsequent methods that utilize the same UNet backbone, but with additional supervision signals.

$\alpha_{2,3,4}=1$, $\alpha_{1}=0$ represents a mixed-supervision objective, novel for its use of distribution-based representations of histopathology. In this experiment, the prediction $\hat{y}_{\text{GGmap}}$ is penalized in regions where the maximum ISUP grade is higher than the ground truth (Eq. 7), e.g. when only the maximum is known for biopsy cores in sextants that represent heterogeneous tissue (i.e. a mixture of background prostate tissue with clinically significant prostate cancer if it were present). When the biopsy is derived from apparent homogeneous tissue, such as MRI-identified lesions, voxels are penalized in histogram bins both higher and lower (Eq. 6) than the ground truth maximum. Specifically, the addition of Equations 6-7 encourages regression by distribution rather than regression by value, representing the first (to our knowledge) deep learning method to incorporate the distribution of cancer histopathology. Note that the number of free model parameters is equal in Experiment 0001, 0011, and 0111. In this setting, and to maintain a clear comparison to the strongly-supervised case, classification is made in the same fashion as in Experiment 0011 by computing the mean or mode value in each region based on thresholded voxels of $\hat{y}_{\text{GGmap}}$.

$\alpha_{1,2,3,4}=1$ represents another mixed-supervision objective that combines the histogram-based penalization of Experiment 0111 together with traditional weak instance-wise or region-wise classification supervision (Eq. 8). A region-wise classification is produced by passing the histogram representation of each region to a small neural network, RegionNet. To drive the clearest interpretation and comparison to Experiment 0111, we choose RegionNet as a dense ReLU-activated 1-layer neural network of input and output dimension $K$ and frozen with weight matrix $I_{K}$, so the $K$ free bias terms represent “optimal” thresholds for declaring clinically-significant cancer in each region. Classification in this setting is achieved by finding the most-significant bit (MSB) of the $K$-bit output $\lceil{\hat{y}_{\text{gg,r}}}\rceil\in\mathbb{Z}^{K}$ for each region $r$, which represents the highest level of cancer.

This experiment aims to show that accounting for tissue heterogeneity can dramatically improve classification accuracy of deep learning cancer classification models. Specifically, the combination of distribution-based penalization (Experiment 0111), together with optimal thresholding and MSB-based classification is designed to enable detection of cancer in heterogeneous tissue where the highest level of cancer is not the most prevalent (e.g. mean or mode).

For each experiment, UCNet was trained twice: (1) by performing an initial learning rate sweep and selecting an optimal point based on decrease in the objective, and (2) by fixing the initial learning rate to 0.0001. Training was performed for 500 epochs, and the checkpoint with the maximum lesion-wise validation accuracy was chosen. In all cases, the fixed learning rate run achieved superior validation performance, and these models were thus chosen for test-set evaluation and reporting.

Table 2 displays the withheld test-set performance of the experiments described in Section 4 with respect to various metrics of success. Experiment 1111, which represents a mixed-supervision objective combining all the aforementioned strong and weak types of supervision, performs the best with respect to lesion classification accuracy as well as gland-wise classification accuracy (based on lesions) compared to the other DNN models, followed by the model trained in Experiment 0111, followed finally by the model in Experiment 0011. This provides strong evidence that weak supervision methods are complementary to strong supervision methods.

Moreover, the mixed supervision strategies in Experiment 1111 achieve 70.3% overall (class-balanced) accuracy in classification of clinically significant cancer in lesions, which exceeds the 66.7% performance of the best PI-RADS cutoff (PI-RADS $\geq$5) classification assigned by radiologists for this dataset. We believe this is achieved not only by learning from hundreds of datasets, but by incorporation of non-salient cancer histopathology acquired through systematic biopsy.

Table 2 indicates that although Experiment 1111 performs the best lesion classification, Experiment 0001 achieves the best overlap with radiologists’ segmentation of lesions, indicating a trade-off between these two metrics for our dataset. However, high accuracy of the lesion segmentation is not a major issue in prostate MRI interpretation, as these are relatively readily visualized by radiologists, but the classification of CS-PCa is the more challenging task. Yet, the radiologist derived lesion segmentation information is still valuable for improving classification performance. For example, Experiment 1110, which includes all mixed-supervision objectives but not lesion segmentation objectives, does not perform as well as Experiment 1111 with respect to lesion classification accuracy (66.2% vs 70.3%). This suggests the lesion segmentation provides some extra attention to suspicious areas during training, improving convergence to desirable minima. We interpret this lesion segmentation versus classification performance tradeoff in the following way: while lesion segmentation maps and corresponding IoU metrics generally correlate with model performance, they do not dictate the lesion classification performance.

A visual comparison of the experiments described in Section 4 is shown in Figures 3-5, including the pre-processed input images and the various outputs which include a lesion risk map, predicted spatial grading map, and results of the regional classification.

In Figure 3, an example is shown where a PI-RADS 4 lesion is identified by the model that is T2 hypointense, markedly hypointense on the ADC map and demonstrating diffusion restriction (high signal intensity) in the right mid posterolateral peripheral zone. This lesion was correctly predicted as clinically significant prostate cancer in all experiments, which corresponds to a biopsy result of ISUP grade group 3.

In Figure 5, an example is shown where a PI-RADS-4 lesion is identified in the right lateral peripheral zone at the apex that is T2 hypointense, hypointense on ADC, and demonstrating restricted diffusion (high signal intensity). This lesion is predicted as not being CS-PCa on all experiments, which corresponds to a biopsy result of benign.

In Figure 5, an example is shown where there is a PI-RADS 5 lesion in the left posterolateral peripheral zone that is subtle to detect on T2 weighted imaging, hypointense on the ADC map and demonstrating restricted diffusion (high signal intensity). In experiments 0011 and 0111, the lesion is accurately segmented but not predicted as clinically significant prostate cancer. With the addition of weak supervision methods in experiment 1111, the lesion is predicted to be CS-PCa, corresponding to the biopsy result of an ISUP grade group 2. Thus, this represents a false negative case that was converted to a true positive case with the addition of weak supervision methods.