Interactive Deep Refinement Network for Medical Image Segmentation

Figure2 shows the architecture of the proposed RefineNet. The network consists of two parts. The first part of the RefineNet is the backbone segmentation network which is used to extract initial segmentation. The second part is the refinement network which uses initial segmentation result of the first network and seed point information to generate refinement result.

The main objective of the proposed network is to refine the segmentation results of a test image based on its seed points. Unlike CRF, graph cut, and GANs, the proposed refinement network is connected and trained with the first part of the segmentation network.

Furthermore, we utilize the U-Net as our initial segmentation backbone, as its performance in medical image segmentation is better than that of other deep learning segmentation methods. Additionally, we use the concept of a pyramid network [24] by adding the 1$\times$1 convolution to generate initial segmentation results before the last three deconvolutional layers which the size of initial segmentation results of last three layers are 1$\times$, 0.5$\times$, and 0.25$\times$ of original input image size. Each initial segmentation result concatenated with seed points images and merged up each scale together using concatenation method and deconvolutional (Up-Conv.). So the size of final segmentation result of second network is same as the size of input image.

The output of the second network is the segmentation result that refined using seed points images as refinement guidelines. To use seed points as image guideline in the refinement network, we transform seed point marks using 2D Gaussian. The output of the network is a probability map for a whole image. Each pixel of the probability map contains the probability of foreground and background value.

The seed points in our research are obtained from user input and auto-generated values. Because of the unpredictability of the initial segmentation from the network, manually provided seed points from users cannot be used in each epoch during the training phase. However, they can be used in the testing phase.

Training: To avoid random giving of seed points during the training by the user, which is due to the unknowing of the initial segmentation result from the first network, we provide a solution that does not require seed points during the training phase.

To generate the foreground/background points from the initial segmentation, we use both the ground truth ($f_{y}$) and initial segmentation result ($f_{x}$) generated from the first network. The subtraction of both images generates the subtraction mask ($\delta f$); indicating that each pixel is either $-1$, 0, or $+1$. The $+1$ mask is referred to as the over-segmentation mask, while the $-1$ mask is regarded as the under-segmentation mask.

To generate seed points from those masks, we apply a skeleton to those masks and use the skeleton line as a foreground seed points (under-segmentation mask), and background seed points (over-segmentation mask) as shown in Figure3.

Testing: The main disadvantage of generally interactive segmentation methods are users do not know the region of difficulty, which is mostly under- or over- segmentation until the initial segmentation result is obtained. In our method, the initial segmentation is generated from the first network. With the initial segmentation, the user can observe the segmented, over-segmentation, and under-segmentation region and uses them as input guideline. Unlike seed points in the training phase that auto-generated seed points require ground truth image, manual seed points are inputted from the user.

To develop our network, we use Keras and TensorFlow as our development libraries. The hyper-parameters of the network were tuned using cross-validation with a validation set [29]. The chosen hyper-parameters were those with which the model performed best with the validation set. All computations were performed using an NVIDIA GeForce GTX TITAN X GPU with 12 GB memory.

The leave-one-out method was used in our experiments. We selected one CT volume as a test image, and other CT volumes were used for training. For each method, all data are performed to collect experimental results with a different training set. The overall performance of each method was evaluated using the test set. The overlap index (Dice coefficient), Sensitivity (SEN), and Positive Predictive Value (PPV) were used as evaluation measures.

The proposed method was evaluated using the IRCAD datasets, which are maintained by a French research institute [25].

In this section, we evaluate our network and benchmark on 2D images because most of the existing segmentation methods are done their work on 2D images. We slicing the 3D volume in the horizontal axis to obtain 2D images. With this method, we can obtain approximately 120 slices from each volume. All benchmark methods in this experiment perform segmentation on the 2D slice and form the 2D segmentation result in 3D segmentation volume. The benchmark methods in this experiment are divided into 2 groups. First group is automatic segmentation methods which are Li et al. [27], Cascaded U-Net [28], Cascaded U-Net with 3D CRF[28] FCN[10], and U-Net[26]. Another group is interactive segmentation methods which are graph cut[5], DEXTR[23], and our proposed method.

We retrain all deep learning methods using IRCAD dataset with the leave-one-out technique. For graph cut [5], we use the boundary of dilation method on ground truth as background seed and apply skeletonization on ground truth for foreground seed. For DEXTR [23] network, we use the top, bottom, leftmost, and rightmost points as the guideline points.

The experimental results are shown in Table 1. Graph cut and DEXTR methods use an interactive technique and can obtain high accuracies. However, DEXTR can obtain only 0.907 because the method is prepared for non-medical imaging and met the intensity problem, while the graph cut method is well prepared for medical imaging and obtain 0.936 of dice. In the automatic approach, Li et al., Cascaded U-Net and cascaded U-Net with 3D CRF methods can get very high accuracies which are 0.945, 0.931 and 0.943. Those methods designed for medical liver segmentation with well prepare to preprocess and post-processing, while U-Net and FCN cannot get good results same as other methods which are 0.729 and 0.709.

Based on those results in Table 1, FCN and U-Net are lowest accuracies than other methods. However, both method not include preprocessing and posprocessing in their process which can significantly improve. In this experiment, we applied our refinement method to refine FCN and U-Net results. The results of proposed refinement method on FCN and U-Net are significantly improve which are 0.941 and 0.960.

In order to evaluate proposed network as a post-processing method, we compare proposed network with other post-processing methods which are CRF, GAN, and graph cut on the same segmentation backbone network.

In this section, we evaluate our proposed method and other methods using semi-automatic segmentation. The input of all interactive segmentation methods used in this experiment is the one perfect liver segmentation slice ($Seg_{i}$) that slice located on the middle or largest liver slice in the volume. Where $i$ indicate the slice index in the volume. We use liver segmentation slice ($Seg_{i}$) to generate the seed points for interactive segmentation methods.

In this experiment, we compare our proposed method with graph cut [5], graph cut with shape constraints [30], and DEXTR[23]. For our methods and original graph cut, we use the boundary of dilation method on segmentation slice ($Seg_{i}$) as background seed and apply skeletonization on segmentation slice ($Seg_{i}$) for foreground seed. Then apply those seed point to upper slice ($i-1$) and below slice ($i+1$). This procedure also applies to DEXTR but we extract only top, bottom, leftmost, and rightmost from segmentation slice ($Seg_{i}$) and apply to neighbor slices. On another hand, graph cut with shape constraints method[30] provide their own seed point generator for Semi-automatic 3D segmentation.

Table 2 show the comparison between semi-automatic slice by slice segmentation methods. The result of DEXTR is lowest in the table. The main reason is DEXTR require accurate seed points, while the semi-automatic slice by slice segmentation will auto-generate seed points from segmentation result of the previous slice. So the result of DEXTR is over-segmentation.

On the other methods, graph cut, and graph cut with shape constraints are not required accurate seed point so the result of both methods are better than DEXTR in slice by slice segmentation. Graph cut and graph cut with shape can obtain high results on DICE, SEN, and PPV which are 0.890,0.932, and 0.855 for graph cut and 0.926, 0.908, and 0.937 for graph cut with shape constraints.

Our RefineNet can obtain high result when compared to other methods. Compare to other methods, results of the proposed method in slice by slice in Table 2 and 2D slice segmentation in Table 1 not much difference.

The visualization of the slice by slice segmentation results are shown in Figure 4. Graph cut and graph cut with shape constraints are over-segmentation (blue color) on the slices that have a complex liver shape (slice no. 83 and 92). While graph cut shape have less under segmentation (green) than graph cut method. All methods have a high accurate segmented region (red) but our method has less under- and over-segmentation compare to both graph cut methods.