SHISRCNet: Super-resolution And Classification Network For Low-resolution Breast Cancer Histopathology Image

To better extract and fuse multi-scale features for super-resolution, we propose a new SR network, called SRMFENet. Like SRResNet [11], SRMFENet takes a single low-resolution image as input and uses the pixelshuffle layer to get the restructured image. The difference between SRMFENet and SRResNet is that a Multi-Features Extraction block (MFEblock) is proposed to extract and fuse multi-scale histopathological images’ features. The structure of the MFEblock is shown in Fig. 1(b). The input features $X$ capture multi-scale features $Y_{i}$ through four 3$\times$3 atrous convolutions [4] with different rates:

where $n$ is the number of atrous convolutions and is set to 4 by the experiments. This design not only preserves the depth of the network, but also increases the width of the network. It is beneficial for the network to extract shallow local texture information and global semantic information. After the feature extraction phase, a new fusion method named MSF fuses all of different scale features $Y_{i}$. In the end, the input features $X$ are added with the fused features. The details of MSF show in the Fig. 1(c). Firstly, we conduct Global Average Pooling (GAP) [14] on the multi-scale features to obtain their average channel-wise weights. Then using Sigmoid activation function to map weight to 0 to 1. Next softmax operation normalizes the same position of the obtained multi-scale average channel-wise weights. Finally, the features are multiplied by the corresponding normalized weights and the processed features are added together to generate the new multi-scale features. MFEblock is very applicable to process histopathological images of different magnification factors, as it employs convolution and attention operations to capture local and global image context information and fuse them well.

The task of the CF module is to classify the reconstructed SR images. It can use the method of downsampling to capture multi-scale features. So we combine the multi-scale receptive fields (SKNet as backbone network) with the FPN (a downsampling method) to achieve the feature extraction of this module. In Fig. 1(a), the multi-sacle features extracted by SKNet are the input of FPN. We propose a new fusion method, called cross-scale selective fusion block (CSFblock) to effectively fuse high-resolution and low-resolution features in FPN. After the fused features are processed by GAP, they are aggregated into a new multi-scale feature by concatenate operation. Finally, the aggregated multi-scale features are classified through the fully connected (FC) layer. The structure of CSFblock is shown in Fig. 1(d). The inputs of CSFblock are two-way inputs which are the high-resolution features $X_{h}\in W$×$H$×$C$ and the low-resolution features $X_{l}\in W_{1}$×$H_{1}$×$C$. In CSFblock, the upsampling operations are performed on the low-resolution features $X_{l}$ to realize consistency with $X_{h}$ dimension. $X_{h}$ and restructured $X_{l}$ are fused via an element-wise summation:

Then, using GAP along the channel dimension to get the global information $S$. A FC layer generates a compact feature vector $Z$ which guides the feature selection procedure. And $Z$ is reconstructed into two weight vectors $a$, $b$ of the same dimension as $S$ through two FC layers, which can be defined as:

Where $\delta$ denotes ReLU and $W_{a}$, $W_{b}$, $W_{c}$, means the weight of the FC layers. Specifically, a softmax operator is applied on $a$ and $b$ ’s channel-wise digits:

The fused feature map $F$ is obtained through the attention weights on multi-scale features:

The SR module and the CF module exploit different loss functions for training. In the SR module, $L1$ Loss is used for super-resolution. In the CF module, we introduce HR images to CF module in the training stage for improving the performance of CF module and reducing the error caused by the reconstructed SR images. We use $Focal$ Loss [16] to alleviate the class imbalanced data problem of the HR and SR images’ classification. Inspired by the contrastive learning algorithm SimCLR [5], the HR and SR of the same images are similar to two different views. So the $NT-Xent$ loss [19] is adopted to calculate similarity between SR multi-scale features and HR multi-scale features for CF module’s robustness. The total loss function can be expressed as:

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are the weights of $L1$ Loss, $Focal$ Loss and $NT-Xent$ Loss, respectively. In the inference stage, only SR images are taken as inputs by CF module. In our experiment, $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are set to 0.6, 0.3 and 0.1, respectively. And the temperature parameter in $NT-Xent$ Loss is set to 0.5.

Table 1 shows the results of the super-resolution phase. We adopt Peak Signal to Noise Ratio (PSNR) and structural similarity index (SSIM) [6] to evaluate the performance of the SR model. MRC-Net and our proposed SRMFENet (SR module) achieves better metrics than the other algorithms. This proves the effectiveness of multi-scale features extraction. Compared with MRC-Net, our MFEblocks can extract and fuse multi-scale features well. And the joint training of SRMFENet and CF module improves the performance of super-resolution. Fig. 2 demonstrates that our model can recover more details with less blurring.

We compare our introduced CF module with five state-of-the-art breast cancer histopathological image models and Diagnosis Network with MRC-Net [6], as shown in Table 2. The results illustrate that the CF module reaches the best performance in four different magnification factors. This indicates the effectiveness of our proposed combination of two multi-scale feature extraction methods. SHISRCNet, which uses down sample to half resolution (x2↓) from HR images, outperforms the SSCA at 40x, 200x and 400x. And it gets results close to the CF module at all magnification factors. Meanwhile, compared with the Diagnosis Network that also uses LR images as input, SHISCNet has remarked performance advantages. Table 3 compares our results with the CF module using different resolution images. The performance of the CF module decreases significantly with the reduction of resolution. In contrast, SHISRCNet greatly improves the CF module performance of different scale low-resolution images.

To verify the effectiveness of the proposed components in SHISRCNet, a comparison between SHISRCNet and its five components on x2↓ images is given in Table 3. (1) w/o MSF repalces MSF with concatenate operation and 1$\times$1 convolution. (2) w/o FPN + CSFblock means that only SKNet is used for feature extraction in the CF module. (3) w/o CSFblock, w/o HR images and w/o NT-Xent loss remove the corresponding operation, respectively. As shown in table 3, firstly, the performance of super-resolution in the SHISRCNet is significantly reduced when we remove MSF. It indicates the importance of MSF for multi-scale feature fusion in the SR module. Secondly, only SKNet is used to extract multi-scale features in the CF module, and the accuracy decreased significantly. This again proves the effectiveness of our proposed combination of two multi-scale feature extraction methods. Thirdly, compared with using FPN alone, the performance of SHISRCNet is further improved by adding CSFblock to FPN. Finally, the introduction of HR images further promotes the performance of SHISRCNet. Because the training method of HR and SR images proposed by us helps to improve the generalization of the SHISRCNet.