Recognition of Cardiac MRI Orientation via Deep Neural Networks and a Method to Improve Prediction Accuracy

Due to different data sources and scanning habits, the orientation of different CMR images may be different, and the orientation vector corresponding to the image itself may not correspond correctly. This may cause problems in tasks such as image segmentation or registration. Taking a 2D image as an example, we set the orientation of an image as the initial image and set the four corners of the image to $\boxed{\begin{array}[]{cc}1&2\\ 3&4\\ \end{array}}$ , Then the orientation of the 2D MR image may have the following 8 variations, which is listed in Table 1.

For each image $X_{t}$ from dataset, the target is to find the correct orientation from 8 classes. We denote the correct orientation of image $X_{t}$ as $i_{t}$ and denote the correctly adjusted image $X_{t}$ as $Y_{t}$ If we view each orientation as a function $f$, we can get a function set $\{f_{i},i=0\dots 7\}$ and $(X_{t},Y_{t},i_{t})$ satisfy the equation $f_{i_{t}}(Y_{t})=X_{t}$. In the following, function set $\{f_{i},i=0\dots 7\}$ is referred to as $F$.

Suppose given image $X_{t}$, $X_{t}$ is then normalized. We denote the processed $X_{t}$ as $X_{t}^{\prime}$. CNN take the $(X_{t}^{\prime},i_{t})$ as input. In the proposed framework, the orientation recognition network consists of 3 convolution layers and 2 fully connected layers. The orientation predicted is denoted as $\hat{O_{i}}$. We use the standard categorical loss to calculate the loss between predicted orientation $\hat{O_{i}}$ and orientation label $O_{i}$. The orientation loss is formulated as below:

$$L_{orientation}=\sum_{i=1}^{8}O_{i}log(\hat{O_{i}})$$

(1)

As we can see in 2.1, we regard label $i_{t}$ as function $f_{i_{t}}$. It can be easily proved that $f_{i}\circ f_{j}\in F$ for any $i$ and $j$, so we can not only view $f_{i}$ as a function but an operator whose define domain and value domain are both $F$. For convenience, we denote the operator $f_{i}$ as $g_{i}$. Surpprisingly, we can prove that operator $g_{i}$ is a surjection in $F$ for any $i$, which can be simply explained by the following matrix $A$. In matrix $A$, $A_{ij}=k$ means $g_{i}(f_{j})=f_{k}$.

$$A=\left(\begin{array}[]{cccccccc}0&1&2&3&4&5&6&7\\ 1&0&3&2&5&4&7&6\\ 2&3&0&1&6&7&4&5\\ 3&2&1&0&7&6&5&4\\ 4&6&5&7&0&2&1&3\\ 5&7&4&6&1&3&0&2\\ 6&4&7&5&2&0&3&1\\ 7&5&6&4&3&1&2&0\end{array}\right)$$

Because $F$ is a finate set, $g_{i}$ is a injection and invertible. For any operator $g_{i}$, it exist an inverse operator and we denote it as $g^{-}_{i}$. Operator $g_{i}^{-}$ is also a surjection and injection in $F$, which can be simply explained by the following matrix $A^{-}$. In matrix $A^{-}$, $A^{-}_{ij}=k$ means $g_{i}^{-}(f_{j})=f_{k}$. For simplification, we omit $f$ and use $g_{i}^{-}(j)=k$ to express the results above.

$$A^{-}=\left(\begin{array}[]{cccccccc}0&1&2&3&4&5&6&7\\ 1&0&3&2&5&4&7&6\\ 2&3&0&1&6&7&4&5\\ 3&2&1&0&7&6&5&4\\ 4&6&5&7&0&2&1&3\\ 6&4&7&5&2&0&3&1\\ 5&7&4&6&1&3&0&2\\ 7&5&6&4&3&1&2&0\end{array}\right)$$

Based on the above premise, we built the predicting method by the follow 4 steps. Figure 1 shows the method by a flow chart.

1. 1.

   Apply $(f_{0},f_{1}\dots,f_{7})$ to the image $X_{t}$ to get 8 images. We denote these images as $(X_{t0},X_{t1},\dots,X_{t7})$

2. 2.

   Take these 8 images as input to DNN and get 8 orientations $(i_{t0},i_{t1},\dots,i_{t7})$.

3. 3.

   Apply $(g^{-}_{0},g^{-}_{1},\dots,g^{-}_{7})$ to these 8 labels and get another 8 labels $(g^{-}_{0}(i_{t_{0}}),g^{-}_{1}(i_{t1}),\dots,g^{-}_{7}(i_{t7}))$.

4. 4.

   The labels which occur most in $(g^{-}_{1}(i_{t_{0}}),g^{-}_{2}(i_{t1}),\dots,g^{-}_{7}(i_{t7}))$ is the final result.

We evaluate orientation recognition network on the MyoPS dataset[3, 2]. The MyoPS dataset provides the three-sequence Cardiac Magnetic Resonance (LGE, T2 and C0) and three anatomy masks, including myocardium (Myo), left ventricle (LV), and right ventricle (RV), some of the three-sequence Cardiac Magnetic Resonance is shown as Figure 2 . MyoPS further provides two pathology masks (myocardial infarct and edema) from the 45 patients. For the simplified orientation recognition network, we train model for single modality on the MyoPS dataset, then transfer the model to other modalities. For each sequence, we resample each slice of each 3d image and the corresponding labels to an in-plane resolution of 1.367 × 1.367 mm.

We divide slices into three sub-sets, i.e., the training set, validation set and test set, at the ratio of 50%, 30% and 20%. Three sub-sets don’t have slices from same patient. Then, for each standard 2d image, we apply all function from $F$ to it to expand dataset. For training set, image slices are cropped or padded to $256\times 256$ for the orientation recognition network, and apply random augmentation. For test set and validation set, the images are only resized to $256\times 256$.

During each training iteration, a batch of the three-channel images X’ is fed into the orientation recognition network. Then, the network outputs the predicted orientation network.

Figure 3 and Table 2 shows the training process and accuracy of the three sequences on test set. The results show that the model get quite high accuracy in three modalities. Howeverm the size of test data is small, random factors influence the result heavily. In the following, we redivide the data, retrain the model, and analyses sensitivity of the model.

When using deep learning method to solve the problem, the volume of data is always the most important factor. In medical image processing, it is difficult to get a lot of data, so analysing the sensitivity of model is necessary. We redivided the data set into training set and test set 5 times, and the proportion of training set was 60%, 50%, 40%, 30% and 20% respectively. In each divided data set, we retrain the model in training set and compute accuracy in test set.

The accuracy variation is shown in Table 3. There is not a distinct difference among the accuracy while the volume of data decrease. Therefore DNN is a suitable method in CMR orientation recognition with high accuracy and low sensitivity.

Table 3 shows the difference of accuracy between improved prediction and direct prediction. We can find that improved prediction always have higher accuracy in our experiment. However, it is not inevitable, because the vote may make the original correct decision wrong. Sometimes, the improved prediction have a lower accuracy, but in an average sense, the improved prediction is better than direct prediction.