Deep networks to automatically detect
late-activating regions of the heart

Abstract

This paper presents a novel method to automatically identify late-activating regions of the left ventricle from cine Displacement Encoding with Stimulated Echo (DENSE) MR images. We develop a deep learning framework that identifies late mechanical activation in heart failure patients by detecting the Time to the Onset of circumferential Shortening (TOS). In particular, we build a cascade network performing end-to-end (i) segmentation of the left ventricle to analyze cardiac function, (ii) prediction of TOS based on spatiotemporal circumferential strains computed from displacement maps, and (iii) 3D visualization of delayed activation maps. Our approach results in dramatic savings of manual labors and computational time over traditional optimization-based algorithms. To evaluate the effectiveness of our method, we run tests on cardiac images and compare with recent related works. Experimental results show that the proposed approach provides fast prediction of TOS with improved accuracy.

1 Introduction

Cardiac resynchronization therapy (CRT) is widely used to treat cardiac conduction system disorders, such as left bundle branch block (LBBB) and intrinsic myocardial diseases [1, 14, 16]. However, standard CRT suffers from a high non-response rate (approximately 40%) [7, 9]. Implanting the CRT left ventricle lead at an area with delayed activation may significantly decrease the non-response rate [4, 18]. Therefore, it is critical to develop a method that accurately measures left ventricular activation time to improve CRT lead site selection and its response rate.

Cine DENSE MR imaging is an accurate and reproducible method for imaging regional myocardial displacement and strain [2, 11]. Studies have shown that late mechanical activations can be effectively measured on circumferential strains of tissue displacements imaged by DENSE [4, 18]. Other examples of strain-based cardiac activation estimation algorithms can be found in [13, 20]. These methods measure the activation at each location of the left ventricle separately, which introduce artifacts of spatial inconsistencies. To alleviate this problem, a recent work [3] developed a semi-automatic algorithm based on active contour models with strain vectors concatenating information of all myocardial segments [3]. However,the estimation of [3] becomes unstable when the signal-to-noise ratio is low. Moreover, an additional requirement of fine-tuning the model parameters case by case greatly hinders its applicability in real clinical settings.

To address the issue above, this paper introduces a deep learning-based method that automatically predicts delayed activation time of ventricular arrhythmias (a.k.a. TOS) without the need of manual inputs and parameter-tuning. More specifically, we develop a cascade deep learning framework that integrates multiple functionalities, including automatic segmentation of left ventricles from cardiac images, TOS prediction by analyzing spatiotemporal strains through convolutional neural networks (CNN) , and 3D visualization of activation maps. In contrast to traditional approaches, our method provides more reliable and stable estimation with significantly reduced computational time and intensive labor. To evaluate the effectiveness of our approach, we run tests on DENSE MR images and compare with the state-of-the-art strain-based activation time estimation algorithm [3].

2 Method

In this section, we present a new framework that automatically detects and reconstructs 3D myocardial activation maps of left ventricles from cine DENSE MRIs. We develop a cascade network where an automatic segmentation of left ventricles is followed by abnormal activation detection of the ventricles in patients with cardiac arrhythmias. Given the input of DENSE images, our model learns a way to reconstruct 3D visualization maps of activation time in an end-to-end fashion. Before introducing our network architecture, we will first review the strain analysis of DENSE imaging [10].

2.1 Circumferential Strain Analysis

Circumferential strain has been shown to represent the myocardial contraction (with negative value) along the circular outline in the short axis [5]. Advantages of analyzing strain values include (i) significantly reducing the computational complexity of our network input from displacements (vector-valued spaces $V$ ) to strain images (real-valued functions $S$ ), (ii) more robustness to body motion artifacts that occurred in original signals, and (iii) its effectiveness of dyssynchrony quantification [5] as well as high reproducibility for validation [21]. Following a recent work of [8], we estimate the left ventricle activation time based on strains computed from the displacement maps. Examples of MR images overlaid with displacement maps over time are shown in Fig. 1(a).

Let $\Omega=\mathbb{R}^{d}/\mathbb{Z}^{d}$ be a $d$ -dimensional torus domain with periodic boundary conditions. Given a $d$ -dimensional displacement map $u(x):\Omega\rightarrow\mathbb{R}^{d}$ , a Jacobian matrix $D^{d\times d}$ of $u$ at each spatial location is

D=\begin{pmatrix}\Delta u_{1}^{1}&\cdots&\Delta u_{1}^{d}\\ \vdots&\ddots&\vdots\\ \Delta u_{d}^{1}&\cdots&\Delta u_{d}^{d}\end{pmatrix},

where $\Delta u_{i}^{j}=\frac{\mathbf{d}u_{i}}{\mathbf{d}x_{j}}$ , with $i,j\in\{1,\cdots,d\}$ . A strain tensor $E$ is then computed as $E=\frac{1}{2}(D^{T}D-I)$ , where $T$ denotes a matrix transpose and $I$ is a $d\times d$ identity matrix. This paper focuses on the circumferential strain, which is the component of strain along the myocardium.

In practical clinical applications, the muscle area of the left ventricle is often divided into a variable number of segments [6, 17]. In this paper, we divide each of the basal, mid-plane, and apical slices of short-axis view into $18$ segments (as shown in Fig. 1(b)). The starting location of the segments is at the upper intersection of the left and right ventricles (labeled by a blue circle in Fig. 1(a) and Fig. 1(b)). Each segment covers the same angle with respect to the center of left ventricles. A $18$ -dimensional strain vector is determined from the full-resolution strain of left ventricle, with strain value extracted at the center of each segment for each slice. To facilitate the automatic estimation of TOS, we arrange the temporal strain data into a 2D matrix consisting of all strain vectors over a time-sequence of cardiac phases. Example of spatiotemporal strain matrix overlaid with circumferential TOS curve is shown in Fig. 1(c). Note that we set the starting segment at the upper intersection of the left and right ventricles across all experiments.

2.2 Network Architecture

Now we are ready to introduce the network architecture (as illustrated in Fig. 2. There are three major components in our model, including an automatic segmentation network providing segmentation maps for left ventricles from cine DENSE MRIs, a CNN-based network predicting TOS curves, and 3D reconstruction and visualization of delayed activation maps. Details are elaborated in the following sections.

Segmentation Network. We employ a 2D segmentation neural network proposed in [10] for left ventricle segmentation. The 2D U-Net networks utilize the structure presented by Ronneberger et al. [19] with modifications for better performance. More specifically, we use dilated convolutional layers in the contracting path of standard U-Net, which increases the receptive field size without increasing the number of parameters and shows improved performance in our experiments. Our loss function is the summation of the weighted pixel-wise cross entropy and soft dice loss. During the training process, data augmentation on-the-fly is performed by applying random translations, rotations and scalings followed by B-spline-based transformations on input images and their corresponding ground-truth label maps at each iteration. To improve the accuracy and smoothness of the segmented contours, each testing image is rotated $9$ times with an interval of $40$ degrees. The corresponding output probability maps are then rotated back and averaged.

CNN for activation time estimation. We develop a CNN-based neural network to predict the TOS curves for late activation time detection. Our network directly learns the relationship between time-series circumferential strain matrices of left ventricles and TOS curves via regression. In all experiments, we use standard convolutional layers with typical activation function ReLU and mean square error (MSE) as a loss. Since the activation for each segment happens no earlier than the time of an initial scan $t_{0}$ , we employ a shifted version of the leaky ReLU [15] to enforce such constraints in the last layer of network architecture:

\displaystyle\sigma(x)=\left\{\begin{matrix}x,&\text{ if \;}x\geq t_{0}\\ -\alpha x+(1+\alpha)t_{0},&\text{otherwise}\end{matrix}\right.

(1)

Here, the input variable $x$ is an output of the fully connected layer in our network, and $\alpha$ is a positive constant (i.e., $\alpha=0.01$ is used in our experiments).

Similar to the segmentation network, we augment the training data for a better network performance. However, many commonly used augmentation methods (e.g. rotation, cropping, and flipping) are inapplicable in this task since they may introduce anatomically meaningless images. Therefore, we carefully apply cyclic translations along the segment axis.

3D Visualization of activation Map. After predicting the TOS curve from our network on all 2D slices from the same patient, we reconstruct a 3D activation map by simply interpolating TOS values on 3D surfaces of left ventricles reconstructed from the segmentation network.

3 Experiments

The network training was performed on an Nvidia 2080Ti GPU with $11$ GB RAM over $1000$ epochs using an Adam optimizer [12]. We set the learning rate as $1E-4$ and the mini batch size as $300$ .

Data. Data were acquired using a 1.5T MR scanner (Avanto, Siemens, Erlangen, Germany) with a four-channel phased-array radiofrequency coil. Cine DENSE was performed in 4 short-axis planes at basal, two mid-ventricular, and apical levels. Cine DENSE parameters included a temporal resolution of 17 ms, pixel size of $2.65\times 2.65\text{ mm}^{2}$ and slice thickness = $8$ mm. Displacement was encoded in two orthogonal directions and a spiral k-space trajectory was used with 6 interleaves per image. Other parameters included: field of view $=240\times 240\text{ mm}^{2}$ , displacement encoding frequency $k_{e}=0.1\text{ cycles/mm}$ , flip angle $15^{\circ}$ and echo time $=1.08\text{ ms}$ .

Experiments. We train the segmentation network on $12,415$ 2D images from $64$ patients, with $20\%$ of which are used for model validation. We then test on $5,255$ images from $44$ mixed subjects of patients and healthy volunteers. Four symmetric encoding and decoding blocks are used in the contracting and expanding path of the network, respectively. Each encoding block of the contracting path includes two consecutive sets of dilated convolutional layers with filter size $3\times 3$ and dilation rate as $2$ , a batch normalization layer, and a rectified linear activation layer. Between each encoding block, pooling layers with step size of $3\times 3$ and stride as $2$ are applied to reduce the spatial dimension in all directions. Each decoding block contains two consecutive sets of deconvolutional layers with filter size $3\times 3$ , a batch normalization layer, and a rectified linear activation layer.

To evaluate our network performance on the TOS curve prediction, we run experiments on $96$ cine DENSE MRIs collected from $15$ patients in total. A number of $1512$ MRIs after data augmentation from $12$ patients are used for training, and the rest for testing. Due to the fact that different data may contain different number of time frames, we zero-pad missing values to unify the dimension of strain matrix over time. We compare the predicted activation time of our method with both ground truth (manually delineated by experts) and the state-of-the-art algorithm [3].

Results. Compared to manually labeled segmentation map, our network results in dice coefficient of left ventricles as $0.87\pm 0.04$ , a Hausdorff distance of $2.7\pm 1$ pixel (equivalent to $5.94\pm 2.2mm$ ), and a mean surface distance of $0.41\pm 0.29$ pixels ( $0.9\pm 0.6mm$ ). The computation time for determining the epicardial and endocardial contours for a single DENSE image is $0.16\pm 0.02s$ , $0.15\pm 0.01s$ , respectively. Fig. 3 shows an example of estimated segmentation of left ventricle by our network vs. manually labeled ground truth.

Fig. 5 displays examples of predicted TOS curves of test data. It shows that our approach outperforms the traditional algorithms based on active contours [3]. Aside from eliminating the need of parameter-tuning, our method runs an order of magnitude faster than the baseline algorithm when estimating the TOS ( $0.001s$ vs. $0.676s$ per image).

Examples of reconstructed 3D activation maps from two patients are shown on the top panel of Fig. 6. For visualization purpose, we employ the $17$ -segment American Heart Association cardiac model, which has been shown to provide the best agreement with the available anatomical data [6]. The Basal and mid-plane slices from the short-axis view are divided into six segments, apical into four segments, and apex into one segment (see the Bulls-eye plots on the bottom panel of Fig. 6). Fig. 6(a) indicates that the most severe late activation (colored in dark red) locates at basel inferolateral section but mid inferior section in Fig. 6(b).

4 Conclusion

In this paper, we present a cascade network that automatically predicts delayed activation time of left ventricles from cardiac images. We develop a workflow that effectively integrates automatic segmentation of left ventricles, TOS prediction based on time-series circumferential strain analysis, and 3D visualization of cardiac activation maps. To the best of our knowledge, this is the first time that an end-to-end network is developed for fully automatic activation time detection of heart from cine DENSE MRIs. The ability of our work predicting better TOS to locate regions of severely delayed activation has great potential to improve the CRT response rate. While our experimental results are demonstrated on cine DENSE MRIs, our model is applicable to many other image modalities, such as ultrasound or spatial modulation of magnetization (SPAMM).

References

[1] W. T. Abraham, W. G. Fisher, A. L. Smith, D. B. Delurgio, A. R. Leon, E. Loh, D. Z. Kocovic, M. Packer, A. L. Clavell, D. L. Hayes, et al. Cardiac resynchronization in chronic heart failure. New England Journal of Medicine, 346(24):1845–1853, 2002.
[2] A. H. Aletras, S. Ding, R. S. Balaban, and H. Wen. Dense: displacement encoding with stimulated echoes in cardiac functional mri. Journal of magnetic resonance (San Diego, Calif.: 1997), 137(1):247, 1999.
[3] D. A. Auger, K. C. Bilchick, J. A. Gonzalez, S. X. Cui, J. W. Holmes, C. M. Kramer, M. Salerno, and F. H. Epstein. Imaging left-ventricular mechanical activation in heart failure patients using cine dense mri: Validation and implications for cardiac resynchronization therapy. Journal of Magnetic Resonance Imaging, 46(3):887–896, 2017.
[4] K. C. Bilchick, S. Kuruvilla, Y. S. Hamirani, R. Ramachandran, S. A. Clarke, K. M. Parker, G. J. Stukenborg, P. Mason, J. D. Ferguson, J. R. Moorman, et al. Impact of mechanical activation, scar, and electrical timing on cardiac resynchronization therapy response and clinical outcomes. Journal of the American College of Cardiology, 63(16):1657–1666, 2014.
[5] L. P. Budge, A. S. Helms, M. Salerno, C. M. Kramer, F. H. Epstein, and K. C. Bilchick. Mr cine dense dyssynchrony parameters for the evaluation of heart failure: comparison with myocardial tissue tagging. JACC: Cardiovascular Imaging, 5(8):789–797, 2012.
[6] M. D. Cerqueira. American heart association writing group on myocardial segmentation and registration for cardiac imaging: Standardized myocardial segmentation and nomenclature for tomographic imaging of the heart: a statement for healthcare professionals from the cardiac imaging committee of the council on clinical cardiology of the american heart association. Circulation, 105:539–542, 2002.
[7] E. S. Chung, A. R. Leon, L. Tavazzi, J.-P. Sun, P. Nihoyannopoulos, J. Merlino, W. T. Abraham, S. Ghio, C. Leclercq, J. J. Bax, et al. Results of the predictors of response to crt (prospect) trial. Echocardiography, 2608:2616, 2008.
[8] M. Dandel, H. Lehmkuhl, C. Knosalla, N. Suramelashvili, and R. Hetzer. Strain and strain rate imaging by echocardiography-basic concepts and clinical applicability. Current cardiology reviews, 5(2):133–148, 2009.
[9] D. V. Exner, A. Auricchio, and J. P. Singh. Contemporary and future trends in cardiac resynchronization therapy to enhance response. Heart rhythm, 9(8):S27–S35, 2012.
[10] S. Ghadimi, X. Feng, C. H. Meyer, and F. H. Epstein. Myocardial segmentation and phase unwrapping for automatic analysis of dense cardiac mri using deep learning. International Society for Magnetic Resonance in Medicine Annual Meeting, 2020.
[11] D. Kim, W. D. Gilson, C. M. Kramer, and F. H. Epstein. Myocardial tissue tracking with two-dimensional cine displacement-encoded mr imaging: development and initial evaluation. Radiology, 230(3):862–871, 2004.
[12] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[13] K. F. Kvåle, J. Bersvendsen, E. W. Remme, S. Salles, J. M. Aalen, P. H. Brekke, T. Edvardsen, and E. Samset. Detection of regional mechanical activation of the left ventricular myocardium using high frame rate ultrasound imaging. IEEE transactions on medical imaging, 38(11):2665–2675, 2019.
[14] J. Lindenfeld, A. M. Feldman, L. Saxon, J. Boehmer, P. Carson, J. K. Ghali, I. Anand, S. Singh, J. S. Steinberg, B. Jaski, T. DeMarco, D. Mann, P. Yong, E. Galle, F. Ecklund, and M. Bristow. Effects of Cardiac Resynchronization Therapy With or Without a Defibrillator on Survival and Hospitalizations in Patients With New York Heart Association Class IV Heart Failure. Circulation, 115(2):204–212, Jan. 2007.
[15] A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3, 2013.
[16] A. J. Moss, W. J. Hall, D. S. Cannom, H. Klein, M. W. Brown, J. P. Daubert, N. M. Estes III, E. Foster, H. Greenberg, S. L. Higgins, et al. Cardiac-resynchronization therapy for the prevention of heart-failure events. New England Journal of Medicine, 361(14):1329–1338, 2009.
[17] A. H. A. W. G. on Myocardial Segmentation, R. for Cardiac Imaging:, M. D. Cerqueira, N. J. Weissman, V. Dilsizian, A. K. Jacobs, S. Kaul, W. K. Laskey, D. J. Pennell, J. A. Rumberger, T. Ryan, et al. Standardized myocardial segmentation and nomenclature for tomographic imaging of the heart: a statement for healthcare professionals from the cardiac imaging committee of the council on clinical cardiology of the american heart association. Circulation, 105(4):539–542, 2002.
[18] R. Ramachandran, X. Chen, C. M. Kramer, F. H. Epstein, and K. C. Bilchick. Singular value decomposition applied to cardiac strain from mr imaging for selection of optimal cardiac resynchronization therapy candidates. Radiology, 275(2):413–420, 2015.
[19] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
[20] B. T. Wyman, W. C. Hunter, F. W. Prinzen, and E. R. McVeigh. Mapping propagation of mechanical activation in the paced heart with mri tagging. American Journal of Physiology-Heart and Circulatory Physiology, 276(3):H881–H891, 1999.
[21] A. A. Young, B. Li, R. S. Kirton, and B. R. Cowan. Generalized spatiotemporal myocardial strain analysis for dense and spamm imaging. Magnetic resonance in medicine, 67(6):1590–1599, 2012.

Deep networks to automatically detect late-activating regions of the heart