CardioGAN: Attentive Generative Adversarial Network with Dual Discriminators for Synthesis of ECG from PPG

The idea of synthesizing ECG has been explored in the past, utilizing both model-driven (e.g. signal processing or mathematical modelling) and data-driven (machine learning and deep learning) techniques. As examples of earlier works, (McSharry et al. 2003; Sayadi, Shamsollahi, and Clifford 2010) proposed solutions based on differential equations and Gaussian models for generating ECG segments.

Despite deep learning being employed to process ECG for a wide variety of different applications, for instance biometrics (Zhang, Zhou, and Zeng 2017), arrhythmia detection (Hannun et al. 2019), emotion recognition (Sarkar and Etemad 2020a, b), cognitive load analysis (Sarkar et al. 2019; Ross et al. 2019), and others, very few studies have tackled synthesis of ECG signals with deep neural networks (Zhu et al. 2019a; Golany and Radinsky 2019; Golany et al. 2020). Synthesizing ECG with GANs was first studied in (Zhu et al. 2019a), where a bidirectional LSTM-CNN architecture was proposed to generate ECG from Gaussian noise. The study performed by (Golany and Radinsky 2019), proposed PGAN or Personalized GAN to generate patient-specific synthetic ECG signals from input noise. A special loss function was proposed to mimic the morphology of ECG waveforms, which was a combination of cross-entropy loss and mean squared error between real and fake ECG waveforms.

A few other studies have targeted this area, for example, EmotionalGAN was proposed in (Chen et al. 2019), where synthetic ECG was used to augment the available ECG data in order to improve emotion classification accuracy. The proposed GAN generated the new ECG based on input noise. Lastly, in a similar study performed by (Golany et al. 2020), ECG was generated from input noise to augment the available ECG training set, improving the performance for arrhythmia detection.

With respect to the very specific problem of PPG-to-ECG translation, to the best of our knowledge, only (Zhu et al. 2019b) has been published. This work did not use deep learning, instead used discrete cosine transformation (DCT) technique to map each PPG cycle to its corresponding ECG cycle. First, onsets of the PPG signals were aligned to the R-peaks of the ECG signals, followed by a de-trending operation in order to reduce noise. Next, each cycle of ECG and PPG was segmented, followed by temporal scaling using linear interpolation in order to maintain a fixed segment length. Finally, a linear regression model was trained to learn the relation between DCT coefficients of PPG segments and corresponding ECG segments. In spite of several contributions, this study suffers from few limitations. First, the model failed to produce reliable ECG in a subject-independent manner, which limits its application to only previously seen subject’s data. Second, often the relation between PPG segments and ECG segments are not linear, therefore in several cases, this model failed to capture the non-linear relationships between these $2$ domains. Lastly, no experiments have been performed to indicate any performance enhancement gained from using the generated ECG as opposed to the available PPG (for example a comparison of measured HR).

In order to not be constrained by paired training where both types of data are needed from the same instance in order to train the system, we are interested in an unpaired GAN, i.e. CycleGAN-based architectures. We propose CardioGAN whose main objective is to learn to estimate the mapping between PPG ($P$) and ECG ($E$) domains. In order to force the generator to focus on regions of the data with significant importance, we incorporate an attention mechanism into the generator. We implement generator $G_{E}:P\rightarrow E$ to learn forward mapping, and $G_{P}:E\rightarrow P$ to learn the inverse mapping. We denote generated ECG and generated PPG from CardioGAN as $E^{\prime}$ and $P^{\prime}$ respectively, where $E^{\prime}=G_{E}(P)$ and $P^{\prime}=G_{P}(E)$. According to (Penttilä et al. 2001) and a large number of other studies, cardiac activity is manifested in both time and frequency domains. Therefore, in order to preserve the integrity of the generated ECG in both domains, we propose the use of a dual discriminator strategy, where $D^{t}$ is employed to classify time domain and $D^{f}$ is used to classify the frequency domain response of real and generated data.

Figure 1 shows our proposed architecture, where $G_{E}$ takes $P$ as an input and generates $E^{\prime}$ as the output. Similarly, $E$ is given as an input to $G_{P}$ where $P^{\prime}$ is generated as the output. We employ $D^{t}_{E}$ and $D^{t}_{P}$ to discriminate $E$ versus $E^{\prime}$, and $P$ versus $P^{\prime}$, respectively. Similarly, $D^{f}_{E}$ and $D^{f}_{P}$ are developed to discriminate $f(E)$ versus $f(E^{\prime})$, as well as $f(P)$ versus $f(P^{\prime})$, respectively, where $f$ denotes the spectrogram of the input signal. Finally, $E^{\prime}$ and $P^{\prime}$ are given as inputs to $G_{P}$ and $G_{E}$ respectively, in order to complete the cyclic training process.

In the following subsections, we expand on the dual discriminator, the notion of integrating an attention mechanism into the generator, and the loss functions used to train the overall architecture. The details and architectures of each of the networks used in our proposed solution are provided in Section 4.3.

As mentioned above, to preserve both time and frequency information in the generated ECG, we use a dual discriminator approach. Dual discriminators have been used earlier in (Nguyen et al. 2017), showing improvements in dealing with mode collapse problems. To leverage the concept of dual discriminators, we perform Short-Time Fourier Transformation (STFT) on the ECG/PPG time series data. Let’s denote $x[n]$ as a time-series, then $STFT(x[n])$ can be denoted as $X(m,\omega)=\sum^{\infty}_{n=-\infty}x[n]w[n-m]e^{-j\omega n}$, where $m$ is the step size and $w[n]$ denotes Hann window function. Finally, the spectrogram is obtained by $f(x[n])=log(|X(m,\omega)|+\theta)$, where we use $\theta=1e^{-10}$ to avoid infinite condition. As shown in Figure 1 the time-domain and frequency-domain discriminators operate in parallel, and as we will discuss in Section 3.4, to aggregate the outcomes of these two networks, the loss terms of both of these networks are incorporated into the adversarial loss.

We adopt Attention U-Net as our generator architecture, which has been recently proposed and used for image classification (Oktay et al. 2018; Jetley et al. 2018). We chose attention-based generators to learn to better focus on salient features passing through the skip connections. Let’s assume $x^{l}$ are features obtained from the skip connection originating from layer $l$, and $g$ is the gating vector that determines the region of focus. First, $x^{l}$ and $g$ are mapped to an intermediate-dimensional space $\mathbb{R}^{F_{int}}$ where $F_{int}$ corresponds to the dimensions of the intermediate-dimensional space. Our objective is to determine the scalar attention values ($\alpha_{i}^{l}$) for each temporal unit $x^{l}_{i}\in\mathbb{R}^{F_{l}}$, utilizing gating vector $g_{i}\in\mathbb{R}^{F_{g}}$, where $F_{l}$ and $F_{g}$ are the number of feature maps in $x^{l}$ and $g$ respectively. Linear transformations are performed on $x^{l}$ and $g$ as $\theta_{x}=W_{x}x^{l}_{i}+b_{x}$ and $\theta_{g}=W_{g}g_{i}+b_{g}$ respectively, where $W_{x}\in\mathbb{R}^{F_{l}\times F_{int}}$, $W_{g}\in\mathbb{R}^{F_{g}\times F_{int}}$, and $b_{x}$, $b_{g}$ refer to the bias terms. Next, non-linear activation function ReLu (denoted by $\sigma_{1}$) is applied to obtain the sum feature activation $f=\sigma_{1}(\theta_{x}+\theta_{g})$, where $\sigma_{1}(y)$ is formulated as $\max(0,y)$. Next we perform a linear mapping of $f$ onto the $\mathbb{R}^{F_{int}}$ dimensional space by performing channel-wise $1\times 1$ convolutions, followed by passing through a sigmoid activation function $(\sigma_{2})$ in order to obtain the attention weights in the range of $[0,1]$. The attention map corresponding to $x^{l}$ is obtained by $\alpha_{i}^{l}=\sigma_{2}(\psi*f)$ where $\sigma_{2}(y)$ can be formulated as $\frac{1}{1+exp^{-y}}$, $\psi\in\mathbb{R}^{F_{int}}$ and $*$ denotes convolution. Next, we perform element-wise multiplication between $x^{l}_{i}$ and $\alpha^{l}_{i}$ to obtain the final output from the attention layer.

Our final objective function is a combination of an adversarial loss and a cyclic consistency loss as presented below.

We use $4$ very popular ECG-PPG datasets, namely BIDMC (Pimentel et al. 2016), CAPNO (Karlen et al. 2013), DALIA (Reiss et al. 2019), and WESAD (Schmidt et al. 2018). We combine these $4$ datasets in order to enable a multi-corpus approach leveraging large and diverse distributions of data for different factors such as activity (e.g. working, driving, walking, resting), age (e.g. $29$ children, $96$ adults), and others. The aggregate dataset contains a total of $125$ participants with a balanced male-female ratio.

Since the above-mentioned datasets have been collected at different sampling frequencies, as a first step we re-sampled (using interpolation) both the ECG and PPG signals with a sampling rate of $128$ Hz. As the raw physiological signals contain a varying amounts and types of noise (e.g. power line interference, baseline wandering, motion artefacts), we perform very common filtering techniques on both the ECG and PPG signals. We apply a band-pass FIR filter with a pass-band frequency of $3$ Hz and stop-band frequency of $45$ Hz on the ECG signals. Similarly, a band-pass Butterworth filter with a pass-band frequency of $1$ Hz and a stop-band frequency of $8$ Hz is applied on the PPG signals. Next, person-specific z-score normalization is performed on both ECG and PPG. Then, the normalized ECG and PPG signals are segmented into $4$-second windows ($128$ Hz $\times 4$ seconds $=512$ samples), with a $10\%$ overlap to avoid missing any peaks. Finally, we perform min-max $[-1,1]$ normalization on both ECG and PPG segments to ensure all the input data are in a specific range.

Our proposed CardioGAN network is trained from scratch on an Nvidia Titan RTX GPU, using TensorFlow 2.2. We divide the aggregated dataset into a training set and test set. We randomly select $80\%$ of the users from each dataset (a total of $101$ participants, equivalent to $58$K segments) for training, and the remaining $20\%$ of users from each dataset (a total of $24$ participants, equivalent to $15$K segments) for testing. The training time was approximately $50$ hours. To enable CardioGAN to be trained in an unpaired fashion, we shuffle the ECG and PPG segments from each dataset separately eliminating the couplings between ECG and PPG followed by a shuffling of the order of datasets themselves for ECG and PPG separately. We use a batch size of $128$, unlike the original CycleGAN where a batch size of $1$ is used. We notice performance gain with a larger batch size. Adam optimizer is used to train both the generators and discriminators. We train our model for $15$ epochs, where the learning rate ($1e^{-4}$) is kept constant for the initial $10$ epochs and then linearly decayed to $0$. The values of $\alpha$, $\beta$, and $\lambda$ are empirically set to $3$, $1$ and $30$ respectively.

Heart rate is measured as number of beats per minutes (BPM) by dividing the length of ECG or PPG segments in seconds by the average of the peak intervals multiplied by 60 (seconds). Let’s define the mean absolute error (MAE) metric for the heart rate (in BPM) obtained from a given ECG or PPG signal ($HR^{Q}$) with respect to a ground-truth HR ($HR^{GT}$) as $MAE_{HR}(Q)={\frac{1}{N}}\sum_{i=1}^{N}|HR^{GT}_{i}-HR^{Q}_{i}|$, where $N$ is the number of segments for which the HR measurements have been obtained. In order to investigate the merits of CardioGAN, we measure $MAE_{HR}(E^{\prime})$, where $E^{\prime}$ is the ECG generated by CardioGAN. We compare these MAE values to $MAE_{HR}(P)$ (where $P$ denotes the available input PPG) as reported by other studies on the 4 datasets. The results are presented in Table 1 where we observe that for $3$ of the $4$ datasets, the HR measured from the ECG generated by CardioGAN is more accurate than the HR measured from the input PPG signals. For CAPNO dataset in which our ECG shows higher error compared to other works based on PPG, the difference is quite marginal, especially in comparison to the performance gains achieved across the other datasets.

Different studies in this area have used different window sizes for HR measurement which we report in Table 1. To evaluate the impact of our solution based on different window sizes, we measure $MAE_{HR}(E^{\prime})$ over different $4,8,16,32$, and $64$ second windows and present the results in comparison to $MAE_{HR}(P)$ across all the subjects available in the $4$ datasets in Table 2. In these experiments, we utilize two popular algorithms for detecting peaks from ECG (Hamilton 2002) and PPG (Elgendi et al. 2013) signals. We observe a clear advantage in measuring HR from $E^{\prime}$ as opposed to $P$. We notice a very consistent performance gain across different window sizes, which further demonstrates the stability of the results produce by CardioGAN.

In Figure 2 we present a number of samples of ECG signals generated by CardioGAN, clearly showing that our proposed network is able to learn to reconstruct the shape of the original ECG signals from corresponding PPG inputs. Careful observation shows that in some cases, the generated ECG signals exhibit a small time lag with respect to the original ECG signals. The root cause of this time delay is the Pulse Arrival Time (PAT), which is defined as the time taken by the PPG pulse to travel from the heart to a distal site (from where PPG is collected, for example, wrist, fingertip, ear, or others) (Elgendi et al. 2019). Nonetheless, this time-lag is consistent for all the beats across a single generated ECG signal as a simple offset, and therefore does not impact HR measurements or other cardiovascular-related metrics. This is further evidenced by the accurate HR measurements presented earlier in Tables 1 and 2.

The proposed CardioGAN consists of attention-based generators and dual discriminators, as discussed earlier. In order to investigate the usefulness of the attention mechanisms and dual discriminators, we perform an ablation study of $2$ variations of the network by removing each of these components individually. To evaluate these components, we perform the same $MAE_{HR}$ along with a number of other metrics to quantify the quality of ECG waveforms. We use metrics similar to those used in (Zhu et al. 2019a), which are Root Mean Squared Error (RMSE), Percentage Root Mean Squared Difference (PRD), and Fréchet Distance (FD). We briefly defined these metrics as follows:

RMSE: In order to understand the stability between $E$ and $E^{\prime}$, we calculate $RMSE=\sqrt{{\frac{1}{N}}\sum_{i=1}^{N}(E_{i}-E^{\prime}_{i})^{2}}$ where $E_{i}$ and $E^{\prime}_{i}$ refer to the $i^{th}$ point of $E$ and $E^{\prime}$ respectively.

PRD: To quantify the distortion between $E$ and $E^{\prime}$, we calculate $PRD=\sqrt{\frac{\sum_{i=1}^{N}(E_{i}-E^{\prime}_{i})^{2}}{\sum_{i=1}^{N}(E_{i})^{2}}\times 100}$.

FD: Fréchet distance (Alt and Godau 1995) is calculated to measure the similarity between the $E$ and $E^{\prime}$. While calculating the distance between two curves, this distance considers the location and order of the data points, hence, giving a more accurate measure of similarity between two time-series signals. Let’s assume $E$, a discrete signal, can be expressed as a sequence of $\{e_{1},e_{2},e_{3},\dots,e_{N}\}$, and similarly $E^{\prime}$ can be expressed as $\{e^{\prime}_{1},e^{\prime}_{2},e^{\prime}_{3},\dots,e^{\prime}_{N}\}$. We can create a $2$-D matrix $M$ of corresponding data points by preserving the order of sequence $E$ and $E^{\prime}$, where $M\subseteq\{(e,e^{\prime})|e\in E,e^{\prime}\in E^{\prime}\}$. The discrete Fréchet distance of $E$ and $E^{\prime}$ is calculated as $FD=\min_{M}\max_{(e,e^{\prime})\in M}d(e,e^{\prime})$, where $d(e,e^{\prime})$ denotes the Euclidean distance between corresponding samples of $e$ and $e^{\prime}$.

The results of our ablation study are presented in Table 3. We present the performance of different variants of CardioGAN for all the subjects across all $4$ datasets. CardioGAN w/o DD is the variant with only the time domain discriminator and no change in the generator architecture. CardioGAN w/o attn is the variant where the generator does not contain an attention mechanism. The results presented in the table evidently show the benefit of using the proposed CardioGAN over it’s ablation variants.

Apart from the interest to the AI community, we believe our proposed solution has the potential to make a larger impact in the healthcare and wearable domains, notably for continuous health monitoring. Monitoring cardiac activity is an essential part of continuous health monitoring systems, which could enable early diagnosis of cardiovascular diseases, and in turn, early preventative measures that can lead to overcoming severe cardiac problems. Nonetheless, as discussed earlier, there are no suitable solutions for every-day continuous ECG monitoring. In this study we bridge this gap by utilizing PPG signals (which can be easily collected from almost every wearable devices available in the market) in our proposed CardioGAN to capture the cardiac information of users and generate accurate ECG signals. We perform a multi-corpus subject-independent study, where the subjects have gone through a wide range of activities including daily-life tasks, which assures us of the usability of our proposed solution in practical settings. Most importantly, our proposed solution can be integrated into an existing PPG-based wearable device to extract ECG data without any required additional hardware. To demonstrate this concept, we have implemented our model to perform in real-time and used a wrist-based wearable device to feed it with PPG data. The video²²2https://youtu.be/z0Dr4k24t7U presented as supplementary material demonstrates CardioGAN producing realistic ECG from wearable PPG in real-time.