Hiding Your Signals: A Security Analysis of PPG-based Biometric Authentication

In the biometrics field, PPG signals are mainly used in two areas: user authentication and communication protocols.

User Authentication: The PPG signal contains a wealth of personal cardiac data. The earliest studies used clinical features commonly existed in PPG signals as biometric features  [9, 10]. For example, as shown in Fig. 2, the ratio of $b$ to $a$ is related to arterial stiffness [11]. $A1$ and $A2$ areas are related to systemic vascular resistance [12]. However, some characteristics of these features are sensitive to changes over time. For example, $\Delta T$ changes with an individual’s age [13]. As a result, researchers have applied a few techniques to get stable features, such as filtering noise-sensitive features using feature selection methods like Kernel Principal Component Analysis [14], transforming robust features by leveraging wavelet transform [15] and Short-time Fourier Transform [16]. GAN has also been used to synthesize temporally stable features of individual human [17].

After constituting an individual template using the extracted signal features, the process of identifying the individual template is similar to other biometric authentication systems. The early generations of authentication systems match individuals by calculating the distance between features and templates [18, 9]. Gradually, machine learning models replace simple distance calculations to improve the performance of template matching. However, many machine learning models require manual feature extraction so that the system designer and engineers may introduce bias. Recently, deep learning methods have provided an end-to-end authentication strategy, for example, CNN and LSTM are used to learn features from raw data automatically in the context of user recognition [19, 3].

Communication Protocols: In body area sensor networks (BASN) with strict computational resource constraints, traditional encryption schemes for information transmission are difficult to apply because PPG signals can get similar measurements in different parts of the body. When used as an entity identifier in a BASN symmetric encryption scheme, the distribution of pre key-agreement could be avoided [20]. IPI information extracted from the PPG signal is the most prevalent entity identifier. IPI indicates the time difference between two consecutive heartbeats, while PPG signal is usually regarded as the time difference between two pulse peaks. The timing of systolic peaks in the PPG signal is sympathetically related and influenced by various physiological factors. This characteristic allows the researcher to extract entity identifiers from the IPI information [21].

After obtaining the IPI, the randomness of the entity identifier needs to be extracted by quantization algorithms. A quantization algorithm encodes the signal into a binary representation. Two quantization methods exist. The first method was proposed in [22] to extract a random value from the intermediate bits of IPI. However, this method suffers from obvious issues, that is, the high bits of IPI are not random enough while the low bits have too much noise. The second method was proposed in [21] to use the trend of the IPI sequence as a random source of information. The second method improves the randomness attribute at the cost of excessive processing time.

Most rPPG signals are obtained by analyzing subtle differences in the facial colour channels from a video clip. The green channel is more readily absorbed by hemoglobin than red and blue channels. In most circumstances, the green channel contains the strongest PPG signal, providing a high signal-to-noise ratio solution for signal acquisition [23]. Two blind source separation methods, Independent Component Analysis and Principal Component Analysis were prposed in [24, 25] to isolate the approximately clean signals from the RGB channels. Unlike contact PPG measurements, the signal reflected by the skin in rPPG measurements includes the specular reflection of natural light, leading to unpredictable normalization errors. CHROM [26] introduces chromatic aberration to eliminate specular reflection effects with the assumption that the pixels in the face region contribute equally to the rPPG signal. Nevertheless, pixels collected from the face region may be affected by different levels of noise.

A recent research trend is to apply deep learning in rPPG acquisition for an end-to-end solution framework. DeepPhys [27] uses an attention network with CNN to capture the differences in the spatial and temporal distribution of signals between video frames. PhysNet [28] uses a spatio-temporal convolutional network to restore the rPPG signal from the video clip. Meta-rPPG [29] presents a transduction meta-learner that allows the deep learning model to adapt the data distribution during deployment. A fully self-supervised training method was introduced in [30] to acquire the rPPG signals. In addition to existing progress, deep learning-based methods have good research potentials for rPPG signal acquisition.

In summary, the research trend for rPPG signal acquisition is towards automated end-to-end solutions with high accuracy and resilience to noise. However, many existing research works are evaluated by the extracted heart rate’s accuracy, neglecting the signal’s morphological characteristics.

The first attack against IPI-based authentication using rPPG was proposed by Calleja et al. [7] in 2015. IPI signals help derive secure random sequences in the key distribution component of IPI-based authentication systems whose security conditions rely on the assumption that IPI can be measured by contact devices only. In fact, most of the IPI information acquired by PPG signals could also be obtained by non-contact techniques (rPPG). By analyzing the face’s colour change information in the video clip, the rPPG signal can be extracted in a manner similar to PPG. The extracted rPPG signal can be used to generate IPI, violating the assumptions in IPI-based authentication protocols [6]. Although rPPG does not accurately describe IPI generated from ECG signals, it has similar accuracy to contact PPG. Currently, PPG-based biometric authentication is mainly based on the morphological characteristics of the PPG signal rather than the IPI information.

In previous attacks on PPG-based biometric authentication, researchers almost always assume that the victim’s PPG signal has been compromised. Karimian et al. [8] utilized a non-linear dynamic model [31] to extract model parameters from the victim’s PPG signal before transforming the model parameters using two Gaussian functions as mapping functions. Thereafter, a spoofing attack was launched using the forged victim’s PPG signal. Another study involved stealing a victim’s PPG signal stealthily by installing a malicious PPG sensor on a device that the victim would touch [32]. Based on the recorded signals, the attacker can use a waveform generator to simulate the victim’s PPG signal and compromise the authentication. However, contact-based PPG signals are challenging to obtain without the victim’s awareness, significantly limiting these attacks’ application scenarios.

PPG-based biometrics, including user authentication and IPI-based security protocols, are vulnerable to spoofing attacks. This paper assumes that the attacker could obtain a video clip with the victim’s face. In this age of social networking, it is easy to retrieve many videos with human faces on social media sites. For instance, YouTube and Facebook (Meta) host numerous HD video clips that are downloadable to the attacker. Furthermore, an attacker can record a video clip of a targeted victim’s face through a surveillance camera, especially when hidden cameras are often not perceived by the victim. In contrast to the previous attack assumptions, our method requires no leaked PPG signals. In addition, we assume that the attacker knows the inputs to the user authentication model. For instance, a PPG signal with one heartbeat cycle is taken as input. The length of the signal is resampled to 60. The amplitude of the signal is normalized to 0-1.

Before acquiring the rPPG signal, we use MTCNN [33] to detect the face region in each video frame before isolation. Then the non-skin part of the face region is filtered by the skin detection algorithm proposed in [34]. For example, clothes, hair, and other parts that do not provide PPG signal information will be filterd. We will get a sequence of video frames $F(n):0<n<total\ number\ of\ frames$ containing only the facial skin. $n$ denotes the $n$th frame in the video clip. Each frame $F(n)$ is stored as a matrix of $w\times h\times 3$, where $w$ and $h$ represent the frame’s width and height, and 3 is the number of color channels. The average value of RGB of the whole skin area is used as the R, G, B values of the current frame. We apply a bandpass filter to obtain the portion of the frame sequence with signal frequencies between 0.65Hz and 4.0Hz (i.e., between 39 and 240 bpm). The method in [35] was used to remove the breadth component of the signal. This preprocessing removes the signal’s noise and breathing components (signal trend).

Upon removing noises, we extract the rPPG signal from the consecutive skin frames. As listed in Section II-B, there are multiple methods for obtaining rPPG signals. For instance, CHROM, POS, LGI, PCA, CL_rPPG.

POS is similar to CHROM, which is also based on the skin reflection model assumption to obtain low signal-to-noise ratio PPG signals by removing specular reflections. First, POS uses a sliding window to normalize the RGB pixel values within the window temporally. $C(t)=[R(t),G(t),B(t)]^{T}$. Time normalization:

LGI extracts the Local Group Invariance features of heart rates from face videos. The singular value decomposition of $C_{tn}(t)$ is first performed, such that $C_{tn}(t)=U\Sigma V^{T}$. The projection plane is defined as $P$:

Principal component analysis (PCA) is a statistical analysis method to reduce features’ dimensionality. PCA uses an orthogonal transformation to project the observed values of potentially correlated variables into a set of linearly uncorrelated variables. PCA separates the periodic pulse signals from the noisy signals.

CL_rPPG is a deep learning-based rPPG signal extraction method. Traditional methods may lose vital information related to the heartbeat through manually designed features. The deep learning-based approach recovers the rPPG signal directly from the original face video. CL_rPPG uses PhysNet as the rPPG estimator to generate the rPPG signal. PhysNet is a spatiotemporal convolutional network representing both long-term and short-term spatio-temporal contextual features. CL_rPPG outputs the PPG signal through the upsampling interpolation and 3D convolution. CL_rPPG has two versions: supervised learning (CL_rPPG_P) and self-supervised learning (CL_rPPG_F). For the supervised learning mode, the CL_rPPG_P model is trained using the maximum cross-correlation between the ground truth PPG signal and the estimated PPG signal as the loss function. In the self-supervised learning model CL_rPPG_F, negative samples with high heart rates are artificially generated by a frequency resampler for contrast training. The power spectral density means the squared error is used to measure the difference between two signals.

To spoof IPI-based security protocols, we need to recover the IPI information from the signal. We mark the peak locations of the original signal before calculating the time difference between the peaks as IPI. As the standard heart rate in adults is 50-120 beats per minute [17], there are at most two heartbeats per second. To minimize sample loss, we exclude data where the distance between heartbeats is less than $FPS/2$.

For the trend-based quantification, we divide each IPI value into 16 parts of an equal length following a normal distribution. Our setting is identical to IMDGuard [36]. Then we encode the IPI sequence according to Algorithm 1.

To analyze the security of PPG-based authentication, we mainly consider the dataset, UBFC-PHYS [37]. Due to video quality, the rPPG signal recovered in other datasets is insufficient to threaten the biometrics.

UBFC-PHYS captures facial videos ($1024\times 1024$ resolution, 35 FPS, 227,474 Kbps) of 56 participants. Participants are about 1 meter away from the camera and the light source. The light source uses artificial light to ensure the same lighting conditions. Each participant recorded a one-minute video of the three states (‘resting’, ‘talking’, or ‘calculating’). Meanwhile, the Empatica E4 wristband was used to collect the PPG signal from the wrist synchronously with the sampling rate of 65 Hz. We exclude some videos in which faces are not correctly recognized in the video frames due to the participants’ activity. We perform repeated independent experiments for each user. Other users are treated as non-victims. As shown in Table I, we extracted rPPG signals ranging from 150 to 287 cardiac cycles for each state. The number of cardiac cycles varies depending on the individual.

To analyze the threat of rPPG for IPI-based protocols, we considered the four most commonly used rPPG datasets, PURE [38], UBFC_rPPG [39], LGI_PPGI [40] and COHFACE [41].

PURE captures 10 participants’ facial videos. Each participant’s videos were recorded in six states (Steady, Talking, Small/Medium Rotation, and Slow/Fast Translation). Each video clip is one-minute long with a resolution of 640 $\times$ 480 and a frame rate of 30 FPS. Meanwhile, the finger clip pulse oximeter acquired the PPG signal at a 65Hz sampling rate. Although there was facial movement in the video, all signals were captured during the rest state.

UBFC_rPPG uses a webcam to record video (640 $\times$ 480 resolution, 30 FPS). Transmissive pulse oximetry (62 Hz) to obtain the PPG signal. It consists of two sub-datasets. In UBFC_1, participants were asked to remain stationary; in UBFC_2, participants played a mathematical game in front of a green screen.

LGI_PPGI records facial videos in four scenes (resting, head movement, fitness, and conversation in an urban background). The video clips were captured by a webcam (640 $\times$ 480 resolution, 25 FPS). The sampling rate of the pulse oximeter is 60 Hz. However, only 6 of the original 25 participants’ videos are publicly available. Our experiments were conducted on these 6 participants.

COHFACE contains facial video clips (640 $\times$ 480 resolution, 20 FPS) of 40 subjects, including the simultaneous acquisition of PPG signals (256 Hz). Subjects were asked to sit still in front of the camera. The video was heavily compressed.

We use Keras¹¹1https://www.tensorflow.org/ to implement the most advanced PPG-based user authentication model [3]. We use the virtual heart rate pyVHR²²2https://github.com/phuselab/pyVHR [42] to implement the different traditional rPPG methods (CHROM, POS, LGI, PCA). It provides a uniform standard of evaluation. For deep learning methods CL_rPPG, we use the open-source code³³3https://github.com/ToyotaResearchInstitute/RemotePPG/tree/master/iccv provided in the article.

For the user authentication model, we mark the signals of the victims as class 1 and other users as 0. We aim to compare the threat of rPPG signals with other users’ PPG signals for the user authentication system. We treat each user as a victim and one-tenth of other users are used to train the user authentication model, which is realistic for authentication systems. It is possible to collect a large amount of data about the target user, but only a small amount of data from other users. The remaining data from other users were used as a random attack to evaluate the spoofing attack by rPPG signals. To explore the impact of signals collected in different states on spoofing attacks, we exclude the PPG signals in the ‘talking’ and ‘calculating’ states while training the user authentication model. The performance of the user authentication model is determined by the Equal Error Rate (EER). We also call the model for the false acceptance rate (FAR) of the rPPG signal as the success rate of spoofing. A higher success rate indicates that the authentication system is more vulnerable to such spoofing attacks. Finally, we use the average results of all users.

For IPI-based security protocols, we evaluate them by using heart rate (HR) and IPI, respectively. The primary metrics are mean absolute error (MAE) and root-mean-square deviation (RMSE). We also use Pearson’s coefficient (PC) to compare the similarity between the PPG signal and the rPPG signal. Since IPI-based security protocols require multi-bit encoding from IPI, we use the bit hit rate (BHR) to show how many codes can be recovered by the rPPG signal. BHR is the percentage of recovered codes for all codes.

Tab. II shows the results of a random attack versus an attack using the victim’s rPPG signal. The success rate of using the victim’s rPPG signal is almost doubled compared to random attacks. The success rate of the Mean-treated signal is even higher, which is 0.34. Although the success rate of rPPG signal is reduced in the Talking and Calculating states, the mean treatment can help mitigate this effect. The attack’s success rate is 0.35 on average. It is a threat to the authentication model, since real-world authentication systems often allow users to make multiple attempts. By analyzing the results, we also found that individual differences were also evident in addition to the quality of the video, which significantly affected the attack’s success rate. The highest success rate of rPPG signal can reach 0.98 for users. In contrast, some users’ rPPGs cannot be used for spoofing attacks. Through the analysis, we found a significant difference in the morphology of some users’ rPPG and PPG signals.

We report MAE and RMSE for the video extracted IPIs with the PPG signal extracted IPI in Tab. III. The MAE of the recovered IPI in the high-quality original video is below 0.1. CL_P reaches a minimum MAE of 0.03. It also recovered an IPI that the Pearson correlation coefficient (PC) is as high as 0.99 with the IPI of PPG. It means the IPIs extracted by CL_P and PPG signals are very similar.

Tab. IV shows the BHR of the IPI-trend encoding extracted from the original video n different datasets. The BHR of the original video clip is around 0.6, implying that an attacker has a 60% success rate of breaking the PPG-based IPI security protocol. In the dataset UBFC_1, CL_F reached the highest BHR of 0.65. Due to the compression, the results in COHEFACE are almost equivalent to a random guess in every rPPG extraction method. Nevertheless, the BHR of CL_F and CL_P still reaches 0.58 and 0.57.

For quantile-based IPI coding, the randomness of the bit usually increases as the location of bits decreases. Because the performance of the original video extraction from COHFACE was poor in the previous experiments, we choose not to conduct the experiments on COHFACE. As shown in Fig. 4, the BHR of the encoding extracted from the original video varies with the position of bits. As the bit position decreases, so does the BHR. In the datasets LGI and UBFC_1, it is obvious to find that the BHR shows a decreasing trend from 0.7 to 0.5 with the decrease of bits. In UBFC_2, the CL_F high BHR even reached above 0.8. Although the BHR in CL_P does not vary with bits, their overall BHR is higher than 0.65.

According to the results in Section IV, we conducted an exhaustive analysis on broadly used datasets. We observed that with the improvement of rPPG extraction methods, state-of-the-art methods have been able to recover the IPI from rPPG signals accurately. Both rPPG and PPG signals reflect individual cardiac information, making them contain a great deal of similar information. While this has facilitated the development of telemedicine, rPPG poses an actual threat to PPG-based biometric authentication, both for user authentication and IPI-based security protocols. In user authentication, the spoofing attack achieves a success rate of 35%, which is a significant improvement of 20% compared to random attacks. Even when the user is in a talking and calculating state, the success rate of rPPG attacks is still much higher than that of random attacks. In IPI-based security protocols, the state-of-the-art rPPG method recovered IPI with a BHR above 0.6. These results suggest numerous similar morphological features between the rPPG signal and the PPG signal. The exposure of these features makes it easier for attackers to compromise PPG signal-based authentication systems.

Tab. V shows the performance of the passive defence strategy. These results show that all passive defence strategies significantly reduce the attack’s success rate. The success rate of rPPG attacks has been reduced from 0.25 to at least 0.04. Mean-treated rPPG was reduced to 0.05. However, we observe that the passive defence strategy seriously affects the performance of the authentication model system. The incremental updating method has the least impact on the authentication system. The F1-Score of the model dropped from 0.87 to 0.82. It reduces the success rate of rPPG attacks and makes the model less generalized. IsolationForest and OneClassSVM have a significant impact. The F1-Score of the model is around 0.73. Unfortunately, it propagates the error rate to the authentication model while blocking the attack.

We propose an active defense strategy to solve the problems in passive defence. To maximize the active defence strategy’s performance, we selected 13 video clips from the original videos with the success rate of being attacked greater than 0.5. As shown in Tab. VI, the attack’s success rate using the rPPG signal extracted from the original video reaches 0.7, and the mean-treated rPPG signal even reaches 1. The success rate of video rPPG attacks processed by the active defence strategy is reduced to 0.02, and the mean-treated rPPG is decreased to 0.05, almost achieving the same performance as the passive defence. Moreover, the active defence strategy does not affect the original model’s performance.

Compared to the best results (CL_P, dataset PURE) of the original video extracted IPI, in our processed video, MAE increased to 0.2, PC was reduced to 0.28. In the other datasets, MSE increased to some extent. However, we found that the variation was minimal in COHFACE because the video clips in COHFACE are compressed. The MAE of the IPI recovered from COHFACE has reached 0.2s. Their PC is around 0.05 for all methods except CL_P and CL_F. It indicates that the IPI recovered by other methods has a low correlation with the IPI of PPG. After processing the video, even the state-of-the-art method PC is reduced to approximately 0.3.

Contrary to the result in Section IV-D, Tab. VIII shows that the BHR of videos processed by active defense methods is approximately 0.5, which is almost equal to random guessing. As shown in Fig. 7, in the processed video, BHR remains steady at approximately 0.5 with the bit position. For example, as shown in Fig. 4(a), the LGI and POS methods gradually decrease in BHR from 0.7 to 0.5 as the bits go down. In contrast, as shown in Fig. 7(a), BHR does not change with the bits. It means that our proposed active defense method completely hides the PPG signal of the user in the video.

A common strategy to combat spoofing attacks is by introducing detection components. Nevertheless, the detection component is usually located in front of the recognition pipeline, and its errors will be propagated to the recognition model, affecting the overall model recognition performance. In particular, when the spoofed signal is similar to a real signal, the model’s false-negative rate increases significantly. We observed that low-quality videos lower the success of spoofing attacks. For example, in the COHFACE dataset, the quality of the obtained rPPG signals is relatively poor, and the success rate of the attack is lower than in other datasets. Compared to other datasets, the video clips in COHFACE have a frame rate of 20 FPS, $640\times 480$ resolution, and 255 Kbps bit rate. However, preventing attacks by sacrificing video quality is not always practical. Thus, we propose an active defence to prevent leaking the target’s PPG signals by modifying the RGB pixel values of the facial skin in the video frame. Video hosting platforms like YouTube and TikTok can successfully prevent signal leakage by batch processing users’ videos.