Biometric Authentication Based on Enhanced Remote Photoplethysmography Signal Morphology

Abstract

Remote photoplethysmography (rPPG) is a non-contact method for measuring cardiac signals from facial videos, offering a convenient alternative to contact photoplethysmography (cPPG) obtained from contact sensors. Recent studies have shown that each individual possesses a unique cPPG signal morphology that can be utilized as a biometric identifier, which has inspired us to utilize the morphology of rPPG signals extracted from facial videos for person authentication. Since the facial appearance and rPPG are mixed in the facial videos, we first de-identify facial videos to remove facial appearance while preserving the rPPG information, which protects facial privacy and guarantees that only rPPG is used for authentication. The de-identified videos are fed into an rPPG model to get the rPPG signal morphology for authentication. In the first training stage, unsupervised rPPG training is performed to get coarse rPPG signals. In the second training stage, an rPPG-cPPG hybrid training is performed by incorporating external cPPG datasets to achieve rPPG biometric authentication and enhance rPPG signal morphology. Our approach needs only de-identified facial videos with subject IDs to train rPPG authentication models. The experimental results demonstrate that rPPG signal morphology hidden in facial videos can be used for biometric authentication. The code is available at https://github.com/zhaodongsun/rppg_biometrics.

1 Introduction

Facial videos contain invisible skin color changes induced by remote photoplethysmography (rPPG) signals, providing valuable cardiovascular information, such as heart rate. Similar to rPPG, contact photoplethysmography (cPPG) captures color changes in fingertips to monitor blood volume changes. cPPG signals, obtained using contact sensors, have been used for biometric authentication [12, 11]. Given the similar nature and measurement principles of rPPG and cPPG [26], rPPG has the potential for biometric authentication. However, the feasibility of rPPG biometric authentication still needs to be validated. Hence, our research questions are: 1) Can rPPG signals be employed for biometric authentication? 2) If so, how can an rPPG-based biometric system be developed? 3) What are the advantages associated with utilizing rPPG biometrics?

(a) rPPG Authentication System

(b) rPPG Morphology Enhancement

Figure 1: (a) rPPG Authentication System. (b) Our method can improve rPPG morphology information. The fiducial points [22] like the systolic peaks and diastolic peaks are the main subject-specific biometric characteristics in rPPG signals.

We first examine the quality and discriminative power of rPPG signals. rPPG signals are derived from subtle changes in facial color caused by blood volume changes during heartbeats. Recent advances [49, 20] have achieved high-quality rPPG measurement, especially when the face has minimal or no movement. Hence, it is feasible to obtain high-quality rPPG signals. However, the question remains whether these high-quality rPPG signals contain subject-specific biometric characteristics. One work [32] has tried using rPPG for biometrics, but the preliminary study was limited by a small-scale dataset and low-quality rPPG, offering inadequate authentication performance for practical applications.

In this paper, we propose an rPPG-based method for biometric authentication, as shown in Fig. 1(a). Considering facial appearance and rPPG are mixed together in facial videos, we first de-identify facial videos while preserving the rPPG information. This step can guarantee that only rPPG information is used for biometric authentication while facial appearance cannot be used. In addition, this step can also conceal sensitive facial appearance information for privacy protection. The first module is the rPPG model that can extract rPPG signals from the de-identified facial videos. The second module is the rPPG-Authn model that utilizes the rPPG morphology to output person authentication results. We design a two-stage training strategy and rPPG-cPPG hybrid training by incorporating external cPPG datasets to exploit rPPG morphology for biometric authentication. Fig. 1(b) illustrates the rPPG morphology enhancement. Note that we only use de-identified videos with subject IDs for rPPG biometrics.

There are several advantages of rPPG biometrics. Compared with facial appearances, the rPPG biometric system only utilizes de-identified facial videos, eliminating the need for sensitive facial appearance. Moreover, rPPG biometrics offers an additional degree of resistance to spoofing, as rPPG inherently serves as a countermeasure to presentation attacks [21, 19]. In contrast, without dedicated presentation attack detection (PAD) methods, conventional face recognition algorithms are vulnerable to presentation attacks and less secure than rPPG-based biometrics. Additionally, since both rPPG biometrics and face recognition use facial videos as data sources, combining both biometric modalities can potentially enhance both accuracy and security. When compared with cPPG biometrics, rPPG biometrics offers the advantages of being non-contact and only requiring off-the-shelf cameras, while cPPG biometrics necessitates specific contact sensors like pulse oximeters. Compared with iris recognition [46, 5] which requires iris scanners, rPPG biometrics only requires cheap RGB cameras and is robust to presentation attacks.

Our contributions include:

1. We propose a new biometric authentication method based on rPPG. We utilize two-stage training to achieve rPPG morphology enhancement and accurate biometric authentication performance. We illustrate that utilizing de-identified facial videos is effective for rPPG biometric authentication and ensures the protection of facial appearance privacy.

2. We conduct comprehensive experiments on multiple datasets to validate the discriminative power of rPPG biometrics. We demonstrate that rPPG biometrics can achieve comparable performance with cPPG biometrics. We also investigate factors that may influence the performance of rPPG biometrics.

3. We discover that our rPPG-based biometric method can enhance rPPG morphology, which opens up possibilities for rPPG morphology learning from facial videos.

2 Related Work

2.1 rPPG Measurement

[41] initially proposed measuring rPPG from face videos via the green channel. Subsequent handcrafted methods have been introduced to enhance the quality of the rPPG signal [34, 6, 18, 40, 45]. Recently, there has been rapid growth in deep learning (DL) approaches for rPPG measurement. Several studies [4, 37, 20, 31, 16] utilize 2D convolutional neural networks (CNN) to input consecutive video frames for rPPG measurement. Another set of DL-based methods [28, 29, 23, 24, 7] employ a spatial-temporal signal map obtained from different facial regions, which is then fed into 2DCNN models. 3DCNN-based methods [50] and transformer-based methods [52, 51] have been proposed to enhance spatiotemporal performance and long-range spatiotemporal perception.

Additionally, multiple unsupervised rPPG methods [8, 43, 39, 36, 47, 53] have been proposed. Since GT signals are expensive to collect and synchronize in rPPG datasets, unsupervised rPPG methods only require facial videos for training without any GT signal and achieve performance similar to the supervised methods. However, most works on rPPG measurement primarily focus on the accuracy of heart rate estimation, while neglecting the rPPG morphology.

2.2 cPPG-based Biometrics

[10] was the first attempt to utilize cPPG for biometric authentication. They extracted some fundamental morphological features, such as peak upward/downward slopes, for cPPG biometrics. Subsequently, other studies have explored additional morphological features, including cPPG derivatives [48] and fiducial points [22]. More recently, researchers have focused on employing DL methods to automatically extract morphological features. [25, 2, 15] directly input cPPG signals into 1DCNN or long short-term memory (LSTM) architectures to conduct biometric authentication, while [12, 11] cut cPPG signals into periodic segments and utilize multiple representations of these periodic segments as inputs to a 1DCNN model. Furthermore, [12] has collected datasets for cPPG biometrics and investigated the permanence of cPPG biometrics. There exists one preliminary work on rPPG biometrics [32], but only a traditional independent component analysis (ICA) based method [34] was applied for rPPG extraction, which yields low-quality rPPG morphology for biometric authentication.

3 Method

Our method consists of facial video de-identification and two training stages. As the rPPG signal does not rely on facial appearance, we first de-identify the input video to avoid facial appearance being used by our method. In the first training stage, we perform unsupervised rPPG training on the de-identified videos to achieve basic rPPG signal measurement. In the second training stage, we use rPPG-cPPG hybrid training for biometric authentication and rPPG morphology enhancement.

3.1 Face De-identification for rPPG Biometrics

We propose to de-identify facial videos using spatial downsampling and pixel permutation. This step aims to obfuscate facial appearances while preserving the rPPG information. Since rPPG signals are spatially redundant at different facial regions and largely independent of spatial information as shown by [40, 27], rPPG signals can be well preserved in this step while facial appearances are completely erased. The reasons for face de-identification are twofold. First, the facial appearance and rPPG information are intertwined in facial videos. We remove facial appearance to make sure that the biometric model performs recognition solely based on the rPPG information. Second, this step can remove facial appearances to protect facial privacy information during rPPG authentication.

Figure 2: Face de-identification for rPPG biometrics. The facial appearance is obfuscated while rPPG information is retained.

The facial video is de-identified as shown in Fig. 2. Faces in the original videos are cropped using OpenFace [1] by locating the boundary landmarks. The cropped facial video $v\in\mathbb{R}^{T\times H\times W\times 3}$, where $T$, $H$, and $W$ are time length, height, and width, is downsampled by averaging the pixels in a sample region to get $v_{d}\in\mathbb{R}^{T\times 6\times 6\times 3}$. It has been demonstrated that such downsampled facial videos are still effective in rPPG estimation [40, 27]. Since rPPG signal extraction does not largely depend on spatial information [40], we further permutate the pixels to completely obfuscate the spatial information to get $v_{de}\in\mathbb{R}^{T\times 6\times 6\times 3}$. Note that the permutation pattern is the same for each frame in a video but distinct for different videos. Since the spatial information is eliminated, we reshape the de-identified video $v_{de}$ into a spatiotemporal (ST) map $M\in\mathbb{R}^{36\times T\times 3}$ for compact rPPG representation like [27].

3.2 The 1st training stage: rPPG Unsupervised Pre-training

This stage aims to train a basic rPPG model capable of extracting rPPG with precise heartbeats. We use unsupervised training to obtain the basic rPPG model. The main reasons for unsupervised training are: 1) Unsupervised rPPG training does not require GT PPG signals from contact sensors, which means only facial videos with subject IDs are required in our entire method. 2) The performance of unsupervised rPPG training [8, 39] is on par with supervised methods.

Figure 3: The diagram of Contrast-Phys-2D (CP2D) for rPPG unsupervised pre-training based on contrastive learning.

We adopt and customize the unsupervised Contrast-Phys (CP) architecture [39] to 2D ST-map inputs since CP can only use face videos as inputs. The modified method called Contrast-Phys-2D (CP2D) is shown in Fig. 3. Two different ST maps $M,M^{\prime}\in\mathbb{R}^{36\times T\times 3}$ from two different videos are the inputs of the rPPG model $G$, where $T$ is 10 seconds. The rPPG model $G$ is based on a 2D convolutional neural network to output rPPG ST maps $M_{rppg},M^{\prime}_{rppg}\in\mathbb{R}^{S\times T}$ where rPPG signals are stacked vertically. Similar to CP, the spatial dimension $S$ is set as four. The architecture of the rPPG model $G$ is presented in the supplementary materials. Inspired by spatiotemporal rPPG sampling in CP, we use a patch with the shape $(1,T/2)$ to randomly get $N=16$ rPPG ST samples $\{m_{rppg(1)},...,m_{rppg(N)}\},\{{m^{\prime}_{rppg(1)},...,m^{\prime}_{rppg(N)}}\}$ from rPPG ST maps $M_{rppg},M^{\prime}_{rppg}$, respectively. The rPPG ST samples are averaged along the spatial dimension to get rPPG samples $\{p_{1},...,p_{N}\},\{p^{\prime}_{1},...,p^{\prime}_{N}\}$ and the corresponding power spectral densities (PSDs) $\{f_{1},...,f_{N}\},\{f^{\prime}_{1},...,f^{\prime}_{N}\}$. We use rPPG prior knowledge [39] including rPPG spatiotemporal similarity and cross-video rPPG dissimilarity to make positive pairs ($(f_{i},f_{j})$ or $(f^{\prime}_{i},f^{\prime}_{j})$) and negative pairs $(f_{i},f^{\prime}_{j})$, which can be used in the positive and negative terms in the contrastive loss $L$. The contrastive loss $L$ is used to pull together the PSDs originating from the same videos and push away the PSDs from different videos. The loss function $L$ is shown below. During inference, the rPPG ST map $M_{rppg}\in\mathbb{R}^{S\times T}$ is averaged along the spatial dimension to get the rPPG signal $s_{rppg}\in\mathbb{R}^{T}$.

Figure 4: GT cPPG signal and rPPG signal extracted by CP2D. After the first training stage, the rPPG signal has accurate heartbeats but lacks morphology information.

However, since CP2D does not utilize any prior knowledge about morphology, the resulting rPPG signals lack morphology information. Fig. 4 shows a GT cPPG signal and an rPPG signal produced by CP2D. CP2D generates an rPPG signal with accurate heartbeats that align with those of the cPPG signal. However, the morphological features, such as the dicrotic notch and diastolic peak evident in the cPPG morphology, are not clearly discernible in the rPPG signals. Since these morphological features play a crucial role in differentiating individuals, we aim to further refine the rPPG signal morphology at the second training stage.

3.3 The 2nd training stage: rPPG-cPPG Hybrid Training

At the second training stage, we further refine rPPG signals to obtain morphology information. Fig. 5 shows the rPPG-cPPG hybrid training, where the rPPG branch utilizes face videos and ID labels during training. On the other hand, the cPPG branch uses external cPPG biometric datasets to encourage the PPG-Morph model $H$ to learn morphology information, which can be incorporated into the rPPG branch through the PPG-Morph model $H$. The PPG-Morph model $H$ comprises 1DCNN layers and transformer layers that extract morphological features from periodic segments. The two branches are trained alternately to facilitate the sharing of morphology information between the rPPG and cPPG branches. Note that our method only requires de-identified facial videos with subject IDs during training (enrollment) and only needs de-identified facial videos during inference.

Figure 5: rPPG-cPPG hybrid training. The rPPG branch and cPPG branch are trained alternatively to utilize external cPPG signals to enhance the rPPG morphology fully.

3.3.1 rPPG Branch

The rPPG branch can extract rPPG morphology and use it to differentiate individuals. This branch only requires a de-identified facial video $v_{de}$ and the ID label $i_{rppg}$ and does not need any GT cPPG signal for training. Therefore, de-identified facial videos with ID labels are sufficient for enrollment in the proposed rPPG biometrics scheme. The ST map $M$ derived from the de-identified facial video $v_{de}$ is fed into the pre-trained rPPG model $G$ to obtain the rPPG signal $s_{rppg}$. Note that the rPPG model $G$ is the pre-trained model from the first unsupervised training stage. To segment the signal, the systolic peaks are located, and the signal $s_{rppg}\in\mathbb{R}^{T}$ is divided into K clips. Due to heart rate variability, the K clips may have different lengths, so the clip length is interpolated to 90 in order to obtain rPPG periodic segments. The choice of a length of 90 is based on the fact that the minimum heart rate (40 beats per minute) for a 60 Hz signal produces the longest periodic segment with a length of 90. Consequently, we obtain $C_{rppg}\in\mathbb{R}^{K\times 90}$. To predict an authentication score for an individual, we use the PPG-Morph model $H$ and the rPPG classification head $h_{rppg}$, which provides the rPPG morphology representation $f_{rppg}\in\mathbb{R}^{64}$ and ID probability $y_{rppg}\in[0,1]^{K\times N_{rppg}}$, where $N_{rppg}$ is the number of individuals in the rPPG biometric dataset. The cross-entropy loss is used for ID classification, which is

$$L_{rppg-id}(y_{rppg},i_{rppg})=-\frac{1}{K}\sum_{k=0}^{K}\log(y_{rppg}^{k,i_{rppg}})$$ (2)

where $y_{rppg}^{k,i_{rppg}}$ is the predicted probability of the kth periodic segment belonging to the ID label $i_{rppg}\in\{1,2,...,N_{rppg}\}$.

3.3.2 cPPG Branch

The cPPG branch utilizes external cPPG biometric datasets including Biosec2 [12], BIDMC [33, 9], and PRRB [14], to learn PPG morphology. Note that the external cPPG biometric datasets are available online and are not related to the facial videos in the rPPG branch. Similar to the rPPG branch, the cPPG signal $s_{cppg}\in\mathbb{R}^{T}$ is processed to obtain $K^{\prime}$ cPPG periodic segments $C_{cppg}\in\mathbb{R}^{K^{\prime}\times 90}$. The PPG-Morph model $H$ and cPPG classification head $h_{cppg}$ are employed to generate the cPPG morphology representation $f_{cppg}\in\mathbb{R}^{64}$ and the ID probability prediction $y_{cppg}\in[0,1]^{K^{\prime}\times N_{cppg}}$, where $N_{cppg}$ is the number of individuals in the external cPPG biometric datasets. Note that the PPG-Morph model $H$ is shared by both the rPPG branch and cPPG branch, allowing the cPPG branch to transfer the learned morphology information to the rPPG branch. The cross-entropy loss is utilized in this branch, which is

$$L_{cppg-id}(y_{cppg},i_{cppg})=-\frac{1}{K^{\prime}}\sum_{k=0}^{K^{\prime}}\log(y_{cppg}^{k,i_{cppg}})$$ (3)

where $y_{cppg}^{k,i_{cppg}}$ is the predicted probability of the kth periodic segment belonging to the ID label $i_{cppg}\in\{1,2,...,N_{cppg}\}$.

3.3.3 Alternate Backpropagation

We alternately train the two branches and backpropagate the gradient of the two loss functions $L_{rppg-id}$ and $L_{cppg-id}$ to achieve rPPG-cPPG hybrid training. During the first step, de-identified facial videos and ID labels are sampled from the rPPG biometric dataset to calculate the loss $L_{rppg-id}$, and the gradient of $L_{rppg-id}$ is backpropagated to update the rPPG model $G$, the PPG-Morph model $H$, and the rPPG classification head $h_{rppg}$. During the second step, cPPG signals and ID labels are sampled from external cPPG biometric datasets to calculate the loss $L_{cppg-id}$, and the gradient of $L_{cppg-id}$ is backpropagated to update PPG-Morph model $H$ and the cPPG classification head $h_{cppg}$. These two steps are repeated in an alternating manner, allowing the two branches to be trained in turns. The cPPG branch uses external cPPG datasets to encourage the PPG-Morph model $H$ to learn morphology information. The morphology features learned from the cPPG branch can then be incorporated into the rPPG branch since the PPG-Morph model $H$ is shared by both cPPG and rPPG branches thus rPPG features are enhanced. The supplementary materials provide a detailed description of the algorithm.

4 Experiments

Signal length	EER$\downarrow$/AUC$\uparrow$
	OBF		UBFC-rPPG	PURE
	intra-session	cross-session	intra-session	intra-session	cross-session
20 heartbeats ($\sim$20 sec)	0.17%/99.97%	2.16%/98.10%	0%/100%	0%/100%	9.59%/93.70%
10 heartbeats ($\sim$10 sec)	0.14%/99.98%	2.61%/98.04%	0%/100%	0.33%/99.67%	14.00%/91.17%
5 heartbeats ($\sim$5 sec)	0.33%/99.97%	3.81%/97.89%	0.01%/99.99%	0.58%/99.36%	18.32%/86.81%

Table 1: EER and AUC for rPPG authentication on OBF, UBFC-rPPG, and PURE datasets.

4.1 Implementation Details

Datasets. We considered three public rPPG datasets, namely OBF [17], PURE [38], and UBFC-rPPG [3]. The scales of these rPPG datasets are enough to validate the feasibility of rPPG biometrics since previous cPPG biometric datasets [12, 11] also have similar scales. These rPPG datasets consist of facial videos, GT cPPG signals, and ID labels, but our method does not require the GT cPPG. OBF dataset [17] consists of data from 100 healthy subjects. Two 5-minute RGB facial videos were recorded for each participant. For each subject, the first facial video was recorded at rest, while the second was recorded after exercise. During the recording, participants remained seated without head or facial motions. Videos have a resolution of 1920×1080 at 60 frames per second (fps). UBFC-rPPG dataset [3] was captured using a webcam at a resolution of 640x480 at 30 fps. In each recording, the subject was positioned 1 meter away from the camera and playing a mathematical game, with the face centrally located within the video frame. The database consists of data from 42 participants, with each one having a 1-minute video. PURE dataset [38] contains data from 10 subjects. Face videos for each subject were captured in 6 distinct scenarios: steady, talking, slow translation, fast translation, small rotation, and medium rotation, leading to a total of 60 one-minute RGB videos. Videos have a resolution of 640×480 at 30 fps.

Additionally, we combined the Biosec2 [12], BIDMC [33, 9], and PRRB [14] datasets to create the external cPPG biometric dataset. These datasets contain cPPG signals from 195 subjects for the cPPG branch in the rPPG-cPPG hybrid training. More details about datasets are provided in the supplementary materials.

Experimental Setup. Our rPPG biometric experiments follow the previous cPPG biometric protocol [12, 11] where the training and test sets have the same persons but might be recorded in the same session (intra-session test) or recorded in different sessions (cross-session test). For the OBF dataset, we divide each pre-exercise video into three parts: the first 60% length is used for training, the following 20% length is used for validation, and the last 20% length is used for intra-session testing. The post-exercise videos are reserved for cross-session testing. As for the UBFC-rPPG dataset, the same division is applied to each video. Since each subject only contributes one video, only intra-session testing can be conducted on this dataset. Moving on to the PURE dataset, the same division is applied to each steady video. The videos involving head motion tasks are used exclusively for cross-session testing. At the first training stage, we select the best rPPG model with the lowest irrelevant power ratio (IPR) in the validation set, as conducted in [8, 39]. At the second training stage, we choose the best-performing models based on the lowest equal error rate (EER) in the validation set. Both training stages are carried out on a single Nvidia V100 GPU and employ the Adam optimizer with a learning rate of 1e-3. During inference, the predicted probabilities from consecutive periodic segments (5 beats, 10 beats, and 20 beats) are averaged.

Evaluation Metrics. Since the model does multi-class classification, we use the one-vs-rest strategy to get the authentication results for each person. Therefore, each person has a binary classification. For each person, we can change the threshold of the model prediction output for that person to get the binary predictions, and we can plot false positive rates and true positive rates in a graph, which is the receiver operating characteristic (ROC) curve. Areas under curve (AUC) is the area under the ROC curve. If we change the threshold, we can find the threshold where the false positive rate and the false negative rate are equal. The EER is the false positive rate or false negative rate at this threshold. The final EER and AUC are averaged across all subjects. To evaluate the rPPG morphology, we calculate the Pearson correlation between the means of periodic segments from rPPG and the GT cPPG. More details are in the supplementary materials.

4.2 Results and discussions

4.2.1 Results and discussions about rPPG authentication.

Table 1 presents the results of rPPG authentication with varying signal lengths. The performance of rPPG authentication improves with longer signal lengths, such as 20 heartbeats, compared to shorter signal lengths like 10 or 5 beats. On all three datasets, the intra-session performance is satisfactory, with EERs below 1% and AUCs above 99%. However, the performance decreases during cross-session testing. On the OBF dataset, the cross-session (pre-exercise $\to$ post-exercise) performance is slightly lower than the intra-session (pre-exercise $\to$ pre-exercise) performance, but still achieves EER of 2.16%. On the PURE dataset, there is a significant drop in performance during cross-session (steady $\to$ motion tasks) compared to intra-session (steady $\to$ steady) due to the adverse impact of motion tasks on the quality of rPPG signals. Conversely, although the OBF dataset includes exercises to increase heart rates, it does not involve facial movements. This indicates that rPPG biometrics is sensitive to low-quality rPPG caused by facial motions but rPPG has reliable and unique biometric information evidenced by the varying heart rates from the same people. In practical usage, users will face the camera and keep still (like face recognition), thus such large intended head motions will not be a concern.

The observed rPPG periodic segments from different subjects (subject A-I) in Fig. 6 align with the aforementioned quantitative results. The rPPG periodic segments from the OBF dataset exhibit consistent morphology before and after exercises in Fig. 6(a). Conversely, the motion tasks in the PURE dataset significantly alter morphology in Fig. 6(c), resulting in noisy rPPG signals and a drop in performance during cross-session testing. Furthermore, the rPPG periodic segments from all three datasets display distinct morphologies for different subjects, highlighting the discriminative power of rPPG morphology. Fig. 7 shows the subject-specific biometric characteristics of rPPG morphology in detail. The rPPG periodic segments from two subjects have distinct fiducial points [22] such as the systolic peaks, diastolic peaks, dicrotic notch, and onset/offset, which contain identity information.

(a) rPPG periodic segments from OBF dataset

(b) rPPG periodic segments from UBFC-rPPG dataset

(c) rPPG periodic segments from PURE dataset

Figure 6: rPPG periodic segments from (a) OBF dataset, (b) UBFC-rPPG dataset, and (c) PURE dataset. The red curves are the means of periodic segments.

Figure 7: rPPG periodic segments and fiducial points from two subjects.

Regarding fairness, prior studies [30, 42] highlighted skin bias in rPPG signal quality. Dark skin may yield lower-quality rPPG signals, impacting authentication performance. We assess authentication performance for light and dark skin groups in the OBF dataset with a 20-heartbeat signal length and cross-session testing. For light skin, EER and AUC are 2.52% and 97.79%, respectively. For dark skin, EER and AUC are 4.04% and 96.74%. The performance of dark skin slightly falls behind that of light skin, indicating a skin tone bias in rPPG biometrics. Addressing this fairness issue may involve collecting more data from dark-skinned people or developing new algorithms, which remains a topic for future research.

Biometric Methods	EER$\downarrow$/AUC$\uparrow$
	OBF		UBFC-rPPG	PURE
	intra-sess	cross-sess	intra-sess	intra-sess	cross-sess
FaceNet [35]$\blacktriangle$	32.07%/65.87%	36.58%/60.84%	36.15%/61.03%	31.67%/66.67%	35.67%/65.11%
Privacy-preserving FR [13]$\blacktriangle$	6.46%/91.24%	6.52%/91.92%	7.26%/90.25%	6.88%/91.27%	7.82%/90.77%
Hwang2021 [11]$\blacklozenge$	1.21%/99.30%	16.72%/84.74%	6.30%/94.02%	0%/100%	4.23%/98.14%
Patil2018 [32]$\bigstar$	14.97%/89.42%	39.79%/62.14%	8.53%/88.70%	4.00%/92.00%	32.68%/72.11%
Ours w/ rPPG training$\bigstar$	0%/100%	3.23%/96.92%	$-*$	0%/100%	11.68%/92.61%
Ours w/ rPPG-cPPG hybrid training$\bigstar$	0.17%/99.97%	2.16%/98.10%	0%/100%	0%/100%	9.59%/93.70%

• $\blacktriangle$: face recognition (FR), $\blacklozenge$: cPPG biometrics, $\bigstar$: rPPG biometrics, $*$: Training does not converge.

Table 2: Performance comparison between biometric methods including face recognition, cPPG biometrics, and rPPG biometrics. Note that de-identified videos proposed in the paper are used for face recognition and rPPG biometrics.

4.2.2 Comparison with other biometrics.

In Table 2, we compare rPPG biometrics with related biometric methods, including face and cPPG biometrics, when the signal length is 20 beats. For face recognition, we choose the highly cited face recognition method (FaceNet [35]) to prove how general face recognition works on de-identified facial videos. We use FaceNet to extract embeddings from de-identified images and train two fully connected layers on the embeddings to get the classification results. Table 2 demonstrates that FaceNet [35] fails to work on de-identified videos, indicating that there is no facial appearance information in the de-identified videos. Since our rPPG biometric method is privacy-preserving for facial appearances, we also compare our method with the recent privacy-preserving face recognition [13]. The results show that our method can achieve better performance than privacy-preserving face recognition [13]. Our rPPG biometric authentication completely gets rid of facial appearance while the privacy-preserving face recognition [13] only adds noises to partially remove facial appearances to guarantee face recognition performance, which may still have risks of privacy leakage. In addition, we also compare our method w/ rPPG-cPPG hybrid training to our method w/ rPPG training (only rPPG branch is used for training in Fig. 5, and the cPPG branch is disabled during training).

On the OBF dataset, ours w/ rPPG-cPPG hybrid training achieves similar intra-session performance to ours w/ rPPG training, but achieves the best cross-session performance. This means external cPPG datasets introducing morphology information can improve generalization, such as cross-session performance. Furthermore, our rPPG biometrics exhibits better performance than cPPG biometrics [11]. This is primarily because rPPG signals are extracted from both spatial and temporal representations, allowing for the utilization of more information compared to cPPG signals, which are measured from a single spatial point in the temporal dimension. However, this holds true only when the rPPG signals are of high quality.

On the UBFC-rPPG dataset, ours w/ rPPG-cPPG hybrid training achieves 100% AUC but ours w/ rPPG training does not converge. The reason might be that it is difficult for the model to learn rPPG morphology from the small-scale UBFC-rPPG dataset without the help of the external cPPG dataset. This suggests that the external cPPG dataset can help the model to learn discriminative rPPG morphology information. Moreover, the performance of cPPG biometrics is still lower than that of our rPPG biometrics.

On the PURE dataset, both rPPG and cPPG biometrics demonstrate good performance in intra-session testing. However, in cross-session testing, our rPPG biometrics are surpassed by cPPG biometrics. This is likely due to significant facial motions in the test videos, which negatively impact the quality of rPPG signals and morphology, as shown in Figure 6(c). On the other hand, cPPG signals measured from fingertips are less affected by facial motions, allowing for better performance in this scenario.

4.2.3 Results and discussions about rPPG morphology.

We also made an interesting finding that the rPPG-cPPG hybrid training can significantly improve rPPG morphology reconstruction. Table 3 shows the Pearson correlations between the mean periodic segments of GT cPPG and rPPG. High Pearson correlations mean rPPG morphology better resembles the corresponding GT cPPG. Note that our method does not require any GT cPPG for rPPG morphology reconstruction, so we choose unsupervised rPPG methods including POS [44], ICA [34], and [8] for comparison. Ours w/ rPPG-cPPG hybrid training achieves significantly higher Pearson correlation than the baseline methods, CP2D, and ours w/ rPPG training, as the external cPPG datasets introduce helpful morphology information via the hybrid training to refine the rPPG morphology. Such cPPG datasets are publicly available, and thus do not introduce extra costs of data collection.

Methods	Pearson Correlations$\uparrow$
POS [44]$*$	0.78
ICA [34]$*$	0.77
Gideon2021 [8]$*$	0.77
After the 1st training stage
CP2D	0.78
After the 1st and 2nd training stages
Ours w/ rPPG training	0.70
Ours w/ rPPG-cPPG hybrid training	0.87

Table 3: Pearson correlations between GT cPPG periodic segments and the rPPG periodic segments.

5 Conclusion

In this paper, we validated the feasibility of rPPG biometrics from facial videos. We proposed a two-stage training scheme and novel cPPG-rPPG hybrid training by using external cPPG biometric datasets to improve rPPG biometric authentication. Our method achieves good performance on both rPPG biometrics authentication and rPPG morphology reconstruction. In addition, our method uses de-identified facial videos for authentication, which can protect sensitive facial appearance information. Future work will focus on collecting a large-scale rPPG biometric dataset and studying influencing factors like temporal stability, lighting, and recording devices.