Research map / Video / Video segmentation and tracking
Fusion-Vital: Video-RF Fusion Transformer for Advanced Remote Physiological Measurement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fusion-Vital, a two-branch RGB-RF transformer that fuses time-difference video and STFT-based RF inputs for remote respiration and heart-rate estimation, and releases a new video-RF rPPG dataset (MMD-rPPG).
Problem
Remote physiological measurement (respiration and heart rate) from single sensors (camera or RF radar) is hurt by illumination, occlusion, and global motion; the paper asks whether fusing RGB video and RF reflections improves vital-sign estimation.
Approach
Project RGB and RF into a shared time-difference (motion/Doppler) domain via pairwise inputs, encode each with parallel 2D/TSM encoders, and fuse them with a cross-attention transformer applied before every pooling layer at multiple temporal resolutions.
Key results
- Respiration MAE 0.44 BPM and RMSE 1.07 on RRM-static, versus 0.66 MAE for the best single-sensor RF-Vital baseline.
- Heart rate MAE 1.61 BPM and RMSE 3.05 on MMD-rPPG, versus 2.41 MAE for the best RGB-only MTTS-CAN.
- Under dark (RGB) or occluded (RF) conditions, Fusion-Vital keeps heart rate MAE near 2.4 to 2.7 BPM, while single-sensor models fail.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse RGB and RF modalities | Remote vital estimation · RRM-static respiration | MAE: 0.66 -> 0.44 | Helped |
| Fuse RGB and RF modalities | Remote vital estimation · MMD-rPPG heart rate | MAE: 2.19 -> 1.61 | Helped |
| Use paired time-difference inputs for both RGB and RF instead of time-domain inputs (ablation) | Fusion-Vital · MMD-rPPG heart rate | MAE: 1.75 -> 1.61 | Helped |
| Use time-difference input for RF instead of time-domain input (ablation) | Fusion-Vital · MMD-rPPG heart rate | MAE: 2.12 -> 1.61 | Helped |
| Use time-difference input for RGB instead of time-domain input (ablation) | Fusion-Vital · MMD-rPPG heart rate | MAE: 1.86 -> 1.61 | Helped |
| Use mismatched temporal input types across RGB and RF (ablation) | Fusion-Vital · MMD-rPPG heart rate | MAE: 1.61 -> 2.12; 1.61 -> 1.86 | Hurt |
| Use the proposed temporal fusion instead of Wang et al. fusion (ablation) | Fusion-Vital · MMD-rPPG heart rate | MAE: 1.99 -> 1.61 | Helped |
| Use the proposed temporal fusion instead of Prakash et al. fusion (ablation) | Fusion-Vital · MMD-rPPG heart rate | MAE: 1.87 -> 1.61 | Helped |
| Use the proposed temporal fusion instead of Li et al. fusion (ablation) | Fusion-Vital · MMD-rPPG heart rate | MAE: 1.71 -> 1.61 | Helped |
| Add sensor-wise projection to the fusion module (ablation) | Fusion-Vital · MMD-rPPG heart rate | MAE: 1.66 -> 1.61 | Helped |
| Use Fusion-Vital under dark conditions rather than relying on RGB-only input | Heart-rate estimation · MMD-rPPG dark | MAE: 2.35 -> 2.39 | Mixed |
| Use Fusion-Vital under occlusion rather than RGB-only input | Heart-rate estimation · MMD-rPPG occluded | MAE: 2.72 -> 2.68 | Helped |
Surprising findings
- Fusion-Vital trained with only one informative sensor still gives estimates comparable to the single-sensor counterparts when the other sensor is corrupted.
Limitations
- Dataset is small (15 participants for heart rate, 13 for respiration), limiting generalization.
- Respiration and heart-rate evaluation uses stationary or limited-motion settings and a single dataset pair.
Experimental setup
Datasets: RRM-static MMD-rPPG
Benchmarks: RRM-static MMD-rPPG
Metrics: MAE RMSE Pearson correlation standard deviation
Baselines: CAN Nowara et al. inverse attention MTTS-CAN Tu et al. Doppler radar Mercuri et al. 2019 Zheng et al. 2021 Ha et al. 2020 RF-Vital
Builds on: CAN (Chen and McDuff 2018) MTTS-CAN (Liu et al. 2020) RF-Vital (Choi, Kang, and Kim 2022) temporal shift module
Closest papers by meaning
- FacePhys: State of the Heart Learning (NeurIPS 2026 · similarity 0.85)
- PhysDiff: Physiology-based Dynamicity Disentangled Diffusion Model for Remote Physiological Measurement (AAAI 2025 · similarity 0.84)
- RhythmMamba: Fast, Lightweight, and Accurate Remote Physiological Measurement (AAAI 2025 · similarity 0.84)
- Remote Photoplethysmography in Real-World and Extreme Lighting Scenarios (CVPR 2025 · similarity 0.83)
- Multi-Modal Hand-to-Mouth Gesture Recognition in Activity-Oriented RGB-Thermal Footage (Student Abstract) (AAAI 2025 · similarity 0.82)
- FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.