Research map / Video / Video segmentation and tracking
FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FSAM, a nonnegative matrix factorization-based module that jointly computes spatial, temporal, and channel attention, and uses it in a 3D-CNN (FactorizePhys) and an EfficientPhys variant for cross-dataset remote photoplethysmography estimation.
Problem
End-to-end rPPG methods compute attention separately across spatial, temporal, and channel dimensions, which squeezes features and limits generalization of blood volume pulse estimation from raw video.
Approach
Factorized Self-Attention Module (FSAM) maps voxel embeddings so temporal features form the factorization vector dimension, applies rank-1 NMF (approximated with one-step gradient, no backprop through NMF) to get a low-rank reconstruction, and uses it as multidimensional attention with a residual connection.
Key results
- Trained on UBFC-rPPG and tested on PURE, FactorizePhys with FSAM reports MAE 0.48 ± 0.17 and SNR 14.16 dB, versus 7.78 for PhysNet trained on iBVP (cross-dataset results vary by setting).
- Trained on SCAMPS and tested on UBFC-rPPG, FactorizePhys with FSAM reports MAE 1.17 ± 0.40 versus 2.18 for EfficientPhys with SASN.
- Trained on iBVP and tested on PURE, FactorizePhys with FSAM reports MAE 1.66 vs 3.80 for EfficientPhys with SASN; within-dataset performance is at par with SOTA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FSAM to FactorizePhys (ablation) | FactorizePhys · UBFC-rPPG → PURE | MAE (HR): 1.37 ± 1.02 -> 0.71 ± 0.39 | Helped |
| Add a residual connection around FSAM (ablation) | FactorizePhys + FSAM · UBFC-rPPG → PURE | MAE (HR): 0.71 ± 0.39 -> 0.48 ± 0.17 | Helped |
| Map temporal features to M and channel-spatial features to N (ablation) | FactorizePhys + FSAM · UBFC-rPPG → PURE | MAE (HR): 0.77 ± 0.42 -> 0.48 ± 0.17 | Helped |
| Use rank 1 rather than rank 2 (ablation) | FactorizePhys + FSAM · UBFC-rPPG → PURE | MAE (HR): 1.40 -> 0.48 | Helped |
| Use rank 1 rather than rank 4 (ablation) | FactorizePhys + FSAM · UBFC-rPPG → PURE | MAE (HR): 2.25 -> 0.48 | Helped |
| Use rank 1 rather than rank 8 (ablation) | FactorizePhys + FSAM · UBFC-rPPG → PURE | MAE (HR): 1.44 -> 0.48 | Helped |
| Skip FSAM during inference after training with FSAM (ablation) | FactorizePhys + FSAM · UBFC-rPPG → PURE | MAE (HR): 0.48 ± 0.17 -> 0.48 ± 0.17 | No clear effect |
| Replace EfficientPhys SASN with FSAM | EfficientPhys · SCAMPS → PURE | MAE (HR): 6.21 ± 2.26 -> 8.03 ± 2.25 | Helped |
| Replace EfficientPhys SASN with FSAM | EfficientPhys · UBFC-rPPG → PURE | MAE (HR): 4.71 ± 1.79 -> 3.69 ± 1.66 | Helped |
| Use FactorizePhys + FSAM instead of EfficientPhys + SASN | rPPG estimation model · SCAMPS → iBVP | MAE (HR): 26.19 ± 3.47 -> 2.71 ± 0.54 | Helped |
| Use FactorizePhys + FSAM instead of PhysFormer | rPPG estimation model · PURE → UBFC-rPPG | MAE (HR): 1.01 ± 0.38 -> 1.04 ± 0.38 | No clear effect |
| Use FactorizePhys + FSAM instead of PhysNet | rPPG estimation model · UBFC-rPPG → iBVP | MAE (HR): 1.63 ± 0.33 -> 1.66 ± 0.30 | No clear effect |
Surprising findings
- FSAM can be dropped at inference for FactorizePhys without losing performance, reducing latency, whereas dropping it for the 2D EfficientPhys variant degraded results.
- Mapping temporal dimension to the factorization vector dimension (M) outperformed mapping channels or spatial features, and higher ranks than 1 performed no better than the baseline without FSAM.
Limitations
- Performance degrades under active head movements, occlusions, and changing ambient lighting.
- Evaluated only on rPPG; generalization to other spatial-temporal tasks (video understanding, tracking, segmentation) not tested.
- FactorizePhys has higher latency than EfficientPhys despite fewer parameters.
Experimental setup
Datasets: iBVP PURE UBFC-rPPG SCAMPS
Benchmarks: iBVP PURE UBFC-rPPG SCAMPS
Metrics: MAE (HR) RMSE (HR) MAPE (HR) Corr (HR) SNR (BVP) MACC (BVP)
Baselines: PhysNet PhysFormer EfficientPhys with SASN
Builds on: Hamburger EfficientPhys Temporal Shift Module rPPG-Toolbox PhysNet
Closest papers by meaning
- PhysDiff: Physiology-based Dynamicity Disentangled Diffusion Model for Remote Physiological Measurement (AAAI 2025 · similarity 0.88)
- RhythmMamba: Fast, Lightweight, and Accurate Remote Physiological Measurement (AAAI 2025 · similarity 0.87)
- Constraint Latent Space Matters: An Anti-anomalous Waveform Transformation Solution from Photoplethysmography to Arterial Blood Pressure (AAAI 2024 · similarity 0.87)
- FacePhys: State of the Heart Learning (NeurIPS 2026 · similarity 0.87)
- PaPaGei: Open Foundation Models for Optical Physiological Signals (ICLR 2025 · similarity 0.86)
- Remote Photoplethysmography in Real-World and Extreme Lighting Scenarios (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.