Research map / Vision-language and multimodal / Human and embodied modeling
Capturing the Unseen: Vision-Free Facial Motion Capture Using Inertial Measurement Units
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAPUS, a vision-free facial motion capture method that uses miniaturized IMUs on the face with a DiT-based network, plus IMU signal simulation from ICT Face Model meshes.
Problem
Facial motion capture typically relies on visual signals, which are sensitive to environmental conditions; IMU-based facial MoCap remains underexplored.
Approach
Attach miniaturized IMUs to the face, calibrate their signals into a unified local coordinate space using a neutral pose and an auxiliary IMU behind the ear to cancel head motion, and train a DiT-based network to infer facial motion; simulate facial IMU data from ICT Face Model meshes to augment training, though body-MoCap-style simulation was found not viable for faces.
Key results
- The DiT-based network achieved the lowest errors versus RNN and LSTM (PVE 0.060 mm, PVE_LMK 0.101 mm, MSE 0.00526 vs. 0.00665 for RNN and 0.00648 for LSTM).
- Training on simulated facial IMU data derived from body-MoCap-style simulation was reported as not viable for real captured data (qualitative comparison in Fig. 6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiT instead of LSTM | Facial IMU motion capture network · Network architecture comparison | PVE (mm): 0.066± 0.072 -> 0.060± 0.069 | Helped |
| Use DiT instead of LSTM | Facial IMU motion capture network · Network architecture comparison | PVE_LMK (mm): 0.112± 0.122 -> 0.101± 0.119 | Helped |
| Use DiT instead of LSTM | Facial IMU motion capture network · Network architecture comparison | MSE: 0.00648 -> 0.00526 | Helped |
| Train on simulated facial IMU data instead of captured IMU data (ablation) | CAPUS · Captured-data test set | Facial motion reconstruction accuracy | Hurt |
Surprising findings
- Body-IMU simulation approach does not transfer to facial IMU data.
Limitations
- Complete dataset and code are promised only after acceptance; submitted supplementary data is a sample.
- Simulated facial IMU data did not yield viable training for real captured signals.
Experimental setup
Datasets: ICT Face Model ARKit sequences (simulated facial IMU data) captured facial IMU data from participants
Benchmarks: captured facial IMU data (held-out participant split)
Metrics: PVE PVE_LMK MSE
Baselines: RNN LSTM
Builds on: DiT ICT Face Model IMU-based full-body motion capture
Closest papers by meaning
- MotionPRO: Exploring the Role of Pressure in Human MoCap and Beyond (CVPR 2025 · similarity 0.86)
- I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions (CVPR 2024 · similarity 0.85)
- Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera (CVPR 2024 · similarity 0.83)
- PTUS: Photo-Realistic Talking Upper-Body Synthesis via 3D-Aware Motion Decomposition Warping (AAAI 2024 · similarity 0.83)
- DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion (CVPR 2024 · similarity 0.83)
- X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.