Research map / 3D vision / Face and shape
3D Face Tracking from 2D Video through Iterative Dense UV to Image Flow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlowFace, a two-stage monocular 3D face tracker that predicts dense per-vertex UV-to-image flow with a recurrent RAFT-style network trained on 3D scans, then fits FLAME with per-vertex deformations, and introduces the SSME screen-space motion metric on a Multiface benchmark.
Problem
Monocular 3D face tracking from 2D video is imprecise in capturing facial motion, and existing benchmarks rely on sparse keypoints or depth/reconstruction errors that ignore temporal motion accuracy.
Approach
Replace sparse landmarks and photometric losses with a dense, uncertainty-aware UV-to-image flow predicted by a Segformer encoder plus iterative RAFT-style update block, trained on high-quality 3D scan annotations, followed by a 3D fitting stage that combines FLAME, MICA neutral priors, and per-vertex deformations; also proposes the screen-space motion error (SSME) metric.
Key results
- On the Multiface benchmark, the method beats the best public method by 54% in face-region SSME for single images and 46% for sequences.
- On FaceScape, CD of 2.21 mm versus 3.56 mm for the best prior regression method (38% improvement).
- On speech-driven animation, augmenting CodeTalker training with generated data reduces VOCASET lip vertex error from 3.13e-5 to 2.85e-5 (8.8%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FlowFace instead of the strongest prior method | 3D face reconstruction · FaceScape | CD (mm): 3.67 -> 2.21 | Helped |
| use FlowFace instead of the strongest prior method | 3D face reconstruction · NoW Challenge, multi-view | Mean error (mm): 1.01 -> 0.88 | Helped |
| use FlowFace instead of TokenFace | 3D face reconstruction · NoW Challenge, single-view | Mean error (mm): 0.95 -> 1.07 | Hurt |
| use FlowFace instead of prior trackers | face tracking · Multiface, single image | SSME (px), face: 5.75 -> 2.58 | Helped |
| augment CodeTalker training with FlowFace-generated MEAD meshes | CodeTalker · VOCASET | lip vertex error: 3.13 × 10−5 -> 2.85 × 10−5 | Helped |
| replace the MiT-b5 backbone with MiT-b1 (ablation) | FlowFace 2D alignment network · Multiface | SSME: 2.58 -> 3.21 | Hurt |
| replace the MiT-b5 backbone with MiT-b2 (ablation) | FlowFace 2D alignment network · Multiface | SSME: 2.58 -> 2.78 | Hurt |
| use 1 update iteration instead of 3 (ablation) | FlowFace 2D alignment network · Multiface | CD: 1.18 -> 1.25 | Hurt |
| use 2 update iterations instead of 3 (ablation) | FlowFace 2D alignment network · Multiface | SSME: 2.58 -> 2.61 | Hurt |
| use 4 update iterations instead of 3 (ablation) | FlowFace 2D alignment network · Multiface | SSME: 2.58 -> 2.62 | Hurt |
| add the MICA neutral shape prior (ablation) | FlowFace 3D model fitting · NoW validation, single-view | Mean error (mm): 1.23 -> 1.02 | Helped |
| add per-vertex deformations (ablation) | FlowFace 3D model fitting · NoW validation, multi-view | Mean error (mm): 0.84 -> 0.83 | Helped |
Surprising findings
- MPT, a photometric optimization method, performs worse on motion error than some regression-based models, which the authors attribute to uniform lighting and texture in Multiface.
Limitations
- The two-stage pipeline is not fully differentiable, preventing end-to-end learning.
- Training data is limited to in-the-lab captures under controlled lighting.
- Relies on 3D scan annotations, which are costly to obtain.
Experimental setup
Datasets: FaceScape Stirling FaMoS Multiface NoW MEAD VOCASET
Benchmarks: Multiface tracking benchmark FaceScape NoW Challenge VOCASET INSTA dataset
Metrics: SSME chamfer distance NME median/mean/std scan-to-mesh error LPIPS lip vertex error
Baselines: DECA EMOCA HRN 3DDFAv2 PRNet SADRNet MPT MICA TokenFace Dense INSTA MGCNet
Builds on: RAFT Dense landmark alignment (Dense) FLAME MICA SegFormer
Closest papers by meaning
- DeepFaceFlow: In-the-Wild Dense 3D Facial Motion Estimation (CVPR 2020 · similarity 0.89)
- RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild (CVPR 2020 · similarity 0.86)
- A Generative Model for Depth-Based Robust 3D Facial Pose Tracking (CVPR 2017 · similarity 0.86)
- AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatars (NeurIPS 2022 · similarity 0.85)
- TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels (NeurIPS 2025 · similarity 0.84)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.