Research map / 3D vision / Face and shape
RigAnyFace: Scaling Neural Facial Mesh Auto-Rigging with Unlabeled Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RigAnyFace, a DiffusionNet-based FACS-conditioned facial auto-rigger that handles multi-component meshes and scales training with unrigged neutral meshes via 2D image, mask and optical-flow supervision.
Problem
Automatically generating animatable FACS blendshape rigs for facial meshes of diverse topologies, including meshes with multiple disconnected components (e.g., eyeballs, teeth), without a predefined template, while overcoming the scarcity of artist-rigged 3D ground truth.
Approach
Modify DiffusionNet with FACS-conditioned diffusion blocks and a global encoder that summarizes the whole mesh (including disconnected parts), and train on unrigged neutral meshes using differentiable-rendering-based image, mask and 2D displacement (optical-flow-like) losses supervised by a 2D face animation model (MegActor) and RAFT.
Key results
- Full model reaches MAE 1.92 mm and MAE Q95 5.63 mm on the rigged test set, improving over the ablation without unrigged data (2.01 / 5.81) and without global encoder (2.14 / 6.64).
- On 12 artist-annotated humanoid heads, RAF achieves MAE 1.01 mm vs. 2.77 mm for NFR and 2.93 mm for Deformation Transfer (which needs extra inputs).
- The model has only 5.4M parameters and runs inference in 3.1 s on a T4 GPU and 8.72 s on an Apple M2 Max CPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the global encoder (ablation) | RigAnyFace · Artist-crafted dataset, rigged test heads | MAE: 2.14 -> 1.92 | Helped |
| Add the global encoder (ablation) | RigAnyFace · Artist-crafted dataset, rigged test heads | MAE Q95: 6.64 -> 5.63 | Helped |
| Add the 2D image loss (ablation) | RigAnyFace · Artist-crafted dataset, rigged test heads | MAE: 2.08 -> 1.92 | Helped |
| Add the 2D image loss (ablation) | RigAnyFace · Artist-crafted dataset, rigged test heads | MAE Q95: 5.84 -> 5.63 | Helped |
| Train with unrigged heads (ablation) | RigAnyFace · Artist-crafted dataset, rigged test heads | MAE: 2.01 -> 1.92 | Helped |
| Train with unrigged heads (ablation) | RigAnyFace · Artist-crafted dataset, rigged test heads | MAE Q95: 5.81 -> 5.63 | Helped |
| Add 2D displacement loss (ablation) | RigAnyFace · Artist-crafted dataset, rigged test heads | MAE: 1.95 -> 1.92 | Helped |
| Add 2D displacement loss (ablation) | RigAnyFace · Artist-crafted dataset, rigged test heads | MAE Q95: 5.89 -> 5.63 | Helped |
| Use RigAnyFace instead of NFR | Artist-crafted dataset, 12 humanoid heads · Artist-crafted dataset, 12 humanoid heads | MAE: 2.77 -> 1.01 | Helped |
| Use RigAnyFace instead of Deformation Transfer | Artist-crafted dataset, 12 humanoid heads · Artist-crafted dataset, 12 humanoid heads | MAE: 2.93 -> 1.01 | Helped |
Surprising findings
- NFR leaves a non-humanoid head largely undeformed, while RAF generalizes to it.
- Without the global encoder, disconnected components such as teeth and gums penetrate the face surface during deformation.
Limitations
- Performance degrades for shell-like meshes lacking fine geometric detail that deviate from training data.
- Meshes with poor discretization that break into multiple disconnected components lose spatial coherence after deformation.
- Requires artist-rigged subset for 3D supervision; the 2D pipeline depends on fine-tuned 2D animation and flow models.
Experimental setup
Datasets: artist-crafted facial mesh dataset (161 rigged + 175 unrigged heads) ICT FaceKit Objaverse CGTrader
Benchmarks: artist-crafted rigged test set (24 heads, 96 FACS poses) artist-annotated humanoid heads (12) artist-crafted unrigged OOD test set (37 heads) in-the-wild ICT FaceKit, Objaverse, CGTrader samples
Metrics: MAE (mm) MAE Q95 (mm) penetration percentage
Baselines: NFR Deformation Transfer
Builds on: DiffusionNet MegActor RAFT NFR
Closest papers by meaning
- LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example (CVPR 2024 · similarity 0.87)
- ControlFace: Harnessing Facial Parametric Control for Face Rigging (CVPR 2025 · similarity 0.86)
- Implicit-ARAP: Efficient Handle-Guided Neural Field Deformation via Local Patch Meshing (NeurIPS 2025 · similarity 0.86)
- EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation (ICLR 2025 · similarity 0.85)
- Anatomically Constrained Implicit Face Models (CVPR 2024 · similarity 0.85)
- ARMesh: Autoregressive Mesh Generation via Next-Level-of-Detail Prediction (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.