Research map / Image generation / Editing and style transfer
ControlFace: Harnessing Facial Parametric Control for Face Rigging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ControlFace, a 3DMM-rendering-conditioned diffusion face rigging method using dual U-Nets (FaceNet + denoiser), a control mixer module, and reference control guidance, trained on CelebV-HQ video pairs to rig pose, expression and lighting without per-identity fine-tuning.
Problem
Face rigging (editing a face image to match pose, expression, or lighting controls) while preserving identity and fine details; existing 3DMM-conditioned diffusion methods rely on single-image datasets, forcing reconstruction training that ignores the control and requiring per-identity fine-tuning.
Approach
Train on paired reference/target frames from facial videos (CelebV-HQ) instead of reconstruction on single images; use a dual-branch U-Net where FaceNet injects detailed reference features via augmented self-attention, a control mixer module models correlations between target and reference 3DMM renderings, and reference control guidance replaces the null condition in CFG with the reference control
Key results
- Average DECA re-inference error of 4.85 across light, shape, expression, and pose, the lowest among compared methods (DiffusionRig 5.06, Arc2Face 10.77, GIF 8.91, CapHuman 10.40).
- FID 15.50 and LPIPS 0.1429 are the best in the comparison, while Arc2Face has higher ID (0.7825 vs 0.7586 for ControlFace).
- User study overall score 0.8486 vs 0.4545 (CapHuman) and 0.3473 (DiffusionRig).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CMM and RCG to FaceNet (ablation) | ControlFace · FFHQ | DECA re-inference error: 7.13 -> 4.85 | Helped |
| Add CMM to FaceNet (ablation) | ControlFace · FFHQ | DECA re-inference error: 7.13 -> 5.78 | Helped |
| Replace no guidance with RCG (ablation) | FaceNet + CMM · FFHQ | DECA re-inference error: 5.78 -> 4.85 | Helped |
| Replace RCG with CFG (ablation) | FaceNet + CMM · FFHQ | DECA re-inference error: 4.85 -> 5.77 | Hurt |
| Replace RCG with CFG* on the face controller input (ablation) | FaceNet + CMM · FFHQ | DECA re-inference error: 4.85 -> 5.75 | Hurt |
| Replace face controller with ControlNet (ablation) | ControlFace architecture without CMM · FFHQ | DECA re-inference error: 7.87 -> 8.37 | Hurt |
| Replace face controller with ControlNeXt (ablation) | ControlFace architecture without CMM · FFHQ | DECA re-inference error: 7.87 -> 8.06 | Hurt |
| Use the proposed method instead of DiffusionRig | FFHQ face rigging · FFHQ | DECA re-inference error (Average): 5.06 -> 4.85 | Helped |
Surprising findings
- Naive CFG with a null face-controller input gives negligible improvement, while the proposed reference-control guidance improves control adherence.
- Increasing guidance scale greatly worsens DiffusionRig's control adherence, while it improves RCG and other CFG variants to some extent.
Limitations
- Relies on DECA for 3DMM renderings, which may miss finer facial details.
- Trained only on a single facial video dataset (CelebV-HQ), limiting diversity of conditions and identities.
Experimental setup
Datasets: CelebV-HQ FFHQ
Benchmarks: FFHQ
Metrics: DECA re-inference RMSE ID cosine similarity FID LPIPS user study SC/PQ/OV
Baselines: GIF CapHuman Arc2Face DiffusionRig ControlNet ControlNeXt
Builds on: Stable Diffusion classifier-free guidance DECA FLAME ControlNet
Closest papers by meaning
- StyleRig: Rigging StyleGAN for 3D Control Over Portrait Images (CVPR 2020 · similarity 0.89)
- PortraitDirector: A Hierarchical Disentanglement Framework for Controllable and Real-time Facial Reenactment (CVPR 2026 · similarity 0.87)
- PerformRecast: Expression and Head Pose Disentanglement for Portrait Video Editing (CVPR 2026 · similarity 0.86)
- RigAnyFace: Scaling Neural Facial Mesh Auto-Rigging with Unlabeled Data (NeurIPS 2025 · similarity 0.86)
- Neural Implicit Morphing of Face Images (CVPR 2024 · similarity 0.85)
- Data Synthesis with Diverse Styles for Face Recognition via 3DMM-Guided Diffusion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.