Research map / 3D vision / Face and shape
DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DICE, a Transformer-based end-to-end method that regresses hand-face meshes, contacts and face deformations from a single image in real time, trained with Decaf data plus 500 weakly-labeled in-the-wild images using depth and adversarial priors.
Problem
Recovering 3D hand-face interactions with facial deformations and contacts from a single RGB image, which prior optimization-based methods (Decaf) solve too slowly and generalize poorly beyond studio data that has 3D annotations.
Approach
A two-branch Transformer (MeshNet for global vertex positions, InteractionNet for local deformation and contact) followed by neural inverse-kinematics to get animatable FLAME/MANO parameters, plus a weakly-supervised scheme that uses 2D keypoints, Marigold depth of keypoints, and hand/face pose discriminators on in-the-wild images without 3D labels.
Key results
- DICE achieves PVE 8.32 on Decaf, a 7.5% reduction vs Decaf's 9.65, and MPJPE 9.95 and PAMPJPE 7.27.
- DICE runs at about 20 fps (0.049 s per image) on an RTX 4090, versus Decaf's 19.59 s per image on an A6000.
- DICE F-Score is 72.7 versus Decaf's 89.6, so it is lower than the optimization-based Decaf on overall plausibility, though best among regression methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Decaf with DICE | hand-face reconstruction · Decaf | PVE: 9.65 -> 8.32 | Helped |
| replace PIXIE-R with DICE | hand-face reconstruction · Decaf | PVE: 11.0 -> 8.32 | Helped |
| replace PIXIE-R with DICE | hand-face reconstruction · Decaf | F-Score: 72.0 -> 72.7 | Helped |
| replace Decaf with DICE | face contact estimation · Decaf face contacts | F-score: 0.57 -> 0.61 | Helped |
| replace Decaf with DICE | hand contact estimation · Decaf hand contacts | F-score: 0.47 -> 0.50 | Helped |
| add the two-branch architecture separating interaction estimation from mesh regression (ablation) | DICE · Decaf | PVE*: 9.29 -> 8.32 | Helped |
| add in-the-wild training data and weak supervision (ablation) | DICE · Decaf | PVE*: 8.93 -> 8.32 | Helped |
| add in-the-wild training data and weak supervision (ablation) | DICE · Decaf | F-Score: 73.3 -> 72.7 | Hurt |
| supervise the unrefined face and hand mesh features before inverse kinematics (ablation) | DICE · Decaf | PVE*: 12.2 -> 8.32 | Helped |
| add depth loss (ablation) | DICE · Decaf | PVE*: 15.6 -> 8.32 | Helped |
| add direct parameter supervision (ablation) | DICE · Decaf | F-Score: 64.7 -> 72.7 | Helped |
| add adversarial supervision (ablation) | DICE · Decaf | PVE*: 11.1 -> 8.32 | Helped |
Surprising findings
- Decaf attains higher overall physical plausibility (F-Score 89.6) than DICE (72.7), which the authors attribute to test-time optimization.
- Removing depth supervision, which is applied only to in-the-wild data, significantly degrades performance on the Decaf validation set (PVE 15.6 vs 8.32).
Limitations
- Lower F-Score than optimization-based Decaf, indicating weaker physical plausibility.
- Fails under extreme hand-face distance or complete occlusion, and degrades on out-of-distribution inputs such as gloves or infants.
- Only 500 in-the-wild images used; scaling to larger in-the-wild data is untested.
Experimental setup
Datasets: Decaf RenderMe-360 FreiHand in-the-wild internet images (500)
Benchmarks: Decaf
Metrics: PVE MPJPE PAMPJPE Collision Distance Non-Collision Ratio Touchness F-Score contact F-score running time
Baselines: Decaf Benchmark (Decaf baseline) PIXIE (whole-body) PIXIE (hand+face) PIXIE-R METRO FastMETRO
Builds on: Decaf METRO FastMETRO PIXIE Marigold
Closest papers by meaning
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.87)
- Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars (NeurIPS 2024 · similarity 0.86)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.86)
- PICO: Reconstructing 3D People In Contact with Objects (CVPR 2025 · similarity 0.85)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.85)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.