Research map / 3D vision / Face and shape
VINECS: Video-based Neural Character Skinning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns pose-dependent skinning weights for a fully rigged clothed human avatar solely from multi-view video, using a coordinate-based MLP (SkinNet) supervised via differentiable rendering with an auxiliary albedo/shadow appearance field.
Problem
Automatically rigging and skinning clothed human characters with pose-dependent skinning weights, without dense 4D scans or manual editing, using only multi-view video.
Approach
A coordinate-based MLP predicts pose-dependent skinning weights over canonical-space points, initialized from Pinocchio weights on a NeuS-reconstructed template, and supervised only by silhouette and rendering losses through an auxiliary pose- and view-dependent appearance field (AlbedoNet and ShadowNet).
Key results
- On D2, the full method reaches Chamfer 3.034 cm versus 3.760 for Pinocchio and 3.750 for SCANimate*, outperforming all skinning-only baselines.
- Compared to dense-point-cloud methods, the method is close to SNARF (2.591) and SCANimate (2.842) on D2 while using only multi-view video.
- Ablation on D2 shows static weights reach 3.354 Chamfer and removing the rendering loss gives 3.116, versus 3.034 for the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use pose-dependent skinning weights instead of initial static weights (ablation) | VINECS · D2 | Chamfer distance (cm): 3.761 -> 3.034 | Helped |
| use pose-dependent skinning weights instead of optimized static weights (ablation) | VINECS · D2 | Chamfer distance (cm): 3.354 -> 3.034 | Helped |
| add rendering loss (ablation) | VINECS · D2 | Chamfer distance (cm): 3.116 -> 3.034 | Helped |
| add shadow estimation to the albedo-only appearance field (ablation) | VINECS · D2 | Chamfer distance (cm): 3.047 -> 3.034 | Helped |
| use the learned pose- and view-dependent appearance field instead of static NeuS appearance (ablation) | VINECS · D2 | Chamfer distance (cm): 3.152 -> 3.034 | Helped |
| use learned albedo instead of NeuS albedo (ablation) | VINECS · D2 | Chamfer distance (cm): 3.137 -> 3.034 | Helped |
| predict pose-dependent skinning instead of per-vertex displacement (ablation) | VINECS · D2 | Chamfer distance (cm): 3.191 -> 3.034 | Helped |
| train using all reported views instead of 10 views (ablation) | VINECS · D2 | Chamfer distance (cm): 3.129 -> 3.034 | Helped |
| use VINECS instead of the strongest reported baseline | VINECS · D2 | Chamfer distance (cm): 2.591 -> 3.034 | Hurt |
| use VINECS instead of the strongest reported baseline | VINECS · D5 | Chamfer distance (cm): 4.320 -> 4.512 | Hurt |
| use VINECS instead of the strongest reported baseline | VINECS · V6 | Chamfer distance (cm): 2.760 -> 2.993 | Hurt |
Surprising findings
- Pose-dependent skinning from video alone is close in accuracy to SNARF and SCANimate, which require dense point clouds.
Limitations
- Queries SkinNet for every canonical vertex, which may be inefficient
- No control over facial expressions
- Residual artifacts since skinning alone cannot model high-frequency deformations
Experimental setup
Datasets: DynaCap D2 DynaCap D5 V6 (newly captured)
Benchmarks: D2 D5 V6
Metrics: Chamfer distance M2S S2M
Baselines: Pinocchio SCANimate SNARF RigNet
Builds on: Pinocchio NeuS SCANimate SNARF DynaCap
Closest papers by meaning
- Pose Modulated Avatars from Video (ICLR 2024 · similarity 0.88)
- 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting (CVPR 2024 · similarity 0.87)
- HumanRig: Learning Automatic Rigging for Humanoid Character in a Large Scale Dataset (CVPR 2025 · similarity 0.87)
- RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control (CVPR 2024 · similarity 0.86)
- FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images (CVPR 2025 · similarity 0.86)
- Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.