Research map / 3D vision / 3D generation
Puppeteer: Rig and Animate Your 3D Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Puppeteer automatically rigs 3D meshes (auto-regressive joint-token skeleton generation plus attention-based skinning with topology-aware joint attention) and animates them via differentiable optimization guided by reference videos from video generators, trained on an expanded 59.4k-model Articulation-XL2.0 dataset.
Problem
Turning static 3D models into animated assets requires expert-driven rigging (skeleton and skinning weights) and animation, and existing automatic rigging methods generalize poorly, are slow, or only handle rigging without animation.
Approach
Combines a joint-based tokenization with hierarchical ordering and randomized joint-group order for an auto-regressive skeleton transformer, a skeleton-graph-distance attention (TAJA) for skinning weights, and a training-free differentiable optimization that fits joint rotations and root motion to video tracks and rendering losses.
Key results
- Skeleton generation: Ours achieves CD-J2J 3.033 on Articulation-XL2.0 vs 3.264 for MagicArticulate and 3.305 for UniRig (x10^-2).
- Skinning weights: Ours reaches 0.335 L1 on Articulation-XL2.0 vs 0.451 for MagicArticulate, and 87.6% precision.
- User study with 21 participants preferred the proposed animation over L4GM and MotionDreamer (e.g., 83.33% motion quality vs 16.67% for L4GM).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diverse-pose training data | Puppeteer skeleton generation · Diverse-pose | CD-J2J (10^-2): 3.212 -> 2.514 | Helped |
| remove pose augmentation (ablation) | Puppeteer skeleton generation · Diverse-pose | CD-B2B (10^-2): 2.027 -> 3.629 | Hurt |
| remove order randomization (ablation) | Puppeteer skeleton generation · Articulation-XL2.0 | CD-B2B (10^-2): 1.923 -> 2.057 | Hurt |
| replace joint tokenization with bone tokenization (ablation) | Puppeteer skeleton generation · Articulation-XL2.0 | CD-B2B (10^-2): 1.923 -> 1.939 | No clear effect |
| replace hierarchical ordering with spatial ordering (ablation) | Puppeteer skeleton generation · Diverse-pose | CD-B2B (10^-2): 2.027 -> 2.295 | Hurt |
| add bone embeddings instead of joint embeddings (ablation) | Puppeteer skinning prediction · Diverse-pose | L1 error: 0.427 -> 0.405 | Helped |
| add topology-aware joint attention (TAJA) (ablation) | Puppeteer skinning prediction · Diverse-pose | L1 error: 0.414 -> 0.405 | Helped |
| add part-aware features (ablation) | Puppeteer skinning prediction · Diverse-pose | L1 error: 0.414 -> 0.405 | Helped |
| add pose augmentation (ablation) | Puppeteer skinning prediction · Diverse-pose | L1 error: 0.444 -> 0.405 | Helped |
| increase attention block depth from 1 to 2 (ablation) | Puppeteer skinning prediction · Articulation-XL2.0 | L1 error: 0.335 -> 0.316 | Helped |
| increase attention block depth from 1 to 2 (ablation) | Puppeteer skinning prediction · ModelsResource | L1 error: 0.443 -> 0.453 | Hurt |
| increase attention block depth from 1 to 3 (ablation) | Puppeteer skinning prediction · Articulation-XL2.0 | L1 error: 0.335 -> 0.317 | Helped |
Surprising findings
- Spatial ordering of joints, used by prior work, frequently yields disconnected skeletons, motivating hierarchical ordering.
- Training with a larger skinning network (depth 2 or 3) degraded performance on ModelsResource, attributed to orientation distribution differences.
Limitations
- Cannot capture fine-scale deformations (hair, cloth) because no skeleton joints are generated for those parts.
- Animation relies on per-scene optimization, preventing real-time deployment.
- Animation quality depends on the quality of the video generator and on single-view occlusion.
Experimental setup
Datasets: Articulation-XL2.0 Objaverse-XL Diffusion4D SMALR ModelsResource
Benchmarks: Articulation-XL2.0-test ModelsResource-test Diverse-pose subset
Metrics: CD-J2J CD-J2B CD-B2B precision recall L1 error average distance error inference time user study preference
Baselines: Pinocchio RigNet MagicArticulate UniRig GVB L4GM MotionDreamer
Builds on: MagicArticulate RigNet PartField OPT CoTracker3
Closest papers by meaning
- Rigel3D: Rig-aware Latents for Animation-Ready 3D Asset Generation (NeurIPS 2026 · similarity 0.92)
- HumanRig: Learning Automatic Rigging for Humanoid Character in a Large Scale Dataset (CVPR 2025 · similarity 0.88)
- MagicArticulate: Make Your 3D Models Articulation-Ready (CVPR 2025 · similarity 0.88)
- Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready 3D Characters (CVPR 2025 · similarity 0.87)
- TapMo: Shape-aware Motion Generation of Skeleton-free Characters (ICLR 2024 · similarity 0.85)
- DRiVE: Diffusion-based Rigging Empowers Generation of Versatile and Expressive Characters (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.