Research map / Video / Video generation
Articulated Kinematics Distillation from Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills articulated skeleton motions for rigged 3D Gaussian-splatting assets from CogVideoX-5B via video SDS, with ground rendering and physics-based motion tracking.
Problem
Text-to-4D generation with neural deformation fields has many DoFs, causing shape inconsistency and physically implausible motion (foot-skating, ground penetration) when animating 3D characters.
Approach
Optimize low-DoF joint-angle sequences of a rigged asset (Gaussian splatting with linear blend skinning) using video score distillation from a pretrained video diffusion transformer, adding checkerboard ground rendering, ground-penetration and smoothness regularizers, and optional differentiable physics tracking.
Key results
- On VideoPhy, AKD scores SA 0.81 vs TC4D 0.40 and PC 0.39 vs 0.31.
- User study (20 evaluators) prefers AKD in 51% (motion amount), 53% (physical plausibility), and 53% (text alignment) of comparisons, against TC4D.
- Ablation on VideoPhy: removing ground rendering drops SA to 0.777 and PC to 0.107 on the tiger example.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AKD instead of TC4D | Text-to-4D motion synthesis · VideoPhy | Semantic Adherence (SA): 0.40 ± 0.34 -> 0.81 ± 0.26 | Helped |
| use AKD instead of TC4D | Text-to-4D motion synthesis · VideoPhy | Physical Commonsense (PC): 0.31 ± 0.15 -> 0.39 ± 0.17 | Helped |
| use AKD instead of TC4D | Text-to-4D motion synthesis · User study | Motion Amount (MA) preference | No clear effect |
| use AKD instead of TC4D | Text-to-4D motion synthesis · User study | Physical Plausibility (PP) preference | No clear effect |
| use AKD instead of TC4D | Text-to-4D motion synthesis · User study | Text Alignment (TA) preference | No clear effect |
| remove ground rendering (ablation) | AKD on Ostrich · Ostrich | Semantic Adherence (SA): 0.982 -> 0.777 | Hurt |
| remove ground rendering (ablation) | AKD on Ostrich · Ostrich | Physical Commonsense (PC): 0.294 -> 0.107 | Hurt |
| remove ground penetration penalty (ablation) | AKD on Tiger · Tiger | Semantic Adherence (SA): 0.989 -> 0.979 | Hurt |
| remove ground penetration penalty (ablation) | AKD on Tiger · Tiger | Physical Commonsense (PC): 0.321 -> 0.269 | Hurt |
| remove smoothness loss (ablation) | AKD on Tiger · Tiger | Semantic Adherence (SA): 0.989 -> 0.984 | Hurt |
| remove smoothness loss (ablation) | AKD on Tiger · Tiger | Physical Commonsense (PC): 0.321 -> 0.294 | Hurt |
Surprising findings
- Ground-rendering ablation shows SA and PC both drop sharply without it, and user-study preferences over TC4D are only modestly above chance (51-53%).
Limitations
- Visual quality of generated assets remains suboptimal.
- Motion diversity depends on the video model's capabilities.
- Requires manually rigged skeletons; physics tracking can still float or slide relative to ground.
Experimental setup
Benchmarks: VideoPhy
Metrics: Semantic Adherence Physical Commonsense user preference (motion amount, physical plausibility, text
Baselines: TC4D
Builds on: Score Distillation Sampling PhysGaussian TC4D 3D Gaussian Splatting threestudio
Closest papers by meaning
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.86)
- CAGE: Controllable Articulation GEneration (CVPR 2024 · similarity 0.86)
- URDF-Anything+: End-to-End Generation for Simulation-Ready Articulated Assets (NeurIPS 2026 · similarity 0.86)
- Arbitrary Motion Style Transfer with Multi-condition Motion Latent Diffusion Model (CVPR 2024 · similarity 0.86)
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.86)
- TapMo: Shape-aware Motion Generation of Skeleton-free Characters (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.