Research map / Video / Video generation
CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Combines an autoregressive text- and target-conditioned motion diffusion planner (DiP) with a PHC motion-tracking RL controller in a closed loop, fine-tuned with PPO across multiple physics tasks.
Problem
Kinematic text-to-motion diffusion models are slow, offline, and not grounded in physical scenes, while physics-based RL controllers struggle to scale to large motion data and multi-task object interaction.
Approach
Use a fast autoregressive diffusion model (DiP) conditioned on text and per-joint target locations as an on-the-fly planner that is closed-loop with a PHC tracking policy, and fine-tune the tracker with the diffusion planner in the loop across all tasks at once without task-specific reward engineering.
Key results
- CLoSD achieves success rates of 1.0 (goal reaching), 0.9 (object striking), 0.88 (sitting), and 0.98 (getting up), outperforming UniHSI on striking and getting up (0.02 and 0.08).
- DiP generates 40-frame plans at 3,500 fps (11.4 ms per 2 s of motion), about 175x real time, using 10 diffusion steps instead of MDM's 50.
- On HumanML3D, CLoSD reaches Top-3 R-precision 0.689 and FID 1.798 versus MoConVQ's 0.614 and 3.279, while having lower penetration, floating and skating than MoConVQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the tracking policy in the closed loop (ablation) | CLoSD · Object striking | success rate: 0.81 -> 0.9 | Helped |
| fine-tune the tracking policy in the closed loop (ablation) | CLoSD · Getting up | success rate: 0.53 -> 0.98 | Helped |
| replace open-loop planning with closed-loop replanning (ablation) | CLoSD · Sitting | success rate: 0.19 -> 0.88 | Helped |
| replace open-loop planning with closed-loop replanning (ablation) | CLoSD · Getting up | success rate: 0.23 -> 0.98 | Helped |
| use longer plans (ablation) | CLoSD · Sitting | success rate: 0.88 -> 0.56 | Hurt |
| use shorter plans (ablation) | CLoSD · Goal reaching | success rate: 1.0 -> 0.86 | Hurt |
| use shorter plans (ablation) | CLoSD · Getting up | success rate: 0.98 -> 0.95 | No clear effect |
| use closed-loop physics-based tracking rather than DiP alone (ablation) | CLoSD · HumanML3D | Skating [mm]: 629 · 10^{-3} -> 2 · 10^{-3} | Helped |
| use closed-loop physics-based tracking rather than DiP alone (ablation) | CLoSD · HumanML3D | R-precision top-3: 0.777 -> 0.689 | Hurt |
| use closed-loop physics-based tracking rather than DiP alone (ablation) | CLoSD · HumanML3D | FID: 0.283 -> 1.798 | Hurt |
| use a longer planning horizon (ablation) | DiP · HumanML3D | R-precision top-3: 0.586 -> 0.674 | Mixed |
| use a longer planning horizon (ablation) | DiP · HumanML3D | FID: 3.481 -> 1.671 | Mixed |
Surprising findings
- Open-loop execution of the same planner achieves 1.0 goal reaching but only 0.19 and 0.23 on sitting and getting up, showing closed-loop feedback is critical for interaction.
- The DiP-only kinematic planner has better text-to-motion metrics (FID 0.283) than the full physics-based CLoSD (FID 1.798), reflecting the expected cost of physical grounding.
Limitations
- Planning relies only on target locations and text, without exteroception such as vision or height maps.
- Focuses on mid- and low-level planning; longer-horizon planning is not addressed.
- Occasional artifacts from the diffusion plans may persist because the tracking horizon is short.
Experimental setup
Datasets: HumanML3D AMASS
Benchmarks: HumanML3D goal reaching object striking sitting down getting up
Metrics: success rate R-precision FID multi-modal distance diversity penetration floating skating runtime
Baselines: UniHSI AMP InterPhys MoConVQ MDM open-loop DiP CLoSD without fine-tuning
Builds on: MDM PHC AMP PPO DistilBERT
Closest papers by meaning
- One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation (ICML 2025 · similarity 0.85)
- Model-based Diffusion for Trajectory Optimization (NeurIPS 2024 · similarity 0.85)
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.85)
- DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control (ICLR 2025 · similarity 0.85)
- Improving Generative Behavior Cloning via Self-Guidance and Adaptive Chunking (NeurIPS 2025 · similarity 0.85)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.