Research map / Image generation / Autoregressive and latent generation
Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LCD, a hierarchical planner where a language-conditioned diffusion model (Temporal U-Net, T5-XXL conditioning) generates latent goal plans in a frozen HULC encoder space, executed by a HULC low-level policy, evaluated on CALVIN and CLEVR-Robot.
Problem
Scaling diffusion-based planning to high-dimensional pixel observations, long horizons, and language-specified novel tasks for generalist robot agents is computationally prohibitive and inaccurate in low-level control.
Approach
Run the diffusion planner in the low-dimensional latent space of a goal-conditioned low-level policy's encoder (32-d), subsample latent plans with temporal stride c, condition on frozen T5-XXL language embeddings, and use DDIM sampling, so that diffusion handles only high-level planning while a HULC LLP handles low-level control.
Key results
- On CALVIN MT-LHC, LCD achieves 88.7% success on horizon-one tasks and average horizon length 2.88, beating HULC (2.57) and SPIL (2.67).
- LCD inference time is 0.333 s versus 1.11 s for Diffuser-1D and 5.02 s for Diffuser-2D, reported as 3.3x-15x faster inference than diffusion baselines.
- On five held-out CALVIN tasks, LCD averages 61.00% success versus 12.67% for Diffuser-1D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LCD instead of SPIL | CALVIN MT-LHC · CALVIN MT-LHC, One | success rate: 84.6\pm0.6 -> 88.7\pm1.5 | Helped |
| Use LCD instead of SPIL | CALVIN MT-LHC · CALVIN MT-LHC, average horizon length | average horizon length: 2.67\pm0.01 -> 2.88\pm0.19 | Helped |
| Use LCD instead of the GCBC Transformer | CLEVR-Robot · CLEVR-Robot Seen | success rate: 45.7 \pm 2.5 -> 53.7 \pm 1.7 | Helped |
| Use LCD instead of the GCBC Transformer | CLEVR-Robot · CLEVR-Robot Unseen | success rate: 46.0 \pm 2.9 -> 54.0 \pm 5.3 | Helped |
| Use LCD instead of the GCBC Transformer | CLEVR-Robot · CLEVR-Robot Noisy | success rate: 42.7 \pm 1.7 -> 48.0 \pm 4.5 | Helped |
| Use LCD instead of Diffuser-1D | CALVIN held-out tasks · CALVIN held-out tasks | average success rate: 12.67\pm3.56 -> 61.00\pm7.79 | Helped |
| Use hierarchical LCD instead of Diffuser-1D | CALVIN inference · CALVIN inference | inference time (sec): 1.11 -> 0.336 | Helped |
| Use hierarchical LCD instead of Diffuser-2D | CALVIN inference · CALVIN inference | inference time (sec): 5.02 -> 0.336 | Helped |
| Replace the diffusion high-level policy with an MLP (ablation) | LCD · CALVIN MT-LHC average | average horizon length: 2.60 \pm 0.33 -> 2 . 8 8 \pm 0 . 1 9 | Hurt |
| Replace the diffusion high-level policy with a Transformer (ablation) | LCD · CALVIN MT-LHC average | average horizon length: 2.36 \pm 0.08 -> 2 . 8 8 \pm 0 . 1 9 | Hurt |
| Augment goal sampling with frame offset o (ablation) | LCD · CALVIN | performance | No clear effect |
| Increase the model dimensions (ablation) | LCD · CALVIN | overfitting | No clear effect |
Surprising findings
- Diffusion trained directly in CALVIN image space failed to converge after two weeks on 8 GPUs, motivating the latent-space hierarchy.
- MLP and Transformer high-level policies with ground-truth language underperform HULC, suggesting gains come from diffusion modeling of the goal space.
Limitations
- Inference is slower than HULC, which needs only a single forward pass.
- Relies on a pretrained hierarchical HULC LLP rather than a flat policy; authors note LCD could work with a flat policy but did not test it.
- Theoretical suboptimality bound assumes optimal expert data, which does not hold for CALVIN's suboptimal data.
Experimental setup
Datasets: CALVIN CLEVR-Robot
Benchmarks: CALVIN CLEVR-Robot
Metrics: success rate average horizon length inference time training wall-clock time model size
Baselines: GCBC MCIL HULC SPIL Diffuser-1D Diffuser-2D Hierarchical Transformer MLP high-level policy
Builds on: Diffuser HULC DDIM Imagen Hindsight relabelling
Closest papers by meaning
- Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks (ICML 2025 · similarity 0.89)
- Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.89)
- Controlling Large Language Model with Latent Action (ICML 2025 · similarity 0.89)
- ALPINE: Unveiling The Planning Capability of Autoregressive Learning in Language Models (NeurIPS 2024 · similarity 0.89)
- System 1.x: Learning to Balance Fast and Slow Planning with Language Models (ICLR 2025 · similarity 0.88)
- Non-myopic Generation of Language Models for Reasoning and Planning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.