Research map / Image generation / Autoregressive and latent generation
Behavior Generation with Latent Actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQ-BeT, which tokenizes continuous actions with a residual vector-quantized VAE and predicts code tokens with a GPT-style transformer, improving multimodal behavior generation over BeT and Diffusion Policy while running ~5x faster.
Problem
Generative modeling of multimodal, continuous, temporally correlated robot and driving behaviors from uncurated demonstrations; existing k-means action tokenization (BeT) does not scale to high-dimensional or chunked actions, and diffusion policies are slow multi-pass samplers.
Approach
Replace k-means action binning in BeT with a learned residual vector-quantized action tokenizer (primary and secondary codes) plus an observation-conditioned offset head, so a single-pass transformer predicts hierarchical action codes while retaining full action fidelity.
Key results
- VQ-BeT achieves the best unconditional results in 5 of 7 environments and the best goal-conditional results in 6 of 7 environments versus BeT, Diffusion Policy, C-BeT and BESO.
- On real-world two-phase tasks VQ-BeT outperforms all baselines by a relative margin of 73% (19/30 vs 11/30 for DiffusionPolicy-T).
- VQ-BeT inference is about 5x faster than Diffusion Policy in simulation (15.1 ms vs 100.5 ms single step) and about 25x faster on the real robot's onboard CPU (207 ms vs 5244 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VQ-BeT instead of the strongest reported baselines | goal-conditional behavior generation · Simulated manipulation and locomotion tasks | task performance | Mixed |
| Use VQ-BeT instead of the strongest reported baselines | unconditional behavior generation · Simulated manipulation and locomotion tasks | task performance | Mixed |
| Use VQ-BeT instead of the diffusion-based trajectory model | nuScenes trajectory planning · nuScenes | Avg. L2 (m): 0.96 -> 0.73 | Helped |
| Use VQ-BeT instead of DiffPol-T | real-world robot policy · Two-task sequences | total success: 11/30 -> 19/30 | Helped |
| Replace residual VQ with a single vanilla VQ layer (ablation) | VQ-BeT · Conditional Kitchen, unconditional Ant, and nuScenes | performance | Hurt |
| Set primary and secondary code loss weights equal (β = 1) (ablation) | VQ-BeT · Conditional Kitchen, unconditional Ant, and nuScenes | performance | Hurt |
| Predict VQ codes autoregressively (ablation) | VQ-BeT · Kitchen | performance | Hurt |
| Use autoregressive prediction of primary and secondary codes (ablation) | VQ-BeT · Real-world robot tasks | performance | Helped |
| Use action chunking (ablation) | VQ-BeT · Environments where action chunking was tested | performance | Hurt |
| Remove the offset prediction head (ablation) | VQ-BeT · Conditional Kitchen, unconditional Ant, and nuScenes | performance | Hurt |
| Increase residual VQ codebook size (ablation) | VQ-BeT · Simulation environments | task performance | Mixed |
| Mask code combinations absent from the dataset at sampling (ablation) | VQ-BeT with extended codebooks · Ant, PushT, and Kitchen | task performance | Mixed |
Surprising findings
- Receding-horizon (open-loop chunk) control of Diffusion Policy fails completely on the low-cost real robot (0/30), so all policies were run closed-loop.
- Extending the residual VQ codebook by 10 to 250 times in combinations had little impact on performance in most environments.
Limitations
- Real-world evaluation is limited to a low-cost mobile manipulator with a small number of tasks and 45 demos per task.
- Performance on BlockPush and Image Kitchen is not best; VQ-BeT is outperformed by DiffusionPolicy-T or C-BESO in some environments.
- Autonomous driving uses preprocessed object-level perception inputs (partial information), so collision avoidance is weaker without lane information.
Experimental setup
Datasets: Franka Kitchen PushT BlockPush UR3 BlockPush Multimodal Ant nuScenes GPT-Driver nuScenes processed dataset real-world kitchen robot demonstrations
Benchmarks: Franka Kitchen Image Kitchen PushT Image PushT BlockPush UR3 BlockPush Multimodal Ant nuScenes real-world robot kitchen tasks
Metrics: expected task completions final/max IoU coverage success rate behavior entropy L2 error collision rate inference time
Baselines: BC BeT Diffusion Policy (CNN) Diffusion Policy (Transformer) GCBC C-BeT BESO C-BESO CFG-BESO GPT-Driver UniAD VAD
Builds on: Behavior Transformer VQ-VAE Residual VQ Diffusion Policy GPT-Driver
Closest papers by meaning
- Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstraction (ICLR 2025 · similarity 0.86)
- VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model (ICML 2026 · similarity 0.85)
- Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.85)
- Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient (NeurIPS 2024 · similarity 0.85)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.85)
- Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.