Research map / Probabilistic and scientific modeling / Molecules and proteins
STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
STAR learns discrete robot skill abstractions with rotation-augmented residual vector quantization (RaRSQ) to avoid codebook collapse, and a causal skill transformer (CST) that autoregressively predicts skill codes with offset refinement for action generation.
Problem
Discrete skill abstractions learned with VQ-VAE for robotic manipulation suffer from codebook collapse and fail to model causal dependencies between skills during composition for long-horizon tasks.
Approach
Replace the straight-through gradient in residual VQ with a rotation-based gradient mechanism that preserves relative angles between encoder outputs and codes, so points in the same code region receive geometry-aware updates; pair it with an autoregressive coarse-to-fine skill transformer plus BeT-style offset refinement.
Key results
- STAR reaches 93.6% overall success on LIBERO, versus 81.5% for the strongest baseline QueST (+12.1 points).
- On LIBERO-Long STAR scores 88.5% vs 69.1% for QueST; on LIBERO-Goal 95.0% vs 76.7%.
- Codebook utilization: RaRSQ uses all 16 codes while naive VQ-VAE uses 7 of 16 (43.8%); removing rotation drops overall performance to 91.0% and removing AR to 89.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use STAR instead of QueST | LIBERO · LIBERO overall | success rate: 81.5% -> 93.6% | Helped |
| add autoregressive skill prediction to CST (ablation) | STAR · LIBERO | overall success rate: 89.5% vs 93.6% | Helped |
| add rotation-augmented gradients to RaRSQ (ablation) | STAR · LIBERO | overall success rate: 91.0% vs 93.6% | Helped |
| add both rotation-augmented gradients and autoregressive prediction (ablation) | STAR · LIBERO | overall success rate: 87.8% vs 93.6% | Helped |
| add action refinement (ablation) | STAR · LIBERO | overall success rate: 59.0% vs 93.6% | Helped |
| use RaRSQ instead of naive VQ-VAE | skill codebook · skill codebook utilization | active code count: 7 out of 16 -> all 16 | Helped |
| use RaRSQ instead of standard VQ-VAE | skill quantization · LIBERO-Long | quantization loss: 0.0076 -> 0.00025 | Helped |
Surprising findings
- Removing the action refinement (offset) head drops LIBERO overall success from 93.6% to 59.0%, a much larger effect than the rotation or AR ablations.
Limitations
- Requires manually pre-defined codebook sizes and quantization depths for each domain.
- As an imitation learning method, performance depends on quality and quantity of expert demonstrations.
- Real-world evaluation is limited to two tasks with 10 trials each.
Experimental setup
Datasets: LIBERO MetaWorld MT50
Benchmarks: LIBERO-Object LIBERO-Spatial LIBERO-Goal LIBERO-Long LIBERO-90 MetaWorld MT50
Metrics: success rate codebook utilization quantization loss
Baselines: VQ-BeT QueST ResNet-T Diffusion Policy ACT Octo OpenVLA
Builds on: VQ-VAE residual vector quantization rotation trick Behavior Transformer (BeT) VQ-BeT
Closest papers by meaning
- QueST: Self-Supervised Skill Abstractions for Learning Continuous Control (NeurIPS 2024 · similarity 0.89)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.88)
- SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution (CVPR 2024 · similarity 0.86)
- Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.86)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.86)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.