Research map / Reinforcement learning / RL from feedback
CSO: Refining Robotic Policies via Skill Distribution Alignment and Skill-Grained Optimization
Zhiyuan Xiang, Xiang Deng, Wanjie Tao, Xu Chen, Zhao Jielun, Weili Guan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSO, a method for refining skill-based robotic manipulation policies by aligning the skill distribution and optimizing at the skill granularity to address quantization errors from VQ-VAE action discretization.
Problem
Discretizing continuous robot actions into skills with VQ-VAE introduces quantization errors that give the skill prior a suboptimal training distribution, degrading performance; direct RL refinement is unstable due to encoder updates.
Approach
Refines the skill prior via skill distribution alignment and skill-grained optimization, rather than directly applying RL through the unstable VQ-VAE encoder. Details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the full CSO two-stage post-training (R-SFT then SPO) | QueST-style policy · LIBERO-Long | success rate (%): 69.1 -> 96.0 | Improved |
| add the full CSO post-training | QueST-style policy · LIBERO overall | success rate (%): 80.1 -> 97.5 | Improved |
| add SPO after R-SFT (ablation) | CSO (QueST-style) · LIBERO-Long | success rate (%): 85.3 -> 96.0 | Improved |
| add R-SFT before SPO (ablation) | CSO (QueST-style) · LIBERO-Long | success rate (%): 91.7 -> 96.0 | Improved |
| freeze the observation encoder after R-SFT (ablation) | CSO (QueST-style) · LIBERO-Long | success rate (%): 89.6 -> 96.0 | Improved |
| use skill-level ratio (SPO) instead of token-level (GRPO) (ablation) | Stage-2 from R-SFT checkpoint · LIBERO-Long | success rate (%): 94.5 -> 96.0 | Improved |
| use skill-level ratio (SPO) instead of episode-level (GSPO) (ablation) | Stage-2 from R-SFT checkpoint · LIBERO-Long | success rate (%): 93.2 -> 96.0 | Improved |
| use critic-free SPO instead of a learned value critic (PPO) (ablation) | Stage-2 from R-SFT checkpoint · LIBERO-Long | success rate (%): 94.3 -> 96.0 | Improved |
| use a wider clipping range [0.97, 1.04] for GSPO (ablation) | GSPO · LIBERO-Long | success rate (%): 93.2 -> 86.4 | Hurt |
| alternate R-SFT and SPO iteratively instead of two stages (ablation) | CSO (QueST-style) · LIBERO-Long | success rate (%): 96.0 -> 96.3 | No clear effect |
| reduce R-SFT success buffer from 200 to 50 trajectories (ablation) | CSO (QueST-style) · LIBERO-Long | success rate (%): 96.0 -> 91.5 | Hurt |
| change skill length from 8 tokens to 4 or 16 (ablation) | CSO (QueST-style) · LIBERO-Long | success rate (%): 96.0 -> 93.5 (4 tokens), 96.0 -> 91.6 (16 tokens) | Hurt |
Experimental setup
Builds on: VQ-VAE
Closest papers by meaning
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.88)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.87)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.87)
- Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training (NeurIPS 2025 · similarity 0.87)
- Active Fine-Tuning of Multi-Task Policies (ICML 2025 · similarity 0.87)
- Rapidly Adapting Policies to the Real-World via Simulation-Guided Fine-Tuning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
