AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

CSO: Refining Robotic Policies via Skill Distribution Alignment and Skill-Grained Optimization

Zhiyuan Xiang, Xiang Deng, Wanjie Tao, Xu Chen, Zhao Jielun, Weili Guan

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CSO, a method for refining skill-based robotic manipulation policies by aligning the skill distribution and optimizing at the skill granularity to address quantization errors from VQ-VAE action discretization.

Problem

Discretizing continuous robot actions into skills with VQ-VAE introduces quantization errors that give the skill prior a suboptimal training distribution, degrading performance; direct RL refinement is unstable due to encoder updates.

Approach

Refines the skill prior via skill distribution alignment and skill-grained optimization, rather than directly applying RL through the unstable VQ-VAE encoder. Details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add the full CSO two-stage post-training (R-SFT then SPO)QueST-style policy · LIBERO-Longsuccess rate (%): 69.1 -> 96.0Improved
add the full CSO post-trainingQueST-style policy · LIBERO overallsuccess rate (%): 80.1 -> 97.5Improved
add SPO after R-SFT (ablation)CSO (QueST-style) · LIBERO-Longsuccess rate (%): 85.3 -> 96.0Improved
add R-SFT before SPO (ablation)CSO (QueST-style) · LIBERO-Longsuccess rate (%): 91.7 -> 96.0Improved
freeze the observation encoder after R-SFT (ablation)CSO (QueST-style) · LIBERO-Longsuccess rate (%): 89.6 -> 96.0Improved
use skill-level ratio (SPO) instead of token-level (GRPO) (ablation)Stage-2 from R-SFT checkpoint · LIBERO-Longsuccess rate (%): 94.5 -> 96.0Improved
use skill-level ratio (SPO) instead of episode-level (GSPO) (ablation)Stage-2 from R-SFT checkpoint · LIBERO-Longsuccess rate (%): 93.2 -> 96.0Improved
use critic-free SPO instead of a learned value critic (PPO) (ablation)Stage-2 from R-SFT checkpoint · LIBERO-Longsuccess rate (%): 94.3 -> 96.0Improved
use a wider clipping range [0.97, 1.04] for GSPO (ablation)GSPO · LIBERO-Longsuccess rate (%): 93.2 -> 86.4Hurt
alternate R-SFT and SPO iteratively instead of two stages (ablation)CSO (QueST-style) · LIBERO-Longsuccess rate (%): 96.0 -> 96.3No clear effect
reduce R-SFT success buffer from 200 to 50 trajectories (ablation)CSO (QueST-style) · LIBERO-Longsuccess rate (%): 96.0 -> 91.5Hurt
change skill length from 8 tokens to 4 or 16 (ablation)CSO (QueST-style) · LIBERO-Longsuccess rate (%): 96.0 -> 93.5 (4 tokens), 96.0 -> 91.6 (16 tokens)Hurt

Experimental setup

Builds on: VQ-VAE

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy