Research map / Reinforcement learning / RL from feedback

Human-assisted Robotic Policy Refinement via Action Preference Optimization

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Action Preference Optimization (APO), a human-in-the-loop framework that fine-tunes autoregressive VLA models (OpenVLA, pi0-FAST) on human-corrected deployment trajectories using a KTO-style desirability loss with adaptive action-error reweighting.

Problem

Post-deployment refinement of Vision-Language-Action models is limited because they rely on offline expert demonstrations; learning from sub-optimal human-intervention trajectories with preference optimization is hard due to irreversible interaction (no paired positive/negative actions) and mismatch

Approach

Adapts prospect-theory-based binary desirability preference optimization (KTO-style) to VLA models, labeling the K steps before human intervention as undesirable and human-corrected actions as desirable, and uses normalized per-sample L1 action errors to reweight the importance coefficients so the discrete-token objective focuses on failure-prone or high-error actions.

Key results

• On four RoboMimic tasks, APO reaches a 48.0% mean success rate versus 40.5% for the base OpenVLA policy and 43.5% for the best baseline (KTO).
• APO achieves the top score on Coffee_D0 (60%), StackThree_D0 (54%, tied with TPO), ThreePieceAssembly_D0 (46%, tied with KTO), and Square_D0 (32%, tied with KTO).
• Generalization to pi0-FAST and real-world insertion tasks is reported as consistent gains over baselines, with 20 trials per method in the real world.

Measured effects reported in the paper

Surprising findings

• Standard behavior cloning fine-tuning on mixed expert and interaction data failed to outperform the base policy (DAgger 39.0%, Sirius 39.5% vs. base 40.5%).
• DPO with synthetic Gaussian-noise negatives performed worst among preference methods (37.0% mean), below the base policy.

Limitations

• Experiments only on autoregressive VLA models (OpenVLA, pi0-FAST); regression-based and diffusion policies not tested.
• Evaluation limited to RoboMimic simulation tasks and a small number of real-world tasks with 20 trials each.
• Reliance on decoding action tokens to continuous actions for reweighting; pi0-FAST decoding failures are handled by a heuristic weight of 1.

Experimental setup

Datasets: RoboMimic expert demonstrations human intervention trajectories collected via SpaceMouse

Benchmarks: RoboMimic Coffee_D0 RoboMimic StackThree_D0 RoboMimic ThreePieceAssembly_D0 RoboMimic Square_D0

Metrics: success rate

Baselines: Base policy DAgger Sirius DPO TPO KTO

Builds on: KTO DPO DAgger Sirius OpenVLA

Closest papers by meaning

• VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model (ICML 2026 · similarity 0.90)
• VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
• RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
• ActionUNet: Improving Robustness of VLA Models with Efficient Multi-scale Fine-tuning (NeurIPS 2026 · similarity 0.88)
• ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.88)
• PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.88)