Research map / Vision-language and multimodal / Robot action and navigation
What Can RL Bring to VLA Generalization? An Empirical Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically compares PPO, GRPO, DPO and SFT for fine-tuning OpenVLA on a ManiSkill pick-and-place benchmark with vision, semantic and execution OOD splits, and distills an efficient PPO recipe.
Problem
Understanding what generalization benefits RL fine-tuning gives vision-language-action models compared with supervised fine-tuning, which suffers from compounding errors under distribution shift.
Approach
Fine-tunes OpenVLA with online PPO (shared actor-critic backbone, warm-up from motion-planner demos, one PPO epoch) and evaluates on a new OOD benchmark across vision, semantics and execution, finding RL gains mainly in semantics and execution while matching SFT on vision.
Key results
- RL (PPO) reaches 0.938 in-distribution success vs 0.781 for SFT-16k, and on OOD object/table split RL is reported about 42.6% better than SFT.
- On execution splits, RL success on unseen robot initial pose is 0.797 vs 0.339 for SFT and on object-and-receptacle position 0.745 vs 0.286.
- Vision-robustness is comparable between RL and SFT (e.g., Texture-s success 0.630 RL vs 0.557 SFT), and PPO outperforms GRPO and DPO in the preliminary comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune OpenVLA with RL instead of SFT | OpenVLA · Unseen Objects | success rate: 0.453 -> 0.714 | Helped |
| Fine-tune OpenVLA with RL instead of SFT | OpenVLA · Unseen Robot Init Pose | success rate: 0.339 -> 0.797 | Helped |
| Fine-tune OpenVLA with RL instead of SFT | OpenVLA · Mid-Episode Object Reposition | success rate: 0.286 -> 0.745 | Helped |
| Fine-tune OpenVLA with RL instead of SFT | OpenVLA · Unseen Table | success rate: 0.719 -> 0.844 | Helped |
| Fine-tune OpenVLA with RL instead of SFT | OpenVLA · Open articulated objects Vision Avg. | Success Rate: 0.646 -> 0.591 | Hurt |
| Fine-tune OpenVLA with RL instead of SFT | OpenVLA · Real-world pick-and-place | Pick-and-Place Success Rate: 0.00 -> 0.27 | Helped |
| Use PPO rather than GRPO or DPO | OpenVLA fine-tuning · Pick-and-place | performance | Helped |
| Use the first action-token embedding h^0 as critic input rather than h^n or concatenated token embeddings (ablation) | PPO with OpenVLA · Pick-and-place training | returns | Helped |
| Use a separate actor-critic Transformer backbone rather than a shared backbone (ablation) | PPO with OpenVLA · Pick-and-place training | training speed: 35\,\% slower | Hurt |
| Use a separate actor-critic Transformer backbone rather than a shared backbone (ablation) | PPO with OpenVLA · Pick-and-place training | VRAM usage: 83\,\% more VRAM ($81.3$ GB vs. $44.4$ GB) | Hurt |
| Warm up OpenVLA with 140 demonstration trajectories (ablation) | PPO with OpenVLA · Pick-and-place training | environment steps to convergence: roughly 50\,\% fewer environment steps | Helped |
| Increase PPO epochs beyond one (ablation) | PPO with OpenVLA · Pick-and-place training | return | No clear effect |
Surprising findings
- Neither RL nor SFT gains visual robustness beyond the randomized training tables, so both perform similarly on vision splits.
- The original OpenVLA checkpoint performs poorly on the benchmark, so a warm-up with 140 motion-planner demonstrations was needed.
Limitations
- Demonstrations come only from a motion planner, not human data.
- Evaluation is limited to pick-and-place tasks in simulation.
- Sim-to-real evidence is preliminary (30 real-world trials on Franka Panda).
Experimental setup
Datasets: ManiSkill pick-and-place assets Objaverse Stable Diffusion/ControlNet table appearances MPLib motion planner demonstrations
Benchmarks: pick-and-place OOD benchmark (vision, semantics, execution splits) open articulated objects task
Metrics: success rate continuous grasp accuracy grasp accuracy relative performance drop
Baselines: SFT GRPO DPO (TPO) PPO-ORZ Octo-Small OpenVLA-OFT
Builds on: PPO GRPO DPO OpenVLA Open-Reasoner-Zero ManiSkill
Closest papers by meaning
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (ICML 2025 · similarity 0.90)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networks (CVPR 2025 · similarity 0.89)
- Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.89)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.