Research map / Reinforcement learning / Visual and representation RL
ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReinboT, a GPT-style VLA that treats dense, decomposed returns-to-go as an extra modality and predicts maximized returns via expectile regression to improve imitation on mixed-quality robot data.
Problem
VLA models trained by imitation learning struggle with uneven quality in robot training data, and offline RL ideas (return maximization) have not been integrated into VLA models with widely applicable dense rewards.
Approach
Decompose trajectories into sub-goal segments and design a four-part dense reward (sub-goal achievement, task progress, behavior smoothness, task completion); then predict maximized ReturnToGo with an [RTG] token trained via expectile regression, and condition action prediction on the predicted return within a single inference pass, without an explicit RL loss or manual initial return.
Key results
- On CALVIN environment D with mixed-quality data, ReinboT (dense, full) reaches average length 2.26 versus 1.82 for RWR (dense, single) and 1.41 for GR-MG.
- Removing ReturnToGo entirely drops average length from 2.26 to 1.36 (-39.8%) in the ablation.
- Real-world UR5 experiments report that ReinboT outperforms GR-1 and RWR in few-shot and OOD settings, with exact numbers only shown in figures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ReinboT with dense full rewards instead of RWR with dense single rewards | CALVIN mixed-quality training and CALVIN D testing · CALVIN D | Avg. Length: 1.82 -> 2.26 | Helped |
| use dense rewards instead of sparse rewards (ablation) | ReinboT · CALVIN D | Avg. Length: 1.74 -> 1.90 | Helped |
| use dense rewards instead of sub-goal sparse rewards (ablation) | ReinboT · CALVIN D | Avg. Length: 1.80 -> 1.90 | Helped |
| predict each ReturnToGo component rather than a single scalar return (ablation) | ReinboT · CALVIN D | Avg. Length: 1.90 -> 2.26 | Helped |
| add ReturnToGo modality information (ablation) | ReinboT · CALVIN D | Avg. Length: 1.36 -> 2.26 | Helped |
| add sub-goal achievement reward r1 (ablation) | ReinboT · CALVIN D | Avg. Length: 1.87 -> 2.26 | Helped |
| add task progress reward r2 (ablation) | ReinboT · CALVIN D | Avg. Length: 1.79 -> 2.26 | Helped |
| add behavior smoothness reward r3 (ablation) | ReinboT · CALVIN D | Avg. Length: 1.90 -> 2.26 | Helped |
| add task completion reward r4 (ablation) | ReinboT · CALVIN D | Avg. Length: 1.93 -> 2.26 | Helped |
| increase expectile regression parameter m from 0.9 to 0.99 (ablation) | ReinboT · CALVIN D | Avg. Length | Hurt |
Surprising findings
- Without the ReturnToGo modality, performance drops to the level of plain GR-1 trained on annotated data (1.36 vs 1.41), suggesting imitation on mixed data can hurt.
- Setting expectile parameter m=0.99 yields a higher predicted return but lower performance than m=0.9, indicating over-optimistic return estimates hurt.
Limitations
- Expectile parameter m must be tuned; too large values cause overestimated returns and degraded performance.
- Real-world results are reported mainly in figures with limited trial counts described in text.
- Sub-goal decomposition relies on a heuristic (joint velocity near zero and gripper changes).
Experimental setup
Datasets: CALVIN Ego4d
Benchmarks: CALVIN
Metrics: success rate per instruction chain length (1-5) average length
Baselines: RoboFlamingo GR-1 PIDM GR-MG RWR Reinformer
Builds on: Decision Transformer Reinformer GR-1 RWR CVAE
Closest papers by meaning
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.88)
- Vision-Language Foundation Models as Effective Robot Imitators (ICLR 2024 · similarity 0.88)
- VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.