Research map / Reinforcement learning / RL from feedback
Self-Improving Embodied Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage post-training recipe for embodied foundation models (PaLI-3B policies) that combines behavioral cloning plus steps-to-go prediction (SFT) with online REINFORCE self-improvement using steps-to-go-derived rewards and success detection.
Problem
Embodied foundation models for low-level robot control are trained almost exclusively with behavioral cloning, and reward engineering and success detection are hard for real-world robot RL post-training.
Approach
Use a steps-to-go prediction objective during SFT so the same model yields a dense reward (decrease in expected steps-to-go) and a success detector (steps-to-go below a threshold), enabling reward-engineering-free on-policy REINFORCE self-improvement of a VLM policy.
Key results
- Real-world LanguageTable: Self-Improvement lifts success from ~62-63% to ~87-88% with ~3% additional Block2Block episodes.
- Simulated LanguageTable: 10% data plus 1% self-improvement episodes outperforms BC on 20% and 80% data; success gains at least 1.5x across dataset sizes.
- BananaTable novel-skill transfer: success improves from ~63% to ~85% within ~8 hours of self-improvement on 2 robot stations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Self-Improvement after SFT | PaLI-3B policy · LanguageTable (real-world, 20% and 80% imitation data) | success rate: ~62-63% up to ~87%-88% | Helped |
| Add Self-Improvement with 10% additional robot time | LanguageTable policy · LanguageTable | success rate: 45% → 75% | Helped |
| Increase imitation data collection by 8× | LanguageTable behavioral-cloning policy · LanguageTable | success rate: 45% → 60% | Helped |
| Add Self-Improvement after SFT | PaLI-3B policy · LanguageTable (simulated, 10%, 20%, and 80% data) | success rate: minimum 1.5x performance boost | Helped |
| Add Self-Improvement using 1% additional episodes | PaLI-3B policy trained on 10% data · LanguageTable (simulated) | success rate: 1% additional episodes | Helped |
| Add Self-Improvement with 3% additional Block2Block episodes | PaLI-3B policy · LanguageTable (real-world, 20% and 80% data) | success rate: ~62-63% to ~87%-88% | Helped |
| Add Self-Improvement with 2.5K episodes | PaLI-3B policy trained on 5K imitation episodes · Aloha Single Insertion (simulated) | success rate: 5K (imitation) + 2.5K (Self-Improvement) episodes | Helped |
| Replace the PaLI reward model with a Uni-PaLI reward model (ablation) | PaLI-3B Self-Improvement policy · LanguageTable (simulated) | success rate | Hurt |
| Replace the PaLI reward model with a Scratch reward model (ablation) | PaLI-3B Self-Improvement policy · LanguageTable (simulated, low-data 10% and 20% regimes) | success rate | Hurt |
| Use a Scratch reward model instead of a PaLI reward model (ablation) | PaLI-3B Self-Improvement policy · LanguageTable (simulated) | success rate | Hurt |
| Replace the PaLI reward model with a Uni-PaLI reward model (ablation) | PaLI-3B Self-Improvement policy · LanguageTable (real-world to simulated transfer) | success rate | Hurt |
| Add Self-Improvement in the simulated target domain | PaLI-3B policy trained on real-world LanguageTable · LanguageTable (real-world to simulated transfer) | success rate: ~22% to ~59% | Helped |
Surprising findings
- Scratch and Uni-PaLI variants significantly underperformed PaLI in behavioral cloning, so ablations were run only at the self-improvement stage.
- Pushing self-improvement beyond its performance peak can degrade success rates.
Limitations
- Relies on on-policy REINFORCE without data reuse, which is sample-inefficient relative to off-policy methods
- Steps-to-go reward model may be unreliable for out-of-distribution failure states absent from the dataset
- Performance degrades if self-improvement runs past its peak, requiring better stopping criteria
Experimental setup
Datasets: LanguageTable dataset (181,020 trajectories) Aloha single insertion synthetic dataset (5K/10K/15K episodes) BananaTable online rollouts
Benchmarks: simulated LanguageTable real-world LanguageTable simulated Aloha single insertion BananaTable
Metrics: success rate sample efficiency (episodes)
Baselines: RT-2 (behavioral cloning) Uni-PaLI reward model Scratch reward model
Builds on: RT-2 PaLI REINFORCE LanguageTable
Closest papers by meaning
- SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents (NeurIPS 2025 · similarity 0.89)
- Universal Actions for Enhanced Embodied Foundation Models (CVPR 2025 · similarity 0.89)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.89)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.89)
- Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (NeurIPS 2026 · similarity 0.88)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.