Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Rapidly Adapting Policies to the Real-World via Simulation-Guided Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Simulation-Guided Fine-Tuning (SGFT), which reshapes real-world RL fine-tuning with a frozen simulation-learned value function as a potential for dense rewards and short H-step horizons, instantiated with SAC and TD-MPC2 on five real Franka manipulation tasks.
Problem
Sim-to-real policy fine-tuning is sample-inefficient because standard RL fine-tuning uses unstructured exploration, and simulators are misspecified so zero-shot transfer fails on contact-rich dexterous manipulation.
Approach
Use a value function learned in simulation as a potential function for potential-based reward shaping and optimize only short H-step returns in the real world, so the simulator supplies where-to-go guidance while real data determines how-to-get-there; the frozen V_sim prevents catastrophic forgetting and keeps model-based rollouts short.
Key results
- SGFT reaches 100% success on hammering and pushing within an hour of real-world fine-tuning and 70% success on inserting within two hours.
- SGFT is reported to require up to an order of magnitude fewer real-world samples than baseline fine-tuning methods, and succeeds on tasks where prior approaches fail entirely.
- Zero-shot sim-to-real transfer performs poorly on all five tasks, and direct sim-to-real methods (domain randomization, ASID, DROPO, DOREAMON) also perform poorly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SGFT fine-tuning instead of standard fine-tuning methods | Simulation-pretrained policies · Real-world manipulation tasks | fine-tuning efficiency and asymptotic performance | Helped |
| Combine short model rollouts with value-driven reward shaping | Real-world fine-tuning · Real-world manipulation tasks | fine-tuning performance | Helped |
| Increase horizon H (ablation) | SGFT · Walker and Peg-in-Hole | performance | Mixed |
| Use SGFT instead of zero-shot sim-to-real transfer | Simulation-pretrained policies · Five real-world manipulation tasks | task success | Helped |
Surprising findings
- Simulation-data co-training during real-world fine-tuning did not help in the authors' settings and was dropped.
Limitations
- Dense reward computation relies on object state estimates (color thresholding or ArUco markers), so the method does not yet work directly from perceptual inputs.
- Tuning baselines and the method for real-world performance was time-consuming.
- Sample-efficiency gains rely on existing RL algorithms; new low-data adaptation algorithms may be needed.
Experimental setup
Datasets: simulation rollouts (random actions and SAC training) real-world fine-tuning rollouts on Franka FR3
Benchmarks: real-world hammering real-world insertion real-world puck pushing real-world deformable object pushing (towel, squishy ball) sim-to-sim Walker Walk sim-to-sim Cheetah Run sim-to-sim Rope Peg-in-Hole
Metrics: task success rate sample efficiency (real-world samples)
Baselines: SAC TDMPC-2 IQL PBRS RLPD Domain Randomization Recurrent Policy + Domain Randomization Asymmetric Actor-Critic DOREAMON ASID DROPO
Builds on: Potential-Based Reward Shaping SAC TD-MPC2 MPPI RLPD
Closest papers by meaning
- Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL (NeurIPS 2024 · similarity 0.89)
- ASID: Active Exploration for System Identification in Robotic Manipulation (ICLR 2024 · similarity 0.89)
- Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training (NeurIPS 2025 · similarity 0.89)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.88)
- Self-Improving Embodied Foundation Models (NeurIPS 2025 · similarity 0.87)
- DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.