Research map / Reinforcement learning / MDPs and planning
Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that in low-rank MDPs with sim-real dynamics mismatch, learning simulator-trained exploration policies (plus random exploration and least-squares regression) yields polynomial real-world sample complexity where direct sim2real transfer and naive exploration are exponentially inefficient, validated on tabular, simulated robot, and real Franka puck-pushing experiments.
Problem
Real-world RL sample complexity is high; direct sim2real policy transfer can fail when the simulator mismatches the real dynamics, and naive exploration with simple regression oracles is provably inefficient in low-rank MDPs.
Approach
Instead of transferring a policy that solves the task in simulation, learn a set of exploratory policies in simulation that cover the simulator's feature space, deploy them in the real world with random action perturbations after a prefix, and use least-squares regression on the collected real data; this provably yields polynomial sample complexity when direct transfer fails.
Key results
- On real Franka puck pushing over 6 runs, exploration policy transfer solved the task 6/6 times while direct sim2real transfer with finetuning solved it 0/6 times.
- In the Franka hammering sim2sim experiment, exploration policy transfer learned faster and reached a substantially higher final success rate than direct policy transfer.
- Theoretically, the method achieves polynomial sample complexity in low-rank MDPs when sim-real dynamics gap is at most about lambda_min/(64dHA^3), versus exponential in horizon for direct transfer and naive exploration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer exploratory policies from simulation instead of directly transferring the task policy | SAC finetuning on the Franka real-world puck-pushing task · Franka real-world puck pushing | successful trials: 0/6 -> 6/6 | Helped |
| Transfer exploratory policies instead of the task-solving policy from simulation | SAC on TychoEnv reach · TychoEnv sim2sim reach | task learning | Helped |
| Use exploratory-policy data instead of data from rolling out the task policy | SAC finetuning on Franka hammering · Franka simulator hammering | success rate | Helped |
| Transfer exploratory policies instead of direct sim2real task-policy transfer | Real-world puck-pushing task · Franka real-world puck pushing | reward | Helped |
| Transfer the simulator's task-solving policy and finetune it with naive exploration | Real-world puck-pushing task · Franka real-world puck pushing | task success | Hurt |
Surprising findings
- Direct sim2real transfer with finetuning converged to suboptimal solutions in all real-world runs, even with domain randomization in simulation.
- Naive exploration with a least-squares oracle and random exploration can be provably inefficient (exponential in horizon) even when a simulator is available, unless exploration is guided by transferred policies.
Limitations
- Theory restricted to low-rank/linear MDPs with dynamics-only sim-real mismatch and an assumed bound on TV distance epsilon_sim.
- Requires epsilon_sim below a threshold depending on dimension, horizon and reachability constant; practical instantiation uses heuristic diversity-based exploration without the theoretical guarantees.
- Real-world experiment is a single task on one robot platform with 6 runs per method.
Experimental setup
Datasets: simulated TychoEnv reach task custom MuJoCo Franka hammering simulator custom Franka puck-pushing simulator
Benchmarks: combination lock (tabular) TychoEnv sim2sim reach Franka hammering sim2sim Franka puck pushing real-world sim2real
Metrics: success rate return sample complexity
Baselines: direct sim2real transfer zeta-greedy Q-learning SAC from scratch SAC finetuning from sim policy
Builds on: low-rank MDP simulation lemma RND SAC zeta-greedy exploration
Closest papers by meaning
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer (NeurIPS 2025 · similarity 0.89)
- Rapidly Adapting Policies to the Real-World via Simulation-Guided Fine-Tuning (ICLR 2025 · similarity 0.89)
- ASID: Active Exploration for System Identification in Robotic Manipulation (ICLR 2024 · similarity 0.88)
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- EASI: Evolutionary Adversarial Simulator Identification for Sim-to-Real Transfer (NeurIPS 2024 · similarity 0.87)
- The Effective Horizon Explains Deep RL Performance in Stochastic Environments (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.