Research map / Reinforcement learning / Visual and representation RL
Domain Randomization via Entropy Maximization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DORAEMON, which maximizes the entropy of a Beta-parameterized domain randomization distribution under a success-rate constraint, enabling automatic DR for sim-to-real without real-world data.
Problem
Choosing the sampling distribution of dynamics parameters for domain randomization in sim-to-real RL is manual and tricky: too much randomization yields overly conservative policies, too little fails to generalize.
Approach
Formulate DR as a constrained problem that maximizes the differential entropy of the training distribution subject to the in-distribution probability of success staying above a threshold alpha, optimized via KL-trust-region updates reusing training episodes with importance sampling, plus a backup problem that backtracks the distribution when the constraint is violated.
Key results
- On sim-to-sim MuJoCo tasks, DORAEMON consistently achieves better and/or faster convergence of global success rate than AutoDR and LSDR over 10 seeds.
- DORAEMON tracks the desired in-distribution success rate alpha (e.g. 0.5, 0.75, 0.9) on Hopper, Walker2D and HalfCheetah, and maintains it across trust-region sizes thanks to the backup problem.
- On a 7-DoF PandaPush real-robot task with 17 randomized dynamics parameters, DORAEMON-trained policies transfer zero-shot, while Fixed-DR policies fail to learn meaningful behavior and LSDR scales poorly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DORAEMON instead of AutoDR | DORAEMON · Six MuJoCo sim-to-sim tasks | global success rate | Helped |
| Gradually widen the training distribution instead of training with maximum-entropy Fixed-DR | DORAEMON · Six MuJoCo sim-to-sim tasks | global success rate | Helped |
| Use DORAEMON instead of LSDR | DORAEMON · Six MuJoCo sim-to-sim tasks | training distribution entropy | Helped |
| Train on DORAEMON rather than Fixed-DR, AutoDR, or LSDR | PandaPush · PandaPush sim-to-real | successful transfer | Helped |
| Train with Fixed-DR across the full wide dynamics distribution | Fixed-DR · PandaPush | meaningful behavior | Hurt |
| Set the success-rate target to α=0.5 rather than higher values (ablation) | DORAEMON · Hopper, Walker2D, and HalfCheetah | global success rate | Helped |
| Replace Beta dynamics distributions with Gaussian distributions (ablation) | DORAEMON · MuJoCo sim-to-sim tasks | global success rate | No clear effect |
| Replace Beta dynamics distributions with Gaussian distributions (ablation) | DORAEMON · Dynamics-space boundaries | generalization toward boundary corners | Hurt |
| Remove the backup optimization problem (ablation) | DORAEMON · Hopper, Walker2D, and HalfCheetah | in-distribution success rate | Hurt |
| Add history-based policies to SPDL (ablation) | SPDL · MuJoCo sim-to-sim tasks | policy training performance | Mixed |
| Add asymmetric actor-critic to SPDL with history (ablation) | SPDL + History · MuJoCo sim-to-sim tasks | policy training performance | Helped |
| Condition the policy on dynamics parameters (SPDL Oracle) (ablation) | SPDL · MuJoCo sim-to-sim tasks | global success rate | Hurt |
Surprising findings
- Fixed-DR on the maximum-entropy uniform distribution underperforms DORAEMON even when DORAEMON eventually converges to that same distribution, attributed to an induced curriculum.
- Lower in-distribution success rate alpha (0.5) yielded better global generalization than higher alpha (0.9), attributed to higher entropy at convergence.
Limitations
- Backtracking may collapse to easy regions of the dynamics space and cause catastrophic forgetting of previously solved dynamics.
- Requires a binary success indicator defined per task, and a hyperparameter alpha and trust-region size epsilon.
- Parameter search space boundaries must be specified manually for the Beta parameterization.
Experimental setup
Datasets: OpenAI Gym MuJoCo environments (CartPole, SwingUpCartpole, Hopper, Walker2D, HalfCheetah, Swimmer) PandaPush simulated environment
Benchmarks: OpenAI Gym MuJoCo tasks (CartPole, SwingUpCartpole, Hopper, Walker2D, HalfCheetah, Swimmer) PandaPush sim-to-real
Metrics: global success rate distribution entropy in-distribution success rate average test return
Baselines: No-DR Fixed-DR LSDR AutoDR SPDL
Builds on: Soft Actor-Critic Automatic Domain Randomization LSDR Self-Paced Deep Learning (SPDL)
Closest papers by meaning
- Flow-based Domain Randomization for Learning and Sequencing Robotic Skills (ICML 2025 · similarity 0.91)
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.87)
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.87)
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer (NeurIPS 2025 · similarity 0.87)
- Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.86)
- GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.