Research map / Reinforcement learning / Policy gradient and actor-critic
Maximum Entropy Model Correction in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaxEnt Model Correction (MoCo), which corrects an approximate model's next-state distributions via minimum-KL density estimation constrained by a few true-dynamics queries, and builds MoCoVI and sample-based MoCoDyna on it, evaluated on a 6x6 grid world.
Problem
Planning with an approximate world model suffers from model error that can severely degrade value estimates; the paper asks whether next-state expectations from the true dynamics can be used to correct the model cheaply.
Approach
Minimally correct the approximate model's next-state distribution (min KL to the model) so that expectations of basis functions match queried true next-state expectations; iteratively use past value functions as basis functions (MoCoVI, MoCoDyna) to converge to the true value function even with model error.
Key results
- MoCoVI converges to the true value function in a few iterations even under extreme model error (lambda=1) on a 6x6 grid world, whereas VI and OS-VI converge much more slowly.
- MoCoDyna with d=2,3 significantly outperforms QLearning, Dyna, and OS-Dyna; Dyna cannot solve the task under model error.
- Theory shows the MaxEnt correction does not increase KL model error (Pythagorean inequality), and value error scales with approximation error of V by basis functions rather than ||V||.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoCoVI instead of VI | Value iteration · 6×6 grid world, model errors λ=0.1, 0.5, and 1 | normalized value error | Helped |
| Use MoCoVI instead of OS-VI | Value iteration · 6×6 grid world, model errors λ=0.1, 0.5, and 1 | normalized value error | Helped |
| Increase the number of basis functions d (ablation) | MoCoVI · 6×6 grid world | normalized value error | Helped |
| Use MoCoDyna instead of QLearning | Dyna-style learning · 6×6 grid world, model errors λ=0.1, 0.5, and 1 | convergence rate | Helped |
| Use MoCoDyna instead of OS-Dyna | Dyna-style learning · 6×6 grid world, model errors λ=0.1, 0.5, and 1 | convergence rate | Helped |
| Use MoCoDyna instead of Dyna | Dyna-style learning · 6×6 grid world, model errors λ=0.1, 0.5, and 1 | task performance | Helped |
| Use MoCoDyna with d=2 or d=3 (ablation) | Dyna-style learning · 6×6 grid world | task performance | Helped |
| Use the value function from the approximate model without correction | Planning with an approximate model · 6×6 grid world, model errors λ=0.1, 0.5, and 1 | normalized value error | Hurt |
Surprising findings
- Dyna fails to solve the task at all under model error, while MoCo-based methods converge even with extreme error.
Limitations
- Experiments limited to a small tabular 6x6 grid world; deep RL applications left to future work.
- Requires queries to the true dynamics for expectations of basis functions, which are assumed costly; regularization is needed under noisy queries.
- Choice of basis functions and regularization parameter beta affects performance and is not fully resolved.
Experimental setup
Datasets: 6x6 grid world (smoothed true dynamics)
Benchmarks: 6x6 grid world
Metrics: normalized L1 error of V_k against V*
Baselines: VI OS-VI pure MBRL (model-based value) QLearning Dyna OS-Dyna
Builds on: Dyna OS-VI Maximum entropy density estimation value-aware model learning
Closest papers by meaning
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.88)
- SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation (ICLR 2025 · similarity 0.87)
- When Maximum Entropy Misleads Policy Optimization (ICML 2025 · similarity 0.87)
- Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.86)
- State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.