Research map / Reinforcement learning / Visual and representation RL
MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaxInfoRL, which augments Boltzmann (max-entropy) exploration with an information-gain intrinsic bonus, auto-tuning both temperatures, and applies it to SAC, DrQ and DrQv2 for state and visual control.
Problem
Off-policy deep RL relies on undirected exploration (epsilon-greedy or Boltzmann), which fails on hard exploration tasks with sparse rewards or action costs, while combining intrinsic and extrinsic rewards is difficult and task-dependent.
Approach
Boltzmann exploration's target distribution is augmented with an information-gain (upper bound via ensemble epistemic uncertainty) term, which is equivalent to maximizing soft value plus entropy over states, rewards and actions; the intrinsic temperature is auto-tuned with a constraint that the policy matches the target policy's information gain.
Key results
- MaxInfoSAC achieves the best performance across state-based tasks and solves the Pendulum swing-up within 10K environment steps where SAC does not cover the state space as fast.
- MaxInfoDrQ and MaxInfoDrQv2 reach higher rewards and better sample efficiency than DrQ and DrQv2 on visual DMC tasks, with MaxInfoDrQv2 reporting the highest returns on humanoid stand, walk and run tasks.
- The epsilon-greedy variant achieves sublinear regret in the multi-armed bandit (GP bandit) setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add information-gain exploration to SAC | SAC · State-based control tasks | performance | Helped |
| add information-gain exploration to SAC | SAC · Pendulum, CartPole, and Walker with action costs | reward | Helped |
| add information-gain exploration to DrQ | DrQ · DMC visual control tasks | reward | Helped |
| add information-gain exploration to DrQv2 | DrQv2 · DMC humanoid stand, walk, and run | performance | Helped |
| replace information gain with RND intrinsic reward (ablation) | MaxInfoSAC · State-based control tasks | performance | Hurt |
| use disagreement as the intrinsic reward in ε-MaxInfo (ablation) | SAC · State-based control tasks | performance | Helped |
| use curiosity instead of disagreement as ε-MaxInfo's intrinsic reward (ablation) | SAC · State-based control tasks | performance | Hurt |
| switch extrinsic and intrinsic policies every step instead of every 32 steps (ablation) | ε-MaxInfo with disagreement · Pendulum | performance | Hurt |
| use MaxInfoREDQ instead of REDQ (ablation) | REDQ · RL tasks | performance | Helped |
| use MaxInfoDrQv2 with no action noise instead of DrQv2 (ablation) | DrQv2 · DMC humanoid walk | reward | Helped |
| use MaxInfoDrQv2 with fixed action noise σ=0.2 or σ=0.0 (ablation) | DrQv2 · Additional DMC tasks | performance | Helped |
| add information-gain exploration to SAC | SAC · HumanoidBench stand, walk, and run | performance | Helped |
Surprising findings
- SACEipo quickly reduces its intrinsic reward weight to zero, making it overly greedy, whereas the auto-tuned MaxInfo coefficient does not decay.
- MaxInfoDrQv2 without action noise still outperforms DrQv2, suggesting information gain alone provides sufficient exploration.
Limitations
- Requires training an ensemble of forward dynamics models, adding computation overhead.
- Constraint for the information gain temperature requires maintaining a target policy.
- Theoretical sublinear regret is shown only for the bandit setting, not the full MDP.
Experimental setup
Datasets: DeepMind Control Suite OpenAI Gym HumanoidBench
Benchmarks: DeepMind Control Suite OpenAI Gym HumanoidBench Pendulum with action cost CartPole with action cost Walker with action cost
Metrics: episode return sample efficiency normalized performance cumulative regret (bandit theory)
Baselines: SAC REDQ DrQ DrQv2 Disagreement Curiosity SACIntrinsic SACEipo OAC
Builds on: SAC DrQ DrQv2 REDQ K-learning
Closest papers by meaning
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.89)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.89)
- Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL (NeurIPS 2025 · similarity 0.88)
- Random Latent Exploration for Deep Reinforcement Learning (ICML 2024 · similarity 0.88)
- CIG: Exploration via Conditional Information Gain (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.