Research map / Multi-agent systems and games / Human-AI and goal-conditioned

Learning to Explore in POMDPs with Informational Rewards

ICML 2024 · Multi-agent systems and games › Human-AI and goal-conditioned · approach: Intrinsically motivated exploration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PROBE, a POMDP agent that adds an information-gain exploration bonus, computed from privileged information (hidden state or future transitions) passed through an information bottleneck, to learn information-gathering strategies.

Problem

Exploration in partially observed MDPs where the agent must learn information-gathering strategies (e.g., checking signs or maps) when the hidden state can change within an episode, which standard RL and meta-RL methods handle poorly.

Approach

Extends DREAM's mutual-information exploration objective from meta-RL to general POMDPs by applying an information bottleneck to privileged information (hidden state or future trajectory) and adding a per-step information-gain bonus to task reward, so exploration targets only task-relevant information.

Key results

• PROBE matches or exceeds prior POMDP and meta-RL baselines on benchmark tasks, recovering the highest achievable return in all of them, with DREAM the only other method reaching optimal return across the three benchmark domains.
• On Non-stationary Map, PROBE reaches a final return of 0.41, while IMPORT and ELF achieve 0.33 and 0.29 by walking to the goal instead of using buses.
• PROBE (FUTURE), which uses future transitions as privileged information, performs similarly to PROBE on Tiger Door and slightly improves learning efficiency.

Measured effects reported in the paper

Surprising findings

• Plain IMPORT and ELF agents fail to gather information in Tiger Door and Light-Dark, instead guessing the task, while DREAM alone recovers optimal return across the benchmark domains.
• The exploration bonus can be used without any privileged information by relying on future transitions, performing comparably to the privileged-state version.

Limitations

• Only handles partially observable settings where new information must be gathered whenever the hidden state changes; predictable non-stationarity does not require exploration.
• Experiments focus on domains with finite hidden state spaces; larger or continuous state spaces are untested.
• Exploration weight alpha is sensitive and must be hand-chosen.

Experimental setup

Datasets: Tiger Door Light-Dark Map Non-stationary Map Overcooked Construction (Miniworld)

Benchmarks: Tiger Door Light-Dark Map Non-stationary Map Overcooked Construction

Metrics: average return final return

Baselines: Recurrent DQN policy IMPORT ELF DREAM

Builds on: DREAM IMPORT ELF informed POMDP

Closest papers by meaning

• Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.87)
• Exploration by Learning Diverse Skills through Successor State Representations (NeurIPS 2024 · similarity 0.86)
• Reinforcement Learning with Lookahead Information (NeurIPS 2024 · similarity 0.85)
• A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.85)
• Partially Observable Hierarchical Reinforcement Learning with AI Planning (Student Abstract) (AAAI 2024 · similarity 0.85)
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.85)