Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Interpreting Emergent Planning in Model-Free Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows via concept probes, plan visualization, and activation interventions that a model-free DRC agent in Sokoban internally forms and evaluates plans resembling parallelized bidirectional search.
Problem
Whether model-free RL agents learn to perform decision-time planning internally, rather than merely producing planning-like behavior via learned heuristics.
Approach
Applies concept-based interpretability with linear probes for square-level planning concepts (agent approach direction, box push direction) to a DRC agent, then uses activation interventions and training-checkpoint analysis to test whether these internal representations are used for planning.
Key results
- Linear probes from the DRC agent's cell states decode agent approach direction and box push direction far above observation baselines, with 1x1 probes performing nearly as well as 3x3.
- Activation interventions succeed in 94.6% (layer 1) and 98.8% (layer 3) of Agent-Shortcut levels using trained probe vectors, versus about 28-34% for random probes.
- Probe macro F1 for plan concepts correlates strongly with the number of levels solved given extra test-time compute across training checkpoints.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use trained rather than randomly initialized probe representations for the Agent-Shortcut intervention at layer 1 | DRC · Agent-Shortcut | success rate: 33.7 ($\pm$32.7) -> 94.6 ($\pm$0.5) | Helped |
| Use trained rather than randomly initialized probe representations for the Agent-Shortcut intervention at layer 2 | DRC · Agent-Shortcut | success rate: 29.8 ($\pm$36.8) -> 90.1 ($\pm$1.9) | Helped |
| Use trained rather than randomly initialized probe representations for the Agent-Shortcut intervention at layer 3 | DRC · Agent-Shortcut | success rate: 27.8 ($\pm$37.9) -> 98.8 ($\pm$0.0) | Helped |
| Use trained rather than randomly initialized probe representations for the Box-Shortcut intervention at layer 1 | DRC · Box-Shortcut | success rate: 31.5 ($\pm$13.9) -> 56.2 ($\pm$1.4) | Helped |
| Use trained rather than randomly initialized probe representations for the Box-Shortcut intervention at layer 2 | DRC · Box-Shortcut | success rate: 30.9 ($\pm$25.8) -> 72.7 ($\pm$1.1) | Helped |
| Use trained rather than randomly initialized probe representations for the Box-Shortcut intervention at layer 3 | DRC · Box-Shortcut | success rate: 4.1 ($\pm$5.4) -> 80.6 ($\pm$2.4) | Helped |
Surprising findings
- The agent often forms complete plans in early internal ticks that are later revised, including plans that are infeasible before being discarded.
Limitations
- Analysis limited to a single DRC agent in Sokoban; generality to other agents and environments untested.
- Plans decoded by probes contain mistakes, especially for the agent's own movements.
- Interventions succeed less often for Box-Shortcut levels (56-81%).
Experimental setup
Datasets: Boxoban unfiltered training set Boxoban unfiltered validation set Boxoban medium-difficulty levels handcrafted Agent-Shortcut and Box-Shortcut levels
Benchmarks: Boxoban Agent-Shortcut levels Box-Shortcut levels
Metrics: macro F1 intervention success rate percentage of levels solved with extra test-time compute
Baselines: random-initialized probes observation-based baseline probes
Builds on: Deep Repeated ConvLSTM MuZero Concept-based interpretability Boxoban
Closest papers by meaning
- When Can Model-Free Reinforcement Learning be Enough for Thinking? (NeurIPS 2025 · similarity 0.88)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.88)
- Learning Planning Domains from Non-redundant Fully-Observed Traces: Theoretical Foundations and Complexity Analysis (AAAI 2024 · similarity 0.87)
- A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.87)
- Learning Planning Abstractions from Language (ICLR 2024 · similarity 0.86)
- PcLast: Discovering Plannable Continuous Latent States (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.