Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ForageWorld, a partially observable foraging task built on Craftax, and uses neuroethology-inspired behavioral and neural analyses to show that model-free PPO-RNN agents display planning-like revisitation and allocentric position encoding without world models.
Problem
Standard DRL evaluation relies on reward curves and offers little insight into how agents behave and what internal representations support memory and planning in complex, partially observable open-ended tasks.
Approach
Applies behavior-first analysis tools from neuroscience and ethology (path analysis, GLM decision models, ridge-regression decoding of recurrent states, pruning and auxiliary-loss ablations) to PPO-GRU agents in a naturalistic foraging environment, and packages these into a reusable analysis framework.
Key results
- Replacing the 512-unit GRU with a feedforward network significantly impairs ForageWorld performance, indicating memory is required.
- Allocentric position is decodable from RNN hidden states up to roughly 50–100 timesteps into the past and future, while egocentric position remains at chance.
- Pruning to 90% sparsity did not degrade training performance and improved spatial decoding in some runs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace recurrent memory with a feedforward policy (ablation) | PPO-GRU in ForageWorld · ForageWorld | survival time | Hurt |
| Downsize the recurrent network to 64 units (ablation) | PPO-GRU in ForageWorld · ForageWorld | performance | Hurt |
| Use a pruned 128-unit network (ablation) | PPO-GRU in ForageWorld · ForageWorld | performance | Hurt |
| Prune the recurrent network (ablation) | PPO-GRU in ForageWorld · ForageWorld | training performance | No clear effect |
| Prune the recurrent network (ablation) | PPO-GRU in ForageWorld · ForageWorld | spatial decoding accuracy | Helped |
| Remove the auxiliary path integration loss (ablation) | PPO-RNN in ForageWorld · Large arenas | performance | Hurt |
| Remove the auxiliary path integration loss (ablation) | PPO-RNN in ForageWorld · Small arenas | performance | No clear effect |
| Remove the auxiliary path integration loss (ablation) | PPO-RNN in ForageWorld · ForageWorld | allocentric position decoding | Hurt |
| Limit perception to a forward-facing field of view (ablation) | PPO-RNN in ForageWorld · ForageWorld | early learning | Helped |
| Replace PPO-GRU with PQN-LSTM (ablation) | ForageWorld agents · ForageWorld | performance | No clear effect |
| Replace PPO-GRU with PQN-LSTM (ablation) | ForageWorld agents · ForageWorld | predator killing rate | Hurt |
| Remove recurrent memory (ablation) | PPO-GRU in ForageWorld · ForageWorld | survival time | Hurt |
Surprising findings
- Model-free RNN agents show revisitation and allocentric encoding typically associated with model-based planning, without world models.
- Removing the auxiliary path integration loss reduces performance in large arenas but not small ones, and eliminates above-chance position decoding.
Limitations
- Limited to 96x96 grid size from Craftax, so generalization to larger arenas is untested.
- Could not log precise locations of trees, rocks, and lakes, limiting landmark-based navigation analysis.
- Only PPO- and PQN-based architectures with shared components were analyzed; broader architectures were non-trivial to instrument.
Experimental setup
Datasets: ForageWorld (procedurally generated Craftax-based arenas)
Benchmarks: ForageWorld held-out test arenas
Metrics: survival time return RMSE of allocentric position decoding vs. average displaceme GLM coefficients for revisitation choice spatial entropy
Baselines: PPO without recurrence feedforward PPO PQN-LSTM 64-unit and 128-unit RNN variants
Builds on: Craftax PPO JaxPruner PQN
Closest papers by meaning
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.87)
- Intrinsic Goals for Autonomous Agents: Model-Based Exploration in Virtual Zebrafish Predicts Ethological Behavior and Whole-Brain Dynamics (NeurIPS 2025 · similarity 0.87)
- Deep Reinforcement Learning with Time-Scale Invariant Memory (AAAI 2025 · similarity 0.87)
- SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents (NeurIPS 2025 · similarity 0.86)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.86)
- Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.