Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Predicting Future Actions of Reinforcement Learning Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares predictability of future actions and events for explicit planning (MuZero, Thinker), implicit planning (DRC), and non-planning (IMPALA) RL agents in Sokoban, using inner-state and learned world-model simulation-based predictors.
Problem
Predicting future actions and events of deployed RL agents, which is hard because future states are unknown; the paper asks how informative inner computations and world-model simulations are across agent types.
Approach
Treat future action/event prediction as supervised learning from state-action pairs plus auxiliary information, either the agent's inner states (plans, hidden activations) or rollouts unrolled in a learned world model, and compare these across planning categories.
Key results
- For MuZero, action prediction accuracy with 50k training data rises from 40% (state-action baseline) to 87% when given internal plans.
- Simulation-based predictors for DRC and IMPALA perform strongly with an accurate world model, surpassing explicit-planning inner-state predictors.
- Under world-model ablations, inner-state predictions for explicit planning agents drop less in action accuracy than simulation-based predictions for the other agents; event prediction results are mixed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide the most-visited rollout as auxiliary information | MuZero · Sokoban | 5-action sequence accuracy: 40% -> 87% | Helped |
| Provide internal plans as auxiliary information | Explicit planning agents MuZero and Thinker · Sokoban | action prediction accuracy | Helped |
| Provide hidden states as auxiliary information | DRC and IMPALA · Sokoban | action prediction accuracy | Helped |
| Provide hidden states as auxiliary information | DRC and IMPALA · Sokoban | event prediction F1 | No clear effect |
| Use a learned-world-model rollout as auxiliary information | DRC and IMPALA simulation-based predictors · Sokoban | action prediction accuracy | Helped |
| Use a learned-world-model rollout as auxiliary information | DRC and IMPALA simulation-based predictors · Sokoban | event prediction F1 | Helped |
| Use an inaccurate world model instead of the default model | Explicit planning agents with inner states versus DRC and IMPALA with simulation rollouts · Sokoban world-model ablations | action prediction accuracy | Mixed |
| Use an inaccurate world model instead of the default model | Inner-state and simulation-based predictors · Sokoban world-model ablations | event prediction F1 | Mixed |
| Use the top 3 rollouts instead of only the top rollout (ablation) | MuZero inner-state predictor · Sokoban | event prediction accuracy | Hurt |
| Use only the hidden state at the last tick instead of hidden states at all ticks (ablation) | DRC inner-state predictor · Sokoban | prediction performance | No clear effect |
| Use outputs from all three residual blocks instead of only the last residual block (ablation) | IMPALA inner-state predictor · Sokoban | event prediction performance | Helped |
| Reduce world-model RNN channel sizes from 128 to 32 (ablation) | World model used by the predictors · Sokoban small-model ablation | action prediction accuracy | Mixed |
Surprising findings
- Inner states do not improve event prediction for DRC and IMPALA, possibly because the event-relevant blue tile does not affect reward and is ignored by the agent's representation.
- Simulation-based predictors for non-planning and implicit planning agents outperform explicit-planning inner-state predictors when the world model is accurate.
Limitations
- Evaluated only in a single environment (Sokoban).
- Only four RL agents/algorithms are considered.
- Assumes the policy is fixed and the world model is deterministic in the simulation setting.
Experimental setup
Datasets: Sokoban transitions generated by trained agents (50k train, 10k eval, 10k test)
Benchmarks: Sokoban
Metrics: accuracy (all five future actions correct) F1 score (event prediction)
Baselines: state-action only predictor
Builds on: MuZero Thinker DRC IMPALA
Closest papers by meaning
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling (NeurIPS 2026 · similarity 0.87)
- Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (ICLR 2025 · similarity 0.87)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.86)
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.86)
- Towards Provable Emergence of In-Context Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.