Research map / Multi-agent systems and games / LLM agents

Dyna-Mind: Learning to Simulate from Experience for Better AI Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dyna-Mind, a two-stage framework that teaches (V)LM agents to simulate environment dynamics in their reasoning: SFT on search-tree-derived reasoning traces (DFS-distill) followed by online RL with simulation-refinement rollouts (RLWM) built on GRPO.

Problem

Strong reasoning models excel at math and coding but perform poorly on long-horizon interactive tasks (web/phone use, text games) because they lack accurate simulation of environment dynamics; prior simulation-augmented methods rely on strong reasoning models to generate possibly erroneous synthetic

Approach

Build reasoning traces by aggregating real DFS search trees (rollouts with value estimates, grounded in true environment states) into a single response for SFT (DFS-distill), then refine online RL with simulation-refinement rollouts that feed real future states back to the policy, with a modified advantage rewarding refinements that beat standard rollouts.

Key results

• On ALFWorld ID, Distill(DFS)+RLWM reaches 92.5% success vs 62.5% for DeepSeek-R1 ReAct and 87.0% for GRPO on the same SFT init.
• On Sokoban AVG, Distill(DFS)+RLWM achieves 77.1% vs 73.1% for GRPO and 65.8% for Dyna-Think.
• On AndroidWorld AVG, Distill-32B(DFS)+RLWM reaches 31.8% vs 27.8% for GRPO and 19.5% for Qwen2.5-VL-72B ReAct.

Surprising findings

• DFS-distilled 7B model outputs about 11x fewer tokens than the R1-distilled model while matching or exceeding its performance.
• DeepSeek-R1 achieves near-perfect Sokoban success (96.6%) but only 62.5% on ALFWorld, attributed to poor environment-layout simulation.

Limitations

• DFS struggles on AndroidWorld, bottlenecked by the rollout model's GUI understanding and recovery from errors.
• DFS data construction is computationally expensive and depends on multiple modules (rollout model, value function, aggregator).
• Results on AndroidWorld are based on limited training steps (60) and small training sets due to compute cost.

Experimental setup

Datasets: Sokoban ALFWorld AndroidWorld

Benchmarks: Sokoban ALFWorld AndroidWorld

Metrics: success rate simulation score generated tokens per turn

Baselines: ReAct DeepSeek-R1 DeepSeek-V3 GPT-4o Claude-3.7-Sonnet Dyna-Think RLOO GRPO

Builds on: GRPO Dyna-Think Dyna algorithms ReAct Reflexion

Closest papers by meaning

• Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models (ICLR 2025 · similarity 0.88)
• Agentick: A Unified Benchmark for General Sequential Decision-Making Agents (NeurIPS 2026 · similarity 0.87)
• Eliciting Reasoning in Language Models with Cognitive Tools (NeurIPS 2025 · similarity 0.86)
• ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory (ICLR 2026 · similarity 0.86)
• Real-Time Reasoning Agents in Evolving Environments (ICLR 2026 · similarity 0.85)
• Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.85)