Research map / Multi-agent systems and games / LLM agents
VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VAGEN, a multi-turn RL framework that trains VLM agents to reason explicitly about visual states (StateEstimation and TransitionModeling) with a turn-level world-modeling reward and Bi-Level GAE, lifting a 3B model to 0.82 on five agent tasks.
Problem
Training VLM agents in partially observable visual environments requires building internal world models, but existing multi-turn RL approaches lack explicit visual state reasoning and suffer from coarse credit assignment.
Approach
Structure the VLM's reasoning into state estimation and transition prediction, reward these predictions densely per turn with an LLM-as-a-judge world-modeling reward, and apply a two-level (turn then token) GAE so turn-level credit is propagated to action tokens.
Key results
- WorldModeling reasoning with VAGEN-Full on Qwen2.5-VL-3B reaches overall 0.82, versus 0.21 for the untrained model and 0.75 for GPT-5.
- WorldModeling strategy (0.76) outperforms FreeThink (0.67) and NoThink (0.28) under VAGEN-Base.
- Natural-language state representation is best for Sokoban and FrozenLake (0.61 and 0.71), while structured representation is slightly better for PrimitiveSkill (0.94 vs 0.91 average).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add WorldModeling reasoning that combines StateEstimation and TransitionModeling (ablation) | Qwen2.5-VL-3B trained with VAGEN-Base · Five tasks, Overall | score: 0.67 -> 0.76 | Helped |
| Add explicit reasoning rather than NoThink (ablation) | Qwen2.5-VL-3B trained with VAGEN-Base · Five tasks, Overall | score: 0.28 -> 0.76 | Helped |
| Add StateEstimation to TransitionModeling (ablation) | Qwen2.5-VL-3B trained with VAGEN-Base · Five tasks, Overall | score: 0.68 -> 0.76 | Helped |
| Add TransitionModeling to StateEstimation (ablation) | Qwen2.5-VL-3B trained with VAGEN-Base · Five tasks, Overall | score: 0.56 -> 0.76 | Helped |
| Add WorldModeling Reward and Bi-Level GAE (ablation) | Qwen2.5-VL-3B with VAGEN-Base · Five tasks, Overall | score: 0.76 -> 0.82 | Helped |
| Add WorldModeling Reward without Bi-Level GAE (ablation) | VAGEN-Base · Agent tasks | task performance | Helped |
| Add Bi-Level GAE without WorldModeling Reward (ablation) | VAGEN-Base · Agent tasks | task performance | Mixed |
| Use Natural-Language rather than Structured state representation (ablation) | WorldModeling on Sokoban and FrozenLake · Sokoban and FrozenLake | success rate: 0.28 -> 0.61; 0.63 -> 0.71 | Helped |
| Use Structured rather than Natural-Language state representation (ablation) | WorldModeling on PrimitiveSkill · PrimitiveSkill Average | success rate: 0.91 -> 0.94 | Helped |
| Use Symbolic rather than Natural-Language state representation (ablation) | WorldModeling on FrozenLake · FrozenLake | success rate: 0.71 -> 0.49 | Hurt |
Surprising findings
- Symbolic state representation performs worst among the three formats tested on Sokoban and FrozenLake.
- Bi-Level GAE agents show reward hacking, producing generic responses that satisfy the LLM judge without state-specific reasoning.
Limitations
- Restricted to the Qwen2.5-VL model family and a limited set of evaluation environments.
- Future work planned on supervised finetuning for multi-turn visual understanding; SFT not explored.
- Bi-Level GAE can induce reward over-optimization and judge hacking.
Experimental setup
Datasets: Sokoban FrozenLake Navigation PrimitiveSkill starvector/svg-icons-simple
Benchmarks: Sokoban FrozenLake Navigation PrimitiveSkill SVG Reconstruction
Metrics: success rate DreamSim similarity DINO similarity overall average score
Baselines: NoThink FreeThink Vanilla-PPO GRPO w/ Mask Turn-PPO w/ Mask GPT-5 o3 Gemini 2.5 Pro Claude 4.5 Sonnet Qwen2.5-VL-72B VLM-R1-3B
Builds on: PPO GRPO GAE DeepSeek-R1 verl
Closest papers by meaning
- Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs (CVPR 2026 · similarity 0.90)
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling (NeurIPS 2026 · similarity 0.90)
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.89)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.88)
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.