Research map / Reinforcement learning / Visual and representation RL

Beyond Prediction: Steering VLM Agents with Retrospective World Modeling

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: RL for combinatorial routing · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Retrospective World Modeling, where a VLM agent infers the action that caused an observed state transition and uses a margin-based Self-Consistency Reward in PPO, improving success on Sokoban, Navigation, ManiSkill and FrozenLake over prospective-only VAGEN.

Problem

Prospective-only world modeling in VLM agents predicts what happens next but does not verify whether the chosen action is causally consistent with the observed transition, leading to physically incoherent behavior and sparse credit assignment in long-horizon tasks.

Approach

Augment the agent's reasoning with a retrospective step that scores candidate actions as explanations of the observed belief transition; convert the margin between the executed action's score and the strongest alternative into a dense intrinsic Self-Consistency Reward and optimize it with PPO alongside extrinsic rewards.

Key results

• RWM-Full reaches 0.81 overall average success rate versus 0.71 for VAGEN-Full and 0.76 for RWM-Base vs 0.56 for VAGEN-Base (Qwen2.5-VL-3B).
• RWM-Full scores 0.66 on Sokoban average and 0.63 on Sokoban-Hard, versus 0.48 and 0.33 for VAGEN-Full.
• RWM-Full (3B) outperforms GPT-5 (0.70) and GPT-4o (0.53) on the reported overall average.

Surprising findings

• Direct token matching as intrinsic reward saturates and encourages copying action history, a shortcut that history-agnostic inference is designed to block.
• Excessive retrospection weight (alpha=0.50) halves success rate, indicating reward hacking where the agent overfits to explaining transitions.

Limitations

• Causal ambiguity: many-to-one mapping between action sequences and state transitions in stochastic or multi-action turns can yield false negative intrinsic rewards.
• Scaling to extremely long-horizon tasks is limited by memory (OOM) in multi-turn RL training.
• Evaluation limited to four environments with a 3B backbone and max 4 turns.

Experimental setup

Datasets: Sokoban FrozenLake Navigation ManiSkill

Benchmarks: Sokoban Sokoban-Hard FrozenLake Navigation (Base, Common Sense) ManiSkill

Metrics: success rate action validness action effectiveness reward

Baselines: Vanilla-PPO ReAct-RL VAGEN-Base VAGEN-Full Qwen2.5-VL-72B Qwen2.5-VL-7B VLM-R1-3B GPT-5 GPT-4o o4-mini Claude 4.5 Sonnet Claude 3.7 Sonnet

Builds on: PPO GAE VAGEN ReAct

Closest papers by meaning

• VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents (NeurIPS 2025 · similarity 0.90)
• Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction–Reality Gaps (NeurIPS 2026 · similarity 0.89)
• WPT: World-to-Policy Transfer via Online World Model Distillation (CVPR 2026 · similarity 0.89)
• Simplifying Latent Dynamics with Softly State-Invariant World Models (NeurIPS 2024 · similarity 0.89)
• Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.89)
• Reinforcing VLAs in Task-Agnostic World Models (NeurIPS 2026 · similarity 0.88)