Research map / Multi-agent systems and games / LLM agents

RIG: Synergizing Reasoning and Imagination in End-to-End Generalist Policy

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RIG, an end-to-end autoregressive Transformer that jointly generates reasoning, low-level Minecraft actions, and imagined next frames, with a progressive data pipeline and lookahead "dream-review" reasoning at test time.

Problem

Embodied agents need both reasoning before acting and imagining action outcomes (world models), but existing systems either have only one ability or chain separate VLM and video generation models, limiting sample efficiency, generalization, and end-to-end optimization.

Approach

Train one autoregressive model to interleave text reasoning, discrete keyboard/mouse actions, and visual next-frame tokens, built via a progressive data pipeline (GPT-4o relabeling of reasoning and review of failed rollouts) so the agent can imagine several steps ahead and self-correct before acting.

Key results

• RIG-lookahead reaches 94.1% exploration accuracy in the Tool setting and 79.6% in Manual, versus 12.6% and 8.4% for the no-reasoning ablation.
• RIG-lookahead achieves FID 77.6 and PSNR 18.4 on generation, compared with MineDreamer and Oasis baselines.
• RIG trains on 111 hours of video, about 17x less than the roughly 2000 hours used by prior works such as VPT and STEVE-1.

Surprising findings

• Adding action learning alone slightly degraded FID (214.5 to 225.6) before reasoning was added, suggesting inconsistency without synergy.
• Tool-assisted exploration accuracy peaks at 3 lookahead steps and slightly decreases afterward, attributed to ceiling effects and prediction uncertainty.

Limitations

• Evaluated only in Minecraft; generalization to other open-world environments is not shown.
• Relies on GPT-4o for reasoning and review annotation, adding dependence on a closed API.
• Lookahead can accumulate prediction errors or hallucinations as the number of imagined steps grows.

Experimental setup

Datasets: MineRL-V0 STEVE-21K STEVE-1 generated trajectories

Benchmarks: Minecraft embodied tasks (collection and exploration) FVD/FID/PSNR image generation STEVE-21K QA

Metrics: number of collected samples success accuracy FID PSNR QA score

Baselines: VPT STEVE-1 Jarvis-1 MineDreamer Oasis GPT-4o

Builds on: Janus STEVE-1 VPT LlamaGen tokenizer SigLIP

Closest papers by meaning

• Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization (NeurIPS 2025 · similarity 0.88)
• Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.88)
• From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents (ICLR 2026 · similarity 0.88)
• VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents (NeurIPS 2025 · similarity 0.88)
• Open-World Reinforcement Learning over Long Short-Term Imagination (ICLR 2025 · similarity 0.87)
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (ICML 2025 · similarity 0.87)