Research map / Multi-agent systems and games / LLM agents

Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VIGIL, a no-feedback embodied benchmark (1,000 AI2-THOR episodes, eight task families) that separately scores world-state completion (W) and benchmark success with a correct terminal report (B), finding that models with similar W differ by up to 19.7 pp in B.

Problem

Standard embodied evaluations score only evaluator-side task success, conflating execution failures with failures to correctly commit to task completion at episode closure (terminal commitment).

Approach

Requires a mandatory semantic report at episode end, checked deterministically against hidden world state, and decouples world-state completion (W) from benchmark success (B) that additionally requires a correct report; adds closure-failure labels (FR, NR, IL) and an action-feedback intervention to test whether closure failures are explained by execution traps.

Key results

• Doubao-Seed-1.8 and InternVL3.5-38B have nearly identical W (40.4% vs 39.9%) but differ by 19.7 pp in B (37.1% vs 17.4%).
• Action feedback (too_far, path_blocked) raises B by about +13 pp for Gemini-3.1-Pro, Doubao-Seed-1.8, and Qwen3.5-27B, but gains are minimal for InternVL3.5-38B (+1.5 pp) and MiMo-Embodied-7B (+0.0 pp).
• Across 10 anchor models, 65-88% of false-success reports occur at zero task progress.

Surprising findings

• Across 10 anchor models, 65-88% of false-success reports occur at zero task progress, suggesting the report may act as a default exit action.
• Execution feedback improves world-state completion broadly but does not improve benchmark success for models whose reports are not coupled to achieved state (e.g., InternVL3.5-38B, MiMo-Embodied-7B).

Limitations

• Evaluated only in AI2-THOR with ProcTHOR houses; no photorealistic simulators or physical robots.
• The mandatory report protocol may itself elicit closure failures that a stop-only interface would not show.
• B=0 does not uniquely identify an internal cognitive cause; the decomposition is behavioral.

Experimental setup

Datasets: VIGIL 1,000-episode evaluation pack (LLM-proposed, simulator-validated, human-reviewed episodes in AI2-THOR/ProcTHOR sce

Benchmarks: VIGIL

Metrics: W (world-state completion) B (benchmark success) Delta = W - B FR rate NR rate IL rate mean belief lag

Baselines: Gemini-3.1-Pro Doubao-Seed-1.8 GPT-5.4 Claude-Sonnet-4 Qwen3.6-27B Qwen3.5-27B Qwen3-VL-32B InternVL3.5-38B MiMo-Embodied-7B

Builds on: ALFRED EmbodiedBench AI2-THOR ProcTHOR

Closest papers by meaning

• ECHO: Terminal Agents Learn World Models for Free (NeurIPS 2026 · similarity 0.86)
• Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors (NeurIPS 2026 · similarity 0.86)
• The Compliance Trap: Diagnosing How AI Agents Consume Conflicting Memory (NeurIPS 2026 · similarity 0.86)
• Automata from Agent Traces: Failure and Next-Step Prediction (NeurIPS 2026 · similarity 0.86)
• No More, No Less: Task Alignment in Terminal Agents (NeurIPS 2026 · similarity 0.85)
• OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories (NeurIPS 2026 · similarity 0.85)