Research map / Multi-agent systems and games / LLM agents

Implicit Intelligence - Evaluating Agents on What Users Don’t Say

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Implicit Intelligence benchmark (205 iOS-based scenarios across four categories) and Agent-as-a-World, an LLM-simulated YAML environment harness, finding the best of 16 models passes only 48.3% of scenarios.

Problem

Agentic benchmarks test explicit instruction-following with fully specified success criteria, not whether agents infer unstated requirements such as accessibility needs, privacy boundaries, catastrophic risks, and contextual constraints.

Approach

Scenarios are defined in single human-readable YAML files and simulated by a fixed LLM World Model that only executes pre-specified action returns, while hidden execution rules and rubrics stay hidden from the agent; scenarios are generated synthetically with iterative refinement against frontier models and filtered by a difficulty gate plus expert review.

Key results

• GPT-5.2-pro achieves the best scenario pass rate of 48.3% (NSS 72.7%) across 205 scenarios.
• Claude Opus 4.5 as World Model achieved 98.6% consistency; it scores 39.5% SPR as an evaluated agent.
• Best open-weight model DeepSeek V3.1 reaches 27.3% SPR, trailing GPT-5.2-pro by about 21 points.

Surprising findings

• Model progression within families is non-monotonic; GPT-5 outperforms its successors GPT-5.1 and GPT-5.2 on SPR.
• DeepSeek R1 underperforms the general-purpose DeepSeek V3.1 on this benchmark.

Limitations

• Scenarios reflect the authors' perspectives and may not capture cultural, age, and technical diversity of real user expectations.
• Benchmark validity may degrade as iOS features change with software updates.
• Action space limited to roughly 300 native iOS actions; third-party apps not covered.

Experimental setup

Datasets: iOS Actions Library PersonaHub

Benchmarks: Implicit Intelligence

Metrics: Scenario Pass Rate Normalized Scenario Score

Baselines: GPT-4.1 GPT-5 GPT-5.1 GPT-5.2 Claude Sonnet 4.5 Claude Opus 4.5 Gemini 3 Flash Gemini 3 Pro DeepSeek V3.1 DeepSeek R1 Llama 4 Maverick GPT-OSS-120B

Builds on: WebArena Tau-bench PersonaHub

Closest papers by meaning

• Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.88)
• $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (ICML 2026 · similarity 0.88)
• Infer Human’s Intentions Before Following Natural Language Instructions (AAAI 2025 · similarity 0.88)
• Bridging semantics and pragmatics in information-theoretic emergent communication (NeurIPS 2024 · similarity 0.87)
• $\tau$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge (ICML 2026 · similarity 0.86)
• Generalizing Verifiable Instruction Following (NeurIPS 2025 · similarity 0.86)