Research map / Multi-agent systems and games / LLM agents

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Forced-injection study measuring how the value of ground-truth clarification for underspecified long-horizon agent tasks varies with injection timing across goal, input, constraint, and context dimensions, plus a 300-session natural-ask analysis of when frontier models actually ask.

Problem

When instructions are incomplete, long-horizon agents must decide when to ask for clarification; prior work treats clarification as binary and does not measure how its value changes over the course of execution.

Approach

Inject ground-truth clarifications at controlled, oracle-calibrated trajectory points (10% to 90%) and measure pass@3 decay per information dimension, then compare to natural ask timing of frontier models to find timing gaps.

Key results

• Goal clarification at 10% of trajectory yields pass@3 0.78 on MCP-Atlas vs 0.40 no-clarification, falling to baseline by 70%.
• Input clarification retains value through roughly 50% of execution (0.46 at Inj-10 vs 0.33 NC on MCP-Atlas).
• No frontier model asks within the empirically optimal window; GPT-5.2 asks in 52% of sessions at mean 43%, Gemini 3 Flash never asks.

Surprising findings

• Deferring any clarification type past mid-trajectory degrades performance below never asking at all.
• Gemini 3 Flash never asks in 100 natural sessions, and GPT-5.2 over-asks yet achieves lower per-session success (3%) than Claude (11%).

Limitations

• Demand side only; does not address agents' ability to detect ambiguity and ask at the right time.
• Small per-cell sample sizes (n=5-31), with floor effects on TheAgentCompany.
• Forced injection disables ask_user while natural-ask enables it, creating a behavioral confound.

Experimental setup

Datasets: LHAW MCP-Atlas TheAgentCompany SWE-Bench Pro

Benchmarks: MCP-Atlas TheAgentCompany SWE-Bench Pro

Metrics: pass@3 wasted compute point of no return Kendall tau cross-model consistency ask rate mean ask timing

Baselines: no-clarification oracle prompt

Builds on: LHAW MCP-Atlas TheAgentCompany SWE-Bench Pro HumanEval pass@k

Closest papers by meaning

• Uncertainty-Aware Clarification in LLM Agents with Information Gain (ICML 2026 · similarity 0.90)
• Agentic Abstention: Do Agents Know When to Stop Instead of Act? (NeurIPS 2026 · similarity 0.88)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.87)
• ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents (NeurIPS 2026 · similarity 0.86)
• Selective Critique for Cost-Aware LLM Agents in Long-Horizon Decision Making (NeurIPS 2026 · similarity 0.86)
• Learning CLI Agents with Structured Action Credit under Selective Observation (NeurIPS 2026 · similarity 0.86)