Research map / Multi-agent systems and games / LLM agents

Interactive Agents to Overcome Underspecificity in Software Engineering

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Ambig-SWE, an underspecified variant of SWE-Bench Verified with a simulated GPT-4o user proxy, to evaluate whether LLM coding agents detect underspecification, ask clarifying questions, and benefit from interaction.

Problem

LLM agents often act on underspecified instructions by making unwarranted assumptions instead of asking clarifying questions, which causes poor outcomes, tool-misuse risks, and wasted compute. The paper studies this in multi-step agentic software engineering tasks.

Approach

Creates paired fully specified and LLM-summarized (underspecified) SWE-Bench Verified issues and lets agents query a simulated user who holds the full issue, decomposing evaluation into detection, question quality, and leveraging interaction for resolution.

Key results

• Interaction improves resolve rates on underspecified issues by up to 74% over the non-interactive Hidden setting.
• Claude Sonnet 4 reaches 89% accuracy on underspecificity detection with strong encouragement; most other models struggle, and Qwen 3 Coder never interacts (100% FNR).
• Claude Sonnet 4 recovers 89% of full-specification performance via interaction, versus 54% for Llama 3.1 70B and 59% for Deepseek-v2.

Surprising findings

• Qwen 3 Coder's resolve rate worsens after receiving navigational (file location) information, indicating rigid protocol-following.
• Interaction improves effectiveness but not efficiency: Claude Sonnet 4 increases steps from 65 to 75 when interaction is enabled.

Limitations

• The simulated user proxy (GPT-4o) is likely more cooperative than real users.
• Underspecification detection is measured only within the first three turns.
• Question quality is approximated with embedding cosine distance that weighs all information equally.

Experimental setup

Datasets: SWE-Bench Verified

Benchmarks: Ambig-SWE SWE-Bench Verified

Metrics: resolve rate accuracy false positive rate false negative rate information gain (cosine distance) LLM-as-judge score

Baselines: Claude Sonnet 3.5 Claude Haiku 3.5 Deepseek-v2 Llama 3.1 70B Qwen 3 Coder

Builds on: SWE-Bench Verified OpenHands

Closest papers by meaning

• TOM-SWE: User Mental Modeling For Software Engineering Agents (ICML 2026 · similarity 0.88)
• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.86)
• daVinci-Dev: Agent-native Mid-training for Software Engineering (ICML 2026 · similarity 0.86)
• Discovering Hierarchical Software Engineering Agents via Bandit Optimization (ICLR 2026 · similarity 0.86)
• SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering (ICML 2025 · similarity 0.86)
• Kimi-Dev: Agentless Training as Skill Prior for SWE-agents (ICLR 2026 · similarity 0.86)