Research map / Multi-agent systems and games / LLM agents

Agentic Abstention: Do Agents Know When to Stop Instead of Act?

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Agentic Abstention, a benchmark (~28k instances across WebShop, Terminal-Bench 2.0, and AbstentionBench-derived QA) evaluating when tool-using LLM agents should stop acting, and proposes CONVOLVE, a playbook-based context engineering method that improves timely abstention without weight updates.

Problem

LLM agents often fail to abstain when a task is infeasible or underspecified, either acting unnecessarily for many turns or never abstaining; existing abstention work is single-turn and agent work focuses on task completion.

Approach

Formulates agentic abstention as a sequential decision (answer, abstain, or act) with metrics for timely vs. overall abstention recall, and distills full interaction trajectories via reflection and curation into a reusable playbook of stopping rules appended to the agent's context.

Key results

• On WebShop, CONVOLVE raises Llama-3.3-70B timely recall (AbsRec@1) from 26.7 to 57.4 and overall recall (AbsRec@10) from 83.2 to 100.0 without parameter updates.
• Llama-3.3-70B achieves roughly 0.84 AbsRec@10 on WebShop, the highest among tested web models, while most others stay below 0.5 after 10 turns.
• All tested models achieve less than 40% average timely recall, and Codex CLI reaches about 0.38 AbsRec@10 versus 0.18 for Terminus 2 with the same GPT-5.4-mini base on terminal tasks.

Surprising findings

• Larger or more capable models sometimes abstain later; increasing model scale improves overall recall but not timely recall.
• Reasoning improves timely abstention but can reduce overall abstention recall on web tasks.

Limitations

• Environment-based tasks focus on missing targets or prerequisites; other forms of latent infeasibility are not covered.
• Benchmark built by adapting existing datasets and environments, with possible contamination and manual filtering.
• Findings restricted to evaluated models and scaffolds, a snapshot of current systems.

Experimental setup

Datasets: WebShop Terminal-Bench 2.0 AbstentionBench (16 datasets, 27,073 samples)

Benchmarks: Agentic Abstention (WebShop subset) Agentic Abstention (Terminal-Bench 2.0 subset) Agentic Abstention (AbstentionBench QA subset)

Metrics: AbsRec@1 (timely recall) AbsRec@10 (overall recall) SPL over-abstention rate

Baselines: Llama-3.3-70B base Llama-3.3-8B base In-context learning Terminus 2 Codex CLI

Builds on: ACE (Agentic Context Engineering) AbstentionBench WebShop Terminal-Bench 2.0 Reflexion

Closest papers by meaning

• Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents? (NeurIPS 2026 · similarity 0.88)
• Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning (ICML 2026 · similarity 0.88)
• Agent-Omit: Adaptive Context Omission for Efficient LLM Agents (ICML 2026 · similarity 0.88)
• LLM Agents Already Know When to Call Tools - Even Without Reasoning (NeurIPS 2026 · similarity 0.87)
• ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.87)
• AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.87)