Research map / Large language models / Agents, code and math

AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions

NeurIPS 2025 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AbstentionBench, a 20-dataset benchmark of unanswerable and underspecified questions, and finds that frontier LLMs struggle to abstain and that reasoning fine-tuning degrades abstention.

Problem

LLMs need to recognize when a question is unanswerable, underspecified, false-premised, stale, or subjective and abstain, but abstention has been studied only in isolated settings without a holistic benchmark, especially for reasoning models.

Approach

Curate 20 datasets across six abstention scenarios (including new underspecified variants of GSM8K, GPQA, and MMLU-Math) and use an LLM judge (Llama 3.1 8B Instruct) to score abstention recall, then systematically compare reasoning fine-tuned models against their instruct counterparts, post-training stages, and test-time reasoning budgets.

Key results

• Abstention recall is low across frontier models; model scale has almost no effect on abstention (Llama 3.1 8B/70B/405B similar).
• Reasoning fine-tuned models show an average ~24% drop in abstention recall versus their non-reasoning counterparts (DeepSeek R1 Distill Llama 70B vs Llama 3.3 70B; s1.1 32B vs Qwen 2.5 32B).
• The abstention judge (Llama 3.1 8B Instruct) reached 0.88 accuracy against 300 human-annotated responses.

Measured effects reported in the paper

Surprising findings

• Reasoning fine-tuning hurts abstention even on math and science datasets where reasoning models were explicitly optimized.
• Increasing test-time reasoning budget improves accuracy but generally worsens abstention recall.

Limitations

• Only English datasets are covered and the scenario space is not exhaustive.
• Reliance on an LLM judge introduces noise into abstention and correctness scores.
• Possible leakage between evaluation data and post-training data (e.g., CoCoNot in Tulu 3), and no private test set.

Experimental setup

Datasets: ALCUNA BBQ BIG-Bench Known Unknowns BIG-Bench Disambiguate CoCoNot FalseQA FreshQA KUQ MediQ MoralChoice MuSiQue QAQA

Benchmarks: AbstentionBench

Metrics: abstention recall abstention precision abstention F1 response accuracy

Baselines: GPT-4o o1 Gemini 1.5 Pro Llama 3.1 Instruct Llama 3.3 70B Instruct Qwen 2.5 32B Instruct Mistral 7B Instruct OLMo 7B Instruct s1.1 32B DeepSeek R1 Distill Llama 70B

Builds on: CoCoNot Tulu 3 s1 DeepSeek R1

Closest papers by meaning

• Answering the Unanswerable Is to Err Knowingly: Analyzing and Mitigating Abstention Failures in Large Reasoning Models (AAAI 2026 · similarity 0.91)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.91)
• QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks? (NeurIPS 2025 · similarity 0.91)
• Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning (ICML 2026 · similarity 0.90)
• From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.90)
• Statistical Early Stopping for Reasoning Models (ICML 2026 · similarity 0.89)