Research map / Large language models / Language, safety and interpretability

The Commit-Abstain Circuit: Why Language Models Hallucinate Instead of Abstaining

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Task vectors and steering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies a sparse Commit-Abstain Circuit of attention heads and MLPs via causal gating that drives unsupported commitment in LMs, and trains a lightweight MLP policy on CAC activations that improves abstention accuracy by 12.2 points over the zero-threshold margin.

Problem

Language models hallucinate by committing to answers on unanswerable inputs; it is unclear which internal components decide whether to commit or abstain, despite unanswerability signals being present in hidden states.

Approach

Defines a logit-based commit-abstain margin, decomposes it over attention heads and MLP sublayers, and adapts causal head gating with a margin-based objective (extended to MLPs) to localise a sparse circuit; commitment-promoting components accumulate early while abstention-promoting components act late and are insufficient to correct, motivating a CAC-feature abstention policy.

Key results

• The CAC-Informed MLP reaches mean accuracy 0.814 across 30 configurations, +12.2 points over the Zero-Threshold baseline, and reduces mean false abstention from 0.320 to 0.127 (2.5x lower).
• Sparse CAC covers 2.0% to 11.8% of components (median 5.2%) across ten models, with about 25% of CAC components being MLP sublayers.
• All 30 model-dataset AUROC values for the commit-abstain margin exceed chance (range 0.640 to 0.913, mean 0.811), yet 27% of unanswerable instances still have margin at or above zero.

Surprising findings

• Ablating commitment-promoting components also skews flips toward commitment (31.7% A-to-C vs 3.4% C-to-A), and drops AUROC, indicating they sustain the discriminative structure of the margin.
• Amplifying abstention-promoting components cannot eliminate false commitment even at extreme scaling, while commitment-promoting amplification nearly eliminates false abstention.

Limitations

• Scope restricted to unanswerable inputs in instruction-tuned models; incorrect answers to answerable questions are not addressed.
• Main analysis uses the pre-generation logit margin with a 17-token abstention set; generation-time dynamics only partially characterised.
• The CAC is a group-level, causally localised account rather than a complete functional characterisation.

Experimental setup

Datasets: KUQ SQuAD 2.0 MuSiQue

Benchmarks: KUQ SQuAD 2.0 MuSiQue HotpotQA SelfAware

Metrics: AUROC accuracy false abstention rate false commitment rate

Baselines: Zero-Threshold Non-CAC Random INSIDE Multi-LLM Semantic Entropy MERA HaloScope HaMI

Builds on: Causal Head Gating direct logit attribution HaMI

Closest papers by meaning

• Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer (NeurIPS 2026 · similarity 0.90)
• Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models (NeurIPS 2025 · similarity 0.89)
• Mitigating Hallucinations in Large Language Models via Causal Reasoning (AAAI 2026 · similarity 0.88)
• BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation (NeurIPS 2026 · similarity 0.88)
• LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (ICLR 2025 · similarity 0.87)
• InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaboration (AAAI 2026 · similarity 0.87)