Research map / Large language models / Language, safety and interpretability

Absence Bench: Language Models Can’t See What’s Missing

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AbsenceBench, a benchmark asking LLMs to identify deliberately omitted lines from poetry, numerical sequences, and GitHub PR diffs, finding even top models reach only about 70% F1 and that explicit placeholders substantially help.

Problem

LLMs are strong at finding inserted or present information (needle-in-a-haystack), but it is unclear whether they can detect clearly omitted content given original and modified documents.

Approach

Frame absence detection as a controlled omission task: remove 10% of elements from a document and ask models to list the removed elements given both original and modified versions; hypothesize that Transformer attention cannot anchor on gaps, and test this by inserting placeholder tokens at omission sites.

Key results

• Gemini-2.5-flash (thinking) achieves the highest average micro F1 of 71.2%, and Claude-3.7-Sonnet (thinking) reaches 69.6% overall.
• Adding a '<missing line>' placeholder raises performance by an average of 41.9% across models, with Llama-4-Maverick improving 54.5% overall.
• Models average a 56.9% F1 drop on AbsenceBench compared with the insertion-based NIAH setting on poetry and GitHub PRs.

Measured effects reported in the paper

Surprising findings

• Lower omission rates lead to worse model performance, counter-intuitively.
• Inference-time compute gives only a modest 7.9% gain while generating about 8K extra thinking tokens.

Limitations

• Covers only surface-form deletions, not semantic absence.
• Structured task format provides both original and modified documents side by side, which may overestimate real-world performance.
• No error bars or significance testing; limited prompt variation.

Experimental setup

Datasets: Gutenberg Poetry Corpus GitHub pull requests from top 20 repositories Synthetic numerical sequences Harry Potter dataset (NIAH needles)

Benchmarks: AbsenceBench Needle in a Haystack

Metrics: micro F1-score exact match thinking token ratio

Baselines: Needle-in-a-haystack test setting o3-mini DeepSeek-R1 GPT-4.1 Llama-3.3-70B-Instruct Qwen2.5-72B-Instruct

Builds on: Needle in a Haystack

Closest papers by meaning

• What to Forget in Unlearning? Forget Set Curation for Language Models (NeurIPS 2026 · similarity 0.87)
• Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.87)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
• A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.87)
• A Controllable Examination for Long-Context Language Models (NeurIPS 2025 · similarity 0.87)
• What is Wrong with Perplexity for Long-context Language Modeling? (ICLR 2025 · similarity 0.86)