Research map / Large language models / Language, safety and interpretability
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes modeling factual queries as constraint satisfaction problems and a linear probe (SAT Probe) on attention to constraint tokens that predicts factual errors in Llama-2 models, comparable to model confidence.
Problem
Understanding the internal mechanisms behind LLM factual errors, which prior mechanistic work (focused on correct recall) has scarcely examined, and predicting such errors without multiple generations.
Approach
Frame factual queries as CSPs with constraint tokens and show that attention to constraint tokens correlates with factual correctness; train a logistic-regression probe on constraint-token attention weights to predict per-constraint satisfaction, enabling fine-grained and early error prediction.
Key results
- On single-constraint datasets, SAT Probe attention features are comparable to Confidence; e.g., for Llama-2 7B basketball players AUROC 0.81 (SAT Probe) vs 0.82 (Confidence).
- The Combined predictor (attention plus confidence) gets the best results in most scenarios, e.g., AUROC 0.82 on 7B basketball players and 0.86 on 13B basketball players.
- Attention to constraint tokens predicts popularity of the constraining entity with Spearman rho >= 0.65 across all Llama-2 sizes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Probe attention weights to constraint tokens rather than use model confidence | Llama-2 7B factual-error prediction · CounterFact - Cit, CounterFact - MT | AUROC: 0.51 ± 0.01 -> 0.71 ± 0.01; 0.89 ± 0.00 -> 0.98 ± 0.00 | Helped |
| Probe attention weights to constraint tokens rather than use model confidence | Llama-2 7B factual-error prediction · CounterFact - HQ, Movies, Songs | AUROC: 0.77 ± 0.01 -> 0.75 ± 0.01; 0.90 ± 0.00 -> 0.80 ± 0.00; 0.93 ± 0.00 -> 0.88 ± 0.00 | Hurt |
| Probe attention weights to constraint tokens rather than use model confidence | Llama-2 7B factual-error prediction · Basketball Players, Football Teams | AUROC: 0.82 ± 0.00 -> 0.81 ± 0.00; 0.81 ± 0.00 -> 0.79 ± 0.00 | No clear effect |
| Probe attention weights to constraint tokens rather than use model confidence | Llama-2 13B factual-error prediction · CounterFact - Cit, CounterFact - MT | AUROC: 0.58 ± 0.00 -> 0.72 ± 0.01; 0.80 ± 0.00 -> 0.96 ± 0.00 | Helped |
| Probe attention weights to constraint tokens rather than use model confidence | Llama-2 13B factual-error prediction · Basketball Players, CounterFact - HQ, Football Teams, Movies, Songs | AUROC: 0.86 ± 0.00 -> 0.85 ± 0.00; 0.77 ± 0.00 -> 0.70 ± 0.01; 0.83 ± 0.00 -> 0.81 ± 0.00; 0.87 ± 0.00 -> 0.79 ± 0.00; 0.93 ± 0 | Hurt |
| Use attention contribution features instead of attention weights (ablation) | SAT Probe, Llama-2 7B · CounterFact - Cit | AUROC: 0.71 ± 0.01 -> 0.76 ± 0.01 | Helped |
| Use attention contribution features instead of attention weights (ablation) | SAT Probe, Llama-2 7B · CounterFact - HQ, Movies | AUROC: 0.75 ± 0.01 -> 0.76 ± 0.01; 0.80 ± 0.00 -> 0.81 ± 0.00 | Mixed |
| Use attention contribution features instead of attention weights (ablation) | SAT Probe, Llama-2 7B · Basketball Players, CounterFact - MT, Football Teams, Songs | AUROC: 0.81 ± 0.00 -> 0.81 ± 0.00; 0.98 ± 0.00 -> 0.98 ± 0.00; 0.79 ± 0.00 -> 0.81 ± 0.00; 0.88 ± 0.00 -> 0.88 ± 0.00 | No clear effect |
| Stop inference early using intermediate-layer attention (ablation) | Llama-2 7B and 13B failure prediction · Single-constraint datasets | prediction performance | No clear effect |
| Stop inference early using intermediate-layer attention (ablation) | Llama-2 70B failure prediction · Single-constraint datasets | prediction performance | Hurt |
Surprising findings
- Probing only constraint-token attention can match or exceed the model's confidence without using hidden states or non-constraint tokens.
- Attention alone and confidence make different errors, so the two signals are complementary.
Limitations
- Assumes access to constraint tokens, which must be identified manually in the evaluations.
- Exact-match verification is strict and may introduce noise; multi-constraint labels rely on GPT-3.5 in the verification loop.
- Attention alone does not explain all failures, and the content of attention patterns remains opaque.
Experimental setup
Datasets: WikiData basketball players WikiData football teams WikiData movies WikiData songs CounterFact WikiData books Nobel Prize Winners dataset hand-curated words
Benchmarks: 10 constraint datasets (basketball players, football teams, movies, songs, CounterFact citizenship/HQ/mother tongue, boo
Metrics: AUROC Risk Top 20% Risk Bottom 20%
Baselines: Confidence Constant Popularity
Builds on: mechanistic factual recall analysis (attention transfers subject information) selective classification Grouped-Query Attention
Closest papers by meaning
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (ICLR 2025 · similarity 0.88)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.88)
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.88)
- Precise Information Control in Long-Form Text Generation (NeurIPS 2025 · similarity 0.88)
- KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval (ICLR 2024 · similarity 0.88)
- Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.