Research map / Large language models / Language, safety and interpretability
Overthinking the Truth: Understanding how Language Models Process False Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs given incorrect few-shot demonstrations 'overthink': intermediate-layer (logit lens) predictions beat final-layer ones after a critical layer, and ablating a few late 'false induction heads' reduces the accuracy gap.
Problem
Language models imitate incorrect labels in few-shot demonstrations, reducing classification accuracy even though they often answer correctly zero-shot; the internal mechanism behind this harmful context-following is unclear.
Approach
Decode predictions from intermediate layers with the logit lens to find a critical layer where correct and incorrect demonstrations diverge (overthinking), then identify late attention heads that attend to and promote false labels from earlier demonstrations (false induction heads) and ablate them.
Key results
- Ablating 5 false induction heads in GPT-J reduces the gap between correct and incorrect permuted-label prompts by an average of 38.9% (about 31.5% under random labels) with negligible loss on correct prompts (average change -1.26 points).
- For GPT-J, early exit at layer 16 beats the full model on all but one dataset given permuted labels, with accuracy diverging at critical layers 13-14.
- Ablating late attention heads alone recovers nearly the full late-layer ablation effect in GPT-J (47.58 vs 47.72 at permuted labels), while late MLP ablation has little effect (37.97).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Zero out late attention heads (ablation) | GPT-J · 14-task average, permuted labels | calibrated accuracy: 37.42_{1.47} -> 47.58_{1.47} | Helped |
| Zero out late attention heads (ablation) | GPT2-XL · 14-task average, permuted labels | calibrated accuracy: 41.97_{1.49} -> 46.09_{1.49} | Helped |
| Zero out late attention heads (ablation) | GPT-NeoX · 14-task average, permuted labels | calibrated accuracy: 45.19_{1.47} -> 44.44_{1.47} | Hurt |
| Zero out late MLPs (ablation) | GPT-J · 14-task average, permuted labels | calibrated accuracy: 37.42_{1.47} -> 37.97_{1.49} | No clear effect |
| Zero out late layers (ablation) | GPT-J · 14-task average, permuted labels | calibrated accuracy: 37.42_{1.47} -> 47.72_{1.46} | Helped |
| Zero out late attention heads (ablation) | GPT2-XL · 14-task average, correct labels | calibrated accuracy: 54.19_{1.51} -> 54.09_{1.49} | No clear effect |
| Zero out late attention heads (ablation) | GPT-J · 14-task average, correct labels | calibrated accuracy: 65.54_{1.42} -> 64.46_{1.39} | Hurt |
| Zero out late attention heads (ablation) | GPT-NeoX · 14-task average, correct labels | calibrated accuracy: 61.68_{1.41} -> 60.86_{1.43} | Hurt |
| Ablate the 5 heads with highest prefix-matching scores (ablation) | GPT-J · 14-task average, permuted labels | accuracy gap reduction: 38.98_{0.16} | Helped |
| Ablate 5 randomly selected heads (ablation) | GPT-J · 14-task average, permuted labels | accuracy gap reduction: 3.97_{0.03} | No clear effect |
| Use random demonstration labels instead of correct labels | GPT-J · k=40 demonstrations | accuracy gap: 0.15 | Hurt |
| Use half correct and half permuted demonstration labels instead of correct labels | GPT-J · k=40 demonstrations | accuracy gap: 0.12 | Hurt |
Surprising findings
- Overthinking also appears with correct demonstrations: early exiting at the critical layer improves accuracy on a majority of datasets, suggesting a misalignment between pretraining and few-shot performance.
- Ablating random heads sometimes increases the accuracy gap, while ablating false induction heads reliably shrinks it.
Limitations
- Head ablations do not fully close the accuracy gap between correct and incorrect demonstrations.
- Experiments are restricted to text classification with many incorrect few-shot examples.
- Zeroing heads may push intermediate states off-distribution, possibly reducing overall performance.
Experimental setup
Datasets: SST-2 Poem-Sentiment Financial-Phrasebank Ethos TweetEval-Hate TweetEval-Atheism TweetEval-Feminist MedQ-Pairs MRPC SICK RTE AGNews
Benchmarks: SST-2 Poem-Sentiment Financial-Phrasebank Ethos TweetEval-Hate TweetEval-Atheism TweetEval-Feminist MedQ-Pairs MRPC SICK RTE AGNews
Metrics: calibrated accuracy accuracy gap permuted score prefix-matching score false label promoting score
Baselines: full model (no ablation) random head ablation late layer ablation late MLP ablation
Builds on: induction heads logit lens early exiting calibration (contextual calibration)
Closest papers by meaning
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (ICLR 2025 · similarity 0.86)
- Negation Neglect: When models fail to learn negations in training (NeurIPS 2026 · similarity 0.86)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.86)
- Emergence of Linear Truth Encodings in Language Models (NeurIPS 2025 · similarity 0.86)
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models (NeurIPS 2026 · similarity 0.86)
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.