Research map / Large language models / Language, safety and interpretability
Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how LMs (124M–7B) fail on balanced-parentheses closing-token prediction via many reliable and faulty attention heads/FF neurons, and proposes RaSteer, which ranks components by reliability and scales the top ones to raise accuracy to near 100%.
Problem
Language models still fail on simple syntactic tasks such as predicting the correct number of closing parentheses in code, and it is unclear why these errors persist internally.
Approach
Treats each attention head and FF neuron that directly writes to the final logit as an independent predictor, distinguishes reliable from faulty components by their accuracy across sub-tasks, and steers generation by amplifying the most reliable components (RaSteer), rather than using steering vectors or circuit discovery.
Key results
- RaSteer steering top attention heads raised GPT-2 XL from 0% to roughly 100% on three- and four-paren sub-tasks, and boosted some models from 0% to about 100%.
- On HumanEval, steering top-20 heads gave Llama2-7b a 5.49% improvement (11.58% to 17.07%) while CodeLlama-7b and Pythia-6.9b stayed roughly unchanged.
- On arithmetic, RaSteer improved Pythia-6.9b multiplication from 14.41% to 34.66% (gain of about 20.25%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Steer the top-5 attention heads | GPT-2 XL · Three-Paren | accuracy: 0% to 100% | Helped |
| Steer the top-10 attention heads | GPT-2 XL · Four-Paren | accuracy: 0% to ~100% | Helped |
| Use attention heads rather than FF neurons alone (ablation) | GPT-2 XL · Three-Paren | accuracy: 0% to 100% | Helped |
| Use FF neurons alone (ablation) | GPT-2 XL · Three-Paren | accuracy: no effect | No clear effect |
| Rank components by F1-score rather than recall or precision (ablation) | RaSteer · Balanced parentheses | accuracy | Helped |
| Steer the top-20 parentheses-task attention heads | CodeLlama-7b · HumanEval | accuracy: 30.48% → 29.87% | Hurt |
| Steer the top-20 parentheses-task attention heads | Llama2-7b · HumanEval | accuracy: 11.58% → 17.07% | Helped |
| Steer the top-20 parentheses-task attention heads | Pythia-6.9b · HumanEval | accuracy: 10.36% → 10.97% | Helped |
| Steer top-30 recall-ranked attention heads | Pythia-6.9b · Multiplication | accuracy: 14.41% → 34.66% | Helped |
| Steer top-30 recall-ranked attention heads | GPT-2 XL · Multiplication | accuracy: 17.50% → 17.50% | No clear effect |
| Steer top-30 recall-ranked attention heads | GPT-2 XL · Addition | accuracy: 0.00% → 0.00% | No clear effect |
| Add five circuit-identified heads from non-final token positions to the top-10 RaSteer heads (ablation) | GPT-2 Medium · Four-Paren | accuracy: ~100% to 4.00% | Hurt |
Surprising findings
- Attention heads steer more effectively than FF neurons; promoting sixty FF neurons had no effect on GPT-2 XL three-paren while five attention heads raised it from 0% to 100%.
- Circuit-discovery-based ranking of heads underperforms the simple reliability ranking, and steering non-final-position circuit heads can collapse accuracy.
Limitations
- Evaluated mainly on synthetic data, which may exaggerate effectiveness.
- Assumes additive contributions of components and ignores non-additive mechanisms.
- Uses simple heuristics: fixed scalar multipliers and ranking by recall/precision/F1.
Experimental setup
Datasets: synthetic balanced parentheses dataset synthetic two-operand arithmetic dataset HumanEval
Benchmarks: balanced parentheses synthetic test set (one- to four-paren sub-tasks) HumanEval two-operand arithmetic tasks
Metrics: accuracy precision recall F1-score faithfulness
Baselines: circuit discovery baseline (activation patching) unsteered models
Builds on: logit lens activation patching knowledge neurons (FF neuron interpretation) activation steering
Closest papers by meaning
- Interpreting and Improving Large Language Models in Arithmetic Calculation (ICML 2024 · similarity 0.88)
- Controlling Equational Reasoning in Large Language Models with Prompt Interventions (AAAI 2025 · similarity 0.87)
- Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics (ICLR 2025 · similarity 0.87)
- Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models (ICLR 2024 · similarity 0.86)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.86)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.