Research map / Architectures and efficiency / Attention and transformers
A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses token-perturbation sensitivity in transformer final layers to measure semantic dependencies, showing tokens largely retain their own meaning, encode truthful dependencies, and that QA mistakes stem from falsely encoded dependencies in shared attention heads.
Problem
How semantic dependencies are encoded and propagated within transformer tokens, and how such encodings cause incorrect answers in models.
Approach
Measure token-level semantic dependency by randomly replacing one input token and quantifying how much each final-layer token changes; use this to test whether QA mistakes correspond to wrong answer tokens strongly influencing question tokens, and localize attention heads responsible.
Key results
- Self-information retention validation ranges from 75.15% (GPT-2) to 99.62% (DeBERTa) across 10 models.
- Average alignment scores between perturbation-derived and SpaCy-parsed dependency groups range from 82.44% (TinyRoBERTa) to 93.41% (GPT-2).
- For BERT on SQuAD failures, 79.07% of mistakes show the wrong answer tokens exerting stronger dependency on question tokens than correct answer tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Measure whether final-layer tokens retain their corresponding input-token information | Transformer models · Six datasets | self-information retention (%): 75.15 to 99.62 across models | Helped |
| Measure propagation of input-token information to final-layer tokens | Transformer models · Six datasets | sequence-level semantic aggregation (%): 93.42 to 100.00 across models | Helped |
| Compare final-layer dependency groups with SpaCy semantic dependency groups | Transformer models · GLUE dependency cases | average alignment score (%): 82.44 to 93.41 across models | Helped |
| Compare incorrect-answer and correct-answer dependencies on question tokens | Transformer models · SQuAD 1.1 validation failure cases | P(fθ) (%): 48.04 to 81.80 across models | Mixed |
| Use shared attention heads to encode both correct and false semantic dependencies | QA attention head groups · Failed QA cases | semantic dependency encoding | Mixed |
| Add irrelevant context or change the order of independent context sequences | Transformer token dependency encoding · Independent sentence contexts | dependency alteration score | Hurt |
Surprising findings
- Shared attention heads encode both correct and incorrect semantic dependencies, so head pruning is unlikely to fix false dependencies.
- GPT-2 retains original token information in only 75.15% of cases, lower than the BERT-series models.
Limitations
- Relies on answer tokens appearing verbatim in the context, limiting applicability to generated answers.
- Random token replacement may introduce variability from semantically similar substitutes.
- Analysis focuses on final-layer versus first-layer tokens only.
Experimental setup
Datasets: GSM8K Yelp GLUE CNN/DailyMail OpenOrca WikiText SQuAD 1.1
Benchmarks: SQuAD 1.1
Metrics: self-information retention percentage sequence-level aggregation percentage alignment score F1 percentage of false dependency cases
Baselines: attention flow head pruning
Builds on: Gandelsman et al. 2024 attention head decomposition perturbed masking
Closest papers by meaning
- Syntax vs. Semantics: How Transformers Learn Deep Dependencies (ICML 2026 · similarity 0.87)
- Linearity of Relation Decoding in Transformer Language Models (ICLR 2024 · similarity 0.86)
- LLM Layers Immediately Correct Each Other (NeurIPS 2025 · similarity 0.86)
- Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models (AAAI 2025 · similarity 0.86)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.85)
- Order-Independence Without Fine Tuning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.