Research map / Architectures and efficiency / Attention and transformers
Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PA-LRP, an LRP extension that attributes relevance to positional encodings (learnable, sinusoidal, RoPE, ALiBi) by reformulating the input space as position-token pairs, improving Transformer explainability over AttnLRP on NLP and vision.
Problem
Existing LRP-based explainability methods for Transformers ignore positional encoding layers, violating conservation and losing positionally-associated relevance.
Approach
Reformulate the explainability input space as token-position pairs and derive LRP rules that propagate relevance into input-level PE (learnable/sinusoidal via epsilon-LRP on addition) and attention-level PE (RoPE/ALiBi via matrix-multiplication rules), summing positive relevance across layers into a sink, on top of AttnLRP.
Key results
- On zero-shot LLaMa 3-8B MCQA, PA-LRP improves generation AUAC by about 3.2% and AU-MSE by about 7.7% over AttnLRP.
- On DeiT-Tiny ImageNet segmentation, PA-LRP raises mIoU from 52.85 to 55.92 over AttnLRP.
- On fine-tuned Tiny-LLaMa (IMDB), PE-only relevance reduces AU-MSE for pruning-generation by about 50% relative to AttnLRP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add positional relevance to AttnLRP | PA-LRP on LLaMa-2 7B · IMDB generation and pruning | AU-MSE: 7.629 -> 6.521; 6.548 -> 6.325 | Helped |
| Use positional-only relevance instead of combined semantic and positional relevance (ablation) | LLaMa-2 7B explainability · IMDB generation and pruning | AU-MSE: 7.629 -> 6.792; 6.548 -> 6.823 | Mixed |
| Add multi-sink positional relevance aggregation (ablation) | PA-LRP on LLaMa-2-7B · IMDB binary classification | AUAC: 0.759 -> 0.796; 0.758 -> 0.790 | Helped |
| Add multi-sink positional relevance aggregation (ablation) | PA-LRP on LLaMa-2-7B · IMDB binary classification | AU-MSE: 7.124 -> 6.521; 7.158 -> 6.325 | Helped |
| Add positional relevance to AttnLRP | PA-LRP on LLaMa 3-8B · MCQA generation and pruning | AUAC: 0.365 -> 0.377; 0.354 -> 0.368 | Helped |
| Add positional relevance to AttnLRP | PA-LRP on LLaMa 3-8B · Next Token Prediction generation and pruning | AUAC: 0.559 -> 0.562; 0.559 -> 0.561 | Helped |
| Add positional relevance to AttnLRP | PA-LRP on DeiT Base, Small, and Tiny · ImageNet segmentation | mIoU: 50.100 -> 51.400; 50.000 -> 51.700; 52.850 -> 55.920 | Helped |
| Add positional relevance to AttnLRP | PA-LRP on DeiT Base, Small, and Tiny · ImageNet predicted-class positive perturbation | AUC | Mixed |
| Use only positional relevance (ablation) | PE Only on Tiny-LLaMa-2 7B · IMDB generation | AU-MSE: 8.065 -> 3.918 | Helped |
Surprising findings
- Attributing relevance to positional encodings alone (PE Only) performs comparably to the full AttnLRP baseline on several settings.
- PA-LRP is worse than the baseline on positive perturbation for half of DeiT configurations.
Limitations
- Positive perturbation results lag the baseline in half of DeiT cases by up to 1.2 points.
- Authors do not extend the insight to redesigning existing LRP rules for attention and Transformers.
- Multi-sink design and PE rules are built on AttnLRP and only validated on a limited set of models and PEs.
Experimental setup
Datasets: IMDB ImageNet ImageNet segmentation
Benchmarks: IMDB Wikipedia ImageNet ImageNet segmentation
Metrics: AUAC AU-MSE AUC negative/positive perturbation pixel accuracy mIoU mAP
Baselines: AttnLRP PE Only (ablation)
Builds on: AttnLRP LRP-epsilon RoPE Input x Gradient
Closest papers by meaning
- AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers (ICML 2024 · similarity 0.91)
- Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding (NeurIPS 2025 · similarity 0.86)
- DePass: Unified Feature Attributing by Simple Decomposed Forward Pass (NeurIPS 2025 · similarity 0.85)
- How Transformers Learn Structured Data: Insights From Hierarchical Filtering (ICML 2025 · similarity 0.85)
- On the Faithfulness of Vision Transformer Explanations (CVPR 2024 · similarity 0.85)
- Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.