Research map / Architectures and efficiency / Attention and transformers
LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LibraGrad, a post-hoc method that prunes and scales backward paths in Transformers (attention, LayerNorm, gated activations, self-gating) to restore FullGrad-completeness, improving gradient-based ViT attributions without changing the forward pass.
Problem
Gradient-based attribution methods perform poorly on Transformers because non-locally-affine operations (attention, LayerNorm, gated activations, self-gating) break FullGrad-completeness, causing imbalanced gradient flow and unfaithful attributions.
Approach
Theoretically motivated backward-pass modifications: treat nonlinear multiplicands (softmax, LayerNorm denominator, gate) as constants via a constant (stop-gradient) operator and scale the gradients of both branches in self-gating by 1/2, so the Transformer becomes FG-complete while the forward pass is unchanged.
Key results
- Libra FullGrad+ reaches 64.2% average MIF deletion accuracy across 7 ViT-family models vs 40.6% for FullGrad+ (Table 6).
- LibraGrad reduces Completeness Error of FullGrad to 0.0 on all seven tested models, versus 11.0 average for FullGrad (Table 4).
- Libra FullGrad+ achieves 79.4 segmentation AP on EVA2-S ImageNet-S setting (Table 5), and Libra variants improve every gradient method tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LibraGrad to FullGrad+ | FullGrad+ on ViT-B · ImageNet | MIF accuracy (predicted labels): 44.2 -> 63.1 | Helped |
| Apply LibraGrad to FullGrad+ | FullGrad+ on ViT-B · ImageNet-Hard | MIF accuracy (predicted labels): 80.1 -> 87.6 | Helped |
| Apply LibraGrad to FullGrad+ | FullGrad+ on ViT-B · MURA | MIF accuracy (predicted labels): 32.8 -> 43.2 | Helped |
| Apply LibraGrad to FullGrad+ | FullGrad+ on ViT-B · Oxford-IIIT Pet | MIF accuracy (predicted labels): 35.3 -> 57.3 | Helped |
| Add Libra attention gradient handling (ablation) | Libra FullGrad+ on EVA2-S · ImageNet | MIF accuracy (predicted labels): 65.2 -> 71.7 | Helped |
| Add Libra LayerNorm gradient handling (ablation) | Libra FullGrad+ on EVA2-S · ImageNet | MIF accuracy (predicted labels): 49.9 -> 71.7 | Helped |
| Add Libra attention and LayerNorm gradient handling (ablation) | Libra FullGrad+ on EVA2-S · ImageNet | MIF accuracy (predicted labels): 61.2 -> 71.7 | Helped |
| Add Libra gated activation handling (ablation) | Libra FullGrad+ on EVA2-S · ImageNet | MIF accuracy (predicted labels): 71.6 -> 71.7 | No clear effect |
| Add Libra SwiGLU self-gating (ablation) | Libra FullGrad+ on EVA2-S · ImageNet | MIF accuracy (predicted labels): 67.0 -> 71.7 | Helped |
| Include bias terms (ablation) | Libra FullGrad+ on EVA2-S · ImageNet | MIF accuracy (predicted labels): 71.5 -> 71.7 | No clear effect |
| Apply LibraGrad rather than standard FullGrad+ (ablation) | FullGrad+ on EVA2-S · ImageNet | MIF accuracy (predicted labels): 48.0 -> 71.7 | Helped |
Surprising findings
- Gated activations (GELU/SiLU) theoretically break FG-completeness, but removing their correction has negligible practical impact because they often operate in saturated regimes.
- Bias terms are necessary for theoretical completeness but omitting them has only modest practical impact.
Limitations
- Gains are shown mainly for ImageNet-finetuned ViT-family models; the method is tailored to Transformer components and does not address other architectures beyond a MLP-Mixer check.
- Integrated Gradients suffers numerical instability and fails to achieve completeness in practice with a 50-step approximation.
Experimental setup
Datasets: ImageNet-1k ImageNet-Hard MURA Oxford-IIIT Pet ImageNet-S FunnyBirds COCO
Benchmarks: ImageNet-1k ImageNet-Hard MURA Oxford-IIIT Pet ImageNet-S FunnyBirds
Metrics: Most-Influential-First Deletion accuracy AOPC Completeness Error Segmentation AP Least-Influential-First Deletion Symmetric Relevance Gain
Baselines: Integrated Gradients Input × Grad FullGrad+ GenAtt TokenTM AttCAT DecompX AttnLRP AliLRP RawAtt Attention Rollout GradCAM+
Builds on: FullGrad Integrated Gradients GenAtt AttnLRP HiResCAM
Closest papers by meaning
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.87)
- On the Faithfulness of Vision Transformer Explanations (CVPR 2024 · similarity 0.86)
- Benchmarking the Attribution Quality of Vision Models (NeurIPS 2024 · similarity 0.86)
- Comprehensive Information Bottleneck for Unveiling Universal Attribution to Interpret Vision Transformers (CVPR 2025 · similarity 0.86)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.86)
- Pixel-level Certified Explanations via Randomized Smoothing (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.