Research map / Architectures and efficiency / Attention and transformers
Transformer Doctor: Diagnosing and Treating Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transformer Doctor, which diagnoses conjunction errors in Vision Transformers' inter-token (MHSA) and intra-token (FFN) information integration and treats them with heuristic dynamic and rule-based static integration constraint losses applied to already-trained models.
Problem
Vision Transformers are hard to interpret, and their internal error mechanisms are poorly understood; existing interpretability work focuses on token importance, modules, and features rather than how errors arise internally.
Approach
Inspired by biological conjunction errors, it hypothesizes that Transformers dynamically integrate information among tokens in MHSA in deep layers and statically integrate class information within tokens in FFN, and that integrating wrong information causes mispredictions; it then adds gradient-weighted loss terms (IDI, ISI) that suppress background integration and wrong-class integration during f
Key results
- ViT-Tiny with both IDI and ISI treatment improves ImageNet-1K accuracy from 64.77% to 68.86% (+4.09) and ImageNet-50 from 59.62% to 61.02%.
- EVA-Tiny gains +2.94 on ImageNet-1K (72.51% to 75.45%) with joint treatment; PVT-Tiny gains +3.21 on ImageNet-1K.
- Across six Transformers and five datasets, treatment improves accuracy in every reported setting, with larger gains on larger datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Transformer Doctor's dynamic and static integration constraints | ViT-Tiny · ImageNet-1K | accuracy: 64.77 -> 68.86 | Helped |
| Add Transformer Doctor's dynamic and static integration constraints | Eva-Tiny · CIFAR-10 | accuracy: 87.56 -> 88.28 | Helped |
| Add Transformer Doctor's static integration constraint | Eva-Tiny · CIFAR-100 | accuracy: 64.15 -> 64.99 | Helped |
| Add Transformer Doctor's dynamic and static integration constraints | PVT-Tiny · ImageNet-10 | accuracy: 82.00 -> 84.20 | Helped |
| Add Transformer Doctor's dynamic and static integration constraints | PVT-Tiny · ImageNet-50 | accuracy: 71.81 -> 74.53 | Helped |
| Add Transformer Doctor's dynamic and static integration constraints | Eva-Tiny · ImageNet-1K | accuracy: 72.51 -> 75.45 | Helped |
| Set static integration threshold τ to 0.2 (ablation) | ViT-Tiny + ISI · ImageNet-10 | accuracy: 78.8 -> 80.4 | Helped |
| Set static integration threshold τ to 0.8 (ablation) | ViT-Tiny + ISI · ImageNet-10 | accuracy: 80.4 -> 78.4 | Hurt |
Surprising findings
- Forcing correct integration in early blocks does not improve accuracy; constraints only help when applied in deep blocks (after roughly the 7th block for dynamic, after the 11th for static).
Limitations
- Validated only on image classification; not on detection, segmentation, NLP, or multimodal tasks.
- Dynamic constraint requires foreground annotations (manual or ImageNet-S masks), limiting applicability.
- Gains are small on small datasets, where feature-extraction failures dominate.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-10 ImageNet-50 ImageNet-1k ImageNet-S (foreground masks)
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-10 ImageNet-50 ImageNet-1K
Metrics: top-1 accuracy
Baselines: Blank control (continued training without constraints) Original pre-trained Transformers Mean/min/max head aggregation of attention weights Non-gradient integration weights
Builds on: Vision Transformer DeiT CaiT TNT PVT BEiT
Closest papers by meaning
- Adventures of Trustworthy Vision-Language Models: A Survey (AAAI 2024 · similarity 0.85)
- Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization (NeurIPS 2024 · similarity 0.84)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.83)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.83)
- Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits (NeurIPS 2025 · similarity 0.83)
- A General and Efficient Training for Transformer via Token Expansion (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.