Research map / Architectures and efficiency / Attention and transformers
Interpreting vision transformers via residual replacement model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically annotates 6.6K TopK SAE features across all layers of ViT, CLIP and DINOv2, and builds a residual replacement model that yields faithful, parsimonious feature circuits for ViTs, applied to debiasing spurious correlations.
Problem
Vision transformers are hard to interpret: their features are poorly understood across all layers, and attention-based token-to-token circuits are too large for human-scale interpretability.
Approach
Replace ViT computation with SAE features in the residual stream only (bypassing attention), aggregate features over tokens, select circuit nodes by edge importance to already-selected downstream nodes, and correct noisy gradients with a LibraGrad-based method.
Key results
- Residual replacement circuits with gradient correction reach 94.1% faithfulness on ViT vs 24.9% for random neuron circuits, and 85.1% (DINOv2) and 82.3% (CLIP).
- Features are judged significantly more interpretable than raw neurons across all layers in both expert and a 16-participant non-expert user study.
- Intervening by ablating a single spurious feature (L9#2371) raises mAUC from 0.854 to 0.904 with accuracy 0.848 vs original 0.849 (SpuFix upper bound mAUC 0.917).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add edge-based discovery to the naïve feature circuit (ablation) | Residual replacement model · ImageNet val, ViT/DINOv2/CLIP circuits | faithfulness: 64.9 -> 74.2; 58.9 -> 64.1; 50.7 -> 52.2 | Helped |
| Add gradient correction to edge-based feature discovery (ablation) | Residual replacement model · ImageNet val, ViT/DINOv2/CLIP circuits | faithfulness: 74.2 -> 94.1; 64.1 -> 85.1; 52.2 -> 82.3 | Helped |
| Use feature nodes instead of neuron nodes (ablation) | Residual replacement model · ImageNet val, ViT/DINOv2/CLIP circuits | faithfulness: 26.3 -> 74.2; 26.7 -> 64.1; 30.2 -> 52.2 | Helped |
| Replace top-k node selection with top-p selection (ablation) | Residual replacement model · ImageNet val, ViT/DINOv2/CLIP circuits | faithfulness: 94.1 -> 95.0; 85.1 -> 85.4; 82.3 -> 82.9 | No clear effect |
| Replace top-k node selection with threshold selection (ablation) | Residual replacement model · ImageNet val, ViT/DINOv2/CLIP circuits | faithfulness: 94.1 -> 93.6; 85.1 -> 84.6; 82.3 -> 82.2 | No clear effect |
| Ablate the identified spurious feature | ViT · ImageNet val debiasing | mAUC: 0.854 -> 0.904 | Helped |
| Ablate the identified spurious feature | ViT · ImageNet val debiasing | Accuracy: 0.849 -> 0.848 | Hurt |
Surprising findings
- Early-layer features, though seemingly uninterpretable at image level, are consistently interpretable at the patch level.
- ViT curve detector circuits are mechanistically similar to those in InceptionV1 despite architectural differences.
Limitations
- Interpretation of features is qualitative and subjective, relying on manual annotation.
- Residual replacement model only indirectly captures attention and MLP roles via gradients.
- Uses only TopK SAE variant; SAE limitations may affect feature quality.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet validation (1,500 sampled images) ImageNet spurious correlation evaluation
Metrics: faithfulness 1 - completeness causality AUC interpretability score accuracy mAUC
Baselines: Cross-Layer Attribution (CLA) neuron circuit random circuit naive circuit SpuFix
Builds on: TopK SAE replacement model attribution patching LibraGrad cross-layer transcoder
Closest papers by meaning
- Intriguing Properties of Vision Transformers (NeurIPS 2021 · similarity 0.90)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.88)
- Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models (ICML 2024 · similarity 0.88)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.87)
- Improving Interpretation Faithfulness for Vision Transformers (ICML 2024 · similarity 0.86)
- Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.