Research map / Architectures and efficiency / Attention and transformers
IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned informative-token selection instead of random dropping | DeiT-S with IA-RED² · ImageNet-1k | Top-1 accuracy: 78.4 -> 79.1 | Helped |
| Use IA-RED² rather than raw-attention token selection | DeiT-S · ImageNet-1k | Top-1 accuracy: 78.4 -> 79.1 | Helped |
| Use IA-RED² rather than raw attention for localization | DeiT-S · ImageNet-Segmentation | mIoU: 46.37 -> 49.42 | Helped |
| Use IA-RED² token selection rather than temporal-difference selection | TimeSformer JointST · Kinetics-400 | clip-1 error: 32.3 -> 35.3 | Hurt |
| Use 3 interpreter groups instead of 2 (ablation) | IA-RED² with DeiT-S · ImageNet-1k | Top-1 accuracy: 78.6% -> 79.1% | Helped |
| Use 3 interpreter groups instead of 4 (ablation) | IA-RED² with DeiT-S · ImageNet-1k | Top-1 accuracy: 78.8% -> 79.1% | Helped |
| Increase input resolution from 224×224 to 384×384 at matched reduction setting (ablation) | IA-RED² with DeiT-B · ImageNet-1k | Top-1 accuracy: 80.3 -> 81.9 | Mixed |
| Replace REINFORCE with straight-through Gumbel (ablation) | IA-RED² with DeiT-S · ImageNet-1k | Top-1 accuracy: 79.1% -> 78.8% | Hurt |
| Replace the squared kept-token reward term with a linear term at τ=1.5 (ablation) | IA-RED² with DeiT-S · ImageNet-1k | Top-1 accuracy: 79.1% -> 78.2% | Hurt |
| Raise the token-discard threshold from 0.50 to 0.52 (ablation) | IA-RED² with DeiT-B · ImageNet-1k | Top-1 accuracy: 80.9% -> 63.7% | Hurt |
| Raise the token-discard threshold from 0.50 to 0.52 (ablation) | IA-RED² with DeiT-B · ImageNet-1k | FLOPs: 11.8 G -> 4.9 G | Helped |
Closest papers by meaning
- Interpreting vision transformers via residual replacement model (NeurIPS 2025 · similarity 0.88)
- PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer (ICLR 2025 · similarity 0.88)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.86)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.86)
- Sharing Key Semantics in Transformer Makes Efficient Image Restoration (NeurIPS 2024 · similarity 0.86)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.