Research map / Architectures and efficiency / Attention and transformers
MLP Can Be A Good Transformer Learner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Removes non-essential self-attention layers from DeiT-B by selecting them via transfer entropy (NOSE) and diluting them into identity mappings so only MLPs remain in those blocks.
Problem
Vision transformers' self-attention is computationally expensive, and prior token pruning/merging methods still load the full network and keep the same memory footprint, so they do not reduce memory bound.
Approach
Use entropy measures to identify attention layers with low information content and low transfer entropy to the output, then gradually degenerate them into identity mappings (via a decaying sparse mask plus feature compensation) so their residual path feeds directly into the subsequent MLP, yielding MLP-only blocks.
Key results
- Removing 40% of DeiT-B attention layers keeps Top-1 at 81.8% on ImageNet-1k (same as DeiT-B), with 13.7% fewer parameters and 20.5% higher memory-bound working load.
- Combined with ToMe, the 40% variant reaches 81.4% Top-1 with throughput of 507 images/s versus 299 for DeiT-B.
- On CIFAR-100 linear probing, removing 40% of attention layers yields 81.3% versus 80.6% for DeiT-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove 40% of attention layers using NOSE | DeiT-B · ImageNet-1k | Top-1 accuracy: 81.8 -> 81.8 | No clear effect |
| Remove 50% of attention layers using NOSE | DeiT-B · ImageNet-1k | Top-1 accuracy: 81.8 -> 81.5 | Hurt |
| Remove 50% of attention layers | DeiT-B · ImageNet-1k | throughput (images/s): 299 -> 408; 36.5% | Helped |
| Remove 40% of attention layers | DeiT-B · ImageNet-1k | memory bound (images/10GB): 606 -> 730; 20.5% | Helped |
| Remove 40% of attention layers | DeiT-B · CIFAR-100 | fine-tuning accuracy: 90.5 -> 90.3 | Hurt |
| Remove 40% of attention layers | DeiT-B · CIFAR-100 | linear probing accuracy: 80.6 -> 81.3 | Helped |
| Remove 40% of attention layers | DeiT-B · ADE20k, pre-trained on ImageNet-1k | mIoU: 47.0 -> 46.2 | Hurt |
| Use NOSE instead of random layer selection (ablation) | DeiT-B · ImageNet-1k | Top-1 accuracy: 79.9 -> 81.8 | Helped |
| Use linear sparse-mask decay over 200 rather than 300 epochs (ablation) | DeiT-B with 40% attention layers removed · ImageNet-1k | Top-1 accuracy: 81.8 -> 81.4 | Hurt |
| Use linear sparse-mask decay over 400 rather than 300 epochs (ablation) | DeiT-B with 40% attention layers removed · ImageNet-1k | Top-1 accuracy: 81.8 -> 82.0 | Helped |
| Use cosine rather than linear sparse-mask decay over 200 epochs (ablation) | DeiT-B with 40% attention layers removed · ImageNet-1k | Top-1 accuracy: 81.4 -> 81.5 | Helped |
| Use cosine rather than linear sparse-mask decay over 300 epochs (ablation) | DeiT-B with 40% attention layers removed · ImageNet-1k | Top-1 accuracy: 81.8 -> 81.8 | No clear effect |
Surprising findings
- Greedy transfer-entropy selection did not pick the lowest-entropy layer first (layer 0), instead choosing non-consecutive layers such as layer 3 at the first step.
- Removing 40% of attention layers slightly improves CIFAR-100 linear probing over the original DeiT-B.
Limitations
- Experiments are limited to DeiT-B and a few benchmarks; other backbones are only in the appendix.
- Random selection baseline sampled only three combinations due to combinatorial cost of the search space.
- Dense prediction results show a gap of about 0.7% relative to the baseline when pre-trained on ImageNet-1k at 50% removal.
Experimental setup
Datasets: ImageNet-1k CIFAR-100 ADE20K
Benchmarks: ImageNet-1k CIFAR-100 ADE20K
Metrics: Top-1 accuracy Top-5 accuracy FLOPs throughput parameters memory bound mIoU mAcc aAcc linear probing accuracy
Baselines: DeiT-B DynamicViT Evo-ViT EViT TPS ToMe DiffRate Random layer selection
Builds on: DeiT ToMe DynamicViT TinyMIM
Closest papers by meaning
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.89)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
- Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.87)
- Frequency-Aware Token Reduction for Efficient Vision Transformer (NeurIPS 2025 · similarity 0.86)
- Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers (CVPR 2024 · similarity 0.86)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.