Research map / Architectures and efficiency / Attention and transformers
Attentive Multi-Layer Fusion for Vision Transformers
Laure Ciernik, Marco Morik, Lukas Thede, Luca Eyring, Shinichi Nakajima, Zeynep Akata, Lukas Muttenthaler
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fuse CLS and AP tokens from all layers with attention | linear probe on last-layer CLS · 20 datasets, 9 ViTs | accuracy gain: 5.54 pp average gain over linear probing | Improved |
| use attentive fusion instead of concatenating layers (ablation) | multi-layer probe · 20 datasets | accuracy gain | Improved |
| concatenate intermediate layers with a linear classifier (ablation) | linear probe · 20 datasets | accuracy gain | Mixed |
| add more intermediate layers to the fusion pool (ablation) | ALF and linear fusion · CLIP-B-16, DINOv2-B-14, ViT-B-16 | accuracy gain | Improved |
| attend over all last-layer patch tokens (AAT) instead of layer summary tokens | ALF · GTSRB, SVHN, Stanford Cars, FGVC Aircraft | accuracy gain: GTSRB 13.47 -> 18.02 | Mixed |
| use ALF in place of AAT | last-layer attentive probe · 20 datasets | stability | Improved |
| combine spatial aggregation across patches with hierarchical aggregation across quarterly layers (ablation) | ALF · GTSRB, FER2013 | accuracy | Improved |
| use both CLS and AP tokens instead of CLS-only or AP-only in ALF (ablation) | ALF · 19 datasets, 3 base models | accuracy gain | Improved |
| use AP tokens only (ablation) | ALF · 19 datasets | accuracy gain | Mixed |
| use ALF for tasks near pretraining domain | ALF · STL-10 | accuracy gain: 0.04 ± 0.17 | No clear effect |
| apply ALF on PCAM | ALF · PCAM | accuracy gain: 5.32 (linear) vs 2.85 (attentive) | Hurt |
| prune ALF tokens at inference to about half (ablation) | ALF · pruning analysis | accuracy | Mixed |
Closest papers by meaning
- AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers (ICML 2024 · similarity 0.87)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.86)
- LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer (ICML 2026 · similarity 0.86)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.86)
- Transformer Fusion with Optimal Transport (ICLR 2024 · similarity 0.86)
- LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
