Research map / Architectures and efficiency / Attention and transformers
Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-criteria Token Fusion (MCTF), which fuses ViT tokens using similarity, informativeness, and token size criteria with one-step-ahead attention and a token reduction consistency fine-tuning loss.
Problem
Token reduction in Vision Transformers lowers cost but typically loses information and degrades accuracy, since prior token fusion methods use only a single criterion for matching tokens.
Approach
Fuse tokens via bidirectional bipartite soft matching with a multiplicative attraction function combining cosine similarity, informativeness (from next-layer attention, one-step-ahead), and inverse token size, plus a consistency loss between different reduction ratios during fine-tuning.
Key results
- DeiT-T with MCTF (r=16) reaches 72.7% top-1 at 0.7G FLOPs, +0.5% over the 72.2% base model with about 44% fewer FLOPs.
- DeiT-S with MCTF reaches 80.1% top-1 at 2.6G FLOPs, +0.3% over the 79.8% base at 4.6G FLOPs.
- Without training, MCTF on pre-trained DeiT-S at r=20 gets 78.0% vs 74.2% for ToMe.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MCTF token fusion | DeiT-T · ImageNet-1K | Top-1 accuracy (%): 72.2 -> 72.7 | Helped |
| Apply MCTF token fusion | DeiT-S · ImageNet-1K | Top-1 accuracy (%): 79.8 -> 80.1 | Helped |
| Apply MCTF token fusion | T2T-ViTt-14 · ImageNet-1K | Top-1 accuracy (%): 81.7 -> 81.8 | Helped |
| Apply MCTF token fusion | LV-ViT-S · ImageNet-1K | Top-1 accuracy (%): 83.3 -> 83.4 | Helped |
| Apply MCTF without additional training instead of ToMe | DeiT-T · ImageNet-1K | Top-1 accuracy (%): 68.7 -> 71.0 | Helped |
| Apply MCTF without additional training instead of ToMe | DeiT-S · ImageNet-1K | Top-1 accuracy (%): 77.9 -> 79.2 | Helped |
| Add similarity and informativeness criteria to the size criterion (ablation) | MCTF on DeiT-S · ImageNet-1K | Top-1 accuracy (%): 79.8% -> 80.1% | Helped |
| Add the size criterion to similarity and informativeness (ablation) | MCTF on DeiT-S · ImageNet-1K | Top-1 accuracy (%): 79.8% -> 80.1% | Helped |
| Add one-step-ahead attention (ablation) | MCTF on DeiT-S · ImageNet-1K | Top-1 accuracy (%) | Helped |
| Add token reduction consistency (ablation) | MCTF on DeiT-S · ImageNet-1K | Top-1 accuracy (%) | Helped |
| Use bidirectional instead of one-way bipartite matching (ablation) | MCTF on DeiT-S · ImageNet-1K | Top-1 accuracy (%): 80.0 -> 80.1 | Helped |
| Use weighted average pooling instead of average pooling (ablation) | MCTF on DeiT-S · ImageNet-1K | Top-1 accuracy (%): 80.0 -> 80.1 | Helped |
Surprising findings
- Token fusion with MCTF slightly improves accuracy over the full base model while reducing FLOPs, for DeiT-T and DeiT-S.
Limitations
- Gains over the base model are small (+0.3% to +0.5%) and the main experiments are on ImageNet-1K classification only.
- Fine-tuning with token reduction consistency requires additional training, and the training-free setting degrades at high reduction ratios.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: Top-1 accuracy FLOPs (G) Params (M)
Baselines: ToMe EViT BAT SPViT DynamicViT A-ViT IA-RED2 EvoViT ATS
Builds on: ToMe EViT BAT DeiT
Closest papers by meaning
- EViT: Expediting Vision Transformers via Token Reorganizations (ICLR 2022 · similarity 0.90)
- DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification (NeurIPS 2021 · similarity 0.89)
- A Simple Romance Between Multi-Exit Vision Transformer and Token Reduction (ICLR 2024 · similarity 0.89)
- Token Cropr: Faster ViTs for Quite a Few Tasks (CVPR 2025 · similarity 0.87)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
- Frequency-Aware Token Reduction for Efficient Vision Transformer (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.