Research map / Architectures and efficiency / Attention and transformers
ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ImagePiece, a MaxMatch-inspired re-tokenization that merges semantically weak but locally coherent ViT patch tokens (with a convolutional local coherence bias) before token pruning, speeding up DeiT-S by 54% while improving ImageNet accuracy.
Problem
ViT tokens from non-overlapping grid patches often lack semantics, so existing token pruning and merging methods discard or dilute important information, hurting accuracy under token reduction.
Approach
Apply a WordPiece-style MaxMatch re-tokenization inside ViT blocks: bottom-k attentive tokens are merged with similar tokens via bipartite soft matching, re-scored by class attention, and kept or discarded accordingly; an overlapping-convolution local coherence bias stem makes nearby patches more similar.
Key results
- ImagePiece on DeiT-S reaches 80.22% top-1 (+0.39% over 79.83%) with throughput 3891.9 img/s vs 2531.1 (about 54% faster).
- In hyper-speed inference at about 2.51x speedup, ImagePiece loses 6.28% accuracy versus 20.05%, 13.99%, and 9.24% for DynamicViT, EViT, and ToMe.
- Adding ImagePiece to DynamicViT, ToMe, and EViT improves their accuracy by 0.79% on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Retokenize non-semantic tokens before pruning | DeiT-S · ImageNet-1k | top-1 accuracy (%): 79.83 -> 80.22 | Helped |
| Retokenize non-semantic tokens before pruning | DeiT-S · ImageNet-1k | throughput (img/s): 2531.1 -> 3891.9 | Helped |
| Retokenize non-semantic tokens before pruning | DeiT-Ti · ImageNet-1k | top-1 accuracy (%): 72.13 -> 72.61 | Helped |
| Retokenize non-semantic tokens before pruning | DeiT-S · ImageNet-1k | top-1 accuracy (%): 79.36 -> 80.22 | Helped |
| Retokenize non-semantic tokens, then prune | DeiT-S · ImageNet-1k | top-1 accuracy (%): 79.67 -> 80.22 | Helped |
| Add the local coherence bias module (ablation) | ImagePiece on DeiT-S · ImageNet-1k | top-1 accuracy (%): 79.81% -> 80.22% | Helped |
| Integrate ImagePiece as the tokenizer | DynamicViT · ImageNet-1k | top-1 accuracy (%): 79.32 -> 80.11 | Helped |
| Integrate ImagePiece as the tokenizer | ToMe · ImageNet-1k | top-1 accuracy (%): 79.36 -> 80.08 | Helped |
| Integrate ImagePiece as the tokenizer | EViT · ImageNet-1k | top-1 accuracy (%): 79.37 -> 80.22 | Helped |
| Retokenize non-semantic tokens | DeiT-S · ImageNet-1k, 7 random masks | accuracy (%): 79.63 vs 78.21 | Helped |
| Retokenize non-semantic tokens | DeiT-S · ImageNet-1k, 50 random masks | accuracy (%): 71.42 vs 65.41 | Helped |
| Reduce tokens to 26 output tokens | ImagePiece · ImageNet-1k hyper-speed inference | accuracy (%): 79.38 | No clear effect |
Surprising findings
- Merged tokens in ToMe are frequently top-attentive tokens in early layers (65.16% of merged tokens in DeiT-Ti are in top-30% attentive tokens), suggesting important information is diluted.
Limitations
- Hyperparameters (p=0.3, merge ratio 0.08, prune ratio 0.8) were specifically tuned for acceleration and may not transfer.
- Evaluated only on ImageNet-1k classification with DeiT-based models.
Experimental setup
Datasets: ImageNet-1k
Benchmarks: ImageNet-1k
Metrics: top-1 accuracy throughput (img/s)
Baselines: DynamicViT SPViT EViT ToMe Token Pooling Token Learner
Builds on: DeiT ToMe EViT DynamicViT WordPiece
Closest papers by meaning
- Token Cropr: Faster ViTs for Quite a Few Tasks (CVPR 2025 · similarity 0.90)
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.88)
- MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer (CVPR 2024 · similarity 0.88)
- Subobject-level Image Tokenization (ICML 2025 · similarity 0.87)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.87)
- Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.