Research map / Architectures and efficiency / Attention and transformers
Morphing Tokens Draw Strong Masked Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Token Morphing (DTM), which dynamically merges contextually related target tokens via bipartite matching while keeping token count, to reduce spatial inconsistency in MIM targets from CLIP, improving ViT pre-training on ImageNet-1K and ADE20K.
Problem
Token-wise masked image modeling targets from pre-trained models (e.g., CLIP) are spatially inconsistent even among neighboring tokens, which hinders learning discriminative ViT representations.
Approach
Dynamic Token Morphing aggregates related tokens with randomized counts and iteration numbers via bipartite matching, averages them within groups, and aligns morphed online and target tokens while preserving the total token count, yielding contextualized, spatially smoother targets.
Key results
- DTM pre-trained ViT-B/16 reaches 85.4% ImageNet-1K fine-tuning top-1 (85.5% at 800 epochs), outperforming MVP (84.4%) and BEiT v2 (85.0%) at 300 epochs.
- DTM achieves 53.1 mIoU on ADE20K with ViT-B/16, 0.3 above the prior best (52.8 FD-CLIP / 52.7 BEiT v2 at 52.8 max).
- Bipartite-matching DTM costs about 1% throughput (1446 vs 1458 image/s) while improving fine-tuning from 84.3% to 85.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Dynamic Token Morphing | CLIP-supervised MIM with ViT-B/16 · ImageNet-1K | fine-tuning accuracy: 84.3 -> 85.4 | Helped |
| Add Dynamic Token Morphing | CLIP-supervised MIM with ViT-B/16 · ADE20K | mIoU: +0.3%p | Helped |
| Add token aggregation to CLIP-B/16 patch representations | CLIP-B/16 · ImageNet-1K | zero-shot image classification: 26.5 -> 30.8 (+4.3) | Helped |
| Add context-preserving token morphing | CLIP-distillation representation learning · ImageNet-1K | linear probing accuracy: 70.9 -> 72.2 (+1.3) | Helped |
| Replace token-wise supervision with downsampling | CLIP-distillation representation learning · ImageNet-1K | linear probing accuracy: 70.9 -> 68.8 (-2.1) | Hurt |
| Add DTM to SLIP supervision | SLIP with ViT-B/16 · ImageNet-1K | fine-tuning accuracy: 84.0 -> 84.5 (+0.5) | Helped |
| Add DTM to MAE with CLIP teacher | MAE + CLIP, ViT-B/16 · ImageNet-1K | fine-tuning accuracy: 82.6 -> 83.2 (+0.6) | Helped |
| Add DTM to BYOL | BYOL, ViT-B/16 · ImageNet-1K | fine-tuning accuracy: 81.7 -> 82.1 (+0.4) | Helped |
| Add dynamic scheduling to token morphing (ablation) | DTM with ViT-B/16 · ImageNet-1K | fine-tuning accuracy: 84.0 -> 85.4 | Helped |
| Use 2 morphing schedules instead of 1 (ablation) | DTM with ViT-B/16 · ImageNet-1K | fine-tuning accuracy: 84.8 -> 84.9 | Helped |
| Use 2 morphing schedules instead of 1 (ablation) | DTM with ViT-B/16 · ImageNet-100 | fine-tuning accuracy: 87.6 -> 87.9 | Helped |
| Fix the number of morphing tokens instead of varying it (ablation) | DTM with ViT-B/16 · ImageNet-100 | fine-tuning accuracy: 87.9 -> 87.4 | Hurt |
Surprising findings
- Target normalization slightly hurt DTM (87.9% to 87.7%), unlike its reported benefit for MAE.
- Using a single fixed number of morphed tokens or downsampling the target tokens degraded performance, with downsampling dropping linear probing by 2.1 points.
Limitations
- Only verified up to ViT-L/16; larger models such as ViT-G were not tested due to resources.
- K-means variant of DTM greatly reduces throughput (489 image/s).
Experimental setup
Datasets: ImageNet-1K ImageNet-100 ADE20K iNaturalist 2018 iNaturalist 2019 iNaturalist 2021-mini CUB-200 CIFAR-10 CIFAR-100 Birds Dogs
Benchmarks: ImageNet-1K ADE20K iNaturalist 2018 iNaturalist 2019 iNaturalist 2021-mini FGVC datasets
Metrics: top-1 accuracy mIoU linear probing accuracy zero-shot accuracy throughput
Baselines: BEiT v2 MVP DeepMIM FD-CLIP data2vec MAE iBOT dBOT LUT
Builds on: BEiT v2 MAE CLIP data2vec ToMe
Closest papers by meaning
- On the Role of Discrete Tokenization in Visual Representation Learning (ICLR 2024 · similarity 0.90)
- Revealing the Dark Secrets of Masked Image Modeling (CVPR 2023 · similarity 0.90)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.89)
- DDAE: Towards Deep Dynamic Vision BERT Pretraining (AAAI 2024 · similarity 0.87)
- Exploring Token Pruning in Vision State Space Models (NeurIPS 2024 · similarity 0.87)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.