Research map / Label-efficient and robust learning / Self-supervised and contrastive
On the Role of Discrete Tokenization in Visual Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes how discrete tokenization shapes MIM's augmentation graph and downstream generalization, proposes the TCAS metric for tokenizer quality, and introduces ClusterMIM, which uses K-means cluster assignments as MIM reconstruction targets.
Problem
Discrete tokenizers used as reconstruction targets in masked image modeling perform very differently, yet their theoretical role in downstream generalization is not understood, and there is no training-free way to compare tokenizers.
Approach
Views the discrete token as an equivalence relation on masked views; discrete tokens aligned with true classes increase intra-class connectivity in the MIM augmentation graph. Proposes TCAS, a training-free token-class alignment metric, and ClusterMIM, which clusters patches (pixel space or DINO feature space) with K-means to produce label-correlated tokens without labels.
Key results
- ClusterMIM Pixel with ViT-S on ImageNet-100 reaches 52.7% linear probing and 86.4% fine-tuning, versus 45.9% and 81.8% for MAE.
- ClusterMIM DINO improves linear probing over the continuous MaskFeat variant by 2.6 points on ImageNet-100 (59.7 vs 57.1) and fine-tuning by 0.4 points (84.7 vs 84.3).
- TCAS correlates with linear probing accuracy across tokenizers with a reported linear-fit r-value of 0.85.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ClusterMIM Pixel instead of MAE | ViT-S · ImageNet-100 | LP Acc.: 45.9 -> 52.7 (+6.8%) | Helped |
| Use ClusterMIM Pixel instead of MAE | ViT-S · ImageNet-100 | FT Acc.: 81.8 -> 86.4 (+4.6%) | Helped |
| Use discrete DINO-feature tokens instead of continuous DINO features | ViT-S · ImageNet-100 | LP Acc.: +2.6% | Helped |
| Use discrete DINO-feature tokens instead of continuous DINO features | ViT-S · ImageNet-100 | FT Acc.: +0.4% | Helped |
| Use ClusterMIM Pixel instead of MAE | ViT-B · ImageNet-1K | LP Acc.: 55.4 -> 62.1 | Helped |
| Use ClusterMIM DINO instead of MaskFeat continuous | ViT-B · ImageNet-1K | LP Acc.: 63.2 -> 67.4 | Helped |
| Use K=10 rather than the reported best moderate K=50 for ClusterMIM Pixel (ablation) | ClusterMIM Pixel, ViT-S · ImageNet-100 | LP Acc.: 52.7 -> 42.7 | Hurt |
| Use K=25 rather than the reported best moderate K=50 for ClusterMIM Pixel (ablation) | ClusterMIM Pixel, ViT-S · ImageNet-100 | LP Acc.: 52.7 -> 48.3 | Hurt |
| Increase ClusterMIM Pixel clustering count from K=50 to K=1000 (ablation) | ClusterMIM Pixel, ViT-S · ImageNet-100 | LP Acc.: 52.7 -> 50.6 | Hurt |
| Train K-Means for 1 rather than 100 epochs (ablation) | ClusterMIM Pixel, ViT-S · ImageNet-100 | LP Acc.: 52.7 -> 47.2 | Hurt |
| Train K-Means for 10 rather than 100 epochs (ablation) | ClusterMIM Pixel, ViT-S · ImageNet-100 | LP Acc.: 52.7 -> 52.5 | Hurt |
| Train K-Means for 1 rather than 100 epochs (ablation) | ClusterMIM DINO, ViT-S · ImageNet-100 | FT Acc.: 84.3 -> 84.3 | No clear effect |
Surprising findings
- Class-wise-aligned tokens are predicted to help, but a cross-class tokenizer is predicted to hurt substantially, reversing the sign of the gain relative to MAE-like targets.
- Clustering-based tokens with only 1 epoch of K-means already outperform MAE (47.2/82.1 vs 45.9/81.8) on ImageNet-100.
Limitations
- Theoretical analysis relies on a toy model with two (or s) classes, small overlap assumption t << 1, and MSE loss simplification.
- TCAS is evaluated with ground-truth labels, which are unavailable in true self-supervised settings, and the correlation is shown on a limited set of tokenizers.
- Gains on fine-tuning for ClusterMIM DINO over MaskFeat are small (0.4 points).
Experimental setup
Datasets: ImageNet-100 ImageNet-1K
Benchmarks: ImageNet-100 ImageNet-1K
Metrics: linear probing accuracy fine-tuning accuracy TCAS
Baselines: MAE BEiT PeCo MaskFeat dVAE VQGAN DeiT
Builds on: MAE BEiT HaoChen et al. augmentation graph theory of MAE MaskFeat PeCo DINO
Closest papers by meaning
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.90)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.88)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.88)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.87)
- Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking (NeurIPS 2025 · similarity 0.87)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.