Research map / Detection and segmentation / CNN features
Image BERT Pre-training with Online Tokenizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use online-tokenizer masked image modeling with [CLS] self-distillation | ViT-S/16 pre-training · ImageNet-1K linear probing | accuracy: 77.0 -> 77.9 | Helped |
| Use online-tokenizer masked image modeling with [CLS] self-distillation | ViT-B/16 pre-training · ImageNet-1K fine-tuning | accuracy: 83.6 -> 84.0 | Helped |
| Use iBOT pre-training instead of DINO | ViT-S/16 · ImageNet-1K, 1% labels | semi-supervised accuracy: 60.3 -> 61.9 | Helped |
| Use iBOT pre-training instead of DINO | ViT-B/16 · COCO detection | box AP: 50.1 -> 51.2 | Helped |
| Use iBOT pre-training instead of DINO | ViT-B/16 with UPerNet · ADE20K | mIoU: 46.8 -> 50.0 | Helped |
| Use iBOT pre-training instead of DINO | ViT-S/16 · ImageNet-C | mCE: 51.7 -> 48.1 | Helped |
| Add [CLS] self-distillation loss (ablation) | iBOT masked image modeling · ImageNet-1K tokenizer ablation | k-NN accuracy: 9.5 -> 69.1 | Helped |
| Add [CLS] self-distillation loss (ablation) | iBOT masked image modeling · ImageNet-1K tokenizer ablation | linear probing accuracy: 29.8 -> 74.2 | Helped |
| Share the [CLS] and patch projection heads (ablation) | iBOT · ImageNet-1K tokenizer ablation | linear probing accuracy: 73.8 -> 74.2 | Helped |
| Use a jointly learned online tokenizer instead of standalone DINO tokenizer (ablation) | MIM pre-training · ImageNet-1K tokenizer ablation | k-NN accuracy: 44.3 -> 69.1 | Helped |
| Merge semantic self-distillation with the MIM tokenizer rather than run BEiT and DINO as separate objectives (ablation) | ImageNet-1K pre-training · ImageNet-1K tokenizer ablation | linear probing accuracy: 62.7 -> 74.2 | Helped |
| Use ImageNet-22K instead of ImageNet-1K for pre-training (ablation) | ViT-L/16 frozen features · ImageNet-1K linear probing | accuracy: 81.0 -> 82.3 | Mixed |
Closest papers by meaning
- On the Role of Discrete Tokenization in Visual Representation Learning (ICLR 2024 · similarity 0.88)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.88)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.87)
- Masking meets Supervision: A Strong Learning Alliance (CVPR 2025 · similarity 0.86)
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.86)
- DDAE: Towards Deep Dynamic Vision BERT Pretraining (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.