Research map / Detection and segmentation / CNN features
Mc-BEiT: Multi-Choice Discretization for Image BERT Pre-training
Xiaotong Li, Yixiao Ge, Kun Yi, Zixuan Hu, Ying Shan, Ling-Yu Duan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-choice soft targets and patch-similarity refinement (mc-BEiT vs BEiT) | BEiT ViT-B 800 ep · ImageNet-1K fine-tuning | top-1 accuracy: 83.2 -> 84.1 | Improved |
| add multi-choice objective (mc-BEiT vs BEiT) | BEiT ViT-L 800 ep · ImageNet-1K fine-tuning | top-1 accuracy: 85.2 -> 85.6 | Improved |
| use DALL-E tokenizer with multi-choice objective instead of single-choice (ablation) | BEiT, 100 epochs · ImageNet-1K | top-1 accuracy: 82.3 -> 82.6 | Improved |
| use VQGAN tokenizer with multi-choice objective instead of single-choice (ablation) | BEiT, 100 epochs · ImageNet-1K | top-1 accuracy: 82.9 -> 83.3 | Improved |
| use VQGAN tokenizer instead of DALL-E (ablation) | BEiT, 100 epochs · ImageNet-1K | top-1 accuracy: 82.3 -> 82.9 | Improved |
| use VQGAN tokenizer instead of DALL-E (ablation) | mc-BEiT, 800 epochs · ImageNet-1K | top-1 accuracy: 83.7 -> 84.1 | Improved |
| add tokenizer-prediction target (omega 0 -> 0.8) rather than only patch-similarity target (ablation) | mc-BEiT, 100 epochs · ImageNet-1K | top-1 accuracy: 81.8 -> 83.3 | Improved |
| use random 75% masking instead of block 75% (ablation) | mc-BEiT, 100 epochs · ImageNet-1K | top-1 accuracy: 82.8 -> 83.3 | Improved |
| raise random mask ratio from 75% to 90% (ablation) | mc-BEiT, 100 epochs · ImageNet-1K | top-1 accuracy: 83.3 -> 83.0 | Hurt |
| lower random mask ratio from 75% to 45% (ablation) | mc-BEiT, 100 epochs · ImageNet-1K | top-1 accuracy: 83.3 -> 83.0 | Hurt |
| increase pre-training from 100 to 800 epochs (ablation) | mc-BEiT ViT-B · ImageNet-1K | top-1 accuracy: 83.3 -> 84.1 | Improved |
| add intermediate ImageNet fine-tuning before ADE20K (ablation) | mc-BEiT ViT-B · ADE20K | mIoU: 47.0 -> 50.8 | Improved |
Closest papers by meaning
- Image BERT Pre-training with Online Tokenizer (ICLR 2022 · similarity 0.91)
- BEiT: BERT Pre-Training of Image Transformers (ICLR 2022 · similarity 0.91)
- Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling (ICLR 2023 · similarity 0.89)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.87)
- DDAE: Towards Deep Dynamic Vision BERT Pretraining (AAAI 2024 · similarity 0.87)
- On the Role of Discrete Tokenization in Visual Representation Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
