Research map / Vision-language and multimodal / Human and embodied modeling
BEiT: BERT Pre-Training of Image Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with masked image modeling to predict discrete visual tokens from masked patches | ViT-Base · ImageNet-1K | top-1 accuracy: 81.8 -> 83.2 | Helped |
| fine-tune at 384 resolution after 224-resolution training | BEiT-B · ImageNet-1K | top-1 accuracy: 83.2 -> 84.6 | Helped |
| intermediate fine-tune on ImageNet labels | BEiT · ADE20K | mIoU: 45.6 -> 47.7 | Helped |
| intermediate fine-tune on ImageNet-1K | BEiT · CIFAR-100 | top-1 accuracy: 90.1 -> 91.8 | Helped |
| remove blockwise masking (ablation) | BEiT · ADE20K | mIoU: 44.65 -> 42.93 | Hurt |
| replace visual-token prediction with masked-pixel reconstruction (ablation) | BEiT · ImageNet-1K | top-1 accuracy: 82.86 -> 81.04 | Hurt |
| replace visual-token prediction with masked-pixel reconstruction (ablation) | BEiT · ADE20K | mIoU: 44.65 -> 41.38 | Hurt |
| remove blockwise masking while using pixel reconstruction (ablation) | BEiT pixel-reconstruction variant · ADE20K | mIoU: 41.38 -> 37.09 | Hurt |
| recover 100% of visual tokens (ablation) | BEiT · ADE20K | mIoU: 44.65 -> 40.93 | Hurt |
| increase pre-training from 300 to 800 epochs (ablation) | BEiT · ImageNet-1K | top-1 accuracy: 82.86 -> 83.19 | Helped |
| add LayerScale (ablation) | ViT architecture · ADE20K | mIoU: 44.86 -> 45.43 | Helped |
| add relative position bias to the LayerScale architecture (ablation) | ViT architecture · ImageNet-1K | top-1 accuracy: 83.00 -> 83.22 | Helped |
Closest papers by meaning
- Image BERT Pre-training with Online Tokenizer (ICLR 2022 · similarity 0.89)
- PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (AAAI 2023 · similarity 0.88)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.86)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.85)
- DDAE: Towards Deep Dynamic Vision BERT Pretraining (AAAI 2024 · similarity 0.85)
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.