Research map / Vision-language and multimodal / Human and embodied modeling
Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BEiT-3 with unified masked data modeling and Multiway Transformers instead of the previous leading model | vision and vision-language transfer · ADE20K | mIoU (+MS): 61.4 -> 62.8 | Helped |
| Use BEiT-3 with unified masked data modeling and Multiway Transformers instead of the previous leading model | vision and vision-language transfer · NLVR2 test-P | accuracy: 87.00 -> 92.58 | Helped |
| Add intermediate finetuning on pretraining image-text pairs (ablation) | BEiT-3 retrieval model · COCO 5K test set | R@1 image-to-text: 82.7 -> 84.8 | Helped |
| Add intermediate finetuning on pretraining image-text pairs (ablation) | BEiT-3 retrieval model · COCO 5K test set | R@1 text-to-image: 65.1 -> 67.2 | Helped |
| Add intermediate finetuning on pretraining image-text pairs (ablation) | BEiT-3 retrieval model · Flickr30K 1K test set | R@1 image-to-text: 97.5 -> 98.0 | Helped |
| Add intermediate finetuning on pretraining image-text pairs (ablation) | BEiT-3 retrieval model · Flickr30K 1K test set | R@1 text-to-image: 89.1 -> 90.3 | Helped |
| Use masked data modeling to learn image-text alignment without image-text contrastive pretraining | BEiT-3 retrieval model · COCO and Flickr30K retrieval | R@1 | Helped |
Closest papers by meaning
- EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE (AAAI 2024 · similarity 0.89)
- On Scaling Up a Multilingual Vision and Language Model (CVPR 2024 · similarity 0.89)
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.89)
- Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.