Research map / Vision-language and multimodal / Human and embodied modeling
EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EVE, a unified multimodal Transformer with modality-aware sparse MoE, pre-trained only with masked signal modeling (masked pixel and masked text reconstruction) on public image-text pairs.
Problem
Vision-language pre-training is costly and hard to scale when it stacks many objectives (ITC, ITM, MIM, MLM), and unified shared-Transformer architectures ignore the modality gap between image and text.
Approach
Use a shared attention Transformer with Modality-Aware MoE (modality-specific router bias and hard/soft routing) and a single masked signal modeling objective that reconstructs raw image pixels and masked text tokens from visible signals, avoiding visual tokenizers, momentum teachers, and ITC/ITM.
Key results
- EVE-Base reaches 78.00 VQA test-dev and 83.93 NLVR2 test-P with 4M images, outperforming VLMO (76.64) and marginally BEiT3-Base (77.65 on test-dev, pre-trained on 3.1B).
- EVE-Base achieves COCO TR@1 79.6 and IR@1 62.0, improving over VLMO by 6.42 and 8.39 points on COCO retrieval without ITC/ITM pre-training.
- Masked signal modeling is reported to accelerate pre-training by 3.5x compared with pre-training that includes ITC and ITM under the same hardware.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace BEiTv2 token MIM targets with raw pixels (ablation) | EVE · NLVR2 | accuracy: 78.0 -> 79.7 | Helped |
| use MIM and MLM together rather than MIM alone (ablation) | EVE masked-signal pre-training · NLVR2 dev | accuracy: 57.2 -> 79.7 | Helped |
| use MIM and MLM together rather than simultaneously masking both modalities (ablation) | EVE masked-signal pre-training · NLVR2 dev | accuracy: 75.4 -> 79.7 | Helped |
| add ITC to MIM + MLM pre-training (ablation) | EVE · Flickr30K | TR: 94.4 -> 94.0 | Hurt |
| add ITM to MIM + MLM pre-training (ablation) | EVE · Flickr30K | TR: 94.4 -> 94.0 | Hurt |
| add ITC and ITM to MIM + MLM pre-training (ablation) | EVE · Flickr30K | TR: 94.4 -> 94.2 | Hurt |
| use a soft router instead of a hard router in deep layers (ablation) | EVE · Flickr30K | IR: 79.3 -> 81.2 | Helped |
| remove the load-balancing auxiliary loss (ablation) | EVE soft router · Flickr30K | IR: 81.2 -> 79.7 | Hurt |
| remove modality routing (ablation) | EVE · NLVR2 dev | accuracy: 80.3 -> 79.7 | Hurt |
| increase the number of selected experts from 1 to 2 (ablation) | EVE MoE · VQA and retrieval tasks | performance | Helped |
| increase expert count with top-k=1 (ablation) | EVE MoE · retrieval tasks | performance | Hurt |
| increase expert count with top-k=2 (ablation) | EVE MoE · VQA and retrieval tasks | performance | Helped |
Surprising findings
- Pixel reconstruction as MIM target outperforms BEiTv2 and DALL-E visual token targets.
- MLM with a high text masking ratio alone is sufficient; adding MIM gives small gains, and pure MIM yields minimal vision-language fusion.
Limitations
- Results on VQA and retrieval rely on fine-tuning with ITC/ITM losses and reranking, so the pre-training alone does not provide retrieval scoring.
- Pre-training data is limited to about 4M-21M public image-text pairs, far smaller than the billion-scale baselines it compares against.
- Shallow soft-router variant increases computational overhead.
Experimental setup
Datasets: MSCOCO Captions Visual Genome SBU Captions Conceptual Captions CC12M VQA 2.0 NLVR2 Flickr30K MSCOCO Karpathy split
Benchmarks: VQA 2.0 NLVR2 MSCOCO retrieval Flickr30K retrieval
Metrics: accuracy TR@1 IR@1 training time inference time
Baselines: ALBEF VLMO VL-BEiT METER MAMO BEiT3 SimVLM VLMAE MaskVLM BLIP
Builds on: BEiT v2 BERT ViT MAE LIMoE
Closest papers by meaning
- Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts (AAAI 2025 · similarity 0.91)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.86)
- DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.86)
- Efficient Vision-Language Pre-training by Cluster Masking (CVPR 2024 · similarity 0.86)
- ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.