Research map / Vision-language and multimodal / Visual question answering
Generative Multimodal Models are In-Context Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales a generative multimodal model (Emu2, 37B) trained with a unified next-element autoregressive objective over interleaved image/video/text, showing strong few-shot in-context learning and instruction-tuned chat and grounded generation variants.
Problem
Current multimodal systems struggle to perform task-agnostic in-context learning from a few demonstrations or simple instructions, as humans do.
Approach
Scale up a generative multimodal model (EVA-02-CLIP visual encoder, LLaMA-33B backbone, SDXL-based visual decoder) with a unified predict-the-next-element objective over large interleaved multimodal sequences, simplifying the Emu architecture by replacing the C-Former with mean pooling plus a linear projection.
Key results
- Emu2 (37B) achieves 68.8 VQAv2 and 57.0 VizWiz at 16-shot, outperforming Flamingo-80B and IDEFICS-80B on VQAv2, VizWiz and TextVQA with a smaller model.
- Emu2-Chat reaches 48.5 on MM-Vet and 703.8 on TouchStone, and 84.9 VQAv2 among the reported LMM results (with some training-set overlap noted).
- On zero-shot DreamBench single-image subject-driven generation, Emu2-Gen reaches DINO 0.766 and CLIP-I 0.850, above Kosmos-G (0.694 DINO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scaling to 37B parameters with few-shot in-context examples (4 to 16 shots) | Emu2 base model · VQAv2, VizWiz, TextVQA few-shot | accuracy | Helped |
| Increasing number of in-context shots | Emu2 base model · Few-shot VQA benchmarks | accuracy | Helped |
| Adding instruction tuning on conversational and academic data (Emu2-Chat) | Emu2 base model · MM-Vet, TouchStone, OKVQA, GQA etc. | score or accuracy | Helped |
| Fine-tuning on 500k high-quality pairs (quality fine-tuning) for generation | Emu2-Gen · MS-COCO zero-shot text-to-image | CLIP-I and CLIP-T | Mixed |
| Using a simpler encoder-LLM connection (mean pooling plus linear projection) instead of C-Former | Emu2 architecture · not ablated | n/a | No clear effect |
Surprising findings
- Zero-shot VQAv2 performance is very low (33.5) compared to few-shot (67.8 at 8-shot), indicating the model relies on demonstrations.
Limitations
- Zero-shot performance is weak on several tasks without in-context examples
- Struggles with visual prompting at smaller scale or zero-shot
- Potential misuse concerns for generative capabilities discussed in the paper
Experimental setup
Datasets: LAION-2B CapsFusion-120M WebVid-10M Multimodal-C4 YT-Storyboard-1B GRIT-20M CapsFusion-grounded-100M Pile LAION-COCO LAION-Aesthetics SA-1B Unsplash
Benchmarks: VQAv2 OKVQA VizWiz TextVQA Hateful Memes GQA MSVD-QA MSRVTT-QA SEED-Bench MM-Vet TouchStone MMMU
Metrics: accuracy CLIP-I CLIP-T DINO score
Baselines: Flamingo-80B IDEFICS-80B Kosmos-1 Emu LLaVA-1.5-13B CogVLM Qwen-VL-13B-Chat Kosmos-G
Builds on: Emu Flamingo LLaMA SDXL EVA-02-CLIP
Closest papers by meaning
- Emu: Generative Pretraining in Multimodality (ICLR 2024 · similarity 0.94)
- CoDi-2: In-Context Interleaved and Interactive Any-to-Any Generation (CVPR 2024 · similarity 0.88)
- Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (ICLR 2025 · similarity 0.88)
- OmniGen2: Towards Instruction-Aligned Multimodal Generation (CVPR 2026 · similarity 0.88)
- Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners (ICML 2026 · similarity 0.88)
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.