Research map / Vision-language and multimodal / Multimodal LLMs
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Molmo, a family of open-weight VLMs (1B–72B) trained on the new PixMo datasets collected without any VLM-generated data, achieving near-GPT-4o human preference at 72B.
Problem
Strongest open-weight vision-language models depend on synthetic data distilled from proprietary VLMs, leaving the community without knowledge of how to build performant VLMs from scratch with open data.
Approach
Collect fully human-annotated and non-VLM-synthetic datasets (speech-based dense captions, human-edited QA, 2D pointing data) and pair them with a simple two-stage training recipe with overlapping multi-crop tiling, attention pooling of vision tokens, length-conditioned captioning, and pointing-based counting.
Key results
- Molmo-72B achieves the highest academic benchmark average (81.2) among compared models, including GPT-4o (78.5) and Claude 3.5 Sonnet (76.7).
- Molmo-72B ranks second in human Elo (1077) behind GPT-4o (1079), ahead of Gemini 1.5 Pro (1074) and Claude 3.5 Sonnet (1069).
- MolmoE-1B nearly matches GPT-4V on academic average (68.6 vs 71.1) and Elo (1032 vs 1041).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use point-then-count instead of count-only (ablation) | Molmo counting model · CountBenchQA test | accuracy: 87.9 -> 89.4 | Helped |
| use point-then-count instead of count-only (ablation) | Molmo counting model · PixMo-Count validation | accuracy: 80.2 -> 86.3 | Helped |
| train with top-down, left-right ordered points (ablation) | Molmo counting model · CountBenchQA test | accuracy: 85.4 -> 89.4 | Helped |
| train with top-down, left-right ordered points (ablation) | Molmo counting model · PixMo-Count validation | accuracy: 74.1 -> 86.3 | Helped |
| use plain-text coordinates instead of special point tokens (ablation) | Molmo counting model · CountBenchQA test | accuracy: 85.8 -> 89.4 | Helped |
| use plain-text coordinates instead of special point tokens (ablation) | Molmo counting model · PixMo-Count validation | accuracy: 80.9 -> 86.3 | Helped |
| use multiple overlapping crops instead of one crop (ablation) | Molmo-7B-D · 11-benchmark suite | 11-avg: 62.8 -> 76.9 | Helped |
| add length hints during caption pre-training (ablation) | Molmo-7B-D · Captioning evaluation | cap F1: 53.0 -> 54.1 | Helped |
| use mean-query attention pooling instead of feature stacking (ablation) | Molmo-7B-D · 11-benchmark suite | 11-avg: 76.1 -> 76.9 | Helped |
| increase PixMo-Cap training images from 0 to 712k (ablation) | Molmo-7B-D · 11-benchmark suite | 11-avg: 74.9 -> 76.9 | Helped |
| remove PixMo-Docs from fine-tuning (ablation) | Molmo-7B-D · 11-benchmark suite | 11-avg: 76.9 -> 75.8 | Hurt |
| remove the pointing task from fine-tuning (ablation) | Molmo-7B-D · 11-benchmark suite | 11-avg: 76.9 -> 76.2 | Hurt |
Surprising findings
- Human-annotated PixMo-Cap captions perform on par with GPT-4o-generated captions on the same images in pretraining ablations.
- Training without a connector-only pretraining stage works, and noisy web-scale data does not improve pretraining.
Limitations
- Weaker on reasoning benchmarks (MMMU, MathVista) likely due to training mix lacking advanced reasoning data.
- Counting and pointing trained only up to 40 counts due to sequence length limits.
- Performance on academic benchmarks depends on evaluation details; Qwen2-VL scores strongly on benchmarks but underperforms in human evaluation.
Experimental setup
Datasets: PixMo-Cap PixMo-AskModelAnything PixMo-Points PixMo-CapQA PixMo-Docs PixMo-Clocks PixMo-Count VQA v2.0 TextVQA ChartQA DocVQA AI2D
Benchmarks: AI2D ChartQA VQA v2.0 DocVQA InfoVQA TextVQA RealWorldQA MMMU MathVista CountBenchQA PixMo-Count
Metrics: accuracy Elo score cap F1
Baselines: GPT-4o GPT-4V Gemini 1.5 Pro Claude 3.5 Sonnet Qwen2-VL-72B LLaVA OneVision-72B InternVL2-Llama-3-76B Llama-3.2V-90B-Instruct
Builds on: LLaVA CLIP OLMo Qwen2 OLMoE
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.90)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.90)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.89)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.