Research map / Vision-language and multimodal / Multimodal LLMs
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Cambrian-1 systematically studies vision encoders, connectors, instruction-tuning data and recipes for MLLMs, introducing the SVA connector, CV-Bench, and the Cambrian-10M/7M datasets, and releases a family of 8B/13B/34B models.
Problem
Vision components of multimodal LLMs are under-explored and disconnected from visual representation learning, and existing benchmarks poorly measure vision-centric capabilities, which hinders sensory grounding.
Approach
Uses MLLM instruction tuning as a vision-centric evaluation protocol for 23 vision backbones, proposes the Spatial Vision Aggregator (SVA) connector that cross-attends spatially-aligned features from multiple encoders across LLM layers to reduce visual tokens, and curates a balanced instruction-tuning data mix with a new vision-centric benchmark CV-Bench.
Key results
- Cambrian-1-34B reaches 76.8 general, 67.0 knowledge, 60.0 OCR and chart, and 74.0 CV-Bench 2D / 79.7 CV-Bench 3D, using only 576 visual tokens.
- Cambrian-1-8B reports an average vision-centric score of 72.0 on the CV-Bench-type suite, outperforming LLaVA-NeXT-8B (65.3) with 2880 tokens.
- Cambrian-7M improves average score from 54.8 (Cambrian-10M) to 55.9 and from 40.7 for LLaVA-665K under the same protocol.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Cambrian-1 instead of LLaVA-NeXT-8B | 8B MLLM with LLaMA-3-Instruct-8B backbone · OCR & Chart average | average score: 63.9 -> 71.3 | Helped |
| Use Cambrian-1 instead of LLaVA-NeXT-13B | 13B MLLM with Vicuna-1.5-13B backbone · Vision-Centric average | average score: 55.9 -> 62.2 | Helped |
| Use Cambrian-1 instead of LLaVA-NeXT-34B | 34B MLLM with Hermes2-Yi-34B backbone · OCR & Chart average | average score: 67.7 -> 71.9 | Helped |
| Add multi-layer aggregation to SVA (ablation) | SVA connector · OCR & Chart | category score: 55.2 -> 55.5 | Helped |
| Add multi-layer aggregation to SVA (ablation) | SVA connector · Vision-Centric | category score: 52.6 -> 53.2 | Helped |
| Use spatial aggregation instead of interpolation plus MLP (ablation) | OpenAI CLIP ViT-L token compression · OCR & Chart | category score: 28.1 -> 31.4 | Helped |
| Increase SVA depth from D=2 to D=4 (ablation) | SVA connector · OCR & Chart | category score: 52.1 -> 52.8 | Helped |
| Increase SVA query groups from G=1 to G=3 (ablation) | SVA connector · OCR & Chart | category score: 52.4 -> 53.1 | Helped |
| Curate Cambrian-10M into Cambrian-7M (ablation) | instruction-tuned MLLM · Average | average score: 54.8 -> 55.9 | Helped |
| Cap each Cambrian-10M data source at 250k or 350k rather than 150k (ablation) | data balancing procedure · Average | average score: 53.7 -> 54.3 | Helped |
| Increase adapter alignment data from 0M to 1.2M (ablation) | SigLIP ViT-SO400M/14@384 MLLM · Average | average score: 46.79 to 49.72 to 53.09 | Helped |
| Add system prompts to Cambrian-7M instruction examples (ablation) | Cambrian-7M model · benchmark performance | benchmark performance | No clear effect |
Surprising findings
- Language-supervised encoders dominate on most benchmarks, but a well-trained self-supervised DINOv2 gets competitive vision-centric results, and the gap narrows with more data and unfreezing.
- Several popular benchmarks (SQA-I, MMMU, MathVista, AI2D) show under 5% gap between vision-enabled and vision-disabled inputs, suggesting they rely heavily on the LLM.
Limitations
- Does not use any-resolution strategy, so ultra high-resolution or extreme aspect-ratio images are not handled well.
- Unfreezing the vision encoder is computationally costly and prevented testing some larger vision models under current sharding.
- Data curation and benchmark evaluation remain preliminary, and benchmarks do not capture conversational ability fully.
Experimental setup
Datasets: ShareGPT-4V LLaVA-665K COCO ADE20K Omni3D PathVQA ScienceQA
Benchmarks: MME MMBench SEED-Bench GQA ScienceQA-IMG MMMU MathVista AI2D ChartQA OCRBench TextVQA DocVQA
Metrics: accuracy
Baselines: LLaVA-NeXT Mini-Gemini-HD GPT-4V Gemini-1.5 Pro MM-1 Grok-1.5
Builds on: LLaVA-1.5 LLaVA-NeXT Perceiver resampler C-Abstractor LDPv2
Closest papers by meaning
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.92)
- Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation (NeurIPS 2025 · similarity 0.90)
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.90)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)
- LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (ICLR 2025 · similarity 0.90)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.