Research map / Vision-language and multimodal / Multimodal LLMs

Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically studies how text-only LLM pre-training data (scale, source, reasoning vs. visual-world mixtures) gives rise to visual priors that transfer to MLLMs, and derives a vision-aware pre-training data mixture.

Problem

Understanding what gives text-only pre-trained LLMs latent visual priors that transfer to multimodal tasks, and how pre-training data composition shapes those priors.

Approach

Decomposes LLM visual priors into a reasoning prior (driven by reasoning-centric pre-training text) and a perception prior (more diffuse, from diverse web text, and more dependent on vision encoder and visual instruction tuning), and uses controlled pre-training mixtures to derive a data mix that balances language and vision performance.

Key results

• The Balanced 7B mixture (mix6-based, 1T tokens) reaches 38.64 overall VQA vs 37.32 for the language-favorable mixture, with lower perplexity (7.49 vs 8.72).
• Increasing code reasoning data from 0% to 100% raises logical soundness from 4.52% to 9.52% and reasoning depth from 8.31 to 53.25.
• Blind visual instruction tuning raises the Balanced model's overall VQA from 38.64 to 39.56.

Surprising findings

• Blind visual instruction tuning (text-only instruction tuning without images) improves VQA, indicating models can shortcut vision tasks via language priors.
• Frontier models (GPT-5 thinking, Gemini 2.5 Pro, Claude Opus 4.1) still hallucinate answers to questions when no image is provided.

Limitations

• Investigation limited to adapter-style MLLM architectures, not discrete tokenization or end-to-end joint training.
• Safety and bias of learned visual priors not investigated.
• Study limited to static images; video priors not examined.

Experimental setup

Datasets: Cambrian-7M WIT SA-1B ADE20K Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC Copa SciQA

Benchmarks: GQA MME MMBench SEED ScienceQA MMMU AI2D MathVista TextVQA ChartQA OCRBench RealWorldQA

Metrics: accuracy perplexity mutual nearest-neighbor alignment Spearman correlation logical soundness reasoning depth

Baselines: Language-Favorable mixture gpt-5 gpt-4o-mini gemini-2.5-flash claude-opus-4

Builds on: Cambrian-1 Web-SSL Llama-3 Platonic Representation Hypothesis LLaVA

Closest papers by meaning

• In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.93)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.93)
• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.92)
• Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.92)
• VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.92)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.91)