Research map / Vision-language and multimodal / Multimodal LLMs

Exploring Visual Pretraining for Learning Language Intelligence

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MAPLE, a visual pretraining paradigm that jointly trains LLMs on masked autoregressive modeling of document page image latents and standard text continued pretraining, reporting gains over text-only pretraining on math benchmarks.

Problem

Whether LLMs can be pretrained on visual document corpora (raw page images) to reach or exceed the language intelligence obtained from text pretraining on the same corpora, expanding usable data beyond text extraction.

Approach

MAPLE augments an LLM with an autoregressive stream over foreground patch latents from a frozen VAE and masked autoregressive (MAR) encoder/decoder, so the LLM predicts latent hypotheses for future document patches and reconstructs pages, while sharing weights with ordinary text CPT; no image-text pairs are needed.

Key results

• MAPLE improves over text pretraining on base models, e.g., InternLM2-1.8B MATH-500 10.84 vs 8.30 for TP and GSM8K 36.23 vs 18.20.
• On Llama3.1-8B, MAPLE reaches MATH-500 34.82 vs 24.24 for text pretraining; the paper reports relative gains up to 40.2% on average accuracy.
• Image-side loss correlates with MATH-500 accuracy with |r| up to about 0.9; CKA shows text-image layer alignment emerges after MAPLE training.

Surprising findings

• Visual pretraining on document page images outperforms text pretraining on OCR-extracted text, even though the text stream is shared.
• Without the LLM in the loop, page reconstruction is severely degraded, yet with it the reconstructions become readable.

Limitations

• Evaluation limited to math reasoning benchmarks and small base models (1.5B-8B).
• Requires a large compute budget (128 A100 GPUs, ~40B tokens) and a proprietary-style CC-PDF corpus that is not released.
• Some comparisons are mixed (TP leads on one Qwen2.5-1.5B MMLU-Pro-Math entry).

Experimental setup

Datasets: OpenWebMath AutoMathText FineMath-3Plus InfiMM-WebMath CC-PDF math PDF corpus AM-Thinking-v1 math subset

Benchmarks: GSM8K MATH-500 OlympiadBench MMLU-Pro (math subset) AIME-24

Metrics: Pass@1 (4-shot) Pass@1 (0-shot) PSNR SSIM Pearson/Spearman correlation CKA

Baselines: Text pretraining (TP) Base models

Builds on: Masked Autoregressive models (MAR) Harmon Platonic Representation Hypothesis InternVL-style tiling FineMath filters

Closest papers by meaning

• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
• VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.90)
• LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs (ICML 2026 · similarity 0.89)
• Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.88)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.88)
• A Vision Check-up for Language Models (CVPR 2024 · similarity 0.88)