Research map / Vision-language and multimodal / Multimodal LLMs

From Pixels to Words -- Towards Native Vision-Language Primitives at Scale

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NEO, a family of native vision-language models (2.2B and 9B) built from a unified primitive with Native-RoPE, mixed attention, and a pre-Buffer/post-LLM training scheme, trained on 390M image-text pairs.

Problem

Native VLMs (monolithic models without a separate pretrained visual encoder) lag modular VLMs; it is unclear what constraints separate them and how to build competitive native VLMs with accessible training.

Approach

A unified vision-language primitive that enlarges QK head dimensions to add separate H/W rotary dimensions (Native-RoPE with per-axis base frequencies), uses mixed causal/bidirectional attention per image, and trains with a randomly initialized pre-Buffer before a post-LLM initialized from Qwen3, then merges them end-to-end.

Key results

• NEO-2.2B (Qwen3-1.7B LLM) reaches 48.6 on MMMU and 76.0 on MMBench, matching or exceeding InternVL3-2B on several benchmarks and beating native baselines like HoVLE and OneCAT.
• NEO-9B scores 82.1 on MMBench and 54.6 on MMMU, competitive with Qwen2.5-VL-7B and InternVL3-8B among 8B-class models.
• Ablation: Native-RoPE with mixed attention reaches 44.0 average vs 39.1 for causal 1D-RoPE baseline; EVEv1.0/1.5/2.0 on the same Qwen3-1.7B reach 33.4/40.2/41.5.

Surprising findings

• NEO-9B does not outperform NEO-2B on DocVQA and InfoVQA, which the authors attribute to limits of the training corpus.
• Using a pre-Buffer trained on only 22M samples (PB3) is only 2.5% below the full NEO model on average.

Limitations

• Lags on knowledge- and OCR-heavy benchmarks such as MMMU, InfoVQA, and TextVQA, attributed to limited training data.
• Training data scale and compute are limited compared with top modular VLMs, so the full potential is not reached.
• Focuses on dense image-text models; video and generation applications are not evaluated.

Experimental setup

Datasets: LAION-400M COYO-700M BLIP3o long captions OpenImages LAION-COCO Wukong InternVL-1.5 pretraining corpus Mono-InternVL SFT instructions

Benchmarks: MMMU MMBench-EN MMVet MMStar SEED-I POPE HallusionBench AI2D DocVQA ChartQA InfoVQA TextVQA

Metrics: accuracy

Baselines: Qwen2-VL InternVL2.5 InternVL3 Qwen2.5-VL Mono-InternVL HoVLE OneCAT EVE EVEv2 SAIL BREEN VoRA

Builds on: Qwen3 Mono-InternVL EVE M-RoPE FlexAttention

Closest papers by meaning

• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.92)
• VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.91)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.91)
• Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.91)
• Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.91)
• What matters when building vision-language models? (NeurIPS 2024 · similarity 0.90)