Research map / Vision-language and multimodal / Multimodal LLMs

UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UVU, a vision-language unified autoregressive MLLM that adds pixel-level visual supervision via a 200K pixel-patch codebook (built by iterative hierarchical clustering) during pre-training, avoiding VQ tokenization and external decoders.

Problem

Visual supervision in MLLMs is usually added post-training, when visual representations are already fixed, so it acts only as an auxiliary constraint; AR+VQ visual supervision also appears to hurt multimodal understanding.

Approach

Replace VQ tokens with a pixel-level codebook of 200K cluster centers over raw 32x32x3 patches (built with positional encoding, deduplication, hierarchical sampling and iterative Faiss K-means), and train an autoregressive LLM to predict these pixel-patch tokens jointly with text during pre-training, with continuous SigLIP-2 input features.

Key results

• UVU (3B) reaches 91.8 on RefCOCO vs 84.1 for Qwen2.5-VL-3B, and 71.7 on LISA-Grounding vs 57.4.
• Ablation: AR+VQ visual supervision lowers MMStar from 52.9 to 46.5 and RefCOCO from 85.6 to 79.6 relative to no visual supervision.
• Codebook size 200K gives MMBench 76.1 and RefCOCO 91.8; 500K gives only marginal gains (76.4, 91.7), so 200K is chosen.

Surprising findings

• Naive AR+VQ visual supervision substantially degrades understanding relative to no visual supervision, while the pixel-level codebook approach improves it.
• The PSC metric stays nearly flat (about 0.34) across codebook sizes from 50K to 500K even as PRP rises, which the paper reads as robustness of the clustering.

Limitations

• Ablations and comparisons use a single 3B backbone; no scaling results are reported.
• Reconstruction quality and generation are shown only qualitatively; the paper gives no quantitative generation metrics.
• Key design claims (gradient orthogonality, manifold argument) are supported by limited analysis and some numbers are inconsistent (e.g., PSC defined as a min-max over clusters divided by active count).

Experimental setup

Datasets: LLaVA-OneVision FineVision Cauldron Cambrian-7M proprietary dataset

Benchmarks: SEED-Bench VisuLogic MMStar MME CVBench-2D CVBench-3D VLMBlind RefCOCO LISA-Grounding ScienceQA BLINK HallusionBench

Metrics: accuracy score PSC PRP

Baselines: Qwen2.5-VL GLM-4v InternVL2 LLaVA-OV DeepSeek-VL ShareGPT4V LLaVA-v1.5 AR+VQ

Builds on: VQVAE SigLIP-2 Faiss Qwen2.5

Closest papers by meaning

• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.92)
• Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.91)
• MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.91)
• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.90)