Research map / Vision-language and multimodal / Multimodal LLMs

Perception Tokens Enhance Visual Reasoning in Multimodal Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces perception tokens (VQVAE-tokenized depth maps and bounding-box pixel tokens) and the AURORA curriculum training method so LLaVA-1.5-13B can reason via intermediate visual tokens, improving relative depth and counting benchmarks.

Problem

Multimodal language models struggle with low- and mid-level perception tasks such as depth ordering and counting, since language cannot express intermediate intrinsic representations; fine-tuning generalizes poorly and external tool use is costly.

Approach

Augment an MLM's vocabulary with perception tokens (VQVAE codebook indices for depth maps, discrete pixel-coordinate tokens for boxes) and train it with a curriculum of atomic token-prediction, chain-of-thought, and direct-labeling tasks so it can generate these tokens as intermediate reasoning steps.

Key results

• AURORA improves BLINK counting by +10.8 points over LLaVA 1.5 13B fine-tuned baseline (45.8 vs 35.0 base), CV-Bench counting +11.3 and SEED-Bench counting +8.3 over fine-tuned baseline.
• On HardBLINK relative depth, LLaVA-AURORA averages 61.6% vs 55.2% for fine-tuned LLaVA, surpassing GPT-4 Turbo and GPT-4 Turbo + Tool.
• Perception tokens beat text-number tokens on all three counting benchmarks (e.g., BLINK counting 45.8 vs 38.3).

Measured effects reported in the paper

Surprising findings

• Adding a VQVAE reconstruction loss gives only marginal gains over token cross-entropy alone (MSE 0.087 vs 0.092 on BLINK).
• LLaVA-AURORA does not surpass GPT-4o on counting (54.6 vs 64.6 on SEED-Bench) despite large gains over its base.

Limitations

• Does not surpass advanced closed-source models such as GPT-4o on counting tasks.
• Generated depth maps are imperfect and evaluated only on relative depth subtask images.
• Evaluated on only two perception task families (relative depth and counting) using a single base model, LLaVA 1.5 13B.

Experimental setup

Datasets: ADE20k LVIS BLINK CV-Bench SEED-Bench Visual Genome

Benchmarks: BLINK HardBLINK CV-Bench SEED-Bench

Metrics: accuracy mean squared error

Baselines: LLaVA 1.5 13B Fine-tuned LLaVA LLaVA OneVision GPT-4o GPT-4 Turbo GPT-4 Turbo + Tool Unified-IO Unified-IO 2

Builds on: LLaVA-1.5 AiT chain-of-thought prompting Depth Anything

Closest papers by meaning

• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.93)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.92)
• Latent Implicit Visual Reasoning (CVPR 2026 · similarity 0.92)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.91)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.91)