Research map / Vision-language and multimodal / Multimodal LLMs

ViCToR: Improving Visual Comprehension via Token Reconstruction for Pretraining LMMs

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ViCToR, a pretraining stage for LMMs that swaps random visual tokens for entries from a learnable visual token pool matched via the Hungarian algorithm, and trains with a visual token reconstruction loss plus dense caption supervision.

Problem

LMMs suffer a modality representation gap during pretraining: language embeddings are stable while visual representations are sensitive to contextual noise, so the LLM struggles to internalize visual structure.

Approach

Replace a random proportion (75%) of visual tokens with entries from a learnable visual token pool, assigned by Hungarian matching for better pool utilization, and have the LLM reconstruct the original visual tokens while also generating detailed captions, forming an extra visual comprehension pretraining stage between alignment and instruction tuning.

Key results

• ViCToR-7B improves over LLaVA-NeXT-8B by 10.4% on MMStar, 3.2% on SEED-I, and 7.2% on RealWorldQA (abstract).
• With CLIP-L/14 and Vicuna-1.5-7B under LLaVA-1.5 data (1.2M pretrain), ViCToR-7B reaches 70.9 on AI2D and 41.4 on MMStar versus 37.6 for LLaVA-NeXT-7B.
• Hungarian matching outperforms nearest-neighbor assignment (MME 2071 vs 1919; RealWorldQA 65.6 vs 63.2).

Surprising findings

• Pixel reconstruction yields low training loss but worse downstream performance than feature reconstruction (MME 1764 vs 2071).
• Initializing the token pool with VQ-GAN or K-means features does not help and sometimes degrades performance.

Limitations

• Only static image token reconstruction is studied; video (spatial-temporal) reconstruction is not addressed.
• Gains reported against baselines that differ in resolution and training data, so comparisons are not fully controlled in the main table.

Experimental setup

Datasets: LLaVA-1.5 558K LLaVA-ReCap-CC3M LLaVA-NeXT 780K instruction data

Benchmarks: MMStar RealWorldQA MMBench OCRBench POPE MMMU AI2D MME SEED-Bench-I

Metrics: accuracy score

Baselines: LLaVA-1.5-13B LLaVA-NeXT-8B Cambrian-13B IDEFICS2-8B Mantis-8B Ross VILA1.5 ShareGPT4V LLaVA-NeXT-7B

Builds on: LLaVA-1.5 LLaVA-NeXT Masked Autoencoders Hungarian algorithm

Closest papers by meaning

• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• Beyond Text: Frozen Large Language Models in Visual Signal Comprehension (CVPR 2024 · similarity 0.90)
• Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference (ICML 2026 · similarity 0.90)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.89)
• Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information (AAAI 2025 · similarity 0.89)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.89)