Research map / Vision-language and multimodal / Multimodal LLMs

Deep Pre-Alignment for VLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Deep Pre-Alignment (DPA), which replaces a VLM's ViT encoder with a small VLM perceiver (Qwen3-0.6B language blocks) so visual features are aligned to the text space before entering the target LLM, reducing text forgetting and modality gap.

Problem

Standard VLMs map ViT features to the LLM via a lightweight projector, so visual features remain distant from the text space in early LLM layers; this wastes depth on modality alignment and causes text-capability forgetting.

Approach

Replace the ViT encoder with a small VLM whose internal language blocks progressively align visual tokens with text space upstream, so the target LLM receives text-aligned visual features and needs less destructive adaptation.

Key results

• On the 4B scale DPA-Qwen3-4B improves the multimodal average by 1.9 points over the controlled LLaVA-NeXT-Qwen3-4B baseline (51.2 to 53.1), and by 3.0 points at 32B (58.1 to 61.1).
• DPA reduces text-capability forgetting by 32.9% relative at 4B and 21.6% at 32B; MATH-500 rises from 36.4 to 54.2 for the 4B model.
• Gains persist with LLaMA-3.2-3B as target LLM (+3.6 all-average), and an untrained perceiver still yields about +3.5 overall improvement.

Surprising findings

• An untrained perceiver with randomly initialized projections still improves over the ViT-based baseline, and perceiver standalone performance is weakly related to final DPA performance.
• Cambrian-1-8B shows near-zero text-only scores largely due to empty or degenerate generations rather than wrong answers.

Limitations

• Text forgetting is only partially mitigated; DPA does not fully eliminate the degradation.
• Correlation between perceiver standalone performance and final DPA gains is weak, so the mechanism is not fully explained.
• 32B models trained with LoRA for stage-2 due to resource limits, and comparisons use controlled self-trained baselines rather than public checkpoints.

Experimental setup

Datasets: 558K LLaVA image-caption pairs 1M visual instruction samples

Benchmarks: MM-Vet MMStar SEED-Bench2-Plus MMMU MathVista MathVision OCRBench AI2D MATH-500 MMLU-Redux GPQA-Diamond

Metrics: accuracy average score per capability group modality gap (MIR) update density intrusion dimension throughput training FLOPs

Baselines: LLaVA-NeXT-Qwen3-4B LLaVA-NeXT-Qwen3-32B LLaVA-NeXT-LLaMA-3.2-3B LLaVA-1.5-7B LLaVA-NeXT-7B Qwen2-VL-2B Idefics2-8B Cambrian-1-8B

Builds on: LLaVA-NeXT Qwen3 LoRA

Closest papers by meaning

• Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.90)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.90)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
• Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.89)
• VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.89)
• The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss (ICLR 2026 · similarity 0.89)