Research map / Vision-language and multimodal / Multimodal LLMs

DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeepAlign, a post-training framework that trains lightweight adapters to shift visual representations toward the text space (representation intervention) and distills DINOv2 patch-similarity structure into MLLM visual hidden states, improving LLaVA-v1.5-7B, Qwen2.5-VL-7B, and InternVL3-8B.

Problem

Modality misalignment in MLLMs: visual and textual representations are misaligned in modality-specific components, and autoregressive text-only supervision depletes modality-specific visual details, causing modality conflict.

Approach

Decouple modality-specific components via Grad-CAM-style gradient weighting of a modality classifier, compute instance- and global-level modality shift directions, train an adapter to apply the shift to visual states, and add a structure-induced loss that matches intra-image patch similarity of MLLM visual states to DINOv2's similarity structure.

Key results

• On Qwen2.5-VL-7B, DeepAlign improves MMBench from 82.2 to 83.2, MMStar from 64.1 to 65.7, and TextVQA from 76.7 to 78.4.
• On LLaVA-v1.5-7B, DeepAlign raises MMStar from 34.6 to 40.1 and MMVet from 32.2 to 38.5.
• On BLINK perceptive tasks, DeepAlign lifts Qwen2.5-VL-7B's average from 65.0 to 69.2.

Surprising findings

• Replacing the image with a BLIP-2 dense caption does not notably hurt LLaVA-v1.5 on perceptive VQA tasks, which the authors read as evidence of modality-shared-only comprehension.
• Before DeepAlign, LLaVA-v1.5 performs worse with visual input than text-only on MNER, MMT and MRE.

Limitations

• Gains on strong backbones are modest (often about 1 point), and the paper does not report variance across runs.
• Training data size and compute cost are not stated in the main text; details are deferred to the appendix.
• Adapter module is about 200M parameters, which is not small relative to the backbone.

Experimental setup

Datasets: CC3M COCO Caption MMLU Wikipedia text segments ImageNet

Benchmarks: MMBench MMStar MMMU HallusionBench OCRBench MMVet ScienceQA TextVQA RealWorldQA MTVQA BLINK OKVQA

Metrics: accuracy perplexity

Baselines: RLHF HADPO DataTailor POVID SIMA VISTA

Builds on: Grad-CAM DINOv2 LLaVA-v1.5

Closest papers by meaning

• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.92)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.91)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.90)
• Mind the Heads: Topological Representation Alignment for Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• SAE-V: Interpreting Multimodal Models for Enhanced Alignment (ICML 2025 · similarity 0.90)