Research map / Vision-language and multimodal / Cross-modal learning

Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that merging a well-aligned vision MLLM into data-scarce audio or video MLLMs improves target performance, explains this via an MI lower bound on an Alignment score, and proposes DCAT, a closed-form layer-wise weight update using a small calibration set.

Problem

Improving an MLLM's capability for a data-scarce modality (audio, video) normally needs costly modality-specific training, and model merging is often infeasible because same-modality variants are rare. The paper asks whether a donor MLLM from another modality can transfer useful alignment to a recip

Approach

Derives a mutual-information lower bound monotonic in an Alignment score (SEO^γ · SD^(1-γ)) measuring how much modality-token energy lies in the text principal subspace and how spread it is; then DCAT solves a closed-form quadratic weight update combining a donor-guided text-alignment term with a norm-preserving spectral target in the text subspace.

Key results

• DCAT on Vicuna-based audio model reaches 56.20 overall on MMAU vs 48.90 original, and up to 100.92% gain on MuCho-music.
• On LLaMA 3.1-based video model, DCAT reaches 44.44 on Video-MME overall vs 38.19 original (about 16.4% gain) and 23.83% gain on MVBench.
• Alignment score correlates significantly with downstream performance across 114 checkpoints (Pearson and Spearman, p<1e-3); reported average gains of 29.55% on audio and 24.92% on video.

Surprising findings

• Merging a vision MLLM as donor into audio or video recipients improves recipient performance even though the modalities differ, while the vision model itself often degrades under merging.
• Standard merging methods at the default lambda=1.0 degrade LLaMA 3.1-based performance, so the paper reports lambda=0.2 for fair comparison.

Limitations

• Theorem relies on Gaussian assumptions and a spectral non-degeneracy factor delta that is only empirically checked.
• Mean-orthogonality assumption on text subspace is only approximately validated.
• Donor is restricted to vision models in experiments; generality to other donor modalities is conjectured rather than shown.

Experimental setup

Datasets: AVQA TextVQA

Benchmarks: MMAU AIR-Bench Foundation MuCho-music Clotho-AQA VocalSound Video-MME TempCompass MVBench MLVU TextVQA

Metrics: accuracy relative improvement Pearson correlation Spearman correlation Wilcoxon signed-rank p-value

Baselines: Task Arithmetic TIES-merging ISO-C TSV WUDI OptMerge

Builds on: Task Arithmetic TIES-merging ISO-C Task Singular Vector WUDI OptMerge

Closest papers by meaning

• DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.92)
• ModaVerse: Efficiently Transforming Modalities with LLMs (CVPR 2024 · similarity 0.92)
• SAE-V: Interpreting Multimodal Models for Enhanced Alignment (ICML 2025 · similarity 0.92)
• Learning Modality Knowledge Alignment for Cross-Modality Transfer (ICML 2024 · similarity 0.91)
• Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.91)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.91)