Research map / Vision-language and multimodal / Multimodal LLMs

DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiM^3, a training-free method that merges multilingual and multimodal residuals in the shared LLM backbone of an MLLM using per-column magnitude and direction salience, keeping the vision encoder and projector fixed.

Problem

Extending an existing multimodal LLM to many languages usually needs costly multilingual multimodal data and retraining; the task is to inject multilingual capability training-free while keeping visual grounding.

Approach

Treats multilingual and multimodal task vectors as heterogeneous residuals and composes them column-wise with weights from rank-normalized softmax salience over magnitude and direction deviations (DoRA-style decomposition), averaged across the two branches, rather than uniform task-vector addition.

Key results

• On text-only multilingual benchmarks, DiM^3 reaches Avg_mul 63.38 on XCOPA, 64.82 on XStoryCloze, and 42.93 on XNLI, the best of all merging baselines on the first two.
• In the Qwen3 setting, DiM^3 raises Qwen3-VL-8B-Instruct Afri-MCQA average from 0.383 to 0.452.
• On general multimodal benchmarks, DiM^3-7B scores 34.86 on MMStar and 37.22 on MMMU versus 32.95 and 33.11 for Palo-7B.

Surprising findings

• Several existing merging baselines (e.g., PCB-Merging) become unstable and produce degenerate outputs on generative multilingual multimodal tasks such as MaXM.

Limitations

• Requires a reasonably compatible shared backbone; tokenizer or architecture mismatches may make composition unstable.
• Symmetric branch averaging and rank-based salience are heuristic design choices, validated only by ablation.

Experimental setup

Datasets: XCOPA XStoryCloze XNLI xMMMU MaXM MaRVL xGQA CVQA Afri-MCQA MMStar MMMU SEED-Bench-2-Plus

Benchmarks: XCOPA XStoryCloze XNLI xMMMU MaXM MaRVL xGQA CVQA Afri-MCQA MMStar MMMU SEED-Bench-2-Plus

Metrics: accuracy average multilingual score (Avg_mul)

Baselines: Task Arithmetic DARE TIES-Merging Breadcrumbs PCB-Merging STF NeuroMerging Palo-7B PLAST-7B

Builds on: Task Arithmetic DoRA TIES-Merging DARE

Closest papers by meaning

• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.90)
• LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages (AAAI 2025 · similarity 0.90)
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.90)
• Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.90)