Research map / Vision-language and multimodal / Multimodal LLMs
Evaluating and Steering Modality Preferences in Multi-modal LLMs
Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the MC^2 benchmark, which builds controlled evidence-conflict scenarios to evaluate whether multi-modal LLMs prefer one modality over another, and studies how to steer that preference.
Problem
Whether multi-modal LLMs exhibit a tendency to favor one modality over another when processing multi-modal contexts remains insufficiently explored.
Approach
A benchmark (MC^2) built on controlled evidence-conflict scenarios, used to systematically measure modality preference in MLLMs and to steer it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add representation-engineering steering vector at probed layer instead of prompting | MLLMs · MC2 steering toward text | S_text (%): Qwen2.5VL-7B 35.4 -> 63.6 | Improved |
| add steering toward vision preference | Qwen2VL-7B · PhD-icc average | accuracy: 6.0 -> 18.4 | Improved |
| add few-shot prompting | Qwen2VL-7B · MC2 steering toward vision | S_vision (%): 15.3 -> 17.2 | No clear effect |
| apply token-wise steering (ICV) instead of a unified steering vector (ablation) | Qwen2VL-7B · PhD-icc / PhD-iac | accuracy: 13.6 on Phd-icc and 30.1 on Phd-iac | Hurt |
| steer toward vision preference on reasoning task | Qwen2.5VL-7B · MathVista | accuracy: 59.0 (InstDesign) -> 62.3 | Improved |
| add chain-of-thought prompting | Qwen2.5VL-7B · MathVista | accuracy: CoT 50.0 vs InstDesign 59.0 | Hurt |
| add instruction guidance toward vision or text | Qwen2.5VL-7B · MC2 Vision Ratio | Vision Ratio (%): 56.2 (Inst-text) / 59.6 (original) / 63.2 (Inst-vision) | Improved |
| increase model size within a family | Qwen2.5VL and InternVL3 · MC2 Vision Ratio | Vision Ratio | Improved |
| increase proportion of conflicting-context fine-tuning data (ablation) | Qwen2.5VL, LLaVA-OneVision · MC2 Vision Ratio | preference shift | Improved |
| higher Vision Ratio vs visual understanding ability | 10 MLLMs · seven visual benchmarks average | Spearman correlation: rho = 0.964 | Improved |
| steer text-dominant translation toward vision | Qwen2.5VL-7B · AmbigCaps En->Tr | BLEU: 9.45 -> 10.2 | Improved |
| use too large steering weight (ablation) | steering method · preliminary experiments | output quality | Hurt |
Experimental setup
Benchmarks: MC^2
Closest papers by meaning
- LLMs Can Evolve Continually on Modality for $\mathbb{X}$-Modal Reasoning (NeurIPS 2024 · similarity 0.89)
- Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.89)
- Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
