AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Evaluating and Steering Modality Preferences in Multi-modal LLMs

Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the MC^2 benchmark, which builds controlled evidence-conflict scenarios to evaluate whether multi-modal LLMs prefer one modality over another, and studies how to steer that preference.

Problem

Whether multi-modal LLMs exhibit a tendency to favor one modality over another when processing multi-modal contexts remains insufficiently explored.

Approach

A benchmark (MC^2) built on controlled evidence-conflict scenarios, used to systematically measure modality preference in MLLMs and to steer it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add representation-engineering steering vector at probed layer instead of promptingMLLMs · MC2 steering toward textS_text (%): Qwen2.5VL-7B 35.4 -> 63.6Improved
add steering toward vision preferenceQwen2VL-7B · PhD-icc averageaccuracy: 6.0 -> 18.4Improved
add few-shot promptingQwen2VL-7B · MC2 steering toward visionS_vision (%): 15.3 -> 17.2No clear effect
apply token-wise steering (ICV) instead of a unified steering vector (ablation)Qwen2VL-7B · PhD-icc / PhD-iacaccuracy: 13.6 on Phd-icc and 30.1 on Phd-iacHurt
steer toward vision preference on reasoning taskQwen2.5VL-7B · MathVistaaccuracy: 59.0 (InstDesign) -> 62.3Improved
add chain-of-thought promptingQwen2.5VL-7B · MathVistaaccuracy: CoT 50.0 vs InstDesign 59.0Hurt
add instruction guidance toward vision or textQwen2.5VL-7B · MC2 Vision RatioVision Ratio (%): 56.2 (Inst-text) / 59.6 (original) / 63.2 (Inst-vision)Improved
increase model size within a familyQwen2.5VL and InternVL3 · MC2 Vision RatioVision RatioImproved
increase proportion of conflicting-context fine-tuning data (ablation)Qwen2.5VL, LLaVA-OneVision · MC2 Vision Ratiopreference shiftImproved
higher Vision Ratio vs visual understanding ability10 MLLMs · seven visual benchmarks averageSpearman correlation: rho = 0.964Improved
steer text-dominant translation toward visionQwen2.5VL-7B · AmbigCaps En->TrBLEU: 9.45 -> 10.2Improved
use too large steering weight (ablation)steering method · preliminary experimentsoutput qualityHurt

Experimental setup

Benchmarks: MC^2

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy