AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Beyond Fixed Biases: Decoding the Role of Reasoning Uncertainty in MLLM Modality Conflicts

Zhuoran Zhang, Tengyue Wang, Xilin Gong, Yang Shi, Haotian Wang, Di Wang, Lijie Hu

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Decomposes MLLM modality-following under cross-modal conflicts into case-specific relative preference uncertainty and a stable inherent preference, showing that following a modality decreases as its relative preference uncertainty grows.

Problem

MLLMs must resolve conflicts when modalities provide contradictory information (modality following), but the factors driving which modality is followed are not well understood.

Approach

Splits modality-following behavior into a case-specific relative preference uncertainty term and a stable inherent preference term, and links the probability of following a modality to how uncertain the model is about its relative preference.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
SFT on Text-easier subset instead of no trainingLLaVA-1.5-7B · DriveBench long-form reasoningoverall score: 13.93 -> 22.88Improved
SFT on Text-easier subset instead of Vision-easier subset (ablation)LLaVA-1.5-7B · DriveBench long-form reasoningoverall score: 18.25 -> 22.88Improved
SFT on Text-easier subset instead of no trainingQwen2.5-VL-7B · DriveBench long-form reasoningoverall score: 32.81 -> 45.35Improved
SFT on Text-easier subset instead of Vision-easier subset (ablation)Qwen2.5-VL-7B · DriveBench long-form reasoningoverall score: 34.64 -> 45.35Improved
SFT on Text-easier subset instead of BaseLLaVA-1.5-7B · DriveBench MCQ perception (conflict)accuracy %: 3.33 -> 50.00Improved
SFT on Text-easier subset instead of BaseQwen2.5-VL-7B · DriveBench MCQ perception (conflict)accuracy %: 6.67 -> 63.33Improved
SFT on Text-easier subset instead of Vision-easier subset (ablation)LLaVA-1.5-7B · DriveBench MCQ perception (conflict)accuracy %: 33.33 -> 50.00Improved
SFT on Text-easier subset instead of Vision-easier subset (ablation)Qwen2.5-VL-7B · DriveBench MCQ perception (conflict)accuracy %: 43.33 -> 63.33Improved
add misleading text caption to vision inputLLaVA-1.5-7B · DriveBench MCQ perceptionaccuracy %: 76.67 -> 3.33Hurt
add misleading text caption to vision inputQwen2.5-VL-7B · DriveBench MCQ perceptionaccuracy %: 83.33 -> 6.67Hurt
patch oscillating middle-to-late layer activations with clear-condition activations (ablation)MLLMs · modality conflictmodality-following rate: 70% reduction in oscillation; rates to 95%Improved
train on easy Pos (vision-dominant) data for vision steering (ablation)LLaVA-1.5-7B · Synthetic dataset Neg regionoscillation change: +164%Hurt

Experimental setup

Metrics: probability of modality following

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy