Research map / Vision-language and multimodal / Multimodal LLMs
Beyond Fixed Biases: Decoding the Role of Reasoning Uncertainty in MLLM Modality Conflicts
Zhuoran Zhang, Tengyue Wang, Xilin Gong, Yang Shi, Haotian Wang, Di Wang, Lijie Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes MLLM modality-following under cross-modal conflicts into case-specific relative preference uncertainty and a stable inherent preference, showing that following a modality decreases as its relative preference uncertainty grows.
Problem
MLLMs must resolve conflicts when modalities provide contradictory information (modality following), but the factors driving which modality is followed are not well understood.
Approach
Splits modality-following behavior into a case-specific relative preference uncertainty term and a stable inherent preference term, and links the probability of following a modality to how uncertain the model is about its relative preference.
Key results
- The probability of following a modality consistently decreases as its relative preference uncertainty increases, across diverse MLLMs and benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| SFT on Text-easier subset instead of no training | LLaVA-1.5-7B · DriveBench long-form reasoning | overall score: 13.93 -> 22.88 | Improved |
| SFT on Text-easier subset instead of Vision-easier subset (ablation) | LLaVA-1.5-7B · DriveBench long-form reasoning | overall score: 18.25 -> 22.88 | Improved |
| SFT on Text-easier subset instead of no training | Qwen2.5-VL-7B · DriveBench long-form reasoning | overall score: 32.81 -> 45.35 | Improved |
| SFT on Text-easier subset instead of Vision-easier subset (ablation) | Qwen2.5-VL-7B · DriveBench long-form reasoning | overall score: 34.64 -> 45.35 | Improved |
| SFT on Text-easier subset instead of Base | LLaVA-1.5-7B · DriveBench MCQ perception (conflict) | accuracy %: 3.33 -> 50.00 | Improved |
| SFT on Text-easier subset instead of Base | Qwen2.5-VL-7B · DriveBench MCQ perception (conflict) | accuracy %: 6.67 -> 63.33 | Improved |
| SFT on Text-easier subset instead of Vision-easier subset (ablation) | LLaVA-1.5-7B · DriveBench MCQ perception (conflict) | accuracy %: 33.33 -> 50.00 | Improved |
| SFT on Text-easier subset instead of Vision-easier subset (ablation) | Qwen2.5-VL-7B · DriveBench MCQ perception (conflict) | accuracy %: 43.33 -> 63.33 | Improved |
| add misleading text caption to vision input | LLaVA-1.5-7B · DriveBench MCQ perception | accuracy %: 76.67 -> 3.33 | Hurt |
| add misleading text caption to vision input | Qwen2.5-VL-7B · DriveBench MCQ perception | accuracy %: 83.33 -> 6.67 | Hurt |
| patch oscillating middle-to-late layer activations with clear-condition activations (ablation) | MLLMs · modality conflict | modality-following rate: 70% reduction in oscillation; rates to 95% | Improved |
| train on easy Pos (vision-dominant) data for vision steering (ablation) | LLaVA-1.5-7B · Synthetic dataset Neg region | oscillation change: +164% | Hurt |
Experimental setup
Metrics: probability of modality following
Closest papers by meaning
- Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
- Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization (CVPR 2025 · similarity 0.89)
- Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.89)
- Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
- Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.88)
- Same Words, Different Judgments: How Preferences Vary Across Modalities (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
