Research map / Vision-language and multimodal / Multimodal LLMs

Words or Vision: Do Vision-Language Models Have Blind Faith in Text?

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 12 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that VLMs exhibit a "blind faith in text" bias, over-trusting inconsistent textual context over images, which hurts accuracy under corrupted text; analyzes influencing factors, mitigates via text-augmented SFT, and gives a theory linking the bias to pure-text vs multimodal data imbalance.

Problem

VLMs receive inconsistent visual and textual inputs (matching, corrupted, or irrelevant text) in vision-centric tasks, but how they resolve modality conflicts and whether they are robust to corrupted text is underexplored.

Approach

Introduces a text preference ratio (TPR) metric over inconsistent image/text answers, using GPT-4o-generated matched and corrupted descriptions plus WikiText irrelevant passages, then analyzes factors (prompt, LLM size, relevance, token order, uni-modal certainty) and proposes text-augmented SFT with a theoretical account based on pure-text vs multimodal data imbalance.

Key results

• Qwen2-VL-7B accuracy on VQAv2, DocVQA, and MathVista under corruption falls to about 59%, 63%, and 52% of its base levels, roughly a 50% reduction.
• Claude Sonnet is most robust, with VQAv2 corruption accuracy 68.17% and TPR 9.58%, while open models like Molmo-7B-D show TPR around 59-60% on VQAv2 and brand detection.
• SFT with text augmentation raises LLaVA-NeXT-7B VQAv2 corruption accuracy from 28.69% to 71.25% and Qwen2-VL-7B corruption accuracy from 50.79% to 82.72%, with macro gains.

Measured effects reported in the paper

Surprising findings

• Text placed before image tokens increases text bias consistently, suggesting token order inherited from language models drives modality preference.
• Scaling LLaVA-NeXT from 7B to 34B reduces text bias only marginally, and the matched-vs-corrupted gap stays stable at larger scale.

Limitations

• Text variations for matched and corrupted cases are generated by GPT-4o and not human-verified in the main text.
• Theoretical analysis relies on specific assumptions and does not directly validate the imbalance mechanism empirically at scale.
• SFT experiments are limited to two 7B models and a small 1,000-sample training set.

Experimental setup

Datasets: VQAv2 DocVQA MathVista WikiText phishing brand detection dataset

Benchmarks: VQAv2 DocVQA MathVista brand detection

Metrics: accuracy text preference ratio normalized accuracy macro accuracy

Baselines: GPT-4o GPT-4o mini Claude Sonnet Claude Haiku LLaVA-NeXT-7B LLaVA-NeXT-13B LLaVA-NeXT-34B Phi3.5 Molmo-7B-D Qwen2-VL-7B

Builds on: LLaVA-NeXT BM25

Closest papers by meaning

• Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.91)
• Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.91)
• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
• See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.90)
• Can We Talk Models Into Seeing the World Differently? (ICLR 2025 · similarity 0.90)