Research map / Vision-language and multimodal / Multimodal LLMs

Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes language bias in LVLM training (VIT and DPO) and proposes LBR, a sequence-level L1 regularizer on text-only log-ratio during instruction tuning, and LBP, a sigmoid penalty on text-only log-ratio during DPO, reducing hallucinations on LLaVA-v1.5 models.

Problem

LVLMs over-rely on text priors and neglect visual inputs, causing hallucinations; prior work describes language bias qualitatively without a principled quantitative definition or mitigation tied to training dynamics.

Approach

Defines language bias as the text-only log-likelihood gain relative to a reference model and adds a regularizer (LBR) that penalizes its magnitude during visual instruction tuning, and a penalty (LBP) that pushes it negative during DPO, requiring no extra data or models.

Key results

• LBR improves LLaVA-1.5-7B on MME from 1490 to 1525 and MMMU from 35.7 to 37.2, with gains across most of over ten general benchmarks.
• LBP on LLaVA-v1.5-7B (RLHF-V) raises MMHalBench score from 2.11 (DPO) to 2.91 and cuts HalRate from 0.66 to 0.43, and reduces AMBER generative HalRate to 18.5.
• LBP reduces hallucination rate of LLaVA-v1.5-7B by 27% on MMHalBench according to the authors.

Surprising findings

• Text-only gain for preferred DPO responses exceeds the multimodal gain, indicating DPO can exacerbate language bias.
• LBR shows only modest gains on Object HalBench and MMHalBench, which the authors attribute to limitations of existing automated hallucination benchmarks.

Limitations

• LBR gains on automated hallucination benchmarks are modest.
• CHAIR metric is susceptible to incomplete ground truth and reward hacking, limiting evaluation reliability.
• LBR is sensitive to the regularization strength alpha.

Experimental setup

Datasets: LLaVA-v1.5 instruction tuning data RLHF-V VLFeedback COCO 2014 val

Benchmarks: MME MMBench SEED-Bench MMMU MMT-Bench MM-Star TextVQA ChartQA InfographicVQA OCRBench GQA ScienceQA

Metrics: accuracy F1 CHAIR hallucination rate CIDEr score

Baselines: LLaVA-v1.5 VIT baseline DPO DPO with margin loss V-DPO MFPO GPT-4V RLAIF-V OPA-DPO HSA-DPO mDPO

Builds on: DPO LLaVA-v1.5 V-DPO MFPO RLHF-V

Closest papers by meaning

• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.92)
• Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.91)
• Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.91)
• Calibrated Self-Rewarding Vision Language Models (NeurIPS 2024 · similarity 0.91)
• See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.91)
• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)