Research map / Vision-language and multimodal / Multimodal LLMs

Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Visual Information Gain (VIG), a perplexity-based log-ratio measuring how much an image reduces answer loss, and uses it to select top-70% samples and visually informative tokens for LVLM instruction tuning, improving vision and hallucination benchmarks with ~34-79% fewer active tokens.

Problem

LVLMs often suffer from language bias, answering from text priors rather than visual evidence, while existing mitigations (decoding, architecture changes, curated data) do not quantify how much each training sample or token depends on the image.

Approach

Define VIG as the log-ratio of answer perplexity with a blurred versus real image, decomposable to tokens, and use it as a data-centric signal to prune weakly grounded samples and mask low-VIG tokens from the loss during instruction tuning, reusing one threshold at both levels.

Key results

• VIG training on LLaVA-1.5 7B with 70% samples and token masking (38.45M active tokens, -34%) improves LLaVA-Bench from 59.02 to 61.22 and MMVet from 28.62 to 32.71, and reduces MMHal hallucination rate from 71.25 to 62.78.
• On LLaVA-1.5 13B, VIG training uses 79% fewer active tokens (12.14M) and still improves all benchmarks, e.g., CV-Bench from 60.16 to 62.89.
• Compared with training-free and training-based baselines on LLaVA-1.5 7B, VIG training alone gives gains on all vision benchmarks where LACING degrades DocVQA, and VIG combined with LACING gives the best MMVet (37.01) and MMBench (67.89).

Surprising findings

• Random 70% subsampling slightly degrades performance relative to the full vanilla model, while VIG-selected 70% subsets outperform it, with token-level selection giving further gains.
• Training on as few as ~12M active tokens for LLaVA-1.5 13B still improves all reported benchmarks.

Limitations

• Computing VIG requires additional forward passes over the dataset (about 6 hours on 8 RTX 4090 GPUs for the analysis setup).
• Empirical evaluation is limited to LLaVA-1.5, ShareGPT4V, and Open-Qwen2VL families and does not test other architectures or domains.
• The VIG score depends on a blurred image as the no-vision proxy, and the threshold is shared across sample and token levels without tuning, which may not be optimal.

Experimental setup

Datasets: LLaVA-665K instruction tuning data LLaVA 558K caption pretraining ShareGPT4V captions 1.2M MAmmoTH-VL 1M subset MS-COCO validation

Benchmarks: LLaVA-Bench (In-the-Wild) MM-Vet MMBench CV-Bench DocVQA POPE CHAIR MMHal-Bench VQAv2 (text corruption) GSM8K MMLU HellaSwag

Metrics: score accuracy F1 CHAIR_S CHAIR_I hallucination rate normalized corruption accuracy

Baselines: VCD PAI VAR LACING Random 70% subset

Builds on: LLaVA-1.5 ShareGPT4V Open-Qwen2VL visual contrastive decoding selective modeling for LLMs

Closest papers by meaning

• VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.91)
• A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.90)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.90)
• In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.90)
• Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.90)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)