Research map / Vision-language and multimodal / Multimodal LLMs

Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VC-STaR, a self-improving framework that uses contrastive VQA pairs (visually similar images with synonymous questions) to rectify visual hallucinations in VLM rationales, producing the 55K-sample VisCoR dataset for SFT.

Problem

Self-improving reasoning methods developed for text cannot verify or rectify visual hallucinations in VLM reasoning paths, so visual rationales remain unfaithful.

Approach

VLMs see visual cues more precisely when comparing a contrastive VQA pair; the method uses a think-contrast-rethink pipeline where a contrastive analysis from a similar image pair guides an LLM to rewrite the coarse rationale, with contrastive pairs curated by image and question embedding similarity and filtered to median-difficulty samples.

Key results

• VC-STaR on Qwen2.5-VL-7B reaches 58.1 average across six benchmarks vs 55.5 base, with MMVP 75.7 (+5.7) and Hallusion 56.3 (+3.2).
• VC-STaR outperforms reproduced self-improving baselines STaR (55.4 avg), Verifier (54.9), and Feedback (56.4) and off-the-shelf reasoning datasets such as LPT (56.9) and LLaVA-CoT (56.2).
• Gains generalize to Qwen2.5-VL-3B (Hallusion +6.3) and InternVL2.5-8B (Hallusion +7.2).

Surprising findings

• Adding easy samples to VisCoR-55K consistently hurts performance on Hallusion, MathVision, and MMStar.
• Existing self-improving methods improve hallucination benchmarks but degrade math and general benchmarks, while VC-STaR does not show this trade-off.

Limitations

• Performance on the proposed dataset is data-dependent; alternative pair curation strategies (HQ-Edit, DOCCI) yield biased gains.
• Rethinking relies on a large external LLM (Qwen2.5-72B) rather than the VLM itself.
• Gains on some benchmarks are small (e.g., MMStar +0.6, MathVision +1.3).

Experimental setup

Datasets: 21 VQA datasets spanning reasoning, graph/chart, math, general, and OCR GQA HQ-Edit DOCCI

Benchmarks: MMVP HallusionBench MathVista MathVision MMStar MME-RealWorld GQA

Metrics: accuracy

Baselines: STaR Verifier Feedback Virgo LLaVA-CoT R1-OV LPT VQA SFT

Builds on: STaR LLaMA-Factory GTE

Closest papers by meaning

• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.92)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.92)
• VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning (CVPR 2025 · similarity 0.91)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.91)
• Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.91)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.91)