Research map / Vision-language and multimodal / Multimodal LLMs

VES-RFT: Rewarding Visual Evidence Sensitivity to Mitigate Hallucinations in Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VES-RFT, which rewards image-attributable entropy reduction (Visual Evidence Sensitivity) plus a verifiable object-correctness reward, optimized with critic-free GRPO and KL regularization to reduce object hallucination in VLMs.

Problem

Vision-language models over-rely on linguistic priors and hallucinate objects even when an image is provided; existing fixes are inference-time interventions or costly retraining with discrete preference labels.

Approach

Defines VES as the entropy gap between no-image and with-image predictive distributions over a task decision variable, converts its positive part into an annotation-free reward, and combines it with a verifiable object-matching reward inside GRPO so that confidence gains are both image-attributable and factually correct.

Key results

• On LLaVA-1.5, VES-RFT lowers CHAIRS from 55.6 to 42.8 and CHAIRI from 15.8 to 14.0 on MS-COCO CHAIR.
• On POPE average, VES-RFT reaches 86.96 accuracy on LLaVA-1.5 (baseline 82.04) and 88.93 on Qwen2.5-VL (baseline 84.84), using 2.8k pairs.
• Ablation shows removing the VES reward drops LLaVA-1.5 POPE accuracy from 86.96 to 86.03 and raises CHAIRS to 47.6.

Surprising findings

• Training cost is about 1.72x per iteration relative to SFT (83.12s vs 48.34s per step) due to the no-image branch and GRPO sampling.

Limitations

• Evaluated only on hallucination and captioning benchmarks, not broader VLM tasks due to resource limits.
• Verifier relies on closed-set object matching against annotations in the main experiments.
• Training is about 1.72x slower per iteration than SFT.

Experimental setup

Datasets: MS-COCO POPE AMBER

Benchmarks: POPE CHAIR AMBER

Metrics: Accuracy F1 Yes ratio CHAIRS CHAIRI Coverage HalRate Cognition

Baselines: VCD M3ID ICD RAR HIO Octopus SFT LLaVA-RLHF LURE OPERA DPO OPA-DPO

Builds on: GRPO DPO VCD M3ID OPA-DPO

Closest papers by meaning

• Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.93)
• On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models (NeurIPS 2025 · similarity 0.92)
• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.92)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.92)
• Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance (ICML 2025 · similarity 0.92)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.92)