Research map / Vision-language and multimodal / Multimodal LLMs
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLHF-V, which collects 1.4k fine-grained segment-level corrections of hallucinations and trains Muffin-13B with a dense DPO variant (DDPO) to reduce MLLM hallucination.
Problem
MLLMs frequently generate text not grounded in the image (hallucinations), and coarse overall-ranking RLHF feedback is ambiguous and data-hungry for aligning multimodal behavior.
Approach
Annotators directly correct hallucinated segments of model responses, yielding segment-level preference pairs; DDPO extends DPO by weighting corrected segments more heavily (gamma>1) in the sequence log-likelihood score, and the paper also removes image cropping and adds VQAv2 instruction tuning to reduce vision-language mismatch.
Key results
- RLHF-V reduces Muffin's overall object hallucination on MHumanEval by 34.8% (74.7 to 55.5) using 1.4k preference samples, outperforming LLaVA-RLHF trained on 10k.
- On Object HalBench, RLHF-V reaches 12.2% response-level hallucination rate versus 25.9% for the best open-source baseline InstructBLIP.
- Training takes less than 1 hour on 8 A100 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Dense DPO (DDPO) with segment-level weighting instead of vanilla DPO (ablation) | RLHF-V on Muffin · MHumanEval | All-type response-level hallucination rate: 57.5 -> 55.5 | Helped |
| Fine-tuning on VQAv2 only (no preference learning) (ablation) | Muffin · MHumanEval | All-type response-level hallucination rate: 74.7 -> 65.1 | Helped |
| Adding image cropping (untrustworthy augmentation) to training (ablation) | RLHF-V · MHumanEval | All-type response-level hallucination rate: 55.5 -> 59.6 | Hurt |
| Adding image cropping augmentation to training (ablation) | RLHF-V · VQAv2 | testdev accuracy: 80.0 -> 77.1 | Hurt |
| Replacing RLHF-V fine-grained correctional data with overall-ranking LLaVA-RLHF data | RLHF-V · MHumanEval hallucination vs data amount | hallucination rate | No clear effect |
| Instruction-tuning Muffin-based model on GPT-4V distilled responses (ablation) | RLHF-V · MHumanEval | All-type response-level hallucination rate: 55.5 -> 75.3 | Hurt |
| Applying RLHF-V data and method to LLaVA | LLaVA · hallucination benchmarks | hallucination count and rate: hallucination count reduced by 13.8 relative points, rate by 5.9 relative points | Helped |
Surprising findings
- Vanilla DPO on the same data degrades performance relative to DDPO (MHumanEval all 57.5 vs 55.5), and including image cropping augmentation hurts both hallucination and VQAv2.
- Distilling GPT-4V responses into the model substantially increases hallucination (MHumanEval all 75.3 vs 55.5).
Limitations
- Evaluation relies partly on GPT-4/ChatGPT judges and small human-annotated benchmarks (MHumanEval has 146 samples).
- Gains are measured mainly on hallucination benchmarks; helpfulness is comparable rather than improved.
- Hallucination on long-form outputs remains substantial relative to GPT-4V on some metrics.
Experimental setup
Datasets: VQAv2 COCO
Benchmarks: Object HalBench MHumanEval MMHal-Bench LLaVA Bench VQAv2
Metrics: response-level hallucination rate mention-level hallucination rate hallucination rate by type informativeness score VQAv2 accuracy
Baselines: LLaVA Muffin LRV LLaVA-RLHF InstructBLIP Qwen-VL-Chat LLaVA 1.5 GPT-4V
Builds on: DPO Muffin RLHF
Closest papers by meaning
- Automated Multi-level Preference for MLLMs (NeurIPS 2024 · similarity 0.91)
- Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.91)
- IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination (ICML 2026 · similarity 0.90)
- Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization (AAAI 2026 · similarity 0.90)
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.90)
- Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.