Research map / Large language models / AI and society
RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLAIF-V, a fully open-source AI-feedback framework that builds preference data with deconfounded sampling and claim-level divide-and-conquer scoring, trains MLLMs with iterative DPO, and uses the DPO-aligned model's length-normalized reward for best-of-N inference-time scaling.
Problem
Hallucination in multimodal LLMs is commonly reduced with feedback learning, but high-quality feedback has relied on costly human annotation or proprietary models like GPT-4V; open-source MLLMs lack a recipe to produce reliable feedback, and inference-time use of feedback is underexplored.
Approach
Generate candidate responses under identical decoding conditions to remove style confounds, score each response by splitting it into atomic claims converted to yes/no questions answered by an open-source labeler, iterate DPO with refreshed feedback, and reuse the aligned model's length-normalized implicit reward for best-of-N selection.
Key results
- RLAIF-V 7B reduces Object HalBench object hallucination by 80.7% relative to LLaVA 1.5 7B (54.5 to 10.5 response-level).
- RLAIF-V 12B reaches 35.6% overall hallucination on MHumanEval, surpassing GPT-4V's 13.6%-level comparison in trustworthiness win rate and other benchmarks, with self-alignment using OmniLMM 12B as labeler.
- Deconfounded sampling plus divide-and-conquer labeling yields 96.7% human agreement on constructed pairs versus 66.7% without divide-and-conquer (Table 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align LLaVA 1.5 7B with open-source LLaVA-NeXT feedback | LLaVA 1.5 7B · Object HalBench | response hallucination rate: 54.5 -> 10.5 | Helped |
| Align OmniLMM 12B with its own feedback | OmniLMM 12B · MHumanEval | response hallucination rate: 52.7 -> 35.6 | Helped |
| Apply self-feedback best-of-N selection | RLAIF-V 7B · Object HalBench | response hallucination rate: 10.5 -> 6.8 | Helped |
| Apply self-feedback best-of-N selection | RLAIF-V 12B · Object HalBench | response hallucination rate: 4.5 -> 4.5 | No clear effect |
| Use deconfounded candidate responses instead of original human annotations (ablation) | RLAIF-V · Object HalBench | response hallucination rate: 25.7 -> 10.1 | Helped |
| Use deconfounded candidate responses instead of original human annotations (ablation) | RLAIF-V · AMBER | accuracy: 73.3 -> 80.1 | Helped |
| Use claim-level divide-and-conquer evaluation instead of direct overall response evaluation (ablation) | RLAIF-V · Object HalBench | response hallucination rate: 53.1 -> 20.6 | Helped |
| Use claim-level divide-and-conquer evaluation instead of direct overall response evaluation (ablation) | RLAIF-V · AMBER | accuracy: 73.5 -> 80.5 | Helped |
| Use LLaVA-NeXT rather than OmniLMM 12B as feedback labeler (ablation) | RLAIF-V · Object HalBench | response hallucination rate: 33.0 -> 20.6 | Helped |
Surprising findings
- RLAIF-V 12B, labeling its own responses, outperforms GPT-4V on trustworthiness across several hallucination benchmarks.
- Replacing deconfounded preferred responses with human-annotated responses on RLHF-V data significantly hurts performance.
Limitations
- Focuses on hallucination/trustworthiness; the authors note the framework does not yet address complex logical reasoning feedback.
- Best-of-N inference scaling requires sampling many candidates, increasing evaluation cost.
- Self-feedback with length normalization still shows reliance on the model's own reward quality; the paper notes poor feedback from direct overall-score labeling.
Experimental setup
Datasets: MSCOCO ShareGPT-4V MovieNet Google Landmark v2 VQA v2 OKVQA TextVQA RLHF-V dataset
Benchmarks: Object HalBench MMHal-Bench MHumanEval AMBER RefoMB MMStar
Metrics: response-level hallucination rate mention-level hallucination rate accuracy F1 trustworthiness win rate overall win rate MMStar overall score
Baselines: VCD OPERA Less-is-more CCA-LLaVA Qwen-VL-Chat LLaVA-NeXT MiniGemini HA-DPO POVID LLaVA-RLHF Silkie RLHF-V
Builds on: direct preference optimization self-rewarding language models RLHF-V iterative DPO
Closest papers by meaning
- U-trustworthy Models. Reliability, Competence, and Confidence in Decision-Making (AAAI 2024 · similarity 0.81)
- LiSA: Lifelong Safety Adaptation via Conservative Policy Induction (NeurIPS 2026 · similarity 0.79)
- Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.79)
- Agentic Confidence Calibration (ICML 2026 · similarity 0.79)
- OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.79)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.