Research map / Vision-language and multimodal / Multimodal LLMs
RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoVRM, a visual reward model trained via three-phase progressive training on textual then caption-based then visual preference data, with optimal-transport preference data selection, improving best-of-n, RL and DPO alignment of LLaVA-1.5.
Problem
Visual reward models for aligning large vision-language models with human preferences are hard to train because visual preference data is scarce and expensive.
Approach
Transfer preferences from abundant textual preference data to a VRM through three progressive phases (textual preference pretraining, caption-based VL preference fine-tuning, visual preference fine-tuning with a visual projector), and select training samples per phase using optimal-transport distances over LoRA-warmup gradient features.
Key results
- With best-of-8 sampling on LLaVA-1.5-7B, RoVRM scores 82.0 on LLaVA-Bench vs 73.6 for VRM-Vanilla and reduces AMBER hallucination rate to 23.9 from 29.0.
- With RL on LLaVA-1.5-7B, RoVRM reaches 58.69 MM-Instruct win rate vs 51.11 for VRM-Vanilla.
- Integrating the method with DPO (RoDPO) on LLaVA-1.5-13B raises MM-Instruct win rate to 78.72 from 65.63 for DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RoVRM reward modeling instead of VRM-Vanilla for best-of-n sampling | LLaVA-1.5-7B · LLaVA-Bench (In-the-Wild) | Score: 73.6 -> 82.0 | Helped |
| Add RoVRM reward modeling instead of VRM-Vanilla for best-of-n sampling | LLaVA-1.5-7B · AMBER | HalRate: 29.0 -> 23.9 | Helped |
| Add RoVRM reward modeling instead of VRM-Vanilla for RL | LLaVA-1.5-7B · LLaVA-Bench (In-the-Wild) | Score: 72.8 -> 78.3 | Helped |
| Integrate progressive training and preference data selection with DPO | LLaVA-1.5-13B · MM-Instruct | WinRate: 65.63 -> 78.72 | Helped |
| Add preference data selection rather than using all data (ablation) | RoVRM with best-of-n sampling · AMBER | HalRate: 25.1 -> 23.9 | Helped |
| Add preference data selection rather than using all data (ablation) | RoVRM with RL · AMBER | HalRate: 32.2 -> 23.4 | Helped |
| Add phase one textual preference pretraining (ablation) | RoVRM with best-of-n sampling · LLaVA-Bench (In-the-Wild) | Score: 71.3 -> 82.0 | Helped |
| Add phase one textual preference pretraining (ablation) | RoVRM with RL · AMBER | HalRate: 35.0 -> 23.4 | Helped |
| Add phase two image-caption preference training (ablation) | RoVRM with best-of-n sampling · MM-Instruct | WinRate: 54.76 -> 61.91 | Helped |
| Add phase two image-caption preference training (ablation) | RoVRM with RL · AMBER | HalRate: 28.2 -> 23.4 | Helped |
| Use optimal transport rather than random selection | RoVRM · LLaVA-Bench (In-the-Wild), best-of-n sampling | Score: 77.2 -> 82.0 | Helped |
| Use optimal transport rather than cosine similarity for preference selection (ablation) | RoVRM with RL · AMBER | HalRate: 25.2 -> 23.4 | Helped |
Surprising findings
- Using only 5k visual preference samples with textual pretraining reaches performance comparable to VRM-Vanilla trained on 83k samples.
- The LLaVA-1.5-7B model with RoVRM best-of-n surpasses LLaVA-1.5-13B on most benchmarks.
Limitations
- Three-phase training adds extra time cost due to an additional preference training session and data selection overhead.
- Evaluation relies heavily on GPT-4 judging and GPT-4o-mini generated captions.
- Textual preference data quality strongly affects gains; fusion of datasets gave inconsistent improvements.
Experimental setup
Datasets: UltraFeedback RLAIF-V LLaVA-Instruct-150K COCO Caption 2017
Benchmarks: MMHalBench Object HalBench AMBER LLaVA-Bench (In-the-Wild) MM-Instruct
Metrics: HalRate informativeness score response-level hallucination rate mention-level hallucination rate object coverage win rate
Baselines: VRM-Vanilla RoVRM-Random GPT-4V GPT-4o LURE HA-DPO VCD Silkie LLaVA-RLHF RLHF-V DPO
Builds on: LLaVA-1.5 DPO LESS gradient-based data selection RLAIF-V UltraFeedback
Closest papers by meaning
- Calibrated Self-Rewarding Vision Language Models (NeurIPS 2024 · similarity 0.91)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.89)
- Systematic Reward Gap Optimization for Mitigating VLM Hallucinations (NeurIPS 2025 · similarity 0.89)
- MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models (ICLR 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.