Research map / Vision-language and multimodal / Multimodal LLMs
VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VL-RewardBench, a 1,250-pair benchmark for vision-language generative reward models built with small-model ensemble filtering, AI-aided labeling and human verification, showing even GPT-4o reaches only ~65% overall accuracy.
Problem
Existing evaluations of vision-language generative reward models (VL-GenRMs) rely on AI-annotated preferences or simple academic tasks that fail to challenge state-of-the-art LVLM judges.
Approach
Curate hard preference pairs by keeping only samples that an ensemble of small LVLM judges consistently misjudges, add AI-labeled reasoning pairs (MMMU-Pro, MathVerse), and verify all labels by humans; also show BoN correlation with MMMU-Pro and that critic training boosts judging.
Key results
- GPT-4o reaches 65.8% overall and 62.4% macro accuracy on VL-RewardBench, while Qwen2-VL-72B reaches only 43.0% macro accuracy.
- Performance on VL-RewardBench correlates with MMMU-Pro Best-of-N gains (Pearson r=0.946 for Qwen2-VL-7B, r=0.914 for LLaVA-OneVision-7B-ov).
- Critic training raises LLaVA-OneVision-7B judging accuracy by +14.7% (pointwise) and +9.2% (pairwise).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale Llama-3.2 from 11B to 90B | Llama-3.2 · VL-RewardBench | Macro average accuracy: 42.8% to 53.9% | Helped |
| Scale Qwen2-VL from 7B to 72B | Qwen2-VL · VL-RewardBench | Macro average accuracy: 33.9% to 43.0% | Helped |
| Scale Molmo from 7B to 72B | Molmo · VL-RewardBench | Macro average accuracy: 39.7% to 43.7% | Helped |
| Use GPT-4o instead of GPT-4o-mini | GPT-4o family · VL-RewardBench | Macro average accuracy: 62.4% vs 44.8% | Helped |
| Increase independent judgments from K=1 to K=7 (ablation) | GPT-4o · VL-RewardBench | Macro accuracy: 60.3% to 62.7% | Helped |
| Increase independent judgments across K values (ablation) | GPT-4o-mini · VL-RewardBench | Accuracy | No clear effect |
| Increase independent judgments from K=1 to K=5 (ablation) | Qwen2-VL-72B · VL-RewardBench | Accuracy: 1.7 percentage points | Hurt |
| Increase independent judgments from K=1 to K=5 (ablation) | Molmo-72B · VL-RewardBench | Accuracy: 2.6 percentage points | Hurt |
| Train the base model as a pointwise critic (ablation) | LLaVA-OneVision-7B-ov · VL-RewardBench subset | Overall accuracy: +14.7% | Helped |
| Train the base model as a pairwise critic (ablation) | LLaVA-OneVision-7B-ov · VL-RewardBench subset | Overall accuracy: +9.2% | Helped |
| Use pointwise rather than pairwise critic scoring (ablation) | LLaVA-Critic · VL-RewardBench subset | Overall accuracy: 52.9% vs 47.4%; +9.1% on hallucination | Mixed |
| Use pairwise rather than pointwise critic scoring (ablation) | LLaVA-Critic · VL-RewardBench reasoning subset | Accuracy: 60.0% | Helped |
Surprising findings
- Many open-source LVLMs (e.g., Qwen2-VL-72B, Molmo-72B) lose accuracy as the number of majority-vote judgments K increases, unlike GPT-4o.
- Most open-source 7B-scale judges barely exceed random chance on the benchmark.
Limitations
- Hard-sample filtering uses only small models, so difficulty may not transfer perfectly to future models.
- Reasoning-task preference labels are AI-drafted by GPT-4o and then human-verified, which may introduce bias.
- Only single-image, single-turn interactions are covered.
Experimental setup
Datasets: VLFeedback WildVision POVID RLAIF-V RLHF-V MMMU-Pro MathVerse
Benchmarks: VL-RewardBench MMMU-Pro
Metrics: overall accuracy macro average accuracy Pearson correlation error rate Cohen's kappa
Baselines: GPT-4o GPT-4o-mini Gemini-1.5-Pro Gemini-1.5-Flash Claude-3.5-Sonnet Qwen2-VL-72B Llama-3.2-90B Molmo-72B NVLM-D-72B LLaVA-Critic
Builds on: LLaVA-Critic VLFeedback WildVision POVID RLAIF-V RLHF-V MMMU-Pro MathVerse
Closest papers by meaning
- Calibrated Self-Rewarding Vision Language Models (NeurIPS 2024 · similarity 0.91)
- ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs (NeurIPS 2025 · similarity 0.91)
- SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning (ICML 2026 · similarity 0.90)
- ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning (CVPR 2026 · similarity 0.90)
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.90)
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.