Research map / Vision-language and multimodal / Multimodal LLMs
VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models
Jiacheng Ruan, Wenzhen Yuan, Xian Gao, Ye Guo, Daoxin Zhang, Zhe Xu, Yao Hu, Ting Liu, Yuzhuo Fu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase model size from small group to middle group | open-source LVLMs as VLRMs · VLRMBench step-based | W-Avg F1: 29.8 -> 45.9 | Improved |
| increase model size from middle group to large group | open-source LVLMs as VLRMs · VLRMBench step-based | W-Avg F1: 45.9 -> 46.1 | No clear effect |
| add confident wording (CM) to erroneous steps | small-group VLRMs · VLRMBench SC vs CM | F1: 6.8% degradation | Hurt |
| add hallucinated information (EH task) to reasoning | small-group VLRMs · VLRMBench SC vs EH | F1: 14.6% degradation | Hurt |
| add hallucinated attributes (AH task) to reasoning | small-group VLRMs · VLRMBench SC vs AH | F1: 14.1% degradation | Hurt |
| swap numerical errors (DE) for general errors (SC) | large-group VLRMs · VLRMBench DE vs SC | F1: decreased by 2.5% | No clear effect |
| use spatial relationship errors (SR) instead of SC errors | large-group VLRMs · VLRMBench SR vs SC | F1: dropped by 30.4% | Hurt |
| use image reference errors (IC) instead of SC errors | large-group VLRMs · VLRMBench IC vs SC | F1: dropped by 15.8% | Hurt |
| place better reasoning second instead of first (ablation) | Llama3.2-11B · VLRMBench MJ | accuracy: 52.8 -> 8.4 | Hurt |
| increase reasoning length from <=10 to >15 steps | GPT-4o · VLRMBench SC | F1: 77.8 -> 68.6 | Hurt |
| add one-shot in-context example (ablation) | Llama3.2-11B · VLRMBench SC | F1: 33.9 -> 42.2 | Improved |
| add one-shot in-context example (ablation) | Qwen2.5VL-3B · VLRMBench SC | F1: 29.9 -> 31.5 | Improved |
Closest papers by meaning
- VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (CVPR 2025 · similarity 0.94)
- What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning (AAAI 2026 · similarity 0.93)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.91)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.91)
- Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
