Research map / Vision-language and multimodal / Multimodal LLMs

BaseReward: A Strong Baseline for Multimodal Reward Model

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematic ablation study of multimodal reward model (MRM) design choices, culminating in BaseReward, a Qwen2.5-VL-7B Naive reward model with a two-layer SiLU head trained on a curated mix of multimodal and text-only preference data, which reaches SOTA on several MRM benchmarks and improves RL training.

Problem

Building state-of-the-art multimodal reward models is poorly understood; there is no systematic guide on reward modeling paradigms, head architecture, training regularization, data curation, backbone choice, and ensembling for MLLM alignment.

Approach

A simple Naive reward model (linear-scored head on an MLLM) with an optimized two-layer SiLU MLP head, trained on a curated mixture of multimodal and text-only preference data, plus model ensembling, outperforms more complex critic-based and generative reward models on multimodal reward benchmarks; text-only data is shown to improve multimodal judgment.

Key results

• BaseReward (Qwen2.5-VL-7B) reaches 91.76 Acc and 80.43 Acc+ on MM-RLHF-Reward Bench, and an ensemble reaches 92.94 Acc, versus 82.0/63.0 for MM-RLHF-Reward-7B.
• On VL-Reward Bench, the BaseReward ensemble achieves 84.41 Overall Acc versus 71.92 for R1-Reward, and reports a ~14.2% overall improvement over the previous best.
• In RL with a hybrid rule-based plus BaseReward reward on Qwen-2.5-VL-3B, the model improves MathVista to 64.3 (vs 61.8 base and 61.2 R1-Reward) and WildVision win rate to 54.0 (vs 51.4 R1-Reward).

Surprising findings

• Text-only preference data improves multimodal reward modeling, matching or beating multimodal data such as MMPR on average multimodal benchmarks.
• Training a Qwen2.5-VL-7B MRM on the full mixed multimodal data gives no gain on pure-text reward benchmarks over text-only training, and LLM backbones (Qwen 2.5 8B, Qwen 3 8B) outperform the MLLM on text RM tasks.

Limitations

• Does not explore reward models on backbones of 72B parameters or larger due to compute constraints.
• LLM-based models still outperform MLLM-based reward models on pure-text reward benchmarks.
• Weaker on coding preference tasks because no coding preference data is included in training.

Experimental setup

Datasets: R1-Reward MM-RLHF training data MMIF Omni-Align RLAIF-V MMPR v1.2 UltraFeedback SHP Tulu-3 Olmo-2 V* arXivQA ThinkLite-VL

Benchmarks: VL-Reward Bench Multimodal RewardBench MM-RLHF-Reward Bench RewardBench RM Bench MMbench v1.1 MME-RealWorld-Lite MMStar MathVista V* LLaVA-Wild WildVision

Metrics: accuracy macro accuracy Acc+ win rate score

Baselines: R1-Reward MM-RLHF-Reward IXC-2.5-Reward LLaVA-Critic Claude 3.7 Sonnet GPT-4o Gemini 1.5 Pro

Builds on: Bradley-Terry reward modeling GRPO Qwen2.5-VL R1-Reward MM-RLHF

Closest papers by meaning

• The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models (ICML 2025 · similarity 0.92)
• MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning (CVPR 2026 · similarity 0.91)
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.91)
• MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.91)
• RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.90)
• Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.90)