Research map / Vision-language and multimodal / Multimodal LLMs

Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Omni-Reward, a benchmark (Omni-RewardBench, 3,725 pairs over 9 tasks and 5 modalities with free-form criteria), a 317K+69K preference dataset (Omni-RewardData), and discriminative (BT) and generative (GRPO-trained R1) omni-modal reward models.

Problem

Existing reward models focus mainly on text and image modalities and are trained on fixed binary preferences, so they handle video, audio, 3D and personalized free-form preferences poorly.

Approach

Combine a multi-modality benchmark with free-form textual criteria, a dataset augmented with GPT-4o-generated instruction-style preference criteria, and two reward models: a Bradley-Terry scorer on MiniCPM-o-2.6 and a GRPO-trained generative judge on Qwen2.5-VL-7B.

Key results

• Omni-RewardModel-BT reaches 73.68% accuracy on Omni-RewardBench w/o Ties and 65.36% w/ Ties.
• Best proprietary model Claude-3.5-Sonnet scores 66.54% w/ Ties; best open-source Gemma-3-27B-it 65.12%.
• Omni-RewardModel-BT achieves 76.3% overall accuracy on VL-RewardBench, reported as SOTA in the paper.

Surprising findings

• Pairwise scoring substantially outperforms pointwise scoring for generative RMs (e.g., Gemma-3-4B 66.61 vs 37.61).
• Chain-of-thought helps weaker generative RMs but yields little or slightly negative gains for stronger models.

Limitations

• Benchmark scale (3,725 pairs) is small relative to millions-scale datasets.
• Preference data is limited to single-turn interactions.
• Annotations done by three PhD-student authors, limited demographic diversity; RL for R1 is preliminary.

Experimental setup

Datasets: Skywork-Reward-Preference UltraFeedback RLAIF-V OmniAlign-V-DPO VLFeedback HPDv2 EvalMuse VideoDPO VisionReward Open-Image-Preferences RMB RPR

Benchmarks: Omni-RewardBench VL-RewardBench Multimodal RewardBench

Metrics: accuracy

Baselines: GPT-4o Gemini-2.0-Flash Claude-3-5-Sonnet Gemma-3-27B-it Qwen2.5-VL-72B-Instruct UnifiedReward InternLM-XComposer2.5-7B-Reward PickScore HPSv2 Skywork-VL-Reward

Builds on: Bradley-Terry GRPO MiniCPM-o-2.6 Qwen2.5-VL RewardBench

Closest papers by meaning

• BaseReward: A Strong Baseline for Multimodal Reward Model (ICLR 2026 · similarity 0.89)
• Omni-Perception Policy Optimization for Multimodal Emotion Reasoning (ICML 2026 · similarity 0.88)
• RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.88)
• The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models (ICML 2025 · similarity 0.87)
• Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.87)
• Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.87)