Research map / Vision-language and multimodal / Multimodal LLMs

Generative RLHF-V: Learning Principles from Multi-modal Human Preference

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Generative RLHF-V, which trains multi-modal generative reward models with RL (GRPO) to produce pairwise principle-based scores and then uses grouped pairwise comparisons among n sampled responses as RL rewards for MLLM alignment.

Problem

Score-only reward models for multi-modal LLM alignment have low accuracy, weak OOD generalization, and poor interpretability; pairwise generative reward models do not readily yield point-wise scores needed for RL optimization.

Approach

Train a pairwise GRM via RL (rule-based reward from preference labels) that autonomously reasons about principles and outputs scores, then convert pairwise judgments into per-response rewards by averaging pairwise scores over a group of candidate responses, so RL gains improve near-linearly with the number of candidates.

Key results

• GRLHF-V on Qwen2-VL-2B improves the average across 7 benchmarks by about 18.1% versus 5.3% for the best score-only RLHF baseline (as stated in abstract).
• On Qwen2.5-VL-7B-Instruct, GRLHF-V reaches 113.71 on LLaVA-Wild vs 97.05 base, and 71.86 on MM-Vet-v2 vs 67.23 base.
• Grouped comparison with GRM+RL reaches Pearson correlation near GPT-4o expert level on the MLLM-as-a-judge Score task.

Measured effects reported in the paper

Surprising findings

• Providing annotation principles to the GRM at inference hurts GRM+RL performance on OOD tasks, and omitting principles during GRM+RL training improves generalization.
• Over-trained GRMs lead RL policies to adopt self-praise behavior that inflates scores on MLLM-as-judge benchmarks (reward hacking).

Limitations

• Potential reward hacking (self-praise) under overfitting of GRM and RL, which may be hard to detect with MLLM-as-judge evaluation.
• Evaluated mainly on helpfulness and harmlessness with 2B-7B Qwen-VL models; limited compute acknowledged.
• Grouped comparison requires O(n^2) pairwise GRM calls per prompt, increasing cost.

Experimental setup

Datasets: Align-Anything BeaverTails-V MIA-Bench LLaVA-Bench-In-The-Wild LLaVA-Bench-Wilder MM-Vet MM-Vet-v2 MM-SafetyBench MSS-Bench LLaVA-Critic MLLM-as-a-Judge

Benchmarks: MIA-Bench LLaVA-Bench-In-The-Wild LLaVA-Bench-Wilder MM-Vet MM-Vet-v2 MM-SafetyBench MSS-Bench MLLM-as-a-Judge LLaVA-Critic Beaver-V

Metrics: accuracy pearson correlation benchmark score

Baselines: score-only RM DPO PPO GRM+SFT LLaVA-Critic Beaver-V

Builds on: SPCT GRPO Bradley-Terry reward modeling LLaVA-Critic

Closest papers by meaning

• Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.92)
• Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.91)
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.91)
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.91)
• Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)