Research map / Large language models / Alignment and preferences

Auto-Rubric as Reward: From Implicit Preference to Explicit Generative Criteria

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Reward model design and preference models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Auto-Rubric as Reward (ARR), a training-free method that generates verified, prompt-specific rubrics from a frozen VLM for pairwise judging, and Rubric Policy Optimization (RPO), which uses rubric-conditioned binary preferences as RL rewards to fine-tune FLUX.1-dev and Qwen-Image-Edit-2509.

Problem

Reward signals for multimodal generative alignment collapse multi-dimensional human preferences into scalar or pairwise labels, which are opaque and prone to reward hacking and positional bias; existing rubric-as-reward methods rely on fixed or supervised rubric construction.

Approach

Externalize a VLM's internalized preference knowledge as instance-conditioned rubrics via a generate-verify-refine pipeline, consolidate them into a hierarchical structured rubric used as judge context, and use the frozen rubric-conditioned judge's binary preferences as the reward for online policy optimization.

Key results

• Gemini 3.1 Pro + ARR reaches 78.3% on HPDv3 and 78.9% on MM-RewardBench2 T2I, improving over the direct judge by 1.7 to 6.3 points across benchmarks.
• RPO on FLUX.1-dev with Gemini 3.1 Pro-based ARR raises GenEval from 0.66 to 0.80 and DPG-Bench from 83.84 to 85.76.
• On Qwen-Image-Edit-2509, ARR-RPO improves GEdit-Bench from 7.54 to 7.85 and ImgEdit from 4.35 to 4.43.

Surprising findings

• Zero-shot ARR only modestly reduces positional bias (about 3 to 5 points), while guidance with human preference exemplars reduces it much more (to about 9 to 10 points).
• Rubrics generated by the weakest generator (Qwen3-VL-8B) recover more than half the gain of same-family rubrics when judged by Gemini 3.1 Pro.

Limitations

• Rubric cardinality increases inference cost linearly, since each rubric needs separate generation, verification and evaluation calls.
• Residual positional bias remains even with human-guided rubrics (Delta about 8.9 for Gemini 3.1 Pro).
• Only pairwise formulation studied; pointwise and listwise extensions not evaluated.

Experimental setup

Datasets: ShareGPT-4o-Image HPDv3 MM-RewardBench2 EditReward-Bench

Benchmarks: HPDv3 MM-RewardBench2 EditReward-Bench GenEval DPG-Bench TIIF UniGenBench++ GEdit-Bench ImgEdit

Metrics: pairwise preference accuracy forward-reverse positional bias gap GenEval score DPG-Bench score TIIF score UniGenBench++ accuracy GEdit-Bench score ImgEdit score

Baselines: PickScore ImageReward HPSv3 UnifiedReward UnifiedReward-Thinking EditReward Qwen3-VL-8B GPT-5 Gemini 3.1 Pro

Builds on: Bradley-Terry reward modeling Flow-GRPO-style online RL PPO clipping AutoRubric

Closest papers by meaning

• Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR (NeurIPS 2026 · similarity 0.89)
• ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning (CVPR 2026 · similarity 0.88)
• RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data (AAAI 2025 · similarity 0.87)
• RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.87)
• Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling (NeurIPS 2026 · similarity 0.87)
• Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.87)