Research map / Reinforcement learning / RL from feedback

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes POW3R, a policy-aware rubric reward that keeps human criterion weights and category balance fixed but reweights criteria within categories by their rollout-level contrast (judge verdict variance) to make GRPO rewards more informative.

Problem

Static weighted-sum rubric rewards for GRPO conflate a criterion's human importance with its current learnability; many high-weight criteria are saturated or dead for the policy and give no group-relative advantage signal.

Approach

Keep the human-weighted rubric as the evaluation target but multiply each criterion's weight by a bounded, EMA-smoothed factor derived from the standard deviation of its judge verdicts across rollouts, renormalized within each rubric category so human weight priors and category mass are preserved.

Key results

• POW3R wins 24 of 30 base-policy/metric comparisons against binary, static scalar, and category-balanced rewards across MM and HealthBench.
• On Qwen3-VL-4B MM, POW3R reaches 46.0 dev reward at step 83 vs 249 for static scalar and 332 for category-balanced (up to 4x fewer steps).
• On HealthBench, POW3R gains +4.7 pp over base for Qwen3-4B (32.7 vs 28.0 overall), the largest gain among tested settings.

Surprising findings

• About half of rubric criteria are non-contrastive (saturated or dead) for a fresh policy, and human weight and rollout variance are essentially uncorrelated.
• Binary all-or-nothing reward never improves over base on the MM rubric objective for Qwen3-VL-4B.

Limitations

• Relies on LLM judges whose verdict bias propagates into policy-aware factors; judge calibrated only against a single reference judge.
• MM is an in-house dataset and HealthBench is the only public rubric set; untested on coding, scientific writing or multilingual domains.
• Gains are modest on some categories and POW3R trails baselines on a few HB strict and external benchmark cells.

Experimental setup

Datasets: MM (in-house 10K multimodal rubric dataset) HealthBench English

Benchmarks: MM test (1K tasks) HealthBench English hard test (500 tasks) HallusionBench POPE MM-IFE MMVetV2 MathVista RealWorldQA

Metrics: rubric reward strict completion overall score per-category pass rate

Baselines: Binary reward Static scalar rubric reward Category-balanced rubric reward

Builds on: GRPO HealthBench rubric-based rewards

Closest papers by meaning

• Reward Hacking in Rubric-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
• Step-wise Rubric Rewards for LLM Reasoning (NeurIPS 2026 · similarity 0.91)
• Deep Research as Rubric (NeurIPS 2026 · similarity 0.90)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.89)
• Auto-Rubric as Reward: From Implicit Preference to Explicit Generative Criteria (NeurIPS 2026 · similarity 0.89)
• Detecting RLVR Training Data via Structural Convergence of Reasoning (NeurIPS 2026 · similarity 0.89)