Research map / Reinforcement learning / RL from feedback
Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, Lifeng Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| refine rubrics with a great pair instead of a good pair (ablation) | rubric RFT (Qwen3-8B-Base) · Medical-o1 win rate | win rate: 22.4 -> 26.5 | Improved |
| refine rubrics with one great pair vs prompt-only rubric (ablation) | rubric RFT (Qwen3-8B-Base) · Finance win rate | win rate: 37.2 -> 42.2 | Improved |
| iterate refinement with 4 great pairs instead of 1 (ablation) | rubric RFT (Qwen3-8B-Base) · Medical-o1 win rate | win rate: 26.5 -> 31.4 | Improved |
| add diverse candidate responses to 4 great pairs (ablation) | rubric RFT (Qwen3-8B-Base) · Medical-o1 win rate | win rate: 31.4 -> 34.4 | Improved |
| add diverse candidate responses to 4 great pairs (ablation) | rubric RFT (Qwen3-8B-Base) · Medical-o1 HealthBench | score: 0.3348 -> 0.3513 | Improved |
| refine rubric with one good pair vs prompt-only (ablation) | rubric RFT (Qwen3-8B-Base) · Medical-o1 HealthBench | score: 0.3004 -> 0.2912 | No clear effect |
| refine with great pairs instead of good pairs (ablation) | rubric accuracy · high-reward region pairs | accuracy: 42.2% -> 45.8% | Improved |
| refine with good pairs instead of great pairs (ablation) | rubric accuracy · low-reward region pairs | accuracy: 66.7% -> 67.9% | Improved |
| add diverse responses to 4 great pairs (ablation) | rubric accuracy · high-reward region pairs | accuracy: 49.2% -> 47.9% | Hurt |
| train on rubrics refined with great diverse pairs rather than initial rubrics, long training (ablation) | rubric RFT · health and finance extended training | win rate and benchmark score | Improved |
| Bradley-Terry reward model trained on off-policy great responses vs on-policy (ablation) | BT reward model (Llama-3.1-8B) · Health domain | win rate: 26.8% -> 22.4% | Hurt |
| add off-policy to on-policy responses for BT reward model (ablation) | BT reward model (Llama-3.1-8B) · Health domain | HealthBench: 0.3036 -> 0.3032 | No clear effect |
Closest papers by meaning
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.91)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.91)
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.91)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
- ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.90)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
