Research map / Reinforcement learning / RL from feedback
Unbiased Principles, Robust Rewards
Qingnan Ren, Zhen Fang, Shiting Huang, Yu Zeng, Lin Chen, Zehui Chen, Feng Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate principles from question only (Q->P) instead of Q+R->P | GRM-16B (fine-tuned) · RewardBench (pointwise) | overall accuracy (%): 61.65 -> 68.22 | Improved |
| decouple principle generation from response | GRM-16B · RewardBench Safety | accuracy (%): 67.41 -> 78.22 | Improved |
| decouple principle generation from response | GRM-16B · RewardBench Reasoning | accuracy (%): 60.04 -> 65.67 | Improved |
| decouple principle generation from response | GRM-16B · RewardBench Chat Hard | accuracy (%): 55.83 -> 56.48 | Improved |
| condition principles on chosen response R+ (biased principles) (ablation) | DeepSeek-GRM-16B · RewardBench | score of R+ vs unbiased P0: +6.6% | Hurt |
| condition principles on rejected response R- (ablation) | DeepSeek-GRM-16B · RewardBench | score of R- vs unbiased P0: +4.6% | Hurt |
| train actor with IP-GRM reward instead of GRM | Qwen3-8B · WritingBench (all) | Avg@3 score: 85.2 -> 87.6 | Improved |
| train actor with IP-GRM reward instead of GRM | Qwen3-8B · Creative Writing v3 | Avg@3 score: 74.2 -> 77.8 | Improved |
| train actor with IP-GRM reward instead of GRM | Qwen3-8B-base · WritingBench (all) | Avg@3 score: 76.9 -> 79.7 | Improved |
| train actor with IP-GRM reward instead of GRM | Qwen3-8B-base · Creative Writing v3 | Avg@3 score: 68.7 -> 74.5 | Improved |
| train actor with IP-GRM reward instead of scalar RM | Qwen3-8B · WritingBench (all) | Avg@3 score: 85.0 -> 87.6 | Improved |
| use IP-GRM instead of GRM as GRPO reward | Qwen3-8B · WritingBench held-out | test score (0-10): 8.05 -> 8.75 | Improved |
Closest papers by meaning
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.90)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.89)
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.89)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.89)
- The Trickle-down Impact of Reward Inconsistency on RLHF (ICLR 2024 · similarity 0.89)
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
