Research map / Reinforcement learning / RL from feedback
Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
Zhiwei Zhang, Hui Liu, Xiaomin Li, Zhenwei Dai, Jingying Zeng, Fali Wang, Minhua Lin, Ramraj Chandradevan, Linlin Wu, Zhen Li, Chen Luo, Zongyu Wu and 3 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly train a multi-objective head with the Bradley-Terry head (SMORM-F) | single-objective reward model (gemma-2B-it) · RewardBench (UF 400k / UltraFeedback) | RewardBench avg: 68.2 -> 72.8 | Improved |
| jointly train multi-objective head with BT head (SMORM-F) | single-objective RM (gemma-2B-it) · RewardBench (UF 40k / HelpSteer2) | RewardBench avg: 64.2 -> 71.0 | Improved |
| jointly train multi-objective head with BT head (SMORM-F) | single-objective RM (Mistral-7B) · RewardBench (UF 40k / HelpSteer2) | RewardBench avg: 68.5 -> 77.8 | Improved |
| add BT head to multi-objective reward model (SMORM-L) | MORM (gemma-2B-it) · RewardBench (UF 400k / UltraFeedback) | RewardBench score: 50.6 -> 63.6 | Improved |
| add BT head to multi-objective reward model (SMORM-L) | MORM (Mistral-7B) · RewardBench (UF 40k / HelpSteer2) | RewardBench score: 66.0 -> 79.9 | Improved |
| add BT head to multi-objective reward model (SMORM-L) | MORM (Mistral-7B) · RM-Bench (UF 40k / HelpSteer2) | RM-Bench score: 52.0 -> 64.4 | Improved |
| add BT head to MORM on Chat-Hard subset | MORM (Mistral-7B) · RewardBench Chat-Hard (UF 400k / UltraFeedback) | Chat-Hard score: 65.1 -> 61.0 | Hurt |
| add BT head to MORM (binarized HelpSteer2 aggregate) | MORM (Llama-3.2-3B) · RewardBench (HelpSteer2 binarized / HelpSteer2) | RewardBench score: 51.3 -> 56.8 | Improved |
| add BT head to MORM (binarized UltraFeedback aggregate) | MORM (Llama-3.2-3B) · RewardBench (UltraFeedback binarized / UltraFeedback) | RewardBench score: 50.0 -> 56.3 | Improved |
| use SMORM-F with D_S and D_M from different datasets | SORM (Llama-3.2-3B) · RewardBench (Skywork80K / HelpSteer2) | RewardBench avg: 65.8 -> 69.6 | Improved |
| use SMORM-F vs GRM (linear) w/ sft at larger data | GRM (linear) w/ sft · RewardBench (Mistral, UF 400k) | RewardBench avg: 79.5 -> 79.8 | Improved |
| use SMORM reward in PPO instead of baseline reward | baseline reward model · PPO gold score (ID) | gold reward | Improved |
Closest papers by meaning
- On the Robustness of Reward Models for Language Model Alignment (ICML 2025 · similarity 0.91)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.91)
- InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling (NeurIPS 2024 · similarity 0.90)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.90)
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.90)
- Rectifying Shortcut Behaviors in Preference-based Reward Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
