Research map / Reinforcement learning / RL from feedback
Rectifying Shortcut Behaviors in Preference-based Reward Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PRISM, a kernel-based regularizer that mitigates multiple shortcut behaviors (verbosity, tone, sycophancy) in Bradley-Terry reward models via approximated group-invariant kernels and a batch decorrelation term.
Problem
Preference-based reward models exploit spurious features (length, sycophancy, tone) that correlate with human preference labels, causing reward hacking and poor out-of-distribution generalization.
Approach
Model shortcut attributes as group-invariant kernels over responses, approximate them with random feature maps (empirical CDF of projections onto random templates), and add a kernel-distance margin penalty plus a global reward-feature decorrelation term to the Bradley-Terry loss.
Key results
- On RewardBench, PRISM (Llama-3 8B) reaches 87.8 overall score, versus 87.0 for GRM and 86.2 for SSRM.
- On RM-Bench, PRISM (Llama-3.1-8B) reaches 71.0 average accuracy, above Skywork-Reward-Llama-3.1-8B at 70.1.
- PRISM reward model shows near-zero Pearson correlation with length, tone, and sycophancy on RM-Bench, while the BT model is strongly correlated with length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PRISM instead of GRM | Llama-3 8B reward model · RewardBench | Score: 87.0 -> 87.8 | Helped |
| use PRISM instead of Skywork-Reward-Llama-3.1-8B | Llama-3.1-8B reward model · RM-Bench | Avg: 70.1 -> 71.0 | Helped |
| regularize against response length, tone, and sycophancy shortcuts with PRISM | Llama-3.1-8B reward model · RM-Bench | Pearson Correlation Coefficient | Helped |
Surprising findings
- Adding the shortcut-invariance penalty yields a modest gain on RM-Bench (71.0 vs 70.1 for the strongest baseline), and PRISM is slightly lower than the baseline on RewardBench Chat (98.7 vs 99.4 for RLHFlow).
Limitations
- Requires prior knowledge of which shortcuts to mitigate (length, tone, sycophancy), so subtle or evolving shortcuts are not automatically detected.
- LLM-based feature extraction with GPT-4o is costly for larger tasks.
- Evaluated only on English text-based preference tasks; multimodal and low-resource languages untested.
Experimental setup
Datasets: RLHFlow preference mixture (HH-RLHF, SHP, HelpSteer, PKU-SafeRLHF, UltraFeedback, UltraInteract, Distilabel-Capybara, Di UltraFeedback RewardBench RM-Bench AlpacaEval-2
Benchmarks: RewardBench RM-Bench AlpacaEval-2
Metrics: accuracy score win rate (WR) length-controlled win rate (LC) average length Pearson correlation coefficient
Baselines: Bradley-Terry reward model RRM GRM SSRM RLHFlow reward model HelpSteer2 RM LLM-as-a-judge GPT-4 Turbo Skywork-Reward-Llama-3.1-8B URM-LLaMa-3.1-8B Llama-3-OffsetBias-RM-8B
Builds on: Bradley-Terry model direct preference optimization invariant risk minimization group-invariant kernels
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.91)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.91)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.90)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.90)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.