Research map / Large language models / Alignment and preferences
When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models
Tong Xie, Andrew Bai, Yuanhao Ban, Yunqi Hong, Haoyu Li, Cho-Jui Hsieh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace BT loss with NormBT | gemma-2b-it, Unified-Feedback · RewardBench Reasoning | accuracy: 75.41 -> 80.71 | Improved |
| replace BT loss with NormBT | Llama-3.2-3B-Instruct, Unified-Feedback · RewardBench Reasoning | accuracy: 71.70 -> 76.93 | Improved |
| replace BT loss with NormBT | Llama-3.2-3B-Instruct, Skywork-80K · RewardBench Reasoning | accuracy: 67.05 -> 74.60 | Improved |
| replace BT loss with NormBT | gemma-2b-it, Unified-Feedback · RewardBench Average | accuracy: 72.25 -> 73.57 | Improved |
| replace BT loss with NormBT | Llama-3.2-3B-Instruct, Skywork-80K · RewardBench Average | accuracy: 80.31 -> 81.48 | Improved |
| replace BT loss with NormBT | Llama-3.2-3B-Instruct, Skywork-80K · RewardBench Chat | accuracy: 86.03 -> 83.80 | Hurt |
| add ground-truth margin to BT loss (ablation) | gemma-2b-it, Unified-Feedback · RewardBench Average | accuracy: 72.25 -> 71.23 | Hurt |
| add label smoothing to BT loss (ablation) | gemma-2b-it, Unified-Feedback · RewardBench Reasoning | accuracy: 75.41 -> 72.28 | Hurt |
| remove EMA normalization from NormBT (ablation) | gemma-2b-it, Unified-Feedback · RewardBench Average | accuracy: 67.78 -> 73.57 | Improved |
| use last-token L2 distance instead of cosine similarity (ablation) | NormBT · RewardBench Average | accuracy: 71.88 -> 73.57 | Improved |
| use last-token L2 distance instead of average pooling (ablation) | NormBT · RewardBench Average | accuracy: 69.68 -> 73.57 | Improved |
| add LayerNorm before reward head to BT (ablation) | gemma-2b-it, Unified-Feedback · RewardBench Average | accuracy: 72.25 -> 72.25 | No clear effect |
Closest papers by meaning
- On the Robustness of Reward Models for Language Model Alignment (ICML 2025 · similarity 0.89)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.89)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.89)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.88)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.88)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
