Research map / Large language models / Alignment and preferences
Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios
Jianxiang Zang, Yongda Wei, Ruxue Bai, Shiyu Jiang, Nijia Mo, Binhong Li, Qiang Sun, Hui Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply synonym transformation to responses | most reward models · RM-Bench Chat | preference confidence: e.g. URM-LLaMa-3-8B 0.370 effect size | Hurt |
| translate responses to another language | most reward models · RM-Bench Chat | preference confidence: e.g. URM-LLaMa-3-8B 0.490, Llama-3-OffsetBias-RM-8B 0.741 | Hurt |
| restructure/extend responses (structured presentation, length extension) | most reward models · RM-Bench Chat | preference confidence: SOLAR 0.774/0.906 vs Skywork-Reward-V2-Llama-3.1-8B -0.382/-0.489 | Mixed |
| apply controlled prompt perturbations (typos, formatting, irrelevant tokens) | reward models · RM-Bench Chat | preference confidence | Mixed |
| move evaluation from objective to subjective domains | reward models · RM-Bench subsets | suitability degradation | Hurt |
| use low-suitability-risk RM for PPO instead of high-risk RM | Llama3-8B policy · perturbed vs clean RLHF | perturbed policy win rate: Spearman -0.881 | Improved |
| add an irrelevant username or weblink to prompt | FsfairX-LLaMA3-RM-v0.1 · RM-Bench Chat | preference confidence: IU 0.502 | Hurt |
Closest papers by meaning
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)
- Interpreting Language Reward Models via Contrastive Explanations (ICLR 2025 · similarity 0.91)
- Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? (ICLR 2025 · similarity 0.90)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.90)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.89)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
