AI Paper Atlas

Research map / Large language models / Alignment and preferences

Unbiased Reward Modeling from Implicit Feedback for LLM Alignment

Hao Wang, Haocheng Yang, Licheng Pan, Zhichao Chen, Xiaoxi Li, Yinuo Wang, Lei Shen, Yuan Lu, Haoxuan Li, Zhouchen Lin

ICML 2026 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
model unlabeled samples as possibly positive (latent positives) rather than all negative (ablation)ImplicitRM · PKU-SafeRLHFR2: 0.6848 -> 0.7872Improved
adaptively estimate action propensities (unfreeze propensity estimator) (ablation)ImplicitRM · HelpSteerR2: 0.2186 -> 0.3114Improved
handle both false negatives and selection bias together (ablation)ImplicitRM · UltraFeedbackR2: 0.4457 / 0.3948 -> 0.5207Improved
use ImplicitRM objective instead of Naive (unlabeled as negative)reward model on frozen LLM backbones · PKU-SafeRLHF, Qwen3-8BR2: 0.4786 -> 0.7546Improved
scale backbone from 8B to 32B (Qwen3) (ablation)ImplicitRM relative gain over Naive · PKU-SafeRLHFR2 relative gain: 57.7% -> 40.0% -> 31.5% relative gainMixed
use smaller learning rate (5e-6) (ablation)ImplicitRM · HelpSteerR2Improved
increase batch size from 64 to 256 (ablation)ImplicitRM · HelpSteerR2Improved
increase action propensity alpha from 0.1 to 0.9 (ablation)all methods · UltraFeedbackR2 gap between Naive and ImplicitRM: gap 0.65 -> 0.10Improved
train policy with ImplicitRM reward vs Naive reward in GRPOQwen2.5-Instruct-7B · HarmBenchSafety score: 0.8381 -> 0.9258Improved
train policy with ImplicitRM reward vs Naive reward in GRPOQwen3-Instruct-4B · WildGuardMixSafety score: 0.8094 -> 0.8974Improved
use debiased (propensity-based) baselines vs NaiveSDR on implicit feedback · UltraFeedbackR2: 0.1459 -> 0.3646Improved
ImplicitRM vs best baseline SelectMix on MAEHelpSteer · HelpSteerMAE: 0.2624 (SelectMix) vs 0.2919 (ImplicitRM)Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy