AI Paper Atlas

Research map / Learning theory and algorithms / Optimal transport

Optimal Transport for LLM Reward Modeling from Noisy Feedback

Licheng Pan, Haocheng Yang, Haoxuan Li, Yunsheng Lu, Yongqi Tong, Yinuo Wang, Shijian Wang, Zhixuan Chu, Lei Shen, Yuan Lu, Hao Wang

ICML 2026 · Learning theory and algorithms › Optimal transport · approach: Neural optimal transport

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace naive pointwise training with full optimal-transport alignment using preference cost only (ablation)reward model on noisy labels · HelpSteer (noisy labels)MSE: 0.083 -> 0.079Improved
add semantic-embedding term to the transport cost (joint cost) under full transport (ablation)SelectiveRM full-transport variant · HelpSteer (noisy labels)MSE: 0.079 -> 0.067Improved
add semantic-embedding joint cost under full transport (ablation)SelectiveRM full-transport variant · PKU-SafeRLHF (noisy labels)R2: 0.684 -> 0.710Improved
add mass relaxation (partial transport) to preference-cost transport (ablation)SelectiveRM preference-cost variant · HelpSteer (noisy labels)MSE: 0.079 -> 0.065Improved
add mass relaxation (partial transport) to preference-cost transport (ablation)SelectiveRM preference-cost variant · PKU-SafeRLHF (noisy labels)R2: 0.684 -> 0.751Improved
add mass relaxation on top of joint-cost full transport (ablation)SelectiveRM joint-cost variant · HelpSteer (noisy labels)MSE: 0.067 -> 0.063Improved
add joint cost on top of partial transport (ablation)SelectiveRM partial-transport variant · UltraFeedback (noisy labels)MSE: 0.109 -> 0.107Improved
use SelectiveRM instead of naive trainingreward model on noisy labels · PKU-SafeRLHF (noisy labels)MSE: 0.089 -> 0.057Improved
use SelectiveRM instead of naive trainingreward model on noisy labels · HelpSteer (noisy labels)MSE: 0.083 -> 0.063Improved
use SelectiveRM instead of naive training with a larger backboneQwen2.5-72B reward model · PKU-SafeRLHF (backbone study)MSE: 0.078 -> 0.056Improved
use SelectiveRM instead of naive training on LLaMA2-13BLLaMA2-13B reward model · PKU-SafeRLHF (backbone study)MSE: 0.088 -> 0.062Improved
use SelectiveRM reward model instead of naive for GRPO policy trainingQwen2.5-7B policy · DAN (GRPO policy safety)safety score: 0.217 -> 0.268Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy