Research map / Learning theory and algorithms / Prediction and decision losses

Preference Learning with Response Time: Robust Losses and Guarantees

NeurIPS 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Losses for decision-making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Neyman-orthogonal loss that jointly uses binary preferences and response times under the EZ-diffusion model to learn reward functions, with asymptotic and finite-sample guarantees and experiments on synthetic, neural, and Pick-a-Pic image preference tasks.

Problem

Learning reward models from pairwise human preferences is sample-inefficient because binary choices carry little information about preference strength; response times could supply this signal but their integration into preference learning lacks robust estimators.

Approach

Uses the EZ-diffusion identity r(X)=E[Y|X]/E[T|X] to build a Neyman-orthogonal loss over choices and response times, so errors in the response-time nuisance only affect the reward estimate at second order, yielding variance that scales polynomially rather than exponentially in reward magnitude.

Key results

• For linear rewards, the orthogonal loss has asymptotic variance that is pointwise smaller than the logistic-loss estimator and avoids the exponential scaling in reward magnitude.
• On the barrier-a experiment with a neural network reward, the orthogonal loss attains MSE around 9.3 to 10.1 versus 13.1 to 20.2 for log-loss across a in [0.5, 2.5].
• On Pick-a-Pic, the orthogonal loss consistently achieves lower MSE and regret than the non-orthogonal and log-loss baselines, per the paper's reported figures.

Measured effects reported in the paper

Surprising findings

• The naive non-orthogonal response-time loss performs worse than the orthogonal loss and in some settings is compared against log-loss with mixed regret gaps, e.g., regret for log-loss, non-orthogonal, and orthogonal losses are nearly tied at a=0.5 (0.298, 0.254, 0.254).

Limitations

• Experiments use synthetically generated response times from the EZ model rather than real human response-time data.
• Assumes homogeneous reward across users and known or fixed barrier a and non-decision time in main experiments.
• Finite-sample bounds are loose in S and require capped response times (breve B).

Experimental setup

Datasets: Pick-a-Pic synthetic linear rewards on unit sphere synthetic random 3-layer neural network rewards

Benchmarks: Pick-a-Pic (semi-synthetic) synthetic linear reward estimation synthetic neural network reward estimation

Metrics: mean squared error estimation error ||theta_hat - theta_o|| cumulative regret

Baselines: logistic log-loss (preference-only) non-orthogonal response-time loss Li et al. response-time preference bandit

Builds on: EZ diffusion model Bradley-Terry model Neyman orthogonal statistical learning DPO

Closest papers by meaning

• Enhancing Preference-based Linear Bandits via Human Response Time (NeurIPS 2024 · similarity 0.92)
• ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning (NeurIPS 2025 · similarity 0.91)
• Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.90)
• Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice (NeurIPS 2024 · similarity 0.90)
• PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.89)
• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)