Research map / Large language models / Alignment and preferences

When Can Proxies Improve the Sample Complexity of Preference Learning?

ICML 2025 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives sufficient conditions under which a large proxy preference dataset provably reduces the true-data sample complexity of DPO-style preference learning, via a shared-encoder plus low-dimensional adapter parametrisation; purely theoretical with no experiments.

Problem

Reward hacking arises when a proxy preference signal differs from the true reward; when only sparse expert data exists, it is unclear whether abundant proxy preference data can provably improve learning of the ground-truth policy.

Approach

Identifies four sufficient conditions (shared level sets, image inclusion, finite-dimensional bi-Lipschitz encoding, Lipschitz adapter between proxy and true policies) under which the true policy is a low-dimensional adapter on top of a proxy-learned decoder/encoder, so covering-number based sample complexity depends on the small latent dimension D rather than the embedding dimension D'.

Key results

• Shows the true policy can be written as a proxy decoder/encoder composed with a Lipschitz adapter on the D-simplex (Theorem on factorisation).
• Covering-number bound for the proxy-structured hypothesis class depends on latent dimension D, versus embedding dimension D' without proxy knowledge, giving a sample complexity improvement.
• Proposition: the topological dimension of the space of distributions over token sequences is infinite for unbounded length and O(k^l) for length at most l.

Measured effects reported in the paper

Limitations

• No empirical evaluation; authors state empirical validation is ongoing.
• Conditions are strong and may not hold for crowdsourced or adversarial raters.
• Analysis assumes proxy components are learned exactly (large proxy data limit) before the adapter stage.

Experimental setup

Builds on: direct preference optimisation

Closest papers by meaning

• Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.88)
• Inference-Time Reward Hacking in Large Language Models (NeurIPS 2025 · similarity 0.88)
• Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.88)
• Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.87)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.87)
• RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.87)