Research map / Large language models / Alignment and preferences
Clone-Robust AI Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces robustness to approximate clones for RLHF, shows regularized MLE violates it, and proposes a similarity-weighted MLE (weights from Voronoi cell areas) that satisfies it.
Problem
Standard RLHF reward learning from pairwise preferences is sensitive to near-duplicate alternatives in unbalanced datasets, so adding approximate clones can shift the learned reward function.
Approach
Define robustness to approximate clones (adapted from social choice's independence of clones) and propose the weighted MLE, which weights each alternative and each pair by its Voronoi-cell area in context space so near-duplicates share weight rather than multiply it.
Key results
- Regularized MLE's top-reward topic flips from romance (Original) to art (Clones), showing non-robustness to approximate clones.
- Weighted MLE's mean rewards for food, art, and romance do not change significantly between Original and Clones datasets.
- Theorem shows weighted MLE is robust to approximate clones under Lipschitz annotator rewards, while regularized MLE is not, even with exact clones (Borda count winner flips).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add approximate clones to the preference dataset | Regularized MLE · Paris responses | reward stability across topics | Hurt |
| Add approximate clones to the preference dataset | Weighted MLE · Paris responses | reward stability across topics | No clear effect |
| Use weighted MLE instead of regularized MLE | RLHF reward estimation · Representative preference datasets with approximate clones | robustness to approximate clones | Helped |
Surprising findings
- Regularized MLE fails even for exact clones, since it is equivalent to Borda count which is not independent of clones.
- Impossibility result: for any RLHF algorithm with two BTL types, some population makes the output arbitrarily far from the mean reward.
Limitations
- Assumes sufficient comparisons per pair of alternatives, which is unrealistic for LLM-generated single-response alternatives.
- Case study is a single synthetic prompt and does not show how frequent clones are in practice.
- Weighting scheme is one choice among many; the weights require a chosen alternative space S.
Experimental setup
Datasets: synthetic Paris-description preference data generated with gpt-4o-mini
Benchmarks: synthetic Paris case study (Original vs Clones datasets)
Metrics: mean reward per topic (food, art, romance) sample standard deviation
Baselines: regularized MLE
Builds on: Bradley-Terry-Luce model regularized maximum likelihood reward learning independence of clones (social choice) Borda count
Closest papers by meaning
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
- On a Connection Between Imitation Learning and RLHF (ICLR 2025 · similarity 0.91)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.91)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.91)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.