Research map / Large language models / Alignment and preferences
Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPA, an iterative self-alignment framework that starts from 3.3% gold UltraFeedback labels, judges self-generated response pairs with the model's own logit-based implicit reward, and trains with DPO plus a noise-aware self-refined label smoothing loss.
Problem
Aligning LLMs with human preferences requires costly large-scale human preference annotation, and existing LLM-as-judge or external reward-model labeling is weak or distribution-mismatched when iterating.
Approach
Derive preference labels directly from the training model's implicit DPO reward (log-ratio against the SFT reference) rather than external judges, and use a de-coupled noise detector built from an extrapolated logit (1+λ)h_θ − λh_ref to refine labels via confidence-based label smoothing.
Key results
- With 3.3% gold labels on Mistral-7B, SPA reaches 15.39% length-controlled and 21.13% raw win rate on AlpacaEval 2.0 versus 9.03% and 7.68% for DPO on the same seed data.
- SPA reaches MT-Bench 6.94 versus 6.81 for DPO and 6.98 for Iterative DPO with PairRM, and it beats Zephyr-7b-beta (11.75 LC win rate, 10.03 raw) using 100% of UltraFeedback.
- Gains persist across Phi-2 (LC 7.02 to 9.10), LLaMA-3-8B-Instruct (LC 24.17 to 25.03), and Phi-3-14B (LC 27.70 to 28.77) relative to DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPA instead of standard DPO with 3.3% gold labels | Mistral-7B-v0.1 · AlpacaEval 2.0 | win rate vs. GPT-4 (%): 7.68% -> 21.13% | Helped |
| Use SPA instead of standard DPO with 3.3% gold labels | Mistral-7B-v0.1 · AlpacaEval 2.0 | length-controlled win rate (%): 9.03% -> 15.39% | Helped |
| Use SPA instead of standard DPO with 3.3% gold labels | Mistral-7B-v0.1 · MT-Bench | average score (0-10): 6.81 -> 6.94 | Helped |
| Use SPA instead of Iterative DPO with PairRM judgment | Mistral-7B-v0.1 · AlpacaEval 2.0 | win rate vs. GPT-4 (%): 9.46% -> 21.13% | Helped |
| Use SPA instead of Iterative DPO with LLM-as-judge | Mistral-7B-v0.1 · AlpacaEval 2.0 | length-controlled win rate (%): 9.28 -> 15.39 | Helped |
| Add self-refinement to data expansion (ablation) | SPA without de-coupled noise detection · AlpacaEval 2.0 | win rate vs. GPT-4 (%): 19.91% -> 19.94% | Helped |
| Add self-refinement to data expansion (ablation) | SPA without de-coupled noise detection · AlpacaEval 2.0 | length-controlled win rate (%): 14.41% -> 14.7% | Helped |
| Add de-coupled noise detection to self-refinement (ablation) | SPA · AlpacaEval 2.0 | win rate vs. GPT-4 (%): 19.91% -> 21.13% | Helped |
| Add de-coupled noise detection to self-refinement (ablation) | SPA · AlpacaEval 2.0 | length-controlled win rate (%): 14.41% -> 15.39% | Helped |
| Add de-coupled noise detection to self-refinement (ablation) | SPA · MT-Bench | average score (0-10): 7.09 -> 6.94 | Hurt |
| Use the SFT model as the reference for direct preference judgment rather than the previous-iteration model (ablation) | SPA data expansion · AlpacaEval 2.0 | length-controlled win rate (%): 13.73% -> 15.08% | Helped |
| Use the SFT model as the reference for direct preference judgment rather than the previous-iteration model (ablation) | SPA data expansion · AlpacaEval 2.0 | win rate vs. GPT-4 (%): 17.66% -> 19.56% | Helped |
Surprising findings
- Without any seed preference data, the framework still improved win rate from 6.31% to 9.79% in the reported setting.
- Using an external reward model (PairRM) for judgment underperformed self-judgment, which the authors attribute to distribution shift across iterations.
Limitations
- Trained models tend to produce longer responses, which the authors note may be dispreferred by some users.
- Gains on coding are near zero and math degrades on MT-Bench category breakdown.
- Length-controlled win rate has relatively high variance across seed samples (2.10).
Experimental setup
Datasets: UltraFeedback (argilla cleaned binarized) UltraChat-200k PairRM training data
Benchmarks: AlpacaEval 2.0 MT-Bench
Metrics: length-controlled win rate win rate vs GPT-4 MT-Bench average score
Baselines: SFT DPO Zephyr-7b-beta Iterative DPO (PairRM) Iterative DPO (LLM-as-judge)
Builds on: DPO Iterative DPO Zephyr PairRM label smoothing
Closest papers by meaning
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.93)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.93)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.92)
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment (NeurIPS 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.