Research map / Large language models / Alignment and preferences
Active Preference Learning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Active Preference Learning, an iterative loop that selects prompt/completion pairs for oracle labeling using DPO's implicit preference certainty (optionally combined with predictive entropy), and fine-tunes GPT-2 and Pythia with GPT-4 as the oracle.
Problem
Preference labels for aligning LLMs via DPO are expensive, yet standard DPO collects them upfront at random; the paper asks how to use a limited labelling budget more effectively.
Approach
Embed an active learning loop around DPO: at each step, sample prompts, generate completion pairs from the current model, score them by the implicit Bradley-Terry preference certainty |r_hat(y1)-r_hat(y2)| (optionally after an entropy pre-filter), acquire GPT-4 labels for the top M, and re-fine-tune DPO from the base model on all acquired data.
Key results
- Preference certainty acquisition beats random by roughly 1-6% average win-rate; on IMDB at size 256 it reaches 0.82 vs 0.76 for random.
- On TLDR at size 512, preference certainty reaches 0.60 win-rate vs 0.56 for random.
- For the first step (M=128) there is no discernible difference between acquisition strategies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use preference-certainty acquisition instead of random acquisition | DPO fine-tuning with GPT-2 · IMDB, 256 acquired | win-rate: 0.76 ± 0.008 -> 0.82 ± 0.005 | Helped |
| Use preference-certainty acquisition instead of random acquisition | DPO fine-tuning with GPT-2 · IMDB, 512 acquired | win-rate: 0.84 ± 0.004 -> 0.87 ± 0.007 | Helped |
| Use preference-certainty acquisition instead of random acquisition | DPO fine-tuning with GPT-2 · IMDB, 768 acquired | win-rate: 0.88 ± 0.004 -> 0.90 ± 0.004 | Helped |
| Use preference-certainty acquisition instead of random acquisition | DPO fine-tuning with Pythia · TLDR, 384 acquired | win-rate: 0.55 ± 0.009 -> 0.58 ± 0.01 | Helped |
| Use preference-certainty acquisition instead of random acquisition | DPO fine-tuning with Pythia · TLDR, 512 acquired | win-rate: 0.56 ± 0.012 -> 0.60 ± 0.009 | Helped |
| Use entropy acquisition instead of random acquisition | DPO fine-tuning with GPT-2 · IMDB, 256 acquired | win-rate: 0.76 ± 0.008 -> 0.74 ± 0.009 | Hurt |
| Use entropy acquisition instead of random acquisition | DPO fine-tuning with Pythia · TLDR, 256 acquired | win-rate: 0.55 ± 0.01 -> 0.57 ± 0.01 | Helped |
| Add entropy filtering before preference-certainty acquisition | DPO fine-tuning with GPT-2 · IMDB, 512 acquired | win-rate: 0.87 ± 0.007 -> 0.89 ± 0.006 | Helped |
| Add entropy filtering before preference-certainty acquisition | DPO fine-tuning with GPT-2 · IMDB, 768 acquired | win-rate: 0.90 ± 0.004 -> 0.92 ± 0.005 | Helped |
| Add entropy filtering before preference-certainty acquisition | DPO fine-tuning with Pythia · TLDR, 256 acquired | win-rate: 0.57 ± 0.01 -> 0.55 ± 0.01 | Hurt |
| Add entropy filtering before preference-certainty acquisition | DPO fine-tuning with Pythia · TLDR, 512 acquired | win-rate: 0.60 ± 0.009 -> 0.59 ± 0.01 | Hurt |
| Use an active acquisition strategy rather than random acquisition | DPO fine-tuning · IMDB and TLDR, first step (128 acquired) | win-rate | No clear effect |
Surprising findings
- GPT-3.5-turbo was only about 60% self-consistent as an oracle versus over 90% for GPT-4, so the authors used GPT-4 despite cost.
Limitations
- Re-initialising and fine-tuning to convergence at every acquisition step is computationally expensive and impractical for larger models.
- Evaluation relies on expensive GPT-4 calls, so evaluations were run sparsely and with limited seeds for some configurations.
- Experiments are limited to two small models (~0.8B parameters) and two tasks.
Experimental setup
Datasets: IMDB TLDR (CarperAI openai_summarize_tldr)
Benchmarks: IMDB held-out prompts TLDR held-out test set
Metrics: win-rate standard error
Baselines: random acquisition entropy acquisition preference + entropy hybrid
Builds on: Direct Preference Optimization Bradley-Terry model Reward rAnked FineTuning
Closest papers by meaning
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.93)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.92)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.92)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.