Research map / Large language models / Alignment and preferences
Deep Bayesian Active Learning for Preference Modeling in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BAL-PM, a stochastic Bayesian active learning policy for preference modeling in LLMs that combines ensemble-of-adapters epistemic uncertainty with a KSG-based entropy term over LLM feature embeddings, reducing required preference labels by 33% (Reddit TL;DR) and 68% (CNN/DM OOD).
Problem
Selecting informative prompt-completion pairs for human preference labeling in LLMs is costly; prior Bayesian active learning (BALD) yields redundant samples and no gain over random selection for preference modeling in batch settings.
Approach
A stochastic batch acquisition objective that adds to the BALD epistemic uncertainty of an ensemble-of-adapters Bradley-Terry preference model a task-agnostic entropy term, estimated with a KSG-style kNN estimator over the frozen LLM's feature space, to favor diverse prompts.
Key results
- BAL-PM reduces required preference labels by about 33% versus random sampling on Reddit TL;DR test split.
- BAL-PM reduces required labels by about 68% versus random sampling in the CNN/DM out-of-distribution setting.
- BALD shows no benefit over random sampling on Reddit TL;DR, while BAL-PM consistently outperforms SoftmaxBALD, SoftRankBALD and PowerBALD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BAL-PM instead of random acquisition | Preference modeling · Reddit TL;DR test | required preference labels: 33% reduction | Helped |
| Use BAL-PM instead of random acquisition | Preference modeling · CNN/DM News OOD | required preference labels: 68% reduction | Helped |
| Use BALD instead of random acquisition | Preference modeling · Reddit TL;DR test | required preference labels | No clear effect |
| Use BALD instead of random acquisition | Preference modeling · CNN/DM News OOD | required preference labels | Helped |
| Add the preference-model uncertainty score to the objective (ablation) | BAL-PM · Reddit TL;DR test and CNN/DM News OOD | required preference labels | Helped |
| Add the prompt-distribution entropy score to the objective (ablation) | BAL-PM · Reddit TL;DR test and CNN/DM News OOD | required preference labels | Helped |
| Use epistemic instead of predictive uncertainty (ablation) | BAL-PM · Reddit TL;DR test and CNN/DM News OOD | required preference labels | Helped |
| Use the KSG instead of the KL entropy estimator (ablation) | BAL-PM · Reddit TL;DR test | required preference labels: approximately 20% fewer samples | Helped |
| Use the KSG instead of the KL entropy estimator (ablation) | BAL-PM · CNN/DM News OOD | required preference labels | No clear effect |
| Scale the base LLM from 7-billion to 70-billion or 140-billion parameters | BAL-PM · Reddit TL;DR | required preference labels: 31%--33% reduction of required samples | No clear effect |
| Use BAL-PM instead of other stochastic Bayesian acquisition policies | Preference modeling · Reddit TL;DR test and CNN/DM News OOD | test Log-Likelihood | Helped |
Surprising findings
- Naive BALD acquisition gives no benefit over random sampling on the Reddit TL;DR preference task.
- Predictive uncertainty underperforms epistemic uncertainty in the objective, attributed to high label noise in preference data.
Limitations
- Relies heavily on the quality of base LLM feature representations; may assign high entropy to nonsensical prompts (Noisy-TV problem).
- Evaluated only on two preference datasets, Reddit TL;DR and CNN/DM, with pool-based setting.
- Does not evaluate downstream preference optimization performance of fine-tuned policies beyond a simplified setup.
Experimental setup
Datasets: Reddit TL;DR preference dataset CNN/DM preference dataset
Benchmarks: Reddit TL;DR test split CNN/DM (OOD)
Metrics: average test log-likelihood number of labels required unique prompt ratio
Baselines: Random Sampling BALD SoftmaxBALD SoftRankBALD PowerBALD Full dataset training
Builds on: BALD BatchBALD Bradley-Terry model Kozachenko-Leonenko entropy estimator KSG estimator
Closest papers by meaning
- MAPLE: A Framework for Active Preference Learning Guided by Large Language Models (AAAI 2025 · similarity 0.91)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.91)
- Self-Evolutionary Large Language Models Through Uncertainty-Enhanced Preference Optimization (AAAI 2025 · similarity 0.89)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.88)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.88)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.