Research map / Large language models / Alignment and preferences
Efficient Exploration for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that active exploration (double Thompson sampling with an epistemic neural network reward model) substantially reduces the human feedback queries needed to tune an LLM's best-of-N responses compared with passive, Boltzmann and infomax exploration.
Problem
RLHF typically gathers preference feedback passively by sampling response pairs from the language model, which is data-hungry; the paper asks whether actively tailoring queries can greatly reduce the number of human queries needed to reach high win rates.
Approach
Apply efficient exploration to query selection in RLHF: an agent samples N candidate responses from the LLM and picks a pair via double Thompson sampling using uncertainty from an epistemic neural network (ensemble of MLP heads on the frozen LLM embedding), with best-of-N response selection instead of policy-gradient tuning.
Key results
- Double TS with an ENN reward model is the top performer over passive, Boltzmann and infomax exploration, averaged over 5 seeds.
- For the performance level reached by 30,000 passive queries, double TS reduces data requirements by an order of magnitude (as stated).
- Oracle reward model reaches 0.755 accuracy on Anthropic Helpfulness eval; ENN gives much better dyadic joint NLL than the point estimate model at similar marginal NLL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use double Thompson sampling with ENN uncertainty instead of passive exploration | LLM preference-feedback exploration · Anthropic Helpfulness Base eval | win rate | Helped |
| Use ENN uncertainty estimates with double Thompson sampling instead of a point-estimate reward model with Boltzmann exploration | LLM preference-feedback exploration · Anthropic Helpfulness Base eval | win rate | Helped |
| Use double Thompson sampling instead of infomax | ENN-based LLM preference-feedback exploration · Anthropic Helpfulness Base eval | win rate | Mixed |
| Use double Thompson sampling rather than alternatives to reach the performance level attained by 30,000 passive queries | LLM preference-feedback exploration · Anthropic Helpfulness Base eval | queries required: 30,000 passive queries; reduces data requirements by an order of magnitude | Helped |
| Increase ENN ensemble size (ablation) | Double Thompson sampling agent · Anthropic Helpfulness Base eval | win rate | Helped |
| Increase ENN ensemble size beyond 10 (ablation) | Double Thompson sampling agent · Anthropic Helpfulness Base eval | win rate | No clear effect |
| Increase point-estimate MLP hidden units from 128 up to 2048 (ablation) | Point-estimate reward model · Uncertainty estimation experiments | performance | No clear effect |
| Use smaller Boltzmann temperatures (ablation) | Boltzmann exploration · Anthropic Helpfulness Base eval | win rate | Helped |
| Use very high Boltzmann temperatures instead of smaller temperatures (ablation) | Boltzmann exploration · Anthropic Helpfulness Base eval | win rate | Hurt |
| Choose one response greedily and the other with Boltzmann sampling instead of standard Boltzmann sampling (ablation) | Greedy-Boltzmann exploration · Anthropic Helpfulness Base eval | win rate | No clear effect |
| Increase SGD steps for infomax (ablation) | Infomax agent · Anthropic Helpfulness Base eval | performance | Helped |
| Increase SGD steps beyond a point (ablation) | Agents other than infomax · Anthropic Helpfulness Base eval | performance | Hurt |
Surprising findings
- Infomax performs very well early but falls far short of double TS later, attributed to information-seeking on irrelevant responses.
- Greedy-Boltzmann does not improve over standard Boltzmann after temperature tuning.
Limitations
- ENN is a simple ensemble of MLP heads on a frozen LLM last-layer embedding; LLM torso not tuned.
- Experiments use a simulated human preference oracle (Gemini Pro-based) rather than real humans.
- Only single-turn helpfulness prompts; multi-turn deep exploration left for future work.
Experimental setup
Datasets: Anthropic Helpfulness Base Anthropic Harmlessness Anthropic Helpfulness and Harmlessness preference data
Benchmarks: Anthropic Helpfulness Base eval (2048 prompts)
Metrics: win rate relative to llm queries required to reach performance level dyadic joint negative log-likelihood
Baselines: passive exploration Boltzmann exploration infomax
Builds on: double Thompson sampling epistemic neural network Bradley-Terry model Boltzmann exploration
Closest papers by meaning
- EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration (ICML 2025 · similarity 0.90)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.89)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.88)
- Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval (ICLR 2025 · similarity 0.88)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.88)
- The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.