Research map / Large language models / Reasoning and chain of thought
LASeR: Learning to Adaptively Select Reward Models with Multi-Arm Bandits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LASeR, which uses a LinUCB contextual multi-armed bandit to select one reward model per training batch during iterative DPO training of LLMs, using bandit rewards derived from the training loss.
Problem
Choosing a single reward model for LLM preference training is suboptimal because RMs generalize poorly across tasks, rankings are unreliable, and over-optimization occurs; ensembling multiple RMs is computationally expensive and suffers from conflicting signals.
Approach
Frame reward model selection as a contextual multi-armed bandit (LinUCB) where each arm is an RM, the context is the mean last-token embedding of the batch's prompts, and the reward is the normalized negative DPO/NLL training loss, so only one RM is loaded and used per batch.
Key results
- On reasoning tasks with Llama-3-8B, LASeR reaches 76.32% average accuracy over StrategyQA, GSM8K, and MMLU, +2.67 points over RM Score Ensemble and +1.45 over Best RM.
- On WildChat, LASeR achieves a 72.69% length-controlled AlpacaEval win rate vs. RM Score Ensemble and 56.34% vs. Best RM.
- LASeR is roughly 2x faster in wall-clock training than the RM Score Ensemble on GSM8K, and about 3x faster than sequential RM selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adaptively select one reward model per batch with a contextual bandit instead of using RM Agreement Ensemble | Llama-3-8B training · Reasoning average (StrategyQA, GSM8K, MMLU) | accuracy: +0.91% | Helped |
| Adaptively select one reward model per batch instead of using RM Score Ensemble | Llama-3-8B training · Reasoning average (StrategyQA, GSM8K, MMLU) | accuracy: 2.67% | Helped |
| Adaptively select a reward model for each query during best-of-n sampling instead of RM Score Ensemble | Llama-3-8B · LongBench single-document QA | F1: 2.96 F1 points | Helped |
| Adaptively select a reward model for each query during best-of-n sampling instead of RM Score Ensemble | Llama-3-8B · LongBench few-shot learning | F1: 2.97 F1 points | Helped |
| Adaptively select reward models instead of using the single Best RM | Llama-3-8B · WildChat instruction-following | length-controlled AlpacaEval win rate: 56.34% | Helped |
| Adaptively select one RM per batch rather than sequentially select RMs | Llama-3-8B training · Training efficiency | speedup: 3× | Helped |
| Select one RM per batch rather than score with the RM Score Ensemble | Llama-3-8B training · Training efficiency | speedup: 2× | Helped |
| Reuse the converged WildChat bandit policy for cold-start reasoning training | Llama-3-8B · Reasoning average (StrategyQA, GSM8K, MMLU) | accuracy: 76.32% vs 76.24% | No clear effect |
| Use batch size 1 rather than batch size 16 (ablation) | LASeR training · Training performance | performance | No clear effect |
Surprising findings
- Training with a single selected RM (LASeR, Best RM) outperforms multi-RM score ensembles, which the authors attribute to conflicting preference signals among RMs.
- RM Score Ensemble performs worse than Best RM on reasoning tasks despite combining several RMs.
Limitations
- Relies on the quality of the RM pool; performance may degrade if all RMs are weak or biased.
- Does not explicitly address safety, fairness, or bias in RM selection.
- Main results are from single runs; multi-seed analysis only covers reasoning tasks with 5 seeds.
Experimental setup
Datasets: StrategyQA GSM8K MMLU WildChat LongBench RewardBench
Benchmarks: StrategyQA GSM8K MMLU WildChat AlpacaEval LongBench
Metrics: accuracy length-controlled AlpacaEval win rate F1 Rouge-L
Baselines: Best RM Avg. RM Random RM Selection Sequential RM Selection Classifier Selection RM Score Ensemble RM Agreement Ensemble RM Online Ensemble SFT
Builds on: DPO LinUCB iterative preference optimization best-of-n sampling multiplicative weights
Closest papers by meaning
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.89)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.88)
- ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.88)
- Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.88)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.87)
- Consolidating Rewarded Perturbations for LLM Post-Training (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.