Research map / Large language models / Alignment and preferences
ARGS: Alignment as Reward-Guided Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARGS, a decoding-time alignment method that adds a scaled reward-model score to LM token log-probabilities over top-k candidates, avoiding RL training and improving HH-RLHF reward with LLaMA-7B.
Problem
Aligning LLMs with human preferences via RLHF/PPO is unstable and resource-intensive, and retraining is needed whenever the reward model changes.
Approach
At each decoding step, add w times a reward-model score of each top-k candidate token to its LM score, then select greedily or stochastically, so alignment happens at inference time without updating the base model.
Key results
- ARGS-greedy improves average reward by 19.56% over greedy decoding on LLaMA-7B-SFT with HH-RLHF (6.872 vs 5.732).
- In GPT-4 evaluation, ARGS-greedy achieves a win-tie of 64.33% against greedy decoding and 54.33% against top-k sampling.
- On SHP with OPT-1.3b, ARGS achieves average reward 5.98 comparable to PPO at 5.88, with higher diversity (0.322 vs 0.247) and coherence (0.390 vs 0.264).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reward-guided scoring during greedy decoding | LLaMA-7B-SFT · HH-RLHF test | Average Reward: 5.732 -> 6.872 | Helped |
| add reward-guided scoring during greedy decoding | LLaMA-7B · HH-RLHF test | Average Reward: 3.981 -> 5.026 | Helped |
| use stochastic token sampling instead of greedy token selection (ablation) | ARGS · HH-RLHF test | Average Reward: 6.87 -> 6.56 | Hurt |
| use stochastic token sampling instead of greedy token selection (ablation) | ARGS · HH-RLHF test | Diversity: 0.670 -> 0.772 | Helped |
| use stochastic token sampling instead of greedy token selection (ablation) | ARGS · HH-RLHF test | Coherence: 0.526 -> 0.525 | No clear effect |
| increase reward weight w (ablation) | ARGS · HH-RLHF test | Coherence | Hurt |
| vary candidate count k from 10 to 40 (ablation) | ARGS · HH-RLHF test | Average Reward | No clear effect |
| use ARGS decoding instead of PPO | OPT-1.3b · SHP test | Average Reward: 5.88 -> 5.98 | No clear effect |
| use ARGS decoding instead of greedy decoding | OPT-2.7b · SHP test | Generation time per response: 1.9 times | Hurt |
| use ARGS decoding instead of greedy decoding | OPT-2.7b · SHP test | Average Reward: +6.8% | Helped |
Surprising findings
- Decoding-time ARGS matches PPO average reward on SHP without any RL training, while also scoring higher on diversity and coherence than PPO and DPO.
Limitations
- Inference cost grows with top-k candidate count and reward model calls, roughly 1.9x generation time on OPT-2.7b.
- Larger reward weight w degrades coherence and reward beyond a point, requiring hyperparameter tuning on validation data.
Experimental setup
Datasets: HH-RLHF SHP
Benchmarks: HH-RLHF test prompts SHP test samples
Metrics: average reward diversity coherence win-tie rate
Baselines: Greedy decoding Top-k sampling Nucleus sampling Contrastive search PPO DPO
Builds on: RLHF PPO DPO contrastive search top-k sampling nucleus sampling
Closest papers by meaning
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.90)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.89)
- GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment (ICLR 2025 · similarity 0.89)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.88)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.88)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.