Research map / Large language models / Alignment and preferences
Decoding-time Realignment of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeRa, which shows KL-regularized aligned models at different strengths are geometric mixtures of a reference and one aligned model, and uses this to realign at decoding time by linearly combining logits, avoiding retraining.
Problem
Choosing the KL regularization strength in LM alignment (RLHF/DPO) requires retraining many aligned models across regularization strengths, which is computationally expensive for large models.
Approach
Aligned models trained at varying KL strengths are geometric mixtures of the SFT reference and one aligned model; DeRa approximates these autoregressively by sampling from softmax of a linear combination of reference and aligned logits, with mixing weight lambda setting an effective regularization beta/lambda at decoding time.
Key results
- On TL;DR summarization, DeRa identifies KL strengths beta/lambda (e.g., 0.05 and 0.02) that outperform the default beta=0.1 per PaLM 2 win rates, and DeRa agrees well with retrained models even for lambda>1.
- On Zephyr-7b, DeRa sweeps lambda 0 to 2.0 and MT-Bench scores show lambda controls alignment; retraining at beta/lambda=0.1/0.5 gives MT-bench 7.23 vs 7.37 for original.
- Retrained Zephyr at 0.1/0.5 achieves average Open LLM leaderboard score 61.55 vs 59.23 for original (GSM8K 30.02 vs 12.74).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Retrain Zephyr-7b with KL strength 0.1/0.5 instead of 0.1 | Zephyr-7b-beta · GSM8K | score: 12.74 -> 30.02 | Helped |
| Retrain Zephyr-7b with KL strength 0.1/0.5 instead of 0.1 | Zephyr-7b-beta · Open LLM leaderboard | Average: 59.23 -> 61.55 | Helped |
| Retrain Zephyr-7b with KL strength 0.1/0.5 instead of 0.1 | Zephyr-7b-beta · Open LLM leaderboard | score across benchmarks | Mixed |
| Increase decoding-time λ to reduce KL regularization | T5-Large summarization model aligned with policy gradient · Reddit TL;DR summarization | PaLM 2 preference win rate | Helped |
| Increase decoding-time λ beyond the useful range | PaLM 2 hallucination-mitigation model · Argument rewriting | naturalness and coherence | Hurt |
| Increase decoding-time λ | PaLM 2 hallucination-mitigation model · Argument rewriting | hallucination mitigation | Helped |
| Use the length reward to increase output length | T5-small summarization model · XSum toy length-reward task | summarization quality | Hurt |
| Increase KL strength by reducing λ | Zephyr-7b · GSM8K | performance | Helped |
| Replace logit combination with linear parameter combination (ablation) | DeRa on Zephyr-7b · MT-Bench | score | Hurt |
| Increase λ into the extrapolation regime | DeRa on T5-Large summarization · Pairwise accuracy | pairwise accuracy | Hurt |
Surprising findings
- Retrained Zephyr at a stronger effective KL (beta/lambda=0.2) scores higher on Open LLM leaderboard overall, largely from recovering GSM8K math performance (30.02 vs 12.74).
- Weight-combining variant of DeRa (rewarded-soup style) performs worse than logit combination, consistent with prior weight-combining findings.
Limitations
- Decoding requires running two models, doubling decoding time and memory.
- Approximation quality degrades for lambda>1 (extrapolation regime), with gap to retrained models growing in pairwise accuracy.
- Evaluation relies largely on LLM auto-raters (PaLM 2, GPT-4) and small, in-house hallucination datasets.
Experimental setup
Datasets: XSum Reddit TL;DR summarization UltraFeedback hallucination dataset (1888 examples, in-house) GSM8K
Benchmarks: MT-Bench Open LLM leaderboard (ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K) TL;DR summarization auto-evaluation length reward toy task
Metrics: win rate MT-bench score average reward pairwise accuracy accuracy on Open LLM leaderboard
Baselines: retrained-from-scratch aligned models rewarded soup weight interpolation SFT reference model
Builds on: RLHF DPO PPO emulated fine-tuning rewarded soups
Closest papers by meaning
- Decoding-Time Language Model Alignment with Multiple Objectives (NeurIPS 2024 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.90)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.90)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.90)
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.90)
- Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.