Research map / Large language models / Alignment and preferences
Alignment-Aware Decoding
Frédéric Berdoz, Luca A Lanzendörfer, René Caky, Roger Wattenhofer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decode with DPO/SFT log-ratio token advantage (AAD) instead of greedy DPO | Llama 3B DPO · Skywork | average oracle reward: 8.45 -> 13.71 | Improved |
| AAD instead of EFT and Bo2 at similar compute | Llama 3B DPO · Argilla | average oracle reward: EFT 4.54, Bo2 3.02 -> AAD 5.64 | Improved |
| replace SFT reference with external amateur Llama Instruct models (contrastive decoding) (ablation) | AAD on Llama 8B DPO · Skywork | average oracle reward: 19.27 -> 13.08 | Hurt |
| set min-alpha token filter around 0.1 to 0.2 (ablation) | AAD · Skywork, Nectar, Argilla (Llama 3B LoRA) | oracle reward | Mixed |
| use full fine-tuning instead of LoRA for the DPO model (ablation) | AAD · six datasets, Llama 1B/3B | oracle reward: HHRLHF 1B: EFT 0.26 vs AAD -0.06 | Mixed |
| AAD instead of Bo2 as training data grows (ablation) | AAD on Llama 3B full FT · Skywork | oracle reward: win rate 0.511 -> 0.746 (1% -> 100% data) | Improved |
| iterative DPO using AAD-generated chosen samples, trained from SFT (Iter 1') (ablation) | Llama DPO · UltraFeedback 10% | R score: R 2.21 for AAD decoding | Improved |
| second iteration of iterative DPO (ablation) | Llama DPO · Argilla 5% | R score (AAD decoding): 2.25 -> -2.13 | Hurt |
| entropy thresholding in beam search on AAD score (ablation) | AAD beam search · Skywork alpha 0.7 | score | Improved |
| use standard beam search with more beams (ablation) | beam search · Argilla, Skywork, Nectar | score | No clear effect |
| train DPO for more epochs (ablation) | AAD · Skywork Llama 3B | oracle reward | Hurt |
| raise DPO beta (ablation) | AAD vs Bo2 and greedy · Argilla | relative loss of oracle score | Mixed |
Closest papers by meaning
- Decoding-Time Language Model Alignment with Multiple Objectives (NeurIPS 2024 · similarity 0.90)
- InfAlign: Inference-aware language model alignment (ICML 2025 · similarity 0.90)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.89)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.89)
- MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
