Research map / Large language models / Language, safety and interpretability

Advantage-Guided Distillation for Preference Alignment in Small Language Models

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DCKD (dual-constrained KD from a DPO-aligned teacher to a student on preferred and dispreferred responses) and ADPA (advantage-guided distillation using the log-ratio of DPO teacher vs. reference teacher), plus ADPA+ which initializes ADPA from DCKD, to align small language models.

Problem

Small language models often gain little or even lose performance from preference alignment (alignment tax), and sequence-level RLHF rewards are sparse, making alignment of SLMs hard.

Approach

Use a DPO-aligned teacher to distill preference knowledge into a small student via two KL constraints (preferred and dispreferred responses), and derive a distribution-level advantage log(pi_dpo/pi_ref) from the teacher pair as a dense reward for student policy optimization without online RL.

Key results

• On LLaMA-3.2-1B with DPO-MIX-7K, ADPA improves MT-Bench from 3.40 (DPO) to 3.88 and ADPA+ reaches 4.02; AlpacaEval WR 53.8% for ADPA+.
• On Danube2-1.8B with DPO-MIX-7K, ADPA+ achieves MT-Bench 4.40 vs DPO 3.87 and DCKD 4.09.
• Distribution-level ADPA achieves 50.0% AlpacaEval win rate vs 40.0% for token-level DPPO and 27.7% for sequence-level DPPO.

Measured effects reported in the paper

Surprising findings

• Removing the DPO teacher from DCKD or the reference teacher from ADPA noticeably degrades results, and on LLaMA-2-7B the w/o-reference-teacher variant gets higher AlpacaEval WR (46.2) than some baselines but lower MT-Bench.
• Some baselines (e.g., DPO on Danube3-500M) slightly decrease MT-Bench, supporting the alignment tax claim.

Limitations

• Advantage derived from top-50 truncated token distributions, approximating the full vocabulary.
• Evaluation relies heavily on GPT-4 judges and a reward model (FsfairX) that may favor certain outputs.
• Gains are mixed across benchmarks; some OLL scores (e.g., GSM8K) drop for ADPA variants.

Experimental setup

Datasets: Deita-10K-V0 DPO-MIX-7K HelpSteer2 RewardBench

Benchmarks: MT-Bench AlpacaEval Open LLM Leaderboard

Metrics: MT-Bench average rating AlpacaEval win rate average accuracy on OLL tasks reward accuracy FsfairX average score

Baselines: DPO SimPO WPO VanillaKD SeqKD ATKD PLaD DDPO DPKD

Builds on: Direct Preference Optimization knowledge distillation MiniLLM DPKD PLaD

Closest papers by meaning

• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
• Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.91)
• Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.91)
• Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.91)
• Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.91)
• Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.90)