Research map / Reinforcement learning / RL from feedback

Preference Distillation via Value based Reinforcement Learning

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TVKD, an offline preference distillation method that adds a potential-based shaping term built from a DPO-trained teacher's soft value function to the DPO loss, improving small student models without extra rollouts.

Problem

Binary win-or-loss DPO supervision is insufficient for small language models with limited capacity, and prior distillation methods mimic teacher behavior (KL or behavior cloning) without modeling teacher reward structure, potentially conflicting with the preference data.

Approach

Define a soft value function V(s)=β log Σ exp(Q/β) from the DPO teacher's token logits and add the potential-based shaping term V(s')−V(s) to the student's DPO reward, which by PBRS theory preserves the optimal policy while injecting soft sequence-level guidance; implemented as a modified DPO loss with precomputed teacher logits and no extra rollouts.

Key results

• On DPOMIX with LLaMA-3.1-8B teacher and LLaMA-3.2-1B student, TVKD reaches RM -1.15, MT-Bench 3.97, AlpacaEval 52.18 win rate vs ADPA, outperforming all baselines including ADPA (50.00 AE).
• On HelpSteer2 the same setting gives TVKD RM -1.18, MT-Bench 3.98, AlpacaEval 54.85, the best among compared methods.
• TVKD requires no student rollouts and uses precomputed teacher logits, with the paper reporting ~2.5-3 hours wall time on four RTX 4090 GPUs.

Measured effects reported in the paper

Surprising findings

• Action-dependent shaping (teacher logits or log-probabilities) yields lower margin accuracy (~18%) than state-based shaping terms, consistent with the theoretical argument about policy invariance.
• Standard DPO student shows lower MT-Bench (3.70) than the SFT-based distillation variants in some settings, and the shaping term alone gives higher sequence-level preference accuracy (25.82%) than raw or length-normalized log-probabilities.

Limitations

• Relies on the teacher value function being accurate; noisy or misestimated values may degrade training, which is not tested.
• Robustness with weak or mismatched teachers is not evaluated.
• Relies on LLM-based judges (gpt-4o-mini and a reward model) for MT-Bench, AlpacaEval, and RM scores; reported gains are small in some metrics.

Experimental setup

Datasets: Deita-10k-V0 DPO-MIX-7K HelpSteer2

Benchmarks: RewardBench-style RM score (FsfairX-LLaMA3-RM-v0.1) MT-Bench AlpacaEval Open LLM Leaderboard

Metrics: RM score MT-Bench score AlpacaEval win rate Open LLM Leaderboard average margin accuracy KL divergence to teacher value

Baselines: SFT DPO SimPO WPO TDPO VanillaKD SeqKD GKD DCKD DDPO DPKD ADPA

Builds on: DPO potential-based reward shaping DCKD ADPA DDPO

Closest papers by meaning

• Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.89)
• AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.89)
• Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.89)
• Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.89)
• Advantage-Guided Distillation for Preference Alignment in Small Language Models (ICLR 2025 · similarity 0.89)
• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.88)