Research map / Reinforcement learning / RL from feedback

Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classification

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TUP, an offline BoN-distillation policy that truncates low-win-rate completions and softly reweights the retained upper tail, trained via BCE with shifted-truncated win-rates as soft labels.

Problem

Rank-based BoN distillation policies use smooth full-support reweighting, leaving low-ranked completions in the support, while sharper reweighting over-relies on brittle top-rank distinctions from a single reward model.

Approach

Decouple lower-tail truncation (threshold lambda on win-rate, zero mass below) from upper-tail sharpness (beta); the target has a closed-form prompt-independent normalizer (incomplete Beta), enabling offline BCE training with shifted-truncated win-rates as soft labels and log-likelihood ratios as logits.</type_placeholder>

Key results

• On Llama 8B Tülu 3 SFT trained on Magpie Air, TUP mid. obtains the best in-dataset Skywork-Llama LC reward (21.24 vs 17.32 for best REBEL and 16.32 for QRPO).
• On AlpacaEval with Magpie Air training, TUP mild achieves 43.29 GPT-judge win rate vs 37.14 for DPO (Mistral table: TUP mild 40.62 vs DPO 37.14).
• Under the Mistral 7B Magpie Air setting, TUP mild achieves the best GPT-judge LC win (36.84 vs 34.45 for REBEL random) and remains competitive on reward metrics.

Surprising findings

• Under ArmoRM (the training reward model), all methods achieve broadly similar results, with larger differences only under the independent Skywork reward models.

Limitations

• Both lambda and beta must be tuned on validation, whereas baselines tune only beta.
• Still exposed to reward hacking from the proxy reward model.
• Theoretical results rely on continuity assumptions and a single global threshold rather than prompt-adaptive truncation.

Experimental setup

Datasets: UltraFeedback Magpie Air

Benchmarks: UltraFeedback eval split Magpie Air eval split AlpacaEval

Metrics: LC reward raw reward LC win rate win rate length

Baselines: DPO REBEL QRPO BoNBoN

Builds on: QRPO BoNBoN DPO REBEL

Closest papers by meaning

• Distillation Robustifies Unlearning (NeurIPS 2025 · similarity 0.85)
• Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.84)
• Theoretical guarantees on the best-of-n alignment policy (ICML 2025 · similarity 0.84)
• Sample-Optimal Agnostic Boosting with Unlabeled Data (ICML 2025 · similarity 0.84)
• Revisiting Agnostic Boosting (NeurIPS 2025 · similarity 0.84)
• On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference (ICML 2026 · similarity 0.84)