Research map / Large language models / Reasoning and chain of thought

Bias-Variance Optimized Preference Optimization for Large Reasoning Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BVPO, which mixes a high-variance trace-based DPO gradient with a low-variance empty-trace (thinking disabled) DPO gradient to reduce trace-sampling variance in preference optimization of large reasoning models.

Problem

Preference alignment of large reasoning models is hard because the marginal answer-level DPO objective is intractable and single-trace surrogates yield high-variance gradients from stochastic reasoning-trace sampling.

Approach

Form a convex combination of the trace-based DPO gradient and an empty-trace DPO gradient (reasoning disabled via empty think tags), with a closed-form MSE-optimal mixing weight; the combined estimator has lower trace-induced variance and no worse MSE than either component.

Key results

• BVPO improves AlpacaEval 2 win rate by up to 7.8 points and Arena-Hard by up to 6.8 points over the best baseline across the three LRMs and two modes.
• On six math reasoning benchmarks, BVPO raises average accuracy by up to 4.0 points over the base model (e.g., R1-Qwen-1.5B avg 44.7 to 48.7).
• Trace generation raises log-probability variance up to 10.17x and mean response length up to 3.17x relative to no-thinking mode.

Surprising findings

• Preference optimization on general conversational data improved math reasoning benchmarks, not only alignment benchmarks.
• The variance ratio of joint log-probabilities between thinking and no-thinking modes varies widely across models (1.23x to 10.17x).

Limitations

• Mixing weight alpha fixed at 0.5 in experiments rather than tuned via the closed-form MSE-optimal value.
• Preference comparisons use only final answers, ignoring reasoning-trace quality.
• Evaluation limited to three distilled Qwen-based LRMs and math reasoning tasks.

Experimental setup

Datasets: UltraFeedback

Benchmarks: AlpacaEval 2 Arena-Hard AIME 2024 AIME 2025 AMC MATH-500 Minerva OlympiadBench

Metrics: win rate length-controlled win rate avg@32 accuracy pass@1

Baselines: DPO SimPO base reasoning model (no preference optimization)

Builds on: DPO SimPO

Closest papers by meaning

• Turning Drift into Constraint: Robust Reasoning Alignment in Non-Stationary Multi-Stream Environments (ICML 2026 · similarity 0.87)
• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization (ICLR 2026 · similarity 0.87)
• TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.87)
• Iterative Reasoning Preference Optimization (NeurIPS 2024 · similarity 0.87)
• Preference Optimization for Reasoning with Pseudo Feedback (ICLR 2025 · similarity 0.86)
• Self-Consistency Preference Optimization (ICML 2025 · similarity 0.86)