Research map / Large language models / Instruction and fine-tuning

Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Maximum Entropy Blackwell Winner solution concept and PROSPER, a regression-based single-player algorithm for multi-objective preference fine-tuning under intransitive LLM-judge rubric feedback.

Problem

Preference fine-tuning breaks down when preferences are intransitive, which arises from inconsistent per-objective judgments and from scalarizing multiple objectives (e.g., rubric-based LLM judges); no well-defined optimal policy exists and the minimax theorem fails in the multi-objective setting.

Approach

Defines the MaxEntBW as the maximizer of a game value where the adversary comparator is KL-regularized to a reference policy and objectives are chosen adversarially; entropy regularization gives a closed-form adversary, reducing the problem to a single-player concave maximization solved by online mirror descent implemented via square-loss regression, with the worst objective chosen per prompt.

Key results

• On Qwen2.5-7B-Instruct, PROSPER reaches 49.2 Arena-Hard vanilla vs 42.5 for RLCF and 42.4 for the base model.
• PROSPER gets 55.4 vanilla AlpacaEval vs 41.4 for RLCF; length-controlled AlpacaEval is 37.61, a close second to PROSPER-JC's 38.21.
• In judge win-rates, PROSPER beats the base policy roughly 3/4 of the time and RLCF roughly 2/3 of the time; on the 7B model, average of the 5 general benchmarks is 71.70 vs 72.55 for the base model.

Surprising findings

• Splitting a checklist into per-item judgments reduces but does not eliminate intransitivity and absence of a Condorcet winner.
• The released RLCF checkpoint performs comparably to the authors' re-implementation, so the reported gains are not due to a weak RLCF re-implementation.

Limitations

• The regret guarantee assumes the regression step is solved accurately in-distribution with error epsilon.
• Evaluation relies on LLM judges (Qwen3-14B and gpt-5-mini), and PROSPER-JC is competitive on length-controlled AlpacaEval.
• Experiments are limited to Qwen2.5 3B and 7B models on one training dataset.

Experimental setup

Datasets: WildChecklists WildChat

Benchmarks: AlpacaEval 2.0 Arena-Hard IFEval MMLU ARC HellaSwag TruthfulQA

Metrics: win rate length-controlled win rate style-controlled win rate accuracy

Baselines: RLCF PROSPER-JC PROSPER-VB

Builds on: Blackwell Winner REBEL RLCF Minimax Winner / von Neumann Winner self-play Online Mirror Descent

Closest papers by meaning

• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.89)
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data (ICML 2024 · similarity 0.88)
• Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.87)
• AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.87)
• INSPO : Unlocking Intrinsic Self-Reflection for LLM Preference Optimization (NeurIPS 2026 · similarity 0.87)
• Mechanism Design for LLM Fine-tuning with Multiple Reward Models (NeurIPS 2025 · similarity 0.87)