Research map / Large language models / Alignment and preferences

INSPO : Unlocking Intrinsic Self-Reflection for LLM Preference Optimization

NeurIPS 2026 · Large language models › Alignment and preferences · approach: DPO design choices · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InSPO, a DPO-family plug-in that trains the policy on preference pairs with symmetric cross-conditioning on the alternative response, claiming invariance to scalarization/reference policy and zero inference overhead.

Problem

DPO-style preference optimization yields a policy that depends on modeling choices (scalarization function, reference policy) and is suboptimal because it ignores comparative information between responses.

Approach

Derive a globally optimal policy conditioned on both the context and an alternative response, and approximate it by training the policy on each response conditioned on the other (symmetric cross-conditioning) inside DPO-family losses; the conditioning is dropped at inference (learning using privileged information).

Key results

• InS-IPO on Mistral-7B-Instruct improves AlpacaEval 2 LC from 20.3 to 28.8 (+8.5).
• InS-DPO on Mistral-7B-Instruct improves Arena-Hard WR from 16.3 to 23.9 (+7.6).
• InS-SimPO on Llama-3-8B-Instruct reaches 46.6 Arena-Hard WR, +6.1 over SimPO's 40.5 raw AlpacaEval WR baseline level (table lists 40.5 WR for SimPO on AlpacaEval 2).

Surprising findings

• Symmetric cross-conditioning yields zero LC gain on AlpacaEval 2 for Llama-3-8B DPO (40.3 to 40.3), while one-sided conditioning gives +3.1.

Limitations

• Training cost increases by roughly 18-25% due to longer cross-conditioned sequences and two forward passes.
• Gains on some variants (e.g., ORPO, SimPO on Mistral) are marginal or negative on certain metrics.
• Invariance and optimality claims rely on an idealized policy class; empirical link is indirect.

Experimental setup

Datasets: UltraFeedback

Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench

Metrics: length-controlled win rate win rate style-controlled win rate MT-Bench score

Baselines: DPO IPO ORPO R-DPO SimPO RRHF SLiC-HF CPO KTO

Builds on: DPO SimPO IPO ORPO R-DPO

Closest papers by meaning

• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.92)
• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.90)
• Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.90)
• InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.90)
• Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.90)