Research map / Large language models / Alignment and preferences

Understanding the Logic of Direct Preference Alignment through Logic

ICML 2025 · Large language models › Alignment and preferences · approach: DPO design choices · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes direct preference alignment losses (DPO, CPO, ORPO, SimPO, etc.) as symbolic logic programs via a generalized semantic loss over "preference structures," enabling decompilation, a loss lattice, and derivation of new losses tested in a small Qwen-0.5B study.

Problem

Direct preference alignment (DPA) losses are numerous and their semantics and relationships are hard to compare or extend without a formal framework for reasoning about what they encode.

Approach

Introduces a probabilistic logic built on generalized semantic loss with "preference structures" (core formula, conditioning constraints, additive constraints), and a decompilation algorithm that translates DPA loss ratios of disjoint multilinear polynomials into such structures, so losses can be compared by logical entailment and new ones derived.

Key results

• The new loss cCPO achieves a 52.0% win-rate against CPO on the UltraFeedback test set with Qwen-0.5B, roughly on par with the baseline.
• Different derived losses perform very differently across data subsets (e.g., qfUNL gets 34.7% on false-qa and 57.9% on flan), suggesting no single loss dominates.
• The formal framework shows the DPA loss space contains up to about 4.3 billion definable DPO-style losses, with preference structures ordered by logical entailment.

Measured effects reported in the paper

Surprising findings

• Unconstrained losses such as unCPO can be satisfied by driving both winner and loser probabilities toward zero, a shortcut behavior revealed by the logical analysis.
• Standard semantic loss cannot compositionally express CPO's loss, so a more expressive preference-structure encoding is needed.

Limitations

• Experiments are small scale: a single 0.5B model and one dataset, with a win-rate that shows only feasibility.
• The framework assumes compositional decompilation over disjoint multilinear polynomials, which excludes some losses such as the original DPOP without variable copying.
• Results are for offline single-pass DPA and do not cover online variants.

Experimental setup

Datasets: UltraFeedback deita

Benchmarks: UltraFeedback test subsets (evol, false-qa, flan, sharegpt, ultrachat)

Metrics: win-rate vs CPO based on reward model score

Baselines: DPO CPO ORPO SimPO IPO SliC RRHF CE CEUnl

Builds on: DPO semantic loss CPO ORPO SimPO

Closest papers by meaning

• Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.89)
• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
• As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.89)
• Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.89)
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.88)
• Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.88)