Research map / Reinforcement learning / RL from feedback
PIPA: Preference Alignment as Prior-Informed Statistical Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PIPA, an RL-free probabilistic framework that casts offline preference alignment as prior-constrained MLE, unifying DPO and KTO and supporting paired/unpaired and answer/step-level data, evaluated on GSM8K and MATH with DeepSeek-Math-7B and Qwen2.5-Math-7B.
Problem
Offline preference alignment methods (DPO, KTO, Step-DPO, Step-KTO, etc.) are designed for specific data settings and lack a unified statistical understanding; they differ in paired vs. unpaired data and answer- vs. step-level annotation.
Approach
Formulate preference alignment as maximum likelihood estimation of the correctness label c given (x,y) subject to a prior constraint on the marginal (PIPA-M) or negative conditional (PIPA-N) distribution; Bayes' rule yields a token-level loss with a learned value head g(x,y<t), and DPO and KTO are recovered as special cases with different priors.
Key results
- PIPA-N reaches 80.29 on GSM8K and 52.32 on MATH with DeepSeek-Math-7B in the paired answer-wise setting, versus 68.39 and 46.94 for DPO.
- PIPA-M achieves 74.91 on GSM8K (Qwen, paired step-wise) vs 66.11 for Step-DPO, and 53.26 on MATH vs 48.38.
- Removing the learned value head (fixing p(c_t|x,y<t)=0.5) drops PIPA-M on GSM8K from 78.24 to 73.54.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PIPA-M or PIPA-N instead of the strongest listed baseline for each setup | preference alignment · GSM8K and MATH | accuracy | Helped |
| fix the value model output at 0.5 instead of learning it (ablation) | PIPA-M · GSM8K, unpaired step-wise | accuracy: 78.24 -> 73.54 | Hurt |
| fix the value model output at 0.5 instead of learning it (ablation) | PIPA-M · MATH, unpaired step-wise | accuracy: 51.82 -> 47.92 | Hurt |
| fine-tune the prior on all samples (ablation) | PIPA-M · GSM8K, unpaired step-wise | accuracy: 78.24 -> 77.86 | Hurt |
| fine-tune the prior on all samples (ablation) | PIPA-M · MATH, unpaired step-wise | accuracy: 51.82 -> 50.60 | Hurt |
| fine-tune the prior on negative samples (ablation) | PIPA-N · GSM8K, unpaired step-wise | accuracy: 79.98 -> 79.83 | Hurt |
| fine-tune the prior on negative samples (ablation) | PIPA-N · MATH, unpaired step-wise | accuracy: 52.78 -> 50.84 | Hurt |
| add SFT loss with coefficient 1.0 (ablation) | PIPA-M · GSM8K, unpaired answer-wise | accuracy: 79.08 -> 78.24 | Hurt |
| add SFT loss with coefficient 1.0 (ablation) | PIPA-M · MATH, unpaired answer-wise | accuracy: 50.82 -> 50.12 | Hurt |
| add step-level rather than answer-level annotations | alignment algorithms · GSM8K and MATH | accuracy | Mixed |
| use PIPA-N rather than PIPA-M | PIPA · across tested settings and models | accuracy | Mixed |
| use Step-KTO loss instead of the original KTO approach for step labels | KTO · math reasoning | performance | No clear effect |
Surprising findings
- Using a more accurate prior (a model fine-tuned on all or negative samples) did not improve PIPA and sometimes hurt performance.
- Step-level annotation does not consistently improve results; it helps on MATH but can hurt GSM8K.
Limitations
- Evaluated mainly on math reasoning; general tasks left for future work
- Offline static dataset only; iterative/online version not explored
- Gains depend on SFT model used as prior; more accurate priors did not help
Experimental setup
Datasets: AlphaMath GSM8K MATH
Benchmarks: GSM8K MATH
Metrics: accuracy
Baselines: DPO IPO KTO Step-DPO Step-KTO
Builds on: DPO KTO Step-DPO Step-KTO
Closest papers by meaning
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.91)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.91)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.91)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.