Research map / Large language models / Alignment and preferences
Self-Consistency Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Consistency Preference Optimization (ScPO), an unsupervised iterative preference training method that builds preference pairs from self-consistency votes and weights the DPO+NLL loss by vote margin, improving reasoning on GSM8K, MATH, and ZebraLogic.
Problem
Self-training LLMs on complex reasoning tasks without gold answers or reliable reward signals; existing self-rewarding and reward-model approaches struggle on verifiable reasoning problems.
Approach
Use self-consistency (majority-vote frequency over sampled responses) as a proxy for correctness: the most consistent response is chosen and the least consistent rejected, with each pair weighted by its vote margin in an iterative weighted DPO+NLL loss; new problems are self-generated and filtered by consistency.
Key results
- On GSM8K, two iterations of unsupervised SCPO (M2) reach 63.91% greedy accuracy vs 41.17% for the zero-shot Llama-3 Base 8B seed model, within 1% of supervised IRPO with gold labels (64.29%).
- On ZebraLogic, SCPO-trained Llama-3 Instruct 8B (M2) reaches 18.1% overall puzzle accuracy, above Llama-3 70B (17.2%), Gemma-2 27B (16.3%) and Claude-3 Haiku (14.3%).
- Semi-supervised SCPO improves GSM8K greedy accuracy over IRPO_Gold by 2.35% (66.64% vs 64.29%) after two iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train for two unsupervised self-consistency preference optimization iterations instead of using the seed model | Llama-3 Base 8B · GSM8K | accuracy (greedy, %): 41.17 -> 63.91 | Helped |
| Train for two unsupervised self-consistency preference optimization iterations instead of using the seed model | Llama-3 Base 8B · MATH | accuracy (greedy, %): 14.46 -> 19.72 | Helped |
| Train for two unsupervised self-consistency preference optimization iterations instead of using the seed model | Llama-3 Instruct 8B · ZebraLogic | overall puzzle accuracy (%): 11.6 -> 18.1 | Helped |
| Use consistency-weighted loss rather than setting w(x)=1 (ablation) | Self-Consistency Preference Optimization M1 · GSM8K | accuracy (greedy, %): 58.53 -> 61.03 | Helped |
| Use consistency-weighted loss rather than setting w(x)=1 (ablation) | Self-Consistency Preference Optimization M2 · GSM8K | accuracy (greedy, %): 62.62 -> 63.91 | Helped |
| Use consistency-weighted loss rather than setting w(x)=1 (ablation) | Self-Consistency Preference Optimization M1 · MATH | accuracy (greedy, %): 15.92 -> 17.36 | Helped |
| Use consistency-weighted loss rather than setting w(x)=1 (ablation) | Self-Consistency Preference Optimization M2 · MATH | accuracy (greedy, %): 18.74 -> 19.72 | Helped |
| Raise the majority-vote filtering threshold from τ=0.1k to τ=0.5k (ablation) | Self-Consistency Preference Optimization M1 · MATH | test accuracy: 15.44 -> 17.36 | Helped |
| Raise the majority-vote filtering threshold from τ=0.5k to τ=0.7k (ablation) | Self-Consistency Preference Optimization M1 · MATH | test accuracy: 17.36 -> 14.76 | Hurt |
| Bootstrap preference pairs from test queries instead of train queries (ablation) | Self-Consistency Preference Optimization M3 · GSM8K | accuracy (greedy, %): 63.91 -> 65.35 | Helped |
| Bootstrap preference pairs from test queries instead of train queries (ablation) | Self-Consistency Preference Optimization M3 · MATH | accuracy (SC 8-way, %): 24.58 -> 25.84 | Helped |
Surprising findings
- Unweighted loss ablation shows the vote-margin weighting matters, and the ArmoRM reward model produced more incorrect pairwise orderings than self-consistency, even though it was trained on in-domain MATH step labels.
- Self-consistency at inference gives diminishing returns after the first training iteration because trained models become more consistent.
Limitations
- Gains depend on the model's consistency; on hard tasks like MATH only a fraction of seed problems yield clear majority answers.
- Self-consistency gains saturate after the first iteration and third iteration yields minimal gains.
- Requires a single parseable final answer, so it does not directly apply to open-ended tasks like summarization.
Experimental setup
Datasets: GSM8K MATH ZebraLogic
Benchmarks: GSM8K MATH ZebraLogic
Metrics: exact match accuracy (greedy) 8-way self-consistency accuracy puzzle accuracy cell accuracy
Baselines: Zero-shot CoT seed model IRPO (gold labels) IRPO with ArmoRM reward model LMSI Llama-3 Instruct 70B Gemma-2 27B Claude-3 Haiku
Builds on: DPO IRPO Self-Rewarding LMs Language Models Self-Improved (LMSI) self-consistency
Closest papers by meaning
- Optimal Self-Consistency for Efficient Reasoning with Large Language Models (ICML 2026 · similarity 0.88)
- Iterative Reasoning Preference Optimization (NeurIPS 2024 · similarity 0.87)
- TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.87)
- Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning (AAAI 2026 · similarity 0.87)
- Preference Optimization for Reasoning with Pseudo Feedback (ICLR 2025 · similarity 0.87)
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.