Research map / Large language models / Alignment and preferences
Preference Optimization for Reasoning with Pseudo Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PFPO, which builds pseudo test-case feedback (from frontier LLMs or multi-test-case self-consistency) to create preference pairs for DPO on math and coding reasoning tasks without human labels.
Problem
Preference optimization for LLM reasoning needs reliable labels, but human-verified reasoning data with answers or test cases is scarce and expensive to produce at scale.
Approach
Frame labeling of reasoning solutions as evaluation against test cases, and generate those test cases as pseudo feedback either from frontier LLM solutions (majority-voted outputs) or from the policy's own self-consistency, extended to multiple test cases for coding; then build outcome and process preference pairs for DPO.
Key results
- Mathstral-7B with PFPO-Self reaches 68.6 on MATH (from 58.3 base, 61.4 SFT), surpassing NuminaMath-72B-CoT and GPT-4-Turbo-1106-preview (64.3).
- Deepseek-coder-7B-v1.5 w/ PFPO reaches 24.6 on LiveCodeBench (from 21.1), above Claude-3-Haiku (24.0).
- On APPs, PFPO-Self Iter. 2 (APPs & M.C. & xCode.) improves from 15.4 SFT to 19.1 overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add frontier-LLM pseudo feedback after SFT | Mathstral-7B-v0.1 · MATH | accuracy: 61.4 -> 66.7 | Helped |
| Add iterative self-consistency pseudo feedback after frontier feedback | Llama-3.1-8B-base · MATH | accuracy: 55.0 -> 57.8 | Helped |
| Add iterative self-consistency pseudo feedback after frontier feedback | Mathstral-7B-v0.1 · MATH | accuracy: 66.7 -> 68.6 | Helped |
| Use frontier-LLM pseudo feedback before self-consistency feedback (ablation) | PFPO training · Mathstral-7B-v0.1, MATH | accuracy: 64.6 -> 65.2 | Helped |
| Use self-consistency pseudo test cases instead of ground-truth test cases (ablation) | Deepseek-coder-v1.5-Instruct SFT · APPs overall | Pass@1: 16.9 -> 17.4 | Helped |
| Use self-consistency pseudo test cases instead of ground-truth test cases (ablation) | Deepseek-coder-7B-v1.5-Instruct SFT · LiveCodeBench overall | Pass@1: 22.9 -> 23.4 | Helped |
| Reduce the number of synthetic test cases to the ground-truth test-case count (ablation) | PFPO-Self · APPs overall | Pass@1: 17.4 -> 14.8 | Hurt |
| Reduce the number of synthetic test cases to the ground-truth test-case count (ablation) | PFPO-Self · LiveCodeBench overall | Pass@1: 23.4 -> 21.1 | Hurt |
| Use pDPO instead of DPO (ablation) | Mathstral-7B-v0.1 SFT · MATH | accuracy: 63.0 -> 66.7 | Helped |
| Use pDPO instead of DPO (ablation) | Deepseek-coder-v1.5-Instruct SFT · HumanEval | Pass@1: 74.4 -> 73.8 | Hurt |
| Add xCodeEval prompts during iterative self-consistency training (ablation) | Deepseek-coder-v1.5-Instruct · HumanEval and MBPP | Pass@1 | Mixed |
| Add self-consistency training iterations on a fixed prompt set (ablation) | Mathstral-7B-v0.1 · MATH | accuracy: 67.8 -> 68.6 -> 68.2 | Mixed |
Surprising findings
- Pseudo test cases built from synthetic inputs outperformed ground-truth test cases on APPs and LiveCodeBench, attributed to having ~10 test cases per problem versus ~5.16 originally.
- Iterative training on a fixed prompt set overfits as pseudo test-case quality saturates.
Limitations
- Pseudo test cases contain errors (e.g., 62.68% pass rate for self-generated tests), causing false positives/negatives in preference labels.
- Iterative self-consistency training saturates on fixed prompts and needs fresh unseen high-quality prompts, which are scarce in coding.
Experimental setup
Datasets: MathScale NuminaMath APPS Magicoder xCodeEval MWPBench GSM8K MATH
Benchmarks: MATH GSM8K College Math APPS HumanEval MBPP LiveCodeBench MATH-Challenging
Metrics: accuracy Pass@1
Baselines: DPO pDPO NuminaMath-72B-CoT GPT-4-Turbo-1106-preview Claude-3-Haiku Llama-3.1-70B-Instruct
Builds on: Direct Preference Optimization pDPO self-consistency MathScale
Closest papers by meaning
- Building Math Agents with Multi-Turn Iterative Preference Learning (ICLR 2025 · similarity 0.89)
- Iterative Reasoning Preference Optimization (NeurIPS 2024 · similarity 0.89)
- Rectify Evaluation Preference: Improving LLMs’ Critique on Math Reasoning via Perplexity-aware Reinforcement Learning (AAAI 2026 · similarity 0.88)
- Advancing LLM Reasoning Generalists with Preference Trees (ICLR 2025 · similarity 0.88)
- TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.88)
- GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.