Research map / Large language models / Reasoning and chain of thought
AutoPSV: Automated Process-Supervised Verifier
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoPSV, which trains an outcome-supervised verifier and then automatically derives step-level process labels from relative changes in its confidence scores, used to retrain the verifier for best-of-5 answer selection on math and commonsense tasks.
Problem
Process-supervised verifiers need costly step-level annotations (human or MCTS-based model-induced), while outcome-supervised verifiers are cheaper but less effective; the paper seeks automatic process labels without ground-truth answers.
Approach
Use the confidence scores of an outcome-supervised verifier at successive reasoning steps; a relative drop in confidence marks a step as erroneous (first-error strategy), producing process labels with no extra sampling or human annotation, which then train a process-supervised verifier.
Key results
- On GSM8K, OSV+PSV improves over OSV for all generators (e.g., Mixtral-Instruct 74.91 to 76.04; Qwen 84.91 to 85.15).
- On MATH, OSV+PSV reaches 16.92 for Mixtral-Instruct versus 15.2 for OSV, but is below Self-Consistency for Qwen-72b (39.36 vs 40.10).
- Adding 7,000 unlabeled WizardLM questions raises Mistral-Instruct GSM8K accuracy from 61.18 (OSV) to 63.11 (OSV+PSV).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Relabel OSV training data with AutoPSV relative step-confidence change (OSV+PSV) instead of outcome-only verifier | verifier, Mistral-Instruct generator · GSM8K | selection accuracy (%): +0.23 over OSV (61.18 to 61.41) | Helped |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mixtral-Instruct generator · GSM8K | selection accuracy (%): +1.13 over OSV (74.91 to 76.04) | Helped |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Qwen-72B generator · GSM8K | selection accuracy (%): +0.24 over OSV (84.91 to 85.15) | Helped |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mistral-Instruct generator · MATH | selection accuracy (%): +0.20 over OSV (5.10 to 5.30) | Helped |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mixtral-Instruct generator · MATH | selection accuracy (%): +1.72 over OSV (15.2 to 16.92) | Helped |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Qwen-72B generator · MATH | selection accuracy (%): -0.70 vs Self-Cons. (40.10 to 39.36); +0.42 over OSV | Hurt |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mistral-Instruct generator · HellaSwag | selection accuracy (%): +0.64 over OSV (73.81 to 74.45) | Helped |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mixtral-Instruct and Qwen-72B generators · HellaSwag | selection accuracy (%): Mixtral +0.79 (82.83 to 83.62); Qwen +0.91 (93.08 to 93.99) | Helped |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mistral-Instruct generator · Winogrande | selection accuracy (%): +0.82 over OSV (79.16 to 79.98) | Helped |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mixtral-Instruct and Qwen-72B generators · Winogrande | selection accuracy (%): Mixtral +0.48 (73.40 to 73.88); Qwen -1.02 (80.34 to 79.32) | Hurt |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mistral-Instruct generator · ANLI | selection accuracy (%): -0.5 (59.8 to 59.3) | Hurt |
| Relabel OSV training data with AutoPSV process labels (OSV+PSV) | verifier, Mixtral-Instruct and Qwen-72B generators · ANLI | selection accuracy (%): Mixtral +1.1 (62.9 to 64.0); Qwen +2.3 (69.1 to 71.4) | Helped |
Surprising findings
- Adding unlabeled questions improves selection performance, showing the method works without ground-truth answers.
- The PSV retrained on AutoPSV labels is only slightly below the OSV it was derived from, suggesting the process labels inherit outcome information.
Limitations
- Process labels are estimates from confidence changes and may misrepresent true step correctness.
- Verifier quality bounds annotation quality.
- Evaluated mainly on math and commonsense multiple-choice-like selection; generalization to other tasks is unclear.
Experimental setup
Datasets: GSM8K MATH HellaSwag Winogrande ANLI WizardLM Evol-Instruct unlabeled questions
Benchmarks: GSM8K MATH HellaSwag Winogrande ANLI
Metrics: accuracy Pass@5 self-consistency accuracy precision recall F1-score
Baselines: Self-Consistency Pass@5 OSV MCTS-based process labeling
Builds on: Math-Shepherd outcome-supervised verifier Monte Carlo tree search annotation self-consistency
Closest papers by meaning
- Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks (ICLR 2025 · similarity 0.86)
- CITE: Anytime Valid Statistical Inference in LLM Self-Consistency (NeurIPS 2026 · similarity 0.86)
- Step-Level Sparse Autoencoder for Reasoning Process Interpretation (ICML 2026 · similarity 0.86)
- SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling (AAAI 2026 · similarity 0.86)
- Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution (ICML 2026 · similarity 0.86)
- ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.