Research map / Large language models / Reasoning and chain of thought
SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCAN, a self-denoising Monte Carlo annotation and noise-tolerant PRM learning framework that uses self-confidence to cheaply build ~101K-197K synthetic step labels, reaching 59.1 F1 on ProcessBench with a 7B PRM.
Problem
Monte Carlo estimation produces noisy step-level labels for training process reward models because the completer model misestimates step correctness (both under- and over-estimation), and synthetic data is costly to annotate at scale.
Approach
Uses a self-confidence metric (the completer's empirical pass rate per question) to select reliable samples, apply MC annotation only to high-confidence negatives, soften labels within a tolerance distance before the first predicted error, and reweight step labels by self-confidence so that annotations from different completers are de-biased.
Key results
- Qwen2.5-Math-7B-SCAN-Pro reaches 59.1 average F1 on ProcessBench, up from 19.9 for Qwen2.5-Math-7B-Instruct and above the PRM800K-trained model at 56.5.
- Qwen2.5-Math-7B-SCAN-Pro achieves 70.1 average best-of-8 accuracy, matching Qwen2.5-Math-PRM-7B (70.1) and surpassing Qwen2.5-Math-7B-PRM800K (69.3).
- SCAN reports 6% of the inference cost of vanilla MC estimation, with 197K samples generated in 374.5 GPU hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the PRM on SCAN-Pro synthetic data | Qwen2.5-Math-7B-Instruct PRM · ProcessBench | average F1: 19.9 to 59.1 | Helped |
| scale synthetic training data from SCAN-Base (101K) to SCAN-Pro (197K) (ablation) | Qwen2.5-Math-7B PRM · ProcessBench | average F1: 56.8 -> 59.1 | Helped |
| add tolerance-distance soft labeling (ablation) | SCAN PRM training · Best-of-N and ProcessBench | performance | Helped |
| add confidence-wise reweighting (ablation) | SCAN PRM training · Best-of-N and ProcessBench | performance | Helped |
| use hard labeling with tolerance distance d = 0 instead of d = 2 (ablation) | SCAN PRM training · training scaling | performance | Hurt |
| use soft labeling with tolerance distance d = n instead of d = 2 (ablation) | SCAN PRM training · training scaling | performance | Hurt |
| increase self-confidence samples per question from k = 64 to k = 128 (ablation) | SCAN data synthesis · Best-of-8 and ProcessBench | performance | Helped |
| add GSM8K data to the training sources (ablation) | SCAN data synthesis · PRM evaluations | performance | Helped |
| add selected synthetic math data subsets from Numina-Math (ablation) | SCAN data synthesis · PRM evaluations | performance | Hurt |
| use Min-Vote instead of Min-Max aggregation (ablation) | SCAN-Base and SCAN-Pro · Best-of-8 | average accuracy: 69.1 -> 69.3; 70.1 -> 70.2 | Mixed |
| add SCAN denoising to vanilla MC annotations (ablation) | synthetic data annotated by Llama-3.1-8B-Instruct · ProcessBench proxy noise evaluation | noise ratio: 56.2% -> 19.1% | Helped |
| add SCAN denoising to vanilla MC annotations (ablation) | synthetic data annotated by Qwen2.5-Math-7B-Instruct · ProcessBench proxy noise evaluation | noise ratio: 51.8% -> 29.4% | Helped |
Surprising findings
- A 1.5B annotator can produce labels that yield a PRM outperforming PRMs trained on 1.5M samples with knowledge distillation from large critics on best-of-8 accuracy.
- Dataset-level noise remains: the authors estimate 19-29% noise after denoising on ProcessBench-based proxies, and MC-only PRMs underperform KD-based PRMs on fine-grained ProcessBench.
Limitations
- Monte Carlo estimation cannot detect false positives where the final answer is correct but intermediate steps contain errors.
- Weaker on fine-grained step-level error detection than KD-based PRMs; error types like logical and conceptual errors are detected less reliably.
- Evaluation is restricted mainly to math; the general-domain GPQA-Diamond test is small and limited.
Experimental setup
Datasets: MATH training set (7,500 questions) GSM8K Numina-Math ProcessBench PRM800K
Benchmarks: GSM8K MATH College Math OlympiadBench ProcessBench GPQA-Diamond
Metrics: best-of-8 accuracy F1 on ProcessBench pass@8 majority vote
Baselines: Math-Shepherd RLHFlow-PRM Skywork-PRM Math-PSA EurusPRM Qwen2.5-Math-PRM-7B UniversalPRM Qwen2.5-Math-7B-PRM800K Majority vote
Builds on: Math-Shepherd ProcessBench Monte Carlo estimation for process supervision vLLM
Closest papers by meaning
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.89)
- Know What You Don't Know: Uncertainty Calibration of Process Reward Models (NeurIPS 2025 · similarity 0.88)
- Free Process Rewards without Process Labels (ICML 2025 · similarity 0.88)
- VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data (ICML 2025 · similarity 0.87)
- Improving Value-based Process Verifier via Low-Cost Variance Reduction (AAAI 2026 · similarity 0.87)
- Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.