Research map / Large language models / Reasoning and chain of thought

SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SPARE, a single-pass, reference-guided LLM annotation framework that jointly aligns model solution steps to reference steps and labels correctness with explicit reasoning, used to train PRMs and to build preference pairs for ORPO fine-tuning.

Problem

Automatic process (step-level) supervision is expensive: MCTS-based annotation is token-costly and ignores step information already in reference solutions, while other reference-based methods depend on step labels or stronger-model rationales.

Approach

A structured system prompt with relevancy directives and few-shot exemplars makes one LLM generation perform step alignment (one-to-one and one-to-many, to reference steps, context, or other output steps) and then assign a correct/incorrect label with explanation and error category, enabling single-pass annotation using only the same model and existing SFT reference solutions.

Key results

• SPARE-PRM with Best-of-N on SpaRP-S reaches 43.7 accuracy vs 41.7 for outcome ORM, and SPARE-ORPO reaches 23.4 on MATH vs 23.1 for outcome ORPO (BoN/SC and greedy results in Table 1).
• On ProcessBench, SPARE-Qwen2.5-3B achieves 85.0 correct-solution accuracy on OlympiadBench and 83.8 on Omni-MATH, using ~16% of the training samples of PRM800K.
• SPARE uses 4,591 tokens vs 10,569 for MCTS on MATH (2.3x fewer tokens) and reduces runtime 2.6x, with label accuracy of 76.2 vs 76.4 for MCTS.

Surprising findings

• Pointwise ORM outperforms pairwise ORM significantly on MATH and SpaRP, and SPARE-ORM underperforms pointwise ORM despite superior pairs.
• Self-consistency aggregation underperforms even ORMs on datasets with limited reasoning variation (MuSiQue, SpaRP-S).

Limitations

• Annotation quality depends on the quality of reference solutions; noisy references not evaluated.
• LLM-based grading is noisy and shows failure modes (ignoring reference, premature termination, lenient grading).
• Experiments limited to selected datasets and models due to computational constraints; no multilingual or multimodal evaluation.

Experimental setup

Datasets: GSM8K MATH MuSiQue-Ans SpaRP PRM800K ProcessBench

Benchmarks: GSM8K MATH-500 MuSiQue-Ans SpaRP-S ProcessBench

Metrics: accuracy F1 macro-F1 label accuracy token count runtime

Baselines: Math-Shepherd ORM Self-Consistency Outcome-ORPO SFT DIVERSE-NLI Rest-MCTS* PRM800K-trained Qwen-2.5-Math-7B-PRM RLHFlow-Deepseek-8B Skywork-7B

Builds on: Math-Shepherd ORPO ProcessBench Self-Consistency

Closest papers by meaning

• AlphaMath Almost Zero: Process Supervision without Process (NeurIPS 2024 · similarity 0.89)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.88)
• Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.88)
• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.88)
• GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · similarity 0.88)
• AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (ICML 2025 · similarity 0.87)