Research map / Large language models / Reasoning and chain of thought

SAPO: Self-Adaptive Process Optimization Makes Small Reasoners Stronger

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAPO, a self-evolving framework for small language models that uses first-error detection and posterior verification (instead of step-wise Monte Carlo rollouts) to produce process supervision, then aligns the reasoner with ORPO, and introduces GSM_Process and MBPP_Process step-verification benchmarks.

Problem

Self-evolution methods for small language models rely mainly on outcome rewards and suffer a growing reasoner-verifier gap, while Monte Carlo process supervision is computationally expensive for step-level labeling.

Approach

Use the verifier's step-score differences to locate a potential first error, verify only the boundary steps with two rollouts from the reasoner, correct labels accordingly, retrain the verifier on these labels, and align the reasoner with ORPO on verifier-scored preference pairs, iterating over rounds.

Key results

• SAPO-iter3 reaches 41.62 on GSM8K and 31.72 on MATH (OOD) with Qwen-2.5-0.5B, versus 40.79 / 28.85 for Online-RFT and 46.24 / 34.53 for SFT+GRPO (which wins on Qwen math).
• On MBPP with Qwen-2.5-0.5B, SAPO-iter3 scores 36.67, the best among methods, above RPO (36.63) and SFT+GRPO (35.20).
• SAPS reportedly gives a 2-3x improvement in FLOPs and time over Shepherd-style step-wise rollout labeling.

Surprising findings

• Naive DPO after RFT substantially degrades performance (e.g., Qwen GSM8K 8.26 vs 37.83 for RFT).
• BoN evaluation of verifiers becomes unreliable: verifiers degrade under BoN across iterations while step-level accuracy steadily improves.

Limitations

• GSM8K verification uses a fixed subset of 100 problems per K-means cluster (800 total) rather than full dataset, to balance efficiency.
• Initial verifier labels rely on binary search (Omega) estimation, with alternative initializations left unexplored.
• GRPO baseline wins on Qwen-2.5-0.5B math, indicating gains are model-dependent.

Experimental setup

Datasets: GSM8K MBPP MATH HumanEval

Benchmarks: GSM8K MATH MBPP HumanEval GSM_Process MBPP_Process

Metrics: accuracy Pass@1 step correction prediction accuracy BoN accuracy FLOPs elapsed time

Baselines: CoT SFT RFT RFT+DPO Online-RFT RPO SFT+GRPO ORM OmegaPRM ShepherdPRM

Builds on: ORPO Math-Shepherd OmegaPRM RPO GRPO Rejection sampling fine-tuning

Closest papers by meaning

• A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula (ICML 2026 · similarity 0.87)
• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.86)
• Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking (ICLR 2026 · similarity 0.86)
• Scalable Best-of-N Selection for Large Language Models via Self-Certainty (NeurIPS 2025 · similarity 0.86)
• StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models (ICLR 2026 · similarity 0.86)
• GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · similarity 0.86)