Research map / Large language models / Reasoning and chain of thought

P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes P2S, a verifier-free RL framework for general-domain QA that combines dynamically synthesized gold chain-of-thought references with a step-wise Path Faithfulness Reward based on conditional probabilities of gold-CoT suffixes.

Problem

Reinforcement learning for reasoning in general-domain QA lacks verifiable rewards, and outcome-only probability rewards (RLPR) ignore step-level supervision, leading to reward sparsity and shortcut solutions.

Approach

Generate gold-CoT references by prompting the policy with the ground-truth answer, select the best by answer log-probability, and reward each reasoning prefix by the masked-baseline-normalized log-probability gain it gives to gold-CoT suffixes; combine with outcome rewards via hierarchical integration and sigmoid step weighting.

Key results

• On DROP, P2S reaches ACC_Avg 70.70 vs 68.40 for the strongest fine-tuned baseline (SFT+GRPO) and 67.60 for RLPR.
• On MedicalQA, P2S achieves ACC_Avg 24.28 vs 22.94 for RLPR.
• Ablation on DROP: removing Gold-CoT filtering drops ACC_Avg by 4.5 points; removing PFR drops it by 2.3 points.

Surprising findings

• The General Reasoner's verifier-based accuracy on MedicalQA (27.45) is inflated relative to Claude (19.18) and GPT (17.20) judgments, suggesting verifier unreliability.
• The 1.5B P2S model outperforms the untuned 3B base model (70.70 vs 62.77 ACC_Avg on DROP).

Limitations

• Reward depends on the policy's own probabilities and a self-generated gold-CoT, so quality of the reference is bounded by the policy's ability.
• Evaluation limited to DROP and MedicalQA with a 1.5B model (plus a 3B scale check) and short answers (1-50 characters).
• Relies on LLM judges (Claude, GPT-4o, 1.5B verifier) for accuracy with no human evaluation.

Experimental setup

Datasets: DROP MedicalQA

Benchmarks: DROP MedicalQA

Metrics: ROUGE-1 F1 ACC_Claude ACC_GPT ACC_Verifier ACC_Avg

Baselines: Chain-of-Thought Self-Consistency Full-SFT GRPO GRPO+SFT-loss SFT+GRPO General Reasoner DRO RLPR VeriFree

Builds on: GRPO RLPR DeepSeek-R1 General Reasoner VeriFree

Closest papers by meaning

• Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.88)
• Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning (ICML 2024 · similarity 0.88)
• RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.88)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.87)
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.87)
• ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs (NeurIPS 2025 · similarity 0.87)