Research map / Large language models / Reasoning and chain of thought

Self-Trained Verification for Training- and Test-Time Self-Improvement

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes self-trained verification (STV), training a verifier via on-policy distillation from a reference-conditioned teacher of itself, and verifier-in-the-loop (ViL) RL training of the generator, improving both test-time refinement and standalone accuracy on hard math and science problems.

Problem

Verifier-guided refinement stalls because verifiers cannot reliably catch errors in plausible but wrong solutions, and self-training on unverified self-generated data is harmful; no human-labeled feedback-quality signal exists for training verifiers.

Approach

A reference-conditioned version of the same model acts as a teacher that is better at diagnosing flaws; the unconditioned student verifier is trained via on-policy distillation (JS divergence) to match the teacher's verdict+feedback distribution, plus verdict RL. The trained verifier then drives RL training of the generator inside the V-R loop (ViL).

Key results

• STV verifier lifts final-round pass@1 on hard DAPO math by up to 2x over the untrained verifier; STV-guided Qwen3-8B (27.4% on Hard) beats Qwen3-32B without verification (17.8%).
• On SciKnowEval Hardest bin, STV raises pass@1 from 1.5% (no verification) to 21.0%, exceeding Qwen3-235B-A22B (8.0%).
• ViL training raises round-0 pass@1 by about 30% relative on Hard (36.7% to 47.7%) without any verifier at test time, while continued RLVR gives no gain.

Surprising findings

• Continuing standard RLVR for the same extra steps yields no gain once converged, whereas ViL training improves standalone round-0 accuracy.
• Meta-verifier RL and SFT on teacher traces gave little or no benefit, unlike on-policy distillation.

Limitations

• Evaluated mainly on math and scientific QA; generalization to code and open-ended reasoning is untested.
• Uses the same training problems for verifier and generator training; data selection for verifier training left open.
• Meta-verifier baseline uses GPT-5.2 as a proxy for the original meta-verifier.

Experimental setup

Datasets: DAPO SciKnowEval

Benchmarks: DAPO hard math splits SciKnowEval

Metrics: pass@1 pass@k precision-coverage

Baselines: No verifier training Verdict-RL verifier Meta-verifier RL verifier SFT verifier Qwen3-32B Qwen3-235B-A22B RLVR-only (longer) Prefix-conditioning

Builds on: on-policy distillation RLVR DAPO prefix-conditioning

Closest papers by meaning

• V1: Unifying Generation and Self-Verification for Parallel Reasoners (ICML 2026 · similarity 0.87)
• Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.87)
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards (NeurIPS 2025 · similarity 0.86)
• Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.86)
• Better, Faster: Harnessing Self-Improvement in Large Reasoning Models (ICML 2026 · similarity 0.86)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.86)