Research map / Large language models / Reasoning and chain of thought
Generative Verifiers: Reward Modeling as Next-Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Generative Verifiers (GenRM), which train LLM verifiers with next-token prediction on 'Yes'/'No' tokens, optionally with synthetic chain-of-thought verification rationales and majority-vote inference, and shows gains over discriminative RMs on algorithmic and math Best-of-N.
Problem
Discriminative LLM verifiers for reasoning are trained as classifiers with a scalar head, so they do not use the text generation abilities of LLMs (CoT, instruction tuning, extra inference compute), and LLM-as-a-judge underperforms trained verifiers.
Approach
Recast verification as next-token prediction: the verifier is SFT-trained to output 'Yes' or 'No' and scores solutions by the 'Yes' probability; it can generate verification CoT rationales (synthetic, reference-guided) and average Yes-probabilities over majority-vote samples, and can be co-trained with solution generation.
Key results
- On GSM8K with Gemma2-9B GenRM-CoT verifier on Gemini 1.0 Pro solutions, problems solved rises from 73% to 93.4%, surpassing GPT-4 and Gemini 1.5 Pro.
- GenRM-CoT matches discriminative verifier's Best-of-32 on MATH using 6.4x fewer solutions, solving about 17% more MATH500 problems with Best-of-32 (easy-to-hard generalization).
- Weighted self-consistency with GenRM-CoT needs 2.5x fewer solutions than with discriminative RM to reach the same performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenRM-CoT instead of Disc-RM | GSM verifiers evaluated on MMLU elementary_mathematics · MMLU elementary_mathematics | accuracy: 90.6% -> 91.1% (+0.5%) | Helped |
| use GenRM-CoT instead of Disc-RM | GSM verifiers evaluated on MMLU high_school_mathematics · MMLU high_school_mathematics | accuracy: 74.8% -> 76.1% (+1.3%) | Helped |
| use GenRM-CoT instead of Disc-RM | GSM verifiers evaluated on MMLU college_mathematics · MMLU college_mathematics | accuracy: 53% -> 56.1% (+3.1%) | Helped |
| use GenRM-CoT instead of Disc-RM | GSM verifiers evaluated on MMLU abstract_algebra · MMLU abstract_algebra | accuracy: 50% -> 53.50% (+3.5%) | Helped |
| use GenRM-CoT verification on Gemini 1.0 Pro solutions instead of the reported 73% result | Gemma2-9B verifier on GSM8K · GSM8K | problems solved: 73% -> 93.4% | Helped |
| add reference guidance when generating synthetic verification rationales (ablation) | GenRM-CoT with Gemma-7B · GSM8K | Best-of-N accuracy: 87.8% -> 91.7% | Helped |
| combine GenRM-CoT scores with weighted self-consistency | GSM8K solution selection · GSM8K | problems solved: 93.4% -> 93.5% | No clear effect |
| train verification only, omitting SFT on correct solutions (ablation) | GenRM and GenRM-CoT · All evaluated tasks | verification performance | Hurt |
| increase the solution-generation data weight too much (ablation) | GenRM · Verification tasks | verification performance | Hurt |
| increase the number of synthetic rationales per solution (ablation) | GenRM-CoT with Gemma-7B · GSM8K | RM accuracy | Helped |
| increase the number of solutions per problem from 8 to 32 (ablation) | GenRM-CoT · Evaluated tasks | RM accuracy: 8 to 32 | Helped |
Surprising findings
- Gemini 1.0 Pro used as an LLM-as-a-Judge underperforms trained Gemma verifiers, highlighting the need to train generative verifiers.
- Synthetic model-generated verification rationales, which can be error-prone, suffice to teach CoT verification on math tasks without a stronger model or human rationales.
Limitations
- Synthetic CoT rationales can contain errors; GSM8K gains from unification are smaller than on algorithmic tasks.
- Majority voting with many CoT samples increases inference cost.
- Experiments rely on Google internal Gemma/Gemini models and a single generator, limiting generality.
Experimental setup
Datasets: Last Letter Concatenation Word Sorting (Big-Bench) GSM8K MATH MMLU
Benchmarks: Last Letter Concatenation BBH Word Sorting GSM8K MATH500 MMLU math subsets
Metrics: Best-of-N accuracy (percentage of problems solved) RM accuracy pass@N
Baselines: Discriminative RM LLM-as-a-Judge DPO verifier self-consistency oracle verifier
Builds on: Best-of-N verification self-consistency DPO chain-of-thought prompting
Closest papers by meaning
- Variation in Verification: Understanding Verification Dynamics in Large Language Models (ICLR 2026 · similarity 0.91)
- Reinforcing General Reasoning Without Verifiers (ICLR 2026 · similarity 0.91)
- RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning (NeurIPS 2025 · similarity 0.90)
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.89)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.89)
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.