Research map / Large language models / Reasoning and chain of thought

VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VerifyBench and VerifyBench-Hard, benchmarks for evaluating reference-based (answer-checking) reward systems used in RL for reasoning models, with human-annotated correctness labels and evaluations of rule-based, LLM-judge, and specialized verifiers.

Problem

Existing reward benchmarks evaluate pairwise preference rankings, not whether a single completion matches a ground-truth reference answer, which is how reference-based rewards are used in RL for reasoning models.

Approach

Builds a reference-based verification benchmark with single-completion correctness labels across four answer types, plus a harder subset selected by disagreement among strong LLM judges, and correlates verifier accuracy with downstream GRPO gains.

Key results

• On VerifyBench, gpt-oss-120b and Qwen3-32B-class judges reach about 95-96% average accuracy, with gpt-oss-120b at 95.85%.
• On VerifyBench-Hard, the best system is gpt-oss-120b at 87.90% average, with CompassVerifier-32B tied at 87.90%; rule-based math-verify reaches 76.00%.
• Removing the reference answer from the judge prompt drops LLM-judge accuracy by roughly 5-18 points on VerifyBench.

Surprising findings

• Removing the reference answer from the judge prompt causes large accuracy drops, showing reference answers are crucial for verification.
• Smaller models (under 3B) perform notably worse, with Llama-3.2-3B-Instruct at 60.95% on VerifyBench.

Limitations

• Covers only general, logical and math reasoning; proof problems and commonsense reasoning excluded.
• Binary correctness labels; no partial credit.
• Human annotation bias and reward hacking not evaluated.

Experimental setup

Datasets: BBH BBEH MMLU-Pro natural_reasoning AbductionRules anli FOLIO gsm8k MATH NuminaMath-CoT DeepMath-103K OlympiadBench-EN

Benchmarks: VerifyBench VerifyBench-Hard RM-Bench RewardBench MATH500 AMC23 MinervaMath

Metrics: accuracy

Baselines: math-verify gpt-4o-2024-11-20 DeepSeek-V3-0324 DeepSeek-R1-0528 gpt-oss-120b xVerify CompassVerifier Tencent-Qwen2.5-7B-RLVR Skywork-Reward-Llama-3.1-8B Qwen2.5-Math-RM-72B

Builds on: GRPO math-verify DeepSeek-R1 VerifierBench

Closest papers by meaning

• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.92)
• RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.90)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
• Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination (AAAI 2026 · similarity 0.89)