Research map / Large language models / Reasoning and chain of thought

From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLVRR, which replaces single-dot RLVR rewards for open-ended generation with rule-based content (LCS over reference keywords) and style (LLM-written Python checks) rewards derived from reference responses, trained with GRPO.

Problem

RLVR's single verifiable answer signal does not apply to open-ended generation, where no unambiguous ground truth exists; reward models are prone to hacking and costly to train with RL.

Approach

Extract an ordered reward chain from high-quality reference responses: a content reward from LCS alignment of LLM-extracted keywords per key point (max over multiple references), and a style reward from weighted LLM-generated Python checks of length/format, aggregated by simple averaging and optimized with GRPO.

Key results

• On Qwen2.5-3B-Instruct, GRPO (RLVRR) with 10K prompts reaches open-ended avg 36.4 vs 33.7 for GRPO (RM) and 29.3 for the Instruct base.
• On Qwen2.5-7B-Base, RLVRR reaches open-ended avg 39.8, above SFT with 100K data (38.5).
• RLVRR adds only 0.71% per-step runtime over random reward, versus 8.28% for the reward model.

Surprising findings

• Direct keyword-percentage matching as content reward causes catastrophic failure (open-ended avg 17.8) due to reward hacking via verbosity.
• Self-data distillation from RLVRR outperforms SFT on the same 10K prompts (29.2 vs 26.7).

Limitations

• Style reward covers only verifiable stylistic features (length, format); tone and other semantic aspects are left to the content reward.
• Relies on GPT-4o-mini or Llama3-70B to generate references and verifiers, and on references from a fixed instruction dataset.
• Gains are reported mainly on a 10K-prompt subset and small models (3B to 8B).

Experimental setup

Datasets: 100K open-ended instruction-response pairs (dataset from prior work, responses regenerated by GPT-4o-mini) MATH (difficulty levels 2-5)

Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench IFEval FollowBench MMLU ARC MATH HumanEval GSM8K MATH 500 Minerva Math

Metrics: length-controlled win rate win rate average score prompt-level strict accuracy hard satisfaction rate accuracy best@5 Self-BLEU

Baselines: SFT Random reward BLEU reward Skywork-Reward-V2 RM GRM (GPT-4o-mini) RLPR DPO

Builds on: RLVR GRPO RLPR SimpleRL-Zoo Rubrics as Rewards

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.93)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.92)
• Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning (ICML 2026 · similarity 0.92)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.92)
• VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models (ICLR 2026 · similarity 0.92)
• Spurious Rewards: Rethinking Training Signals in RLVR (ICML 2026 · similarity 0.91)