Research map / Reinforcement learning / RL from feedback

Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes JEPO, an RL algorithm built on a multi-sample Jensen's evidence lower bound that treats chain-of-thought as a latent variable, enabling RL-style training on unverifiable long-form data without external verifiable rewards.

Problem

RL from verifiable rewards requires short-form, easily checkable ground-truth answers, so it cannot scale to long-form or unverifiable data such as mathematical proofs.

Approach

Views chain-of-thought as a latent variable and optimizes a Jensen's lower bound on the marginal likelihood of the ground-truth answer, using a multi-sample log-mean-exp form with leave-one-out control variates; the resulting update mixes a REINFORCE term (log-likelihood of the answer as reward) with an SFT-like term, requiring no verifier.

Key results

• On MATH, multi-sample JEPO matches online RL on test accuracy and reaches about 70% test accuracy with the 70B model, modestly outperforming RL.
• On semi-verifiable Numina, JEPO and the combined algorithm improve Sympy+LLM-judge accuracy over the RL baseline, especially on the unverifiable subset, while online RL does not improve test-set NLL.
• On Numina-proof, JEPO achieves lower test NLL than SFT at both 8B and 70B scales and asymptotically better NLL.

Measured effects reported in the paper

Surprising findings

• Online RL does not improve proxy NLL on the test set even though its test accuracy improves.
• Length normalization of the log-likelihood reward substantially hurts JEPO on proof data.

Limitations

• Lower bound is loose unless chain-of-thought likelihoods of the answer are equal, and the gap is not characterized in practice.
• Generic match functions are hard to incorporate because the set probability requires enumerating equivalent answers.
• Evaluation on unverifiable data relies on NLL only since long-form answers are hard to assess via sampling.

Experimental setup

Datasets: MATH Numina 1.5 Numina-proof

Benchmarks: MATH Numina (post-processed) Numina-proof

Metrics: accuracy (Sympy) accuracy (Sympy + LLM-as-judge combined) test set negative log likelihood (proxy NLL)

Baselines: online RL with Sympy reward (RLOO-style) SFT VR-CLI single-sample Jensen's lower bound SFT on golden chain-of-thought

Builds on: RLOO GRPO VR-CLI evidence lower bound (ELBO)

Closest papers by meaning

• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
• The Unlearnability Phenomenon in RLVR for Language Models (ICML 2026 · similarity 0.89)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.89)