Research map / Large language models / Reasoning and chain of thought

Coupled Variational Reinforcement Learning for Language Model General Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoVRL, a verifier-free RL method that couples question-only (prior) and answer-guided (posterior) reasoning-trace sampling through a composite distribution and hybrid sampling, improving general and math reasoning on Qwen2.5/Qwen3 base models.

Problem

Verifier-free RL methods sample reasoning traces conditioned only on the question, causing low sample efficiency on hard questions and incoherence between reasoning traces and final answers, while requiring no external verifier.

Approach

Frame reasoning training as variational inference with a composite distribution that mixes prior and posterior token-level probabilities, optimized via off-policy hybrid sampling (prior with probability alpha, posterior otherwise) with importance-weighted GRPO, an NLL term on positive-advantage traces, and a KL regularizer to the prior.

Key results

• CoVRL reaches 50.2 overall average on Qwen2.5-7B vs 37.8 base (+12.4) and 47.9 for the strongest baseline RLPR (about +2.3).
• On Qwen2.5-14B, overall rises from 40.5 to 54.5 (+14.0); on Qwen3-8B, from 47.4 to 56.0 (+8.6).
• On reasoning model Qwen3-8B, overall improves from 55.9 to 59.9, while Qwen3-14B gains only 0.9 (61.6 to 62.5).

Surprising findings

• Training only on non-mathematical WebInstruct questions yields large gains on math benchmarks (e.g., AQuA +21.7 points).
• Reducing the KL coefficient to 0.1 collapses overall accuracy to 27.4, far more harmful than reducing the NLL coefficient.

Limitations

• Gains on reasoning models (Qwen3-14B +0.9) are modest and mixed across benchmarks, with some degradations.
• Hybrid sampling is an off-policy approximation to the composite distribution, introducing distribution mismatch handled only by importance weighting.
• Sensitivity to KL coefficient: insufficient KL regularization causes training instability.

Experimental setup

Datasets: WebInstruct (non-math subset)

Benchmarks: GPQA MMLU-Pro TheoremQA AIME'24 AQuA CARP-EN MATH-500 Minerva SAT-Math

Metrics: Average@N accuracy

Baselines: VeriFree JLB LaTRO RAVR RLPR GRPO

Builds on: GRPO VeriFree LaTRO RAVR RLPR

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.90)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)