Research map / Large language models / Reasoning and chain of thought

Generalization of RLVR Using Causal Reasoning as a Testbed

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirical study of RLVR vs SFT generalization on probabilistic inference over random causal graphical models (RLCausal), fine-tuning Qwen2.5-Instruct 3B-32B across association, intervention, and counterfactual query levels.

Problem

Under what conditions RLVR yields robust generalization beyond training data is poorly understood; the paper studies this on causal inference (probabilistic queries over causal graphical models) across query levels and structural complexity.

Approach

Uses the causal ladder and a controlled synthetic generator of fully specified 10-node binary SCMs with exact reference answers to test within- and across-level generalization of RLVR (GRPO/DAPO) versus SFT, and links RLVR effectiveness to the base model's initial reasoning competence.

Key results

• RLVR outperforms SFT on within-level association and intervention queries for models 7B and larger, but underperforms at 3B and on counterfactual queries for all sizes.
• RLVR outperforms SFT across levels for models 7B and larger.
• RLVR shifts 7B/32B models toward incremental marginalization and reduces probability derivation errors, while 3B models learn to skip marginalization.

Surprising findings

• 3B models regress to directly predicting answers after RLVR, as their step-by-step marginalization attempts are rarely correct.
• Providing twin-network hints for counterfactual queries did not meaningfully change accuracy, and models did not attempt twin-network construction.

Limitations

• Only binary variables and single-variable observations/interventions
• RLVR ineffective when base model reasoning competence is low (3B, counterfactual)
• Qwen2.5-Instruct only; results may not transfer to other model families

Experimental setup

Datasets: RLCausal (synthetic causal graph queries)

Benchmarks: RLCausal test set

Metrics: accuracy within total variation distance 0.01 accuracy vs threshold t

Baselines: SFT

Builds on: GRPO DAPO CLadder

Closest papers by meaning

• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.92)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.92)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.92)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.92)