Research map / Large language models / Reasoning and chain of thought

Compositional Reasoning in Language Models under Reinforcement Learning Post-Training

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes compositional reasoning as dependency graphs over reusable skills and shows, with RL on data-structure tasks, a decomposed-to-composed asymmetry: atomic-skill RL training does not transfer to compound tasks while compound training transfers back to atomic ones.

Problem

Understanding how RL post-training affects compositional reasoning in language models, i.e., whether models trained on decomposed skills can compose them into multi-step tasks at test time.

Approach

A dependency-graph framework with three compositionality levels (single-skill chain, multi-skill chain, branch-merge graph), used for controlled RL comparisons between atomic-skill and composed-task training, plus a coverage-factor theory explaining the asymmetry.

Key results

• RL-Atomic gives only marginal gains on compound tasks while RL-Compound improves both atomic and compound performance across Level I-III with Qwen3-4B-Instruct.
• On Level III branch-merge tasks, RL-Atomic collapses to near-zero compound accuracy while RL-Compound performs substantially better.
• SFT performs well in-domain but degrades sharply on larger out-of-domain instances, whereas RL shows better length generalization.

Surprising findings

• Atomic-skill RL training can fail to transfer to composed tasks even when atomic accuracy is near ceiling (Level III).
• Requiring intermediate outputs or structured JSON outputs reduced final accuracy in preliminary studies.

Limitations

• Evaluation focuses on controlled synthetic data-structure tasks; real-world validation limited to a BFCL pilot.
• Compound tasks evaluated only by final-output accuracy.
• Theory relies on assumptions such as context invariance and coverage that may be optimistic.

Experimental setup

Datasets: DSR-Bench BFCL simple_python BFCL parallel BFCL parallel_multiple

Benchmarks: DSR-Bench data-structure tasks BFCL

Metrics: accuracy

Baselines: SFT Base model (no post-training)

Builds on: GRPO DSR-Bench Composition-RL SKILL-MIX

Closest papers by meaning

• How does RL Post-training Induce Skill Composition? A Case Study on Countdown (NeurIPS 2026 · similarity 0.92)
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.91)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.91)
• From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones (ICLR 2026 · similarity 0.91)
• Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning (ICLR 2026 · similarity 0.90)
• Compositional Causal Reasoning Evaluation in Language Models (ICML 2025 · similarity 0.90)