Research map / Large language models / Reasoning and chain of thought

From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows with a synthetic string-transformation testbed that RL on compositional (Level-2) problems lets Llama-3.1-8B-Instruct learn new compositional skills that generalize to deeper nesting and transfer to Countdown, while RFT and atomic-only RL do not.

Problem

Whether RL teaches LLMs genuinely new skills or only reweights existing ones, given confounds from pretraining contamination and pass@k metrics on mixed-difficulty benchmarks.

Approach

Define skills as string transformation functions with hidden definitions, train atomic skills via rejection fine-tuning, then apply RL on unseen two-function compositions; show compositional incentive in RL (not RFT or atomic-only RL) yields generalization to deeper compositions and a different task (Countdown), and that pass@k gaps widen on hard compositional problems.

Key results

• RL on Level-2 compositions raises held-out Level-3 accuracy from about 5% to around 30% and Level-4 from 1% to about 15%, whereas iterative RFT on the same data never exceeds 2.6% on Level 3.
• On Countdown (no RL on Countdown), Multi-Base + RL L1+2 reaches about 35% at Level 3, outperforming the Multi-Base baseline by more than 18 points.
• On hard compositional levels the RL Level 1+2 gap over the RFT base grows from 4% at pass@1 to about 25% at pass@1024 at Level 5, while easy-level gaps shrink.

Surprising findings

• Iterative RFT on the same Level-2 compositional data fails to generalize even to held-out Level-2 problems (about 15% accuracy), while RL does.
• Atomic-only RL (Level 1) yields near-zero compositional generalization, suggesting RL on atomic tasks alone does not promote composition.

Limitations

• Evaluation relies on synthetic string and Countdown tasks, which may not reflect natural reasoning domains such as math or code.
• Requires a two-stage pipeline with supervised atomic skill acquisition before RL; sufficiency of atomic skills is shown but necessity is not.
• Failure-mode analysis depends on Gemini-2.5-Pro classification of responses.

Experimental setup

Datasets: synthetic string transformation problems (25 functions) Countdown

Benchmarks: synthetic string transformation prediction (Levels 1-8) Countdown (Levels 3-5)

Metrics: accuracy Avg@32 pass@k (k up to 1024)

Baselines: RFT (iterative rejection fine-tuning) RL Level 1 RL Level 2 Multi-Base String-Base + RL L1+2 Multi-Base + RL L1

Builds on: GRPO DAPO rejection fine-tuning

Closest papers by meaning

• How does RL Post-training Induce Skill Composition? A Case Study on Countdown (NeurIPS 2026 · similarity 0.93)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.92)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)
• Compositional Reasoning in Language Models under Reinforcement Learning Post-Training (NeurIPS 2026 · similarity 0.91)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.90)