Research map / Reinforcement learning / Offline RL

How does RL Post-training Induce Skill Composition? A Case Study on Countdown

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Simple practical offline RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies RL (GRPO) post-training on the Countdown task using expression-tree patterns to separate length generalization from compositional generalization, finding structure-dependent learnability and partial compositional transfer.

Problem

Whether RL post-training induces genuine skill composition rather than mere length generalization, and how the compositional structure of a solution affects learnability and transfer.

Approach

Represents Countdown solutions as canonical expression-tree patterns (shape signatures) and builds a balanced dataset sampled uniformly over patterns, enabling separate measurement of length generalization (larger n) and compositional generalization (held-out patterns and shapes).

Key results

• Qwen2.5-1.5B trained on n in {3,4} reaches pass@32 of about 0.68 on held-out n=5 and about 0.36 on n=6 (final checkpoints).
• Balanced shapes [2]∘[2] are mastered more reliably than unbalanced ones; right-heavy [1]÷[3] has per-pattern precision 0.61 for 1.5B versus 0.75-0.80 for other shapes.
• A held-out family of patterns (A/B+C and its n=4 extensions) is recovered by 1.5B model after RL training, with n=3 coverage emerging before n=4.

Surprising findings

• SFT learns n=4 patterns in the opposite structural order (right-heavy easiest) compared with RL, suggesting the hierarchy is specific to RL training.
• Smaller models often exploit a formatting-only reward shortcut, collapsing response length before learning the task.

Limitations

• Scope limited to the Countdown task; no evaluation on other reasoning or compositional tasks.
• Generalization tested only to larger n or a selected held-out set of patterns.
• No data curriculum considered; training uses a random balanced mix.

Experimental setup

Datasets: Countdown balanced pattern dataset (own construction) Countdown dataset from prior work (for comparison)

Benchmarks: Countdown held-out n=3,4 patterns Countdown n=5 and n=6 held-out

Metrics: pass@32 all-correct@32 average accuracy pattern coverage per-pattern precision high-precision coverage

Baselines: SFT without CoT SFT with random CoT PPO base model

Builds on: GRPO verl Countdown task

Closest papers by meaning

• From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones (ICLR 2026 · similarity 0.93)
• Compositional Reasoning in Language Models under Reinforcement Learning Post-Training (NeurIPS 2026 · similarity 0.92)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.90)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.88)
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (ICML 2025 · similarity 0.88)