Research map / Large language models / Reasoning and chain of thought
DELTA-Code: How RL Unlocks and Transfers New Programming Algorithms in LLMs
Yiyou Sun, Yuhan Cao, Pohao Huang, Haoyue Bai, Hannaneh Hajishirzi, Nouha Dziri, Dawn Song
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DELTA-Code, a controlled benchmark of synthetic coding problem families to test whether RL can teach LLMs genuinely new programming algorithms and transfer them.
Problem
Whether LLMs can acquire or generalize genuinely new reasoning strategies in coding beyond skills already encoded during pre-training or post-training.
Approach
A controlled benchmark of synthetic coding problem families designed to probe learnability and transferability of algorithmic reasoning, measured distributionally, to separate new strategy acquisition from sharpening of existing skills.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with binary full-pass reward directly on a pass@K=0 family (ablation) | GRPO · Manufactoria-HAS | full pass rate | Hurt |
| use per-test pass-rate reward instead of full-pass (ablation) | GRPO · Manufactoria-HAS | full pass rate: full-pass remains <0.01% | No clear effect |
| add per-test pass-rate warm-up then switch to full-pass reward (ablation) | GRPO · Manufactoria-HAS | full pass rate: 0% -> 100% | Improved |
| add experience replay of successful traces (ablation) | GRPO · Manufactoria-HAS | grokking time / convergence | Mixed |
| add feedback-in-the-loop (verifier feedback replaces EOS) (ablation) | GRPO · Manufactoria-HAS | grokking time / stability | Mixed |
| curriculum via REGEX family before HAS (ablation) | GRPO curriculum · Manufactoria-HAS | pass rate | Improved |
| curriculum via COMPR family before HAS (ablation) | GRPO curriculum · Manufactoria-HAS | pass rate | No clear effect |
| per-test warm-up on harder PREPEND family (ablation) | GRPO · Manufactoria-PREPEND (MEDIUM) | full pass rate: stays at 0 | No clear effect |
| RL on Basic BouncingSim mixture, test on unseen skill combinations | GRPO · BouncingSim compositional | full pass rate: near-zero -> 60-70% | Improved |
| RL on Basic BouncingSim, test on Hard tier | GRPO · BouncingSim explorative Hard | full pass rate: single digits | No clear effect |
| RL on BouncingSim, test on transformative periodic trajectories | GRPO · BouncingSim transformative | full pass rate: near zero | No clear effect |
Experimental setup
Benchmarks: DELTA-Code
Closest papers by meaning
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.89)
- R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning (ICLR 2026 · similarity 0.88)
- AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.88)
- RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems (ICLR 2026 · similarity 0.88)
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
