Research map / Large language models / Reasoning and chain of thought

R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains Qwen-2.5 models (3/7/14B) with multi-turn code-interpreter SFT then multi-stage GRPO curriculum ordered by a measured improvement-potential score over 144 reasoning/planning tasks, reaching 72.4% average test accuracy for R1-CI-14B.

Problem

Training general-purpose LLM code interpreters across many heterogeneous reasoning and planning tasks with RL yields only marginal gains because of task heterogeneity and sparse effective training samples.

Approach

Estimates each training sample's improvement potential as 4p(1-p) from correctness rates across multiple agent strategies, then runs GRPO in four stages that start with high-potential samples and progressively add lower-potential ones.

Key results

• R1-CI-14B raises average test accuracy on 37 held-out tasks from 44.1% to 72.4%, above GPT-4o text-only (58.6%) and GPT-4o with Code Interpreter (70.9%).
• Multi-stage improvement-potential curriculum raises RL gains from +3.4% to +9.3% average across 3/7/14B Qwen-2.5 models (per abstract).
• Code execution sandbox decoupling reduces RL training time by about 39%, from around 4500 to 1845 GPU hours.

Surprising findings

• Vanilla GRPO on 107 mixed tasks yields only marginal gains, with the max average improvement dropping from 27.4% for a single task to 3.3% with 107 tasks.
• Response length does not increase during GRPO training, contrary to prior findings on RL-driven length growth.

Limitations

• Some tasks still score near zero, so the base model's inherent capability limits performance.
• Training is expensive (about 1845 GPU hours for 14B) and multi-turn code execution adds inference latency and token cost.
• Improvement potential requires sampling 20 answers per question from four agent variants, adding preprocessing cost.

Experimental setup

Datasets: SymBench Big-Bench-Hard Reasoning-Gym GSM MATH

Benchmarks: SymBench Big-Bench-Hard Reasoning-Gym GPQA Diamond AIME 2024 AIME 2025

Metrics: accuracy

Baselines: GPT-4o All Text GPT-4o Code Interpreter All Text All Code Code Agent CodeSteer Qwen3-14B All Text

Builds on: GRPO ToRL ReTool CodeSteer ToolRL

Closest papers by meaning

• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
• Towards Effective Code-Integrated Reasoning (AAAI 2026 · similarity 0.90)
• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.90)
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (NeurIPS 2025 · similarity 0.90)
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Compiler-R1: Towards Agentic Compiler Auto-tuning with Reinforcement Learning (NeurIPS 2025 · similarity 0.89)