Research map / Large language models / Reasoning and chain of thought

Towards Effective Code-Integrated Reasoning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies code-integrated reasoning (CIR) trained with stabilized tool-augmented RL (REINFORCE++, KL removal, clip-higher, zero entropy, feedback masking, progressive tool budget), reaching 52.4 avg accuracy on five math benchmarks with Qwen2.5-Math-7B.

Problem

Tool-augmented RL for math reasoning with a code interpreter is prone to unstable learning dynamics, distribution shift from tool feedback, and response homogenization under fixed interaction budgets.

Approach

Combines stability measures (exact code-block matching for tool triggers, masking tool feedback in the loss, disabling entropy bonus) with exploration measures (removing KL term, clip-higher, progressively increasing tool-call budget from 2 to 4) in a tool-augmented RL recipe for code-integrated reasoning.

Key results

• CIR (Qwen2.5-M7) reaches 52.4 average accuracy across five math benchmarks, above ToRL-7B (50.5) and the best non-code baseline Oat-Zero-7B (39.3).
• CIR (Qwen2.5-M7) scores 42.3% on AIME2024 versus 37.9% for ToRL-7B and 18.8% for Eurus-2-7B-PRIME.
• CIR averages about 1,726 response tokens versus 13,831 for DeepSeek-R1-Distill-Qwen-7B, at 34.4 vs 44.7 average accuracy on AIME24/AIME25/OlymMATH.

Surprising findings

• Code pass rate declines during training even as accuracy improves, which the authors interpret as more code revisions rather than worse performance.
• Non-executable code can still appear in correct responses (39.4%), with error feedback helping the model revise toward correct solutions.

Limitations

• Geometry problems show minimal benefit from code integration.
• Only code execution is studied; generalization to other tools is left to future work.
• Long-CoT reasoning models were not used as the RL base, due to training cost.

Experimental setup

Datasets: STILL-3 curated subset (38000 samples)

Benchmarks: MATH500 AMC23 AIME2024 AIME2025 OlymMATH

Metrics: accuracy avg@16 pass@k average response length

Baselines: Qwen2.5-Math-7B-Instruct Qwen2.5-Math-7B-Instruct-TIR SimpleRL-Zero-7B Eurus-2-7B-PRIME Oat-Zero-7B ToRL-7B Qwen3-8B (Thinking-10K) DeepSeek-R1-Distill-Qwen-7B

Builds on: DAPO REINFORCE++ ToRL STILL-3 veRL

Closest papers by meaning

• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (ICLR 2026 · similarity 0.91)
• SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning (ICLR 2026 · similarity 0.90)
• R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning (ICLR 2026 · similarity 0.90)
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning (ICLR 2026 · similarity 0.89)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.89)