Research map / Reinforcement learning / RL from feedback

Multi-Turn Code Generation Through Single-Step Rewards

ICML 2025 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paper Check your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes μCODE, which trains a generator and a learned verifier for multi-turn code generation from execution feedback by treating the task as one-step recoverable and reducing RL to imitation learning with single-step rewards.

Problem

Multi-turn code generation from execution feedback, where existing RL methods with multi-turn rewards suffer from sparse signals and are complex to train.

Approach

Code generation is a one-step recoverable MDP, so a learned verifier can guide a local search expert that relabels rollouts, and the generator is fine-tuned by imitation on these labels (expert iteration), with multi-turn best-of-N at inference using the verifier.

Key results

With Llama-3.2-1B, μCODE with multi-turn BoN (N=5) reaches 51.1 on MBPP and 41.5 on HumanEval, versus 47.3 and 35.7 for the multi-turn Instruct+BoN baseline.
With Llama-3.1-8B, μCODE with BoN reaches 70.6 on MBPP, 63.8 on HumanEval, and 16.3 on CodeContests, the best among compared methods.
On MBPP (POMDP), μCODE gains 15.9% from turn 1 to turn 6 while base and RFT models fall behind by about 6%.

Measured effects reported in the paper

What was changed	Where	Measured	Effect
Use μCODE training instead of RFT	Llama-3.2-1B-Instruct code generator · MBPP	BoN accuracy: 35.7 -> 37.9	Helped
Use μCODE training instead of RFT	Llama-3.2-1B-Instruct code generator · HumanEval	BoN accuracy: 34.1 -> 35.4	Helped
Use μCODE multi-turn BoN instead of the strongest baseline	Llama-3.2-1B-Instruct code generator · MBPP	BoN accuracy: 47.3 -> 51.1	Helped
Use μCODE multi-turn BoN instead of the strongest baseline	Llama-3.2-1B-Instruct code generator · HumanEval	BoN accuracy: 35.7 -> 41.5	Helped
Use μCODE multi-turn BoN instead of Instruct +BoN	Llama-3.1-8B-Instruct code generator · MBPP	BoN accuracy: 69.7 -> 70.6	Helped
Use μCODE multi-turn BoN instead of Instruct +BoN	Llama-3.1-8B-Instruct code generator · HumanEval	BoN accuracy: 62.9 -> 63.8	Helped
Use μCODE multi-turn BoN instead of RFT +BoN	Llama-3.1-8B-Instruct code generator · CodeContests	BoN accuracy: 14.9 -> 16.3	Helped
Add learned-verifier selection when choosing training rollouts (ablation)	RFT code generator · MBPP	BoN accuracy: 46.7 -> 49.0	Helped
Add learned-verifier selection when choosing training rollouts (ablation)	RFT code generator · HumanEval	BoN accuracy: 36.5 -> 38.9	Helped
Add one-step-recoverability relabeling with oracle verifier (ablation)	RFT code generator · MBPP	BoN accuracy: 46.7 -> 48.2	Helped
Add one-step-recoverability relabeling with oracle verifier (ablation)	RFT code generator · HumanEval	BoN accuracy: 36.5 -> 38.0	Helped
Use both oracle and learned verifier scores for relabeling instead of either verifier alone (ablation)	μCODE code generator · MBPP	BoN accuracy: 48.2 -> 51.1; 48.5 -> 51.1	Helped

Surprising findings

Multi-turn RFT underperforms the single-turn Instruct base on some benchmarks, attributed to training on sub-trajectories with incorrect code at non-terminal steps.
A Bradley-Terry verifier loss consistently outperforms binary cross-entropy for verifier ranking.

Limitations

Only models up to 8B parameters were trained due to budget, so scaling conclusions may not hold for larger models.
Training uses only the small MBPP training set (374 problems) and Python only.

Experimental setup

Datasets: MBPP HumanEval CodeContests

Benchmarks: MBPP HumanEval CodeContests MBPP+ HumanEval+ MBPP (POMDP)

Metrics: BoN accuracy pass@1 (BoN@1)

Baselines: Instruct RFT CodeRL RLEF PPOCoder SCoRe RISE

Builds on: expert iteration DAgger rejection fine-tuning Bradley-Terry reward modeling

Closest papers by meaning

MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation (NeurIPS 2026 · similarity 0.90)
What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.86)
RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.86)
ReVeal: Self-Evolving Code Agents via Reliable Self-Verification (ICLR 2026 · similarity 0.86)
EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.86)
Teaching Language Models to Critique via Reinforcement Learning (ICML 2025 · similarity 0.85)

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft