Research map / Reinforcement learning / RL from feedback

MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MURPHY, a multi-turn GRPO extension that builds feedback-conditioned rollout trees and propagates rewards backward (MaRS/MeRS) plus pruning, for self-correcting code generation.

Problem

Standard GRPO optimizes terminal rewards on isolated single-turn responses, so it cannot credit failed code attempts that surface informative execution feedback enabling later successful refinements in multi-turn code generation.

Approach

Build feedback-conditioned rollout trees where failed programs are expanded with executor feedback, propagate execution rewards backward through the tree (max-backup MaRS or mean-backup MeRS), then apply local GRPO advantages to credit-adjusted rewards; add IntraP/InterP pruning to cut optimization cost.

Key results

• Up to +6.20 pp pass@1 over strongest comparator at Iter-3 on OLMo-2-1124-7B-Instruct (HumanEval), with +6.07 pp on MBPP.
• On LiveCodeBench-v6 with Qwen3-4B, gains of +4.38 (Medium) and +4.20 (Hard) at Iter-5.
• InterP pruning cuts optimization time by 74.1% and per-step training time by 21.4% while staying within 0.4-0.8 pp of unpruned Iter-3 performance.

Surprising findings

• IntraP improves single-turn Iter-1 performance but regresses by 2-2.3 pp at Iter-3 on Qwen3-1.7B.
• Murphy shows a small regression on Qwen3-1.7B MBPP at Iter-1 (-1.27 pp).

Limitations

• Multi-turn rollout-tree training is more expensive than GRPO even with pruning.
• Evaluation limited to code generation with clean execution feedback; noisier feedback and longer horizons untested.
• Training and ablations limited to two turns; reuse of KV cache across shared prefixes not exploited.

Experimental setup

Datasets: KodCode

Benchmarks: HumanEval MBPP LiveCodeBench-v6

Metrics: pass@1

Baselines: Base GRPO-MT muCode ReVeal RLEF

Builds on: GRPO Reflexion RLEF muCode ReVeal

Closest papers by meaning

• Multi-Turn Code Generation Through Single-Step Rewards (ICML 2025 · similarity 0.90)
• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.88)
• Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization (NeurIPS 2025 · similarity 0.88)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.88)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.87)
• ReVeal: Self-Evolving Code Agents via Reliable Self-Verification (ICLR 2026 · similarity 0.87)