Research map / Reinforcement learning / RL from feedback

$\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis

ICLR 2024 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes B-Coder, a value-based deep RL framework for natural-language-to-code generation that initializes Q from a pre-trained LM's logits and uses a conservative Bellman operator, plus a free learned reward model for ranking candidate programs.

Problem

Policy-based RL dominates program synthesis despite program synthesis having cheap verifiable rewards and abundant off-policy data (human and historical programs), which value-based methods could exploit; but value-based RL is hard to train in huge token state-action spaces.

Approach

Initialize a dueling Q-function from the logits of a pre-trained code LM (Q = alpha*log-prob advantage + V), pre-train the value head with TD error, and fine-tune with a conservative Bellman operator that bootstraps through the reference LM's greedy action; the recovered Q yields a reward model used to rank generated programs.

Key results

• On APPS test without example tests, B-Coder reaches 2.30 pass@1 and 3.80 pass@5 overall, the best among compared methods at those k, versus 2.20 for CodeRL and 1.45 for RLTF.
• At pass@1000 on APPS, B-Coder gets 18.12 overall, second only to RLTF's 19.92.
• Using only basic unit-test reward, B-Coder reaches 2.30 pass@1 vs CodeRL's 1.62 on APPS.

Measured effects reported in the paper

Surprising findings

• Vanilla dueling DDQN performance continuously decreases even on the training set, and the Q-initialization and conservative operator are needed to stabilize training.
• Ranking with the free reward model is far from the upper bound: filtering by true hidden-test reward gives 11.06 pass@1-level on 'All' vs 2.30 for ranking.

Limitations

• Raw performance without ranking is mixed compared to CodeRL (CodeRL is better at pass@5 and pass@50 in the limitations table).
• Additional frozen parameters are introduced by the shared encoder and multiple heads.
• Ranking with the recovered reward leaves a large gap to filtering by true reward.

Experimental setup

Datasets: APPS MBPP

Benchmarks: APPS MBPP

Metrics: pass@1 pass@5 pass@1000

Baselines: CodeRL PPOCoder RLTF Codex AlphaCode GPT-J GPT-Neo

Builds on: CodeRL Dueling DQN CodeT T5

Closest papers by meaning

• CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models (NeurIPS 2026 · similarity 0.91)
• Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search (ICLR 2025 · similarity 0.89)
• Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling (NeurIPS 2025 · similarity 0.89)
• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.89)
• R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning (ICLR 2026 · similarity 0.88)
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.88)