Research map / Large language models / Reasoning and chain of thought

CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CodeScaler, a code-specific reward model trained on on-policy RLVR rollouts from DeepCoder problems, used both as an execution-free RL reward (with syntax-aware extraction and validity-preserving shaping) and as a fast Best-of-N ranker.

Problem

Execution-based RLVR for code LLMs is limited by scarce, reliable unit tests, while existing reward models underperform RLVR in RL training and lag unit-test methods in test-time Best-of-N selection.

Approach

Train a Bradley-Terry reward model (initialized from Skywork-Reward-V2-Qwen3-8B) on preference pairs built from on-policy GRPO rollouts on verified DeepCoder problems, augmented with misaligned problem-code pairs; combine with strict code extraction (single code block plus AST check) and a softplus reward transform that gives invalid code a reward of 0.

Key results

• On DeepCoder, CodeScaler improves average Avg@8 over RLVR by +1.55 (Qwen3-8B-Base, 38.56 vs 37.01) and +4.23 (Qwen3-14B-Base, 43.35 vs 39.12), while SkyworkRM and AceCodeRM degrade relative to RLVR.
• Scaling to 44K problems with 20K synthetic problems yields 39.60 average for Qwen3-8B-Base, +14.64 over the base model without test cases.
• On RM-Bench, CodeScaler-8B reaches 84.1 average vs 81.4 for SkyworkRM-8B; test-time BoN with CodeScaler is about 10x faster than CURE (0.31 vs 3.20 s/question) at comparable accuracy.

Surprising findings

• Training a reward model only on code improves general chat (80.6 to 83.0) and math (75.0 to 79.9) judgment on RM-Bench.
• Binary (all-tests-pass) preference labels outperform pass-ratio threshold labels for both RL training and Best-of-N selection.

Limitations

• Reward model shows length and keyword biases: false positives are 28% longer than true positives, and correct concise solutions are undervalued.
• RL with the learned reward still requires a 24K verified problem seed set for reward data; the synthetic-only scaling uses DeepCoder rollouts for training the RM.
• SkyworkRM baseline collapsed in the 44K setting, so comparisons depend on checkpoint selection.

Experimental setup

Datasets: DeepCoder KodCode rStarCoder TACO RM-Bench HumanEval-X

Benchmarks: LiveCodeBench CodeContests MBPP CodeForces RM-Bench HumanEval-X

Metrics: Avg@8 BoN@8 pass@1 AUROC latency (s/question)

Baselines: RLVR SkyworkRM AceCodeRM CURE KodCode rStarCoder

Builds on: GRPO Skywork-Reward-V2 DeepCoder CURE VeRL QueST

Closest papers by meaning

• CURE: Co-Evolving Coders and Unit Testers via Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
• $\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis (ICLR 2024 · similarity 0.91)
• Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.90)
• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.90)
• Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.90)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.90)