Research map / Large language models / Agents, code and math
rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 418K-problem competitive programming dataset with 580K verified long-reasoning solutions, using GPT-4o-generated input generators and a mutual-verification labeling scheme, and fine-tunes Qwen2.5-Coder models (1.5B-14B) that rival much larger reasoning models.
Problem
Code reasoning LLMs lack large-scale, high-difficulty competitive programming problems with reliable, verifiable input-output test cases to validate solutions.
Approach
Synthesize new solvable problems from expert seeds plus their oracle solutions; generate diverse, constraint-valid test inputs via LLM-written generator/validator utilities swept over scale parameters; label outputs by majority agreement among 16 QWQ-32B solutions (mutual verification) rather than by LLM direct output prediction.
Key results
- On LiveCodeBench, rStar-Coder-14B reaches 62.5% (from 23.3% base) and surpasses o3-mini (low) at 59.4%.
- rStar-Coder-7B reaches 16.15% average pass@1 on USACO 2025, above QWQ-32B at 15.62%.
- Mutual verification labels test outputs with 96.8% accuracy versus 12.7% for direct GPT-4o output generation on 3,150 inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-7B on rStar-Coder instead of using the base model | Qwen2.5-7B · LiveCodeBench | pass@1 accuracy: 17.4% to 57.3% | Helped |
| Fine-tune Qwen2.5-14B on rStar-Coder instead of using the base model | Qwen2.5-14B · LiveCodeBench | pass@1 accuracy: 23.3% to 62.5% | Helped |
| Replace direct GPT-4o input-output generation with mutual verification (ablation) | Test output labeling · 64 seed problems; 3,150 test inputs | labeling accuracy: 12.7% -> 96.8% | Helped |
| Use three-step input generation instead of direct GPT-4o prompting (ablation) | Qwen2.5-Coder-7B-Instruct trained on 150K synthetic problems · LiveCodeBench | Average: 42.9 -> 44.6 | Helped |
| Train on the full dataset rather than only synthetic problems (ablation) | Qwen2.5-Coder-7B · LiveCodeBench | accuracy: 46.8 -> 57.3 | Helped |
| Train on the full dataset rather than only seed problems (ablation) | Qwen2.5-Coder-7B · LiveCodeBench | accuracy: 49.7 -> 57.3 | Helped |
| Increase solutions per seed problem from 1 to 8 (ablation) | 7B model trained on seed problems · LiveCodeBench | accuracy: 40.8 -> 51.1 | Helped |
| Increase solutions per seed problem from 8 to 16 (ablation) | 7B model trained on seed problems · LiveCodeBench | accuracy: 51.1 -> 54.7 | Helped |
| Expand problem diversity using the rStar-Coder dataset rather than scaling solutions on seed problems alone (ablation) | 7B model · LiveCodeBench | accuracy: 54.7 -> 57.3 | Helped |
| Expand problem diversity using the rStar-Coder dataset rather than scaling solutions on seed problems alone (ablation) | 7B model · USACO 2025 | average pass@1 accuracy: 10.41 -> 16.15 | Helped |
Surprising findings
- Scaling unique problems with one solution each (580K) outperforms scaling solutions per seed problem to 603K on LiveCodeBench and USACO.
- Dataset of 580K outperforms OCR (736K) and OpenThinker2 (1M) at 7B, indicating data quality matters more than size.
Limitations
- Pipeline relies heavily on GPT-4o and other frontier LLM resources; many synthetic problems are discarded after verification.
- Cannot handle problems whose constraints are implied by context rather than stated explicitly.
Experimental setup
Datasets: TACO APPS CodeContests CodeContests-Python-Submission CodeForces (OpenR1) USACO 2011-2023 IOI 2002-2023
Benchmarks: LiveCodeBench v5 USACO 2025 HumanEval HumanEval+ MBPP MBPP+
Metrics: pass@1 (average of 16 samples) output labeling accuracy
Baselines: GPT-4o Claude 3.5 Sonnet DeepSeek-R1 QWQ-32B OpenAI o3-mini DeepSeek-R1-Distill-Qwen OpenThinker OlympicCoder OCR-Qwen OpenThinker2 Bespoke-Stratos
Builds on: Qwen2.5-Coder QWQ-32B CYaRon KodCode FlashAttention-2
Closest papers by meaning
- CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance (ICML 2025 · similarity 0.90)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests (NeurIPS 2025 · similarity 0.89)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.88)
- MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning (ICLR 2024 · similarity 0.88)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.