Research map / Large language models / Agents, code and math
Rethinking Verification for LLM Code Generation: From Generation to Testing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-dimensional metrics and SAGA, a human-LLM collaborative test-case generation framework that uses correct and incorrect human submissions to build stronger verifiers, plus TCGBench, TCGBench-Lite, CodeCompass, and a distilled TCGCoder-7B.
Problem
Existing code benchmarks and verifiers rely on few homogeneous, LLM-biased test cases, so subtle faults go undetected, inflating measured performance and giving noisy rewards for RLVR.
Approach
SAGA guides an LLM with human insights from correct solutions (multidimensional analysis) and from failed submissions paired with their corrections (differential analysis) to write Python case scripts that generate targeted test inputs, which a ground-truth interpreter then labels with outputs.
Key results
- SAGA (DeepSeek-V3 backbone) reaches DR@50 of 90.62% and VAcc@50 of 32.58% on TCGBench-Lite, above the Input-Interpreter baseline's 81.07% and 16.72%.
- SAGA-distilled TCGCoder-7B achieves AUC@50 of 0.1890, above all baselines including the DeepSeek-V3-backed Input-Interpreter (0.1234).
- On the shared 101-problem AtCoder subset, average Pass@1 of various models drops by a relative 9.56% on CodeCompass versus LiveCodeBench-v6, re-ranking some models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAGA instead of Input-Interpreter random sampling | TCG on TCGBench-Lite · TCGBench-Lite | AUC@50: 0.1234 vs. 0.2228 | Helped |
| add Multidimensional Analysis to SAGA (ablation) | SAGA on TCGBench-Lite · TCGBench-Lite | AUC@50: 0.1923 vs. 0.2228 | Helped |
| add Differential Analysis to SAGA (ablation) | SAGA on TCGBench-Lite · TCGBench-Lite | AUC@50: 0.1926 vs. 0.2228 | Helped |
| replace detailed insight-driven prompts with SimpleCOT (ablation) | SAGA on TCGBench-Lite · TCGBench-Lite | AUC@50: 0.1424 vs. 0.2228 | Helped |
| replace SAGA-generated inputs with random inputs using ground truth (ablation) | SAGA on TCGBench-Lite · TCGBench-Lite | AUC@50: 0.1616 vs. 0.2228 | Helped |
| replace SAGA test generation with EvalPlus using ground truth (ablation) | SAGA on TCGBench-Lite · TCGBench-Lite | AUC@50: 0.1278 vs. 0.2228 | Helped |
| use Qwen2.5-Coder-7B-Instruct backbone instead of DeepSeek-V3-0324 (ablation) | SAGA · TCGBench-Lite | AUC@50: 0.2228 vs. 0.1810 | Hurt |
| use Qwen2.5-72B-Instruct backbone instead of DeepSeek-V3-0324 (ablation) | SAGA · TCGBench-Lite | AUC@50: 0.2228 vs. 0.1943 | Hurt |
| use Qwen2.5-Coder-32B-Instruct backbone instead of DeepSeek-V3-0324 (ablation) | SAGA · TCGBench-Lite | AUC@50: 0.2228 vs. 0.2139 | Hurt |
| add simple human priors to Direct Generation | Direct Generation on AtCoder · AtCoder | Accuracy@50: 11.30% -> 15.06% | Helped |
| add simple human priors to Input-Interpreter | Input-Interpreter on AtCoder · AtCoder | Accuracy@50: 23.36% -> 27.95% | Helped |
Surprising findings
- Direct LLM-generated tests show high self-pass rates, with LLM solutions passing their own generated tests more easily, suggesting blind spots.
- Increasing the number of random LLM-generated test inputs does not fundamentally improve detection rate, which saturates due to inter-test correlation.
Limitations
- Theoretical upper bound relies on an approximating model with averaged parameters and is only approximate.
- Difficulty labels for TCGBench-Lite are assigned from platform tags and contest heuristics rather than a validated measure.
- Main experiments use a single backbone (DeepSeek-V3-0324) and a subset of 101 AtCoder problems for the LiveCodeBench comparison.
Experimental setup
Datasets: TCGBench LiveCodeBench-v6 (AtCoder subset, 101 problems) CodeForces-CoTs
Benchmarks: TCGBench-Lite TCGBench CodeCompass (AtCoder subset vs LiveCodeBench-v6)
Metrics: Detection Rate (DR) Verifier Accuracy (VAcc) AUC@50 Distinct Error Pattern Coverage (DEPC) Diversity Ratio Pass@1
Baselines: TestChain Input-Interpreter (LiveCodeBench-style random sampling) EvalPlus TCGCoder-7B LiveCodeBench-v6
Builds on: LiveCodeBench EvalPlus TestChain DeepSeek-R1
Closest papers by meaning
- CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models (NeurIPS 2026 · similarity 0.90)
- Learning to Generate Unit Test via Adversarial Reinforcement Learning (ICLR 2026 · similarity 0.90)
- QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation (NeurIPS 2025 · similarity 0.89)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.89)
- RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.89)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.