Research map / Large language models / Agents, code and math
CodeT: Code Generation with Generated Tests
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, Weizhu Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CODET dual execution agreement (generated tests + solution consensus) instead of baseline sampling selection | code-davinci-002 · HumanEval | pass@1: +18.8 | Improved |
| Use CODET dual execution agreement instead of baseline selection | code-davinci-002 · MBPP | pass@1: +9.6 | Improved |
| Use CODET dual execution agreement instead of baseline selection | code-davinci-002 · APPS INTRODUCTORY | pass@1: +7.4 | Improved |
| Use CODET dual execution agreement instead of baseline selection | code-davinci-002 · APPS COMPETITION and CodeContests | pass@1: +0.4 (COMPETITION), +1.4 (CodeContests) | No clear effect |
| Use CODET dual execution agreement instead of baseline selection | INCODER-6B and CODEGEN-MONO-16B · HumanEval and MBPP | pass@1: +4.2 to +13.1 | Improved |
| Score consensus sets by number of code solutions only (f' ablation) instead of joint solutions x tests (ablation) | code-davinci-002 · HumanEval | pass@1: -9.9 vs CODET | Hurt |
| Score consensus sets by number of test cases only (f'' ablation) instead of joint solutions x tests (ablation) | code-davinci-002 · HumanEval | pass@1: -7.4 vs CODET | Hurt |
| Reduce generated test cases from 500 to about 10 per problem (ablation) | code-davinci-002 CODET · HumanEval | pass@1: 65.8 to 56.5 | Hurt |
| De-duplicate code solutions and test cases before CODET (ablation) | code-cushman-001 CODET · HumanEval and MBPP | pass@1: HumanEval 44.5 to 42.7; MBPP 55.4 to 54.7 | No clear effect |
| Use test cases generated by code-davinci-002 instead of model's own tests (ablation) | code-cushman-001, code-davinci-001, INCODER, CODEGEN · HumanEval and MBPP | pass@1: absolute gains of 1.8% to 4.3% for Codex cushman/davinci-001; 6.2% to 15.9% for INCODER and CODEGEN | Improved |
| Remove trivial code solutions before CODET (ablation) | code-davinci-002 CODET · APPS INTRODUCTORY and CodeContests | pass@1: +0.3 (INTRODUCTORY), +0.6 (CodeContests) | No clear effect |
| Remove example input-output cases from HumanEval context (ablation) | baseline and CODET · HumanEval original vs modified context | pass@1 | Mixed |
Later papers that compared with this method
14 later accepted papers report CODET as a baseline in their results tables; 13 of them report a better number on most of the results they share with it.
- Dynamic-Static Synergistic Selection Method for Candidate Code Solutions with Generated Test Cases (AAAI 2026 · better on 5 of 5 shared results)
- ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries (AAAI 2026 · better on 5 of 5 shared results)
- SRACG: A Code Generation Framework with Selective Retrieval Augmentation (AAAI 2026 · better on 2 of 2 shared results)
- Towards Functional Correctness of Large Code Models with Selective Generation (ICML 2026 · better on 1 of 1 shared result)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · better on 1 of 1 shared result)
- CLARC: C/C++ Benchmark for Robust Code Search (ICLR 2026 · better on 1 of 1 shared result)
- CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (ICLR 2025 · better on 1 of 1 shared result)
- AST-T5: Structure-Aware Pretraining for Code Generation and Understanding (ICML 2024 · better on 5 of 5 shared results)
6 more not listed.
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.89)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · similarity 0.89)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.88)
- TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark (ICLR 2025 · similarity 0.87)
- CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules (ICLR 2024 · similarity 0.87)
- ChatGPT-Generated Code Assignment Detection Using Perplexity of Large Language Models (Student Abstract) (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
