Research map / Large language models / Agents, code and math
Towards Functional Correctness of Large Code Models with Selective Generation
Jaewoo Jeong, Taesoo Kim, Sangdon Park
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SCG selective generation on top of CodeT | CodeT (GPT-3.5-Turbo) · MBPP-f | FDR-CE: 0.491±0.042 -> 0.145±0.052 | Improved |
| apply SCG on top of LDB | LDB (GPT-3.5-Turbo) · MBPP-f | FDR-CE: 0.442±0.044 -> 0.142±0.043 | Improved |
| apply SCG on top of SFS | SFS (GPT-3.5-Turbo) · HumanEval-f | FDR-CE: 0.288±0.072 -> 0.158±0.058 | Improved |
| add selective generation (SCG) to GPT-4o | GPT-4o · APPS-f | FDR-CE: 0.431±0.011 -> 0.224±0.018 | Improved |
| use fuzzing-generated unit tests instead of 21 APPS tests (SCG vs SCG-small) (ablation) | SCG (GPT-4o) · APPS-f | selection efficiency: 0.339±0.012 -> 0.337±0.011 | No clear effect |
| include false entailment rate term eps_E (SCG vs SCG-H) (ablation) | SCG (GPT-4o) · APPS-f | FDR-CE: 0.280±0.020 -> 0.224±0.018 | Improved |
| include false entailment rate term eps_E (SCG vs SCG-H) (ablation) | SCG (GPT-4o) · HumanEval-f | FDR-CE: 0.145±0.123 -> 0.049±0.111 | Improved |
| use calibrated threshold instead of top-k% manual selection (SCG vs SCG-manual) (ablation) | SCG (GPT-4o) · APPS-f | FDR-CE: 0.291±0.014 -> 0.224±0.018 | Improved |
| use exact-match selective predictor (SCG-EM) rather than unit-test entailment (ablation) | SCG-EM (GPT-4o) · APPS-f | selection efficiency: 0.000±0.000 | Hurt |
| use verbalized confidence scoring function instead of normalized log-prob (ablation) | SCG (GPT-4o) · APPS-f | FDR-CE control | Hurt |
| use smaller eps_E (more unit tests via fuzzing) (ablation) | SCG (GPT-4o) · APPS-f | FDR-CE bound tightness | Improved |
| increase number of fuzzed unit tests used for evaluation (ablation) | FuzzEval · APPS-f | FDR-CE | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- CodeT — CodeT: Code Generation with Generated Tests (ICLR 2023) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generation (NeurIPS 2024 · similarity 0.87)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · similarity 0.86)
- AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (ICML 2025 · similarity 0.86)
- CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification (AAAI 2025 · similarity 0.85)
- Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling (NeurIPS 2026 · similarity 0.85)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
