Research map / Large language models / Agents, code and math
Measuring Coding Challenge Competence With APPS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from GPT-2 0.1B to GPT-2 1.5B | GPT-2 fine-tuned on APPS · APPS test set, Average | Test Case Average: 6.16 -> 7.96 | Helped |
| Increase model size from GPT-2 1.5B to GPT-Neo 2.7B | Fine-tuned code-generation models · APPS test set, Average | Test Case Average: 7.96 -> 10.15 | Helped |
| Fine-tune on APPS rather than use few-shot prompting | GPT-3 175B versus fine-tuned GPT-2 0.1B · APPS test set, Average | Test Case Average: 0.55 -> 6.16 | Helped |
| Generate five candidate programs and use the best-performing one | GPT-Neo 2.7B · APPS test set, Introductory | Strict Accuracy: 3.9% -> 5.5% | Helped |
| Generate five candidate programs and use the best-performing one | GPT-Neo 2.7B · APPS test set, Introductory | Test Case Average: 14.7% -> 19.9% | Helped |
| Use BLEU instead of test-case correctness to compare models | Code-generation evaluation on APPS · APPS difficulty levels | BLEU: 26.8, 29.7, and 30.2; 27.1, 29.1, and 29.3 | Mixed |
Closest papers by meaning
- TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark (ICLR 2025 · similarity 0.87)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.87)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.87)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.87)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.86)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.