Research map / Large language models / Agents, code and math
DocPrompting: Generating Code by Retrieving the Docs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add retrieved documentation to CodeT5 generation | CodeT5 · tldr test | EM (%): 2.18 -> 9.15 | Helped |
| Add retrieved documentation to Codex few-shot prompting | Codex 3-shots · tldr test | charBLEU: 16.94 -> 23.72 | Helped |
| Add retrieved documentation with oracle command name | Codex 3-shots · tldr test | EM (%): 22.44 -> 32.43 | Helped |
| Add retrieved documentation to CodeT5 generation | CodeT5 · CoNaLa test | BLEU: 34.57 -> 36.22 | Helped |
| Add retrieved documentation to CodeT5 generation | CodeT5 · CoNaLa test | Recall unseen: 9.03 -> 18.30 | Helped |
| Add retrieved documentation to Codex few-shot prompting | Codex 3-shots · CoNaLa test | BLEU: 43.16 -> 43.47 | No clear effect |
| Use oracle documentation instead of retrieved documentation | CodeT5 + DocPrompting · CoNaLa test | BLEU: 36.22 -> 49.04 | Helped |
| Use oracle documentation instead of retrieved documentation | Codex + DocPrompting · CoNaLa test | BLEU: 43.47 -> 50.59 | Helped |
| Remove weak supervision on the documentation pool (ablation) | CodeT5 retriever · CoNaLa dev | Recall@10: 55.81 -> 29.34 | Hurt |
| Remove weak supervision on the documentation pool (ablation) | RoBERTa retriever · tldr dev | Recall@10: 60.33 -> 59.84 | No clear effect |
| Use BM25 instead of a trained dense retriever (ablation) | Retriever · CoNaLa dev | Recall@10: 55.81 -> 9.73 | Hurt |
| Retrieve more documents (ablation) | CodeT5 + DocPrompting · CoNaLa | BLEU | Mixed |
Closest papers by meaning
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.87)
- LILO: Learning Interpretable Libraries by Compressing and Documenting Code (ICLR 2024 · similarity 0.85)
- CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (ICML 2025 · similarity 0.85)
- LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models (NeurIPS 2026 · similarity 0.85)
- Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation (NeurIPS 2024 · similarity 0.85)
- Toward Interactive Understanding of Code APIs (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.