Research map / Multi-agent systems and games / LLM agents
Escaping Whack-a-Mole: Optimizing Documentation as Repo-Specific Playbooks for Coding Agents
Yutong Cheng, Haifeng Chen, Wenchao Yu, Xujiang Zhao, Peng Gao, Wei Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames agent-oriented code documentation as repo-specific playbooks optimized via black-box search, with quality judged only by downstream coding-agent task success.
Problem
Code documentation is written for human readability, but coding agents need documentation they can execute; the paper targets generating documentation that specifies precise agent behaviors for a given repository.
Approach
Formulates agent-oriented documentation generation as black-box optimization over the documentation space, where a document's quality is defined solely by its downstream effect on agent performance rather than human readability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reverse topological order (ablation) | DocSearch (GPT-4o) · DevEval+ (10-module subset) | solve rate: 90.7 vs 72.2 (-18.5 without) | Improved |
| add error diversity (diversified batch sampling) (ablation) | DocSearch (GPT-4o) · DevEval+ (10-module subset) | solve rate: 90.7 vs 81.1 (-9.6 without) | Improved |
| add worthy condition (ablation) | DocSearch (GPT-4o) · DevEval+ (10-module subset) | solve rate: 90.7 vs 83.7 (-7.0 without) | Improved |
| add reverse topological ordering to iterative refinement | Iterative (3 LLMs) · DevEval+ | solve rate: 41.4 -> 44.1, 62.6 -> 67.4, 52.9 -> 58.1 | Improved |
| add worthy condition and bi-level search on top of Iterative+Topo | Iterative + Topo (GPT-4o) · DevEval+ | solve rate: 58.1 -> 90.7 | Improved |
| add test generation to Cursor agent | Cursor (Gemini-2.5-Flash) · DevEval+ | solve rate: 17.2 -> 33.9 | Improved |
| add test and topo order to Claude Code | Claude Code (Claude-4.5-Sonnet) · DevEval+ | solve rate: 41.9 -> 64.8 | Improved |
| use optimized documentation instead of source code for Python->Java | GPT-4o · Python->Java (10 modules) | solve rate: 41.4 -> 68.7 | Improved |
| use optimized documentation instead of source code for Python->Go | GPT-4o · Python->Go (5 modules) | solve rate: 39.6 -> 62.3 | Improved |
| add source code to optimized documentation (Java) | GPT-4o · Python->Java (10 modules) | solve rate: 68.7 -> 79.3 | Improved |
| add source code to optimized documentation (Go) | GPT-4o · Python->Go (5 modules) | solve rate: 62.3 -> 72.1 | Improved |
| use DocSearch-optimized documentation instead of initial documentation | GPT-4o · human evaluation Specificity | Likert score: 2.17 -> 4.52 | Improved |
Experimental setup
Metrics: downstream agent task success
Closest papers by meaning
- Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.89)
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.88)
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.88)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.88)
- Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages (NeurIPS 2026 · similarity 0.88)
- RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
