AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

Escaping Whack-a-Mole: Optimizing Documentation as Repo-Specific Playbooks for Coding Agents

Yutong Cheng, Haifeng Chen, Wenchao Yu, Xujiang Zhao, Peng Gao, Wei Cheng

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Frames agent-oriented code documentation as repo-specific playbooks optimized via black-box search, with quality judged only by downstream coding-agent task success.

Problem

Code documentation is written for human readability, but coding agents need documentation they can execute; the paper targets generating documentation that specifies precise agent behaviors for a given repository.

Approach

Formulates agent-oriented documentation generation as black-box optimization over the documentation space, where a document's quality is defined solely by its downstream effect on agent performance rather than human readability.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add reverse topological order (ablation)DocSearch (GPT-4o) · DevEval+ (10-module subset)solve rate: 90.7 vs 72.2 (-18.5 without)Improved
add error diversity (diversified batch sampling) (ablation)DocSearch (GPT-4o) · DevEval+ (10-module subset)solve rate: 90.7 vs 81.1 (-9.6 without)Improved
add worthy condition (ablation)DocSearch (GPT-4o) · DevEval+ (10-module subset)solve rate: 90.7 vs 83.7 (-7.0 without)Improved
add reverse topological ordering to iterative refinementIterative (3 LLMs) · DevEval+solve rate: 41.4 -> 44.1, 62.6 -> 67.4, 52.9 -> 58.1Improved
add worthy condition and bi-level search on top of Iterative+TopoIterative + Topo (GPT-4o) · DevEval+solve rate: 58.1 -> 90.7Improved
add test generation to Cursor agentCursor (Gemini-2.5-Flash) · DevEval+solve rate: 17.2 -> 33.9Improved
add test and topo order to Claude CodeClaude Code (Claude-4.5-Sonnet) · DevEval+solve rate: 41.9 -> 64.8Improved
use optimized documentation instead of source code for Python->JavaGPT-4o · Python->Java (10 modules)solve rate: 41.4 -> 68.7Improved
use optimized documentation instead of source code for Python->GoGPT-4o · Python->Go (5 modules)solve rate: 39.6 -> 62.3Improved
add source code to optimized documentation (Java)GPT-4o · Python->Java (10 modules)solve rate: 68.7 -> 79.3Improved
add source code to optimized documentation (Go)GPT-4o · Python->Go (5 modules)solve rate: 62.3 -> 72.1Improved
use DocSearch-optimized documentation instead of initial documentationGPT-4o · human evaluation SpecificityLikert score: 2.17 -> 4.52Improved

Experimental setup

Metrics: downstream agent task success

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy