Research map / Multi-agent systems and games / LLM agents

Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Build-and-Find, a builder-finder protocol that evaluates whether agent-written repositories let downstream finder agents recover hidden design intent, gating inspection effort on accuracy and stability, and releases harness and artifacts for a two-task Rust panel.

Problem

Behavioral correctness benchmarks do not capture whether generated repositories expose intended behaviors and design choices clearly enough for later agents to inspect and reuse them as working context.

Approach

Treat a generated repository as a communication artifact: a builder implements a hidden spec, a finder sees only the codebase and a spec-traced multiple-choice question bank, and recovery accuracy, repeatability, implementation coverage, and novel inspection bytes are reported with gates so effort is interpreted only after reliable recovery.

Key results

• Artifact-conditioned recovery accuracy is 98.9% on the compile-pass primary panel versus 94.5% for question-only, a +4.4 pp lift.
• Implementation-aware downstream recovery ranges from 84.0% to 99.4% across builders, with builder-side coverage (85.0-100.0%) driving most variation.
• GPT-5.5 high-effort builder has conditional effort R_b=1.033, the lowest among builders in the high-effort panel.

Surprising findings

• Question-only finders reach 94.5% accuracy, so generic engineering priors nearly saturate the benchmark in this task pack.
• Seven of 48 artifact-present builds failed the root-level compile probe, and six of those contained a buildable nested Cargo crate.

Limitations

• High question-only baseline (94.5%) saturates accuracy, so effort carries the main comparison.
• Does not measure runtime correctness, security, deployment readiness, or human maintainability.
• Only two task families (Rust scratch_minidb and scratch_nanoweb) and a panel-local, prior-conditioned comparison; public task packs may be contaminated.

Experimental setup

Datasets: scratch_minidb scratch_nanoweb

Benchmarks: Build-and-Find task pack (scratch_minidb, scratch_nanoweb)

Metrics: exact-match recovery accuracy 3-trial agreement implementation coverage conditional inspection effort R_b AELS

Baselines: question-only control spec-only control

Builds on: HumanEval SWE-bench LLM-as-a-judge

Closest papers by meaning

• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.91)
• RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations (NeurIPS 2026 · similarity 0.90)
• BootstrapAgent: Turning Repository Setup into Reusable Agent Knowledge (NeurIPS 2026 · similarity 0.90)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.90)
• GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.89)
• EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.89)