Research map / Multi-agent systems and games / LLM agents

Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EnConda-Bench, a process-level benchmark for LLM agents' environment configuration that injects realistic errors into READMEs and evaluates error diagnosis, repair, and Docker-validated executability.

Problem

Environment configuration is a bottleneck for SWE agents, and existing benchmarks only report end-to-end build/test success, hiding where and why agents fail; high-quality buildable repositories are also scarce.

Approach

Treat an executable README as ground truth, inject two LLM-generated errors per README with labeled types and fixes, validate effectiveness via Docker and LLM filtering, and evaluate agents on error localization (precision/recall/F1), fix description/fix accuracy, and end-to-end Pass@1 to give process-level diagnosis.

Key results

• Repo2Run + Claude-4 achieves the best error-type F1 of 60.6, fix accuracy 47.3, and Pass@1 of 22.9.
• Zero-shot GPT-4.1 has 90.6 recall but only 33.4 precision on error typing, with Pass@1 of 1.5.
• OpenHands + Claude-4 reaches Pass@1 of 10.6 and description accuracy of 54.2; models often collapse errors into the 'other' category.

Surprising findings

• Models often correctly name the error type without locating the erroneous command in the README.
• Claude-4 uses about three times as many output tokens as DeepSeek-V3 in zero-shot yet gains only about 0.2 Pass@1.

Limitations

• Errors are LLM-synthesized and may not fully reflect real-world configuration failures.
• Only Python repositories from 323 selected repos are covered.
• Pass rates are low and the benchmark relies on an LLM judge for diagnosis matching.

Experimental setup

Datasets: EnvBench SetupBench INSTALLAMATIC bench ExecutionAgent bench

Benchmarks: EnConda-Bench

Metrics: precision recall F1 description accuracy fix accuracy Pass@1

Baselines: GPT-4.1 Claude-4 Gemini2.5-pro DeepSeek-V3 OpenHands SWE-Agent INSTALLAMATIC Repo2Run

Builds on: EnvBench SWE-bench Repo2Run INSTALLAMATIC EXECUTIONAGENT

Closest papers by meaning

• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents (NeurIPS 2026 · similarity 0.87)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.87)
• daVinci-Dev: Agent-native Mid-training for Software Engineering (ICML 2026 · similarity 0.86)
• SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.86)
• RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.86)
• Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.85)