Research map / Multi-agent systems and games / LLM agents

Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering with Generative Optimization

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Frontier-Eng, a 47-task benchmark where LLM agents iteratively edit executable engineering artifacts under frozen verifiers and a fixed budget, scored by rank-based metrics and evaluated across frontier models and search frameworks.

Problem

Existing agent benchmarks test binary pass/fail or one-shot answers, not iterative, budget-aware improvement of feasible engineering artifacts under hard constraints and simulator feedback.

Approach

Formalizes generative optimization as a triple (context, feasible initial solution, frozen evaluator) with best-feasible-score-within-budget as the objective, packages 47 heterogeneous engineering tasks with anti-reward-hacking isolation, and aggregates results via average rank, performance profiles, and win rate.

Key results

• gpt-5.4 attains the best average rank (3.54) after retest, narrowly ahead of claude-opus-4.6 (3.63) across 47 tasks under openevolve with 100 iterations.
• Claude-opus-4.6 outperforms gpt-oss-120b under all three frameworks (openevolve rank 1.28 vs 1.64 in the two-model comparison).
• Under a fixed budget of 256 steps, a single deep chain scores 1.00 normalized versus 0.91 for 16 chains of depth 16, indicating depth matters more than width.

Surprising findings

• Depth dominates width: splitting a fixed evaluation budget across independent chains reduces best-found score.
• Improvement frequency and magnitude follow approximate power laws, with marginal returns near zero after roughly 50-100 iterations.

Limitations

• Ranking-based primary metric discards magnitude of improvement
• Performance profile translation for negative-score tasks changes the meaning of alpha thresholds
• Results rely on a fixed 100-iteration budget and a few search frameworks

Experimental setup

Datasets: ISCSO2015 truss benchmark ISCSO2023 tower benchmark MQT Bench quantum circuits Summit reaction emulators OpenProblems single-cell platform MuJoCo and PyBullet simulators JSSP ABZ/SWV/TA instances

Benchmarks: Frontier-Eng

Metrics: average rank performance profile win rate over baseline best feasible score within budget

Baselines: abmcts openevolve shinkaevolve

Builds on: AlphaEvolve FunSearch Self-Refine Learning to Discover at Test Time OpenEvolve

Closest papers by meaning

• Understanding the Challenges in Iterative Generative Optimization with LLMs (NeurIPS 2026 · similarity 0.88)
• UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios (ICML 2026 · similarity 0.87)
• Huxley-G\"odel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine (ICLR 2026 · similarity 0.86)
• Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench (NeurIPS 2026 · similarity 0.86)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.86)
• AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.86)