Research map / Multi-agent systems and games / LLM agents

Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Phoenix-bench, a 511-instance execution-verified Verilog/SystemVerilog issue-resolution benchmark, and evaluates four commercial coding agents and eight open-source agent structures across four LLM backbones, plus localization and testbench-feedback diagnostics.

Problem

Whether software-engineering coding agents transfer to realistic hardware repository issue resolution, which requires cross-hierarchy navigation, localization, EDA-executed verification and patching; existing hardware benchmarks isolate sub-tasks.

Approach

A GitHub-issue-driven hardware repair benchmark where patches are judged by Docker-pinned Verilator/Icarus fail-to-pass and pass-to-pass testbenches, paired with diagnostics on file-level oracle localization and one round of testbench-log feedback.

Key results

• Claude Code achieves 38.6% resolved rate on Phoenix-bench, with Codex at 38.0% and Gemini CLI at 37.4%.
• OpenHands with GPT-5.2 reaches 33.9% resolved rate, while the same agent class drops 37 to 58 points from SWE-bench Verified.
• A single round of testbench-log feedback raises Claude Code from 38.6% to 83.2% resolved, while a perfect file-level oracle adds only +1.4 points.

Surprising findings

• A perfect file-level oracle yields only +1.4 points because agents then break files that did not need editing.
• The strongest SWE-bench pair (mini-SWE-agent with GPT-5.2) is among the weakest on Phoenix-bench at 14.5%.

Limitations

• Paper's own text states the statistical significance section lacks confidence intervals or bootstrap estimates.
• Scope limited to open-source EDA tools (Verilator, Icarus, Surelog, Yosys); synthesis stages beyond synthesis excluded due to non-determinism.
• Possible public-repository contamination noted in the checklist.

Experimental setup

Datasets: Phoenix-bench corpus collected from GitHub Verilog/SystemVerilog repositories

Benchmarks: Phoenix-bench SWE-bench Verified

Metrics: resolved rate file precision file recall module precision module recall tokens per instance

Baselines: Claude Code OpenAI Codex Gemini CLI GitHub Copilot coding agent mini-SWE-agent SWE-agent OpenHands Aider KGCompass Lingxi ExpeRepair Agentless

Builds on: SWE-bench SWE-agent OpenHands Agentless mini-SWE-agent

Closest papers by meaning

• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions (ICML 2026 · similarity 0.86)
• PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification (ICML 2025 · similarity 0.86)
• daVinci-Dev: Agent-native Mid-training for Software Engineering (ICML 2026 · similarity 0.86)
• Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering with Generative Optimization (NeurIPS 2026 · similarity 0.86)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.85)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.85)