Research map / Large language models / Agents, code and math

HSCO-Bench: An Agent-Driven End-to-End Hardware-Software Co-design Benchmark for Systems-on-Chip

NeurIPS 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HSCO-Bench, an end-to-end benchmark where LLM agents (via mini-swe-agent) must pick kernels, design HLS accelerators, integrate them into an ESP RISC-V SoC, and validate on a VC707 FPGA across 10 workloads.

Problem

Existing LLM benchmarks evaluate software and hardware design separately, with fixed hardware or fixed algorithmic targets; no benchmark tests whether an agent can perform end-to-end HW/SW co-design of accelerator-rich SoCs.

Approach

Places the agent in an open-source ESP SoC workspace where it must analyze C application code, generate HLS (SystemC) accelerators with drivers, integrate them into the SoC configuration, and verify on FPGA, scoring correctness, CPU-cycle speedup, and FPGA resource overhead.

Key results

• Claude Opus 4.6 achieved 70% success rate and 2.32x geometric-mean speedup with +2.22% average resource overhead.
• GPT-5.4 reached peak speedup of 16.22x on TinyCLIP-T at +15.37% resource overhead; only Opus 4.6 and GPT-5.4 produced meaningful successes.
• MiniMax M2.5, Kimi K2.5, and Gemini 3.1 Pro succeeded in 10%, 0%, and 10% of cases respectively.

Surprising findings

• Even successful frontier agents were conservative, mostly instantiating a single accelerator and leaving hardware underutilized (max additional overhead 23.67%).
• Over-ambitious multi-accelerator designs from Kimi K2.5 universally failed during integration.

Limitations

• Only five models evaluated with a single agent scaffold (mini-swe-agent), with small success counts (10 attempts per model).
• Workloads scaled to run within one hour on FPGA, and the testbed is a single ESP 2x2 SoC template on VC707.
• Cost analysis covers only two successful models.

Experimental setup

Datasets: 10 curated workloads from open-source GitHub repos (Autoencoder, BERT-Tiny, MiniLM, TinyCLIP-T, EfficientViT, MiniDeiT,

Benchmarks: HSCO-Bench

Metrics: success rate geometric-mean speedup FPGA resource overhead cost efficiency

Baselines: MiniMax M2.5 Kimi K2.5 Gemini 3.1 Pro GPT-5.4 Claude Opus 4.6

Builds on: ESP mini-swe-agent Catapult HLS

Closest papers by meaning

• KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware (ICML 2026 · similarity 0.85)
• Towards a Comprehensive Benchmark for High-Level Synthesis Targeted to FPGAs (NeurIPS 2023 · similarity 0.85)
• PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models (NeurIPS 2026 · similarity 0.84)
• CUDABench: Benchmarking LLMs for Text-to-CUDA Generation (NeurIPS 2026 · similarity 0.83)
• HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models (NeurIPS 2024 · similarity 0.82)
• Benchmarking End-To-End Performance of AI-Based Chip Placement Algorithms (NeurIPS 2025 · similarity 0.82)