Research map / Large language models / Agents, code and math
Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EngDesign, a multi-domain engineering design benchmark with simulation-based pass/fail and partial-credit scoring, and finds frontier LLMs pass only up to about 34% of tasks.
Problem
Existing LLM benchmarks focus on factual recall, textbook QA, or code, and do not evaluate open-ended engineering design that must satisfy constraints and be validated functionally.
Approach
Design tasks across nine engineering domains are scored by domain-specific simulation pipelines (e.g., SPICE, MATLAB, FEA) with per-task rubrics giving partial credit, replacing static answer matching; the paper also introduces a reasoning-robustness metric and an iterative feedback-based design protocol.
Key results
- o3 achieves the highest overall pass rate of 34.38% on EngDesign, while GPT-4o and Gemini-2.0-Flash reach around 15%.
- Analog IC design tasks yield 0% pass rate for all evaluated models.
- Iterative feedback raises o3 pass rate to almost 60% after ten iterations on a 71-task subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use o3 instead of general-purpose chat models | EngDesign evaluation · EngDesign overall | average pass rate (%): 15.68 -> 34.38 | Helped |
| Use o3 instead of Claude-3.7-Sonnet | EngDesign evaluation · EngDesign overall | average pass rate (%): 22.61 -> 34.38 | Helped |
| Use iterative design with evaluation feedback | o3 · EngDesign 71-task subset | pass rate (%): almost a 60\% after ten iterations | Helped |
| Use iterative design with evaluation feedback | LLMs on Analog IC Design tasks · Analog IC Design | pass rate (%): 0\% | No clear effect |
| Use reasoning-focused models rather than chat-oriented models | LLMs evaluated on EngDesign · EngDesign overall | reasoning robustness: 0.20 -> 0.62 | Helped |
Surprising findings
- Iterative design does not help on Analog IC design tasks, where models still fail after ten iterations.
- Claude-3.7-Sonnet and Claude-3.7-Thinking show similar pass rates, so reasoning mode does not help Claude as it does for other models.
Limitations
- Covers only nine engineering domains and does not span all disciplines.
- Evaluation includes a selected set of models and is not exhaustive.
- Structured output via instructor sometimes fails for some LLMs.
Experimental setup
Datasets: EngDesign
Benchmarks: EngDesign EngDesign-Open
Metrics: pass rate average score reasoning robustness token consumption
Baselines: GPT-4o Claude-3.7-Sonnet Gemini-2.0-Flash DeepSeek-v3 o1 o3 o4-mini Gemini-2.5-Pro DeepSeek-R1
Builds on: instructor
Closest papers by meaning
- EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark (CVPR 2025 · similarity 0.85)
- EngiAgent: Fully Connected Coordination of LLM Agents for Solving Open-ended Engineering Problems with Feasible Solutions (ICML 2026 · similarity 0.85)
- Automated Creation of Reusable and Diverse Toolsets for Enhancing LLM Reasoning (AAAI 2025 · similarity 0.84)
- BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints (NeurIPS 2025 · similarity 0.84)
- ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution (NeurIPS 2024 · similarity 0.84)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.