Research map / Large language models / Agents, code and math
BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction
Tian Xia, Tianrun Gao, Wenhao Deng, Long Wei, Xiaowei Qian, Chenglei Yu, Tailin Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add one strong example (one-shot) to the prompt (ablation) | GPT-5 · Support Lv.1 | success rate (%): 85.9 -> 65.6 (↓20.3) | Hurt |
| add one strong example (one-shot) to the prompt (ablation) | GPT-4o · Support Lv.1 | success rate (%): 40.6 -> 25.0 (↓15.6) | Hurt |
| add one strong example (one-shot) to the prompt (ablation) | GPT-5 · Support Lv.1 | indicator (max load): 324.9 -> 294.0 (↓30.9) | Hurt |
| add one strong example (one-shot) to the prompt (ablation) | GPT-4o · Support Lv.1 | indicator (max load): 181.2 -> 160.6 (↓20.6) | Hurt |
| add simulator feedback closed-loop retries (up to five rounds) (ablation) | Fixed workflow · Support Lv.1 | success rate: to 100% success within five rounds | Improved |
| add an analyst agent that reviews simulator trajectories and failure causes (ablation) | Fixed workflow, three tested models · Lift Lv.2 | success rate: 0% -> non-zero | Improved |
| increase task difficulty from Lv.1 to Lv.3 in Support | GPT-5 · Support | success rate (%): 85.9 -> 10.9 | Hurt |
| increase task difficulty from Lv.1 to Lv.3 in Lift | GPT-5 · Lift | success rate (%): 95.3 -> 17.2 | Hurt |
| use more output tokens per construction run | LLM construction runs · Transport, Support, Lift | normalized performance | No clear effect |
| prompt explicitly for structural complexity and visual elaborateness (ablation) | LLMs with the same module set · Construction complexity | structure intricacy | Improved |
| scale up from GPT-4o-class models to GPT-5 on Lift Lv.2/3 | GPT-5 and Grok-4 vs others · Lift | success rate | Improved |
Closest papers by meaning
- Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs (NeurIPS 2025 · similarity 0.85)
- Scaling-Aware Adapter for Structure-Grounded LLM Reasoning (ICML 2026 · similarity 0.84)
- CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation (CVPR 2025 · similarity 0.82)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.82)
- Text-to-Code Generation for Modular Building Layouts in Building Information Modeling (NeurIPS 2025 · similarity 0.82)
- PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
