Research map / Large language models / Agents, code and math
PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PCEval, a simulation-based benchmark of 50 Arduino physical-computing projects that automatically evaluates LLMs on logical/physical circuit generation and code generation from each, finding code and logical circuits far easier than breadboard layouts.
Problem
LLMs' ability to reason about physical hardware constraints (breadboard layouts, pin assignments) for physical computing projects is unexplored, and prior evaluations relied on manual expert judgment.
Approach
Decompose physical computing into four generation tasks (logical circuit, physical breadboard circuit, and code conditioned on logical or physical circuits) with automatic simulation-based test procedures and circuit validation metrics, avoiding manual evaluation.
Key results
- o3-mini achieves the highest total overall success (61.7%), with 66.0% on logical circuit generation but only 45.2% on physical circuit generation.
- Physical circuit generation success drops sharply with complexity, e.g., o3-mini from 65.0% at L1 to 25.5% at L4 and Claude 3.7 Sonnet from 33.3% to 5.5%.
- Self-improvement raises o3-mini from 61.7% to 76.5%; chain-of-thought improves GPT-4o by 10.4 points on the physical code task.
Surprising findings
- Most models perform better at code generation than at circuit generation, and many score below 10% on physical circuit generation.
- Pin conflicts are the dominant error in physical circuits, with Claude 3.7 Sonnet averaging 7.52 per sample despite a 43.6% physical-code success.
Limitations
- Scope limited to introductory educational projects and the Arduino Uno platform.
- Physical circuit evaluation covers electrical correctness and breadboard constraints but not wiring readability or placement ergonomics.
- Human evaluation involved only two educators and three pre-service teachers.
Experimental setup
Datasets: PCEval projects (50 Arduino projects)
Benchmarks: PCEval
Metrics: success rate simulation success circuit validation error counts CodeBLEU
Baselines: MICRO25 EmbedGenius EmbedTask
Builds on: MICRO25 EmbedGenius HumanEval
Closest papers by meaning
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.86)
- L2MAC: Large Language Model Automatic Computer for Extensive Code Generation (ICLR 2024 · similarity 0.86)
- The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.85)
- CUDABench: Benchmarking LLMs for Text-to-CUDA Generation (NeurIPS 2026 · similarity 0.85)
- VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification (NeurIPS 2025 · similarity 0.84)
- DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.