Research map / Large language models / Agents, code and math
EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EEE-Bench, a 2,860-question multimodal benchmark of electrical and electronics engineering problems with diagrams, and evaluates 17 LLMs/LMMs plus prompting methods and a caption-based "laziness" analysis.
Problem
Existing multimodal benchmarks do not systematically test whether LMMs can solve visually complex, technical engineering problems such as electrical and electronics engineering (EEE) circuit and system diagrams.
Approach
A curated benchmark of EEE exam problems across 10 subdomains, filtered so that text-only GPT-4 cannot answer them, to force reliance on visual understanding; it also probes whether LMMs over-rely on text by injecting spurious captions.
Key results
- GPT-4o achieves the best overall accuracy of 46.78% on EEE-Bench, while the best open-source model InternVL2-26B reaches 26.89%.
- Adding GPT-4o-generated captions lowers GPT-4o accuracy by 1.43 points on average, and spurious captions lower it by 7.79 points.
- Self-Consistency gives GPT-4o a +3.54 point gain (50.32%), the largest among the prompting methods tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add accurate image captions (ablation) | GPT-4o · EEE-Bench | accuracy: 46.78 -> 45.35 | Hurt |
| add spurious image captions (ablation) | GPT-4o · EEE-Bench | accuracy: -7.79% | Hurt |
| add spurious image captions (ablation) | Gemini-Pro 1.5 · EEE-Bench | accuracy: -6.78% | Hurt |
| use Self-Consistency prompting (ablation) | GPT-4o · EEE-Bench | accuracy: +3.54% | Helped |
| use Self-Consistency prompting (ablation) | Gemini-Pro 1.5 · EEE-Bench | accuracy: +3.73% | Helped |
| use Cantor prompting (ablation) | GPT-4o · EEE-Bench | accuracy: +3.16% | Helped |
| use Cantor prompting (ablation) | Gemini-Pro 1.5 · EEE-Bench | accuracy: +3.99% | Helped |
| use DDCot prompting (ablation) | GPT-4o · EEE-Bench | accuracy: +2.34% | Helped |
| use Self-Correction prompting (ablation) | GPT-4o · EEE-Bench | accuracy: +1.48% | Helped |
| use PROGRAM prompting (ablation) | GPT-4o · EEE-Bench | accuracy: -0.52% | Hurt |
| use 1-Shot CoT prompting (ablation) | GPT-4o · EEE-Bench | accuracy: -0.19% | Hurt |
| use PROGRAM prompting (ablation) | Gemini-Pro 1.5 · EEE-Bench | accuracy: -1.35% | Hurt |
Surprising findings
- Adding GPT-4o-generated image captions often hurts performance, with GPT-4o dropping 1.43 points on average.
- Many prompting techniques that help on language-only tasks give little or negative gains on this multimodal benchmark.
Limitations
- Evaluation is zero-shot only, and captions and knowledge are generated by proprietary models.
- Relies on answer extraction by GPT-4o-mini, which adds a judge-model dependency.
Experimental setup
Datasets: EEE-Bench
Benchmarks: EEE-Bench
Metrics: accuracy
Baselines: GPT-4 GPT-4o-mini GPT-4V Gemini-Pro 1.5 Gemini-Flash 1.5 Qwen-VL-Max Qwen-VL-Plus LLaVA-1.6-7B LLaVA-1.6-13B DeepSeek-VL Phi3-Vision MiniCPM-V-2.6
Builds on: MathVista Chain-of-thought prompting Self-Consistency DDCoT
Closest papers by meaning
- CircuitSense: A Hierarchical Circuit System Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process (ICLR 2026 · similarity 0.87)
- Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs (NeurIPS 2025 · similarity 0.85)
- MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals (CVPR 2026 · similarity 0.85)
- LLM Circuit Analyses Are Consistent Across Training and Scale (NeurIPS 2024 · similarity 0.84)
- PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models (NeurIPS 2026 · similarity 0.84)
- CircuitFusion: Multimodal Circuit Representation Learning for Agile Chip Design (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.