Research map / Vision-language and multimodal / Multimodal LLMs
MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAPS, which fine-tunes CogVLM-17B on synthetic circuit diagram/SPICE pairs to convert diagrams into simulator code, then uses a Chain-of-Simulation process with an MLLM and NgSPICE to answer college-level circuit analysis questions.
Problem
Multi-modal LLMs perform poorly on expert-level physical science problems that require parsing complex circuit diagrams and doing precise quantitative analysis, with frequent perception errors and mathematical hallucinations.
Approach
Decompose the task: a fine-tuned physics perception model converts the diagram into SPICE netlists, a physical simulator (NgSPICE) computes exact results, and the MLLM reasons over the simulation description and results (Chain-of-Simulation and Simulation-Aided Reasoning), trained on rule-synthesized diagram/SPICE pairs.
Key results
- GPT-4V + MAPS reaches 32.9% accuracy on SimpleCircuitEval versus 7.6% for GPT-4V alone.
- Claude-3.5 + MAPS improves from 6.33% to 25.3%, and GLM-4V + MAPS from 6.33% to 19.0%.
- PPM achieves 85.4% simulation accuracy on synthetic numerical circuits but only about 53.3% on real-world numerical diagrams.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MAPS perception and simulation framework | GPT-4V · SimpleCircuitEval | accuracy: 7.6 -> 32.9 | Helped |
| Add MAPS perception and simulation framework | Claude-3.5 · SimpleCircuitEval | accuracy: 6.33 -> 25.3 | Helped |
| Add MAPS perception and simulation framework | GLM-4V · SimpleCircuitEval | accuracy: 6.33 -> 19.0 | Helped |
| Replace direct GPT-4V prompting with Multimodal-CoT prompting | GPT-4V · SimpleCircuitEval | accuracy: 7.6 -> 5.1 | Hurt |
| Add simulation language description to final reasoning (ablation) | MAPS · SimpleCircuitEval, 20 sampled problems | accuracy: 45.0 -> 55.0 | Helped |
| Add physical simulator results (ablation) | MAPS · SimpleCircuitEval, 20 sampled problems | accuracy: 15.0 -> 55.0 | Helped |
| Use a physical simulator instead of MLLM-written Python (PoT) (ablation) | MAPS · SimpleCircuitEval, 20 sampled problems | accuracy: 15.0 -> 55.0 | Helped |
Surprising findings
- GPT-4V refused to generate SPICE code in 8 of 10 instances, so direct prompting for simulation descriptions is unreliable.
- Removing the simulator drops accuracy to 15% even with program-of-thought, showing the simulator is essential.
Limitations
- Static workflow with no feedback interaction between the MLLM and the simulator.
- Evaluated only on linear pure resistive circuits (first four chapters), a small set of 79 questions.
- Perception errors in PPM on real diagrams are the main failure source (18 of 20 sampled errors).
Experimental setup
Datasets: ppm-syn-lprc SimpleCircuitEval
Benchmarks: SimpleCircuitEval
Metrics: accuracy component quantity accuracy component value accuracy simulation accuracy
Baselines: GPT-4V GPT-4V + MMCoT Claude-3.5 GLM-4V Gemini-1.5 GPT-4o
Builds on: Multimodal-CoT Program-of-Thought LoRA
Closest papers by meaning
- Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models (ICLR 2026 · similarity 0.88)
- Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs (ICLR 2026 · similarity 0.88)
- SeePhys: Does Seeing Help Thinking? – Benchmarking Vision-Based Physics Reasoning (NeurIPS 2025 · similarity 0.87)
- RBench-V: A Primary Assessment for Visual Reasoning Models with Multimodal Outputs (NeurIPS 2025 · similarity 0.87)
- What MLLMs Learn about When they Learn about Multimodal Reasoning (NeurIPS 2026 · similarity 0.87)
- ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.