Research map / Vision-language and multimodal / Multimodal LLMs
SolidGeo: Measuring Multimodal Spatial Math Reasoning in Solid Geometry
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SolidGeo, a 3,113-problem benchmark of solid geometry math problems with images, fine-grained categories and difficulty levels, and evaluates 27 LLMs/MLLMs against human performance.
Problem
Existing multimodal math benchmarks focus on plane geometry and include very few solid (3D) geometry problems, leaving MLLMs' spatial mathematical reasoning under-evaluated.
Approach
A large, real-world K-12 and competition solid geometry benchmark with 8 expert-defined fine-grained subject categories, 3 difficulty levels, and multi-stage filtering (structural, OpenCV sharpness, LLM semantic, n-gram deduplication), evaluated across zero-shot, CoT, and few-shot prompting plus token-efficiency analysis.
Key results
- OpenAI-o1 achieves the highest overall accuracy at 49.5% on SolidGeo, versus 77.5% for human performance.
- Gemini-2.5-pro reaches 42.7% overall and 80.7% on Level 3 problems while scoring only 22.1% on Level 1.
- Llama-4-Maverick-17B-128E is the best open-source system-1 model at 29.6%, while the best open-source system-2 model QvQ-72B-Preview reaches 26.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Chain-of-Thought prompting (ablation) | Gemini-1.5-pro · SolidGeo | accuracy: 25.3 -> 26.1 (+0.8) | Helped |
| Add Chain-of-Thought prompting and 2-shot examples (ablation) | Gemini-1.5-pro · SolidGeo | accuracy: 25.3 -> 27.5 (+2.2) | Helped |
| Add Chain-of-Thought prompting (ablation) | GPT-4o · SolidGeo | accuracy: 25.5 -> 24.9 (-0.6) | Hurt |
| Add Chain-of-Thought prompting and 2-shot examples (ablation) | GPT-4o · SolidGeo | accuracy: 25.5 -> 22.9 (-2.6) | Hurt |
| Add Chain-of-Thought prompting (ablation) | Qwen2.5-VL-Instruct-72B · SolidGeo | accuracy: 24.2 -> 28.8 (+4.6) | Helped |
| Add Chain-of-Thought prompting and 2-shot examples (ablation) | Qwen2.5-VL-Instruct-72B · SolidGeo | accuracy: 24.2 -> 26.1 (+1.9) | Helped |
| Add Chain-of-Thought prompting (ablation) | LLaVA-OneVision-Chat-7B · SolidGeo | accuracy: 8.6 -> 10.5 (+1.9) | Helped |
| Add Chain-of-Thought prompting and 2-shot examples (ablation) | LLaVA-OneVision-Chat-7B · SolidGeo | accuracy: 8.6 -> 11.4 (+2.8) | Helped |
| Add CoT or CoT with 2-shot examples (ablation) | Evaluated models on SolidGeo · SolidGeo | accuracy | Mixed |
Surprising findings
- Gemini-2.5-pro and OpenAI-o1 perform better on harder (Level 3) problems than on Level 1 problems, contrary to the trend for other models.
- CoT and few-shot prompting do not consistently improve accuracy; GPT-4o and LLaVA-OV-72B degrade under these settings.
Limitations
- Dataset includes only English and Chinese problems.
- No widely adopted formal language exists for representing solid geometry problems, limiting symbolic and programmatic evaluation.
- Category distribution is imbalanced (smallest categories have about 110-135 samples).
Experimental setup
Datasets: MathVerse MATH-Vision DynaMath GeoEval CMMaTH OlympiadBench Zujuan platform K-12 problems
Benchmarks: SolidGeo
Metrics: accuracy average generated tokens human performance comparison
Baselines: GPT-4o Claude-3.7-Sonnet Gemini-2.5-pro OpenAI-o1 Qwen2.5-VL-72B InternVL3-78B Llama-4-Maverick-17B-128E QvQ-72B-Preview
Builds on: MathVista MathVerse MATH-Vision DynaMath OlympiadBench
Closest papers by meaning
- Hilbert-Geo: Solving Solid Geometric Problems by Neural-Symbolic Reasoning (CVPR 2026 · similarity 0.91)
- Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructions (CVPR 2026 · similarity 0.91)
- Do Large Language Models Truly Understand Geometric Structures? (ICLR 2025 · similarity 0.89)
- GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation (ICLR 2026 · similarity 0.89)
- Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces (ICML 2026 · similarity 0.88)
- GeoTikzBridge: Advancing Multimodal Code Generation for Geometric Perception and Reasoning (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.