Research map / Large language models / Agents, code and math
Do Large Language Models Truly Understand Geometric Structures?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GeomRel, a text-only benchmark isolating geometric relationship identification (GRI) from final answers, evaluates LLMs, and proposes GeoCoT prompting that improves GRI accuracy.
Problem
LLMs' geometric problem-solving is evaluated only by final-answer accuracy, which can be right by coincidence, so their true understanding of geometric structures (geometric relationship identification) is not measured.
Approach
Isolate the geometric relationship identification step into a multiple-choice benchmark (GeomRel) built from a 26-relationship pool with basic and advanced (composed, diversified, indeterminate) data, and propose GeoCoT, a two-stage prompt (geometry breakdown then reverse-reasoning relationship observation) to improve identification.
Key results
- GPT-4o achieves 77.86% average accuracy on basic GeomRel but only 47.93% on advanced GeomRel.
- GeoCoT in Few-Shot setting improves GPT-3.5-Turbo by an average of 9.15% on basic and 14.79% on advanced GeomRel.
- Fine-tuned LLaMA3-8B-Base reaches 71.67% on advanced line-based, while fine-tuned LLaMA3-8B-Instruct shows no improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Few-Shot-GeoCoT instead of standard Few-Shot prompting | GPT-3.5-Turbo · GeomRel basic and advanced | accuracy: +9.15% basic; +14.79% advanced | Helped |
| Add geometry breakdown (Stage 1) to GeoCoT (ablation) | GPT-3.5-Turbo with Few-Shot-GeoCoT · GeomRel basic and advanced | accuracy: 63.96 -> 72.55 basic; 44.58 -> 51.98 advanced | Helped |
| Add reverse thinking in Stage 2 (ablation) | GPT-3.5-Turbo with Few-Shot-GeoCoT · GeomRel basic and advanced | accuracy: 68.03 -> 72.55 basic; 42.79 -> 51.98 advanced | Helped |
| Add geometry breakdown (Stage 1) to GeoCoT (ablation) | GPT-3.5-Turbo with Zero-Shot-GeoCoT · GeomRel basic and advanced | accuracy: 62.50 -> 65.04 basic; 37.93 -> 40.64 advanced | Helped |
| Add reverse thinking in Stage 2 (ablation) | GPT-3.5-Turbo with Zero-Shot-GeoCoT · GeomRel basic and advanced | accuracy: 64.37 -> 65.04 basic; 38.81 -> 40.64 advanced | Helped |
| Use Zero-Shot-CoT instead of Zero-Shot prompting | GPT-3.5-Turbo · GeomRel, line/angle/shape domains | accuracy: 60.71 vs 65.57; 46.07 vs 46.23; 46.82 vs 54.17; 21.75 vs 21.75; 71.48 vs 72.22; 40.39 vs 43.26 | Mixed |
| Use Few-Shot-CoT instead of Few-Shot examples without reasoning | GPT-3.5-Turbo · GeomRel basic and advanced, line/angle/shape domains | accuracy: 70.67 vs 64.87; 49.94 vs 45.75; 59.20 vs 53.29; 30.94 vs 24.30; 71.53 vs 72.05; 46.68 vs 41.52 | Mixed |
| Fine-tune LLaMA3-8B-Instruct on GeomRel question-answer pairs | LLaMA3-8B-Instruct · GeomRel basic and advanced | accuracy: 63.16 -> 34.14 line basic; 52.17 -> 32.50 angle basic | Hurt |
| Fine-tune LLaMA3-8B-Base on GeomRel | LLaMA3-8B-Base · GeomRel basic and advanced | accuracy: 86.86 line basic; 71.67 line advanced; 63.16 angle basic; 54.90 angle advanced; 93.75 shape basic; 61.69 shape advanced | Helped |
| Fine-tune Qwen2-7B-Instruct on GeomRel | Qwen2-7B-Instruct · GeomRel basic and advanced | accuracy: 65.55 -> 82.14 line basic; 53.75 -> 78.95 angle basic; 55.38 -> 94.48 shape basic | Helped |
| Fine-tune Qwen2-7B-Base on GeomRel | Qwen2-7B-Base · GeomRel basic and advanced | accuracy: 89.29 line basic; 70.48 line advanced; 73.68 angle basic; 51.51 angle advanced; 95.70 shape basic; 70.13 shape advanced | Helped |
| Increase geometric-description complexity using point relabeling or unrelated information (ablation) | GPT-3.5-Turbo on GeomRel subsets · GeomRel basic and advanced, line/angle/shape domains | accuracy | Mixed |
Surprising findings
- On three existing datasets, GPT-3.5-Turbo's geometric relationship identification accuracy was lower than its final answer accuracy (e.g., Geometry3K 0.12 vs 0.36), indicating correct answers are often reached by coincidence.
- Zero-shot chain-of-thought and few-shot prompting did not improve geometric relationship identification over zero-shot prompting, and CoT even hurt in some domains.
Limitations
- Scope limited to planar geometry and text-only descriptions.
- Open-source models perform near random on the advanced subset.
- Fine-tuning gains may be affected by small dataset size and split randomness.
Experimental setup
Datasets: GeomRel Geometry3K PGPS9K MATH
Benchmarks: GeomRel
Metrics: accuracy precision recall F1 pass@k
Baselines: GPT-4o GPT-4-Turbo GPT-3.5-Turbo Claude-3-Opus QwenMax Qwen1.5-110B LLaMA3-70B-Instruct LLaMA3-8B-Instruct LLaMA2-13B-Chat Zero-Shot-CoT
Builds on: chain-of-thought prompting few-shot in-context learning
Closest papers by meaning
- G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model (ICLR 2025 · similarity 0.92)
- SolidGeo: Measuring Multimodal Spatial Math Reasoning in Solid Geometry (NeurIPS 2025 · similarity 0.89)
- GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training (ICLR 2025 · similarity 0.89)
- Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructions (CVPR 2026 · similarity 0.88)
- GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation (ICLR 2026 · similarity 0.88)
- GeoTikzBridge: Advancing Multimodal Code Generation for Geometric Perception and Reasoning (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.