Research map / Large language models / Agents, code and math
GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMs
Shixian Luo, Zhu zezhou, Yu Yuan, Yuncheng Yang, Lianlei Shan, Yong Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add procedural code to text-only geometry problems | DeepSeek-R1 · AIME24 | accuracy: -23.5% | Hurt |
| Add procedural code to text-only geometry problems | DeepSeek-R1 · MATH-500 | accuracy: -10.9% | Hurt |
| Switch drawing code from Asymptote to Matplotlib (ablation) | QwQ-32B · AIME24 and MATH-500 geometry subsets | accuracy: less than 1% difference | No clear effect |
| Add 100 GeoGramBench samples to s1k fine-tuning (ablation) | s1.1-32B · GeoGramBench 200-problem subset | average accuracy: 38.18 -> 41.20 | Improved |
| Add 300 instead of 100 GeoGramBench samples (ablation) | s1.1-32B · GeoGramBench 200-problem subset | average accuracy: 41.20 -> 41.79 | No clear effect |
| Add 50 GeoGramBench samples (ablation) | s1.1-32B · GeoGramBench 200-problem subset, Primitive | accuracy: 59.38 -> 57.99 | Hurt |
| Apply Token Budget Forcing (append Wait tokens) (ablation) | s1.1-32B · GeoGramBench | accuracy: 54.60 -> 54.90 | No clear effect |
| Increase Token Budget Forcing N-Ignore to 6 (ablation) | s1.1-32B · GeoGramBench | accuracy | Hurt |
| Use reasoning/thinking models over non-reasoning models | GPT-5, GPT-o1 vs GPT-4o · GeoGramBench ALL | accuracy: 21.40 -> 70.92 | Improved |
Closest papers by meaning
- GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation (ICLR 2026 · similarity 0.91)
- Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructions (CVPR 2026 · similarity 0.91)
- Hilbert-Geo: Solving Solid Geometric Problems by Neural-Symbolic Reasoning (CVPR 2026 · similarity 0.90)
- SolidGeo: Measuring Multimodal Spatial Math Reasoning in Solid Geometry (NeurIPS 2025 · similarity 0.89)
- GeoTikzBridge: Advancing Multimodal Code Generation for Geometric Perception and Reasoning (CVPR 2026 · similarity 0.89)
- Do Large Language Models Truly Understand Geometric Structures? (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
