Research map / Large language models / Agents, code and math
Can Large Language Models Understand Symbolic Graphics Programs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SGP-Bench, a procedurally built benchmark testing whether LLMs can answer semantic questions about SVG and CAD programs without seeing rendered images, and proposes Symbolic Instruction Tuning (SIT) on GPT-4o-captioned renderings, which improves both program understanding and some general reasoning benchmarks.
Problem
Whether LLMs can semantically understand symbolic graphics programs (SVG, CAD) and answer questions about their rendered images using only the program text, without a vision encoder.
Approach
Generate multiple-choice semantic questions from rendered images with GPT-4o, then pose them to LLMs with only the symbolic program as input; add SE(2)-perturbed programs to test consistency; and finetune LLMs on GPT-4o detailed captions of renderings (Symbolic Instruction Tuning).
Key results
- Claude 3.5 Sonnet achieves the best SVG understanding average of 0.674 and CAD average of 0.742; GPT-4o reaches 0.633 on SVG.
- SIT raises Llama-3.1-8B semantic-program accuracy from 46.5 to 51.4 with 55K pairs.
- Mixing original and reverse SIT data into Open-Instruct improves average general benchmark scores over the OI baseline; e.g., AGIEval 23.7 to 31.6 with rev-SIT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from 8B to 70B | Llama-3 · SGP-Bench SVG Understanding | accuracy: 0.429 to 0.548 | Helped |
| Increase model size from 7B to 110B | Qwen-1.5 · SGP-Bench SVG Understanding | accuracy: 0.376 to 0.499 | Helped |
| Fine-tune with 55K original SIT instruction question-answer pairs | Llama-3.1-8B · SGP-Bench SVG Understanding | accuracy: 46.7% to 51.4% | Helped |
| Increase SIT dataset size from 10K to 55K (ablation) | Llama-3.1-8B · SGP-Bench SVG Understanding | accuracy: 48.0 to 51.4 | Helped |
| Use SIT-10k data (ablation) | Llama3-8B · SGP-Bench SVG Understanding | accuracy: 43.20 -> 48.16 | Helped |
| Use SIT-25k data (ablation) | Llama3-8B · SGP-Bench SVG Understanding | accuracy: 43.20 -> 51.43 | Helped |
| Use SIT-40k data (ablation) | Llama3-8B · SGP-Bench SVG Understanding | accuracy: 43.20 -> 45.62 | Helped |
| Use SIT-55k data (ablation) | Llama3-8B · SGP-Bench SVG Understanding | accuracy: 43.20 -> 40.99 | Hurt |
| Use SIT-10k data (ablation) | Gemma-7B · SGP-Bench SVG Understanding | accuracy: 39.33 -> 45.60 | Helped |
| Use SIT-25k data (ablation) | Gemma-7B · SGP-Bench SVG Understanding | accuracy: 39.33 -> 46.87 | Helped |
| Use SIT-40k data (ablation) | Gemma-7B · SGP-Bench SVG Understanding | accuracy: 39.33 -> 45.21 | Helped |
| Use SIT-55k data (ablation) | Gemma-7B · SGP-Bench SVG Understanding | accuracy: 39.33 -> 47.28 | Helped |
Surprising findings
- On SGP-MNIST, even GPT-4o reaches only 13.0% accuracy on 4-way digit recognition, near chance, despite humans recognizing the rendered digits easily.
- SIT on Llama-3.1-8B reduces instruction-following format compliance, requiring LLM-based answer extraction.
Limitations
- LLMs show low consensus on symbolic-program answers compared with humans, and finetuning does not close the gap to larger models.
- Semantic questions are generated by GPT-4o and only manually inspected, so label noise is possible.
- SIT degrades instruction-following format compliance, complicating evaluation.
Experimental setup
Datasets: SVG Icons (Kaggle) DeepCAD Fusion360 Reconstruction Dataset SketchGraphs MNIST-SVG Open-Instruct
Benchmarks: SGP-Bench (SVG understanding) SGP-Bench (SVG invariance) SGP-Bench (CAD) SGP-MNIST XNLI IFEval HellaSwag C-Eval MMLU BBH GSM8k MathQA
Metrics: accuracy consistency score
Baselines: GPT-4o GPT-4 Turbo GPT-3.5 Turbo Claude 3.5 Sonnet Llama3.1-405B Open-Instruct (OI)
Builds on: visual instruction tuning Alpaca LoRA orthogonal finetuning
Closest papers by meaning
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.88)
- Do Vision-Language Models Really Understand Visual Language? (ICML 2025 · similarity 0.88)
- Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs (ICLR 2026 · similarity 0.88)
- LTD-Bench: Evaluating Large Language Models by Letting Them Draw (NeurIPS 2025 · similarity 0.88)
- A Vision Check-up for Language Models (CVPR 2024 · similarity 0.88)
- GENOME: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.