Research map / Large language models / Agents, code and math
LTD-Bench: Evaluating Large Language Models by Letting Them Draw
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LTD-Bench, a 183-item benchmark that evaluates LLMs by having them draw characters and objects via dot matrices or Python code, with generation and recognition tasks at three difficulty levels, finding large gaps in spatial imagination and perception.
Problem
Standard LLM benchmarks report opaque numerical scores that hide limits in spatial reasoning and bidirectional language-spatial mapping, with no intuitive view of what models can do spatially.
Approach
Evaluate LLMs by requiring visual outputs: models draw via 0-1 dot matrices or executable drawing code, and recognize characters from matrices or code, across easy, normal and hard levels; hard-level outputs are scored by GPT-4.1 and visual style similarity is used to probe model similarity.
Key results
- DeepSeek-R1 achieves the best average of 71.54% across LTD-Bench, the only model above 70%; GPT-4.1-mini averages 64.38%.
- Qwen2.5-72B-Instruct and Llama3.3-70B-Instruct average about 30.75% and 27.07%.
- Llama3.3-70B distilled with DeepSeek-R1 data gains 18.05 points on recognition but drops 2.91 points on generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Deepseek-r1 instead of GPT-4.1-mini | LTD-Bench models · LTD-Bench recognition | accuracy: 47.22 -> 73.61 | Helped |
| Use Deepseek-r1 instead of GPT-4.1-mini | LTD-Bench models · LTD-Bench generation | score: 77.46 -> 72.88 | Hurt |
| Distill Llama3.3-70B-Instruct with Deepseek-r1 data | Llama3.3-70B-Instruct · LTD-Bench recognition | accuracy: +18.05% | Helped |
| Distill Llama3.3-70B-Instruct with Deepseek-r1 data | Llama3.3-70B-Instruct · LTD-Bench generation | score: 2.91% decline | Hurt |
| Distill Llama3.3-70B-Instruct with Deepseek-r1 data | Llama3.3-70B-Instruct · LTD-Bench | average score: +7.57% | Helped |
Surprising findings
- Multimodal GPT-4.1-mini and GPT-4o do not consistently outperform text-only DeepSeek models on the spatial tasks.
- Deep reasoning improves recognition but not generation; distillation can degrade generation performance.
Limitations
- Small dataset (183 items) and narrow range of abilities limit generalizability.
- Model similarity analysis is preliminary, based on a small set of 22 style comparisons.
- Hard-level scoring relies on GPT-4.1 judge, which can hallucinate and shows output variance.
Experimental setup
Benchmarks: LTD-Bench
Metrics: accuracy score (0-1) for hard generation human vs GPT-4.1 agreement style similarity rate
Baselines: DeepSeek-R1 DeepSeek-V3 GPT-4.1-mini GPT-4o QwQ-32B Qwen2.5-72B-Instruct Llama3.3-70B-Instruct
Closest papers by meaning
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.90)
- Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark (AAAI 2024 · similarity 0.89)
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.89)
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.89)
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.89)
- Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.