Research map / Image generation / Text-to-image diffusion
Evaluating Numerical Reasoning in Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GECKO-NUM, a human-annotated benchmark of 1386 prompts evaluating numerical reasoning (exact counts, approximate quantities/zero, fractions) in 12 text-to-image models, and uses it to test auto-eval metrics and PaliGemma counting.
Problem
Text-to-image models often fail to generate correct numbers of objects, approximate quantities, zero, and fractional/partial quantities, and no comprehensive benchmark exists to evaluate numerical reasoning in them.
Approach
A controlled, template-based prompt suite across three numerical tasks (exact, approximate/zero, conceptual quantitative reasoning) with twelve prompt types, paired with large-scale human annotations of generated images, used to evaluate T2I models and auto-eval metrics and to serve as a VQA counting benchmark.
Key results
- DALL-E 3 achieves the highest Task 1 exact-number accuracy (45.2%) and Task 2 accuracy (48.7%), while Task 3 accuracy is near random chance (~50%) for most models.
- Accuracy drops with each successive number: for DALL-E 3, 18 pp from 1 to 2, 9 pp from 2 to 3, and 23 pp from 3 to 4.
- Fine-tuning PaliGemma on TallyQA improves GECKO-NUM accuracy from 68.4% to 73.3%; Gecko and VNLI were the only auto-eval metrics that reliably separated correct from incorrect images for all models tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| represent numbers with words instead of digits (ablation) | Text-to-image models on numeric-simple prompts (numbers 1–4) · Task 1: Exact Number Generation | accuracy: 64.6 -> 78.4 | Helped |
| represent numbers with words instead of digits (ablation) | DALL·E on numeric-simple prompts (numbers 1–4) · Task 1: Exact Number Generation | accuracy: 69.8 -> 70.4 | No clear effect |
| use frequent nouns instead of rare nouns (ablation) | Text-to-image models on numeric-simple prompts · Task 1: Exact Number Generation | accuracy: 67.6 -> 75.0 | Helped |
| use frequent nouns instead of rare nouns (ablation) | DALL·E on numeric-simple prompts · Task 1: Exact Number Generation | accuracy: 71.7 -> 68.6 | No clear effect |
| increase the requested count from one to the next successive number (ablation) | DALL·E on numeric-simple prompts · Task 1: Exact Number Generation | accuracy: 18 pp decrease for 1 → 2, 9 pp for 2 → 3, and 23 pp for 3 → 4 | Hurt |
| add another number–noun combination (2-additive instead of numeric-simple) (ablation) | Text-to-image models · Task 1: Exact Number Generation | accuracy: -9.7 to -35.4 | Hurt |
| add a third number–noun combination (3-additive instead of 2-additive) (ablation) | Text-to-image models · Task 1: Exact Number Generation | accuracy: -0.1 to -13.0 | Hurt |
| add color terms to numeric-simple prompts (ablation) | Text-to-image models · Task 1: Exact Number Generation | accuracy: 0.81 -> 0.76; 0.64 -> 0.72 | Mixed |
| add color terms to 2-additive prompts (ablation) | Text-to-image models · Task 1: Exact Number Generation | accuracy: 0.42 -> 0.31; 0.51 -> 0.38 | Mixed |
| add spatial relationships to 2-additive prompts (ablation) | Text-to-image models · Task 1: Exact Number Generation | accuracy: 0.31 -> 0.25 | Hurt |
| use two entities rather than one in approximate quantity prompts (ablation) | Text-to-image models · Task 2: Approximate Number Generation and Zero | accuracy | Hurt |
| fine-tune on TallyQA (train) (ablation) | PaliGemma · GeckoNum counting VQA | accuracy: 68.4{\scriptstyle\pm0.4}\% -> 73.3{\scriptstyle\pm0.4}\% | Helped |
Surprising findings
- Ten of twelve models were more accurate when numbers were written as words rather than digits.
- Models most often failed on the concept of zero and 'no X' prompts, even DALL-E 3 consistently failing on 'a watermelon with no seeds'.
Limitations
- Relies on costly human annotation that does not scale.
- Small noun vocabulary (N=40) for frequency analysis may not generalize.
- Task 3 evaluation via DSG questions yields a 50% baseline for some prompts, making results hard to interpret.
Experimental setup
Datasets: TallyQA GECKO-NUM prompts and generated images
Benchmarks: GECKO-NUM TallyQA (test)
Metrics: accuracy standard error of the mean Mann-Whitney U test Wilcoxon signed-rank test chi-squared test
Baselines: CLIPscore TIFA Gecko DSG VNLI random chance
Builds on: Gecko DSG TIFA TallyQA CountBench
Closest papers by meaning
- MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning (NeurIPS 2025 · similarity 0.88)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.87)
- GIR-Bench: Versatile Benchmark for Generating Images with Reasoning (ICLR 2026 · similarity 0.86)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.86)
- Text-to-Image Generation for Abstract Concepts (AAAI 2024 · similarity 0.86)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.