Research map / Vision-language and multimodal / Human and embodied modeling
AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes using TikZ as an intermediate representation for generating scientific vector figures from captions, releasing a 120k TikZ-caption dataset (AutomaTikZ) and CLiMA, a LLaMA variant augmented with CLIP embeddings.
Problem
Generating scientific vector graphics from text is hard because low-level SVG primitives are difficult for models to produce directly and prior methods handle only simple icons or fonts; no large TikZ dataset existed.
Approach
Use the high-level TikZ graphics language as an intermediate representation for text-conditioned figure generation, and inject CLIP multimodal embeddings as soft prompts into LLaMA (CLiMA) so the model gets a visual interpretation of captions; add iterative resampling guided by compiler logs to fix errors.
Key results
- CLiMA-13b reaches CLIPScore 26.966 and CrystalBLEU 3.369 versus LLaMA-13b 26.898 and 3.258, with CLiMA-7b improving over LLaMA-7b on five of seven metrics.
- GPT-4 and Claude obtain up to 2.1pp higher CLIPScore but tend to copy caption text into images (typographic attacks), and are 6.6pp lower on CLIPScore-img with KID over six times larger.
- In best-worst scaling human evaluation, CLiMA-13b is the only model besides humans with a mode above 0 on caption similarity, and GPT-4 lags behind.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate CLIP multimodal embeddings into LLaMA | LLaMA[7b] · Human-created test set | CrystalBLEU: by up to 0.4pp | Helped |
| Integrate CLIP multimodal embeddings into LLaMA | LLaMA[13b] · Human-created test set | CLIPScore[img]: 0.1pp higher CLIPScore[img] for LLaMA[13b] | Hurt |
| Add reference images as inputs at inference | CLiMA[13b] · Human-created test set | CLIPScore[img]: 1.2pp | Helped |
| Add reference images as inputs at inference | CLiMA[13b] · Human-created test set | KID | No clear effect |
| Remove caption augmentation (ablation) | CLiMA[7b] · Human-created test set | CLIPScore: 26.776 -> 25.703 | Hurt |
| Remove caption augmentation (ablation) | CLiMA[7b] · Human-created test set | CSR: 1.202 -> 1.169 | Helped |
| Remove image inputs during training (ablation) | CLiMA[7b] · Human-created test set | CLIPScore: 26.776 -> 26.604 | Hurt |
| Remove Arxiv papers (86k) from training (ablation) | CLiMA[7b] · Human-created test set | CrystalBLEU: 2.82 -> 0.886 | Hurt |
| Remove Stack Exchange (29k) from training (ablation) | CLiMA[7b] · Human-created test set | CrystalBLEU: 2.82 -> 2.936 | Helped |
| Remove artificial examples (4k) from training (ablation) | CLiMA[7b] · Human-created test set | CrystalBLEU: 2.82 -> 2.537 | Hurt |
| Remove curated examples (1k) from training (ablation) | CLiMA[7b] · Human-created test set | CrystalBLEU: 2.82 -> 2.711 | Hurt |
| Use GPT instead of CLiMA[13b] | TikZ generation · Human-created test set | CLIPScore[img]: 81.017 vs 78.63 | Hurt |
Surprising findings
- GPT-4 and Claude achieve higher CLIPScore largely through typographic attacks, copying caption text into generated images, so CLIPScore is misleading for text-rich figures.
- Human-created TikZ code contains roughly twice as many tokens as GPT-4 or Claude outputs, and GPT-4 and Claude generate simpler figures.
Limitations
- Training context limited to 1200 tokens, filtering out longer examples and preventing multi-turn compiler feedback.
- Arxiv-derived data excludes over 120k drawings that failed to compile; arXiv data redistribution restricted by license, so only generation code is released for it.
- Human evaluation uses only 100 test items with expert annotators; crowdsourcing was discarded due to low agreement.
Experimental setup
Datasets: AutomaTikZ MSCOCO LVIS VisOr OpenMoji LAION-style 595K text-image pairs
Benchmarks: AutomaTikZ test set (1k human-created items)
Metrics: CLIPScore CLIPScore-img KID CrystalBLEU EED compilation sampling rate best-worst scaling
Baselines: GPT-4 Claude LLaMA-7b LLaMA-13b
Builds on: LLaMA CLIP LoRA CapFilt CLIPScore
Closest papers by meaning
- DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ (NeurIPS 2024 · similarity 0.92)
- ScImage: How good are multimodal large language models at scientific text-to-image generation? (ICLR 2025 · similarity 0.88)
- Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models (CVPR 2025 · similarity 0.86)
- VecGlypher: Unified Vector Glyph Generation with Language Models (CVPR 2026 · similarity 0.86)
- Empowering LLMs to Understand and Generate Complex Vector Graphics (CVPR 2025 · similarity 0.85)
- VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.