Research map / Large language models / Agents, code and math
DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DeTikZify, a multimodal LM (SigLIP encoder + code LLMs) that synthesizes TikZ graphics programs from scientific figures and sketches, with MCTS-based self-refinement guided by compiler diagnostics or self-assessed perceptual similarity, plus three new datasets.
Problem
Recovering semantics-preserving TikZ graphics programs from compiled scientific figures and generating TikZ from hand-drawn sketches, where existing tools and commercial LLMs perform poorly.
Approach
Combine a SigLIP vision encoder with TeX-pretrained code LLMs, train on figures plus synthetic sketches (InstructPix2Pix-generated), and apply a tailored MCTS at inference that uses compiler diagnostics or the model's own visual embedding similarity as reward to iteratively refine its outputs without extra training.
Key results
- DeTikZify-DS-7B (OI) reaches average score 0.965 on reference figures vs 0.612 for GPT-4o and 0.148 for Claude 3 Opus in output-driven inference.
- Time-budgeted MCTS improves SSIM by 2.2-5.6 points and DreamSim by 4.9-8.7 points across models.
- In human BWS evaluation, DeTikZify-DS-7B (TI) scores mean 0.39 vs 0.09 for GPT-4o (OI), while GPT-4o (OI) beats DeTikZify (OI) (-0.32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeTikZify DS 7B instead of GPT for output-driven generation | Scientific figure-to-TikZ generation · Reference figures | average: 0.612 -> 0.965 | Helped |
| Use DeTikZify DS 7B instead of GPT for output-driven generation | Scientific figure-to-TikZ generation · Synthetic sketches | average: 0.15 -> 0.965 | Helped |
| Increase inference budget using MCTS | DeTikZify DS 7B · Reference figures | average: 0.965 -> 0.76 | Hurt |
| Increase inference budget using MCTS | DeTikZify DS 7B · Synthetic sketches | average: 0.965 -> 0.994 | Helped |
| Use MCTS time-budgeted refinement instead of output-driven inference | DeTikZify DS 7B · Human evaluation on sketches | human preference score: $\mu=-0.32$ -> $\mu=0.39$ | Helped |
| Use Visual Self-Refine instead of output-driven inference | GPT · Human evaluation on sketches | human preference score: $\mu=0.09$ -> $\mu=-0.16$ | Hurt |
| Add synthetic-sketch training (ablation) | DeTikZify DS 1.3B · Reference figures | average performance across reported metrics: MTE -1.957; CBLEU +0.327; TED -0.822; DSIM +2.433; SSIM +1.318; KID +3.296 | Mixed |
| Add synthetic-sketch training (ablation) | DeTikZify DS 1.3B · Synthetic sketches | average performance across reported metrics: MTE -13.358; CBLEU +0.171; TED +1.369; DSIM -3.993; SSIM -3.332; KID +34.18 | Helped |
| Pretrain the connector on MetaFig (ablation) | DeTikZify DS 1.3B · Reference figures | average performance across reported metrics: MTE -1.846; CBLEU -0.096; TED -0.356; DSIM +0.398; SSIM -0.046; KID +0.115 | No clear effect |
| Pretrain the connector on MetaFig (ablation) | DeTikZify DS 1.3B · Synthetic sketches | average performance across reported metrics: MTE -0.132; CBLEU +0.053; TED -0.378; DSIM +0.084; SSIM -0.181; KID +2.773 | No clear effect |
Surprising findings
- GPT-4o fails to improve with visual self-refinement in time-budgeted inference (mean BWS drops to -0.16), while DeTikZify improves with MCTS.
- Self-assessed SSIM from the model's own vision encoder correlates best with humans at segment level, outperforming trained metrics like LPIPS and DISTS.
Limitations
- Proprietary baselines lack transparency and their performance changes over time, complicating fairness and data-leakage checks.
- Public DaTikZ v2 release omits some programs due to licensing; artificial subset restricted to non-commercial use.
- Models may inherit biases and could be misused to produce misinformation and fake science.
Experimental setup
Datasets: DaTikZ v2 MetaFig SketchFig DaTikZ v1
Benchmarks: DaTikZ v2 test split (reference figures) SketchFig (synthetic and real sketches)
Metrics: CrystalBLEU TED DreamSim SSIM KID average normalized score MTE MST Bradley-Terry-style BWS human score Spearman correlation
Baselines: GPT-4o Claude 3 Opus LIVE Self-Refine Visual Self-Refine
Builds on: MCTS SigLIP LLaVA InstructPix2Pix DeTikZify v1 (DaTikZ)
Closest papers by meaning
- AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ (ICLR 2024 · similarity 0.92)
- Factuality Matters: When Image Generation and Editing Meet Structured Visuals (ICLR 2026 · similarity 0.85)
- ScImage: How good are multimodal large language models at scientific text-to-image generation? (ICLR 2025 · similarity 0.85)
- GeoTikzBridge: Advancing Multimodal Code Generation for Geometric Perception and Reasoning (CVPR 2026 · similarity 0.83)
- Can Large Language Models Understand Symbolic Graphics Programs? (ICLR 2025 · similarity 0.83)
- VecGlypher: Unified Vector Glyph Generation with Language Models (CVPR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.