Research map / Large language models / Instruction and fine-tuning
Empowering LLMs to Understand and Generate Complex Vector Graphics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLM4SVG, which adds 55 learnable SVG semantic tokens to pretrained LLMs and fine-tunes them on the 580k-sample SVGX-SFT dataset to both understand and generate complex SVG graphics.
Problem
LLMs produce hallucinated, semantically ambiguous SVG primitives and occlusion-prone outputs because SVG is tokenized as plain text, and existing SVG generators handle only simple graphics and lack instruction data.
Approach
Replace SVG tags and attributes with dedicated learnable semantic tokens initialized from the average embedding of their descriptions, and train pretrained LLMs (GPT-2, Phi-2, Falcon, LLaVA) with a two-stage scheme (embedding alignment then SFT) on a large instruction-following SVG dataset.
Key results
- LLM4SVG (GPT-2-XL) reaches FID 64.11 and CLIPScore 0.3496, outperforming all LLM baselines such as Claude-3.5 (FID 82.89, CLIPScore 0.3083) in the table.
- In human evaluation, LLM4SVG scores 0.89 prompt alignment and 0.92 visual quality versus 0.49/0.61 for GPT-4o and 0.56/0.74 for Claude 3.5.
- LLM4SVG generates SVGs in roughly 12-25 seconds for 10 SVGs, far faster than optimization-based methods (up to 49 min).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM4SVG (GPT-2-XL) instead of SVGDreamer | Text-to-SVG generation · Text-to-SVG generation | FID: 72.68 -> 64.11 | Helped |
| Use LLM4SVG (GPT-2-XL) instead of SVGDreamer | Text-to-SVG generation · Text-to-SVG generation | CLIPScore: 0.3001 -> 0.3496 | Helped |
| Use LLM4SVG (GPT-2-XL) instead of SVGDreamer | Text-to-SVG generation · Text-to-SVG generation | Aesthetic: 5.5432 -> 5.9836 | Helped |
| Use LLM4SVG (GPT-2-XL) instead of SVGDreamer | Text-to-SVG generation · Text-to-SVG generation | HPS: 0.2685 -> 0.2485 | Hurt |
| Use LLM4SVG (GPT-2-XL) instead of Claude-3.5 | LLM-based SVG generation · Text-to-SVG generation, LLM comparison | FID: 82.89 -> 64.11 | Helped |
| Use LLM4SVG (GPT-2-XL) instead of Claude-3.5 | LLM-based SVG generation · Text-to-SVG generation, LLM comparison | CLIPScore: 0.3083 -> 0.3496 | Helped |
| Use LLM4SVG (GPT-2-XL) instead of Claude-3.5 | LLM-based SVG generation · Text-to-SVG generation, LLM comparison | Aesthetic: 5.2370 -> 5.9836 | Helped |
| Use LLM4SVG (GPT-2-XL) instead of Claude-3.5 | LLM-based SVG generation · Text-to-SVG generation, LLM comparison | HPS: 0.1912 -> 0.2485 | Helped |
| Use LLM4SVG instead of Claude-3.5 | Human-evaluated SVG generation · Human evaluation | Prompt Alignment: 0.56 -> 0.89 | Helped |
| Use LLM4SVG instead of Claude-3.5 | Human-evaluated SVG generation · Human evaluation | Visual Quality: 0.74 -> 0.92 | Helped |
| Use LLM4SVG instead of Claude-3.5 | Human-evaluated SVG generation · Human evaluation | Pick Score: 0.69 -> 0.88 | Helped |
Surprising findings
- Qwen models tokenize numbers and decimal points as units, which hurts coordinate and color generation, while GPT-2 enumerates numbers up to 1000 as single tokens and performs better.
Limitations
- Optimization-based methods still achieve better FID and visual quality than LLM4SVG.
- Training sequences truncated at 4096 tokens, which may cut off long complex SVGs.
- Human evaluation is limited to comparisons with GPT-4o and Claude 3.5 and a small set of examples.
Experimental setup
Datasets: SVGX-SFT
Benchmarks: SVGX-SFT
Metrics: FID CLIPScore Aesthetic score HPS generation time Avg.Tok
Baselines: CLIPDraw VectorFusion DiffSketcher SVGDreamer LIVE DeepSVG Iconshop StrokeNUWA GPT-4o Claude 3.5 Sonnet Gemini 1.5 Pro Llama 3.1 70B
Builds on: LLaVA LoRA QLoRA LlamaFactory BLIP
Closest papers by meaning
- Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models (CVPR 2025 · similarity 0.90)
- InternSVG: Towards Unified SVG Tasks with Multimodal Large Language Models (ICLR 2026 · similarity 0.90)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.87)
- StarVector: Generating Scalable Vector Graphics Code from Images and Text (AAAI 2025 · similarity 0.87)
- LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.86)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.