Research map / Image generation / Autoregressive and latent generation
Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GRIMOIRE, a text-conditioned SVG generator trained only on raster images, using a VQ-style Visual Shape Quantizer (FSQ codebook with DiffVG rasterization) and an autoregressive Transformer over shape, position and text tokens.
Problem
Generating SVG vector graphics typically requires SVG-level supervision, which needs heavy preprocessing and lacks visual supervision; prior image-supervised SVG models lack text conditioning and produce redundant shapes.
Approach
A finite-scalar-quantized autoencoder maps raster patches to discrete codes decoded into cubic Bézier shapes via a differentiable rasterizer (DiffVG), trained with raster MSE plus a geometric constraint; a second autoregressive Transformer models text, positions and codes jointly.
Key results
- On FIGR-8 full-dataset generation, GRIMOIRE with post-processing reaches FID 3.58 and CLIP 27.45, versus Im2Vec's no-text baseline (Im2Vec results on full data reported only for subsets).
- On FIGR-8 reconstruction, VSQ + PI reaches MSE 0.002 and FID 0.05 versus Im2Vec's MSE 0.050 and FID 13.90.
- On Fonts (A) text-conditioned generation, GRIMOIRE achieves CLIP 30.60 versus Im2Vec 25.39 (FID 5.61 vs 5.36).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VSQ instead of Im2Vec (filled) for reconstruction | Raster-supervised SVG reconstruction · MNIST | MSE: 0.140 -> 0.090 | Helped |
| Use VSQ instead of Im2Vec for reconstruction | Raster-supervised SVG reconstruction · Fonts | MSE: 0.050 -> 0.014 | Helped |
| Use VSQ instead of Im2Vec for reconstruction | Raster-supervised SVG reconstruction · FIGR-8 | MSE: 0.050 -> 0.004 | Helped |
| Use text-conditioned Grimoire instead of Im2Vec (filled) | Text-guided SVG generation · Fonts (A) | CLIPScore: 25.81 -> 30.60 | Helped |
| Use text-conditioned Grimoire instead of Im2Vec | Text-guided SVG generation · FIGR-8 (Star) | CLIPScore: 31.88 -> 32.24 | Helped |
| Add path clipping post-processing (ablation) | Grimoire · Fonts | FID: 4.44 -> 1.67 | Helped |
| Add path clipping post-processing (ablation) | Grimoire · FIGR-8 | FID: 4.20 -> 3.58 | Helped |
| Add path interpolation instead of no post-processing (ablation) | Grimoire · FIGR-8 | FID: 4.20 -> 4.57 | Hurt |
| Add 3 stroke context (ablation) | Grimoire text-guided generation · FIGR-8 | CLIPScore: 27.45 -> 25.31 | Hurt |
| Increase stroke context from 3 to 24 strokes (ablation) | Grimoire text-guided generation · FIGR-8 | CLIPScore: 25.31 -> 26.39 | Helped |
| Use two codes per centered stroke rather than one (ablation) | VSQ reconstruction · Fonts and FIGR-8 | MSE: 18.28%, 41.46%, and 26.09% | Helped |
| Increase the number of segments beyond 15 (ablation) | VSQ shape reconstruction · MNIST | reconstruction quality | No clear effect |
Surprising findings
- Im2Vec achieves better FID on filled MNIST and some icon classes, which the authors attribute to CLIP-based FID being biased for sparse images.
- About 39.91% of the 4,375-code stroke codebook goes unused on FIGR-8.
Limitations
- Relies on post-processing (path clipping/interpolation) to connect fragmented strokes in the final output.
- Only closed filled shapes validated on MNIST and outline strokes on icons and fonts; no broader SVG primitives tested.
- Image-supervised reconstruction FID can be unreliable for sparse, low-resolution images, as the authors note.
Experimental setup
Datasets: MNIST SVG-Fonts FIGR-8
Benchmarks: MNIST SVG-Fonts FIGR-8
Metrics: MSE FID CLIPScore
Baselines: Im2Vec Im2Vec (filled)
Builds on: VQ-VAE finite scalar quantization DiffVG Im2Vec IconShop
Closest papers by meaning
- Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models (CVPR 2025 · similarity 0.89)
- SVGDreamer: Text Guided SVG Generation with Diffusion Model (CVPR 2024 · similarity 0.87)
- OmniSVG: A Unified Scalable Vector Graphics Generation Model (NeurIPS 2025 · similarity 0.87)
- VecGlypher: Unified Vector Glyph Generation with Language Models (CVPR 2026 · similarity 0.86)
- StarVector: Generating Scalable Vector Graphics Code from Images and Text (AAAI 2025 · similarity 0.86)
- Towards High-fidelity Artistic Image Vectorization via Texture-Encapsulated Shape Parameterization (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.