Research map / Vision-language and multimodal / Multimodal LLMs

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VFig-Data (66K diagram-centric figure-SVG pairs) and VFig-Bench evaluation suites, and trains VFig VLMs with simple-to-complex SFT followed by GRPO using rendering-aware rubric VLM-judge rewards.

Problem

Raster-to-SVG conversion for complex diagram-centric figures (model architectures, flowcharts, schematics) is underserved: existing SVG datasets are icon-heavy and benchmarks rely on pixel or embedding similarity that misses structural errors like broken arrow connectivity.

Approach

Combine a large diagram-centric figure-to-SVG dataset (real arXiv figures converted via a describe-and-generate pipeline plus programmatic shapes-and-arrows diagrams) with a structure-aware benchmark, and train VLMs using a simple-to-complex SFT curriculum followed by GRPO with a four-axis rubric VLM-judge reward computed on rendered SVGs.

Key results

• VFig-4B (SFT+RL) reaches Gemini-Judge 78.2 and GPT-Judge 87.5 on VFig-Bench, versus 76.7/87.4 for Claude-Sonnet-4.6, and 91.5/94.8 for Gemini-3-Pro.
• Simple-to-complex SFT raises Render rate by 18.4 points over one-stage SFT on Qwen3-VL-4B (74.9 to 93.3).
• Rubric-based VLM-judge reward outperforms DINO embedding reward and pixel-augmented rewards on judge metrics; no-rubric variant loses 2-3 judge points.

Surprising findings

• Adding pixel-level objectives (SSIM) to the rubric reward slightly improves SSIM but lowers VLM-judge scores, indicating a tradeoff.
• Human Elo ranks VFig (SFT+RL) above Claude-Sonnet-4.6, reversing the VLM-judge ordering.

Limitations

• Models still struggle with fine-grained connectors and arrowheads.
• Real-world training targets are Gemini-3-Pro renders, so the data inherits a stylistic prior.
• Connectivity-aware filtering of generated samples is left for future work; judges come from only two model families.

Experimental setup

Datasets: VFig-Data-Complex-Diagrams VFig-Data-Shapes-and-Arrows SVG-Diagrams Molmo2-Diagram Paper2Fig

Benchmarks: VFig-Bench VFig-Bench-OOD Molmo2-Diagram SVG-Diagrams

Metrics: SSIM LPIPS VisualSim Gemini-Judge GPT-Judge Clean Render rate Elo

Baselines: VTracer Claude-Sonnet-4.6 GPT-5.2 Gemini-3-Flash Gemini-3-Pro OmniSVG StarVector InternSVG Qwen3-VL-4B InternVL3.5-4B

Builds on: GRPO LoRA StarVector Paper2Fig

Closest papers by meaning

• Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models (CVPR 2025 · similarity 0.90)
• OmniSVG: A Unified Scalable Vector Graphics Generation Model (NeurIPS 2025 · similarity 0.89)
• InternSVG: Towards Unified SVG Tasks with Multimodal Large Language Models (ICLR 2026 · similarity 0.89)
• SVGDreamer: Text Guided SVG Generation with Diffusion Model (CVPR 2024 · similarity 0.89)
• SVGBuilder: Component-Based Colored SVG Generation with Text-Guided Autoregressive Transformers (AAAI 2025 · similarity 0.88)
• StarVector: Generating Scalable Vector Graphics Code from Images and Text (AAAI 2025 · similarity 0.87)