Research map / Vision-language and multimodal / Multimodal LLMs
DaVinci: Reinforcing Visual-Structural Syntax in MLLMs for Generalized Scientific Diagram Parsing
Xingchen ZENG, Zhewei Su, Hengming Zhang, Juyong Jiang, Jiazhi Xia, Wei Zeng
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DaVinci, an MLLM for parsing raster scientific diagrams into structured representations, trained with a two-stage supervised-then-reinforcement framework.
Problem
Parsing raster-based scientific diagrams into structured, editable representations is hard for existing MLLMs due to diverse visual primitives, complex layouts, and strict syntax requirements.
Approach
A two-stage training framework for diagram parsing: supervised learning on visual primitives followed by reinforcement learning to reinforce visual-structural syntax in the MLLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reorder TikZ code into a logical drawing order (ablation) | Qwen2.5-VL-7B SFT · DaTikZv3 | Pass@1 compile rate: 69.74 -> 78.78 | Improved |
| add LLM-written planning comments to TikZ training code (ablation) | Qwen2.5-VL-7B SFT · DaTikZv3 | Pass@1 compile rate: 78.78 -> 84.50 | Improved |
| add GRPO RL with hybrid reward after SFT | DaVinci-SFT-7B · DaTikZv3 | Pass@1 compile rate: 84.50 -> 97.60 | Improved |
| add GRPO RL after SFT | DaVinci-SFT-7B · DaTikZv3 | cBLEU: 7.52 -> 6.57 | Hurt |
| add spatio-textual reward Rtext (ablation) | DaVinci RL · DaTikZv3 | Textual match score: 37.23 -> 41.58 | Improved |
| add geometric reward Rgeom on top of Rtext (ablation) | DaVinci RL · DaTikZv3 | Geometry match score: 42.12 -> 44.10 | Improved |
| add SSIM reward to base image reward (ablation) | DaVinci RL · DaTikZv3 | Textual/Geometry match: 37.23 -> 37.45 (textual), 41.44 -> 41.87 (geometry) | No clear effect |
| enable explicit thinking mode (ablation) | GLM-4.5V · DaTikZv3 | Pass@1 compile rate: 67.90 -> 62.92 | Hurt |
| enable explicit thinking mode (ablation) | Claude-Sonnet-4 · DaTikZv3 | Pass@1 compile rate: 84.87 -> 86.90 | No clear effect |
Closest papers by meaning
- Primitive Vision: Improving Diagram Understanding in MLLMs (ICML 2025 · similarity 0.87)
- DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ (NeurIPS 2024 · similarity 0.86)
- VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing (ICML 2026 · similarity 0.86)
- Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation (ICLR 2026 · similarity 0.86)
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.86)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
