Research map / Vision-language and multimodal / Human and embodied modeling
Image2Struct: Benchmarking Structure Extraction for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Image2Struct, a round-trip, fully automatic benchmark where VLMs generate structured code (LaTeX, HTML, sheet music) from images, which is rendered and compared to the input image using image similarity metrics.
Problem
Evaluating VLM outputs for structure extraction (e.g., image to LaTeX/HTML) is hard because many valid structures exist and human grading is costly, while MCQA cannot capture real-world code generation tasks.
Approach
Round-trip evaluation: the VLM generates structure from an image, a task-specific renderer produces an image from it, and the rendered image is scored against the original with automatic image metrics (including newly proposed CIS and EMS), using fresh data continually downloaded from online communities.
Key results
- Evaluated 14 prominent VLMs; scores vary widely, differentiating model performance.
- Best score is 0.402 on sheet music versus 0.830 on LaTeX equations, indicating varying task difficulty across domains.
- CIS and EMS metrics show high correlation with Levenshtein edit distance to ground-truth structures.
Limitations
- Relies on fresh data from online communities, so data distribution may shift over time.
- Round-trip similarity may not fully capture semantic correctness of generated structure.
Experimental setup
Datasets: fresh data downloaded from active online communities (webpages, LaTeX, musical scores)
Benchmarks: Image2Struct
Metrics: pixel similarity cosine similarity between Inception vectors (CIS) learned perceptual image patch similarity (LPIPS) structural similarity index measure (SSIM) earth mover similarity (EMS)
Builds on: Levenshtein edit distance Inception LPIPS SSIM
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.88)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.88)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.88)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.