Research map / Vision-language and multimodal / Human and embodied modeling

Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Layout-aware document understanding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Read-Parse-Describe, a unified page-level document parsing benchmark (layout, reading order, OCR, captions, alt-text) and SARA, a frozen-VLM spatial adapter that reuses multi-layer visual features with crop-aware grid alignment.

Problem

Document understanding is evaluated as isolated components (layout, OCR, reading order, captioning), so no benchmark checks whether a single page parse is structurally faithful and whose visual elements carry accessible descriptions that agree with the rest of the parse.

Approach

A unified ordered-element output schema binding bbox, category, reading order, text/HTML, caption link and alt-text per element, plus SARA, a lightweight residual adapter that concatenates intermediate ViT layers, aligns them to crop-wise merged token grids, and adds them to the frozen VLM's visual tokens via a learnable gate.

Key results

• Fine-tuned Dolphin-v2 reaches 0.9826 F1, 0.9588 mIoU and 0.9857 reading-order score, the best localization and RO in Table 2.
• Qwen3.5-2B + SARA (83.99M trainable params) raises F1 from 0.7336 (full FT) to 0.9189 and RO from 0.9628 to 0.9774 while the base VLM stays frozen.
• Fine-tuned DeepSeek-OCR achieves the highest METEOR (0.3998) and GPT-5.6 the highest zero-shot BERTScore (0.9004).

Surprising findings

• SARA's gains are concentrated in structural metrics; its alt-text scores are mixed, and on Gemma4-E4B it slightly decreases RO and description metrics versus full fine-tuning.
• At 10% of training data SARA trails full fine-tuning on all structural metrics, with its advantage emerging only at 50% and 100%.

Limitations

• Benchmark limited to open-access ACM scientific PDFs and mainly the first eight pages, underrepresenting later pages and references.
• Alt-text annotations rely on LLM (Gemini) generation and scoring with human refinement; closed-source pretraining exposure cannot be ruled out.
• Generated alt-text should not be treated as deployment-ready accessibility metadata.

Experimental setup

Datasets: Read-Parse-Describe

Benchmarks: Read-Parse-Describe

Metrics: F1 (IoU>=0.5) mIoU OCR accuracy reading-order score METEOR BERTScore

Baselines: GOT-OCR 2.0 Nougat Dolphin-v2 MonkeyOCR VGT PaddleOCR-VL PP-StructureV3 MinerU DeepSeek-OCR DeepSeek-OCR2 Gemma4-E4B Qwen3.5-2B full fine-tuning

Builds on: LayoutLMv3 Qwen2.5-VL PubLayNet DocLayNet LoRA

Closest papers by meaning

• Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding (CVPR 2025 · similarity 0.88)
• Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training (CVPR 2026 · similarity 0.88)
• MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing (NeurIPS 2026 · similarity 0.88)
• Boosting Document Parsing Efficiency and Performance with Coarse-to-Fine Visual Processing (CVPR 2026 · similarity 0.87)
• DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming (AAAI 2025 · similarity 0.87)
• ColPali: Efficient Document Retrieval with Vision Language Models (ICLR 2025 · similarity 0.86)