Research map / Vision-language and multimodal / Multimodal LLMs
Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DocMark, an InternVL-based MLLM pretrained on 3.8M markup-generation pairs (DocMark-Pile) and fine-tuned with two-round chain-of-thought markup-context generation on 624k DocMark-Instruct examples, reporting gains over InternVL2 and other MLLMs on document VQA benchmarks.
Problem
Visual document understanding with MLLMs suffers from short-answer fine-tuning data that causes hallucinations and weak modeling of layout and spatial relations across diverse document types.
Approach
Pretrain the MLLM to convert document images into markup languages (Markdown, LaTeX, HTML, JSON, TikZ, plain text), then fine-tune with a two-round CoT where the model first generates a markup-formatted context and then answers, with the CoT data built by having ChatGPT-3.5 extract answer-relevant context from ground-truth markup.
Key results
- DocMark-2B reaches 87.8 on DocVQA and 79.8 on ChartQA, above InternVL2-2B (86.0 and 76.4).
- DocMark-8B reaches 84.2 on ChartQA and 823 on OCRBench, above InternVL2-8B (82.2 and 794) on those benchmarks, while InternVL2-8B remains higher on DocVQA (90.8 vs 89.8) and InfoVQA (72.8 vs 68.3).
- On specialized tasks DocMark-8B scores 73.4 on HME100K versus 64.3 for CAN-ABM, and 68.5 on FUNSD versus 55.4 for Nougat.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DocMark-2B instead of InternVL2-2B | InternVL2-2B · AI2D | accuracy: 74.2 -> 82.5 | Helped |
| Use DocMark-8B instead of InternVL2-8B | InternVL2-8B · ChartQA | accuracy: 82.2 -> 84.2 | Helped |
| Use DocMark-8B instead of InternVL2-8B | InternVL2-8B · DocVQA_val | accuracy: 90.8 -> 89.8 | Hurt |
| Use DocMark-8B instead of InternVL2-8B | InternVL2-8B · OCRBench | score: 794 -> 823 | Helped |
| Add DocMark-CoT fine-tuning (ablation) | Qwen2-VL-2B · ChartQA | accuracy: 69.3 -> 77.1 | Helped |
| Add DocMark-CoT fine-tuning (ablation) | Qwen2-VL-2B · DocVQA | accuracy: 88.2 -> 89.8 | Helped |
| Add DocMark-CoT fine-tuning (ablation) | LLaVA-OneVision-7B · ChartQA | accuracy: 80.4 -> 83.4 | Helped |
| Add DocMark-CoT fine-tuning (ablation) | LLaVA-OneVision-7B · MathV | score: 22.0 -> 23.2 | Helped |
| Use CoT prompting without DocMark training (ablation) | Qwen2-VL-2B · ChartQA | accuracy: 73.5 -> 55.5 | Hurt |
| Use CoT prompting without DocMark training (ablation) | LLaVA-OneVision-7B · DocVQA | accuracy: 88.7 -> 87.2 | Hurt |
| Pretrain on DocMark-Pile and fine-tune on DocMark-Instruct (ablation) | DocMark-2B · Four document-related QA datasets | accuracy | Helped |
Surprising findings
- Vanilla Qwen2-VL-2B drops from 89.2 to 84.9 on DocVQA and from 73.5 to 55.5 on ChartQA when prompted with CoT without training.
- DocMark-8B is nearly on par with DT-CL 7B on DaTikZ (0.64 vs 0.68) and DT-DS 1.3B is lower (0.22), while DocMark-2B scores 0.39.
Experimental setup
Datasets: DocMark-Pile DocMark-Instruct LLaVA-558K TextVQA DocVQA InfographicsVQA ChartQA DVQA XFUND Geo170K ShareGPT4V PubTabNet
Benchmarks: TextVQA DocVQA InfoVQA ChartQA AI2D OCRBench VisualWebBench WebQA MathVision HME100K ChartQA-SE FUNSD SROIE
Metrics: accuracy AP@strict AP@slight AP@high image and code similarity average
Baselines: InternVL2-2B InternVL2-8B MiniCPM-V-2 Vary-toy Monkey TextMonkey DocOwl2 CogAgent Qwen2-VL-2B LLaVA-OneVision-7B OneChart Nougat
Builds on: InternVL2 InternViT LLaVA-OneVision
Closest papers by meaning
- Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation (NeurIPS 2026 · similarity 0.88)
- Harnessing Webpage UIs for Text-Rich Visual Understanding (ICLR 2025 · similarity 0.87)
- Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension (NeurIPS 2026 · similarity 0.87)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.87)
- DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming (AAAI 2025 · similarity 0.86)
- Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.