Research map / Vision-language and multimodal / Multimodal LLMs
ProcTag: Process Tagging for Assessing the Efficacy of Document Instruction Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProcTag, which assesses document instruction data by tagging the pseudo-code execution process generated by GPT-3.5 (with a layout-aware DocLayPrompt representation), and samples data by tag complexity and diversity.
Problem
Existing instruction-data evaluation methods look only at instruction text, so they cannot distinguish document VQA instructions whose execution differs across document layouts, leading to poor data selection for training LLMs/MLLMs on document VQA.
Approach
Model the instruction execution process (CoT plus pseudo-code generated by GPT-3.5 over a layout-aware document representation) instead of instruction text, derive tags from function names, and use tag-based complexity/diversity to select data; DocLayPrompt integrates OCR and layout detection into a semi-structured text prompt.
Key results
- Sampling with ProcTag reaches 100% efficacy using only 30.5% of the generated document instructions, per the abstract.
- On DocVQA, the DocLayPrompt representation raises GPT-3.5 direct-prompting ANLS to 73.84 vs 71.03 for SpatialFormat and 63.66 for PlainText.
- Tag quality: precision of 96% (GPT-4) and 92% (human); consistency 80% (GPT-4) and 88% (human).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace plain-text / SpatialFormat document prompt with DocLayPrompt (layout-aware semi-structured representation) (ablation) | GPT-3.5 direct prompting · DocVQA test | ANLS: +2.81 over SpatialFormat | Helped |
| Use DocLayPrompt instead of PlainText or SpatialFormat document representation (ablation) | Qwen fine-tuning · DocVQA | ANLS: 81.80 vs 80.93 (SpatialFormat) and 77.14 (PlainText) | Helped |
| Generate process as pseudo-code tags instead of natural-language tags (ablation) | ProcTag sampling · DocVQA | sampling performance vs random | Helped |
| Sample training data with ProcTag tags instead of InsTag or random sampling | Qwen and Qwen-VL on DocVQA and four generated datasets · DocVQA and generated datasets (DocILE, RVL-CDIP, PubLayNet, PubTabNet) | ANLS | Helped |
| Sample with InsTag (text-only tags) vs random sampling | Qwen and Qwen-VL on generated datasets · generated document instruction datasets | ANLS | No clear effect |
| Sample with ProcTag, reaching 100% efficacy with only a fraction of data | generated document instruction datasets · data ratio analysis | data efficacy (performance relative to best): 30.5% of instructions reach 100% efficacy | Helped |
| Increase tag coverage rate in the sampled data (ablation) | ProcTag-sampled data · DocVQA/generated datasets | ANLS | Helped |
| Use ProcTag on more diverse document datasets (e.g., DocILE) vs less diverse ones (PubTabNet, PubLayNet) | ProcTag sampling · generated datasets | ANLS gain | Mixed |
| Use chain-of-thought step-by-step reasoning with ProcTag process before answering (ablation) | GPT-3.5 on DocVQA · DocVQA | ANLS: 73.84 vs 78.68 reported as increase from direct reasoning | Helped |
Surprising findings
- InsTag-based and random sampling show similar performance trends, suggesting text-only tagging cannot distinguish document instructions.
Limitations
- Relies on text-only document representation with GPT-3.5, losing visual information; not applicable to visually rich datasets such as InfographicVQA.
- Gains on PubTabNet and PubLayNet are weaker because these datasets are narrow in layout diversity.
- Only Qwen-7B-Chat and Qwen-VL-Chat tested; the Qwen-VL evaluation on DocVQA was avoided because it was likely trained on it.
Experimental setup
Datasets: DocVQA RVL-CDIP DocILE PubLayNet PubTabNet
Benchmarks: DocVQA RVL-CDIP DocILE PubLayNet PubTabNet
Metrics: ANLS tag precision tag consistency Cohen's kappa data efficacy
Baselines: InsTag random sampling IFD SpatialFormat PlainText
Builds on: InsTag LayoutLLM LATIN-Prompt SpatialFormat Instruction-Following Difficulty
Closest papers by meaning
- LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding (CVPR 2024 · similarity 0.87)
- InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · similarity 0.86)
- Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models (ICLR 2025 · similarity 0.85)
- DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA (ICML 2026 · similarity 0.85)
- #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models (ICLR 2024 · similarity 0.84)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.