Research map / Vision-language and multimodal / Visual question answering
Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQAMask, a stage-1 vision-language alignment that jointly trains VQA-based text parsing and an auxiliary mask generator on a 6M-sample mask dataset (MTMask6M), then SFT for Marten, an 8B document MLLM.
Problem
Existing document MLLM pre-training tasks (text transcription, visual text grounding) emphasize semantic alignment and lack spatial supervision, which may cause hallucinations when parsing visual text in document images.
Approach
Add an auxiliary mask generation module that takes cross-attention between visual tokens (queries) and answer/question tokens (keys) from an intermediate LLM layer, deconvolves them to image resolution, and supervises them with text-region masks obtained from a K-means binarization pipeline; the module is discarded at inference.
Key results
- Marten (8.1B) reaches 92.0 on DocVQA, 75.2 InfoVQA, 84.4 TabFact, and 80.4 SROIE, outperforming prior best by 0.4%, 0.4%, 4.0% and 10.1% respectively.
- Marten scores 820 on OCRBench, 26 points above InternVL2 and 18 above MiniMonkey.
- Adding the mask generation module reduces edit distance by about 0.06 on natural and document scenes in stage 1 and improves DocVQA by 2.5 to 4.7 points in stage 2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MGM during alignment training (ablation) | Marten with InternViT and InternLM2 · DocVQA | score: 89.52 -> 92.01 | Helped |
| add MGM during alignment training (ablation) | Marten with InternViT and InternLM2 · InfoVQA | score: 71.65 -> 75.21 | Helped |
| add MGM during alignment training (ablation) | Marten with InternViT and InternLM2 · ChartQA | score: 79.26 -> 81.72 | Helped |
| add MGM during alignment training (ablation) | Marten with InternViT and InternLM2 · TextVQA val | score: 71.25 -> 74.38 | Helped |
| add MGM during alignment training (ablation) | Marten with ViT and InternLM2 · ICDAR15 | Edit Distance: 0.339 -> 0.278(-0.061) | Helped |
| add MGM during alignment training (ablation) | Marten with ViT and InternLM2 · TotalText | Edit Distance: 0.451 -> 0.381(-0.07) | Helped |
| add MGM during alignment training (ablation) | Marten with ViT and InternLM2 · IIT-CDIP | Edit Distance: 0.271 -> 0.193(-0.078) | Helped |
| add MGM during alignment training (ablation) | Marten with ViT and InternLM2 · DocGenome | Edit Distance: 0.319 -> 0.211(-0.108) | Helped |
Surprising findings
- Marten underperforms InternVL2 on chart and scene-text benchmarks despite its document gains, which the authors attribute to InternVL2's much larger training data.
Limitations
- Weaker on chart and natural scene text benchmarks than InternVL2, which uses far more training data.
- Mask generation gains on text recognition are limited because recognition lacks layout information.
- Mask acquisition relies on PaddleOCR detection and K-means thresholding, which can be imperfect for complex document images.
Experimental setup
Datasets: MTMask6M DocStruct4M IIT-CDIP DocGenome ICDAR13 ICDAR15 SynthText TextOCR OpenVINO DocVQA InfoVQA DeepForm
Benchmarks: DocVQA InfoVQA DeepForm KLC ChartQA TextVQA WTQ TabFact FUNSD SROIE POIE OCRBench
Metrics: accuracy ANLS F1 edit distance OCRBench score
Baselines: InternVL2 DocOwl-1.5 Mini-Monkey TextHawk2 MM1.5 UReader TextMonkey Park et al.
Builds on: DocOwl 1.5 InternVL Swin Transformer CCD
Closest papers by meaning
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.90)
- FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering (NeurIPS 2025 · similarity 0.90)
- Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.89)
- GRAM: Global Reasoning for Multi-Page VQA (CVPR 2024 · similarity 0.89)
- DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA (ICML 2026 · similarity 0.89)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.