Research map / Vision-language and multimodal / Human and embodied modeling
M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M2Doc, a pluggable multi-modal fusion approach that adds BERT-based textual grid features to visual detectors via early (pixel-level gated) and late (block-level IoU-matched) fusion for document layout analysis.
Problem
Document layout analysis methods rely mainly on visual features, and existing multi-modal layout models often underperform unimodal detectors on complex logical layout datasets.
Approach
Builds a 2D grid of BERT text embeddings aligned to the page image, uses a single shared backbone for visual and textual features, fuses them per scale with a gate (early fusion), and adds IoU-matched block-level text embeddings to RoI or decoder query features (late fusion), so any two-stage or end-to-end detector can be plugged in.
Key results
- DINO with M2Doc reaches 89.0 mAP on DocLayNet (up from 77.7), 69.9 mAP on M6Doc (+1.9), and 95.5 mAP on PubLayNet.
- Cascade Mask R-CNN with M2Doc improves from 75.6 to 86.7 mAP on DocLayNet and 59.7 to 61.8 on M6Doc.
- M2Doc plugged into Faster R-CNN, Mask R-CNN, and Deformable DETR gives gains of 2.6 to 2.9 mAP on M6Doc.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add M2Doc fusion to DINO | DINO · DocLayNet | mAP: 77.7 -> 89.0 | Helped |
| Add M2Doc fusion to Cascade Mask R-CNN | Cascade Mask R-CNN · DocLayNet | mAP: 75.6 -> 86.7 | Helped |
| Add M2Doc fusion to DINO | DINO · M6Doc | mAP: 68.0 -> 69.9 | Helped |
| Add M2Doc fusion to DINO | DINO · PubLayNet | mAP: 95.4 -> 95.5 | No clear effect |
| Add M2Doc fusion to DINO | DINO · PubLayNet | mAP: 95.7 -> 95.5 | Hurt |
| Add early fusion (ablation) | M2Doc with DINO · M6Doc | mAP: 68.5 -> 69.9 | Helped |
| Add late fusion (ablation) | M2Doc with DINO · M6Doc | mAP: 68.4 -> 69.9 | Helped |
| Add early and late fusion (ablation) | Cascade Mask R-CNN · M6Doc | mAP: 59.7 -> 61.8 | Helped |
| Add early fusion (ablation) | M2Doc with Cascade Mask R-CNN · M6Doc | mAP: 60.5 -> 61.8 | Helped |
| Replace gate with concatenation in early fusion (ablation) | M2Doc with DINO · M6Doc | mAP: 69.9 -> 69.4 | Hurt |
| Replace gate with sum in early fusion (ablation) | M2Doc with DINO · M6Doc | mAP: 69.9 -> 69.1 | Hurt |
| Replace sum with concatenation in late fusion (ablation) | M2Doc with DINO · M6Doc | mAP: 69.9 -> 69.2 | Hurt |
Surprising findings
- Gains on Table and Figure categories were smaller or negative with multi-modal fusion, attributed to overlaid text degrading boundary detection.
- On PubLayNet, DINO with M2Doc does not significantly outperform DINO alone (95.5 vs 95.4 mAP).
Limitations
- Marginal gains on simple physical layout datasets such as PubLayNet where unimodal methods suffice.
- Multi-modal fusion can degrade detection for categories with overlaid text such as Figure and Table.
- Does not solve the low-recall problem on M6Doc.
Experimental setup
Datasets: DocLayNet M6Doc PubLayNet
Benchmarks: DocLayNet M6Doc PubLayNet
Metrics: mAP@IoU[0.50:0.95] AP50 AP75 Recall
Baselines: VSR DINO Cascade Mask R-CNN TransDLANet SwinDocSegmenter LayoutLMv3 DiT StructTextv2 UDoc Mask R-CNN Faster R-CNN Deformable DETR
Builds on: DINO Cascade Mask R-CNN BERTgrid LAVT BERT
Closest papers by meaning
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.85)
- LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding (CVPR 2024 · similarity 0.85)
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.85)
- Towards Unified Multi-granularity Text Detection with Interactive Attention (ICML 2024 · similarity 0.85)
- Graph-based Document Structure Analysis (ICLR 2025 · similarity 0.84)
- A Simple yet Effective Layout Token in Large Language Models for Document Understanding (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.