Research map / Architectures and efficiency / Attention and transformers
DocMamba: Efficient Document Pre-training with State Space Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DocMamba, a purely state-space-model (bidirectional Mamba) encoder for visually-rich document understanding that uses a Segment-First Bidirectional Scan, achieving SOTA-level F1 on FUNSD/CORD/SROIE with linear cost and length extrapolation.
Problem
Transformer-based VrDU pre-trained models have quadratic self-attention cost, limiting efficiency and the ability to process long, text-dense documents; it is unexplored whether SSMs like Mamba can work for VrDU.
Approach
Adapt Mamba to 2-D document layouts by scanning tokens segment-by-segment (layout-analysis segments, each scanned in Y-then-X order) in a bidirectional manner, using text and 2-D layout embeddings without 1-D position embeddings, and pre-training with MLM using variable batch sizes by sequence length.
Key results
- DocMamba (135M, text+layout only) reaches 91.7 F1 on FUNSD, 97.0 on CORD and 96.8 on SROIE, above LayoutLMv3 (90.3, 96.6).
- On HRDoc at long inputs, DocMamba saves up to 88.3% GPU memory and runs 2.4x faster in FPS than LayoutLMv3 (input length 4096 for speed).
- Segment-First Bidirectional Scan gives 91.7 F1 on FUNSD versus 80.8 for word-first bidirectional scan.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DocMamba instead of LayoutLMv3 | Document understanding models · FUNSD | F1: 90.3 -> 91.7 | Helped |
| use DocMamba instead of LayoutLMv3 | Document understanding models · CORD | F1: 96.6 -> 97.0 | Helped |
| use DocMamba instead of LayoutLMv2 | Receipt understanding models · SROIE | F1: 96.3 -> 96.8 | Helped |
| use DocMamba instead of TILT | Receipt understanding models · SROIE | F1: 97.7 -> 96.8 | Hurt |
| replace SFBS with WFBS (ablation) | DocMamba · FUNSD | F1: 91.7 -> 80.8 | Hurt |
| limit pre-training input length to 512 instead of using variable lengths (ablation) | DocMamba · FUNSD | F1: 91.7 -> 90.8 | Hurt |
| limit pre-training input length to 512 instead of using variable lengths (ablation) | DocMamba · CORD | F1: 97.0 -> 96.6 | Hurt |
| limit pre-training input length to 512 instead of using variable lengths (ablation) | DocMamba · SROIE | F1: 96.8 -> 96.8 | No clear effect |
| increase test input length (ablation) | DocMamba_512 · HRDoc-Hard | F1 | Helped |
| use DocMamba instead of LayoutLMv3 | Document understanding models · HRDoc-Hard, input length 2,560 | GPU memory savings: 88.3% | Helped |
| use DocMamba instead of LayoutLMv3 | Document understanding models · HRDoc-Hard, input length 4096 | inference speed: 2.4 times | Helped |
Surprising findings
- A text+layout-only SSM model outperforms Transformer models that also use image modality, such as LayoutLMv3 and UDoc-style multimodal baselines.
- Vanilla Mamba pre-trained from scratch on the same data scores far below BERT-style models (58.3 F1 on FUNSD with ours data), showing the importance of the proposed scan and layout embeddings.
Limitations
- No image modality is used, so visual cues are not exploited.
- Relies on an off-the-shelf layout analysis system to extract segments and on PaddleOCR for words and boxes.
- Length extrapolation evaluated only on HRDoc semantic unit classification.
Experimental setup
Datasets: IIT-CDIP Test Collection 1.0 FUNSD CORD SROIE HRDoc
Benchmarks: FUNSD CORD SROIE HRDoc
Metrics: entity-level F1 GPU memory usage frames per second F1 vs input length
Baselines: BERT RoBERTa Mamba LayoutLM BROS LiLT SelfDoc UDoc TILT DocFormer XYLayoutLM LayoutLMv2
Builds on: Mamba Vim LayoutLMv3 BERT
Closest papers by meaning
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.86)
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.86)
- DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding (CVPR 2026 · similarity 0.85)
- DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.85)
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.84)
- URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding (AAAI 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.