Research map / Detection and segmentation / Open-set and few-shot detection

DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learning

CVPR 2025 · Detection and segmentation › Open-set and few-shot detection · approach: Interactive and referring segmentation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

DocSAM is a transformer-based unified model that segments diverse document images (layout analysis, multi-granularity text, table structure) by decomposing tasks into instance and semantic queries, trained jointly on ~48 heterogeneous datasets.

Problem

Document image segmentation tasks (layout analysis, text detection, table structure recognition) are handled by separate specialized models, which limits generalization and wastes compute and storage.

Approach

Embed dataset class names with Sentence-BERT into semantic queries that act as class prototypes alongside learnable instance queries; instance classes come from dot-product similarity with semantic queries, turning the closed-set classifier into an open-set one so one model can train on heterogeneous datasets.

Key results

• On M6Doc, DocSAM reaches 0.661 instance mAP, below DAT (0.657)-level only partially; it outperforms Deformable DETR (0.556) and TransDLANet (0.638) but not the DAT-reported 0.712 object mAP.
• Ablation: removing masked attention drops instance mAP from 0.3804 to 0.2322 averaged over 15 datasets; SimMIM pretraining collapses it to 0.1002.
• On CTW1500 and Total-Text, DocSAM (F 0.842 and 0.770) trails dedicated scene text spotters such as DAT-SEG (0.917 and 0.920), which the authors attribute to scene text diversity.

Measured effects reported in the paper

Surprising findings

• SimMIM vision-encoder pretraining on all 48 datasets significantly degraded performance (instance mAP 0.1002 vs 0.3804).
• The model underperforms on logical layout analysis (D4LA, DocLayNet, M6Doc, SCUT-CAB-logical), which the authors attribute to its single-modal design.

Limitations

• Single-modal model underperforms on logical layout analysis requiring multimodal cues.
• Weak on scene text detection and complex scenarios; long training time.
• Performance gaps remain versus specialized state-of-the-art on several benchmarks.

Experimental setup

Datasets: PubLayNet M6Doc D4LA DocLayNet SCUT-CAB HJDataset CASIA-HWDB SCUT-HCCDoc FinTabNet PubTabNet TableBank CTW1500

Benchmarks: M6Doc SCUT-CAB CTW1500 Total-Text D4LA DocLayNet PubLayNet FinTabNet PubTabNet TableBank IIIT-AR-13K CHDAC-2022

Metrics: mAP mAPb mAF mIoU AP50 AP75 precision recall F-score

Baselines: Faster R-CNN Mask R-CNN Deformable DETR ISTR TransDLANet DAT SCNet VSR HierText DPText-DETR ESTextSpotter

Builds on: Mask2Former Deformable DETR SAM Sentence-BERT SimMIM

Closest papers by meaning

• Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark (CVPR 2025 · similarity 0.83)
• Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation (CVPR 2024 · similarity 0.83)
• DocMamba: Efficient Document Pre-training with State Space Model (AAAI 2025 · similarity 0.83)
• Graph-based Document Structure Analysis (ICLR 2025 · similarity 0.82)
• DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.82)
• M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis (AAAI 2024 · similarity 0.82)