Research map / Detection and segmentation / CNN features
Towards Unified Multi-granularity Text Detection with Interactive Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Detect Any Text (DAT), a single end-to-end DINO-based model that unifies word, line, paragraph and page text detection via an across-granularity interactive attention mask and a SAM-style prompt-based segmentation branch.
Problem
Existing OCR and document analysis systems train separate models for scene text detection, layout analysis, and page detection, which is computationally costly; prior unified work (HierText) uses cascaded bottom-up design requiring full multi-granularity labels.
Approach
An interactive across-granularity attention mask in the Transformer decoder lets group queries for word, line, paragraph and page exchange information across adjacent granularities, combined with a mixed-granularity loss weighting that allows training on datasets with incomplete granularity labels and a prompt-based mask decoder (SAM-style) for arbitrary-shaped text and page segmentation.
Key results
- DAT-SEG reaches 92.01 F-score on Total-Text and 91.72 on CTW1500, outperforming ESTextSpotter (90.0 and 90.0).
- DAT-SEG improves M6Doc layout mAP from 63.8 (M6Doc baseline) to 65.7.
- DAT-SEG achieves 98.65 mIoU on DIW, above DeepLabv3+ (98.61) and SAM (84.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add line-level granularity to word-level training (ablation) | DAT · ICDAR2015 | F-score: 81.02 -> 89.15; 8.13-point | Helped |
| add line-level granularity to word-level training (ablation) | DAT · Total-Text | F-score: 89.74 -> 89.61; 0.13-point | Hurt |
| add line-level granularity to word-level training (ablation) | DAT · CTW1500 | F-score: 2.49 points | Helped |
| add paragraph-level granularity to word + line training (ablation) | DAT · M6Doc | mAP: 62.0 to 67.8 (+5.8) | Helped |
| add page-level granularity to word + line + paragraph training (ablation) | DAT · DIW | mIoU: 98.67 to 98.65 | Hurt |
| remove interactive attention (ablation) | DAT · Total-Text | F-score: 90.98 -> 84.78 | Hurt |
| replace bidirectional interactive attention with bottom-up attention (ablation) | DAT · Total-Text | F-score: 87.94 to 90.32 (+2.38) | Helped |
| increase interaction factor from I=1 to I=2 (ablation) | DAT · Total-Text | F-score: 90.98 -> 91.06 | Helped |
| increase interaction factor from I=1 to I=2 (ablation) | DAT · CTW1500 | F-score: 2.96 | Hurt |
| increase interaction factor from I=1 to I=3 (ablation) | DAT · CTW1500 | F-score | Hurt |
| use DAT-SEG instead of DAT-DET | DAT · ICDAR2015 | F-score: 92.66 vs 91.42 | Hurt |
| use DAT-SEG instead of DAT-DET | DAT · Total-Text | F-score: 90.98 vs 92.01 | Helped |
Surprising findings
- Unrestricted cross-granularity attention (interaction factor I=3, no mask) sharply degrades performance across all granularities, nearly matching the model without interactive attention.
- The segmentation branch slightly lowers scores on quadrilateral-annotated ICDAR2015 and MSRA-TD500, attributed to annotation style mismatch.
Limitations
- Larger parameter count and GFLOPS (474 for full model) than single-task detectors, requiring more GPU memory and longer training.
- Limited use of multilingual datasets (MLT, ArT) because their per-language annotation granularity is ambiguous.
- Struggles with blurred small text and severe occlusion (failure cases).
Experimental setup
Datasets: ICDAR2015 Total-Text Curved SynthText ICDAR-MLT ArT CTW1500 MSRA-TD500 M6Doc DIW Doc3d
Benchmarks: ICDAR2015 Total-Text CTW1500 MSRA-TD500 M6Doc DIW
Metrics: precision recall F-score mAP mIoU
Baselines: SAM DeepLabv3+ M6Doc HierText SIR DPText-DETR UNITS ESTextSpotter
Builds on: DINO SAM HierText Deformable DETR Group-DETR
Closest papers by meaning
- Towards End-to-End Unified Scene Text Detection and Layout Analysis (CVPR 2022 · similarity 0.88)
- LayoutFormer: Hierarchical Text Detection Towards Scene Text Understanding (CVPR 2024 · similarity 0.87)
- Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysis (CVPR 2024 · similarity 0.87)
- M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis (AAAI 2024 · similarity 0.85)
- Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding (NeurIPS 2024 · similarity 0.84)
- Kernel Adaptive Convolution for Scene Text Detection via Distance Map Prediction (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.