Research map / Detection and segmentation / CNN features
LayoutFormer: Hierarchical Text Detection Towards Scene Text Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LayoutFormer, a Transformer-based hierarchical scene text detector that jointly detects word-, line-, and paragraph-level text and their layouts, using a Prior Location Sampler to cut decoder cost.
Problem
Scene text detectors typically handle only one level of text (word, line, or paragraph) and ignore inter-level relationships needed for scene text understanding; Transformer decoders on high-resolution multi-scale features are also computationally expensive.
Approach
Use line as the detection unit with LineDecoder, derive words via a WordDecoder (per-line word queries) and paragraphs via a ParaDecoder (cross-attention over line queries), supervised directly by multi-level boxes; a Prior Location Sampler selects Top-K foreground features across multi-scale maps with weighted aggregation to update queries efficiently.
Key results
- Line-paragraph two-level detection on HierText test: 53.76 paragraph PQ and 62.37 line PQ with Nline=384, beating Unified Detector (53.60 paragraph PQ, 62.23 line PQ).
- Word-line-paragraph three-level detection on HierText test achieves PQ of 49.24 (word), 46.21 (line), 42.97 (paragraph) at 3.0 FPS.
- Line detection on MSRA-TD500 reaches 90.10 F-measure, outperforming PCR (87.0) and Unified Detector (87.70); Total-Text word F-measure is 87.12, slightly below FSG and Unified Detector.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use line rather than word or paragraph as the detection unit (ablation) | LayoutFormer · HierText val | PQ: 53.09 -> 57.48; 40.34 -> 57.48 | Helped |
| use weighted Top-K sampling on C1-C4 instead of masked attention (ablation) | LayoutFormer · Total-Text val | F-measure: 86.21 -> 87.12 | Helped |
| use weighted Top-K sampling on C1-C4 instead of masked attention (ablation) | LayoutFormer · HierText val line-level | PQ: 56.74 -> 57.48 | Helped |
| increase Top-K from 16 to 32 (ablation) | LayoutFormer Prior Location Sampler · Total-Text val | F-measure: 86.14 -> 87.12 | Helped |
| increase Top-K from 32 to 64 (ablation) | LayoutFormer Prior Location Sampler · HierText val line-level | PQ: 57.48 -> 58.86 | Helped |
| use sampled Fl instead of Xline for WordDecoder image features (ablation) | LayoutFormer WordDecoder · HierText val | Word PQ: 44.02 -> 50.29 | Helped |
| use sampled Fl instead of Xline for WordDecoder image features (ablation) | LayoutFormer WordDecoder · HierText val | Line PQ: 36.35 -> 48.60 | Helped |
| set WordDecoder depth L3 to 4 instead of 1 (ablation) | LayoutFormer WordDecoder · HierText val | Word PQ: 50.29 -> 51.07 | Helped |
| set WordDecoder depth L3 to 4 instead of 1 (ablation) | LayoutFormer WordDecoder · HierText val | Line PQ: 48.60 -> 44.05 | Hurt |
| reduce Nper line from 10 to 6 (ablation) | LayoutFormer WordDecoder · HierText val | Word PQ: 50.29 -> 46.21 | Hurt |
| use Xline instead of sampled Fl for ParaDecoder image features (ablation) | LayoutFormer ParaDecoder · HierText val | Paragraph PQ: 46.15 -> 51.98 | Helped |
| use LayoutFormer rather than Max-DeepLab-Cluster | HierText line-paragraph detection · HierText test | Line PQ: +0.14% | Helped |
Surprising findings
- Using the entire multi-scale feature map was not required; sampling only 32 Top-K foreground features per query retained or improved accuracy while reducing GPU memory.
- Paragraph-level detection performance is much lower than word and line levels (Table 1: PQ 40.34 vs 53.09 and 57.48), so the authors avoid a top-down paragraph-first approach.
Limitations
- Inference speed of 3.0 FPS for the three-level model.
- Word-level Total-Text results are slightly behind FSG and Unified Detector, which use more epochs or larger training data.
- No prior baselines exist for word-line-paragraph three-level detection, so comparison is limited to self-reported numbers.
Experimental setup
Datasets: HierText MSRA-TD500 CTW1500 Total-Text SynthText
Benchmarks: HierText MSRA-TD500 CTW1500 Total-Text
Metrics: Panoptic Quality (PQ) Recall Precision F-measure
Baselines: Unified Detector FSG DPText-DETR PCR DB PAN GCP API GCN-PP Mask-RCNN-Cluster Max-DeepLab-Cluster TextSnake CUTE
Builds on: Mask2Former DETR Deformable DETR Unified Detector
Closest papers by meaning
- Towards End-to-End Unified Scene Text Detection and Layout Analysis (CVPR 2022 · similarity 0.91)
- Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysis (CVPR 2024 · similarity 0.90)
- Towards Unified Multi-granularity Text Detection with Interactive Attention (ICML 2024 · similarity 0.87)
- Arbitrary Reading Order Scene Text Spotter with Local Semantics Guidance (AAAI 2025 · similarity 0.85)
- Kernel Adaptive Convolution for Scene Text Detection via Distance Map Prediction (CVPR 2024 · similarity 0.85)
- LRANet: Towards Accurate and Efficient Scene Text Detection with Low-Rank Approximation Network (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.