Research map / Detection and segmentation / CNN features
Box2Poly: Memory-Efficient Polygon Prediction of Arbitrarily Shaped and Rotated Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Box2Poly builds a Sparse R-CNN-style cascade that turns learnable boxes into polygons via a Bézier centerline and refines them with a RoIAlign-like PolyAlign, using one instance-level feature per polygon for memory-efficient arbitrary-shape text detection.
Problem
Transformer-based polygon text detectors encode each boundary vertex with a separate query, which incurs high memory and inference cost and struggles to model relations among vertices of the same instance, especially for rotated and curved text.
Approach
Generate polygon proposals from learnable bounding boxes via a Bézier centerline expanded to the box scale, then iteratively refine polygons with scale- and translation-invariant offsets, PolyAlign vertex-level RoI features, and orientation-equivalent annotation, using one instance-level feature per instance.
Key results
- On TotalText and CTW1500, Box2Poly reaches F-scores of 88.35 and 88.13, about 0.7 points below DPText-DETR (89 and 88.8).
- On InverseText with pretraining, Box2Poly achieves F-score 88.7, surpassing DPText-DETR's 87.3 by 1.4 points (paper text says 1.6).
- Without pretraining on TotalText, VRAM drops from 17409MB to 8072MB and FPS rises from 13.6 to 19.6 versus DPText-DETR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use instance-level feature embedding and iterative box-to-polygon regression | Box2Poly versus DPText-DETR · TotalText | F-score: 89 -> 88.35 | Hurt |
| use instance-level feature embedding and iterative box-to-polygon regression | Box2Poly versus DPText-DETR · CTW1500 | F-score: 88.8 -> 88.13 | Hurt |
| use Box2Poly instead of DPText-DETR | text detection · InverseText | F-score: 87.3 -> 88.7 | Helped |
| use Box2Poly instead of DPText-DETR | text detection without pretraining · InverseText | F-score: 86.8 -> 87.41 | Helped |
| use instance-level feature embedding | Box2Poly versus DPText-DETR · TotalText, no pretraining | FPS: 13.6 -> 19.6 | Helped |
| use instance-level feature embedding | Box2Poly versus DPText-DETR · TotalText, no pretraining | VRAM: 17409MB -> 8072MB | Helped |
| replace box-edge sampling with the Box-to-Poly transformation using a Bézier intermediary (ablation) | Box2Poly · InverseText, 50K iterations | F-score: 53.61 -> 87.41 | Helped |
| use PolyAlign-Vertex instead of PolyAlign-Grid (ablation) | Box2Poly · InverseText, 50K iterations | F-score: 86.07 -> 87.41 | Helped |
| use PolyAlign-Vertex instead of PolyAlign-Bezier (ablation) | Box2Poly · InverseText, 50K iterations | F-score: 86.26 -> 87.41 | Helped |
| add orientation-equivalent annotation (ablation) | Box2Poly · InverseText, 50K iterations | F-score: 87.49 -> 87.41 | No clear effect |
| add orientation-equivalent annotation (ablation) | Box2Poly · InverseText, 30K iterations | F-score: 86.94 -> 87.34 | Helped |
| add polygon IoU loss (ablation) | Box2Poly · InverseText, 50K iterations | F-score: 87.41 -> 86.29 | Hurt |
Surprising findings
- Using a single instance-level feature embedding per polygon matches point-level-query methods that use 16 embeddings per instance, with only a small F-score gap.
Limitations
- Performance gap of about 0.7 F-score to DPText-DETR on TotalText and CTW1500.
- Lower recall on scenes dominated by small text; relies on a weak proposal generator.
- PolyAlign implemented with grid_sample is not optimized, so speed could improve with CUDA kernels.
Experimental setup
Datasets: SynthText 150K TotalText CTW1500 ICDAR19 MLT InverseText
Benchmarks: TotalText CTW1500 InverseText
Metrics: precision recall F-score FPS VRAM
Baselines: DPText-DETR TESTR-Polygon SwinTextSpotter DB I3CL ABCNet-v2 Boundary (end-to-end)
Builds on: Sparse R-CNN Cascade R-CNN DPText-DETR DeepSolo RoIAlign
Closest papers by meaning
- Text Spotting Transformers (CVPR 2022 · similarity 0.87)
- Geometry-Aware Scene Text Detection With Instance Transformation Network (CVPR 2018 · similarity 0.84)
- CentripetalText: An Efficient Text Instance Representation for Scene Text Detection (NeurIPS 2021 · similarity 0.84)
- LRANet: Towards Accurate and Efficient Scene Text Detection with Low-Rank Approximation Network (AAAI 2024 · similarity 0.83)
- LayoutFormer: Hierarchical Text Detection Towards Scene Text Understanding (CVPR 2024 · similarity 0.83)
- SRFormer: Text Detection Transformer with Incorporated Segmentation and Regression (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.