Research map / Vision-language and multimodal / Human and embodied modeling
Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DL-CSVTR, a diversified-layout benchmark for Chinese scene text retrieval, and CSTR-CLIP, a two-stage CLIP-based model that uses full-image features with text-region-guided segmentation maps and random multi-granularity alignment.
Problem
Chinese scene text retrieval, where query text appears in diverse layouts (vertical, cross-line, partial), is poorly handled by existing methods that crop detected text regions and use single-granularity alignment; existing CSVTR benchmark mostly has horizontal, isolated queries.
Approach
Replace text-region cropping with full-image CLIP encoding guided by text segmentation maps (Text Position Convolution), train in two stages (single-granularity then multi-granularity with random alignment granularity processing and global feature fusion from the frozen original CLIP encoder).
Key results
- CSTR-CLIP reaches 88.57% mAP on CSVTR, improving over the prior best (Luo et al. with CLIP backbone, 80.74%) by about 18.82% in relative terms per the paper's claim, with 21.5 FPS.
- On DL-CSVTR, CSTR-CLIP achieves 84.44 (vertical), 65.56 (cross-line), and 61.85 (partial) mAP, the best across all three subsets.
- Stage 1 alone already beats cropped-region baselines on DL-CSVTR-V (74.50 vs 62.51 best baseline), but falls below CLIP on cross-line and partial layouts until stage 2 adds global features and RAGP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the strongest previous method with CSTR-CLIP | CSVTR retrieval · CSVTR | mAP (%): 80.74 -> 88.57 | Helped |
| Use CSTR-CLIP instead of the strongest baseline | DL-CSVTR-V retrieval · DL-CSVTR-V | mAP (%): 55.48 -> 84.44 | Helped |
| Use CSTR-CLIP instead of the strongest baseline | DL-CSVTR-CL retrieval · DL-CSVTR-CL | mAP (%): 54.43 -> 65.56 | Helped |
| Use CSTR-CLIP instead of the strongest baseline | DL-CSVTR-P retrieval · DL-CSVTR-P | mAP (%): 37.31 -> 61.85 | Helped |
| Add Text Position Convolution to the CLIP image encoder (ablation) | CSTR-CLIP with frozen visual encoder · CSVTR | mAP (%): 57.32 -> 59.50 | Helped |
| Add Text Position Convolution to the CLIP image encoder (ablation) | CSTR-CLIP with frozen visual encoder · DL-CSVTR-V | mAP (%): 55.48 -> 53.17 | Hurt |
| Add Text Position Convolution to the CLIP image encoder (ablation) | CSTR-CLIP with frozen visual encoder · DL-CSVTR-CL | mAP (%): 54.43 -> 42.15 | Hurt |
| Add Text Position Convolution to the CLIP image encoder (ablation) | CSTR-CLIP with frozen visual encoder · DL-CSVTR-P | mAP (%): 37.30 -> 30.70 | Hurt |
| Fine-tune the image encoder together with Text Position Convolution (ablation) | CSTR-CLIP Stage 1 · CSVTR | mAP (%): 59.50 -> 86.25 | Helped |
| Add global image features (ablation) | CSTR-CLIP Stage 1 · DL-CSVTR-V | mAP (%): 74.50 -> 83.91 | Helped |
| Add global image features (ablation) | CSTR-CLIP Stage 1 · DL-CSVTR-CL | mAP (%): 45.98 -> 57.82 | Helped |
| Add global image features (ablation) | CSTR-CLIP Stage 1 · DL-CSVTR-P | mAP (%): 33.25 -> 43.19 | Helped |
Surprising findings
- Stage 1 single-granularity training underperforms the plain CLIP baseline on DL-CSVTR-CL and DL-CSVTR-P, showing single-granularity alignment hurts diverse layouts.
- Plain CLIP at 640x640 outperforms cropped-region methods on cross-line and partial layouts, indicating cropping is a bottleneck.
Limitations
- Retrieval relies on PaddleOCR detection outputs at inference, so detector failures still affect results.
- DL-CSVTR is small (89 queries, 2,070 images) and used only for testing; data is partly described as a portion released in supplementary material.
Experimental setup
Datasets: SynthText-CH ReCTS CSVTR
Benchmarks: CSVTR DL-CSVTR-V DL-CSVTR-CL DL-CSVTR-P
Metrics: mAP FPS
Baselines: Mishra et al. TDSL VSTR Luo et al. CLIP (640x640)
Builds on: CLIP Chinese CLIP ReCLIP AlphaCLIP TDSL
Closest papers by meaning
- MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling (NeurIPS 2025 · similarity 0.88)
- CLIP is Almost All You Need: Towards Parameter-Efficient Scene Text Retrieval without OCR (CVPR 2025 · similarity 0.86)
- Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.84)
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.84)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.84)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.