Research map / Vision-language and multimodal / Human and embodied modeling
MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSTAR, a box-free scene text retrieval model with progressive attention-shift vision embeddings, style-aware instructions, and multi-instance matching, plus a new MQTR benchmark for four query types.
Problem
Scene text retrieval normally requires costly bounding-box annotations, and existing methods handle only one query type (word or phrase), not diverse multi-query needs. Box-free VLMs tend to overlook small text.
Approach
Progressive vision embedding iteratively shifts attention away from salient regions (using a binarized, inverted cross-attention mask from the multimodal encoder) to surface small text, combined with style-aware text instructions and a Hungarian-based multi-instance matching module, trained without any box supervision.
Key results
- On MQTR, MSTAR achieves 66.78 average MAP, versus 58.11 for the best box-free baseline (BLIP-2 FT) and 54.09 for the best box-based TG-Bridge.
- On the six word retrieval datasets, MSTAR with re-ranking averages 84.18 MAP, and it reports 86.96 on Total-Text versus 79.18 for FDP.
- MSTAR runs at 14.2 FPS versus 6.7 FPS for TG-Bridge on the spotting-style evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MSTAR instead of TG-Bridge | MQTR · MQTR | MAP (%): average of 8.5% | Helped |
| Use MSTAR instead of FDP | MSTAR · Total-Text | MAP (%): 6.4% | Helped |
| Add style-aware instructions (ablation) | MSTAR · CTR | MAP (%): 52.87 -> 54.65 | Helped |
| Add style-aware instructions (ablation) | MSTAR · MQTR | MAP (%): 65.79 -> 66.15 | Helped |
| Add multi-instance matching (ablation) | MSTAR with instructions · STR | MAP (%): +2.19% | Helped |
| Add multi-instance matching (ablation) | MSTAR with instructions · CTW | MAP (%): +1.35% | Helped |
| Add multi-instance matching (ablation) | MSTAR with instructions · MQTR | MAP (%): -0.46% | Hurt |
| Add progressive vision embedding (ablation) | MSTAR with instructions and MIM · CTR | MAP (%): +4.36% | Helped |
| Add progressive vision embedding (ablation) | MSTAR with instructions and MIM · IC15 | MAP (%): +1.98% | Helped |
| Use TH+WS+CC instead of TH+CC for the attention mask (ablation) | MSTAR PVE · CTR | MAP (%): +4.37% | Helped |
| Use TH+WS+CC instead of TH+CC for the attention mask (ablation) | MSTAR PVE · Total-Text | MAP (%): +1.54% | Helped |
| Use TH+WS+CC instead of TH+CC for the attention mask (ablation) | MSTAR PVE · IC15 | MAP (%): +1.98% | Helped |
Surprising findings
- Adding multi-instance matching slightly lowers MQTR performance (66.15 to 65.69) even while improving STR and CTW.
- Off-the-shelf VLMs such as CLIP, BLIP, BLIP-2 and SigLIP achieve only 6.6-13.3% MAP on the small-text CTR dataset.
Limitations
- Lags behind box-based methods on extremely small and dense text such as CTR.
- Re-ranking reduces speed (6.9 FPS versus 14.2 FPS without it).
- Positional supervision is absent, which limits localization precision.
Experimental setup
Datasets: SynthText-900k TextCaps MLT-5K CTR PSTR HierText Total-Text CTW IC15 SVT STR
Benchmarks: MQTR SVT STR CTR Total-Text CTW IC15 Phrase-level Scene Text Retrieval dataset
Metrics: MAP FPS
Baselines: TDSL FDP TG-Bridge Deepsolo SPTSv2 BLIP-2 SigLIP ABCNet MaskTextSpotter Wen et al.
Builds on: BLIP-2 SigLIP Deepsolo
Closest papers by meaning
- Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts (ICML 2025 · similarity 0.88)
- CLIP is Almost All You Need: Towards Parameter-Efficient Scene Text Retrieval without OCR (CVPR 2025 · similarity 0.87)
- OTE: Exploring Accurate Scene Text Recognition Using One Token (CVPR 2024 · similarity 0.86)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.86)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.86)
- ColPali: Efficient Document Retrieval with Vision Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.