Research map / Vision-language and multimodal / CLIP and zero-shot
CLIP is Almost All You Need: Towards Parameter-Efficient Scene Text Retrieval without OCR
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAYN, a strictly OCR-free scene text retrieval method that adapts CLIP with a lightweight visual position adapter, visual context dropout, and parameter-free cross-attention reranking, training under 0.5M extra parameters.
Problem
Scene text retrieval (finding all images containing a query string) typically relies on OCR-style text spotting or localization, which adds pipeline complexity, accumulated errors, and heavy parameter tuning costs.
Approach
Use CLIP directly for OCR-free scene text retrieval: a visual position adapter supplies positional information for high input resolutions, visual context dropout yields locally aligned tokens, and a parameter-free cross-attention (PFCA) transfers global image-text contrastive relations to token-level features for linear-classifier reranking.
Key results
- CAYN-RN50x16 reaches 92.46% SVT, 89.49% IIIT-STR and 85.98% TTR mAP, beating FDP-L by 2.83 points on SVT and 6.80 on TTR.
- CAYN adds only 0.20M to 0.45M tuned parameters, versus 20M+ for FDP-style methods.
- CAYN-RN50 runs at 80.13 FPS, about 3x faster than FDP-L at 11.82 FPS on the same metric table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CAYN to CLIP RN50x16 | CLIP RN50x16 · SVT | mAP: 81.60 -> 92.46 | Helped |
| Add CAYN to CLIP RN50x16 | CLIP RN50x16 · IIIT-STR | mAP: 64.04 -> 89.49 | Helped |
| Add CAYN to CLIP RN50x16 | CLIP RN50x16 · TTR | mAP: 44.26 -> 85.98 | Helped |
| Add VPA to the CLIP retrieval baseline (ablation) | CLIP RN50 · SVT | mAP: 72.81 -> 81.98 | Helped |
| Add VPA to the CLIP retrieval baseline (ablation) | CLIP RN50 · TTR | mAP: 41.87 -> 66.54 | Helped |
| Add reranking with VCD and PFCA (ablation) | VPA retrieval baseline (RN50) · SVT | mAP: 82.83 -> 85.05 | Helped |
| Add reranking with VCD and PFCA (ablation) | VPA retrieval baseline (RN50) · TTR | mAP: 68.02 -> 74.04 | Helped |
| Add visual context dropout (ablation) | CAYN (RN50) · SVT | mAP: +2.22% | Helped |
| Add visual context dropout (ablation) | CAYN (RN50) · TTR | mAP: +6.02% | Helped |
| Use PFCA instead of multi-head cross-attention (ablation) | CAYN (RN50) · SVT | mAP: +5.96% | Helped |
| Use PFCA instead of multi-head cross-attention (ablation) | CAYN (RN50) · TTR | mAP: +5.94% | Helped |
| Keep the residual connection in VPA (ablation) | VPA (RN50) · SVT | mAP: 50.54 -> 81.98 | Helped |
Surprising findings
- Simply interpolating CLIP visual position embeddings at larger resolutions gives a strong zero-shot baseline, with ResNet backbones generalizing better than ViT.
- A quotation-mark prompt template ('"{query}"') outperformed other hand-crafted prompts for RN50.
Limitations
- ViT-based CLIP generalizes poorly to larger input resolutions without image splitting.
- Reranking only applies to the top-K (K=32) retrieved images.
Experimental setup
Datasets: MLT5k
Benchmarks: SVT IIIT-STR TTR
Metrics: mAP FPS
Baselines: FDP-L FDP-B FDP-S Wen et al. TDSL Mask TextSpotter v3 ABCNet EXCLIP CLIP-Adapter CoOP
Builds on: CLIP CLIP-Adapter FDP CoOP VPT
Closest papers by meaning
- OTE: Exploring Accurate Scene Text Recognition Using One Token (CVPR 2024 · similarity 0.88)
- MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling (NeurIPS 2025 · similarity 0.87)
- ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining (AAAI 2024 · similarity 0.87)
- Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts (ICML 2025 · similarity 0.86)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.86)
- Gradient-based Visual Explanation for Transformer-based CLIP (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.