Research map / Vision-language and multimodal / Human and embodied modeling
Heterogeneous Prompt-Guided Entity Inferring and Distilling for Scene-Text Aware Cross-Modal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOPID, a scene-text aware cross-modal retrieval network that uses heterogeneous (visual and text) prompts with a discriminative entity inferring module and a slot-attention-based perceptive entity distilling module.
Problem
Scene-text aware cross-modal retrieval suffers from semantic ambiguity of isolated scene text and fails to align heterogeneous scene-text concepts between images and captions.
Approach
Align scene text across image and caption using a heterogeneous prompt (a text prompt of caption-predicted scene-text words from a fine-tuned BERT, and a visual prompt marking OCR regions in the image), and distill property-centric scene-text representations from multi-modal OCR features (semantic, appearance, position) via slot attention.
Key results
- On CTC-1K, HOPID reaches RSUM 442.4 versus 398.3 for ViSTA-S, the prior best reported in the table.
- On TextCaps, HOPID reaches RSUM 435.4, outperforming VISTA (328.6) and Dual Encoder (387.1).
- Ablation on CTC-1K shows the visual prompt and text prompt together with PED raise RSUM from 360.9 (baseline) to 442.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HOPID instead of ViSTA-S | Scene-text-aware cross-modal retrieval · CTC-1K | RSUM: 398.3 -> 442.4 | Helped |
| Use HOPID instead of ViSTA-S | Scene-text-aware cross-modal retrieval · CTC-5K | RSUM: 273.5 -> 328.6 | Helped |
| Use HOPID instead of Dual Encoder | Scene-text-aware cross-modal retrieval · TextCaps | RSUM: 387.1 -> 435.4 | Helped |
| Add PED to Baseline (ablation) | HOPID · CTC-1K | RSUM: 360.9 -> 365.0 | Helped |
| Add PED to Baseline (ablation) | HOPID · CTC-5K | RSUM: 239.8 -> 240.8 | Helped |
| Add text prompt to Baseline (ablation) | HOPID · CTC-1K | RSUM: 360.9 -> 374.8 | Helped |
| Add text prompt to Baseline (ablation) | HOPID · CTC-5K | RSUM: 239.8 -> 242.5 | Helped |
| Add visual prompt to Baseline (ablation) | HOPID · CTC-1K | RSUM: 360.9 -> 394.9 | Helped |
| Add visual prompt to Baseline (ablation) | HOPID · CTC-5K | RSUM: 239.8 -> 277.9 | Helped |
| Replace the visual-prompt image feature with the original image feature (ablation) | HOPID with text prompt · CTC-1K | RSUM: 407.4 -> 396.1 | Hurt |
| Add PED to the model with visual and text prompts (ablation) | HOPID · CTC-1K | RSUM: 407.4 -> 442.3 | Helped |
| Add PED to the model with visual and text prompts (ablation) | HOPID · CTC-5K | RSUM: 285.1 -> 328.6 | Helped |
Surprising findings
- A 'toxic' text prompt with no scene-text words (template only) performs worse than having no text prompt at all, showing the gain comes from the predicted scene-text words.
Limitations
- Relies on an external OCR tool (paddleOCR) and limited to top-20 OCR tokens.
- Performance on CTC relies on CLIP, which the authors note performs poorly on CTC without the method.
Experimental setup
Datasets: COCO-Text Captioned (CTC) TextCaps TextCaps-OCR
Benchmarks: CTC-1K CTC-5K TextCaps
Metrics: Recall@1 Recall@5 Recall@10 RSUM
Baselines: SCAN VSRN STARNet Dise-OCR ViSTA-S HM VISTA Dual Encoder
Builds on: slot attention ViSTA StacMR CLIP
Closest papers by meaning
- Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval (AAAI 2025 · similarity 0.87)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.87)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.86)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.86)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.