Research map / Vision-language and multimodal / Human and embodied modeling
Beyond Instance-Level Image Retrieval: Leveraging Captions to Learn a Global Visual Representation for Semantic Retrieval
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the ResNet R-MAC visual representation using caption-derived semantic triplets | ResNet-101 R-MAC · Human triplet-ranking study | agreement score: 64.0 -> 76.9 | Helped |
| Use human-caption tf-idf similarity instead of ImageNet-pretrained visual similarity | Semantic ranking proxy · Human triplet-ranking study | agreement score: 64.0 -> 76.3 | Helped |
| Use object-category annotation histograms as the semantic proxy | Semantic ranking proxy · Human triplet-ranking study | agreement score: 64.0 -> 63.4 | Hurt |
| Use generated captions instead of human-annotated captions | Caption tf-idf semantic proxy · Human triplet-ranking study | agreement score: 76.3 -> 62.5 | Hurt |
| Train a joint visual-text embedding rather than visual retrieval alone (ablation) | Proposed visual embedding · Human triplet-ranking study | agreement score: 76.9 -> 77.2 | Helped |
| Train a joint visual-text embedding rather than visual retrieval alone (ablation) | Proposed visual embedding · NDCG AUC | NDCG AUC: 70.1 -> 68.8 | Hurt |
| Train a joint visual-text embedding rather than visual retrieval alone (ablation) | Proposed visual embedding · PCC AUC | PCC AUC: 20.7 -> 21.1 | Helped |
| Add text to the image query using the joint embedding (ablation) | Proposed (V+T, V) · Human triplet-ranking study | agreement score: 77.2 -> 78.6 | Helped |
| Add text to the image query using the joint embedding (ablation) | Proposed (V+T, V) · NDCG AUC | NDCG AUC: 68.8 -> 74.4 | Helped |
| Add text to the image query using the joint embedding (ablation) | Proposed (V+T, V) · PCC AUC | PCC AUC: 21.1 -> 22.5 | Helped |
| Optimize only the joint-embedding loss instead of directly optimizing visual retrieval | WSABIE-like embedding · Human triplet-ranking study | agreement score: 67.8 -> 76.9 | Hurt |
Closest papers by meaning
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.90)
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.89)
- SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval (ICML 2025 · similarity 0.88)
- MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions (ICML 2024 · similarity 0.88)
- Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval (NeurIPS 2025 · similarity 0.88)
- Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.