Research map / Vision-language and multimodal / Human and embodied modeling
Learning to Embed Semantic Similarity for Joint Image-Text Retrieval
Noam Malali, Yosi Keller
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace text-only center loss with semantic centers on both modalities (ablation) | joint embedding · Flickr8K | annotation r@1: 6.6 -> 30.1 | Improved |
| replace text-only center loss with semantic centers (ablation) | joint embedding · Flickr30K | annotation r@1: 43.9 -> 46.7 | Improved |
| add cross-entropy losses alongside semantic centers (ablation) | joint embedding · Flickr8K/Flickr30K | annotation r@1: 30.1 -> 32.3 | No clear effect |
| add quantized center loss (1000 centers) (ablation) | ResNet152 embedding · Flickr30K, no random crops | annotation r@1: 52.9 -> 57.2 | Improved |
| quantize the center loss instead of unquantized (ablation) | ResNet152 embedding · Flickr30K, no random crops | annotation r@1: 55.8 -> 57.2 | Improved |
| add center loss with random crops (ablation) | ResNet152 embedding · Flickr30K, random crops | annotation r@1: 57.1 -> 60.1 | Improved |
| add center loss on VSRN backbone (ablation) | VSRN embedding · Flickr30K | annotation r@1: 68.1 -> 69.8 | Improved |
| vary number of semantic centers (ablation) | ResNet152 embedding · Flickr30K and MS-COCO 1K | accuracy: maximal difference 0.1% | No clear effect |
| replace learnt embedding with bert-large-uncased (ablation) | VSRN embedding · Flickr30K | annotation r@1: 69.8 -> 70.6 | Improved |
| add adaptive margin c=1.03 (ablation) | triplet loss · Flickr30K | average recall: 70.1 -> 70.7 | Improved |
| use adaptive margin of Wu instead of proposed (ablation) | triplet loss · Flickr30K | average recall: 70.7 -> 44.6 | Hurt |
| use adaptive margin of Li instead of proposed (ablation) | triplet loss · Flickr30K | average recall: 70.7 -> 55.9 | Hurt |
Closest papers by meaning
- Learning Deep Structure-Preserving Image-Text Embeddings (CVPR 2016 · similarity 0.89)
- Image-Text Embedding Learning via Visual and Textual Semantic Reasoning (TPAMI 2022 · similarity 0.89)
- Beyond Instance-Level Image Retrieval: Leveraging Captions to Learn a Global Visual Representation for Semantic Retrieval (CVPR 2017 · similarity 0.88)
- A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval (NeurIPS 2022 · similarity 0.88)
- Divide and Conquer the Embedding Space for Metric Learning (CVPR 2019 · similarity 0.88)
- Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
