Research map / Vision-language and multimodal / Human and embodied modeling
Learning Semantic Concepts and Order for Image and Sentence Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 10 crops rather than 1 to extract global context (ablation) | context-only matching · Flickr30k and MSCOCO | mR: 48.0 -> 53.8; 63.8 -> 67.9 | Helped |
| Use generated caption sentences as image representations (ablation) | ctx + sen · Flickr30k and MSCOCO | mR: 53.8 -> 42.8; 67.9 -> 64.0 | Hurt |
| Add sentence-generation supervision for semantic order learning (ablation) | context-only matching · Flickr30k and MSCOCO | mR: 53.8 -> 55.3; 67.9 -> 68.9 | Helped |
| Add scheduled sampling during generation supervision (ablation) | ctx + gen · Flickr30k and MSCOCO | mR: 55.3 -> 54.6; 68.9 -> 68.4 | No clear effect |
| Share the word embeddings for sentence representation and generation (ablation) | ctx + gen · Flickr30k and MSCOCO | mR: 55.3 -> 54.3; 68.9 -> 68.7 | No clear effect |
| Use semantic concepts rather than context alone (ablation) | image-sentence matching · Flickr30k and MSCOCO | mR: 53.8 -> 50.8; 67.9 -> 76.8 | Mixed |
| Add generation supervision to semantic concepts without global context (ablation) | cnp · Flickr30k and MSCOCO | mR: 50.8 -> 51.8; 76.8 -> 78.4 | Helped |
| Add global context and combine it with concepts by summation (ablation) | semantic-concept matching · Flickr30k and MSCOCO | mR: 50.8 -> 59.7; 76.8 -> 79.8 | Helped |
| Replace summation with gated fusion of concepts and context (ablation) | cnp + ctx · Flickr30k and MSCOCO | mR: 59.7 -> 61.1; 79.8 -> 80.8 | Helped |
| Add generation supervision to gated concept-context matching (ablation) | cnp + ctx · Flickr30k and MSCOCO | mR: 61.1 -> 62.3; 80.8 -> 81.8 | Helped |
| Remove global context from concept-plus-generation matching (ablation) | cnp + ctx + gen · Flickr30k and MSCOCO | mR: 62.3 -> 51.8; 81.8 -> 78.4 | Hurt |
| Remove semantic concepts from context-plus-generation matching (ablation) | cnp + ctx + gen · Flickr30k and MSCOCO | mR: 62.3 -> 55.3; 81.8 -> 68.9 | Hurt |
Closest papers by meaning
- Comprehending and Ordering Semantics for Image Captioning (CVPR 2022 · similarity 0.90)
- End-to-End Convolutional Semantic Embeddings (CVPR 2018 · similarity 0.89)
- Instance-Aware Image and Sentence Matching With Selective Multimodal LSTM (CVPR 2017 · similarity 0.88)
- Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations (NeurIPS 2019 · similarity 0.87)
- Expressing Objects Just Like Words: Recurrent Visual Embedding for Image-Text Matching (AAAI 2020 · similarity 0.87)
- Composing Object Relations and Attributes for Image-Text Matching (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.