Research map / Vision-language and multimodal / Human and embodied modeling
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, Wei-Ying Ma
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| factorize caption meaning into objects, attributes, relations, and scenes, then align semantic components contrastively with image regions | UniVSE · MS-COCO 1K test image-to-sentence retrieval | R@1: 57.7 -> 64.3 | Improved |
| factorize caption meaning into objects, attributes, relations, and scenes, then align semantic components contrastively | UniVSE · MS-COCO 5K test | rsum: 299.1 -> 324.8 | Improved |
| add a bag-of-components coverage embedding to the sentence embedding (ablation) | UniVSE · MS-COCO adversarial image-to-sentence retrieval | total sum: 572.4 -> 603.0 | Improved |
| add object components to the coverage embedding (ablation) | UniVSE · MS-COCO object attack | R@1: 40.7 -> 42.9 | Improved |
| add attribute components to the coverage embedding (ablation) | UniVSE · MS-COCO attribute attack | R@1: 30.0 -> 37.4 | Improved |
| add relation components to the coverage embedding (ablation) | UniVSE · MS-COCO relation attack | R@1: 32.6 -> 42.8 | Improved |
| combine object, attribute, and relation components in the coverage embedding instead of using only one type (ablation) | UniVSE · MS-COCO adversarial image-to-sentence retrieval, total across attacks | total sum: 603.0 vs. 589.7 | Improved |
| train with contrastive examples from all semantic component types instead of object examples only (ablation) | UniVSE · MS-COCO unified text-to-image retrieval | mAP sum: 164.30 -> 170.42 | Improved |
| train with contrastive examples from all semantic component types instead of attribute examples only (ablation) | UniVSE · MS-COCO unified text-to-image retrieval | mAP sum: 155.81 -> 170.42 | Improved |
| train with contrastive examples from all semantic component types instead of relation examples only (ablation) | UniVSE · MS-COCO unified text-to-image retrieval | mAP sum: 168.94 -> 170.42 | Improved |
| use image matching scores to resolve ambiguous word dependencies | UniVSE · MS-COCO attribute dependency recovery | accuracy: 37.41 -> 64.82 | Improved |
| use image matching scores to resolve ambiguous relational dependencies | UniVSE · MS-COCO relational phrase dependency recovery | accuracy: 31.90 -> 62.69 | Improved |
Closest papers by meaning
- Image-Text Embedding Learning via Visual and Textual Semantic Reasoning (TPAMI 2022 · similarity 0.91)
- ViCo: Word Embeddings From Visual Co-Occurrences (ICCV 2019 · similarity 0.91)
- Towards Unsupervised Image Captioning With Shared Multimodal Embeddings (ICCV 2019 · similarity 0.90)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.90)
- Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment (AAAI 2025 · similarity 0.89)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
