Research map / Vision-language and multimodal / Human and embodied modeling
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add image-conditioned GRU initialization with transposed weight sharing (ablation) | Model A · Gold RP10K | average precision: 0.773 -> 0.843; 7.0% | Helped |
| Add image-conditioned GRU initialization with transposed weight sharing (ablation) | Model A · RP10M | average precision: 0.681 -> 0.725; 4.4% | Helped |
| Add visual information through GRU initialization, without weight sharing (ablation) | Pure text RNN · Gold RP10K | average precision: 0.748 -> 0.773 | Helped |
| Add visual information through GRU initialization, without weight sharing (ablation) | Pure text RNN · RP10M | average precision: 0.633 -> 0.681 | Helped |
| Add direct visual supervision to word embeddings (ablation) | Pure text RNN · Gold RP10K | average precision: 0.748 -> 0.771 | Helped |
| Add direct visual supervision to word embeddings (ablation) | Pure text RNN · RP10M | average precision: 0.633 -> 0.687 | Helped |
| Use direct supervision on the final GRU state instead of soft supervision through weight sharing (ablation) | Model A · Gold RP10K | average precision: 0.843 -> 0.705 | Hurt |
| Use direct supervision on the final GRU state instead of soft supervision through weight sharing (ablation) | Model A · RP10M | average precision: 0.725 -> 0.646 | Hurt |
| Use direct visual supervision on embeddings instead of soft supervision through weight sharing (ablation) | Model A · Gold RP10K | average precision: 0.843 -> 0.771 | Hurt |
| Use direct visual supervision on embeddings instead of soft supervision through weight sharing (ablation) | Model A · RP10M | average precision: 0.725 -> 0.687 | Hurt |
| Add weight sharing to Model B or Model C (ablation) | Model B and Model C · RP10K and RP10M | average precision | No clear effect |
| Add visual information through image-conditioned GRU initialization and weight sharing | Word2Vec-GoogleNews · Gold RP10K | average precision: 9.5% | Helped |
Closest papers by meaning
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
- Expressing Objects Just Like Words: Recurrent Visual Embedding for Image-Text Matching (AAAI 2020 · similarity 0.89)
- A Simple but Tough-to-Beat Baseline for Sentence Embeddings (ICLR 2017 · similarity 0.88)
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction (ICLR 2026 · similarity 0.87)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.86)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.