Research map / Vision-language and multimodal / Human and embodied modeling
Learning to Learn Words from Visual Scenes
Dídac Surís, Dave Epstein, Heng Ji, Shih-Fu Chang, Carl Vondrick
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add via-vision attention instead of isolated attention (ablation) | EXPERT · EPIC-Kitchens new-word acquisition 1:1 | top-1 accuracy: 69.0 -> 72.7 | Improved |
| add via-vision attention instead of isolated attention (ablation) | EXPERT · EPIC-Kitchens new-word acquisition 2:1 | top-1 accuracy: 57.8 -> 64.5 | Improved |
| add input pointing on top of via-vision attention (ablation) | EXPERT · EPIC-Kitchens new-word acquisition 1:1 | top-1 accuracy: 72.7 -> 75.0 | Improved |
| add input pointing on top of via-vision attention (ablation) | EXPERT · EPIC-Kitchens familiar words (all) | top-5 accuracy: 74.9 -> 67.0 | Hurt |
| use full quadratic attention instead of via-vision attention (ablation) | EXPERT · EPIC-Kitchens new-word acquisition 1:1 | top-1 accuracy: 72.7 -> 76.6 | Mixed |
| use full attention instead of via-vision attention (ablation) | EXPERT · EPIC-Kitchens familiar words (all) | top-5 accuracy: 74.9 -> 67.3 | Hurt |
| add visual grounding (BERT with Vision vs BERT scratch) | BERT · EPIC-Kitchens new-word acquisition 1:1 | top-1 accuracy: 36.5 -> 63.4 | Improved |
| add meta-learned episodic training (EXPERT vs BERT with Vision) | BERT with Vision · Flickr30k new-word acquisition 1:1 | top-1 accuracy: 32.1 -> 69.3 | Improved |
| add meta-learned episodic training (EXPERT vs BERT with Vision) | BERT with Vision · EPIC-Kitchens new compositions | top-5 accuracy: 37.6 -> 53.0 | Improved |
| add language pretraining (BERT pretrained vs scratch) | BERT · Flickr30k new-word acquisition 1:1 | top-1 accuracy: 31.6 -> 69.4 | Improved |
| remove bounding boxes at test time, keep full image only (ablation) | EXPERT · EPIC-Kitchens new-word acquisition | top-1 accuracy: 4% worse with 1:1, 2% worse with 2:1 | Hurt |
| remove region proposals at test time (ablation) | EXPERT · Flickr30k | accuracy: decreases by at most 7% | Hurt |
Closest papers by meaning
- Context-Aware Meta-Learning (ICLR 2024 · similarity 0.89)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.87)
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.87)
- Causal Graphical Models for Vision-Language Compositional Understanding (ICLR 2025 · similarity 0.87)
- In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.87)
- Compositional Entailment Learning for Hyperbolic Vision-Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
