Research map / Vision-language and multimodal / Human and embodied modeling
Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations
Ashwin Kalyan, Stefan Lee, Anitha Kannan, Dhruv Batra
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer neighboring examples’ annotations using learned semantic similarity | multi-label attribute prediction · AWA and CUB | Average Precision@10: AWA: 54.18 -> 57.12 (1 observed label); 63.71 -> 66.28 (20% observed labels); CUB: 35.62 -> 38.83 (1 observed label); 39 | Improved |
| transfer neighboring annotations with attention selecting relevant sequence portions | image captioning · Flickr-8k, Flickr-30k, COCO | Recall5@100: MLE: 0.71 -> 0.95; 1.30 -> 1.63; 1.45 -> 1.84 | Improved |
| transfer neighboring annotations with attention selecting relevant sequence portions | visually grounded question generation · Flickr, COCO, Bing | Recall5@100: MLE: 0.35 -> 0.51; 0.33 -> 0.49; 0.37 -> 0.55 | Improved |
| add attention to select relevant portions of neighboring captions instead of transferring entire captions (ablation) | Caption-Transfer · COCO | Recall5@100: 1.78 -> 1.84 | Improved |
| append neighboring captions as full ground-truth training targets rather than softly weighting supervision (ablation) | COCO captioning · COCO | Recall5@100: 1.84 -> 1.49 | Hurt |
| refine the representation space jointly for the task instead of holding it fixed (ablation) | COCO captioning · COCO | Recall5@100: 1.63 -> 1.84 | Improved |
| transfer neighboring captions without attention rather than using attention (ablation) | Caption-Transfer · Flickr-8k | Recall5@100: 0.89 -> 0.95 | Hurt |
Closest papers by meaning
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.87)
- Causal Graphical Models for Vision-Language Compositional Understanding (ICLR 2025 · similarity 0.87)
- COMMA: Co-articulated Multi-Modal Learning (AAAI 2024 · similarity 0.87)
- Pre-Trained Vision-Language Models as Noisy Partial Annotators (AAAI 2025 · similarity 0.87)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.86)
- Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
