Research map / Vision-language and multimodal / Human and embodied modeling
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan Carlos Caicedo, Julia Hockenmaier, Svetlana Lazebnik
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch from classification VGG19 to detection Fast RCNN features | CCA · Flickr30k Entities phrase localization | R@1: 30.83 -> 41.77 | Improved |
| add detector scores for phrases mapped to PASCAL categories (ablation) | CCA · Flickr30k Entities phrase localization | R@1: 41.77 -> 43.84 | Improved |
| use detector alone instead of CCA for phrases that have it | CCA or Detector · Flickr30k Entities phrase localization | R@1: 41.77 -> 42.58 | Improved |
| add region size bias (ablation) | CCA+Detector · Flickr30k Entities phrase localization | R@1: 43.84 -> 49.22 | Improved |
| add color classifier term (ablation) | CCA+Detector · Flickr30k Entities phrase localization | R@1: 43.84 -> 45.79 | Improved |
| add color term on top of size term (ablation) | CCA+Detector+Size · Flickr30k Entities phrase localization | R@1: 49.22 -> 50.89 | Improved |
| add color term on top of size term (clothing phrases) (ablation) | CCA+Detector+Size · Flickr30k Entities clothing phrases | R@1: 39.20 -> 46.88 | Improved |
| fine-tune Fast RCNN on colors instead of linear SVM on detector features | color classifier · Color classification | accuracy: 16% -> 80.47% | Improved |
| train CCA on all region-phrase pairs instead of resampling to at most N=10 per phrase (ablation) | CCA · Flickr30k Entities phrase localization | R@1 | Hurt |
| add GMM Fisher vectors to HGLMM (ablation) | phrase features · Flickr30k Entities phrase localization | R@1 | No clear effect |
| add region-phrase distance to whole-image CCA (VGG19) for retrieval | HGLMM FV+VGG19 · Flickr30k image annotation | R@1: 36.5% -> 37.0% | Improved |
| use full region-phrase model instead of basic VGG model in retrieval | Weighted Distance · Flickr30k image annotation | R@1: 37.0% -> 37.5% | No clear effect |
Closest papers by meaning
- Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment (ICCV 2019 · similarity 0.89)
- Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.87)
- Panoptic Captioning: An Equivalence Bridge for Image and Text (NeurIPS 2025 · similarity 0.87)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.87)
- Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension (NeurIPS 2024 · similarity 0.86)
- Composing Object Relations and Attributes for Image-Text Matching (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
