Research map / Vision-language and multimodal / Human and embodied modeling
Learning to Generate Grounded Visual Captions without Localization Supervision
Chih-Yao Ma, Yannis Kalantidis, Ghassan AlRegib, Peter Vajda, Marcus Rohrbach, Zsolt Kira
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cyclical localization-reconstruction training | unsupervised captioning baseline · Flickr30k Entities (test) | F1_all: 4.08 -> 4.98 | Improved |
| add cyclical localization-reconstruction training | unsupervised captioning baseline · Flickr30k Entities (test) | F1_loc_per_sent: 31.83 -> 35.54 | Improved |
| add cyclical localization-reconstruction training | unsupervised captioning baseline · Flickr30k Entities (test) | CIDEr: 59.6 -> 61.4 | Improved |
| add cyclical localization-reconstruction training | unsupervised captioning baseline · ActivityNet-Entities (val) | F1_loc: 12.00 -> 15.84 | Improved |
| add cyclical localization-reconstruction training | unsupervised captioning baseline · ActivityNet-Entities (val) | F1_loc_per_sent: 31.68 -> 41.56 | Improved |
| add cyclical localization-reconstruction training | unsupervised captioning baseline · ActivityNet-Entities (val) | SPICE: 15.1 -> 14.8 | Hurt |
| add explicit KL attention consistency between attended and localized regions (ablation) | unsupervised baseline · Flickr30k Entities (val) | F1_all: 4.18 -> 4.19 | No clear effect |
| localize from the generated word embedding only instead of h^A (ablation) | cyclical model · Flickr30k Entities (val) | F1_all: 4.58 -> 5.63 | Improved |
| use a single fully-connected localizer instead of MLP (ablation) | cyclical model · Flickr30k Entities (test) | F1_all: 4.40 -> 4.98 | Improved |
| zero the localized region for non noun/verb words (ablation) | cyclical model · Flickr30k Entities (test) | F1_all_per_sent: 15.03 -> 16.30 | Mixed |
| compute reconstruction loss only on noun/verb words (ablation) | cyclical model · Flickr30k Entities (test) | F1_all: 4.98 -> 5.14 | No clear effect |
| add cyclical training with a perfect object detector | unsupervised baseline · Flickr30k Entities (test, perfect object detector) | F1_all: 20.82 -> 25.27 | Improved |
Closest papers by meaning
- Weakly-Supervised Generation and Grounding of Visual Descriptions With Conditional Generative Models (CVPR 2022 · similarity 0.92)
- Cycle-Consistency Learning for Captioning and Grounding (AAAI 2024 · similarity 0.90)
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.90)
- Pixel-Aligned Language Model (CVPR 2024 · similarity 0.89)
- Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.88)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
