Research map / Vision-language and multimodal / Human and embodied modeling
Joint Optimization for Cooperative Image Captioning
Gilad Vered, Gal Oren, Yuval Atzmon, Gal Chechik
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use partial sampling (PSST) instead of always sampling (ST Multinomial) (ablation) | speaker-listener captioner · COCO fixed CIDEr=1.2 | recall@10 (%): 81.8 -> 86.3 | Improved |
| use PSST on Gumbel softmax instead of ST Gumbel softmax (ablation) | speaker-listener captioner · COCO fixed CIDEr=1.2 | recall@10 (%): 80.6 -> 85.7 | Improved |
| train speaker and listener jointly instead of freezing speaker (MLE) then training listener (ablation) | PSST Multinomial · COCO fixed CIDEr=1.13 | recall@10 (%): 61.3 -> 89.9 | Improved |
| train jointly instead of Luo et al. frozen listener approach (ablation) | PSST Multinomial · COCO fixed CIDEr=1.13 | recall@10 (%): 74.6 -> 89.9 | Improved |
| joint training with PSST instead of REINFORCE (ablation) | speaker-listener captioner · COCO fixed CIDEr=1.13 | recall@10 (%): 86.2 -> 89.9 | Improved |
| PSST instead of REINFORCE at fixed recall@5=80% (ablation) | speaker-listener captioner · COCO | CIDEr: 0.902 -> 1.119 | Improved |
| PSST instead of REINFORCE at fixed recall@5=90% (ablation) | speaker-listener captioner · Flickr30K | CIDEr: 0.431 -> 0.488 | Improved |
| PSST instead of ST Multinomial at fixed recall@5=90% (ablation) | speaker-listener captioner · Flickr30K | CIDEr: 0.478 -> 0.488 | No clear effect |
| test listener on human-generated captions instead of speaker captions | PSST listener · COCO val | recall@10 (%): 89.9 -> 59.3 | Hurt |
| train listener on GT captions instead of jointly (ablation) | PSST · COCO val | recall@10 (%): 89.9 -> 66.8 | Hurt |
| use partial sampling rate 0<rho<1 instead of rho=0 or 1 (ablation) | PSST · COCO fixed CIDEr=1.22 | recall@10 | Improved |
| PSST Multinomial captions vs ST Multinomial in human discrimination (ablation) | PSST Multinomial · human 2AFC | accuracy (%): 69% -> 75% | Improved |
Closest papers by meaning
- Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (TPAMI 2016 · similarity 0.88)
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (TPAMI 2022 · similarity 0.88)
- Teaching Machines to Describe Images with Natural Language Feedback (NeurIPS 2017 · similarity 0.88)
- Reflective Decoding Network for Image Captioning (ICCV 2019 · similarity 0.87)
- Noise-Aware Image Captioning with Progressively Exploring Mismatched Words (AAAI 2024 · similarity 0.86)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
