Research map / Vision-language and multimodal / Human and embodied modeling
Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, Bernt Schiele
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace maximum-likelihood training with adversarial training (beam search) | caption generator · MS-COCO test (5000 images) | Vocabulary size: 1085 -> 2176 | Improved |
| replace maximum-likelihood training with adversarial training (sampling) | caption generator · MS-COCO test (5000 images) | Vocabulary size: 1460 -> 2671 | Improved |
| replace maximum-likelihood training with adversarial training (beam search) | caption generator · MS-COCO test (5000 images) | % novel sentences: 44.27 -> 72.53 | Improved |
| replace maximum-likelihood training with adversarial training (sampling) | caption generator · MS-COCO test (5000 images) | mBleu-4: 0.68 -> 0.51 | Improved |
| replace maximum-likelihood training with adversarial training (beam search) | caption generator · MS-COCO test (5000 images) | Meteor: 0.272 -> 0.239 | Hurt |
| replace maximum-likelihood training with adversarial training (beam search) | caption generator · MS-COCO test (5000 images) | SPICE: 0.187 -> 0.167 | Hurt |
| replace maximum-likelihood training with adversarial training (beam search) | caption generator · MS-COCO test (5000 images) | SPICE Count: 0.025 -> 0.080 | Improved |
| adversarial training instead of ML, judged by humans | caption generator · MS-COCO test (5000 images) | human eval: adversarial better vs worse (%), beam search: 36.9 vs 34.8 | No clear effect |
| discriminator evaluates 5 captions instead of 1 (ablation) | adversarial generator (VGG) · MS-COCO val (5000 images) | Meteor: 0.179 -> 0.197 | Improved |
| discriminator evaluates 5 captions instead of 1 (ablation) | adversarial generator (VGG) · MS-COCO val (5000 images) | Vocabulary size: 812 -> 1810 | Improved |
| add feature matching loss (ablation) | adversarial generator (VGG, p=5) · MS-COCO val (5000 images) | Meteor: 0.197 -> 0.207 | Improved |
| add feature matching loss (ablation) | adversarial generator (VGG, p=5) · MS-COCO val (5000 images) | Vocabulary size: 1810 -> 2547 | Improved |
Closest papers by meaning
- Generating Diverse and Descriptive Image Captions Using Visual Paraphrases (ICCV 2019 · similarity 0.90)
- Adversarial Semantic Alignment for Improved Image Captions (CVPR 2019 · similarity 0.90)
- Fast, Diverse and Accurate Image Captioning Guided by Part-Of-Speech (CVPR 2019 · similarity 0.90)
- Improving Image Captioning with Conditional Generative Adversarial Nets (AAAI 2019 · similarity 0.89)
- Partial Off-Policy Learning: Balance Accuracy and Diversity for Human-Oriented Image Captioning (ICCV 2021 · similarity 0.89)
- On Distinctive Image Captioning via Comparing and Reweighting (TPAMI 2022 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
