Research map / Vision-language and multimodal / Human and embodied modeling
Cross-Domain Image Captioning With Discriminative Finetuning
Roberto Dessì, Michele Bevilacqua, Eleonora Gualdoni, Nathanaël Carraz Rakotonirina, Francesca Franzon, Marco Baroni
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune ClipCap with a discriminative retrieval objective | ClipCap-COCO · COCO | P@1: 74.2 -> 84.8 | Improved |
| Fine-tune ClipCap with a discriminative retrieval objective | ClipCap-ConCap · Conceptual Captions | P@1: 82.5 -> 94.4 | Improved |
| Fine-tune ClipCap with a discriminative retrieval objective | ClipCap-ConCap · Flickr | P@1: 76.7 -> 87.8 | Improved |
| Fine-tune ClipCap with a discriminative retrieval objective | ClipCap-ConCap · Concadia | P@1: 59.17 -> 80.49 | Improved |
| Fine-tune ClipCap with a discriminative retrieval objective | ClipCap-COCO · COCO | CIDEr: 108.55 -> 105.40 | Hurt |
| Fine-tune ClipCap with a discriminative retrieval objective | ClipCap-ConCap · Conceptual Captions | CIDEr: 87.22 -> 55.26 | Hurt |
| Fine-tune BLIP with a discriminative retrieval objective | BLIP · Flickr | P@1: 75.1 -> 80.8 | Improved |
| Use discriminatively fine-tuned captions instead of human captions | Conceptual Captions human image identification · Conceptual Captions human retrieval | accuracy: 42.8 -> 47.6 | Improved |
| Remove the REINFORCE baseline (ablation) | ClipCap trained on Flickr · Flickr validation | P@1: 97.8 -> 34.3 | Hurt |
| Replace the running mean with greedy decoding as the REINFORCE baseline (ablation) | ClipCap trained on Flickr · Flickr validation | P@1: 97.8 vs 97.4 | No clear effect |
| Replace log-probability retrieval reward with cosine similarity reward (ablation) | ClipCap trained on Flickr · Flickr validation | P@1: 86.2 -> 85.2 | Hurt |
| Replace log-probability retrieval reward with accuracy reward (ablation) | ClipCap trained on Flickr · Flickr validation | P@1: 86.2 -> 85.3 | Hurt |
Closest papers by meaning
- Show, Adapt and Tell: Adversarial Training of Cross-Domain Image Captioner (ICCV 2017 · similarity 0.89)
- Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data (ECCV 2018 · similarity 0.89)
- SmallCap: Lightweight Image Captioning Prompted With Retrieval Augmentation (CVPR 2023 · similarity 0.89)
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning (ICLR 2026 · similarity 0.87)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.87)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
