Research map / Vision-language and multimodal / Human and embodied modeling
More Grounded Image Captioning by Distilling Image-Text Matching Model
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use noun-only POS-SCAN matching rather than original SCAN matching | Image-text matching model used as grounding guide · Flickr30k Entities val | Attention accuracy: 17.63% -> 28.58% | Helped |
| Distill POS-SCAN attention into Up-Down with KL regularization | Up-Down captioner · Flickr30k Entities val | Attention accuracy: 19.83% -> 29.39% | Helped |
| Distill original SCAN attention instead of POS-SCAN attention (ablation) | Up-Down captioner · Flickr30k Entities val | F1_all: 4.49 -> 4.72 | No clear effect |
| Add POS-SCAN attention supervision during XE pretraining (ablation) | Up-Down captioner · Flickr30k Entities val | F1_all: 5.11 -> 6.47 | Helped |
| Add ground-truth attention supervision during XE pretraining (ablation) | Up-Down captioner · Flickr30k Entities val | F1_loc: 14.67 -> 23.09 | Helped |
| Fine-tune with CIDEr reward after POS-SCAN attention supervision (ablation) | Up-Down captioner · Flickr30k Entities val | F1_loc: 17.96 -> 16.97 | Hurt |
| Use SCAN as the SCST reward in addition to POS-SCAN attention supervision (ablation) | Up-Down captioner · Flickr30k Entities val | F1_all: 6.62 -> 7.30 | Helped |
| Use SCAN instead of CIDEr as the SCST reward (ablation) | Up-Down captioner · MS-COCO Karpathy test | SPICE: 22.0 -> 22.2 | Helped |
| Use POS-SCAN instead of SCAN as the SCST reward (ablation) | Up-Down captioner · Flickr30k Entities val | CIDEr: 68.2 -> 64.3 | Hurt |
| Use SCAN rather than CIDEr as SCST reward (ablation) | Up-Down captioner · Flickr30k Entities val | F1_all: 4.19 -> 4.59 | Mixed |
Closest papers by meaning
- Cycle-Consistency Learning for Captioning and Grounding (AAAI 2024 · similarity 0.87)
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning (ICLR 2026 · similarity 0.86)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.86)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.86)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.86)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.