Research map / Vision-language and multimodal / Human and embodied modeling
Image Captioning with Multi-Context Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICSD, which uses GPT-3.5 to select and summarize simple captions into multi-context captions, generates images with Stable Diffusion v1.4, and trains a BLIP-style captioner solely on this synthetic data.
Problem
Image captioning normally needs many annotated image-text pairs; synthetic images generated from simple captions lack the multi-context richness of real scenes, so captioners trained on them underperform.
Approach
Group similar captions via CLIP retrieval, have an LLM select captions describing the same scene from multiple viewpoints and summarize them into one multi-context caption, then generate images with a diffusion model and train the captioner only on these synthetic pairs.
Key results
- In-domain on MSCOCO, ICSD reaches CIDEr 96.6 and B@4 29.9, outperforming DeCap (92.9 CIDEr) and CapDec (91.8 CIDEr) under text-only training.
- On Flickr30k, ICSD achieves CIDEr 54.3 and B@4 25.2, versus DeCap's 49.3 and 20.0.
- Cross-domain with SS1M+GenT training reaches NoCaps overall CIDEr 53.5 and MSCOCO CIDEr 58.3, versus DeCap's 42.7 overall on NoCaps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uni-context-only training with selected captions and summarized multi-context image-text pairs (ablation) | ICSD · MSCOCO | BLEU@4: 26.9 -> 29.9 | Helped |
| Add selection of compatible captions, retaining uni-context images without summarization (ablation) | ICSD · MSCOCO | CIDEr: 90.8 -> 94.2 | Helped |
| Add summarization and multi-context image generation without selection (ablation) | ICSD · MSCOCO | CIDEr: 90.8 -> 95.8 | Helped |
| Add selection before summarization to the summarization-only variant (ablation) | ICSD · MSCOCO | CIDEr: 95.8 -> 96.6 | Helped |
| Use ground-truth grouping instead of LLM selection (ablation) | ICSD · MSCOCO | CIDEr: 96.6 -> 97.3 | Helped |
| Increase multi-context training images from 10,000 to 50,000 (ablation) | ICSD · MSCOCO test | BLEU@4: 28.0 -> 29.3 | Helped |
| Increase multi-context training images from 150,000 to 200,000 (ablation) | ICSD · MSCOCO test | CIDEr: 96.6 -> 96.5 | No clear effect |
| Retrieve images rather than generate them with diffusion (ablation) | ICSD · MSCOCO | BLEU@4: 29.9 -> 27.3 | Hurt |
| Generate a detailed caption from each input caption instead of merging simple captions (ablation) | ICSD · MSCOCO | CIDEr: 96.6 -> 94.7 | Hurt |
| Expand the training corpus from SS1M to GenT | ICSD · NoCaps val out-of-domain | CIDEr: 35.6 -> 40.3 | Helped |
| Combine SS1M and GenT rather than using SS1M alone | ICSD · MSCOCO (cross-domain) | BLEU@4: 13.6 -> 16.0 | Helped |
| Combine SS1M and GenT rather than using SS1M alone | ICSD · NoCaps val out-of-domain | CIDEr: 35.6 -> 48.3 | Helped |
Surprising findings
- Retrieving real images from LAION-2B with condensed captions performs worse than generating images with Stable Diffusion (CIDEr 85.7 vs 96.6).
- Using ground-truth MSCOCO groups for summarization only modestly improves over LLM selection (CIDEr 97.3 vs 96.6).
Limitations
- Performance on the NoCaps out-domain set is weaker when trained on SS1M, since SS1M is built from MSCOCO object classes.
- Dependence on API-only GPT-3.5-turbo limits large-scale generation (uni-context images were added for SS1M due to API limits).
- Gains saturate at 200,000 multi-context images, attributed to limited corpus diversity.
Experimental setup
Datasets: MSCOCO Flickr30k SS1M NoCaps Visual Genome LAION-2B GenT
Benchmarks: MSCOCO Flickr30k NoCaps
Metrics: BLEU@4 METEOR ROUGE-L CIDEr
Baselines: ZeroCap Magic ESPER-Style CLIPRe CapDec DeCap CLOSE ConZIC
Builds on: BLIP DeCap DPM-Solver chain-of-thought
Closest papers by meaning
- Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models (ICLR 2025 · similarity 0.90)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.89)
- Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning (AAAI 2024 · similarity 0.88)
- Sieve: Multimodal Dataset Pruning using Image Captioning Models (CVPR 2024 · similarity 0.87)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.87)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.