Research map / Vision-language and multimodal / CLIP and zero-shot
Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a text-only-trained zero-shot image captioner that mitigates the CLIP modality gap via noise injection, sub-region feature aggregation, and sampling-and-filtering at inference.
Problem
Zero-shot captioning with CLIP trained only on text suffers from the modality gap between text and image embeddings, causing poor transfer from training (text) to inference (image).
Approach
Inject Gaussian noise into CLIP text features to mimic image-embedding misalignment during training, and at inference aggregate CLIP-attention-selected image sub-region features with their mean, plus sampling and filtering to reduce noise-induced uncertainty.
Key results
- On MSCOCO, the full method reaches CIDEr 0.697 versus 0.221 for the baseline (rebuttal ablation table).
- Adding noise injection raises CIDEr from 0.221 to 0.286; adding sub-region aggregation raises it to 0.599.
- Using informative sub-regions (CLIP attention) gives CIDEr 0.599 versus 0.520 for random sub-regions and 0.340 when only half are kept.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add noise injection (ablation) | CLIP zero-shot captioning baseline · MSCOCO | CIDEr: 0.221 -> 0.286 | Helped |
| add subregion aggregation (ablation) | CLIP zero-shot captioning with noise injection · MSCOCO | CIDEr: 0.286 -> 0.599 | Helped |
| add sampling and filtering (ablation) | CLIP zero-shot captioning with noise injection and subregion aggregation · MSCOCO | CIDEr: 0.599 -> 0.697 | Helped |
| drop half of the informative subregions (ablation) | CLIP zero-shot captioning using informative subregions · MSCOCO | CIDEr: 0.599 -> 0.340 | Hurt |
| select random subregions instead of informative subregions (ablation) | CLIP zero-shot captioning using informative subregions · MSCOCO | CIDEr: 0.599 -> 0.520 | Hurt |
Surprising findings
- Randomly selecting 10 sub-regions gives slightly lower performance than the mix representation, and random single sub-region has the lowest similarity, suggesting that choosing the right sub-regions matters.
Limitations
- Noise injection adds uncertainty that requires sampling and filtering to offset.
- Sub-region selection depends on CLIP attention; random selection degrades performance.
Experimental setup
Datasets: MSCOCO
Benchmarks: MSCOCO
Metrics: BLEU-1 BLEU-4 METEOR ROUGE-L CIDEr SPICE
Baselines: Baseline (CLIP text-trained captioner without gap mitigation)
Builds on: CLIP ClipCap
Closest papers by meaning
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.91)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.91)
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning (CVPR 2025 · similarity 0.90)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.90)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.90)
- Zero-Shot Image Captioning with Multi-type Entity Representations (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.