Research map / Vision-language and multimodal / Cross-modal learning
Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SynTIC, a text-only image captioning method that generates synthetic images from training texts with Stable Diffusion, refines their CLIP features with a contrastive loss, projects them into the CLIP text space, and adds DETR object tags as auxiliary prefixes.
Problem
Text-only image captioning relies on CLIP's cross-modal alignment, but the modality gap between CLIP text and image features and the training/inference discrepancy (text-only training vs. real images at inference) hinder alignment.
Approach
Use a pre-trained text-to-image model to create synthetic images for training texts, optimize their CLIP features toward paired texts via contrastive loss, project them as weighted combinations of CLIP text features, and feed them (plus detected object tags) as decoder prefixes, so training and inference use the same image-feature pipeline.
Key results
- In-domain MSCOCO: SynTIC reaches CIDEr 101.1 and BLEU-4 29.9, outperforming DeCap (CIDEr 91.2, BLEU-4 24.7).
- Zero-shot on MSCOCO with SS1M training: BLEU-4 13.3 and CIDEr 55.6, versus DeCap (SS1M) at 8.9 and 50.6.
- Cross-domain Flickr30K to MSCOCO with target-domain projection variant (SynTIC-TT) reaches CIDEr 64.4, against DeCap-TT's 63.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add feature optimization to the generated pseudo-image feature baseline (ablation) | SynTIC · MSCOCO in-domain | CIDEr: 88.5 -> 89.8 | Helped |
| add feature projection to the generated pseudo-image feature baseline (ablation) | SynTIC · MSCOCO in-domain | CIDEr: 88.5 -> 98.2 | Helped |
| add auxiliary object features to the generated pseudo-image feature baseline (ablation) | SynTIC · MSCOCO in-domain | CIDEr: 88.5 -> 91.2 | Helped |
| add auxiliary object features alongside feature projection and optimization (ablation) | SynTIC · MSCOCO in-domain | CIDEr: 100.0 -> 101.1 | Helped |
| add feature optimization alongside feature projection and auxiliary features (ablation) | SynTIC · MSCOCO in-domain | CIDEr: 100.3 -> 101.1 | Helped |
| add auxiliary object features to the generated pseudo-image feature baseline (ablation) | SynTIC · MSCOCO→Flickr30K | CIDEr: 34.8 -> 34.7 | Hurt |
| add feature projection to the generated pseudo-image feature baseline (ablation) | SynTIC · MSCOCO→Flickr30K | CIDEr: 34.8 -> 36.2 | Helped |
| use target-domain captions for feature projection | SynTIC · MSCOCO→Flickr30K | CIDEr: 38.4 -> 43.2 | Helped |
| use target-domain captions for feature projection | SynTIC · Flickr30K→MSCOCO | CIDEr: 47.0 -> 64.4 | Helped |
| train using fewer MSCOCO text samples (ablation) | SynTIC · MSCOCO in-domain | captioning performance | Hurt |
Surprising findings
- Feature projection alone already beats DeCap on MSCOCO in the ablation, showing the value of synthetic pairs even without other components.
- Adding object-tag features directly to the baseline gives only limited gains, which the authors attribute to redundancy with image features.
Limitations
- Relies on a text-to-image generator and an object detector, adding pipeline cost and dependence on their quality.
- Gains over strong baselines such as DeCap are mixed on some metrics (e.g., Flickr30K CIDEr, METEOR), and SynTIC-TT uses target-domain captions.
Experimental setup
Datasets: MSCOCO Flickr30K SS1M
Benchmarks: MSCOCO Flickr30K
Metrics: BLEU-4 METEOR ROUGE-L CIDEr SPICE
Baselines: BUTD ClipCap CLIPRe ZeroCap MAGIC CapDec CLOSE DeCap
Builds on: CLIP CapDec DeCap ClipCap Stable Diffusion
Closest papers by meaning
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.90)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.89)
- SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger (AAAI 2024 · similarity 0.89)
- Image Captioning with Multi-Context Synthetic Data (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.