Research map / Vision-language and multimodal / Human and embodied modeling
Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPARCL, which generates subtly varied synthetic image-caption pairs (LLM captions, feature-injected fast T2I with AdaIN style transfer) and trains CLIP with a sigmoid contrastive loss plus an adaptive margin loss to improve compositional understanding.
Problem
VLMs like CLIP lack compositional understanding (attributes, relations, objects), and collecting paired image-text data with subtle variations is costly; synthesizing such data is hard due to efficiency, text alignment, and image fidelity issues, plus noisy synthetic pairs.
Approach
Inject real image embeddings into the padding positions of a fast T2I model's text embeddings to preserve fidelity, apply AdaIN style transfer to reduce domain gap, and use an adaptive margin loss that zeroes out likely mislabeled pairs and emphasizes hard samples.
Key results
- SPARCL with ViT-B/32 reaches 77.2 on ARO, 79.2 on VL-CheckList, 87.1 on SugarCrepe, and 66.1 on SugarCrepe++, versus CLIP zero-shot 61.1/73.2/73.4/59.8.
- Synthetic captions drive most of the gain: ablation average rises from 69.33 to 73.99 (ViT-B/32) when adding synthetic captions alone, while synthetic images alone add about 0.5 points.
- Adaptive margin loss improves average accuracy over a fixed margin (77.38 vs 76.97 for ViT-B/32), and removing synthetic positives costs nearly 3 points on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with SPARCL synthetic data and adaptive-margin loss | CLIP ViT-B/32 · VL-CheckList | accuracy: 73.2 -> 79.2 | Helped |
| train with SPARCL synthetic data and adaptive-margin loss | CLIP ViT-B/32 · SugarCrepe | accuracy: 73.4 -> 87.1 | Helped |
| train with SPARCL synthetic data and adaptive-margin loss | CLIP ViT-B/32 · ARO | accuracy: 61.1 -> 77.2 | Mixed |
| add synthetic captions to real training samples (ablation) | SPARCL training, CLIP ViT-B/32 · four compositional benchmarks, average accuracy | accuracy: 69.33 -> 73.99 | Helped |
| add synthetic images without synthetic captions (ablation) | SPARCL training, CLIP ViT-B/32 · four compositional benchmarks, average accuracy | accuracy: 69.33 -> 69.79 | Helped |
| add synthetic images alongside synthetic captions (ablation) | SPARCL training, CLIP ViT-B/32 · four compositional benchmarks, average accuracy | accuracy: 73.99 -> 74.51 | Helped |
| add image feature injection (ablation) | SPARCL image generation, CLIP ViT-B/32 · four compositional benchmarks, average accuracy | accuracy: 74.51 -> 75.32 | Helped |
| add AdaIN style transfer (ablation) | SPARCL image generation, CLIP ViT-B/32 · four compositional benchmarks, average accuracy | accuracy: 74.51 -> 74.95 | Helped |
| add image feature injection and AdaIN (ablation) | SPARCL image generation, CLIP ViT-B/32 · four compositional benchmarks, average accuracy | accuracy: 74.51 -> 75.58 | Helped |
| add fixed-margin loss (ablation) | SPARCL, CLIP ViT-B/32 · four compositional benchmarks, average accuracy | accuracy: 75.58 -> 76.97 | Helped |
| replace fixed margin with adaptive margin (ablation) | SPARCL, CLIP ViT-B/32 · four compositional benchmarks, average accuracy | accuracy: 76.97 -> 77.38 | Helped |
| use inversed adaptive margin instead of adaptive margin (ablation) | SPARCL, CLIP ViT-B/32 · SugarCrepe | accuracy: 87.11 -> 86.68 | Hurt |
Surprising findings
- Synthetic images alone provide only marginal benefit (~0.5 average points), whereas synthetic captions give a large gain.
- Using only positive and hard-negative sets in the margin loss improves ARO and VL-CheckList but hurts SugarCrepe and SugarCrepe++, suggesting learned artifacts.
Limitations
- Synthetic images still contain generative artifacts, and synthetic captions may not perfectly match images, so some pairs are incorrect.
- Performance on ARO is lower than some baselines, possibly due to artifacts in ARO test samples.
- Gains from scaling training data diminish beyond 50% of the data.
Experimental setup
Datasets: COCO-2014 ARO VL-CheckList SugarCrepe SugarCrepe++
Benchmarks: ARO VL-CheckList SugarCrepe SugarCrepe++
Metrics: accuracy
Baselines: CLIP zero-shot CLIP finetune NegCLIP CE-CLIP CE-CLIP+ MosaiCLIP COMO FSC-CLIP SDS-CLIP syn-CLIP
Builds on: CLIP NegCLIP syn-CLIP AdaIN sigmoid contrastive loss (SigLIP)
Closest papers by meaning
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.90)
- Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning (AAAI 2024 · similarity 0.90)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.90)
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.