Research map / Vision-language and multimodal / CLIP and zero-shot

CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning

ICML 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Recaptioning and text-supervised pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CtrlSynth, a training-free, closed-loop, modular pipeline that decomposes images into visual tags, applies user control policies, and recomposes them via LLMs and diffusion models to synthesize image-text data that improves CLIP pretraining.

Problem

Web-crawled image-text data for CLIP pretraining is noisy, misaligned, and long-tailed, and existing synthetic data pipelines are ad hoc and offer little fine-grained control over generated text or images.

Approach

Decompose an image into visual tags (objects, attributes, relations), apply user-specified control policies (remove/add/replace, style), and recompose them with an LLM for text or a text-to-image model for images, with a closed-loop tag-consistency filter; no training required.

Key results

• On CC3M-trained CLIP, CtrlSynth raises average zero-shot classification on 20 datasets from 19.4 to 27.1 and COCO/Flickr recall@1 average from 13.7 to 37.1 (+23.4).
• CtrlSynth improves SugarCrepe average accuracy by 4.5 points for CC3M and 3.0 for CC12M models.
• On ImageNet-LT, tail-class accuracy improves from 13.8 to 35.1 (+21.3) and on Places-LT from 8.2 to 24.4 (+16.2).

Measured effects reported in the paper

Surprising findings

• Using original noisy captions to generate images (SP(4), CtrlSynth-img) gives little or negative compositional reasoning gains, since the source texts are noisy.
• Setting the self-filter threshold above 50% yields no accuracy gain, and CtrlSynth-img samples are the least helpful.

Limitations

• Relies on large pretrained LLM and diffusion models, and the quality of synthetic samples depends on them (smaller LLM Qwen2-7B degrades performance).
• Synthetic images from noisy original captions are often low quality, so image-only synthesis path helps less.
• Head-class accuracy slightly decreases on long-tail tasks.

Experimental setup

Datasets: CC3M CC12M ImageNet-LT Places-LT ImageNet-1K COCO Flickr30k SugarCrepe VTAB CIFAR-10 CIFAR-100

Benchmarks: ImageNet-1K ImageNet-V2 ImageNet-S ImageNet-A ImageNet-O ImageNet-R VTAB CIFAR-10 CIFAR-100 Food-101 COCO Flickr30k

Metrics: top-1 zero-shot accuracy recall@1 mean per class recall tail/medium/head accuracy

Baselines: CLIP baseline trained on original data VeCLIP LaCLIP

Builds on: CLIP VeCLIP LaCLIP stable diffusion

Closest papers by meaning

• Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.90)
• Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.88)
• SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.88)
• Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
• CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.88)
• CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning (AAAI 2026 · similarity 0.88)