Research map / Image generation / Text-to-image diffusion
Semantic-Aware Data Augmentation for Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SADA, a semantic-aware data augmentation framework for text-to-image synthesis combining implicit textual semantic preserving augmentation (ITA) with an image semantic regularization loss (L_r) to avoid semantic mismatch and collapse.
Problem
Existing data augmentation for text-to-image synthesis causes semantic mismatch between augmented text-image pairs and semantic collapse in generated images, hurting text-image consistency and image quality.
Approach
Augment text embeddings in semantic space with covariance-scaled noise (closed-form ITA_C, or a learnable ITA_T) so augmented embeddings keep the main semantics, and regularize generated image semantic embeddings with L_r, which constrains both shift direction and magnitude via a variance-matching target.
Key results
- On CUB with AttnGAN, ITA_C + L_r reduces FID from 23.98 to 13.17 (CS 68.00 to 68.20).
- Tuning SD on Pokemon BLIP with SADA improves CLIPScore from 72.72 to 73.80 and reduces FID from 55.98 to 46.07.
- Tuning CLIP with ITA raises COCO image-retrieval Top-1 from 44.43 to 44.88 and text-retrieval Top-1 from 61.20 to 62.76.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ITA (textual semantic preserving augmentation) when tuning CLIP | CLIP ViT-B/16 retrieval tuning · COCO text-image retrieval | Top1 retrieval accuracy: +0.45 image, +1.56 text | Helped |
| add SADA (ITA_C + L_r) to AttnGAN and DF-GAN training | AttnGAN, DF-GAN · CUB | FID: AttnGAN 23.98 to 13.17; DF-GAN 12.10 to 10.45 | Helped |
| add SADA (ITA_C + L_r) when tuning AttnGAN | AttnGAN tuned on COCO · COCO | FID: 29.60 to 22.70 | Helped |
| add SADA (ITA_C + L_r) when tuning AttnGAN | AttnGAN tuned on COCO · COCO | CLIPScore: 62.59 to 64.59 | Helped |
| add SADA ITA_C tuning to Stable Diffusion with LoRA | Stable Diffusion · Pokemon BLIP | CLIPScore and FID: CS 72.72 to 73.80; FID 55.98 to 46.07 | Helped |
| add SADA to conditional DDPM | conditional DDPM · MNIST | FID: 8.61 to 7.78 | Helped |
| add Mixup augmentation (ablation) | AttnGAN · CUB | FID: 23.98 to 41.47 | Hurt |
| add Random Mask augmentation (ablation) | AttnGAN and DF-GAN · CUB | FID and CS: AttnGAN FID 23.98 to 15.59 (CS 68.00 to 67.80); DF-GAN FID 12.10 to 15.17 | Mixed |
| add Add Noise (Gaussian noise on text embedding) (ablation) | DF-GAN · CUB | FID: 12.10 to 42.07 (Task 3 Train) | Hurt |
| add ITA_C (closed-form, training-free) text augmentation (ablation) | AttnGAN · CUB | CLIPScore and FID: FID 23.98 to 13.68; CS 68.42 vs 68.00 | Helped |
| add GisC constraint L_r on top of ITA_C versus L_db (ablation) | DF-GAN, AttnGAN · CUB and COCO | FID: DF-GAN CUB FID 10.77 (L_db) vs 10.45 (L_r); AttnGAN COCO FID 25.82 (L_db) vs 22.70 (L_r) | Helped |
| add ITA_T learnable augmentation with L_db (ablation) | DF-GAN · CUB Task 2 | FID: ITA_T 14.03 vs ITA_T+L_db 11.74 | Mixed |
Surprising findings
- Mixup and DiffAug often worsen FID relative to baselines, and Random Mask and Add Noise give inconsistent gains across frameworks and datasets.
- ITA_T converges more slowly and weakens semantic enhancement early in training, improving consistency only with longer training.
Limitations
- ITA relies on covariance estimates of semantic embeddings, which may be unreliable on unbalanced datasets.
- ITA_T cannot be combined with L_r because the range of the augmentation is not closed-form.
- Several theoretical propositions rely on strong assumptions such as exact invariance and linear encoders; the non-linear case is not proven.
Experimental setup
Datasets: CUB COCO MNIST Pokemon BLIP Emoji
Benchmarks: CUB COCO MNIST Pokemon BLIP
Metrics: CLIPScore FID Top-1 retrieval accuracy Top-5 retrieval accuracy
Baselines: Mixup DiffAug Random Mask Add Noise I2T2I RiFeGAN Stylegan-nada direction bounding loss
Builds on: StyleGAN-NADA AttnGAN DF-GAN Group-theoretic data augmentation framework LoRA
Closest papers by meaning
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.85)
- Reproducibility Study of "ITI-GEN: Inclusive Text-to-Image Generation" (NeurIPS 2024 · similarity 0.84)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.83)
- CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning (ICML 2025 · similarity 0.83)
- Salient Concept-Aware Generative Data Augmentation (NeurIPS 2025 · similarity 0.83)
- Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.