Research map / Image generation / Text-to-image diffusion
Not Just Pretty Pictures: Toward Interventional Data Augmentation Using Text-to-Image Generators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that off-the-shelf text-to-image generators (mainly Stable Diffusion v1.5 via SDEdit and Text2Image) can serve as interventional data augmentation for single domain generalization and reducing reliance on spurious features, beating classical augmentation baselines.
Problem
Neural image classifiers degrade under covariate shift from environmental conditions absent in training data; existing augmentations hard-code specific invariances and cannot simulate arbitrary interventions.
Approach
Use text-to-image generators with natural-language interventional prompts (via SDEdit, Text2Image, ControlNet, InstructPix2Pix) to synthesize or edit training images that simulate environmental interventions, zero-shot and without task-specific fine-tuning, then train classifiers on the augmented data.
Key results
- On SDG with ResNet-50, SDEdit(M) averages 61.04 across four benchmarks vs. 55.89 for the strongest baseline PixMix and 54.78 for ERM.
- Text2Image(M) reaches 83.72 on PACS with ResNet-50 vs. 67.12 for PixMix, and 40.77 on DomainNet vs. 26.46 for ACVC.
- Retrieval from Stable Diffusion's training set reaches 70.14 on Office-Home with ResNet-50, outperforming all generative methods there.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SDEdit synthetic target-domain data with minimal prompts | ResNet-50 · PACS, OfficeHome, NICO++, and DomainNet SDG | average accuracy: 54.78 -> 61.04 | Helped |
| Replace conventional augmentation with minimal-prompt SDEdit | ResNet-50 SDG · Average across the four SDG benchmarks | accuracy: 55.89 -> 61.04 | Helped |
| Use diverse or deterministic language-enhanced prompts instead of minimal prompts (ablation) | SDEdit · PACS, OfficeHome, and DomainNet SDG | accuracy: 76.43 -> 75.65; 64.66 -> 63.14; 31.94 -> 30.94 | Hurt |
| Use language-enhanced prompts instead of minimal or handcrafted prompts (ablation) | SDEdit · NICO++ SDG | accuracy: 71.12 -> 73.69 | Helped |
| Remove access to synthetic target domains (ablation) | SDEdit(M) · PACS SDG | accuracy: 76.43 -> 76.03 | Hurt |
| Generate with text only rather than edit a source image (ablation) | Stable Diffusion IDA · PACS, OfficeHome, and DomainNet SDG | accuracy: 76.43 -> 83.72; 64.66 -> 67.63; 31.94 -> 40.77 | Helped |
| Use CLIP post-hoc filtering of generated images (ablation) | T2I-based IDA · SDG and RRSF | performance | No clear effect |
| Use retrieval instead of generation (ablation) | T2I-based IDA · OfficeHome SDG | accuracy: 67.63 -> 70.14 | Helped |
| Use retrieval instead of generation (ablation) | T2I-based IDA · NICO++ and DomainNet SDG | accuracy: 71.54 -> 70.42; 40.77 -> 31.07 | Hurt |
| Use targeted SDEdit prompts to intervene on the relevant spurious feature | ResNet-18 · ImageNet-9, CCS, and CelebA-sub | spurious-feature reliance | Helped |
| Use PixMix, which mixes images with fractals | ResNet-18 · CelebA-sub demographic bias | demographic bias | Hurt |
Surprising findings
- Post-hoc CLIP filtering does not consistently improve results, unlike prior reports on earlier generators.
- The simple Text2Image conditioning (no input image) outperforms image-conditioned SDEdit, suggesting image conditioning can hinder intervention.
Limitations
- Failure modes of ControlNet and InstructPix2Pix conditioning can hurt spurious-feature reduction.
- Generation computational cost remains a bottleneck.
- Only image classification is studied; extending to detection/segmentation needs spatial conditioning.
Experimental setup
Datasets: PACS Office-Home NICO++ DomainNet ImageNet-9 Cue-Conflict Stimuli CelebA
Benchmarks: PACS Office-Home NICO++ DomainNet ImageNet-9 Cue-Conflict Stimuli CelebA-sub
Metrics: average test accuracy across target domains background gap texture bias RandGap/FlipGap
Baselines: ERM AugMix RandAugment CutOut PixMix ACVC MixUp CutMix L2D MEADA RSC
Builds on: SDEdit ControlNet InstructPix2Pix PixMix AugMix
Closest papers by meaning
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.87)
- Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following (CVPR 2024 · similarity 0.86)
- Instruct Where the Model Fails: Generative Data Augmentation via Guided Self-contrastive Fine-tuning (AAAI 2025 · similarity 0.85)
- Effective Data Augmentation With Diffusion Models (ICLR 2024 · similarity 0.85)
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.85)
- Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.