Research map / Image generation / Fast diffusion sampling
Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGID, a semantic-guided generative image augmentation method that uses BLIP captions and class labels as prompts for Stable Diffusion image-to-image generation to balance diversity and semantic consistency for image classification.
Problem
Generative image augmentation with diffusion models yields diverse images but can alter essential semantics of the original image, while perturbation-based augmentation preserves semantics but lacks diversity; the paper targets balancing these for image classification.
Approach
Build a textual prompt from the ground-truth label and a BLIP-generated caption (optionally CLIP-filtered) and feed it with the original image into Stable Diffusion image-to-image generation, with prompt weighting and a similarity-based guidance-scale mapping, so generated variants stay semantically consistent yet diverse.
Key results
- SGID achieves average accuracy 67.92% on ResNet-50 (from scratch) vs 63.03% for the best perturbation baseline RandAugment and 66.20% for the best generative variant (+DiverseCaption).
- On ViT (ImageNet-21k), SGID averages 90.88%, outperforming Text2Img-based and RandAugment baselines by a small margin (about 0.33% over DiverseCaption variant).
- On CLIP-ViT (LAION-2B), SGID averages 87.99% vs 87.80% for RandAugment; combining SGID with RandAugment reaches 88.46%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SGID instead of SGID + DiverseCaption | ResNet-50 (from scratch) · Avg. accuracy across seven datasets | accuracy: 66.20 -> 67.92 | Helped |
| use SGID instead of SGID + DiverseCaption | ViT (ImageNet-21k) · Avg. accuracy across seven datasets | accuracy: 90.55 -> 90.88 | Helped |
| use SGID instead of SGID + DiverseCaption | CLIP-ViT (LAION-2B) · Avg. accuracy across seven datasets | accuracy: 87.85 -> 87.99 | Helped |
| add label and caption as a complete semantic prompt (ablation) | SGID, ResNet-50 (from scratch) · Pets | accuracy: 55.76 -> 76.15 | Helped |
| add label and caption as a complete semantic prompt (ablation) | SGID, ResNet-50 (from scratch) · Cars | accuracy: 82.09 -> 84.31 | Mixed |
| use the complete prompt instead of caption only (ablation) | SGID, ResNet-50 (from scratch) · Pets | accuracy: 63.80 -> 76.15 | Helped |
| use the complete prompt instead of caption only (ablation) | SGID, ResNet-50 (from scratch) · Cars | accuracy: 83.89 -> 84.31 | Helped |
| use the label only instead of the complete prompt (ablation) | SGID, ResNet-50 (from scratch) · Cars | accuracy: 84.31 -> 88.53 | Helped |
| use the complete prompt instead of label only (ablation) | SGID, ResNet-50 (from scratch) · Pets | accuracy: 75.90 -> 76.15 | Helped |
| add prompt weighting to beam-sampled captions (ablation) | SGID, ResNet-50 (from scratch) · CIFAR-100 | accuracy: 74.69 -> 75.60 | Helped |
| add prompt weighting to beam-sampled captions (ablation) | SGID, ResNet-50 (from scratch) · Pets | accuracy: 76.15 -> 75.06 | Hurt |
| add prompt weighting to nucleus-sampled captions (ablation) | SGID, ResNet-50 (from scratch) · DTD | accuracy: 37.04 -> 37.16 | No clear effect |
Surprising findings
- Label-only prompts outperform the complete label+caption prompt on Cars, attributed to BLIP struggling with fine-grained captions.
- Text2Img combined with SGID underperforms SGID alone, suggesting Text2Img alters essential semantics.
Limitations
- Performance is sensitive to the noise rate: increasing noise adds diversity but lowers accuracy.
- Captions from BLIP are unreliable for fine-grained datasets and textures, and the caption filter helps only in some cases.
- Diversity is limited to one augmented image per original image in main experiments, with expansion costs not deeply analyzed.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Caltech101 Stanford Cars Flowers102 OxfordPets DTD
Benchmarks: CIFAR-10 CIFAR-100 Caltech101 Stanford Cars Flowers102 OxfordPets DTD
Metrics: classification accuracy human-rated semantic consistency and diversity (1-5) average cosine similarity (CLIP image embeddings)
Baselines: Random Erasing CutMix MoEx RandAugment Text2Img ALIA+DiverseCaption ALIA+InstructPix2Pix
Builds on: Stable Diffusion BLIP CLIP RandAugment ALIA
Closest papers by meaning
- Effective Data Augmentation With Diffusion Models (ICLR 2024 · similarity 0.89)
- Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model (CVPR 2024 · similarity 0.86)
- Salient Concept-Aware Generative Data Augmentation (NeurIPS 2025 · similarity 0.86)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.85)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.85)
- Data Augmentation with Diffusion for Open-Set Semi-Supervised Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.