Research map / Image generation / Text-to-image diffusion
Salient Concept-Aware Generative Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a salient concept-aware image embedding (trained with an angular margin loss on DINOv2) that conditions an SDXL-based personalized generator for generative data augmentation, improving fine-grained classification especially in long-tail settings.
Problem
Generative data augmentation with image and text prompts struggles to balance fidelity and diversity because image embeddings entangle non-essential background attributes that conflict with text edits.
Approach
Train a domain-specific salient concept-aware image embedding model with an additive angular margin loss so that embeddings focus on class-discriminative content; feed these frozen embeddings with text embeddings into an SDXL adapter plus LoRA synthesis model without per-subject optimization.
Key results
- Conventional setting: average Top-1 of 92.97% (RN50) and 93.74% (ViT-B/16) across six FGVC datasets, +0.73% and +0.55% over the best peer method (Diff-Mix).
- Long-tail IN-LT and iNat: tail-class accuracy of 69.93% (IN-LT) and 73.20% (iNat) for the ViT-B/16 model, with about 6.5% average gain over prior augmentation methods; 80.58% and 78.54% with LWS.
- Few-shot 10-shot setting: average 85.02% on ViT-B/16 versus 83.17% for DA-Fusion; OOD Waterbirds average 51.16% vs 45.48% for DA-Fusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SCA generative augmentation rather than the strongest reported baseline | FGVC classifiers · Conventional FGVC (6 datasets, RN50) | average Top-1 accuracy (%): 91.23 -> 92.97 | Helped |
| use SCA generative augmentation rather than Diff-Mix | ViT-B/16 classifier · Conventional FGVC (6 datasets) | average Top-1 accuracy (%): 93.19 -> 93.74 | Helped |
| use SCA generative augmentation rather than DA-Fusion | ViT-B/16 classifier · Few-shot FGVC (10-shot, 5 datasets) | average Top-1 accuracy (%): 83.17 -> 85.02 | Helped |
| use SCA generative augmentation rather than CutMix | ViT-B/16 classifier · ImageNet-LT | Top-1 accuracy (%): 72.69 -> 79.17 | Helped |
| use SCA generative augmentation rather than Mixup | ViT-B/16 classifier · iNat long-tail | Top-1 accuracy (%): 67.59 -> 74.11 | Helped |
| use SCA generative augmentation rather than DA-Fusion | ViT-B/16 few-shot classifier · Waterbird OOD (average) | Top-1 accuracy (%): 45.48 -> 51.16 | Helped |
| replace SCA embeddings with CE classifier embeddings (ablation) | SDXL generation and iNat downstream classifier · iNat | Cls. Top1 (%): 74.01% -> 66.24% | Hurt |
| replace SDXL with Stable Diffusion 1.5 (ablation) | generation pipeline · iNat | Cls. Top1 (%): 74.01% -> 71.84% | Hurt |
| remove LoRA (ablation) | generation pipeline · iNat | Cls. Top1 (%): 74.01% -> 68.86% | Hurt |
| replace the adapter with a naive MLP (ablation) | generation pipeline · iNat | Cls. Top1 (%): 74.01% -> 73.43% | Hurt |
| replace image classifier-free null guidance with a zero vector (ablation) | generation pipeline · iNat | Cls. Top1 (%): 74.01% -> 73.77% | Hurt |
| increase synthetic-to-real ratio beyond the useful range (ablation) | downstream classifier training · FGVC classification | Top-1 accuracy | Hurt |
Surprising findings
- Classifier embeddings trained with cross-entropy give marginal downstream gains compared with the angular-margin SCA embedding.
- Standard FID and KID were intentionally excluded because they favor identity mapping and do not fit the goal of meaningfully different augmentations.
Limitations
- Pipeline built on SDXL is optimized for object-centric images and may not suit text-rich VQA or graph tasks, per the authors.
- Training cost grows with dataset size (about 3 days for 2.7M images on 8 A100 GPUs).
- SCA is described as a mitigation rather than a precise concept isolator; the margin loss does not fully remove irrelevant information.
Experimental setup
Datasets: ImageNet-1K iNaturalist 2018 Flowers102 FGVC Aircraft CUB-200-2011 Stanford Cars Stanford Dogs Food-101 Places365 Waterbirds
Benchmarks: ImageNet-1K iNaturalist 2018 Flowers102 FGVC Aircraft CUB-200-2011 Stanford Cars Stanford Dogs Food-101 ImageNet-LT Waterbirds Places365
Metrics: Top-1 accuracy Top-5 accuracy DreamSim Vendi score CLIP-T generated-image classifier accuracy
Baselines: Vanilla CutMix Mixup Real-Guidance DA-Fusion Diff-Mix LWS IP-Adapter ELITE E4T FastComposer
Builds on: Textual Inversion DreamBooth DA-Fusion SDXL DINOv2
Closest papers by meaning
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.88)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.87)
- CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization (AAAI 2025 · similarity 0.87)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.87)
- GMAIL: Generative Modality Alignment for generated Image Learning (ICML 2025 · similarity 0.87)
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.