Research map / Image generation / Text-to-image diffusion
Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diff-Mix, which personalizes Stable Diffusion (TI+DB with LoRA) and performs SDEdit-style inter-class image translation to create faithful yet background-diverse synthetic data for classification.
Problem
Generative data augmentation with text-to-image models for domain-specific image classification produces samples that are either unfaithful to foreground concepts (vanilla distillation) or lack background diversity (intra-class augmentation).
Approach
Fine-tune Stable Diffusion with Textual Inversion plus LoRA-based U-Net tuning to get faithful domain concepts, then translate reference images from other classes using a class prompt, so the foreground changes class while the background is retained, with soft labels via a nonlinear interpolation rule.
Key results
- Diff-Mix + CutMix reaches 92.46% average accuracy on six fine-grained datasets with ViT-B/16@384 vs 91.74% baseline; Diff-Mix alone 92.22%.
- On CUB-LT (IF=100), Diff-Mix (s=0.7) achieves 64.48% overall accuracy vs 58.13% for CE baseline and 62.27% for Real-Mix.
- On Waterbird, Diff-Mix gives +6.5% accuracy on the landbird-water-counterfactual-group (waterbird on land) challenge group relative to the baseline, averaging 72.47 vs 70.19.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Diff-Mix instead of the strongest listed baseline augmentation | ResNet50 classifier · Conventional classification across five fine-grained datasets | average accuracy gain: +0.56 | Helped |
| Use Diff-Mix instead of the strongest listed baseline augmentation | ViT-B/16 classifier · Conventional classification across five fine-grained datasets | average accuracy gain: +0.48 | Helped |
| Add CutMix to Diff-Mix | Diff-Mix classifier · Conventional classification across five fine-grained datasets | average accuracy gain: +0.63; +0.72 | Helped |
| Use Diff-Mix rather than CutMix | CUB-trained classifier evaluated on Waterbird · Waterbird average | accuracy: 71.23 -> 72.47 | Helped |
| Use Diff-Mix rather than CutMix | CUB-trained classifier evaluated on Waterbird · Waterbird: waterbird with land background | accuracy: 60.12 -> 63.24 | Helped |
| Fine-tune both the text identifier and U-Net instead of only the identifier (ablation) | Diff-Mix · CUB, 5-shot | accuracy: 57.64 -> 59.41 | Helped |
| Fine-tune both the text identifier and U-Net instead of only the U-Net (ablation) | Diff-Mix · CUB, 5-shot | accuracy: 56.11 -> 59.41 | Helped |
| Fine-tune both the text identifier and U-Net instead of only the identifier (ablation) | Diff-Mix · CUB, all-shot | accuracy: 81.86 -> 82.85 | Helped |
| Fine-tune both the text identifier and U-Net instead of only the U-Net (ablation) | Diff-Mix · CUB, all-shot | accuracy: 81.99 -> 82.85 | Helped |
| Increase the number of referable classes used as image sources (ablation) | Diff-Mix · CUB | accuracy | Helped |
Surprising findings
- Vanilla (non-fine-tuned) generative distillation (Real-Gen) falls below baseline accuracy at 10-shot and above on CUB, showing unfaithful synthetic data can hurt.
- Da-fusion and Real-Filtering degrade average accuracy relative to the no-augmentation baseline in the conventional setting.
Limitations
- Inter-class translation produces more undesirable samples with incomplete foregrounds when classes differ in shape or pose, requiring CLIP-based filtering.
- Little gain on the Stanford Dogs dataset, attributed to multiple subjects per image hindering foreground editing.
- Hyperparameters (translation strength, gamma) are sensitive to shot number and need tuning.
Experimental setup
Datasets: CUB-200-2011 Stanford Cars Oxford Flowers Stanford Dogs FGVC Aircraft CUB-LT Flower-LT Waterbird
Benchmarks: CUB-200-2011 Stanford Cars Oxford Flowers Stanford Dogs FGVC Aircraft CUB-LT Flower-LT Waterbird
Metrics: top-1 classification accuracy FID
Baselines: Real-Gen Real-Aug Real-Mix Diff-Gen Diff-Aug Real-Filtering Real-Guidance Da-fusion CutMix Mixup CMO CMO+DRW
Builds on: Stable Diffusion DreamBooth Textual Inversion SDEdit Da-fusion CutMix Mixup SYNAuG
Closest papers by meaning
- Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentation (NeurIPS 2024 · similarity 0.88)
- Inversion Circle Interpolation: Diffusion-based Image Augmentation for Data-scarce Classification (CVPR 2025 · similarity 0.88)
- Effective Data Augmentation With Diffusion Models (ICLR 2024 · similarity 0.86)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.86)
- Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification (AAAI 2024 · similarity 0.86)
- Diffusion Classifiers Understand Compositionality, but Conditions Apply (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.