Research map / Image generation / Fast diffusion sampling
DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIFFUSEMIX, an image augmentation that concatenates half of an input image with an InstructPix2Pix-generated filter-style variant and blends in a random fractal, improving classification, fine-grained, fine-tuning, data-scarcity and adversarial robustness results.
Problem
Image-mixing augmentations (Mixup, CutMix, etc.) can omit salient regions and create label ambiguity by mixing images across classes, while naive diffusion-generated augmentations give limited gains.
Approach
Concatenate a masked half of the original image with a diffusion-generated counterpart produced by a filter-like conditional prompt (instead of mixing two different-class images), then blend a random fractal image into the hybrid to add structural diversity.
Key results
- On Tiny-ImageNet-200, DIFFUSEMIX reaches 65.77% Top-1 vs 57.23% vanilla and 64.63% for the second-best Guided-AP.
- On ImageNet-1K with ResNet-50 (100 epochs), DIFFUSEMIX achieves 78.64% Top-1 vs 75.97% vanilla and 77.53% for GuidedMixup.
- On Stanford Cars fine-grained classification, DIFFUSEMIX reaches 91.26% Top-1 vs 85.52% vanilla.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DiffuseMix instead of Guided-AP | PreActResNet-18 · Tiny-ImageNet-200 | Top-1 accuracy (%): 64.63 -> 65.77 | Helped |
| use DiffuseMix instead of Guided-AP | PreActResNet-18 · CIFAR-100 | Top-1 accuracy (%): 81.20 -> 82.50 | Helped |
| use DiffuseMix instead of YOCO | ResNet-50 · ImageNet-1K | Top-1 accuracy (%): 77.88 -> 78.64 | Helped |
| use DiffuseMix instead of PuzzleMix | PreActResNet-18 · CIFAR-100 | FGSM error rate (%): 19.62 -> 17.38 | Helped |
| use DiffuseMix instead of PuzzleMix | PreActResNet-18 · Tiny-ImageNet-200 | FGSM error rate (%): 36.52 -> 34.53 | Helped |
| use DiffuseMix instead of Guided-AP | ResNet-50 · CUB-200-2011 | Top-1 accuracy (%): 77.08 -> 79.37 | Helped |
| use DiffuseMix instead of Guided-AP | ResNet-50 · FGVC Aircraft | Top-1 accuracy (%): 84.32 -> 85.76 | Helped |
| use DiffuseMix instead of Guided-AP | ResNet-50 · Stanford Cars | Top-1 accuracy (%): 90.27 -> 91.26 | Helped |
| add fractal blending to original images (ablation) | DiffuseMix · Stanford Cars | Top-1 accuracy (%): 85.52 -> 86.73 | Helped |
| use generated images directly, omitting concatenation and fractal blending (ablation) | DiffuseMix · Flower102 | Top-1 accuracy (%): 78.73 -> 77.38 | Hurt |
| add fractal blending to generated images only (ablation) | DiffuseMix · Flower102 | Top-1 accuracy (%): 77.38 -> 77.81 | Hurt |
| concatenate original and generated images, omitting fractal blending (ablation) | DiffuseMix · Stanford Cars | Top-1 accuracy (%): 85.52 -> 90.59 | Helped |
Surprising findings
- Using diffusion-generated images directly as augmentations gives only 87.63% Top-1 on Cars (vs 85.52% vanilla) and degrades Flowers accuracy in combination with fractal blending, showing that concatenation with original images is needed.
Limitations
- Generation depends heavily on text prompts; wrong prompts may yield unrealistic images.
- Additional computational overhead for generating augmented images with a diffusion model.
Experimental setup
Datasets: ImageNet-1K CIFAR-100 Tiny-ImageNet-200 Oxford Flowers-102 Stanford Cars FGVC Aircraft CUB-200-2011
Benchmarks: ImageNet-1K CIFAR-100 Tiny-ImageNet-200 CUB-200-2011 Stanford Cars FGVC Aircraft Oxford Flowers-102
Metrics: Top-1 accuracy Top-5 accuracy FGSM error rate
Baselines: Vanilla Mixup CutMix SaliencyMix PuzzleMix Co-Mixup Guided-AP AugMix SnapMix AdaAug AutoAugment RandAugment
Builds on: Mixup CutMix AugMix PixMix PuzzleMix InstructPix2Pix
Closest papers by meaning
- Adversarial AutoMixup (ICLR 2024 · similarity 0.87)
- Effective Data Augmentation With Diffusion Models (ICLR 2024 · similarity 0.87)
- DiffAug: A Diffuse-and-Denoise Augmentation for Training Robust Classifiers (NeurIPS 2024 · similarity 0.86)
- Inversion Circle Interpolation: Diffusion-based Image Augmentation for Data-scarce Classification (CVPR 2025 · similarity 0.86)
- Data Augmentation with Diffusion for Open-Set Semi-Supervised Learning (NeurIPS 2024 · similarity 0.86)
- Efficient Image-to-Image Diffusion Classifier for Adversarial Robustness (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.