Research map / Image generation / Autoregressive and latent generation
Generative Data Mining with Longtail-Guided Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Longtail Guidance, which uses a differentiable single-pass Epistemic Head signal from a deployed classifier to steer latent diffusion toward rare or hard synthetic training examples, improving generalization without retraining either model.
Problem
Deployed predictive models encounter longtail (rare or hard) inputs that are hard to anticipate; existing synthetic data methods do not condition generation on the specific deployed model's weaknesses.
Approach
A lightweight Epistemic Head (duplicated classifier heads trained with an oracle loss, stop-grad on the backbone) gives a differentiable single-pass epistemic uncertainty signal; Longtail Guidance backpropagates this signal through decoded predicted clean latents of a latent diffusion model, so the predictor never sees noisy diffusion states and neither model is retrained.
Key results
- On the in-distribution ImageNet-100 task, LTG (Epistemic) reaches 64.64 average top-1 vs 60.33 for Dream-ID and 59.37 for SD, and gains over Dream-ID are about 5.6 points on average versus GIF or Dream-ID.
- On Dataset Expansion, LTG (Epistemic) averages 82.4 top-1 across Caltech, Cars, Flowers, Pets, versus 75.6 for the best GIF baseline (GIF-SD), with LTG reaching GIF-level accuracy at 20-30% of the synthetic volume.
- On ImageNet-LT with LiVT, LTG (Epistemic) improves overall accuracy to 61.9 (from 60.6 for LiVT) and Few-class accuracy to 50.0 (from 40.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| augment training with LTG (Epistemic) synthetic data instead of GIF-SD synthetic data | ResNet-50 · Caltech | top-1 accuracy: 65.1 -> 71.5 | Helped |
| augment training with LTG (Energy) synthetic data instead of GIF-SD synthetic data | ResNet-50 · Cars | top-1 accuracy: 75.7 -> 85.3 | Helped |
| augment training with LTG (Epistemic) synthetic data instead of GIF-SD synthetic data | ResNet-50 · Flowers | top-1 accuracy: 88.3 -> 90.9 | Helped |
| augment training with LTG (Epistemic) synthetic data instead of GIF-SD synthetic data | ResNet-50 · Pets | top-1 accuracy: 73.4 -> 82.0 | Helped |
| augment training with LTG (Epistemic) synthetic data instead of DREAM-ID synthetic data | ResNet-34 · ImageNet-100 | top-1 accuracy: 88.46 -> 90.30 | Helped |
| augment training with LTG (Entropy) synthetic data instead of DREAM-ID synthetic data | ResNet-34 · ImageNet-A | top-1 accuracy: 12.13 -> 22.70 | Helped |
| augment training with LTG (Epistemic) synthetic data instead of DREAM-ID synthetic data | ResNet-34 · ImageNet-V2 | top-1 accuracy: 80.40 -> 81.54 | Helped |
| fine-tune LiVT with LTG (Epistemic) synthetic data instead of Stable Diffusion synthetic data | LiVT · ImageNet-LT Few | top-1 accuracy: 44.3 -> 50.0 | Helped |
| use Epistemic rather than Energy as the LTG guidance signal (ablation) | ResNet-50 with LTG · Pets | top-1 accuracy: 80.0 -> 82.0 | Helped |
| use Epistemic rather than Entropy as the LTG guidance signal (ablation) | ResNet-50 with LTG · Pets | top-1 accuracy: 81.6 -> 82.0 | Helped |
| continue LTG training from 100 to 200 epochs (ablation) | ResNet-50 with LTG · Caltech | top-1 accuracy: 71.5 -> 72.9 | Helped |
| continue LTG training from 200 to 300 epochs (ablation) | ResNet-50 with LTG · Caltech | top-1 accuracy: 72.9 -> 72.9 | No clear effect |
Surprising findings
- The predictor guiding generation never sees noisy diffusion states; decoded predicted clean terminal states are closer in FID to real data than decoded noisy states, which explains why guidance works without training on intermediate states.
- CLIP zero-shot and the production classifier have similar aggregate accuracy on Flowers (74.1% vs 71.2%) but per-class accuracy differences average about 34%, so CLIP-difficult classes differ from production-model-difficult classes.
Limitations
- Longtail Guidance is expensive: differentiating through the VAE decoder costs about 45GB VRAM at batch size 8 and reduces generation throughput to about 1 image/s.
- Performance gains saturate with continued fine-tuning and synthetic generation (300-epoch results plateau).
- Results rely on reproduced baseline checkpoints because originals were unavailable, and gains over Epistemic/Entropy/Energy are described as mild.
Experimental setup
Datasets: ImageNet-LT ImageNet-100 ImageNet-A ImageNet-V2 Stanford Cars Oxford Flowers Oxford Pets Caltech101
Benchmarks: ImageNet-LT ImageNet-100 ImageNet-A ImageNet-V2 Stanford Cars Oxford Flowers Oxford Pets Caltech101
Metrics: top-1 accuracy FID precision and recall per-class accuracy
Baselines: GIF Dream-ID Stable Diffusion prompt-tuned DALL-E2 prompt-tuned MAE data generation Cutout GridMask RandAugment AutoAugment CutMix AugMix DeepAugment
Builds on: classifier guidance DDIM Universal Guidance Monte Carlo dropout Hydra architecture
Closest papers by meaning
- Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model (ICLR 2025 · similarity 0.87)
- TFG: Unified Training-Free Guidance for Diffusion Models (NeurIPS 2024 · similarity 0.87)
- Token Perturbation Guidance for Diffusion Models (NeurIPS 2025 · similarity 0.86)
- Transfer Learning for Diffusion Models (NeurIPS 2024 · similarity 0.86)
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.86)
- Provable Efficiency of Guidance in Diffusion Models for General Data Distribution (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.