Research map / Image generation / Autoregressive and latent generation
UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UtilGen, a utility-centric generative data augmentation framework that uses a meta-learned sample-weight network as a utility signal to drive DPO fine-tuning of Stable Diffusion and instance-level optimization of prompt embeddings and initial noise.
Problem
Generative data augmentation mostly optimizes intrinsic data attributes (fidelity, diversity) and ignores task-specific utility, which varies across downstream tasks and architectures; evaluating utility by full retraining is too costly.
Approach
A meta-learned weight allocation network scores each synthetic sample's utility from its classification loss; these scores build preference pairs for Diffusion-DPO (model-level) and drive gradient-based prompt embedding optimization and CFG-discrepancy-based initial noise optimization (instance-level), forming a feedback loop between generation and training.
Key results
- Training on synthetic data only, UtilGen averages 37.17% accuracy vs 33.30% for DataDream (+3.87%) across eight datasets with ResNet-50.
- With real + synthetic data, UtilGen averages 62.04% vs 58.67% for DataDream (+3.37% per table; text states +3.54%).
- On ImageNet-100, UtilGen at 5x synthetic data reaches 40.94% vs 35.48% for DataDream and surpasses the real-data-only baseline of 38.58% in the joint table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use utility-guided UtilGen instead of DataDream | ResNet-50 training on synthetic data only · Eight benchmarks, 5× synthetic data | average accuracy: 33.30 -> 37.17; +3.87 | Helped |
| Use utility-guided UtilGen instead of DataDream | ResNet-50 training on real + synthetic data · Eight benchmarks, 5× synthetic data | average accuracy: 58.67 -> 62.04 | Helped |
| Increase the UtilGen synthesis budget | ResNet-50 · ImageNet-100-S, synthetic data only | accuracy: 18.04 -> 28.52 -> 40.94 | Helped |
| Use UtilGen rather than Stable Diffusion v2.1 | Synthetic data generation · ImageNet-100 | mean intra-class diversity: 0.5815 -> 0.6054 | Helped |
| Use synthetic data generated by UtilGen rather than DataDream | WideResNet downstream model · ImageNet-100 | accuracy: 31.76 -> 36.40 | Helped |
| Use synthetic data generated by UtilGen rather than DataDream | CLIP downstream model · ImageNet-100 | accuracy: 71.42 -> 72.14 | Helped |
| Use UtilGen rather than DataDream | ResNeXt-50 · ImageNet-100 | accuracy: 31.24 -> 37.62 | Helped |
| Use UtilGen rather than DataDream | WideResNet-50 · ImageNet-100 | accuracy: 31.76 -> 37.82 | Helped |
| Use UtilGen rather than DataDream | MobileNetV2 · ImageNet-100 | accuracy: 35.48 -> 40.59 | Helped |
| Add noise optimization (ablation) | UtilGen generation pipeline without MLCO or prompt optimization · ImageNet-100 | accuracy: 27.96 -> 28.68 | Helped |
| Add MLCO (ablation) | UtilGen generation pipeline without prompt or noise optimization · ImageNet-100 | accuracy: 27.96 -> 36.42 | Helped |
| Add prompt optimization (ablation) | UtilGen generation pipeline without MLCO or noise optimization · ImageNet-100 | accuracy: 27.96 -> 37.96 | Helped |
Surprising findings
- ResNet-50 trained solely on 3x synthetic data from UtilGen surpasses its real-data-trained counterpart on several benchmarks.
- Ablation on ImageNet-100 shows MLCO alone combined with noise optimization (32.08%) is lower than MLCO with prompt optimization (39.73% with MLCO+prompt), suggesting component interactions are non-additive.
Limitations
- Dual-level optimization incurs computational overhead over conventional augmentation methods.
- Performance depends on the base generative model's ability to produce viable initial samples.
- Synthesis is guided by a small set of real images, so generated data may inherit and amplify biases.
Experimental setup
Datasets: ImageNet-1k-Subset ImageNet-100-Subset Caltech 101 Oxford Pets Food101-Subset Flowers 102 CUB-200-2011 DTD
Benchmarks: ImageNet-1k-Subset ImageNet-100-Subset Caltech 101 Oxford Pets Food101-Subset Flowers 102 CUB-200-2011 DTD
Metrics: top-1 accuracy average accuracy intra-class diversity (CLIP cosine distance) data influence (influence functions)
Baselines: Stable Diffusion v2.1 GIF GAP DataDream
Builds on: Diffusion-DPO textual inversion GIF DataDream GAP
Closest papers by meaning
- Instruct Where the Model Fails: Generative Data Augmentation via Guided Self-contrastive Fine-tuning (AAAI 2025 · similarity 0.87)
- DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data (CVPR 2024 · similarity 0.87)
- AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition (NeurIPS 2025 · similarity 0.87)
- Salient Concept-Aware Generative Data Augmentation (NeurIPS 2025 · similarity 0.86)
- The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.85)
- Improving the Scaling Laws of Synthetic Data with Deliberate Practice (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.