Research map / Image generation / Autoregressive and latent generation

Instruct Where the Model Fails: Generative Data Augmentation via Guided Self-contrastive Fine-tuning

AAAI 2025 · Image generation › Autoregressive and latent generation · approach: Synthetic training data from generators · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a task-aware generative data augmentation where a GPT-4o VLM coordinator detects misalignment between generated and original failed images, and a LoRA-finetuned Stable Diffusion is contrastively tuned at selected diffusion steps between epochs, evaluated on few-shot class-incremental learning.

Problem

Generative data augmentation with Stable Diffusion trained on web data produces biased, misaligned images that hurt downstream classifiers, since the generator is unaware of where the downstream model fails. This is studied in few-shot class-incremental learning.

Approach

Use downstream model failures as feedback: a VLM captioner compares correctly classified augmented images with misclassified original images, rewrites prompts for semantic mismatches, and fine-tunes the diffusion trajectory (early steps for semantics, late steps for details) with a self-contrastive SSIM triplet loss.

Key results

• On miniImageNet the method reaches 97.35 base, 95.23 last, and 96.54 average accuracy, versus 95.05/57.14/73.31 for DualPrompt.
• On CUB200 it reports 79.63 average accuracy, below PriViLege's 77.50 only in the sense of being higher; it is best on average but second-best on last accuracy (73.82 vs 75.08).
• Ablation on miniImageNet: fixed SD raises last accuracy from 9.87 to 48.37, and adding VLM re-captioning raises it to 94.87.

Measured effects reported in the paper

Surprising findings

• Fixed-prompt Stable Diffusion augmentation with biased content still substantially improves last-session accuracy by making the network converge.
• GPT-4o zero-shot classification on CUB200 achieves only about 56.8% accuracy, so the VLM alone is insufficient for fine-grained detail.

Limitations

• Performance on CUB200 is weaker on last-session accuracy, attributed to fine-grained bird details being hard for the VLM captioner.
• Relies on proprietary GPT-4o as the VLM coordinator, incurring API cost and reproducibility concerns.
• Evaluated only on few-shot class-incremental learning with small benchmarks.

Experimental setup

Datasets: miniImageNet CUB200 CIFAR-100 LAION (pretraining of Stable Diffusion, not used directly)

Benchmarks: miniImageNet CUB200 CIFAR-100

Metrics: base session accuracy last session accuracy average accuracy across sessions

Baselines: Fine-Tuning + Proto CEC L2P DualPrompt NC-FSCIL WaRP PriViLege Stable Diffusion fixed augmentation GPT-4o zero-shot

Builds on: Stable Diffusion LoRA classifier-free guidance SSIM

Closest papers by meaning

• Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.87)
• UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation (NeurIPS 2025 · similarity 0.87)
• The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.87)
• Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.86)
• Data Augmentation with Diffusion for Open-Set Semi-Supervised Learning (NeurIPS 2024 · similarity 0.86)
• Boosting Alignment for Post-Unlearning Text-to-Image Generative Models (NeurIPS 2024 · similarity 0.86)