Research map / Image generation / Text-to-image diffusion

APT: Adaptive Personalized Training for Diffusion Models with Limited Data

CVPR 2025 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Adaptive Personalized Training (APT), which fine-tunes SDXL with LoRA on few reference images using an overfitting indicator for adaptive augmentation and loss weighting, plus feature mean/variance and cross-attention regularization to preserve prior knowledge.

Problem

Personalizing text-to-image diffusion models from limited reference images causes overfitting, which shifts noise predictions, disrupts the denoising trajectory, and degrades prior knowledge and text alignment.

Approach

An time-bin-wise overfitting indicator computed from EMA denoising-loss gaps between pretrained and fine-tuned models drives adaptive augmentation probability and loss down-weighting, combined with regularizing intermediate activation statistics and aligning cross-attention maps to the pretrained model.

Key results

• Full APT achieves CLIP-T 0.664 and HPSv2 0.288, versus 0.661 and 0.272 for DreamBooth baseline.
• APT reaches FID 41.967 and recall 0.738 against SDXL-generated references, better than DCO (52.298 FID) and DreamBooth (53.130 FID).
• In a 20-participant user study, 56.1% preferred APT versus 22.8% DreamBooth and 21.1% DCO.

Measured effects reported in the paper

Limitations

• Trade-off between prior preservation and learning new concepts; monster-toy example shows class prior dominates.
• Sensitive to hyperparameters such as the attention regularization weight lambda_attn, requiring per-concept tuning.
• Image similarity (DINOv2) is slightly lower than some baselines, which the authors attribute to broader scene context.

Experimental setup

Datasets: DreamBooth dataset Textual Inversion dataset MS COCO captions

Benchmarks: DreamBooth dataset Textual Inversion dataset MS COCO captions

Metrics: CLIP-T HPSv2 DINOv2 similarity FID precision recall user study preference

Baselines: DreamBooth Custom Diffusion DCO SDXL (prior)

Builds on: DreamBooth StyleGAN-ADA Direct Consistency Optimization

Closest papers by meaning

• Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.91)
• On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.89)
• PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.89)
• Efficient Personalization of Quantized Diffusion Model without Backpropagation (CVPR 2025 · similarity 0.88)
• Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models (NeurIPS 2026 · similarity 0.88)
• Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.88)