Research map / Image generation / Text-to-image diffusion
Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Subject-Agnostic Guidance (SAG), a training-free dual classifier-free guidance scheme that uses a subject-agnostic condition early in denoising to improve text alignment in subject-driven text-to-image synthesis.
Problem
Subject-driven text-to-image synthesis is dominated by reference subject embeddings, so the output often ignores attributes specified in the text prompt (content ignorance).
Approach
Construct a subject-agnostic condition by replacing the subject token with a generic noun (or zeroing the subject embedding) and apply dual classifier-free guidance that uses this condition with a time-varying weight in early denoising steps, then restores the subject-aware condition for later steps; requires no retraining.
Key results
- On ELITE, ELITE-SAG improves CLIP-T from 0.342 to 0.344, CLIP-I from 0.751 to 0.790, and DINO from 0.586 to 0.671.
- In the user study, 56% to 64% of raters preferred SAG-enhanced outputs on subject alignment versus baselines, and 68% to 80% on text alignment, with Textual Inversion at 76% and ELITE at 80% for text alignment.
- SAG applies to Textual Inversion, ELITE, SuTI and DreamSuTI without retraining, with T and r adjustable at inference time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Subject-Agnostic Guidance to ELITE | ELITE · ELITE evaluation | CLIP-T: 0.342 -> 0.344 | Helped |
| Add Subject-Agnostic Guidance to ELITE | ELITE · ELITE evaluation | CLIP-I: 0.751 -> 0.790 | Helped |
| Add Subject-Agnostic Guidance to ELITE | ELITE · ELITE evaluation | DINO: 0.586 -> 0.671 | Helped |
| Apply SAG with a smaller guidance timing T (ablation) | DreamSuTI · Style customization | Style alignment | Helped |
| Increase guidance timing T (ablation) | DreamSuTI · Style customization | Subject alignment | Helped |
| Lower guidance weight r (ablation) | DreamSuTI · Style customization | Style alignment | Helped |
| Add Subject-Agnostic Guidance | Textual Inversion · Subject-driven image synthesis | Text alignment | Helped |
| Add Subject-Agnostic Guidance | SuTI · Subject-driven image synthesis | Text alignment | Helped |
| Add Subject-Agnostic Guidance | DreamSuTI · Style customization | Style alignment | Helped |
Surprising findings
- Subject-agnostic guidance improved subject fidelity (DINO) as well as text alignment on ELITE, not only text alignment.
Limitations
- Output quality is bounded by the underlying generation model, which struggles with uncommon content.
- Evaluation of ELITE-SAG relies on an internal dataset that is not released.
- Societal risk of misuse for misleading imagery is noted.
Experimental setup
Datasets: internal text-image dataset meta-dataset (dogs and cats subset)
Benchmarks: ELITE comparison set (qualitative) Textual Inversion/SuTI/DreamSuTI qualitative sets
Metrics: CLIP-T CLIP-I DINO user preference rate
Baselines: DreamBooth Textual Inversion ELITE Stable Diffusion
Builds on: classifier-free guidance Textual Inversion ELITE SuTI DreamBooth
Closest papers by meaning
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.86)
- Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (CVPR 2025 · similarity 0.85)
- Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation (ICLR 2025 · similarity 0.84)
- CustomContrast: A Multilevel Contrastive Perspective for Subject-Driven Text-to-Image Customization (AAAI 2025 · similarity 0.84)
- MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.