Research map / Image generation / Text-to-image diffusion
High-fidelity Person-centric Subject-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Face-diffuser, a tuning-free person-centric subject-to-image pipeline that uses two separately fine-tuned Stable Diffusion models (a text-driven scene model and a subject-augmented model) blended per-pixel during DDIM sampling via saliency-adaptive noise fusion driven by classifier-free guidance responses.
Problem
Subject-driven text-to-image methods that jointly fine-tune one diffusion model for scene and person generation suffer from a training imbalance (persons need long tuning, which causes forgetting of scene priors) and a quality compromise in person fidelity.
Approach
Decouple scene and person generation into two specialized diffusion models and collaborate at sampling time in three stages (scene construction, subject-scene fusion, subject enhancement), where a per-pixel mask is computed from the absolute, smoothed CFG responses of each model (saliency-adaptive noise fusion).
Key results
- Single-subject identity preservation 0.708 and prompt consistency 0.325 for Face-diffuser vs 0.605/0.228 for Subject-Diffusion (best baseline in IP).
- Multi-subject IP 0.593 and PC 0.319, beating Fastcomposer (0.461/0.235) by 0.132 IP and 0.084 PC.
- Ablations show removing the subject enhancement stage (beta=1) degrades person fidelity; optimal alpha=0.3 and beta=0.6 chosen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Face-Diffuser instead of Fastcomposer | Person-centric subject-to-image generation · Single-benchmark | IP: 0.514 -> 0.708 | Helped |
| Use Face-Diffuser instead of Fastcomposer | Person-centric subject-to-image generation · Single-benchmark | PC: 0.243 -> 0.325 | Helped |
| Use Face-Diffuser instead of Fastcomposer | Person-centric subject-to-image generation · FFHQ-face test set | IP: 0.461 -> 0.593 | Helped |
| Use Face-Diffuser instead of Fastcomposer | Person-centric subject-to-image generation · FFHQ-face test set | PC: 0.235 -> 0.319 | Helped |
| Add the middle subject-scene fusion stage (SNF), rather than setting β=α (ablation) | Face-Diffuser · Person-centric image generation | person-scene coherence | Helped |
| Vary α from 0 to 0.3 while β=α (ablation) | Face-Diffuser · Person-centric image generation | semantic scene quality | Helped |
| Set α=0 (no initial TDM scene construction) (ablation) | Face-Diffuser · Person-centric image generation | subject overfitting | Hurt |
| Increase β from 0.4 toward 0.6 with α=0.3 (ablation) | Face-Diffuser · Person-centric image generation | person-scene coherence | Helped |
| Set β=1, removing the last subject-enhancement stage (ablation) | Face-Diffuser · Person-centric image generation | character fidelity | Hurt |
Surprising findings
- Fine-tuning Fastcomposer and Subject-Diffusion longer raises training-set prompt consistency while test-set prompt consistency drops, indicating scene-prior forgetting.
Limitations
- Evaluation on a single-benchmark of 15 subjects and an FFHQ-face test split; qualitative comparisons only on unseen persons.
- Hyperparameters alpha and beta are tuned by visual inspection of ablations.
- Requires training two separate diffusion models, doubling the training effort.
Experimental setup
Datasets: FFHQ-face FFHQ-wild Single-benchmark
Benchmarks: Single-benchmark FFHQ-face test set
Metrics: identity preservation (MTCNN + FaceNet similarity) prompt consistency (CLIP-L/14 image-text similarity)
Baselines: ELITE DreamBooth Custom-Diffusion Subject-Diffusion Fastcomposer
Builds on: Stable Diffusion DDIM classifier-free guidance Fastcomposer Subject-diffusion
Closest papers by meaning
- IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation (NeurIPS 2024 · similarity 0.87)
- Limitations of Face Image Generation (AAAI 2024 · similarity 0.87)
- Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation (CVPR 2024 · similarity 0.86)
- PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention (CVPR 2025 · similarity 0.86)
- MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts (NeurIPS 2024 · similarity 0.86)
- Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.