Research map / Image generation / Text-to-image diffusion
OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OneActor, a one-shot tuning method that trains only a small projector to produce cluster-offset semantic embeddings, enabling prompt-driven consistent subject generation on SDXL without backbone tuning.
Problem
Text-to-image diffusion models generate inconsistent appearances of the same subject across images; existing consistent-subject methods either need external images/datasets, expensive backbone fine-tuning, or extra modules that slow inference.
Approach
Formulates consistent subject generation as cluster-guided score estimation (target attraction minus auxiliary exclusion) and implements it by training a lightweight projector that maps U-Net encoder features and prompt embeddings into a semantic offset added to the base word embedding, with auxiliary samples for regularization and semantic interpolation/cluster guidance at inference.
Key results
- Pareto-dominates IP-Adapter, Textual Inversion and BLIP-Diffusion in the TheChosenOne setting on identity consistency vs prompt similarity.
- Achieves DINO-fg 0.756 (highest) vs DreamBooth-LoRA 0.735 in subject consistency table.
- Tuning takes about 5 minutes versus 20 minutes for TheChosenOne on a single A100, a claimed at least 4x speedup; with eta2=0 inference matches SDXL at 8.3 s per image.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OneActor instead of the baselines | SDXL subject generation · Single-subject generation | DINO-fg: 0.735±0.078 -> 0.756±0.073 | Helped |
| Use OneActor instead of DreamBooth-LoRA | SDXL subject generation · Single-subject generation | CLIP-T-score: 0.594±0.058 -> 0.620±0.038 | Helped |
| Add auxiliary loss Laux (ablation) | OneActor tuning · Character-centric generation | image quality and diversity | Helped |
| Add auxiliary losses Laux and Laver (ablation) | OneActor tuning · Character-centric generation | identity consistency | Helped |
| Add auxiliary losses Laux and Laver (ablation) | OneActor tuning · Character-centric generation | content diversity | Helped |
| Increase semantic scale v (ablation) | OneActor inference · Subject-centric generation | subject consistency | Helped |
| Increase semantic scale v too much (ablation) | OneActor inference · Subject-centric generation | content diversity | Hurt |
| Set semantic scale v to 0.8 (ablation) | OneActor inference · Subject-centric generation | consistency-diversity balance: v = 0.8 | Helped |
| Increase cluster guidance scale η1 (ablation) | OneActor inference · Subject-centric generation | prompt alignment | Helped |
| Increase cluster guidance scale η2 while keeping η1 − η2 = 7.5 (ablation) | OneActor inference · Subject-centric generation | consistent details | Helped |
| Apply cluster guidance to early denoising steps (ablation) | OneActor inference · Subject-centric generation | target sub-cluster navigation: steps 1-20 | Helped |
| Set η2 = 0 to avoid auxiliary cluster guidance (ablation) | OneActor inference · Subject-centric generation | image quality: η2 = 0 | Hurt |
Surprising findings
- Applying the auxiliary exclusion term is not required for good results: performance with eta2=0 is already satisfying, and eta2 mainly adds consistent details.
Limitations
- Struggles to generate more than about 4 subjects in one image (tie neglected in 5-subject case).
- Inherits entangled latent clusters of the base diffusion model, causing bias (e.g., umbrella biasing indoor scenes) and out-of-distribution failures for actions not in the sub-cluster (e.g., swimming).
- Consistency comparisons with TheChosenOne and ConsiStory rely on figures from their papers because no official code exists.
Experimental setup
Datasets: self-generated base images from SDXL per target prompt
Benchmarks: TheChosenOne setting ConsiStory setting custom 30-subject prompt set
Metrics: CLIP-I CLIP-T DreamSim CLIP-score DINO-fg CLIP-I-fg LPIPS-fg DINO-bg CLIP-I-bg LPIPS-bg
Baselines: Textual Inversion DreamBooth-LoRA BLIP-Diffusion IP-Adapter ELITE TheChosenOne ConsiStory
Builds on: classifier-free guidance latent diffusion models TheChosenOne ConsiStory
Closest papers by meaning
- Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion (CVPR 2025 · similarity 0.89)
- Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection (ICLR 2025 · similarity 0.89)
- Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation (NeurIPS 2024 · similarity 0.89)
- Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects (CVPR 2025 · similarity 0.89)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
- Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.