Research map / Image generation / Text-to-image diffusion
FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SuDe, a plug-and-play regularizer that models a subject as a derived class of its category by penalizing generated images that fail to classify as the category under the diffusion model's implicit classifier, improving attribute-related one-shot subject-driven generation.
Problem
One-shot subject-driven text-to-image generation fails on attribute-related prompts (e.g., running, burning, golden material) because a single example image does not supply category-level public attributes that the subject should inherit.
Approach
Borrowing the object-oriented derived-class concept, the subject is treated as a derived class of its semantic category; a Subject Derivation Regularization loss, computed from the diffusion model's implicit classifier (via conditional vs. unconditional noise predictions) with a loss-truncation threshold, pushes subject-generated images to be classified as the category so the subject inherits publ
Key results
- On ViCo with SD-v1.4, SuDe raises BLIP-T from 39.1 to 43.3 (+4.2) and the flexible-ws variant reaches 44.4 (+5.3).
- On DreamBooth with SD-v1.5, SuDe raises BLIP-T from 41.8 to 43.3 (+1.5) while DINO-I stays roughly unchanged (64.5 to 63.3).
- On Custom Diffusion with SD-v1.5, SuDe raises BLIP-T by +2.0 and the flexible-ws variant by +2.3, with subject fidelity (DINO-I) roughly preserved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SuDe to ViCo | ViCo · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | BLIP-T: 39.1 -> 43.3; 40.7 -> 43.3 | Helped |
| Add SuDe to Custom Diffusion | Custom Diffusion · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | BLIP-T: 45.2 -> 46.1; 44.6 -> 46.6 | Helped |
| Add SuDe to DreamBooth | DreamBooth · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | BLIP-T: 42.1 -> 43.3; 41.8 -> 43.3 | Helped |
| Add SuDe to ViCo | ViCo · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | DINO-I: 53.5 -> 56.8; 55.7 -> 59.4 | Helped |
| Add SuDe to Custom Diffusion | Custom Diffusion · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | DINO-I: 59.6 -> 59.1; 59.8 -> 60.0 | Mixed |
| Add SuDe to DreamBooth | DreamBooth · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | DINO-I: 59.7 -> 59.9; 64.5 -> 63.3 | Mixed |
| Use flexible SuDe weight ws, selecting from [0.5, 1.0, 2.0] · ws (ablation) | ViCo w/ SuDe · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | BLIP-T: +5.3%; +3.9% | Helped |
| Use flexible SuDe weight ws, selecting from [0.5, 1.0, 2.0] · ws (ablation) | Custom Diffusion w/ SuDe · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | BLIP-T: +1.1%; +2.3% | Helped |
| Use flexible SuDe weight ws, selecting from [0.5, 1.0, 2.0] · ws (ablation) | DreamBooth w/ SuDe · DreamBench attribute-related prompts, SD-v1.4 and SD-v1.5 | BLIP-T: +3.2%; +2.0% | Helped |
| Remove loss truncation from SuDe (ablation) | Custom Diffusion w/ SuDe · Training generations | generation stability | Hurt |
| Increase SuDe weight ws beyond an appropriate range (ablation) | DreamBooth w/ SuDe · Generated images across example subjects | subject fidelity | Hurt |
Surprising findings
- Without loss truncation, generations distort at epoch 2 and collapse by epoch 3, showing that unbounded optimization of the derivation loss damages the diffusion model's semantic space.
Limitations
- Requires per-baseline tuning of the loss weight ws (0.4, 1.5, 2.0), since a fixed value does not suit all baselines.
- Gains on attribute alignment are modest for some baselines (e.g., +0.9 BLIP-T for Custom Diffusion on SD-v1.4), and the attribute metric relies on BLIP-T rather than human judgment.
Experimental setup
Datasets: DreamBench
Benchmarks: DreamBench
Metrics: CLIP-I DINO-I CLIP-T BLIP-T
Baselines: DreamBooth Custom Diffusion ViCo
Builds on: DreamBooth Custom Diffusion ViCo Stable Diffusion DDPM
Closest papers by meaning
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network (AAAI 2025 · similarity 0.82)
- Limitations of Face Image Generation (AAAI 2024 · similarity 0.81)
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.81)
- Tree of Attributes Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.81)
- SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation (CVPR 2024 · similarity 0.80)
- Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning (AAAI 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.