Research map / Image generation / Editing and style transfer
Foundation Cures Personalization: Improving Personalized Models’ Prompt Consistency via Hidden Foundation Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeCure, a training-free method that uses a personalization model's identity-free (foundation) denoising branch to inject well-aligned facial attributes into the personalized generation via foundation-aware self-attention with attribute masks and asymmetric prompt guidance.
Problem
Facial personalization models using identity embeddings in cross-attention degrade prompt consistency (attribute control such as hair, expression, accessories), while identity fidelity must be preserved.
Approach
Dual inference with and without identity embedding reveals that the foundation model retains precise attribute control; FASA concatenates the foundation branch's self-attention keys/values into the personalized branch, masked to attribute regions, leaving cross-attention untouched, and APG restores abstract attributes like expressions via inversion with identity-free prompts.
Key results
- FastComposer + FreeCure raises PC from 18.14% to 21.02% and hMean from 25.55 to 27.80.
- PuLID (FLUX) + FreeCure improves PC from 22.42% to 24.78% (+10.53%) while IF drops 3.15%.
- Across seven baselines, FreeCure improves PC by 3.55% to 15.91% with identity fidelity drops of at most about 5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FreeCure to facial personalization models | Eight personalization baselines across SDv1.5, SDXL, and FLUX · Facial personalization evaluation | PC (%): FastComposer 18.14 -> 21.02; Face-Diffuser 20.67 -> 22.48; Face2Diffusion 21.92 -> 23.26; InstantID 21.89 -> 23.62; Phot | Helped |
| Add FreeCure to facial personalization models | Eight personalization baselines across SDv1.5, SDXL, and FLUX · Facial personalization evaluation | PC × IF (hMean): FastComposer 25.55 -> 27.80; Face-Diffuser 30.52 -> 32.32; Face2Diffusion 28.31 -> 29.20; InstantID 32.61 -> 34.21; Phot | Helped |
| Add FreeCure to facial personalization models | Eight personalization baselines across SDv1.5, SDXL, and FLUX · Facial personalization evaluation | IF (%): FastComposer 43.19 -> 41.02; Face-Diffuser 58.34 -> 57.51; Face2Diffusion 39.98 -> 39.23; InstantID 63.94 -> 62.01; Phot | Hurt |
| Add FASA without APG (ablation) | SDv1.5, SDXL, and FLUX personalization models · Personalization models grouped by foundation model | PC (%): SDv1.5 21.83; SDXL 24.55; FLUX 24.50 | Helped |
| Add APG without FASA (ablation) | SDv1.5, SDXL, and FLUX personalization models · Personalization models grouped by foundation model | PC (%): SDv1.5 20.58; SDXL 23.97; FLUX 24.09 | Helped |
| Use FASA only in upsampling blocks (ablation) | Denoising model · Facial attribute enhancement | Prompt consistency | Helped |
| Add FASA to middle and downsampling blocks when upsampling blocks already use FASA (ablation) | Denoising model · Facial attribute enhancement | Prompt consistency | No clear effect |
| Vary the FASA scaling factor ω (ablation) | FreeCure · Facial attribute enhancement | Generated face quality: ω=2.0 | Mixed |
| Vary the APG starting timestep γ (ablation) | FreeCure · Facial attribute enhancement | Identity fidelity and attribute improvement: γ=0.5 | Mixed |
Surprising findings
- Identity embeddings suppress attention to other prompt tokens, so deactivating identity embedding restores attribute control that the personalized model overrides.
- Injecting foundation attention into cross-attention layers quickly destroys identity fidelity, motivating modifying only self-attention.
Limitations
- Performance bounded by the inherent bias and capability of the underlying personalization models.
- Struggles with transparent objects such as glass bottles and shows some attribute entanglement between FD and PD branches.
- Not explored on auto-regressive personalization architectures.
Experimental setup
Datasets: CelebA-HQ
Benchmarks: CelebA-HQ custom 50-identity facial prompt set
Metrics: CLIP-T prompt consistency MTCNN+FaceNet identity fidelity LPIPS face diversity PC x IF harmonic mean
Baselines: FastComposer Face-Diffuser Face2Diffusion InstantID PhotoMaker PuLID InfiniteYou
Builds on: OminiControl Face2Diffusion PhotoMaker PuLID
Closest papers by meaning
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.87)
- PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium (AAAI 2025 · similarity 0.86)
- DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation (CVPR 2024 · similarity 0.86)
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network (AAAI 2025 · similarity 0.86)
- PersonaHOI: Effortlessly Improving Face Personalization in Human-Object Interaction Generation (CVPR 2025 · similarity 0.85)
- Visual Persona: Foundation Model for Full-Body Human Customization (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.