Research map / Image generation / Editing and style transfer
Interpreting the Weight Space of Customized Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a dataset of over 60,000 LoRA-customized identity diffusion models and models their weights as a PCA subspace (weights2weights), enabling sampling, linear editing, and single-image inversion of new identities.
Problem
Whether the weight space of many customized (personalized) diffusion models contains an interpretable, low-dimensional manifold that can be navigated like a GAN latent space to create new models rather than images.
Approach
Treat flattened LoRA weights of many identity-personalized diffusion models as data, model their manifold as a PCA linear subspace (w2w), find linear attribute directions via classifiers on weights, and invert single images by optimizing only PCA coefficients.
Key results
- w2w edits achieve ID score 0.45 on gender vs 0.33 for Concept Sliders and 0.39 for prompting, with LPIPS 0.31 (lower is better).
- Single-image w2w inversion reaches ID score 0.64±0.01, versus 0.43±0.03 for single-image DB-LoRA and 0.69±0.01 for multi-image DB-LoRA.
- In a 2AFC user study, w2w edits win 71.6% against Concept Sliders and 87.2% against prompting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use w2w weight-space edits instead of prompting or Concept Sliders | Personalized identity models · Gender, chubby, and eyes edits | ID Score: Gender: 0.39, 0.33 -> 0.45; Chubby: 0.29, 0.33 -> 0.45; Eyes: 0.52, 0.53 -> 0.72 | Helped |
| Use w2w weight-space edits instead of prompting or Concept Sliders | Personalized identity models · Gender, chubby, and eyes edits | LPIPS: Gender: 0.30, 0.39 -> 0.31; Chubby: 0.41, 0.38 -> 0.36; Eyes: 0.32, 0.30 -> 0.19 | Mixed |
| Use w2w weight-space edits instead of prompting or Concept Sliders | Personalized identity models · Gender, chubby, and eyes edits | CLIP Score: Gender: 1.98, 3.50 -> 4.13; Chubby: 1.12, 2.21 -> 2.16; Eyes: 0.17, 0.01 -> 0.59 | Mixed |
| Constrain single-image inversion to the w2w weight subspace | Identity inversion · FFHQ | ID Score: 0.43\pm0.03 -> 0.64\pm0.01 | Helped |
| Constrain inversion to the w2w weight subspace rather than use multi-image Dreambooth | Identity inversion · FFHQ | ID Score: \mathbf{0.69}\pm0.01 -> 0.64\pm0.01 | Hurt |
| Constrain inversion to the w2w weight subspace | Identity inversion · FFHQ | Opt. Time (s): 200 -> 55 | Helped |
| Use w2w instead of Celeb-Basis or IP-Adapter FaceID | Single-shot personalization · FFHQ | ID Score: 0.60\pm0.02, 0.62\pm0.02 -> 0.64\pm0.01 | Helped |
| Increase the number of identity models used to construct w2w (ablation) | w2w identity subspace · Identity edit directions | Absolute cosine similarity between edit directions | Helped |
| Increase the number of identity models used to construct w2w (ablation) | w2w identity subspace · 100 inverted FFHQ identities | ID Score | Helped |
| Use the first 100 principal components instead of the first 1000 to find edit directions (ablation) | w2w identity editing · Goatee edit | Semantic alignment and disentanglement | Hurt |
| Use the first 10,000 principal components instead of the first 1000 to find edit directions (ablation) | w2w identity editing · Goatee edit | Semantic alignment and disentanglement | Hurt |
| Optimize the first 1000 principal components instead of the first 10,000 during inversion (ablation) | w2w identity inversion · FFHQ identities | Identity preservation | Hurt |
Surprising findings
- Inversion into the w2w subspace (about 10k PC coefficients) is faster (55 s) than single-image DB-LoRA (200 s) and nearly matches multi-image DB-LoRA identity fidelity.
- Scaling the number of models reduces entanglement among CelebA attribute directions such as Male-Beard.
Limitations
- Span of w2w is limited by the identity models used; complex or out-of-distribution identities are projected onto the closest manifold point.
- Attribute directions are limited to CelebA labels and can entangle correlated attributes such as gender and facial hair.
- CLIP score is noisy for fine-grained facial attributes.
Experimental setup
Datasets: CelebA FFHQ ImageNet Stanford Cars
Benchmarks: CelebA attribute edits FFHQ inversion (100 identities)
Metrics: ID score (FaceNet similarity) LPIPS CLIP score user study win rate
Baselines: Prompting Concept Sliders DB-LoRA (single-image) DB-LoRA (multi-image) Celeb-Basis IP-Adapter FaceID
Builds on: Dreambooth LoRA Concept Sliders latent diffusion models GAN latent space linear directions
Closest papers by meaning
- Unsupervised Region-Based Image Editing of Denoising Diffusion Models (AAAI 2025 · similarity 0.88)
- Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.88)
- Isometric Representation Learning for Disentangled Latent Space of Diffusion Models (ICML 2024 · similarity 0.87)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.87)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.87)
- Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.