Research map / Image generation / Editing and style transfer
NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoiseCLR, an unsupervised contrastive method that learns disentangled editing directions in a frozen Stable Diffusion v1.5 from about 100 unlabeled images per domain.
Problem
Discovering interpretable, disentangled latent editing directions in large text-to-image diffusion models without text prompts, labels, or masks, which existing diffusion methods do not handle well for Stable Diffusion.
Approach
Learn K direction embeddings by contrasting noise-prediction feature divergences across unlabeled images: edits from the same direction attract, edits from different directions repel. The directions act as conditional embeddings on a frozen denoiser, so they can be composed within and across domains.
Key results
- User study (50 participants, 1-5 scale): NoiseCLR scored 2.65 edit quality and 3.05 disentanglement, above Cycle-Diffusion (1.87, 2.73) and SEGA (1.52, 1.81).
- LPIPS on Age/Mustache/Gender/Race edits: NoiseCLR gave lower or comparable values (e.g., 0.13 race) than Composable Diffusion, Cycle-Diffusion and SEGA; it was 0.17 on age, versus 0.10 for Cycle-Diffusion.
- Training on one domain takes about 7 hours for 100 face directions on a single NVIDIA L40 GPU; an edit takes about 5 seconds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NoiseCLR instead of the compared editing methods | Diffusion-model image editing · User study | edit quality rating: 2.65 vs 2.19 (Composable Diffusion), 1.20 (Unsupervised Concept Discovery), 1.52 (SEGA), 1.87 (Cycle-Diffusion) | Helped |
| Use NoiseCLR instead of the compared editing methods | Diffusion-model image editing · User study | disentanglement rating: 3.05 vs 2.15 (Composable Diffusion), 1.28 (Unsupervised Concept Discovery), 1.81 (SEGA), 2.73 (Cycle-Diffusion) | Helped |
| Use NoiseCLR instead of Composable Diffusion and SEGA | Image editing · Age, Mustache, Gender, and Race edits | LPIPS: Age: 0.17 vs 0.19, 0.11; Mustache: 0.17 vs 0.40, 0.23; Gender: 0.20 vs 0.42, 0.27; Race: 0.13 vs 0.40, 0.27 | Mixed |
| Use NoiseCLR instead of Cycle-Diffusion | Image editing · Age, Mustache, Gender, and Race edits | LPIPS: Age: 0.17 vs 0.10; Mustache: 0.17 vs 0.21; Gender: 0.20 vs 0.23; Race: 0.13 vs 0.26 | Mixed |
Surprising findings
- Applying the 'Child' edit noticeably shifts race-based edits (Indian, Asian), which the authors attribute to internal biases in Stable Diffusion.
Limitations
- Capabilities are bounded by the training data of Stable Diffusion and by CLIP's biases.
- Some directions remain entangled, e.g., background and car-shape edits.
- Potential for misuse in malicious image synthesis.
Experimental setup
Datasets: FFHQ AFHQ-Cats Stanford Cars paintings subset
Benchmarks: CLIP re-scoring analysis LPIPS user study on Amazon MTurk
Metrics: CLIP classification probability change LPIPS user-rated edit quality user-rated disentanglement
Baselines: Cycle-Diffusion SEGA Composable Diffusion Unsupervised Concept Discovery
Builds on: LatentCLR classifier-free guidance DDIM inversion Stable Diffusion
Closest papers by meaning
- Unsupervised Region-Based Image Editing of Denoising Diffusion Models (AAAI 2025 · similarity 0.92)
- Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing (NeurIPS 2024 · similarity 0.91)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
- UniFL: Improve Latent Diffusion Model via Unified Feedback Learning (NeurIPS 2024 · similarity 0.89)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.88)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.