Research map / Image generation / Fast diffusion sampling
Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskUNet, which learns timestep- and sample-dependent binary masks over frozen Stable Diffusion U-Net weights, trained either via diffusion loss with a small MLP mask generator or training-free via reward-model guidance.
Problem
Diffusion U-Nets reuse the same parameters across all denoising steps, forcing one network to learn both structure (early steps) and texture (late steps), which may limit generation quality.
Approach
Rather than updating U-Net weights, learn a binary mask over pretrained U-Net weights that varies with timestep and prompt/sample, using either a diffusion-loss-trained MLP mask generator or a training-free per-step optimization guided by ImageReward and HPSv2 reward models.
Key results
- On COCO 2017, MaskUNet reaches FID-5k 21.88 vs 23.39 for SD 1.5 and 23.18 for LoRA, with similar CLIP scores (0.33).
- On T2I-CompBench, SynGen+MaskUNet improves color BLIP-VQA from 0.6288 to 0.6989 over SynGen.
- On GenEval, SynGen+MaskUNet raises overall score from 0.43 to 0.50 and color attribution from 0.05 to 0.26 (SynGen alone is 0.05 for color attribution per the table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add timestep- and sample-dependent parameter masks | SD 1.5 · COCO 2014 | FID-30k: 12.85 -> 11.72 | Helped |
| Add timestep- and sample-dependent parameter masks | SD 1.5 · COCO 2017 | FID-5k: 23.39 -> 21.88 | Helped |
| Add timestep- and sample-dependent parameter masks | SD 1.5 · COCO 2014 | CLIP: 0.32 -> 0.32 | No clear effect |
| Add timestep- and sample-dependent parameter masks | SD 1.5 · COCO 2017 | CLIP: 0.33 -> 0.33 | No clear effect |
| Add MaskUNet to SynGen | SynGen · T2I-CompBench color | BLIP-VQA: 0.6288 -> 0.6989 | Helped |
| Add MaskUNet to SynGen | SynGen · T2I-CompBench texture | BLIP-VQA: 0.5796 -> 0.6209 | Helped |
| Add MaskUNet to SynGen | SynGen · T2I-CompBench shape | BLIP-VQA: 0.3881 -> 0.4644 | Helped |
| Add MaskUNet to SynGen | SynGen · GenEval | Overall: 0.43 -> 0.50 | Helped |
| Add timestep embedding to the mask generator (ablation) | MaskUNet · COCO 2017 | FID: 22.30 -> 21.88 | Helped |
| Add sample input to the mask generator (ablation) | MaskUNet · COCO 2017 | FID: 22.14 -> 21.88 | Helped |
| Add timestep embedding to the mask generator (ablation) | MaskUNet · COCO 2017 | CLIP: 0.32 -> 0.33 | Helped |
| Add sample input to the mask generator (ablation) | MaskUNet · COCO 2017 | CLIP: 0.32 -> 0.33 | Helped |
Surprising findings
- Full fine-tuning of the U-Net degrades FID relative to SD 1.5 on COCO 2014 (14.06 vs 12.85), suggesting overfitting.
Limitations
- Dynamic masking does not enable learning of new knowledge.
- Training-free semantic binding gains are not uniform (e.g., MaskUNet alone underperforms SD 2.0 on color).
Experimental setup
Datasets: Laion-art subset (20.1k pairs) COCO 2014 COCO 2017 T2I-CompBench GenEval
Benchmarks: COCO 2014 COCO 2017 T2I-CompBench GenEval
Metrics: FID CLIP score BLIP-VQA GenEval score user preference
Baselines: SD 1.5 Full Fine-tune LoRA SD 2.0 SynGen Attend-and-excite DreamBooth Textual Inversion ReVersion Text2Video-Zero
Builds on: Stable Diffusion 1.5 LoRA ReNO SynGen FreeU
Closest papers by meaning
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.90)
- ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation (NeurIPS 2025 · similarity 0.90)
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.90)
- MC-DiT: Contextual Enhancement via Clean-to-Clean Reconstruction for Masked Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Effective and Efficient Masked Image Generation Models (ICML 2025 · similarity 0.89)
- Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.