Research map / Image generation / Fast diffusion sampling
Training-Free Constrained Generation With Stable Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Integrates proximal Langevin dynamics with pretrained Stable Diffusion by correcting latents via decoder-space constraint projections, training-free, with theoretical feasibility/fidelity bounds and experiments on microstructure, metamaterial, and copyright tasks.
Problem
Stable diffusion models generate outputs that do not strictly satisfy physical, functional, or copyright constraints; existing inference-time constraint methods operate in pixel space and do not apply to latent diffusion, and latent-space methods rely on measurement operators or learned penalties.
Approach
Apply proximal Langevin dynamics to the latent of a frozen Stable Diffusion model: after each noisy gradient step, a proximal/projection correction is computed in image space through the frozen decoder and backpropagated to the latent, with black-box simulators handled by finite-difference (DPO-style) gradient estimates.
Key results
- Latent constrained model satisfies porosity constraints exactly with competitive FID, while conditional diffusion only had 31.6% of samples within 10% porosity error.
- With five DPO steps, the method achieves a 4.6x MSE improvement over the prior state-of-the-art and 5.1x over conditional stable diffusion on metamaterial inverse design.
- On copyright-safe generation, the latent constrained model generates the protected character 10% of the time versus 33% for the conditional baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply latent constraint correction instead of conditional Stable Diffusion | Stable Diffusion · Metamaterial inverse design | Stress-strain response MSE: 4.6x improvement | Helped |
| Apply latent constraint correction instead of conditional Stable Diffusion | Stable Diffusion · Metamaterial inverse design | Stress-strain response MSE: 5.1x improvement | Helped |
| Use latent constraint correction rather than text conditioning | Stable Diffusion · Microstructure generation | Porosity error <10% success rate: 31.6\% of the samples had a porosity error less than 10\%; the proposed method satisfies the porosity constraints exactl | Helped |
| Use latent constraint correction rather than conditional Stable Diffusion | Stable Diffusion · Microstructure generation | FID: 10.8 | Mixed |
| Apply latent constraint correction | Stable Diffusion · Copyright-safe generation | Protected-character generation rate: 33\% to 10\% | Helped |
| Apply gradient-based latent constraint correction | Stable Diffusion · Copyright-safe generation | FID | No clear effect |
Surprising findings
- Image-space Projected Diffusion Models failed to produce reasonable samples at 1024x1024, forcing 64x64 sampling plus upscaling.
- Learned latent corrector and image-space re-encoding correction baselines produced samples far from the training distribution.
Limitations
- Classifier-based constraints depend on classifier accuracy and only give statistical guarantees.
- Only applied to image latent diffusion; video/temporal constraints not explored.
- Non-differentiable simulators require costly finite-difference estimates and slow CPU simulation.
Experimental setup
Datasets: microstructure dataset (porous material images) periodic stochastic cellular structures stress-strain dataset Tom and Jerry / Mickey Mouse copyright image set (8,000 images)
Benchmarks: microstructure porosity generation metamaterial stress-strain inverse design (Abaqus simulator) copyright-constrained generation (Mickey Mouse classifier)
Metrics: FID constraint satisfaction rate MSE to target stress-strain curve porosity error within 10%
Baselines: Conditional Stable Diffusion Projected Diffusion Models (PDM) Image Space Correction Learned Latent Corrector Unconstrained Stable Diffusion
Builds on: Projected Diffusion Models Stable Diffusion Differentiable perturbed optimizer Augmented Lagrangian method ReSample
Closest papers by meaning
- Constrained Synthesis with Projected Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Physics-Informed Diffusion Models (ICLR 2025 · similarity 0.89)
- Scalable Diffusion for Materials Generation (ICLR 2024 · similarity 0.86)
- Composition and Alignment of Diffusion Models using Constrained Learning (NeurIPS 2025 · similarity 0.85)
- Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints (NeurIPS 2025 · similarity 0.85)
- Constrained Generative Modeling with Manually Bridged Diffusion Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.