Research map / Image generation / Fast diffusion sampling
Diffusing Differentiable Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDRep, a training-free method that samples differentiable representations (SIRENs, panoramas, NeRFs) from pretrained Stable Diffusion 1.5 by pulling back the reverse PF-ODE to parameter space, with RePaint-style constraint conditioning.
Problem
Existing training-free diffrep generation methods (SDS, SJC) perform mode-seeking rather than true sampling, yielding oversmoothed, low-diversity 3D/image outputs, and ignore consistency constraints imposed by non-invertible renderers.
Approach
Pull back the reverse-time probability flow ODE through the render map using the correct tangent-vector pullback (J^T J)^{-1} J^T, solve per-step parameter updates by a small least-squares/Adam suboptimization, separate fixed noise from the noiseless rendered signal, and apply RePaint forward steps to enforce renderability constraints.
Key results
- On SIREN image generation, samples are nearly indistinguishable from SDv1.5 references (PSNR about 29.7 dB, LPIPS 0.044 at CFG 0).
- At CFG 10, SJC samples look very similar across seeds while the method and SD show significantly more diversity.
- A SIREN panorama takes 218 s and a NeRF about 24 min on one A6000, versus about 34 min for SJC's 10,000 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDRep instead of SJC to sample 8 image SIRENs | SIREN image generation · 8 image SIRENs | sampling time: 694 seconds -> 82 seconds | Helped |
| Use DDRep instead of SJC for NeRF generation | 3D NeRF generation · 3D NeRF | sampling time: approximately 34 minutes -> about 24 minutes | Helped |
| Add RePaint consistency conditioning (ablation) | DDRep SIREN panoramas · SIREN panoramas | panorama quality and consistency: 460 NFEs for both approaches | Helped |
| Add RePaint consistency conditioning (ablation) | DDRep SIREN panoramas · Low-CFG SIREN panoramas | panorama quality and consistency | Helped |
| Use object-centric scene initialization (ablation) | DDRep NeRF generation · 3D NeRF | generation quality | Helped |
Surprising findings
- The pullback of the PF-ODE requires the (J^T J)^{-1} metric term, which SDS and SJC omit; the authors claim omitting it yields incorrect samples rather than just slower convergence.
Limitations
- RePaint forward steps add computational overhead and the number needed is not known a priori.
- Monte Carlo estimate of the pullback score over views is high-variance when the Jacobian is ill-conditioned.
Experimental setup
Datasets: MS-COCO 2017 validation captions
Benchmarks: MS-COCO 2017 captions (100 prompts)
Metrics: KID PSNR SSIM LPIPS
Baselines: Score Jacobian Chaining (SJC) DreamFusion (SDS) SDv1.5 reference samples
Builds on: Score Jacobian Chaining DreamFusion RePaint DDIM
Closest papers by meaning
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.89)
- SITCOM: Step-wise Triple-Consistent Diffusion Sampling For Inverse Problems (ICML 2025 · similarity 0.89)
- Multistep Distillation of Diffusion Models via Moment Matching (NeurIPS 2024 · similarity 0.88)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.88)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.88)
- DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.