Research map / Learning theory and algorithms / Matrix and tensor methods
Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives an outer-product span formula for the diffusion Fisher information and builds two gradient-free access methods (DF-TM trace matching and DF-EA endpoint approximation) with error bounds, plus a numerical optimal-transport test of PF-ODE maps.
Problem
Accessing the diffusion Fisher information (negative Hessian of the diffused log density) in diffusion models is done by auto-differentiating the score network, which is slow and has no accuracy guarantee.
Approach
The diffusion Fisher lies in the span of outer products of the score-related initial data points (weighted sums of y_i y_i^T), with weights depending only on the data and noise schedule; this yields gradient-free trace (learned scalar network) and matrix-vector (replace x0 outer product with endpoint sample) access with provable error bounds.
Key results
- On 2-D chessboard data, DF-TM reduces trace relative error to 3.41%-5.81% across timesteps versus 6.68%-70.95% for VJP.
- DF-TM trace computation takes 0.072 s per iteration versus 2195.48 s for VJP on SD-v1.5 over 10k COCO prompts (99% reduction).
- DF-EA adjoint guidance takes 0.063 s versus 0.155 s for VJP (59% reduction) with better guidance scores on SAC, Pick-Score and other objectives.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DF-TM instead of VJP for trace access | Diffusion Fisher trace evaluation · SD-v1.5, 10k COCO prompts | practical time-cost (s): 2195.48 -> 0.072 | Helped |
| Use DF-EA instead of VJP for Fisher operator access | Adjoint optimization · SD-v1.5, 10k COCO prompts | practical time-cost (s): 0.155 -> 0.063 | Helped |
| Use DF-TM instead of VJP | Diffusion Fisher trace evaluation · 2-D chessboard, VE, t = 1.0 | relative error of NLL evaluation: 6.68% -> 3.41% | Helped |
| Use DF-TM instead of VJP | Diffusion Fisher trace evaluation · 2-D chessboard, VE, t = 0.8 | relative error of NLL evaluation: 5.79% -> 4.56% | Helped |
| Use DF-TM instead of VJP | Diffusion Fisher trace evaluation · 2-D chessboard, VE, t = 0.6 | relative error of NLL evaluation: 10.46% -> 4.13% | Helped |
| Use DF-TM instead of VJP | Diffusion Fisher trace evaluation · 2-D chessboard, VE, t = 0.4 | relative error of NLL evaluation: 20.13% -> 4.28% | Helped |
| Use DF-TM instead of VJP | Diffusion Fisher trace evaluation · 2-D chessboard, VE, t = 0.2 | relative error of NLL evaluation: 51.14% -> 5.33% | Helped |
| Use DF-TM instead of VJP | Diffusion Fisher trace evaluation · 2-D chessboard, VE, t = 0.0 | relative error of NLL evaluation: 70.95% -> 5.81% | Helped |
| Use single-Gaussian or affine initial data rather than non-affine initial data | PF-ODE map · VE, VP, sub-VP, and EDM noise schedules | asymmetry rate: 0.00% versus 25.28%, 23.36%, 13.84%, and 27.09% | Mixed |
| Replace weighted data outer-product sum with the endpoint sample outer product | DF-EA for adjoint guidance · Adjoint guidance sampling | objective scores | Helped |
Surprising findings
- Numerical OT test shows PF-ODE maps are not optimal transport for non-affine three-point initial data across VE, VP, sub-VP and EDM schedules (asymmetry 13.84%-27.09%).
Limitations
- DF-EA replaces the data outer-product sum with the endpoint sample x0 x0^T, an approximation that is only motivated heuristically as t goes to 0.
- DF-TM requires training a separate scalar network per model (SD-1.5 and SD-2base) on large-scale data.
- Error bounds depend on approximation errors of learned networks and bounded data assumption.
Experimental setup
Datasets: LAION-2B-en COCO
Benchmarks: COCO 10k prompts (NLL and CLIP score) 2-D chessboard toy data SAC aesthetic score Pick-Score CLIP loss Face ID loss
Metrics: relative error of NLL/trace NLL CLIP score wall-clock time Hilbert-Schmidt error
Baselines: VJP (auto-differentiation) Hutchinson estimator
Builds on: probability flow ODE adjoint ODE guidance classifier-free guidance VJP / Hutchinson trace estimation
Closest papers by meaning
- Beyond Scores: Proximal Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models (NeurIPS 2024 · similarity 0.87)
- O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.87)
- Score-Optimal Diffusion Schedules (NeurIPS 2024 · similarity 0.87)
- A Unified Sampling Framework for Solver Searching of Diffusion Probabilistic Models (ICLR 2024 · similarity 0.87)
- Accelerating Convergence of Score-Based Diffusion Models, Provably (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.