Research map / Large language models / Reasoning and chain of thought

Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HyperNoise, a LoRA-based noise hypernetwork that predicts a modulated initial noise for a frozen distilled text-to-image generator, trained with a reward-plus-L2 noise-space objective to amortize test-time noise optimization.

Problem

Test-time noise optimization and search improve diffusion outputs but add large inference latency; directly fine-tuning distilled generators with reward objectives risks reward hacking and KL terms are intractable.

Approach

Learn a noise hypernetwork (LoRA on the generator architecture) that maps Gaussian noise to an optimized initial latent; the KL to the base prior in noise space is approximated by an L2 penalty on the noise modification, so the generator stays frozen and inference costs one extra forward pass.

Key results

• SANA-Sprint + HyperNoise improves GenEval mean from 0.70 to 0.75 at 0.3 s versus ReNO's 0.81 at 30 s.
• SD-Turbo + HyperNoise reaches GenEval mean 0.57 vs 0.49 base, surpassing SDXL (0.55).
• FLUX-schnell + HyperNoise improves GenEval mean from 0.68 to 0.72 at 0.9 s, versus ReNO 0.76 at 40 s.

Measured effects reported in the paper

Surprising findings

• Direct LoRA fine-tuning of SANA-Sprint with the same reward objective lowers GenEval mean (0.70 to 0.67 one-step; 0.73 to 0.62 four-step), indicating reward hacking.
• One-step HyperNoise (2 NFEs) outperforms four-step SANA-Sprint (0.75 vs 0.73).

Limitations

• Performance depends strongly on the base model and reward model quality.
• Gains are partial: recovers only about half of ReNO's improvement on SANA-Sprint.
• Theoretical KL-to-L2 approximation relies on a Lipschitz constant L<1 assumption that is not verified for trained networks.

Experimental setup

Datasets: Pick-a-Pic v2 T2I-CompBench ABC-6K

Benchmarks: GenEval ImageReward

Metrics: GenEval mean score sub-task scores (single, two, counting, colors, position, at inference time (s) ImageReward redness reward

Baselines: ReNO Best-of-N Prompt Optimization LoRA fine-tuning SDXL DPO-SDXL Hyper-SDXL SD3-Medium Flux-dev

Builds on: ReNO Consistency Models LoRA DPO Direct reward fine-tuning

Closest papers by meaning

• Scaling Inference Time Compute for Diffusion Models (CVPR 2025 · similarity 0.92)
• Test-Time Scaling of Diffusion Models via Noise Trajectory Search (NeurIPS 2025 · similarity 0.90)
• Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.90)
• TEST-TIME SCALING IN DIFFUSION LLMS VIA HIDDEN SEMI-AUTOREGRESSIVE EXPERTS (ICLR 2026 · similarity 0.89)
• Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models (ICML 2026 · similarity 0.89)
• Inference-Time Alignment of Diffusion Models with Direct Noise Optimization (ICML 2025 · similarity 0.89)