Research map / Image generation / Autoregressive and latent generation
Hyper-Transforming Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDMI, a latent diffusion framework over implicit neural representations using a Transformer-based hyper-decoder (HD) that generates SIREN weights, trainable from scratch or by hyper-transforming a frozen pretrained LDM.
Problem
Learning expressive, scalable probabilistic generative models over implicit neural representation (INR) parameters, where MLP-based hypernetworks bottleneck on large INRs and existing Transformer hypernetworks (Trans-INR) are deterministic.
Approach
A full Transformer hypernetwork (encoder over latent tokens plus a decoder cross-attending to learnable grouped weight tokens) generates INR weights, with a new multiplicative reconstruction (1+w)⊙b replacing normalization to stabilize periodic (SIREN) INRs; this decoder plugs into a VAE and latent diffusion, and can replace the decoder of a pretrained LDM ("hyper-transforming").
Key results
- On CelebA-HQ 64x64, LDMI reaches 24.80 dB PSNR and 18.06 FID with 8.06M hypernetwork params, versus VAMoH's 23.17 dB and 66.27 FID.
- On ImageNet 256x256, LDMI achieves FID 6.94 (PSNR 20.69 dB), though it uses 102.78M hypernetwork params.
- On ERA5, LDMI reaches 44.6 dB PSNR versus VAMoH's 39.0 dB; Functa is better on Chairs (99.51% vs LDMI 97.25%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LDMI instead of VAMoH | INR reconstruction · ShapeNet Chairs | accuracy (%): 96.75 -> 97.25 | Helped |
| use LDMI instead of VAMoH | INR reconstruction · ERA5 | PSNR (dB): 39.0 -> 44.6 | Helped |
| use LDMI instead of Functa | INR reconstruction · ShapeNet Chairs | accuracy (%): 99.51 -> 97.25 | Hurt |
| use LDMI instead of GASP | image generation · CelebA-HQ (64×64) | FID: 7.42 -> 18.06 | Hurt |
| replace an MLP decoder with the HD decoder (ablation) | LDMI · CelebA-HQ | PSNR (dB): 24.93 -> 27.72 | Helped |
| replace an MLP decoder with the HD decoder (ablation) | LDMI · CelebA-HQ | HN Params: 17.53M -> 8.06M | Helped |
Surprising findings
- GASP achieves a lower FID (7.42) than LDMI on CelebA-HQ 64x64, though GASP cannot perform reconstruction.
- Functa's higher PSNR on CelebA-HQ comes from test-time per-image optimization using ground truth, so the comparison is not direct.
Limitations
- Comparisons to Functa are confounded by its test-time optimization, and the paper acknowledges this.
- GASP achieves a better FID on CelebA-HQ 64x64 and Functa achieves better Chairs accuracy.
- ImageNet hypernetwork uses 102.78M parameters, far more than the 8.06M of the CelebA-HQ model.
Experimental setup
Datasets: CelebA-HQ CelebA ImageNet ShapeNet Chairs ERA5
Benchmarks: CelebA-HQ 64x64 ImageNet 256x256 ShapeNet Chairs ERA5
Metrics: PSNR FID accuracy (%) hypernetwork parameter count
Baselines: GASP Functa VAMoH Spatial Functa Trans-INR
Builds on: latent diffusion models Trans-INR SIREN VAE DDPM
Closest papers by meaning
- DDMI: Domain-agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations (ICLR 2024 · similarity 0.89)
- Nested Diffusion Models Using Hierarchical Latent Priors (CVPR 2025 · similarity 0.88)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.87)
- Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture (CVPR 2024 · similarity 0.86)
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.86)
- Neural Diffusion Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.