Research map / Image generation / Autoregressive and latent generation
Neural Residual Diffusion Models for Deep Scalable Vision Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Neural-RDM, a learnable gated residual mechanism (per-block gating weights alpha and beta) applied to U-shaped and flow-shaped diffusion backbones, fine-tuned from LDM and Latte, to enable stable deep scalable diffusion training for image and video generation.
Problem
Deeper stacked U-Net and Transformer diffusion networks suffer numerical propagation errors and loss of sensitivity, which hinders massively deep scalable training of vision generation models.
Approach
Replace the plain residual update with a gated residual z_{i+1} = z_i + alpha*F(z_i) + beta, with learnable time-dependent gates that act as a parameterized mean-variance scheduler matching the PF-ODE; a residual-sensitivity ODE is used to argue the gates counteract sensitivity decay in deep stacks.
Key results
- Neural-RDM-F reports FID 2.12 and IS 295.32 on ImageNet class-to-image at 256x256, outperforming DiT-XL/2-G (FID 2.27) and the baselines in the table.
- On video, Neural-RDM reaches FVD 39.89 on SkyTimelapse and 91.22 on Taichi-HD, versus 59.82 and 159.60 for Latte-XL, reported as 33.3% and 42.8% improvements.
- Neural-RDM-U reaches FID 2.25 and IS 235.35 on JourneyDB text-to-image, the best in the table, while Neural-RDM-F is second-best on class-to-image.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learnable post-processing gated residual weights alpha and beta | Neural-RDM-F vs. Latte-XL · SkyTimelapse | FVD: 33.3% | Helped |
| Add learnable post-processing gated residual weights alpha and beta | Neural-RDM-F vs. Latte-XL · Taichi-HD | FVD: 42.8% | Helped |
| Use flow-shaped gated residual architecture instead of the baseline architecture | ImageNet, Class-to-Image · ImageNet, Class-to-Image | FID: 9.62 -> 2.12 | Helped |
| Use U-shaped Neural-RDM instead of Neural-RDM-F | Neural-RDM · JourneyDB, Text-to-Image | FID: 2.46 -> 2.25 | Helped |
| Use U-shaped Neural-RDM instead of Neural-RDM-F | Neural-RDM · JourneyDB, Text-to-Image | IS: 206.32 -> 235.35 | Helped |
| Use post-processing gated residual Variant-0 rather than the other residual variants (ablation) | Latte-XL backbone trained from scratch · UCF-101 video generation | FVD | Helped |
| Replace post-processing gated residual with AdaLN-style gating (Variant-1) (ablation) | Latte-XL backbone trained from scratch · UCF-101 video generation | FVD | Hurt |
| Replace post-processing gated residual with alpha-scaled skip and beta offset (Variant-2) (ablation) | Latte-XL backbone trained from scratch · UCF-101 video generation | FVD | Hurt |
| Remove both gating parameters, using the ResNet residual (Variant-3) (ablation) | Latte-XL backbone trained from scratch · UCF-101 video generation | FVD | Hurt |
| Remove beta gating, using ReZero-style residual (Variant-4) (ablation) | Latte-XL backbone trained from scratch · UCF-101 video generation | FVD | Hurt |
| Increase the number of residual units (ablation) | Neural-RDM · UCF-101 | FVD | Helped |
Limitations
- The gated residual only inhibits, rather than fully avoids, sensitivity decay and propagation errors in deep stacks.
- Gains are reported on fine-tuned LDM/Latte backbones with approximately matched model size, not from-scratch massive-scale training.
- Theoretical claims rely on continuous-time ODE approximations of discrete networks.
Experimental setup
Datasets: ImageNet MSCOCO JourneyDB SkyTimelapse Taichi-HD UCF-101
Benchmarks: ImageNet JourneyDB SkyTimelapse Taichi-HD UCF-101
Metrics: FID sFID IS FVD
Baselines: BigGAN-deep StyleGAN-XL ADM LDM-8 LDM-4 DiT-XL/2 Latte-XL MoCoGAN MoCoGAN-HD DIGAN StyleGAN-V MoStGAN-V
Builds on: latent diffusion models Latte DiT ReZero probability flow ODE
Closest papers by meaning
- U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models (ICLR 2025 · similarity 0.88)
- Neural Diffusion Models (ICML 2024 · similarity 0.88)
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.88)
- Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding (ICML 2024 · similarity 0.88)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.88)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.