Research map / Image generation / Autoregressive and latent generation
DiMSUM: Diffusion Mamba - A Scalable and Unified Spatial-Frequency Method for Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiMSUM, a Mamba-based diffusion backbone that fuses spatial sweep scanning with wavelet-subband window scanning via a cross-attention fusion layer and adds a globally shared transformer block, reporting strong FID on CelebA-HQ, LSUN Church, and ImageNet.
Problem
Mamba-based diffusion models struggle with scanning order for 2D image tokens and fail to capture global and long-range frequency relations, limiting image generation quality and training convergence.
Approach
Decompose latent features with a multi-level Haar wavelet transform into subbands scanned by window scanning, fuse them with spatial sweep-4 Mamba features through a query-swapped cross-attention fusion layer, condition Mamba by initializing its hidden state with an embedding, and interleave a globally weight-shared transformer block after every four DiM blocks.
Key results
- On ImageNet 256, DiMSUM reaches FID 2.11 with guidance (Ours-G), versus 2.27 for DiT-XL/2-G and 2.15 for SiT-XL/2-G, with 460M parameters.
- On CelebA-HQ ablation, the full model improves FID from 6.19 baseline to 4.65 (recall 0.52), with the cross-attention fusion layer contributing the largest gain among spatial-frequency components.
- On LSUN Church, DiMSUM-XL/2 achieves FID 3.45 with 718M parameters, compared with 3.94 for DIFFUSSM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add conditional Mamba (ablation) | DiMSUM baseline · CelebA-HQ 256×256 | FID: 6.19 to 5.27 | Helped |
| add Wavelet Mamba with concatenation fusion (ablation) | DiMSUM baseline · CelebA-HQ 256×256 | FID: 5.27 to 5.87 | Hurt |
| replace concatenation with cross-attention fusion (ablation) | DiMSUM spatial-frequency Mamba · CelebA-HQ 256×256 | FID: 5.87 to 4.92 | Helped |
| add globally shared transformer block (ablation) | DiMSUM spatial-frequency Mamba · CelebA-HQ 256×256 | FID: 4.92 to 4.65 | Helped |
| use 2 wavelet levels instead of 1 (ablation) | DiMSUM · CelebA-HQ 256×256 | FID: 5.09 to 4.65 | Helped |
| use 3 wavelet levels instead of 2 (ablation) | DiMSUM · CelebA-HQ 256×256 | FID: 4.65 to 5.23 | Hurt |
| use query-swapped cross-attention rather than linear projection (ablation) | DiMSUM fusion layer · CelebA-HQ 256×256 | FID: 6.00 to 4.92 | Helped |
| swap keys instead of queries in cross-attention fusion (ablation) | DiMSUM fusion layer · CelebA-HQ 256×256 | FID: 4.92 to 5.45 | Hurt |
| replace wavelet processing with DCT (ablation) | DiMSUM · CelebA-HQ 256×256 | FID: 4.92 to 5.53 | Hurt |
| replace wavelet processing with EinFFT (ablation) | DiMSUM · CelebA-HQ 256×256 | FID: 4.92 to 5.63 | Hurt |
| use Sweep-4 instead of bidirectional scanning (ablation) | Conditional Mamba · CelebA-HQ 256×256 | FID: 6.39 to 5.27 | Helped |
| use window scanning on frequency features instead of Sweep-4 (ablation) | Spatial-frequency Mamba · CelebA-HQ 256×256 | FID: 5.41 to 4.92 | Helped |
Surprising findings
- Naively concatenating wavelet and spatial features worsens FID (5.87 vs 5.27 without it), showing misaligned fusion can hurt performance.
- Using more scanning orders (sweep-8, zigzag-8, JPEG-8) does not outperform sweep-4 in the ablations.
Limitations
- Relies on manually defined scanning orders, which the authors acknowledge as a limitation.
- Does not use a multiscale architecture.
- Evaluated only on class-conditional and unconditional benchmarks at modest scale; no text-to-image experiments.
Experimental setup
Datasets: CelebA-HQ LSUN Church ImageNet
Benchmarks: CelebA-HQ 256 CelebA-HQ 512 LSUN Church 256 ImageNet 256
Metrics: FID Recall GFLOPs training iterations per second inference latency
Baselines: DiT SiT DIFFUSSM ZigMa LFM LDM StyleGAN-XL BigGAN-deep
Builds on: Mamba ZigMa DiT flow matching SiT
Closest papers by meaning
- MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · similarity 0.88)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.88)
- Scaling Diffusion Mamba with Bidirectional SSMs for Efficient 3D Shape Generation (AAAI 2025 · similarity 0.87)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.86)
- LC-Mamba: Local and Continuous Mamba with Shifted Windows for Frame Interpolation (CVPR 2025 · similarity 0.86)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.