Research map / Image generation / Fast diffusion sampling
Lumina-Next : Making Lumina-T2X Stronger and Faster with Next-DiT
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lumina-Next improves the Flag-DiT text-to-image diffusion transformer with a Next-DiT architecture (3D RoPE, sandwich norm, GQA), frequency- and time-aware scaled RoPE for resolution extrapolation, sigmoid time schedule with higher-order solvers, and time-aware context drop for faster sampling, and extends the framework to recognition, multi-view, audio, music, and point cloud generation.
Problem
Lumina-T2X (Flag-DiT) suffers from training instability, slow inference, and extrapolation artifacts when generating images beyond the training resolution, and needs many sampling steps.
Approach
Replace 1D RoPE with 3D RoPE and add sandwich normalization in a Next-DiT backbone; make the RoPE frequency base depend on both frequency and diffusion time for tuning-free extrapolation; use a sigmoid time schedule for flow ODE sampling with a midpoint solver; and drop redundant key/value tokens over time to speed up each network evaluation.
Key results
- Lumina-Next with a 2B Next-DiT and Gemma-2B text encoder outperforms Lumina-T2X (5B Flag-DiT, LLaMA-7B encoder) on text-to-image generation while reducing training and inference costs.
- On text-to-audio, the proposed system reaches FAD 1.03 and CLAP 0.630 on AudioCaps, versus 1.87 FAD for Tango and 1.75 for AudioLDM 2-Large, with MOS-Q 77.53 and MOS-F 76.52.
- On ImageNet-1K recognition, Next-DiT (86M) reaches 82.3% top-1 at 224x224 under 300 epochs, versus 81.8% for DeiT-base, and 84.2% after any-resolution fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Next-DiT instead of DeiT-base with the same 300-epoch fixed-resolution training | Image classification backbone · ImageNet-1K, 224×224 | Top-1 accuracy (%): 81.8 versus 82.3 | Helped |
| Fine-tune Next-DiT for arbitrary resolutions using dynamic partitioning and padding | Next-DiT image recognition backbone · ImageNet-1K, flexible resolution | Top-1 accuracy (%): 67.2 versus 84.2 | Helped |
| Remove the dual text encoder (ablation) | Next-DiT text-to-audio · AudioCaps validation | CLAP: 0.630 -> 0.601 | Hurt |
| Remove AudioCaps fine-tuning (ablation) | Next-DiT text-to-audio · AudioCaps validation | FAD: 1.03 -> 2.09 | Hurt |
| Replace Next-DiT with Flag-DiT (ablation) | Text-to-audio generation · AudioCaps validation | FAD: 1.03 -> 1.56 | Hurt |
| Use Lumina-Next music generation instead of AudioLDM 2 | Text-to-music generation · MusicCaps Evaluation | FAD: 3.81 -> 3.75 | Helped |
| Use Lumina-Next music generation instead of AudioLDM 2 | Text-to-music generation · MusicCaps Evaluation | KL: 1.22 -> 1.24 | Hurt |
| Add long-skip connections (ablation) | Next-DiT · ImageNet-256 label-conditional generation | FID | Hurt |
| Use Time-Aware Context Drop | Next-DiT image generation · 1K-resolution image generation | inference speed: 2× inference-time speed up | Helped |
| Use Time-Aware Scaled RoPE for resolution extrapolation | Lumina-Next · High-resolution image generation | visual quality | Helped |
| Use a sigmoid time schedule instead of uniform time discretization | Flow-model sampling · Few-step text-to-image generation | sample quality | Helped |
| Use Lumina-Next point-cloud generation instead of PDiffusion | Point-cloud generation · ShapeNet airplane | MMD (×10³ CD): 3.276 -> 3.371 | Hurt |
Surprising findings
- Straightforward application of Token Merging (ToMe) to Next-DiT completely fails to produce satisfactory images, motivating the time-aware context drop.
- Discretization errors near pure noise and clean data are larger than at intermediate timesteps for flow models, contrary to the EDM finding for diffusion models.
Limitations
- Context Drop and time-aware RoPE are heuristic and their similarity to masked attention or dilated convolution is only noted as a possible explanation, not analyzed.
- Multi-view, audio, music and point cloud results are evaluated on limited subsets and mostly with baselines borrowed from prior work.
Experimental setup
Datasets: ImageNet-1K ImageNet-256 AudioSet-SL AudioCaps LP-MusicCaps Objaverse Cap3D ShapeNet
Benchmarks: ImageNet-256 ImageNet-1K AudioCaps MusicCaps ShapeNet
Metrics: FID IS Top-1 accuracy FAD KL CLAP score MOS-Q MOS-F MMD COV
Baselines: SDXL PixArt-alpha Lumina-T2X DemoFusion MultiDiffusion ScaleCrafter DeiT-base AudioLDM 2 MusicGen PDiffusion
Builds on: Lumina-T2X Flag-DiT NTK-Aware Scaled RoPE SiT Token Merging
Closest papers by meaning
- One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.89)
- PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (ICLR 2024 · similarity 0.87)
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.87)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.87)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.87)
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.