Research map / Architectures and efficiency / Attention and transformers
Masked Generative Nested Transformers with Decode Time Scaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaGNeTS, which schedules nested (MatFormer-style) sub-models of increasing size over MaskGIT-style parallel decoding iterations and caches KV of unmasked tokens with intermittent refresh, cutting inference compute ~2.5-3.7x on ImageNet, UCF101, and K600.
Problem
Parallel decoding visual generators (MaskGIT-style) use the same full-size transformer at every iteration and recompute all tokens each step, which is computationally expensive at inference.
Approach
Use a decode-time model size schedule that starts with small nested sub-models and scales up to the full model over iterations, sharing parameters via MatFormer-style slicing, combined with KV caching of unmasked tokens refreshed when the model size changes; nested sub-models are trained with a ground-truth plus progressive self-distillation loss.
Key results
- On ImageNet 256x256 the scheduled model gives roughly 2.65-3x speedup with negligible FID drop; with L-size model and caching+refresh, FID 3.1 at 3.0x FLOP gain (vs baseline FID 2.5 at 1.0x).
- On UCF101 class-conditional generation, MaGNeTS reaches FVD 96.4 with ~1.7k GFLOPs vs MAGVIT-L FVD 74.4 at ~4.3k GFLOPs.
- On Kinetics600 frame prediction, MaGNeTS gets FVD 10.8 at ~1.2k GFLOPs vs MAGVIT-L FVD 7.2 at ~4.3k GFLOPs, a ~3.7x lower compute claim.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add caching (ablation) | Baseline MaskGIT++ · ImageNet256×256 | FID: 2.5 -> 3.4 | Hurt |
| Add intermittent cache refresh after adding caching (ablation) | Baseline MaskGIT++ · ImageNet256×256 | FID: 3.4 -> 2.6 | Helped |
| Add caching to the upscaling schedule (ablation) | Scheduled model · ImageNet256×256 | FID: 3.1 -> 4.8 | Hurt |
| Add intermittent cache refresh to the scheduled model with caching (ablation) | Scheduled model · ImageNet256×256 | FID: 4.8 -> 3.1 | Helped |
| Use an upscaling nested-model schedule instead of the baseline (ablation) | MaskGIT++ · ImageNet256×256 | FLOP gains: 1.0 -> 2.1 | Helped |
| Replace nested models with independently trained standalone models (ablation) | MaGNeTS · ImageNet | FID: 3.1 -> 3.1 | No clear effect |
| Replace nested models with independently trained standalone models (ablation) | MaGNeTS · UCF101 | FVD: 96.4 -> 115.0 | Hurt |
| Linearly transition from ground-truth loss to progressive distillation instead of using ground-truth loss only (ablation) | Nested models · ImageNet256×256 | FID: 6.1 -> 5.7 | Helped |
| Linearly transition from ground-truth loss to progressive distillation instead of using ground-truth loss only (ablation) | Nested models · UCF101 | FVD: 112.6 -> 96.4 | Helped |
| Train using only distillation loss (ablation) | Nested models · ImageNet256×256 and UCF101 | training convergence | Hurt |
| Nest along attention head count rather than head feature dimension (ablation) | MaGNeTS · Image and video generation | performance | Hurt |
| Use MaGNeTS instead of MAGVIT-L | MAGVIT-L · UCF101 | FVD: 74.4± 2 -> 96.4±2 | Hurt |
Surprising findings
- Scaling model size up over decoding iterations is much better than scaling it down, indicating larger models are better utilized in later iterations.
- Training nested sub-models with only distillation loss diverges; a linear GT-to-distillation schedule is required.
Limitations
- Quality depends on the pretrained tokenizer and KV caching requires discrete tokens.
- Cache refresh adds compute overhead and caching alone degrades FID.
- Inherits artifacts of MaskGIT++ on ImageNet.
Experimental setup
Datasets: ImageNet UCF101 Kinetics600
Benchmarks: ImageNet 256x256 UCF101 Kinetics600
Metrics: FID FVD Inception Score precision recall GFLOPs
Baselines: MaskGIT++ MAGVIT-B MAGVIT-L RaMViD Video Diffusion TATS Phenaki PAR-4x
Builds on: MaskGIT MatFormer MAGVIT classifier-free guidance
Closest papers by meaning
- Learnings from Scaling Visual Tokenizers for Reconstruction and Generation (ICML 2025 · similarity 0.88)
- Halton Scheduler for Masked Generative Image Transformer (ICLR 2025 · similarity 0.88)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.87)
- Fixed-Point Masked Generative Modeling (NeurIPS 2026 · similarity 0.87)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.87)
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.