Research map / Image generation / Autoregressive and latent generation
NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NFIG, an autoregressive image generator that uses a frequency-guided residual-quantized VAE to generate tokens from low to high frequency bands, reaching gFID 2.81 on ImageNet-256 with a VAR-style transformer.
Problem
Standard autoregressive image generators produce tokens in a fixed spatial order (raster or multi-scale), which ignores the hierarchical frequency structure of images and is computationally costly.
Approach
Decompose image latents with FFT masks into frequency bands and build a residual-quantized tokenizer (FR-VAE) so that low-frequency bands use fewer tokens; a decoder-only transformer then generates bands from low to high frequency, instead of by spatial scale.
Key results
- NFIG reaches gFID 2.81 and IS 332.42 on ImageNet-256 with a 310M model and 10 steps, versus VAR-d20 (600M) at gFID 2.95.
- FR-VAE reconstruction rFID of 0.85, compared with 1.62 for the VAR-style baseline tokenizer in the ablation.
- Relative wall-clock time of 1 versus 1.25 for VAR-d20, reported as a 1.25x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use frequency-guided generation instead of the VAR-d20 baseline | NFIG on ImageNet-256×256 · ImageNet-256×256 | gFID: 2.95 -> 2.81 | Helped |
| use frequency-guided generation instead of VAR-d20 | NFIG on ImageNet-256×256 · ImageNet-256×256 | IS: 302.6 -> 332.42 | Helped |
| use NFIG instead of VAR-d20 | ImageNet-256×256 inference · ImageNet-256×256 | inference speed: 1.25× speedup | Helped |
| add frequency-guided residual quantization (ablation) | VAR framework image tokenizer · ImageNet validation | rFID: 1.62 -> 1.40 | Helped |
| add DINO discriminator (ablation) | Frequency-guided residual-quantized tokenizer · ImageNet validation | rFID: 1.40 to 0.85 | Helped |
| add Top_k sampling (ablation) | NFIG Transformer with FR-VAE · ImageNet validation | gFID: 9.7 to 6.83 | Helped |
| add classifier-free guidance (CFG) (ablation) | NFIG Transformer with FR-VAE and Top_k · ImageNet validation | gFID: 6.83 to 2.81 | Helped |
| increase training from 200 to 350 epochs | NFIG-310M · ImageNet-256×256 | FID: 3.35 -> 2.81 | Helped |
| increase training from 200 to 350 epochs | NFIG-310M · ImageNet-256×256 | IS: 309.2 -> 332.4 | Helped |
| use NFIG instead of VAR-d16 at 200 epochs | 310M image generator · ImageNet-256×256 | FID: 3.55 -> 3.35 | Helped |
Surprising findings
- The paper reports that NFIG's Precision (0.77) is lower than VAR-d16 (0.84), so gains are concentrated in gFID, IS and recall.
Limitations
- Frequency band division is a simple scale-based heuristic that the authors say inadequately captures the first band.
- Failure cases include anatomical errors, texture abnormalities and fine detail loss, attributed to semantic inconsistency across frequency bands.
- Scaling experiments were limited by compute, so only the 310M model was trained for the full schedule.
Experimental setup
Datasets: ImageNet ILSVRC 2012
Benchmarks: ImageNet 256x256
Metrics: FID IS Precision Recall rFID wall-clock time
Baselines: VAR-d16 VAR-d20 XQGAN-310M FAR-H RCG MaskGIT StyleGAN-XL DiT-XL/2
Builds on: VAR VQGAN XQGAN DINOv2
Closest papers by meaning
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.88)
- HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation (CVPR 2025 · similarity 0.88)
- Guiding Visual Autoregressive Models through Spectrum Weakening (NeurIPS 2026 · similarity 0.87)
- FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction (NeurIPS 2025 · similarity 0.87)
- Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation (NeurIPS 2025 · similarity 0.87)
- Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropy (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.