Research map / Image generation / Super-resolution
Visual Autoregressive Modeling for Image Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VARSR, a visual autoregressive (next-scale prediction) framework for image super-resolution with prefix-token LR conditioning, scale-aligned RoPE, a diffusion refiner for quantization residuals, and image-based CFG, trained on a new 4M-image dataset.
Problem
Diffusion-based image super-resolution is computationally expensive and struggles with pixel-level fidelity versus realism; applying visual autoregressive models to ISR raises issues of LR conditioning, spatial structure, quantization loss, and quality guidance.
Approach
Adapts next-scale visual autoregression to ISR by prepending LR-encoded tokens as a prefix scale, aligning 2D RoPE across scales, predicting continuous quantization residuals with a small diffusion MLP head, and guiding sampling with positive/negative image-quality embeddings (image-based CFG).
Key results
- VARSR achieves the best no-reference scores on DIV2K-Val, RealSR and DRealSR (e.g., MUSIQ 71.27 on DIV2K-Val vs 68.35 for second best), but lags GAN methods on PSNR/SSIM.
- VARSR runs in 0.59s per image with 10 steps and 1101.9M params, about 10.1% of DiffBIR's 5.85s inference time, per the paper's complexity table (which lists DiffBIR at 50 steps).
- In a user study with 20 participants on 50 real-world images, VARSR was selected 55.2% of the time versus 20.3% for the next-best method (SeeSR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VARSR rather than the strongest reported baselines | ISR methods · DIV2K-Val, RealSR, and DRealSR | MUSIQ: DIV2K-Val: 68.35 -> 71.27; RealSR: 69.56 -> 71.26; DRealSR: 64.75 -> 68.15 | Helped |
| use VARSR rather than the strongest reported baseline | ISR methods · DIV2K-Val | PSNR: 24.42 -> 23.91 | Hurt |
| use VARSR rather than the strongest reported baseline | ISR methods · RealSR | PSNR: 26.38 -> 24.61 | Hurt |
| use VARSR rather than the strongest reported baseline | ISR methods · DRealSR | PSNR: 28.70 -> 28.16 | Hurt |
| use Prefix Tokens rather than directly adding LR features, ControlNet, or ControlNet+CA (ablation) | VARSR · DRealSR | SSIM: 0.6902; 0.7188; 0.7314 -> 0.7652 | Helped |
| apply SA-RoPE to condition and token maps rather than original APE (ablation) | VARSR · DRealSR | SSIM: 0.7424 -> 0.7652 | Helped |
| add the Diffusion Refiner (ablation) | VARSR · DRealSR | SSIM: 0.7583 -> 0.7652 | Helped |
| add image-based CFG (ablation) | VARSR · DRealSR | MANIQA: 0.4052 -> 0.5362 | Helped |
| add image-based CFG (ablation) | VARSR · DRealSR | SSIM: 0.8004 -> 0.7652 | Hurt |
| train on large-scale data rather than DIV2K (ablation) | VARSR · DRealSR | MANIQA: 0.4885 -> 0.5362 | Helped |
| use Resize&Crop rather than random cropping (ablation) | VARSR · DRealSR | MANIQA: 0.4577 -> 0.5362 | Helped |
| use VQVAE scale dropout with p_d=0.1 (ablation) | VARSR · DRealSR | SSIM: 0.7567 -> 0.7652 | Helped |
Surprising findings
- Reference-based fidelity metrics (PSNR, SSIM) favor GAN-based methods, and VARSR does not win them, which the authors attribute to metric limitations and the realism-fidelity trade-off.
Limitations
- Semantic coverage limited by the 4M-image training set compared to billion-scale Stable Diffusion data; severe degradations of rare content (e.g., Spider-Man) can fail.
- Reference-based metrics (PSNR, SSIM, LPIPS) lag behind GAN and some diffusion methods.
Experimental setup
Datasets: DIV2K DIV8K LAION DataComp KonIQ-10k CLIVE AVA Flickr2K ImageNet DrealSR RealSR RealLR200
Benchmarks: DIV2K-Val RealSR DRealSR RealLR200
Metrics: PSNR SSIM LPIPS DISTS FID MANIQA CLIPIQA MUSIQ user study selection rate inference time
Baselines: BSRGAN Real-ESRGAN SwinIR LDM StableSR DiffBIR PASD SeeSR
Builds on: VAR MAR HART RoPE Real-ESRGAN degradation pipeline
Closest papers by meaning
- NSARM: Next-Scale Autoregressive Modeling for Robust Real-World Image Super-Resolution (NeurIPS 2026 · similarity 0.91)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.91)
- Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (CVPR 2025 · similarity 0.89)
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.88)
- FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction (NeurIPS 2025 · similarity 0.88)
- Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.