Research map / Image generation / Super-resolution
AdaFormer: Efficient Transformer with Adaptive Token Sparsification for Image Super-resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaFormer, an efficient SR transformer that adds a parallel global cross-attention branch and a confidence-map-based token early-exit mechanism trained with an uncertainty-guided loss to speed up inference on large images.
Problem
Efficient transformer-based image super-resolution is slow on large images (2K-8K) because all tokens are processed equally and window-based attention has a limited receptive field; dynamic token halting from high-level vision does not transfer directly to regression-style SR.
Approach
A texture-relevant sparse attention block combines local window self-attention with a global cross-attention branch driven by learnable prototypes, and a lightweight confidence estimator, trained with an aleatoric-uncertainty (Laplace likelihood) loss, produces a halting score map that progressively exits well-recovered tokens from both branches.
Key results
- On Test8K (4x SR), AdaFormer reaches 35.13 dB PSNR with 1355 ms latency vs. SwinIR-light 35.04 dB at 11766 ms, roughly 90% latency reduction.
- Against ELAN-light on Test8K, AdaFormer cuts latency by 281 ms and gains 0.05 dB PSNR.
- Token sparsity reaches almost 90% in the last TSAB on Test4K; early-exit comparisons show 0.32 dB gain over uniform exit and 0.29 dB over A-ViT at lower FLOPs/latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaFormer instead of ELAN-light | 4× SR · Test8K | Latency: 1636.38 -> 1355.34 | Helped |
| Use AdaFormer instead of ELAN-light | 4× SR · Test8K | PSNR: 35.08 -> 35.13 | Helped |
| Use AdaFormer instead of ELAN-light | 4× SR · Test4K | Latency: 300.31 -> 257.94 | Helped |
| Use AdaFormer instead of ELAN-light | 4× SR · Urban100 | PSNR: 26.54 -> 26.48 | Hurt |
| Add the global cross-attention branch to the local-only model (ablation) | AdaFormer · Test4K | PSNR: +0.03dB | Helped |
| Add local-mask early exit to the local-and-global branch model (ablation) | AdaFormer · Test4K | PSNR: 29.06 -> 29.04 | No clear effect |
| Add early exit to both local and global branches (ablation) | AdaFormer · Test4K | PSNR: 29.04 -> 29.10 | Helped |
| Replace adaptive exit with uniform exit (ablation) | AdaFormer · Test4K | PSNR: 28.78 vs. 29.10 | Hurt |
| Replace adaptive exit with random exit (ablation) | AdaFormer · Test4K | PSNR: 28.85 vs. 29.10 | Hurt |
| Use the A-Vit exit strategy instead of adaptive exit (ablation) | AdaFormer · Test4K | PSNR: 28.81 vs. 29.10 | Hurt |
| Use APE incremental-capacity exit signals instead of adaptive exit (ablation) | AdaFormer · Test4K | Latency: 499.45 vs. 267.24 | Hurt |
| Use a weight-unshared confidence estimator instead of weight-shared (ablation) | AdaFormer · Test4K | Latency: 279.92 vs. 267.24 | Hurt |
Surprising findings
- Halting tokens also slightly improved accuracy (+0.06 dB on Test4K in the ablation), which the authors attribute to reduced overfitting and noise from attending to informative tokens only.
- Directly applying the A-ViT ponder/Gaussian-prior halting from classification to SR underperforms, which the authors attribute to SR's texture-focused rather than center-focused content.
Limitations
- Gains over strong efficient baselines are small in PSNR (about 0.05 dB on Test8K, 0.02 dB on Test2K).
- Latency comparisons are on a single 2080Ti GPU and the ablations use 400-epoch training rather than the full 1000 epochs.
Experimental setup
Datasets: DIV2K DIV8K
Benchmarks: DIV2K Test2K Test4K Test8K Set5 Set14 B100 Urban100
Metrics: PSNR SSIM latency (ms) FLOPs
Baselines: SwinIR-light ELAN-light N-Gram ESRT IMDN LAPAR-A RFDN RLFN ETDS
Builds on: SwinIR A-ViT Adaptive Patch Exiting (APE) Uncertainty-Driven Loss for SISR EViT
Closest papers by meaning
- Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary (CVPR 2024 · similarity 0.90)
- Recursive Generalization Transformer for Image Super-Resolution (ICLR 2024 · similarity 0.88)
- AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-Resolution (CVPR 2024 · similarity 0.87)
- When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution (NeurIPS 2026 · similarity 0.87)
- Iterative Token Evaluation and Refinement for Real-World Super-resolution (AAAI 2024 · similarity 0.87)
- Adapt or Perish: Adaptive Sparse Transformer with Attentive Feature Refinement for Image Restoration (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.