Research map / Architectures and efficiency / Attention and transformers
Adapt or Perish: Adaptive Sparse Transformer with Attentive Feature Refinement for Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Sparse Transformer (AST) for image restoration, combining a two-branch adaptive sparse self-attention (ASSA) with a feature refinement feed-forward network (FRFN), evaluated on deraining, raindrop removal and dehazing.
Problem
Standard Transformers for image restoration attend to all tokens, introducing noisy interactions from irrelevant regions and redundant features, while existing sparse or top-k fixes are sensitive to task-specific settings.
Approach
Combine a ReLU-squared sparse self-attention branch (filters low query-key scores) with a softmax dense branch, fused by learnable adaptive weights, plus an FRFN that enhances informative channels via partial convolution and then eases redundancy with a gate.
Key results
- AST-B reaches 49.51 dB PSNR on SPAD deraining, 0.98 dB above DRSformer (48.53 dB).
- AST-B reaches 32.32 dB PSNR on AGAN-Data raindrop removal, 0.39 dB above AWRCP (31.93 dB).
- AST-B reaches 17.12 dB PSNR on Dense-Haze, outperforming the best prior Transformer baselines by at least 0.46 dB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AST-B instead of DRSformer | AST-B · SPAD | PSNR: 48.53 -> 49.51 | Helped |
| use AST-B instead of AWRCP | AST-B · AGAN-Data | PSNR: 31.93 -> 32.32 | Helped |
| use AST-B instead of MB-TaylorFormer-B | AST-B · Dense-Haze | PSNR: 16.66 -> 17.12 | Helped |
| replace Swin self-attention with ASSA (ablation) | AST-T · SPAD | PSNR: 44.47 -> 45.43 | Helped |
| replace Top-k self-attention with ASSA (ablation) | AST-T · SPAD | PSNR: 44.67 -> 45.43 | Helped |
| replace condensed self-attention with ASSA (ablation) | AST-T · SPAD | PSNR: 44.94 -> 45.43 | Helped |
| replace standard local self-attention with sparse local self-attention (ablation) | AST-T · SPAD | PSNR: 45.09 -> 44.58 | Hurt |
| replace standard channel self-attention with sparse channel self-attention (ablation) | AST-T · SPAD | PSNR: 44.91 -> 44.45 | Hurt |
| combine dense and sparse self-attention with adaptive branch weights (ablation) | AST-T · SPAD | PSNR: 45.09 -> 45.43 | Helped |
| replace softmax attention with ReLU2 sparse attention (ablation) | AST-T · SPAD | PSNR: 45.09 -> 44.58 | Hurt |
| use adaptive sparse-and-dense attention instead of StarReLU attention (ablation) | AST-T · SPAD | PSNR: 45.30 -> 45.43 | Helped |
| replace ACON attention with adaptive sparse-and-dense attention (ablation) | AST-T · SPAD | PSNR: 43.23 -> 45.43 | Helped |
Surprising findings
- Directly replacing softmax attention with ReLU-squared sparse attention lowers PSNR (local SA: 45.09 to 44.58; channel SA: 44.91 to 44.45), so pure sparsity hurts restoration.
Limitations
- Task-specific model; does not handle multiple degradation types with one uniform model.
- Fails on scenes with heavy degradation (failure cases shown).
- Ablations use AST-T trained for only 10 epochs on 128x128 patches.
Experimental setup
Datasets: SPAD AGAN-Data Dense-Haze
Benchmarks: SPAD AGAN-Data Dense-Haze Internet-Data
Metrics: PSNR SSIM NIQE
Baselines: DRSformer Restormer Uformer IDT MB-TaylorFormer-B DehazeFormer-S Top-k SA Condensed SA Swin SA AWRCP
Builds on: Restormer SwinIR Uformer DRSformer PConv
Closest papers by meaning
- AdaFormer: Efficient Transformer with Adaptive Token Sparsification for Image Super-resolution (AAAI 2024 · similarity 0.87)
- Omni-Kernel Network for Image Restoration (AAAI 2024 · similarity 0.86)
- Sharing Key Semantics in Transformer Makes Efficient Image Restoration (NeurIPS 2024 · similarity 0.86)
- Degradation-Aware Feature Perturbation for All-in-One Image Restoration (CVPR 2025 · similarity 0.86)
- Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration (NeurIPS 2026 · similarity 0.86)
- Rethinking Multi-Scale Representations in Deep Deraining Transformer (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.