Research map / Architectures and efficiency / Attention and transformers
Sharing Key Semantics in Transformer Makes Efficient Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemanIR, a transformer for image restoration that builds a sparse KNN key-semantic dictionary once per stage and shares it across all layers, giving linear attention cost per window.
Problem
Self-attention in ViT-based image restoration attends to all global tokens, including semantically unrelated ones, causing quadratic cost at high resolution and inefficiency.
Approach
Construct a top-k semantic dictionary per transformer stage from patch self-similarity (KNN), then share this fixed sparse connection across all layers in the stage so each layer attends only to semantically related patches, reducing attention cost while keeping global receptive field.
Key results
- SemanIR achieves 32.25 dB average PSNR on deblurring (GoPro+HIDE), 0.52 dB above Restormer on GoPro (33.44 vs 32.92).
- On classical 4x SR, SemanIR-B reaches 28.51 dB on Urban100 versus 27.75 for EDT, with fewer FLOPs than HAT.
- For 4x SR on Urban100, SemanIR-S (28.34 dB, 290.20 GFLOPs) matches HAT (28.37 dB, 416.90 GFLOPs) with 41.7% fewer parameters than HAT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SemanIR instead of Restormer | image deblurring · GoPro | PSNR: 32.92 -> 33.44; 0.52 dB | Helped |
| use SemanIR instead of Restormer | image deblurring · HIDE | PSNR: 31.22 -> 31.05 | Hurt |
| use SemanIR instead of the strongest listed multi-task baseline | SemanIR · Test1 (rain+fog) | PSNR: 27.96 -> 29.57 | Helped |
| use SemanIR instead of the strongest listed multi-task baseline | SemanIR · SnowTest100k-L | PSNR: 28.48 -> 30.76 | Helped |
| use SemanIR instead of All-in-One | SemanIR · RainDrop | PSNR: 31.12 -> 30.82 | Hurt |
| increase window size from 8 to 32 (ablation) | SemanIR · Gray denoising, Set12, σ=25 | PSNR: 31.01 -> 31.17 | Helped |
| increase window size from 8 to 32 (ablation) | SemanIR · Color denoising, McMaster, σ=25 | PSNR: 33.20 -> 33.38 | Helped |
| train with random top-k rather than fixed top-k (ablation) | SemanIR · JPEG CAR, denoising, and SR at low inference k | PSNR | Helped |
| increase k beyond 384 (ablation) | SemanIR · image restoration tasks | PSNR | No clear effect |
| use Torch-Gather instead of Triton for key-semantic attention (ablation) | SemanIR · key-semantic attention, N=4096 | GPU memory: 2.61 GB -> 4.41 GB | Hurt |
| use Torch-Mask instead of Triton for key-semantic attention (ablation) | SemanIR · key-semantic attention, N=4096 | GPU memory: 2.61 GB -> 6.83 GB | Hurt |
Surprising findings
- Training with a random k sampled per iteration gives more stable inference across k than fixed-k training, especially for small inference k.
- The Triton kernel saves GPU memory but is slower at inference than the other implementations in the authors' setup.
Limitations
- Each restoration task requires a separate network; multi-degradation handling is limited to denoising and JPEG levels.
- Parameter count is in the tens of millions, making deployment on constrained devices difficult.
- The k value is fixed per patch and not learnable per patch or pixel.
Experimental setup
Datasets: DIV2K Flickr2K WED BSD400 GoPro Snow100K Raindrop Outdoor-Rain
Benchmarks: GoPro HIDE LIVE1 BSD500 CBSD68 McMaster Urban100 Set12 BSD68 Kodak Test1 SnowTest100k-L
Metrics: PSNR SSIM Params FLOPs Runtime
Baselines: SwinIR Restormer GRL HAT CAT EDT ART KiT DRUNet DnCNN FFDNet MPRNet
Builds on: Restormer SwinIR FlashAttention DRSformer KiT
Closest papers by meaning
- Intra and Inter Parser-Prompted Transformers for Effective Image Restoration (AAAI 2025 · similarity 0.87)
- DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation (NeurIPS 2024 · similarity 0.87)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.87)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.86)
- ACL: Activating Capability of Linear Attention for Image Restoration (CVPR 2025 · similarity 0.86)
- UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancement (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.