Research map / Image generation / Image restoration
Soft Superpixel Neighborhood Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes soft superpixel neighborhood attention (SNA), which weights neighborhood attention by per-pixel superpixel probabilities, and shows it is the optimal denoiser under a latent superpixel model and beats neighborhood attention in a small denoising network.
Problem
Square-window local attention mixes features from perceptually unrelated regions with deformable object boundaries, degrading denoising; naive hard superpixel masking is unstable and sensitive to superpixel errors.
Approach
Replace hard superpixel assignments in attention with per-pixel superpixel probabilities, re-weighting neighborhood attention weights by the inner product of superpixel probabilities of query and key pixels; derived as the optimal denoiser under a latent superpixel model where probabilities are sampled per pixel.
Key results
- SNA reaches about 2.8 dB PSNR higher than standard NA at sigma=30 in the single-layer denoiser (27.70 vs 24.91 for NA with fixed scale).
- With matched parameter counts, SNA is about 1.8 dB better than NA with learned attention scale at sigma=30.
- SNA is roughly 15-22 times slower than NA in forward and backward passes, with about 13 times more memory per the paper's table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SNA instead of NA with learned attention scale | Single-layer denoising network · Set5, BSD100, Urban100, Manga109; σ=30 | PSNR: 25.69 -> 27.70 | Helped |
| Use SNA instead of NA with learned attention scale at matched parameter budget | Single-layer denoising network · Set5, BSD100, Urban100, Manga109; σ=30 | PSNR: 25.69 -> 27.51 | Helped |
| Add pixel SSN loss to denoising loss (ablation) | SNA denoiser · BSD500 test | PSNR: 32.77 -> 32.78 | Helped |
| Add pixel SSN loss to denoising loss (ablation) | SNA denoiser · BSD500 test | SSIM: 0.879 -> 0.889 | Helped |
| Add segmentation SSN loss to denoising loss (ablation) | SNA denoiser · BSD500 test | PSNR: 32.77 -> 30.14 | Hurt |
| Add segmentation SSN loss to denoising loss (ablation) | SNA denoiser · BSD500 test | SSIM: 0.879 -> 0.798 | Hurt |
| Learn the attention scale per pixel (ablation) | NA denoiser · Set5, BSD100, Urban100, Manga109; σ=30 | PSNR: 22.37 -> 25.69 | Helped |
| Learn the attention scale per pixel (ablation) | SNA + gϕ,SLIC denoiser · Set5, BSD100, Urban100, Manga109; σ=30 | PSNR: 27.51 -> 26.94 | Hurt |
| Use denoising loss rather than segmentation-label supervision for superpixel probabilities (ablation) | SNA superpixel probabilities · BSD500 test | PSNR: 30.14 -> 32.77 | Helped |
Surprising findings
- Jointly training superpixels for segmentation labels (SSN loss) degraded denoising PSNR by about 2.6 dB, suggesting boundary adherence and denoising utility are adversarial.
- Learning superpixel probabilities purely from the denoising loss gave better superpixel pooling quality than supervised segmentation-label training.
Limitations
- SNA is 15-22 times slower than NA in the unoptimized implementation and uses more memory.
- Evaluation uses a very simple single-attention-layer denoiser, not large-scale networks.
- Identifiability of the latent superpixel model is not addressed.
Experimental setup
Datasets: BSD500
Benchmarks: Set5 BSD100 Urban100 Manga109 BSD500
Metrics: PSNR SSIM ASA BR forward time backward time memory
Baselines: Neighborhood Attention (NA) Hard superpixel neighborhood attention (H-SNA) NA with learned attention scale Superpixel Sampling Network (SSN)
Builds on: Neighborhood Attention SLIC superpixels Superpixel Sampling Network Charbonnier loss
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.85)
- BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model (CVPR 2024 · similarity 0.83)
- Learning Invariant Inter-pixel Correlations for Superpixel Generation (AAAI 2024 · similarity 0.83)
- Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level (NeurIPS 2024 · similarity 0.83)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.83)
- Locality in Image Diffusion Models Emerges from Data Statistics (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.