Research map / Image generation / Super-resolution
SeD: Semantic-Aware Discriminator for Image Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeD, a semantic-aware discriminator for GAN-based image super-resolution that conditions the discriminator on CLIP RN50 layer-3 semantic features via a cross-attention fusion block (SeFB).
Problem
GAN-based super-resolution discriminators learn coarse-grained image/patch/pixel distributions, which leads to virtual textures and counter-intuitive generation results.
Approach
Use pixel-wise semantic features from a frozen pretrained vision model (CLIP RN50 middle layer) as a condition for the discriminator, fused with image features via a semantic-aware fusion block using cross-attention, so the discriminator measures distribution distance per semantic context.
Key results
- On classical SR, RRDB+P+SeD reaches LPIPS 0.048 on Manga109 versus 0.058 for RRDB+P, with up to 1.0 dB PSNR gain on Manga109.
- SeD-our fusion gives LPIPS 0.106 on Urban100 versus 0.131 for channel attention (SeD-B) and 0.125 for concatenation (SeD-A).
- Using CLIP RN50 layer-3 semantics outperforms ResNet-50 ImageNet features (Urban100 LPIPS 0.106 vs 0.117).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SeD to SwinIR with PatchGAN | SwinIR+P · Set5, Set14, DIV2K, Urban100, Manga109 | LPIPS: Set5: 0.070 -> 0.061; Set14: 0.127 -> 0.115; DIV2K: 0.103 -> 0.090; Urban100: 0.107 -> 0.097; Manga109: 0.048 -> 0.044 | Helped |
| Replace SeFB with concatenation (ablation) | RRDB+P+SeD · Set5, Urban100, Manga109 | LPIPS: Set5: 0.064 -> 0.069; Urban100: 0.106 -> 0.124; Manga109: 0.048 -> 0.058 | Hurt |
| Replace SeFB with channel attention (ablation) | RRDB+P+SeD · Set5, Urban100, Manga109 | LPIPS: Set5: 0.064 -> 0.085; Urban100: 0.106 -> 0.131; Manga109: 0.048 -> 0.065 | Hurt |
| Replace SeFB with spatial attention (ablation) | RRDB+P+SeD · Set5, Urban100, Manga109 | LPIPS: Set5: 0.064 -> 0.074; Urban100: 0.106 -> 0.125; Manga109: 0.048 -> 0.055 | Hurt |
| Use CLIP layer 1 features instead of layer 3 (ablation) | RRDB+P+SeD · Set5, Urban100, Manga109 | LPIPS: Set5: 0.064 -> 0.066; Urban100: 0.106 -> 0.120; Manga109: 0.048 -> 0.057 | Hurt |
| Use CLIP layer 2 features instead of layer 3 (ablation) | RRDB+P+SeD · Set5, Urban100, Manga109 | LPIPS: Set5: 0.064 -> 0.068; Urban100: 0.106 -> 0.117; Manga109: 0.048 -> 0.056 | Hurt |
| Use CLIP layer 4 features instead of layer 3 (ablation) | RRDB+P+SeD · Set5, Urban100, Manga109 | LPIPS: Set5: 0.064 -> 0.069; Urban100: 0.106 -> 0.122; Manga109: 0.048 -> 0.055 | Hurt |
| Use CLIP instead of ResNet-50 as semantic extractor (ablation) | RRDB+P+SeD · Set5, Urban100, Manga109 | LPIPS: Set5: 0.066 -> 0.064; Urban100: 0.117 -> 0.106; Manga109: 0.055 -> 0.048 | Helped |
Surprising findings
- Channel attention fusion of semantics performed worst, indicating destroying spatial semantic information hurts guidance.
Limitations
- Gains on real-world tasks are mixed; the paper notes SeD surpasses vanilla discriminators on most but not all real-world datasets.
- Relies on the choice of semantic extractor layer, which was selected by ablation.
Experimental setup
Datasets: DF2K DIV2K ImageNet
Benchmarks: Set5 Set14 DIV2K Urban100 Manga109 DPED OST300 RealSRSet
Metrics: LPIPS PSNR SSIM NIQE
Baselines: SFTGAN ESRGAN USRGAN LDL DualFormer Real-ESRGAN SwinIR
Builds on: ESRGAN Real-ESRGAN SRGAN CLIP SwinIR
Closest papers by meaning
- Exploring Semantic Feature Discrimination for Perceptual Image Super-Resolution and Opinion-Unaware No-Reference Image Quality Assessment (CVPR 2025 · similarity 0.93)
- Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution (CVPR 2022 · similarity 0.90)
- Uncertainty-Aware GAN for Single Image Super Resolution (AAAI 2024 · similarity 0.88)
- Training Generative Image Super-Resolution Models by Wavelet-Domain Losses Enables Better Control of Artifacts (CVPR 2024 · similarity 0.88)
- GSDD: Generative Space Dataset Distillation for Image Super-resolution (AAAI 2024 · similarity 0.87)
- SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.