Research map / Image generation / Super-resolution
SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeeSR, a semantics-aware real-world super-resolution method that fine-tunes a RAM tag model into a degradation-aware prompt extractor giving hard (tags) and soft (embeddings) prompts to a ControlNet-equipped Stable Diffusion 2-base, plus an LR-embedding initialization at inference.
Problem
Diffusion-prior real-world image super-resolution often produces semantic errors because heavy degradation of LR inputs corrupts local structures and object semantics.
Approach
Use a degradation-aware prompt extractor (DAPE), trained by aligning LR-branch tag-model features and logits to HR-branch outputs, to supply tag-style hard prompts and representation-based soft prompts to a pretrained T2I model; additionally initialize sampling from the LR latent rather than pure Gaussian noise.
Key results
- On DIV2K-Val, SeeSR achieves best LPIPS (0.3843) and DISTS (0.2257) and FID 31.93, the best among compared methods.
- In the semantics test on COCO-Val, SeeSR reaches PQ 30.0 vs 16.2 for zoomed LR and 27.2 for the next best (DiffBIR).
- In a 20-participant user study, SeeSR got a 38.6% confusion rate on synthetic data and 57.1% best rate on real-world data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SeeSR with degradation-aware hard and soft semantic prompts and LR embedding | Stable Diffusion 2-base for real-world image super-resolution · DIV2K-Val | FID: 36.57 -> 31.93 | Helped |
| Use SeeSR with degradation-aware hard and soft semantic prompts and LR embedding | Stable Diffusion 2-base for real-world image super-resolution · RealSR | MUSIQ: 68.69 -> 69.77 | Helped |
| Use SeeSR with degradation-aware hard and soft semantic prompts and LR embedding | Stable Diffusion 2-base for real-world image super-resolution · DRealSR | CLIPIQA: 0.6773 -> 0.6804 | Helped |
| Use SeeSR with degradation-aware hard and soft semantic prompts and LR embedding | Stable Diffusion 2-base for real-world image super-resolution · RealLR200 | NIQE: 4.1715 -> 4.1620 | Helped |
| Use SeeSR to restore images before downstream semantic tasks | Real-ISR methods evaluated on COCO-Val · COCO-Val | Semantic Segmentation (mIOU): 37.7 -> 41.3 | Helped |
| Use SeeSR rather than the competing methods | Real-ISR methods · synthetic and real-world user study | selection rate: 38.6%; 57.1% | Helped |
| Add LR latent to the initial sampling noise (ablation) | SD-based Real-ISR methods · smooth regions such as the sky | visual artifacts | Helped |
| Use tag-style prompts instead of caption-style prompts (ablation) | PASD Real-ISR · image with a ship | semantic fidelity | Helped |
| Use degradation-aware DAPE prompts instead of prompts extracted directly from a degraded LR image (ablation) | pretrained T2I model for Real-ISR · severely degraded images | semantic fidelity | Helped |
| Use tag-style prompts instead of classification-style prompts (ablation) | PASD Real-ISR · multi-object scenes | local object perception | Helped |
| Use tag-style prompts instead of caption-style prompts (ablation) | PASD Real-ISR · scenes with diverse objects | object coverage | Helped |
| Rely on the LR image alone instead of adding semantic prompts (ablation) | StableSR and DiffBIR · real-world image super-resolution | semantic detail | Hurt |
Surprising findings
- Classification-style and caption-style prompts from LR images gave results nearly indistinguishable from a null prompt or produced semantic errors, motivating tag-style prompts.
- Embedding the LR latent into the initial noise removes artifacts in smooth regions such as the sky for SD-based SR methods.
Limitations
- DAPE may predict incorrect tags for heavily degraded images, leading to wrongly restored objects.
- Tag-region alignment can be inaccurate under severe degradation.
- Like other SD-based methods, SeeSR struggles with small-scale scene text images.
Experimental setup
Datasets: LSDIR FFHQ DIV2K-Val RealSR DRealSR RealLR200 COCO-Val
Benchmarks: DIV2K-Val RealSR DrealSR RealLR200 COCO-Val
Metrics: PSNR SSIM LPIPS DISTS FID NIQE MANIQA MUSIQ CLIPIQA PQ
Baselines: BSRGAN Real-ESRGAN LDL DASR FeMaSR LDM StableSR ResShift PASD DiffBIR
Builds on: Stable Diffusion ControlNet RAM PASD Real-ESRGAN
Closest papers by meaning
- Text-Aware Real-World Image Super-Resolution via Diffusion Model with Joint Segmentation Decoders (NeurIPS 2025 · similarity 0.89)
- Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach (CVPR 2025 · similarity 0.89)
- Text-guided Explorable Image Super-resolution (CVPR 2024 · similarity 0.89)
- SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning (NeurIPS 2025 · similarity 0.89)
- DP²O-SR: Direct Perceptual Preference Optimization for Real-World Image Super-Resolution (NeurIPS 2025 · similarity 0.88)
- SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.