Research map / Image generation / Super-resolution
CoSeR: Bridging Image and Language for Cognitive Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoSeR, a cognitive super-resolution framework that builds a multi-token cognitive embedding from LR images (via a Q-Former-style adapter aligned to CLIP text tokens) to activate Stable Diffusion priors, generate reference images, and guide SR through an All-in-Attention module.
Problem
Real-world super-resolution often fails to recover semantically correct details because it focuses on local textures and lacks global scene understanding, and existing reference or diffusion-based methods do not fully exploit text-to-image priors.
Approach
A cognitive encoder (lightweight Q-Former-style adapter over CLIP image tokens, supervised with multi-token CLIP text embeddings from BLIP2 captions) yields a cognitive embedding that drives Stable Diffusion's cross-attention and generates a semantically aligned reference image; an All-in-Attention module fuses LR, reference, and cognitive conditions into the denoising U-Net for fidelity.
Key results
- On ImageNet Test2000, CoSeR reaches FID 19.41 vs 22.53 for StableSR (second best), a 13.8% improvement.
- On RealSR and DRealSR, CoSeR's FID is 80.82 and 71.22, improving over the second-best by 3.8% and 4.7%.
- In a user study with 23 subjects on 20 real-world LRs, about 80% of votes chose CoSeR as visually best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-token cognitive encoder rather than omit cognitive encoding (ablation) | CoSeR · ImageNet Test2000 | FID: 23.18 -> 20.27 | Helped |
| add multi-token cognitive encoder rather than class-token encoder (ablation) | CoSeR · ImageNet Test2000 | Gen-score: 0.4881 -> 0.5953 | Helped |
| use a class-token encoder instead of no cognitive encoder (ablation) | CoSeR · ImageNet Test2000 | FID: 23.18 -> 21.35 | Helped |
| add generated reference guidance (ablation) | CoSeR · ImageNet Test2000 | FID: 20.27 -> 19.80 | Helped |
| add generated reference guidance (ablation) | CoSeR · ImageNet Test2000 | MUSIQ: 61.82 -> 64.21 | Helped |
| use generated reference instead of ImageNet reference (ablation) | CoSeR · ImageNet Test2000 | MUSIQ: 63.49 -> 64.21 | Helped |
| use generated reference instead of ImageNet reference (ablation) | CoSeR · ImageNet Test2000 | FID: 19.72 -> 19.80 | No clear effect |
| use All-in-Attention instead of SFT (ablation) | CoSeR · ImageNet Test2000 | FID: 21.50 -> 20.27 | Helped |
| use All-in-Attention instead of SFT (ablation) | CoSeR · ImageNet Test2000 | DISTS: 0.1530 -> 0.1502 | Helped |
| use All-in-Attention instead of SFT (ablation) | CoSeR · ImageNet Test2000 | LPIPS: 0.3101 -> 0.3076 | Helped |
Surprising findings
- A class-token-only cognitive encoder introduces semantic and texture bias in generated references, and multi-token supervision reduces this bias.
- Pre-trained BLIP2 captions from LR inputs were less accurate than the proposed cognitive adapter, which is about 3% of a 7B captioning model's parameters.
Limitations
- Reference image generation relies on the quality of the cognitive embedding and diffusion model, so inaccurate generations can propagate.
- Evaluation relies largely on non-reference and perceptual metrics since PSNR/SSIM correlate weakly with perception.
Experimental setup
Datasets: ImageNet RealSR DRealSR
Benchmarks: ImageNet Test2000 RealSR DRealSR
Metrics: FID DISTS LPIPS CLIP-Score MANIQA MUSIQ PSNR SSIM Gen-score
Baselines: RealSR Real-ESRGAN+ BSRGAN DASR FeMaSR LDM StableSR
Builds on: Stable Diffusion 2.1 ControlNet StableSR Q-Former CLIP
Closest papers by meaning
- SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning (NeurIPS 2025 · similarity 0.88)
- SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (CVPR 2024 · similarity 0.87)
- Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach (CVPR 2025 · similarity 0.87)
- The Power of Context: How Multimodality Improves Image Super-Resolution (CVPR 2025 · similarity 0.86)
- Text-Aware Real-World Image Super-Resolution via Diffusion Model with Joint Segmentation Decoders (NeurIPS 2025 · similarity 0.86)
- Everything at Every Scale: Scale-Invariant Diffusion with Continuous Super-Resolution (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.