Research map / Image generation / Super-resolution
The Power of Context: How Multimodality Improves Image Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMSR, a diffusion-based single-image super-resolution method that conditions Stable Diffusion v2 on multiple modalities (caption, depth, segmentation, edges) via VQGAN tokens and a multimodal latent connector, plus multimodal CFG and per-modality temperature control.
Problem
Single-image super-resolution struggles to recover fine details and avoid hallucinations; text prompts alone cannot represent spatial relationships, so generative SR relies on limited priors.
Approach
Encode depth, segmentation, and edge maps (extracted from the LR input) as discrete VQGAN tokens, compress them with a multimodal latent connector (cross-attention to learnable latents, linear in modality length), and apply multimodal classifier-free guidance plus attention temperature scaling for per-modality control.
Key results
- On DIV2K-Val-100 (1024), MMSR reaches LPIPS 0.2810 and DISTS 0.1492 and MUSIQ 72.31, outperforming SeeSR (LPIPS 0.3273, DISTS 0.1620, MUSIQ 69.79).
- Multimodal latent connector improves MUSIQ from 69.69 to 72.31 and LPIPS from 0.3929 to 0.2810 on DIV2K-Val-100 at a slight throughput cost (3.48 to 3.32 img/s).
- Multimodal CFG keeps LPIPS at 0.3772 and NIQE at 3.9815 at guidance 14, versus 0.5064 and 5.1886 for standard CFG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multimodal guidance (MMSR) | text-guided super-resolution · DIV2K-Val-3k 512 × 512 | LPIPS: 0.3843 -> 0.3707 | Helped |
| add multimodal guidance (MMSR) | text-guided super-resolution · DIV2K-Val-100 1024 × 1024 | MUSIQ: 69.79 -> 72.31 | Helped |
| add multimodal guidance (MMSR) | text-guided super-resolution · RealSR 512 × 512 | LPIPS: 0.3009 -> 0.2952 | Helped |
| add multimodal guidance (MMSR) | text-guided super-resolution · DrealSR 512 × 512 | LPIPS: 0.3174 -> 0.3249 | Hurt |
| add multimodal guidance (MMSR) | text-guided super-resolution · DrealSR 512 × 512 | MUSIQ: 65.09 -> 68.93 | Helped |
| add the Multimodal Latent Connector (ablation) | MMSR · DIV2K-Val-100 1024 × 1024 | LPIPS: 0.3929 -> 0.2810 | Helped |
| add the Multimodal Latent Connector (ablation) | MMSR · DIV2K-Val-100 1024 × 1024 | MUSIQ: 69.69 -> 72.31 | Helped |
| add multimodal classifier-free guidance for the negative prompt (ablation) | MMSR · DIV2K-Val-100 1024 × 1024, guidance 14 | LPIPS: 0.5493 -> 0.3772 | Helped |
| add multimodal classifier-free guidance for the negative prompt (ablation) | MMSR · DIV2K-Val-100 1024 × 1024, guidance 14 | NIQE: 5.9175 -> 3.9815 | Helped |
| add depth input to the text-guided model (ablation) | MMSR · DIV2K-Val-100 1024 × 1024 | MUSIQ: 69.78 -> 72.33 | Helped |
| add all modalities (depth, segmentation, edges, and text) (ablation) | MMSR · DIV2K-Val-100 1024 × 1024 | DISTS: 0.17 -> 0.15 | Helped |
Surprising findings
- Discrete VQGAN tokens reconstruct modalities better than continuous tokens, motivating discrete tokenization for conditioning.
- Modalities are extracted from the LR input at inference, yet the method still outperforms baselines.
Limitations
- Multimodal extraction adds computational overhead, with Gemini Flash captioning the slowest step.
- Noisy or severely degraded LR inputs can produce inaccurate modality predictions.
- Training uses TPUv5 pods, limiting reproducibility for smaller groups.
Experimental setup
Datasets: LSDIR DIV2K DIV2K-Val RealSR DRealSR
Benchmarks: DIV2K-Val-3k DIV2K-Val-100 RealSR DrealSR
Metrics: PSNR SSIM LPIPS DISTS NIQE FID MANIQA MUSIQ CLIPIQA LIQE
Baselines: BSRGAN Real-ESRGAN LDL DASR FeMASR LDM StableSR ResShift PASD DiffBIR SeeSR SUPIR
Builds on: Stable Diffusion ControlNet InstructPix2Pix VQGAN classifier-free guidance
Closest papers by meaning
- Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignment (NeurIPS 2025 · similarity 0.90)
- SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning (NeurIPS 2025 · similarity 0.89)
- RAP-SR: RestorAtion Prior Enhancement in Diffusion Models for Realistic Image Super-Resolution (AAAI 2025 · similarity 0.88)
- When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution (NeurIPS 2026 · similarity 0.88)
- SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (CVPR 2024 · similarity 0.88)
- Text-Aware Real-World Image Super-Resolution via Diffusion Model with Joint Segmentation Decoders (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.