Research map / Image generation / Image restoration
Latent Harmony: Synergistic Unified UHD Image Restoration via Latent Space Regularization and Controllable Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Latent Harmony, a two-stage framework that regularizes a VAE latent space (LH-VAE) and then jointly fine-tunes it with a restoration network via high-frequency-aware LoRA modules to deliver efficient UHD all-in-one image restoration with an adjustable fidelity-perception parameter.
Problem
UHD all-in-one image restoration must balance computational efficiency, detail retention, and generalization across degradations, but standard VAE latents lose degradation-related high-frequency information and are sensitive to degradations.
Approach
Regularize the VAE latent with progressive degradation perturbation, DINOv2 semantic alignment, and latent equivariance, then fine-tune the VAE jointly with the restoration network using separate encoder (fidelity, high-frequency L1) and decoder (perception, high-frequency GAN) LoRA modules trained by alternating optimization, with an inference-time alpha to trade off fidelity and perception.
Key results
- On the six-degradation UHD all-in-one setting, Latent Harmony reports 29.24 average PSNR, 0.920 SSIM and 0.1822 LPIPS, with 3.6G FLOPs and 1.2M parameters at 256x256 input.
- On the four-degradation UHD setting it reports 29.70 average PSNR and 0.877 SSIM, versus 29.23 PSNR for the next best UHDprocesser.
- Plugging LH-VAE into SFHformer, Restormer and NAFNet raises latent restoration PSNR by about 5 dB while cutting parameters by 84-93%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add degradation-invariance visual semantic loss L_Inv (ablation) | Latent Harmony · UHD all-in-one restoration ablation | PSNR: 24.28 -> 29.77 | Helped |
| Add latent equivariance loss L_Eqv (ablation) | Latent Harmony · UHD all-in-one restoration ablation | PSNR: 25.68 -> 29.77 | Helped |
| Add progressive degradation perturbation strategy (ablation) | Latent Harmony · UHD all-in-one restoration ablation | PSNR: 27.82 -> 29.77 | Helped |
| Add fidelity-oriented encoder HF-LoRA (ablation) | Latent Harmony · UHD all-in-one restoration ablation | PSNR: 28.12 -> 29.77 | Helped |
| Add perception-oriented decoder HF-LoRA (ablation) | Latent Harmony · UHD all-in-one restoration ablation | LPIPS: 0.306 -> 0.250 | Helped |
| Add HF-LoRA fine-tuning (ablation) | Latent Harmony · UHD all-in-one restoration ablation | PSNR: 28.68 -> 29.77 | Helped |
| Fine-tune the VAE (ablation) | Latent Harmony · UHD all-in-one restoration ablation | PSNR: 28.48 -> 29.77 | Helped |
| Increase inference control α (ablation) | Latent Harmony · UHD all-in-one restoration | PSNR and LPIPS: 28.94 -> 29.74; 0.2218 -> 0.2904 | Mixed |
Limitations
- Code and models are promised only upon acceptance, not released with the submission.
- Alpha-based control trades PSNR/SSIM against LPIPS; the perceptual metrics degrade as alpha increases.
Experimental setup
Datasets: UHD-LL UHD-blur UHD-haze UHDN UHD-rain UHD-snow
Benchmarks: UHD-LL UHD-blur UHD-haze UHDN UHD-rain UHD-snow
Metrics: PSNR SSIM LPIPS FID
Baselines: AIRNet IDR PromptIR CAPTNet NDR-Restore Gridformer DiffUIR-L Histoformer adaIR HAIR UHDprocesser
Builds on: VAE LoRA REPA-E VA-VAE
Closest papers by meaning
- DreamUHD: Frequency Enhanced Variational Autoencoder for Ultra-High-Definition Image Restoration (AAAI 2025 · similarity 0.91)
- From Zero to Detail: Deconstructing Ultra-High-Definition Image Restoration from Progressive Spectral Perspective (CVPR 2025 · similarity 0.89)
- UHD-processer: Unified UHD Image Restoration with Progressive Frequency Learning and Degradation-aware Prompts (CVPR 2025 · similarity 0.89)
- When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution (NeurIPS 2026 · similarity 0.87)
- ReFIR: Grounding Large Restoration Models with Retrieval Augmentation (NeurIPS 2024 · similarity 0.86)
- Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.