Research map / Image generation / Image restoration
SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeNM-VAE, a hierarchical VAE that learns a degradation model p(y|x) from a small paired set plus unpaired source/target sets via a mixture inference model, then synthesizes paired data for image denoising and super-resolution.
Problem
Real-world image restoration suffers from a data bottleneck: paired degraded/clean data is costly and misaligned, while unpaired data is abundant but hard to exploit for learning the degradation process.
Approach
A specially designed graphical model with latent content z and degradation z_n, optimized under variational inference with a mixture inference model q(z|x,y)=0.5q(z|x)+0.5q(z|y), giving a single objective decomposed into paired, source (clean-only) and target (degraded-only) losses; the hierarchical VAE uses Gaussian conditionals and a degradation-level controllable generator.
Key results
- On SIDD noise modeling with 10 paired samples, SeNM-VAE reaches FID 17.25 and KLD 0.036 versus LUD-VAE's 35.31 and 0.108, and with 1% paired data it beats fully paired NeCA-W (PSNR 37.28 vs 37.04 in DnCNN downstream).
- Downstream SIDD benchmark denoising with DRUNet trained on 1% paired plus SeNM-VAE synthetic data reaches 39.34 PSNR, close to DRUNet fully supervised at 39.51.
- Finetuning NAFNet with SeNM-VAE synthetic data reaches 40.38 PSNR on SIDD benchmark, the best reported in the paper's table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SeNM-VAE with 10 paired samples for noise synthesis | SIDD noise modeling · SIDD validation noise synthesis | FID: 19.96 -> 17.25 | Helped |
| Use SeNM-VAE with 1% paired data rather than NeCA-W | SIDD noise modeling · SIDD validation noise synthesis | PSNR: 37.04 -> 37.28 | Helped |
| Augment DRUNet training with SeNM-VAE synthetic data using 10 paired samples | DRUNet · SIDD validation | PSNR: 34.48 -> 37.96 | Helped |
| Augment DRUNet training with SeNM-VAE synthetic data using 0.1% paired samples | DRUNet · SIDD validation | PSNR: 37.68 -> 38.91 | Helped |
| Augment DRUNet training with SeNM-VAE synthetic data using 1% paired samples | DRUNet · SIDD validation | PSNR: 38.93 -> 39.39 | Helped |
| Finetune NAFNet with synthetic data generated by SeNM-VAE | NAFNet · SIDD validation | PSNR: 40.30 -> 40.49 | Helped |
| Use SeNM-VAE degradation modeling to train ESRGAN with 10 paired samples | ESRGAN · AIM19 validation | PSNR: 21.47 -> 22.48 | Helped |
| Use SeNM-VAE degradation modeling to train ESRGAN with 10 paired samples | ESRGAN · NTIRE20 validation | PSNR: 25.05 -> 25.91 | Helped |
| Add clean-image reconstruction loss (ablation) | SeNM-VAE · SIDD validation noise synthesis (10 paired) | FID: 24.26 -> 17.25 | Helped |
| Add clean-image reconstruction loss (ablation) | SeNM-VAE · SIDD validation noise synthesis (10 paired) | PSNR: 35.60 -> 36.73 | Helped |
| Add source- and target-domain unpaired data (ablation) | SeNM-VAE · SIDD validation noise synthesis (10 paired) | FID: 19.79 -> 17.25 | Helped |
| Add 96 source and 96 target unpaired images (ablation) | SeNM-VAE · SIDD validation noise synthesis (10 paired) | FID: 19.79 -> 17.52 | Helped |
Surprising findings
- With only 10 paired samples, SeNM-VAE's synthetic data already beats several self-supervised denoisers on SIDD validation PSNR.
Limitations
- Degradation model learned on SIDD with heavy reliance on paired-data quantity for best results
- Relies on a fixed mixture weight and a hand-tuned KL weight lambda (1e-7) to balance terms
- Downstream SR uses bicubic downsampling approximation with noise absorbing the mismatch
Experimental setup
Datasets: SIDD DND AIM19 NTIRE20
Benchmarks: SIDD DND AIM19 NTIRE20
Metrics: FID KLD PSNR SSIM LPIPS
Baselines: C2N DeFlow LUD-VAE DANet Flow-sRGB NeCA-W N2V N2S CVF-SID AP-BSN SCPGabNet SDAP
Builds on: VAE beta-VAE CycleGAN KL annealing
Closest papers by meaning
- RestoreGrad: Signal Restoration Using Conditional Denoising Diffusion Models with Jointly Learned Prior (ICML 2025 · similarity 0.89)
- DreamUHD: Frequency Enhanced Variational Autoencoder for Ultra-High-Definition Image Restoration (AAAI 2025 · similarity 0.87)
- SSC-VAE: Structured Sparse Coding Based Variational Autoencoder for Detail Preserved Image Reconstruction (AAAI 2025 · similarity 0.86)
- Noise Conditional Variational Score Distillation (ICML 2025 · similarity 0.86)
- Diffusion Priors for Variational Likelihood Estimation and Image Denoising (NeurIPS 2024 · similarity 0.86)
- Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.