Research map / Image generation / Image restoration
Residual-Conditioned Optimal Transport: Towards Structure-Preserving Unpaired and Paired Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models image restoration as an optimal transport problem with a Fourier residual-guided cost and a two-pass map whose second pass is conditioned on an embedding of the transport residual, trained adversarially for unpaired and partially paired settings.
Problem
Image restoration methods tend to over-smooth or distort structural details, and generative methods often ignore degradation-specific information, making it hard to preserve structure while minimizing distortion.
Approach
Formulates restoration as an OT problem whose transport cost adds an l1/l2 penalty on the Fourier transform of the transport residual (FROT), and a two-pass map where the first-pass residual is encoded as an embedding that conditions the second-pass restoration (RCOT map); solved via minimax dual training.
Key results
- On Kodak24 denoising (sigma=50), RCOT reaches 29.53 PSNR / 0.828 SSIM / 33.13 FID versus 29.25 / 0.799 / 64.26 for Restormer.
- On DIV2K 4x SR, RCOT achieves 28.41 PSNR and 1.22 FID, versus 27.90 PSNR and 6.14 FID for Restormer.
- On real-world SPANet deraining, RCOT reports 43.77 PSNR / 0.993 SSIM, above IR-SDE at 42.56 PSNR; the unpaired RCOT variant is competitive with paired baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the TRC residual-conditioning module (ablation) | RCOT · DIV2K, Rain100L, and Kodak24 | PSNR: 27.24 -> 28.41; 34.20 -> 37.27; 28.45 -> 29.53 | Helped |
| combine FROT with supervised ℓ2 loss (ablation) | RCOT · CBSD68, σ=50 | PSNR: 27.60 (FROT only); 27.69 (supervised ℓ2 only) -> 28.25 | Helped |
| add the Fourier residual penalty g(r) (ablation) | unpaired FROT · Denoising on Kodak24, σ=50; SR on DIV2K; deraining on Rain100L; dehazing on SOTS | PSNR: 28.37 -> 28.64; 25.96 -> 26.78; 35.69 -> 36.22; 29.72 -> 30.34 | Helped |
| condition the second pass on the residual rather than on the base restored image (ablation) | RCOT · Average of SR, deraining, and denoising | FID: 26.52 -> 14.34 | Helped |
| use the ℓ2 Fourier-residual regularizer instead of L1 (ablation) | RCOT · Denoising | PSNR: 28.64 -> 28.79 | Helped |
| use L0.5 instead of L1 for the Fourier residual (ablation) | RCOT · SR | PSNR: 26.78 -> 26.82 | Helped |
| use ℓ2 instead of L1 for the Fourier residual (ablation) | RCOT · Deraining | PSNR: 36.22 -> 35.80 | Hurt |
| use ℓ2 instead of L1 for the Fourier residual (ablation) | RCOT · Dehazing | PSNR: 30.34 -> 29.80 | Hurt |
| add TRC to the MPRNet, NAFNet, or Restormer backbone (ablation) | MPRNet, NAFNet, and Restormer · Rain100L | PSNR: 34.95 -> 36.78; 35.58 -> 37.10; 36.74 -> 38.22 | Helped |
Surprising findings
- The unpaired RCOT variant (trained without paired labels) outperforms several paired methods on perceptual metrics such as LPIPS and FID in dehazing and SR.
- Conditioning on the residual embedding outperforms conditioning on the restored image estimate (PSNR 31.72 vs 30.39 average).
Limitations
- Fourier residual regularizer is handcrafted and task-specific (l1 vs l2 chosen per degradation), which may not be optimal across degradations.
- Not designed as an all-in-one model; the all-in-one test uses only two degradations and the regularizer remains task-specific.
- The paper notes the minimax solution may include saddle points that are not the exact OT map.
Experimental setup
Datasets: BSD400 WED Kodak24 CBSD68 Rain100L SPANet SOTS O-HAZE DIV2K Rain13K
Benchmarks: Kodak24 CBSD68 Rain100L SPANet SOTS O-HAZE DIV2K
Metrics: PSNR SSIM LPIPS FID
Baselines: NOT OTUR MPRNet Restormer IR-SDE PromptIR RCD SFNet Dehazeformer IDM LINF CycleGAN
Builds on: Optimal transport dual formulation (Kantorovich) MPRNet Neural OT (unpaired translation) Wasserstein-1 OT restoration (OTUR)
Closest papers by meaning
- Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image Restoration (ICLR 2025 · similarity 0.87)
- Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration (NeurIPS 2026 · similarity 0.87)
- Reversing Flow for Image Restoration (CVPR 2025 · similarity 0.87)
- A Restoration Network as an Implicit Prior (ICLR 2024 · similarity 0.87)
- Learning Where and What to Restore for Composite Image Restoration (NeurIPS 2026 · similarity 0.86)
- Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.