Research map / 3D vision / 3D generation
SIR-DIFF: Sparse Image Sets Restoration with Multi-View Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIR-Diff, a multi-view latent diffusion model that jointly restores sparse sets of degraded photos (motion deblurring and 4x super-resolution) while keeping them 3D consistent.
Problem
Single-view image restoration ignores complementary information across multiple degraded photos of the same scene and produces view-inconsistent outputs, which hurts 3D downstream tasks.
Approach
Extends a Stable Diffusion 2.1 UNet to jointly denoise a set of views by adding a 3D convolution branch (initialized from Stable Video Diffusion, blended with the 2D branch) and multi-view 3D self-attention at low-resolution layers, conditioned on channel-concatenated degraded image latents.
Key results
- On sparse-view deblurring on ScanNet++, SIR-Diff reaches FID 40.09 and LPIPS 0.160 versus 49.72 FID for Restormer (best baseline).
- On 4x sparse-view super-resolution on ScanNet++, SIR-Diff reaches PSNR 27.38 and LPIPS 0.099, outperforming OSEDiff (25.22 PSNR, 0.154 LPIPS).
- Feature matching with LoFTR finds 1621.3 correspondences on deblurred images versus 1262.6 for single-view methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| restore multiple views jointly rather than process images independently (ablation) | SIR-Diff · ScanNet++, ETH3D, and CO3D motion deblurring | FID: 81.58 -> 40.09; 109.6 -> 67.31; 137.9 -> 85.67 | Helped |
| restore multiple views jointly rather than process images independently (ablation) | SIR-Diff · ScanNet++, ETH3D, and CO3D super-resolution | PSNR: 25.85 -> 27.38; 21.97 -> 22.58; 24.35 -> 24.53 | Helped |
| use SIR-Diff instead of the strongest listed single-image or video baseline | motion deblurring · ScanNet++, ETH3D, and CO3D | FID: 49.72 -> 40.09; 73.47 -> 67.31; 144.8 -> 85.67 | Helped |
| use SIR-Diff instead of the strongest listed baseline | super-resolution · ScanNet++, ETH3D, and CO3D | PSNR: 25.22 -> 27.38; 21.93 -> 22.58; 24.50 -> 24.53 | Helped |
| use SIR-Diff instead of the strongest listed baseline | sparse-view depth prediction · ScanNet++ and ETH3D | AbsRel: 9.270 -> 9.011; 6.662 -> 6.663 | Mixed |
| predict depth jointly from sparse views with SIR-Diff instead of the strongest listed baseline | sparse-view depth prediction · ScanNet++ and ETH3D | GConsis.: 0.162 -> 0.129; 0.410 -> 0.339 | Helped |
Limitations
- Cross-image attention is computationally heavy, so it is restricted to low-resolution layers.
- No explicit guarantees for 3D view consistency are provided.
Experimental setup
Datasets: Hypersim TartanAir
Benchmarks: ScanNet++ ETH3D CO3Dv2
Metrics: FID LPIPS PSNR SSIM VConsis AbsRel delta1 GConsis novel view synthesis metrics
Baselines: Restormer PromptIR VRT SAM-DiffSR OSEDiff Upscale-A-Video DepthAnythingV2 DepthCrafter Marigold BAD-GS InstantSplat
Builds on: Stable Diffusion 2.1 Stable Video Diffusion MVDream CAT3D Marigold
Closest papers by meaning
- ViewFusion: Towards Multi-View Consistency via Interpolated Denoising (CVPR 2024 · similarity 0.88)
- DMV3D: Denoising Multi-view Diffusion Using 3D Large Reconstruction Model (ICLR 2024 · similarity 0.87)
- Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration (ICLR 2025 · similarity 0.87)
- Residual Denoising Diffusion Models (CVPR 2024 · similarity 0.87)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.87)
- Visual-Instructed Degradation Diffusion for All-in-One Image Restoration (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.