Research map / Image generation / Image restoration
CamFreeDiff: Camera-free Image to Panorama Generation with Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CamFreeDiff, which predicts a 3-DoF homography from a camera-free input view to a canonical perspective view and uses it inside a frozen-SD multi-view diffusion model with correspondence-aware attention to outpaint 360° panoramas.
Problem
360° panorama outpainting from a single image requires known camera intrinsics and pose, which existing multi-view diffusion methods (MVDiffusion, PanoDiffusion) assume; camera-free inputs break them.
Approach
Estimate the input view's homography to a canonical view (parameterized as fov, x-axis rotation, z-axis rotation) with an MLP on a frozen Stable Diffusion U-Net encoder, then feed the resulting point-level correspondences into correspondence-aware attention, with the input added as an extra conditioning branch, so the estimator is trained end-to-end with the diffusion model.
Key results
- CamFreeDiff (new view variant) achieves FID 27.0, IS 5.6, CS 24.4, PSNR 19.3 on Matterport3D, versus MVDiffusion 42.4 FID and PanoDiffusion 48.7 FID under the same rectified inputs.
- On zero-shot Structured3D, CamFreeDiff reaches FID 31.1 vs PanoDiffusion 35.3 (trained directly on Structured3D).
- Ablation on CAA neighbourhood size K shows FID improving from 66.4 (K=1) to 47.3 (K=7), with diminishing gains beyond K=5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Treat the input as a new view and use correspondence-aware attention (ablation) | CamFreeDiff · Matterport3D | FID: 35.2 -> 27.0 | Helped |
| Treat the input as a new view rather than unwarping the latent (ablation) | CamFreeDiff · Matterport3D | PSNR: 15.6 -> 19.3 | Helped |
| Use CamFreeDiff with the new-view strategy instead of MVDiffusion | CamFreeDiff · Matterport3D | FID: 42.4 -> 27.0 | Helped |
| Use CamFreeDiff on out-of-domain data without fine-tuning | CamFreeDiff · Structured3D | FID: 35.3 -> 31.1 | Helped |
| Use a frozen Stable Diffusion encoder with an MLP instead of HomographyNet (ablation) | CamFreeDiff homography estimator · Matterport3D | FID: 29.2 -> 27.0 | Helped |
| Increase CAA neighborhood size from K=1 to K=3 (ablation) | CamFreeDiff · Matterport3D | FID: 66.4 -> 51.1 | Helped |
| Increase CAA neighborhood size from K=3 to K=5 (ablation) | CamFreeDiff · Matterport3D | FID: 51.1 -> 48.0 | Helped |
| Increase CAA neighborhood size from K=5 to K=7 (ablation) | CamFreeDiff · Matterport3D | FID: 48.0 -> 47.3 | Helped |
| Increase CAA neighborhood size from K=5 to K=7 (ablation) | CamFreeDiff · Matterport3D | IS: 6.2 -> 6.1 | Hurt |
Surprising findings
- Predicting only 3 degrees of freedom (fov, x-rotation, z-rotation) instead of the full 8-DoF homography is reported as sufficient for panorama outpainting.
Limitations
- Relies on the assumption of pinhole input camera with centered principal point and zero skew.
- Larger CAA neighbourhood sizes increase computational and time cost with limited gains.
- Evaluated mainly on indoor scenes (Matterport3D, Structured3D).
Experimental setup
Datasets: Matterport3D Structured3D
Benchmarks: Matterport3D Structured3D
Metrics: FID IS CLIP score PSNR MAE
Baselines: MVDiffusion PanoDiffusion HomographyNet
Builds on: MVDiffusion Stable Diffusion inpainting PanoDiffusion BLIP-2
Closest papers by meaning
- PanoDiffusion: 360-degree Panorama Outpainting via Diffusion (ICLR 2024 · similarity 0.87)
- CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation (ICLR 2025 · similarity 0.87)
- Taming Stable Diffusion for Text to 360 Panorama Image Generation (CVPR 2024 · similarity 0.87)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.87)
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.86)
- Unboxed: Geometrically and Temporally Consistent Video Outpainting (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.