Research map / 3D vision / 3D generation
LEAP: Liberate Sparse-View 3D Modeling from Camera Poses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LEAP, a pose-free generalizable NeRF that builds a shared learnable neural volume and updates it via feature-similarity attention over unposed sparse views, outperforming pose-based methods with estimated poses.
Problem
Novel view synthesis and 3D modeling from sparse, unposed wide-baseline views, where estimated camera poses are noisy and degrade existing pose-based generalizable NeRFs.
Approach
Eliminates all pose-based operations: a shared learnable neural volume in the canonical view's local camera frame is updated by cross-attention over multi-view image features (feature-similarity-driven 2D-3D mapping) instead of projecting 3D points with camera poses; poses are used only as training supervision via rendering loss.
Key results
- On object-centric sets, LEAP reaches 29.10 PSNR on OmniObject3D vs 26.56 for FORGE with predicted poses, and matches or beats pose-based methods with ground-truth poses.
- LEAP runs in 0.3 sec per scene versus 15 min for FORGE with predicted poses and 2 min for PixelNeRF.
- On DTU, LEAP-pretrain reaches 18.07 PSNR, comparable to SPARF with noisy poses (18.57 at sigma=0.05), while being pose-free and feed-forward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pose-free LEAP instead of the strongest predicted-pose baseline, FORGE | LEAP · OmniObject3D; Kubric-ShapeNet seen and novel; Objaverse | PSNR: 26.56 -> 29.10; 26.61 -> 29.86; 25.57 -> 28.22; 23.67 -> 26.77 | Helped |
| Use pose-free LEAP instead of the strongest predicted-pose baseline, FORGE | LEAP · OmniObject3D; Kubric-ShapeNet seen and novel; Objaverse | LPIPS: 0.108 -> 0.057; 0.106 -> 0.067; 0.107 -> 0.070; 0.226 -> 0.103 | Helped |
| Pretrain LEAP on large-scale object-centric data before DTU evaluation | LEAP · DTU | PSNR: 15.37 -> 18.07 | Helped |
| Pretrain LEAP on large-scale object-centric data before DTU evaluation | LEAP · DTU | SSIM: 0.535 -> 0.671 | Helped |
| Pretrain LEAP on large-scale object-centric data before DTU evaluation | LEAP · DTU | LPIPS: 0.478 -> 0.371 | Helped |
| Use category-level world coordinate instead of local camera coordinate (ablation) | LEAP · OmniObject3D seen categories | PSNR: 29.10 -> 31.23 | Helped |
| Use category-level world coordinate instead of local camera coordinate (ablation) | LEAP · Kubric-ShapeNet novel categories | PSNR | Hurt |
| Remove the multi-view encoder (ablation) | LEAP · Object-centric evaluation | novel-view synthesis performance | Hurt |
| Keep only global consensus reasoning (GCR) layers (ablation) | LEAP · Object-centric evaluation | novel-view synthesis performance | Hurt |
| Keep only non-canonical view update (NVU) layers (ablation) | LEAP · Object-centric evaluation | novel-view synthesis performance | Hurt |
| Use two 2D-3D mapping layers instead of four (ablation) | LEAP · Object-centric evaluation | novel-view synthesis performance | Hurt |
| Reduce the number of input views from two to one (ablation) | LEAP · Object-centric evaluation | novel-view synthesis performance | Hurt |
Surprising findings
- Pose-free LEAP outperforms several pose-based methods even when those use ground-truth poses on some object-centric sets.
- Learned attention behaves like epipolar geometry: a 2D point is lifted to a ray whose projection in other views matches the epipolar line.
Limitations
- The neural volume spans a uniformly bounded 3D space, so it does not handle unbounded scenes.
- Performance drops noticeably when only one input view is given.
- Scene-level DTU results trail SPARF with noisy poses unless the model is pretrained on object-centric data.
Experimental setup
Datasets: OmniObject3D Kubric-ShapeNet Objaverse DTU
Benchmarks: OmniObject3D Kubric-ShapeNet-seen Kubric-ShapeNet-novel Objaverse DTU
Metrics: PSNR SSIM LPIPS inference time
Baselines: PixelNeRF FORGE SPARF SRT Zero123
Builds on: PixelNeRF DINOv2 Transformer NeRF
Closest papers by meaning
- Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis (NeurIPS 2024 · similarity 0.92)
- Cameras as Rays: Pose Estimation via Ray Diffusion (ICLR 2024 · similarity 0.88)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.88)
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images (ICLR 2025 · similarity 0.88)
- 3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation (ICLR 2024 · similarity 0.88)
- Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.