Research map / 3D vision / Human pose estimation
Cameras as Rays: Pose Estimation via Ray Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes representing cameras as bundles of Plücker rays predicted per image patch, with regression and denoising-diffusion models that achieve state-of-the-art sparse-view camera pose estimation on CO3D.
Problem
Estimating camera poses from sparsely sampled views (fewer than 10 images), where SfM fails and global pose parameterizations are hard for neural networks to regress and ambiguous under symmetry and partial observation.
Approach
Over-parameterize each camera as a set of per-patch 6-D Plücker rays predicted by a transformer over DINOv2 patch features, recovering intrinsics and extrinsics via least squares; a DDPM-style diffusion model denoises these rays to sample plausible modes.
Key results
- Ray Diffusion reaches 93.3% rotation accuracy @15 degrees on seen CO3D categories with 8 images, versus 85.5% for RelPose++.
- On unseen CO3D categories with 8 images, Ray Diffusion gets 88.1% rotation accuracy and 71.4% camera center accuracy, above PoseDiffusion's 67.7% rotation accuracy.
- Ray Regression alone (0.133 s per 8-image inference) already beats prior methods, while Ray Diffusion takes 11.1 s per inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ray regression with ray diffusion | Ray Regression · CO3D seen categories (8 images) | rotation accuracy @15°: 89.2 -> 93.3 | Helped |
| replace ray regression with ray diffusion | Ray Regression · CO3D unseen categories (8 images) | rotation accuracy @15°: 81.9 -> 88.1 | Helped |
| replace ray regression with ray diffusion | Ray Regression · CO3D seen categories (8 images) | camera center accuracy @0.1: 76.2 -> 84.1 | Helped |
| replace ray regression with ray diffusion | Ray Regression · CO3D unseen categories (8 images) | camera center accuracy @0.1: 63.9 -> 71.4 | Helped |
| increase ray resolution from 2 × 2 to 16 × 16 (ablation) | Ray Regression · CO3D (hydrant, wineglass; 3 images) | Rot@15: 52.5 -> 84.0 | Helped |
| increase ray resolution from 2 × 2 to 16 × 16 (ablation) | Ray Regression · CO3D (hydrant, wineglass; 3 images) | CC@0.01: 72.5 -> 89.8 | Helped |
| stop backward diffusion early and use the predicted X0 at T=30 (ablation) | Ray Diffusion · CO3D (8 images) | accuracy | Helped |
| remove stochastic noise during DDPM inference (ablation) | Ray Diffusion · CO3D | accuracy | Helped |
Surprising findings
- On RealEstate10K, a constant identity-rotation baseline performs strongly because the dataset has a forward-facing bias.
- Stopping DDPM backward diffusion early (using predicted x0 at T=30) and removing stochastic noise improves accuracy.
Limitations
- Ray Diffusion inference is much slower than regression and other feedforward methods (11.1 s for 8 images).
- No geometric consistency is enforced across predicted rays, unlike classical pipelines.
- Accuracy drops as the number of images grows beyond the training range for camera center accuracy, and COLMAP is more precise when it converges.
Experimental setup
Datasets: CO3Dv2 MegaDepth RealEstate10K
Benchmarks: CO3Dv2 RealEstate10K
Metrics: rotation accuracy @15 degrees camera center accuracy @0.1 scene scale AUC inference time memory usage
Baselines: COLMAP (SP+SG) RelPose RelPose++ PoseDiffusion PoseDiffusion w/o GGS R+T Regression
Builds on: DiT DINOv2 DDPM PoseDiffusion RelPose++
Closest papers by meaning
- Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis (NeurIPS 2024 · similarity 0.90)
- Object Pose Estimation via the Aggregation of Diffusion Features (CVPR 2024 · similarity 0.89)
- DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion (CVPR 2025 · similarity 0.88)
- LEAP: Liberate Sparse-View 3D Modeling from Camera Poses (ICLR 2024 · similarity 0.88)
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images (ICLR 2025 · similarity 0.87)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.