Research map / 3D vision / Depth and stereo
Cameras as Relative Positional Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares camera-conditioning methods for multiview transformers and proposes PRoPE, a relative attention-level encoding using full projective frustum relationships, which improves novel view synthesis, stereo depth, and spatial cognition.
Problem
How to condition multiview vision transformers on camera geometry (intrinsics and extrinsics) so that they generalize across world frames, intrinsics, and sequence lengths.
Approach
Projective Positional Encoding (PRoPE) applies the relative projective transform P_i1 P_i2^{-1} of full camera frustums inside GTA-style attention, combined with RoPE for patch positions, so attention captures both intrinsics and extrinsics relatively without a global frame.
Key results
- On RealEstate10K with constant intrinsics, PRoPE reaches 22.80 PSNR / 0.146 LPIPS versus 22.51 PSNR for GTA and 20.48 for Plücker raymaps.
- With varying intrinsics, PRoPE gets 21.42 PSNR on RealEstate10K versus 20.56 for naive raymaps and 15.77 for GTA.
- UniMatch + PRoPE reduces abs_rel on RGBD from 0.123 to 0.105 and on SUN3D from 0.131 to 0.117.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace absolute raymaps with relative projective camera encoding | LVSM · RealEstate10K | PSNR: 20.48 -> 22.80 | Helped |
| replace GTA with PRoPE | LVSM · RealEstate10K | PSNR: 22.51 -> 22.80 | Helped |
| replace GTA with PRoPE | LVSM · Objaverse | PSNR: 23.70 -> 23.70 | No clear effect |
| use relative projective camera encoding instead of raymap or relative pose encoding | LVSM · RealEstate10K with varying intrinsics | PSNR: 20.56 -> 21.42 | Helped |
| add token-level CamRay to PRoPE (ablation) | LVSM · RealEstate10K with varying intrinsics | PSNR: 21.42 -> 21.78 | Helped |
| add token-level CamRay to PRoPE (ablation) | LVSM · RealEstate10K, out-of-distribution intrinsics | PSNR: 21.42 -> 21.86; 22.40 -> 22.40 | Mixed |
| add PRoPE to UniMatch | UniMatch · RGBD | Abs Rel: 0.123 -> 0.105 | Helped |
| add PRoPE to UniMatch | UniMatch · SUN3D | Abs Rel: 0.131 -> 0.117 | Helped |
| add PRoPE to UniMatch | UniMatch · Scenes11 | Abs Rel: 0.065 -> 0.049 | Helped |
| use PRoPE+CamRay instead of Plücker raymaps | spatial cognition model · DL3DV, 9 views | accuracy: 76.9% -> 93.0% | Helped |
| add PRoPE to LVSM | LVSM · RealEstate10K, 100× compute | PSNR: 25.64 -> 26.56 | Helped |
| add PRoPE to CAT3D | CAT3D · CAT3D, 3-view | PSNR: 16.62 -> 16.93 | Helped |
Surprising findings
- Plücker raymaps, which encode full camera information, underperform relative encodings across out-of-distribution sequence lengths and focal lengths.
- Adding CamRay to PRoPE hurts performance for intrinsics extrapolation at higher zoom levels.
Limitations
- Numerical stability when multiplying ill-conditioned projective matrices, e.g. telephoto focal lengths.
- Not extended to distorted camera models.
- Multi-frequency encoding of camera parameters is nontrivial due to non-commutativity of projective transforms.
Experimental setup
Datasets: RealEstate10K Objaverse DL3DV RGBD SUN3D Scenes11
Benchmarks: RealEstate10K Objaverse RGBD SUN3D Scenes11 DL3DV
Metrics: PSNR SSIM LPIPS abs_rel sq_rel RMSE accuracy
Baselines: Plücker raymap Naive raymap CAPE GTA GTA+CamRay CAT3D UniMatch
Builds on: LVSM GTA CaPE RoPE UniMatch
Closest papers by meaning
- DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers (NeurIPS 2026 · similarity 0.90)
- GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers (ICLR 2024 · similarity 0.87)
- Understanding Multi-View Transformers (NeurIPS 2026 · similarity 0.87)
- Thinking in Boxes: 3D Editing in Real Images Made Easy (NeurIPS 2026 · similarity 0.87)
- CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation (NeurIPS 2026 · similarity 0.86)
- Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.