Research map / 3D vision / Gaussian splatting
3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces 3DGS's EWA linearized splatting with an Unscented Transform over sigma points and evaluates particles in 3D, enabling rasterized rendering with distorted cameras, rolling shutter, and hybrid secondary-ray tracing.
Problem
3DGS rasterization relies on first-order EWA linearization, which limits it to pinhole cameras with approximation errors under distortion, cannot model time-dependent effects like rolling shutter, and cannot trace secondary rays for reflections and refractions.
Approach
Approximate each 3D Gaussian with sigma points that are projected exactly through any nonlinear camera function and re-fit a 2D Gaussian via the Unscented Transform (no Jacobians), and evaluate particle response in 3D at the max-response point along the ray, aligning with 3DGRT so the same representation can be rasterized and ray-traced.
Key results
- On MipNeRF360, Ours (sorted) reaches 200 FPS and PSNR 27.26 vs 3DGRT 52 FPS at 27.20 PSNR, while plain Ours runs at 265 FPS vs 3DGS 347 FPS.
- On ScanNet++ fisheye, Ours (sorted) achieves PSNR 29.11 and SSIM 0.910 versus FisheyeGS 28.15 and 0.901, using 0.38M Gaussians vs 1.07M.
- On Tanks & Temples, Ours (sorted) achieves SSIM 0.844 and LPIPS 0.172, the best among compared methods in those columns.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3DGUT with the Unscented Transform for nonlinear camera projection | 3D Gaussian Splatting · Scannet++ | PSNR: 28.15 -> 29.11 | Helped |
| Train on the original distorted views rather than undistorted views | 3DGUT · Distorted fisheye views | visual quality | Helped |
| Use per-ray sorting (ablation) | 3DGUT · MipNeRF360 | SSIM: 0.810 -> 0.812 | Helped |
| Use per-ray sorting (ablation) | 3DGUT · Tanks & Temples | PSNR: 23.21 -> 22.90 | Hurt |
Surprising findings
- Ours (sorted) renders at 200 FPS on MipNeRF360 while its sorting approximation is only modestly slower than plain 3DGS-style splatting, yet it matches 3DGRT-level quality at about 4x the speed.
Limitations
- Still marginally slower than 3DGS due to UT evaluation and 3D particle evaluation overhead.
- Projected shape deviates from a 2D Gaussian under large distortion, degrading tile/pixel assignment.
- Single-point evaluation per primitive cannot accurately render overlapping Gaussians.
Experimental setup
Datasets: MipNeRF360 Tanks & Temples ScanNet++ Waymo
Benchmarks: MipNeRF360 Tanks & Temples ScanNet++ Waymo
Metrics: PSNR SSIM LPIPS FPS
Baselines: 3DGS StopThePop 3DGRT EVER ZipNeRF FisheyeGS
Builds on: 3D Gaussian Splatting 3DGRT EWA splatting Unscented Transform StopThePop
Closest papers by meaning
- IBGS: Image-Based Gaussian Splatting (NeurIPS 2025 · similarity 0.89)
- 3D-HGS: 3D Half-Gaussian Splatting (CVPR 2025 · similarity 0.89)
- GaussianPro: 3D Gaussian Splatting with Progressive Propagation (ICML 2024 · similarity 0.89)
- SplatFormer: Point Transformer for Robust 3D Gaussian Splatting (ICLR 2025 · similarity 0.89)
- UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mapping (CVPR 2025 · similarity 0.89)
- Speedy-Splat: Fast 3D Gaussian Splatting with Sparse Pixels and Sparse Primitives (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.