Research map / 3D vision / Gaussian splatting
SplatFormer: Point Transformer for Robust 3D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SplatFormer, a PTv3-based point transformer that refines an initial 3D Gaussian Splatting set in one feed-forward pass to remove artifacts under out-of-distribution camera views.
Problem
3D Gaussian Splatting rendering quality degrades sharply when test views lie far outside the input camera distribution (e.g., top-down views from low-elevation captures), and existing regularization and prior-based methods do not generalize to these OOD-NVS settings.
Approach
Adapt the Point Transformer V3 to operate directly on 3DGS splats as point clouds with Gaussian attributes as features, predicting residual corrections to the splat attributes that are trained with 2D photometric and LPIPS losses on OOD renderings from large synthetic datasets.
Key results
- On Objaverse-OOD (elevation >= 70 degrees), SplatFormer reaches 23.06 PSNR / 0.821 SSIM / 0.170 LPIPS, versus 20.56 PSNR for the next best (2DGS).
- On ShapeNet-OOD, SplatFormer achieves 27.98 PSNR / 0.920 SSIM / 0.136 LPIPS, compared with 23.52 PSNR for 2DGS.
- On GSO-OOD, SplatFormer reaches 25.01 PSNR versus 23.29 for 2DGS, and improves SSIM and LPIPS on real-world iPhone captures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| refine 3DGS with SplatFormer | 3DGS · ShapeNet-OOD | PSNR: 20.21 -> 27.98 | Helped |
| refine 3DGS with SplatFormer | 3DGS · Objaverse-OOD | PSNR: 19.24 -> 23.06 | Helped |
| refine 3DGS with SplatFormer | 3DGS · GSO-OOD | PSNR: 21.78 -> 25.01 | Helped |
| refine 3DGS with SplatFormer | 3DGS · RealWorld-OOD | SSIM: 0.877 -> 0.902 | Helped |
| refine 3DGS with SplatFormer | 3DGS · RealWorld-OOD | PSNR: 23.83 -> 24.33 | Helped |
| add residual prediction of 3DGS attributes (ablation) | PTv3 SplatFormer · Objaverse-OOD | PSNR: 21.36 -> 23.06 | Helped |
| use PTv3 instead of Mink as the backbone (ablation) | residual-prediction SplatFormer · Objaverse-OOD | PSNR: 22.67 -> 23.06 | Helped |
| apply DiffBIR-stage1 denoising to OOD 3DGS renderings and retrain 3DGS | 3DGS · ShapeNet-OOD | PSNR: 20.21 -> 25.16 | Helped |
Surprising findings
- Multi-view-trained 3DGS priors and regularized methods such as 2DGS and SplatFields give only modest gains on OOD views, and Nerfbusters can remove real scene content as floaters.
- 2D denoising with DiffBIR, even with retraining 3DGS on denoised images, underperforms the 3D refinement approach.
Limitations
- Struggles to reconstruct fine-grained details and complex textures.
- Generalization to real-world captures is limited by synthetic training data and lighting realism.
- Training is computationally expensive and scaling to large unbounded scenes is not yet demonstrated.
Experimental setup
Datasets: ShapeNet Objaverse-1.0 Google Scanned Objects MVImgNet
Benchmarks: ShapeNet-OOD Objaverse-OOD GSO-OOD RealWorld-OOD (iPhone captures)
Metrics: PSNR SSIM LPIPS Depth-MAE Normal-MAE
Baselines: MipNeRF360 InstantNGP 3DGS 2DGS SplatFields InstantSplat FSGS SSDNeRF Nerfbusters SyncDreamer EscherNet LaRa
Builds on: 3D Gaussian Splatting Point Transformer V3 LaRa gsplat
Closest papers by meaning
- GaussianPro: 3D Gaussian Splatting with Progressive Propagation (ICML 2024 · similarity 0.92)
- Speedy-Splat: Fast 3D Gaussian Splatting with Sparse Pixels and Sparse Primitives (CVPR 2025 · similarity 0.92)
- Splatter Image: Ultra-Fast Single-View 3D Reconstruction (CVPR 2024 · similarity 0.92)
- SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis (CVPR 2025 · similarity 0.92)
- LP-3DGS: Learning to Prune 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.91)
- IBGS: Image-Based Gaussian Splatting (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.