Research map / 3D vision / Gaussian splatting
SfM-Free 3D Gaussian Splatting via Hierarchical Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HT-3DGS, an SfM-free 3D Gaussian Splatting method for video that trains per-segment 3DGS models, merges them via importance-based pruning and union, and retrains with pseudo-view and video-frame-interpolation supervision.
Problem
Novel view synthesis with 3D Gaussian Splatting normally needs known camera poses and an SfM point cloud; for small-motion video, SfM is slow and fragile, and standard densification leaves sparse Gaussian coverage in regions unseen by the first frame.
Approach
Hierarchically train base 3DGS models on overlapping video segments, align and merge them by pruning low-importance Gaussians and taking the union (acting as densification), and retrain with multi-source supervision from base-model pseudo-views and VFI-interpolated frames; VFI also refines relative pose estimation.
Key results
- On Tanks and Temples, averages 33.53 dB PSNR versus 31.28 dB for CF-3DGS (+2.25 dB), with a best-scene gain of 3.72 dB on Barn.
- On CO3D-V2, averages a 1.74 dB PSNR gain over the strongest prior SfM-free baselines, with a top gain of 3.90 dB on scene 34_1403_4393.
- Hierarchical training alone boosts PSNR by 1.19-1.58 dB over the single-segment baseline across levels L=0 to L=3, and L=2 is chosen as the saturation point.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hierarchical training to the baseline (ablation) | SfM-free 3DGS · Tanks and Temples | PSNR: 31.50 -> 33.02 | Helped |
| Add VFI (ablation) | CF-3DGS · Tanks and Temples | PSNR: 31.28 -> 31.45 | Helped |
| Add VFI to global training (ablation) | Baseline + global training · Tanks and Temples | PSNR: 31.52 -> 31.95 | Helped |
| Add VFI to hierarchical training (ablation) | Hierarchical training · Tanks and Temples | PSNR: 33.02 -> 33.37 | Helped |
| Add supervision from base 3DGS models (ablation) | HT + VFI · Tanks and Temples | PSNR: 33.37 -> 33.53 | Helped |
| Use global training instead of the baseline configuration (ablation) | SfM-free 3DGS · Tanks and Temples | PSNR: 31.50 -> 31.52 | No clear effect |
| Use progressive rather than hierarchical training (ablation) | SfM-free 3DGS · Tanks and Temples | PSNR: 32.82 -> 33.02 | Hurt |
| Increase hierarchical training level from L=0 to L=2 (ablation) | HT-3DGS · Tanks and Temples | PSNR: 31.50 -> 33.02 | Helped |
| Increase hierarchical training level from L=2 to L=3 (ablation) | HT-3DGS · Tanks and Temples | PSNR: 33.02 -> 33.08 | No clear effect |
| Use heuristic intrinsics instead of known camera intrinsics (ablation) | HT-3DGS · Tanks and Temples | PSNR: 33.53 -> 32.17 | Hurt |
| Use HT-3DGS instead of CF-3DGS | SfM-free novel view synthesis · Tanks and Temples | PSNR: 31.28 -> 33.53 | Helped |
| Use HT-3DGS instead of CF-3DGS | SfM-free novel view synthesis · CO3D-V2 | PSNR: +1.74 dB | Helped |
Surprising findings
- Running COLMAP SfM poses with 3DGS (30.20 dB) or NeRF (26.61 dB) performs worse than the proposed SfM-free method on Tanks and Temples, attributed to COLMAP struggling in low-texture scenes.
- Hierarchical merging reduces memory storage (by 0.22-0.26 GB) despite adding Gaussians, because pruning before merging keeps counts stable.
Limitations
- Requires longer training time than baselines.
- Struggles with large camera motion or low-quality inputs, which can cause alignment errors when merging models.
- ATE pose accuracy improves inconsistently, possibly due to VFI interpolation errors.
Experimental setup
Datasets: Tanks and Temples CO3D-V2
Benchmarks: Tanks and Temples CO3D-V2
Metrics: PSNR SSIM LPIPS ATE RPEt RPEr memory size
Baselines: BARF SC-NeRF Nope-NeRF CF-3DGS GGRt InstantSplat COGS
Builds on: 3D Gaussian Splatting CF-3DGS adaptive density control compact 3DGS importance pruning
Closest papers by meaning
- KeyGS: A Keyframe-Centric Gaussian Splatting Method for Monocular Image Sequences (AAAI 2025 · similarity 0.93)
- GaussianPro: 3D Gaussian Splatting with Progressive Propagation (ICML 2024 · similarity 0.93)
- COLMAP-Free 3D Gaussian Splatting (CVPR 2024 · similarity 0.92)
- DOGS: Distributed-Oriented Gaussian Splatting for Large-Scale 3D Reconstruction Via Gaussian Consensus (NeurIPS 2024 · similarity 0.92)
- HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.91)
- FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.