Research map / 3D vision / Gaussian splatting
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Scaffold-GS, which anchors view-adaptive neural Gaussians to a sparse SfM-derived voxel grid, decoding their attributes on the fly with small MLPs and adding anchor growing/pruning, to get 3D-GS-level quality with fewer, more compact Gaussians.
Problem
3D Gaussian Splatting produces highly redundant Gaussians that overfit each training view and ignore scene geometry, making it less robust to large view changes, texture-less regions, lighting effects, and multi-scale scenes.
Approach
Organize 3D Gaussians hierarchically: anchor points on a sparse voxel grid carry learnable features and offsets, and small MLPs decode the attributes (opacity, color, scale, rotation) of the spawned neural Gaussians from the anchor feature and camera-anchor viewing direction and distance, with error-based growing and opacity-based pruning of anchors.
Key results
- On Tanks&Temples and Deep Blending, Scaffold-GS surpasses 3D-GS in PSNR (24.04 vs 23.14; 30.43 vs 29.41) while using about 4-10x less storage (e.g., 66 MB vs 676 MB on Deep Blending).
- On BungeeNeRF, PSNR improves from 24.89 (3D-GS) to 27.01 with storage reduced from 1606 MB to 203 MB.
- Rendering stays real-time (around 100 FPS at 1K resolution), comparable to 3D-GS (e.g., 102 vs 97 FPS on Mip-NeRF360).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use anchor-structured, view-adaptive neural Gaussians instead of 3D-GS | 3D-GS · Tanks&Temples | PSNR: 23.14 -> 24.04 | Helped |
| use anchor-structured, view-adaptive neural Gaussians instead of 3D-GS | 3D-GS · Deep Blending | PSNR: 29.41 -> 30.43 | Helped |
| use anchor-structured, view-adaptive neural Gaussians instead of 3D-GS | 3D-GS · Mip-NeRF360 | PSNR: 27.21 -> 27.72 | Helped |
| use anchor-structured, view-adaptive neural Gaussians instead of 3D-GS | 3D-GS · Tanks&Temples | LPIPS: 0.183 -> 0.172 | Helped |
| use anchor-structured, view-adaptive neural Gaussians instead of 3D-GS | 3D-GS · Mip-NeRF360 | SSIM: 0.815 -> 0.811 | Hurt |
| use anchor growing (ablation) | Scaffold-GS · DB-PLAYROOM | PSNR: 29.12 -> 31.07 | Helped |
| use anchor pruning (ablation) | Scaffold-GS · DB-PLAYROOM | storage size (MB): 71 -> 61 | Helped |
| use anchor growing and pruning (ablation) | Scaffold-GS · DB-PLAYROOM | PSNR: 28.45 -> 31.07 | Helped |
| use anchor growing (ablation) | Scaffold-GS · DB-DRJOHNSON | PSNR: 28.51 -> 29.79 | Helped |
| use anchor pruning (ablation) | Scaffold-GS · DB-DRJOHNSON | storage size (MB): 76 -> 50 | Helped |
| use view-frustum culling and opacity-based selection (ablation) | Scaffold-GS · DB-PLAYROOM | PSNR: 30.4 -> 31.07 | No clear effect |
| use view-frustum culling and opacity-based selection (ablation) | Scaffold-GS · DB-PLAYROOM | FPS: 84 -> 150 | Helped |
Surprising findings
- Filtering strategies (view frustum culling and opacity-based selection) have no notable effect on fidelity but substantially speed up inference.
Limitations
- Initialization from SfM points is suboptimal for scenes with large texture-less regions, since point clouds are extremely sparse there.
- Growing and pruning still rely on heuristic thresholds (e.g., per-scene gradient thresholds chosen for intricate or texture-less scenes).
Experimental setup
Datasets: Mip-NeRF360 Tanks&Temples DeepBlending Synthetic Blender BungeeNeRF VR-NeRF
Benchmarks: Mip-NeRF360 Tanks&Temples Deep Blending Synthetic Blender BungeeNeRF VR-NeRF
Metrics: PSNR SSIM LPIPS FPS storage size (MB)
Baselines: 3D-GS Mip-NeRF360 iNGP Plenoxels
Builds on: 3D Gaussian Splatting Point-NeRF Plenoxels Instant-NGP
Closest papers by meaning
- Toy-GS: Assembling Local Gaussians for Precisely Rendering Large-Scale Free Camera Trajectories (AAAI 2025 · similarity 0.85)
- LITA-GS: Illumination-Agnostic Novel View Synthesis via Reference-Free 3D Gaussian Splatting and Physical Priors (CVPR 2025 · similarity 0.85)
- 3D Geometry-Aware Deformable Gaussian Splatting for Dynamic View Synthesis (CVPR 2024 · similarity 0.85)
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (CVPR 2024 · similarity 0.84)
- MaskGaussian: Adaptive 3D Gaussian Representation from Probabilistic Masks (CVPR 2025 · similarity 0.84)
- Generative Sparse-View Gaussian Splatting (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.