Research map / 3D vision / Gaussian splatting
Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Fuse-and-Refine module that merges and refines pixel-aligned Gaussians from feed-forward 3DGS models in a canonical voxel space using a sparse voxel transformer, enabling zero-shot history-aware streaming reconstruction of dynamic scenes.
Problem
Pixel-aligned feed-forward 3D Gaussian Splatting produces redundant primitives across overlapping views, constrains primitive positions to input rays, and does not naturally handle temporal fusion for dynamic scenes.
Approach
Deposit pixel-aligned Gaussian primitives (with features) onto a coarse-to-fine voxel hierarchy in a canonical 3D cost volume, process coarse voxels with a sparse voxel transformer, and predict residual refinements of Gaussian attributes; for video, warp past primitives with 2D point tracking and triangulation and fuse them with current-frame primitives without any dynamic training.
Key results
- On RealEstate10K, the method reaches 28.47 PSNR / 0.907 SSIM / 0.078 LPIPS, versus 28.10 PSNR for GS-LRM (official).
- On DL3DV, the full model reaches 30.61 PSNR vs 28.59 for GS-LRM, with 52.5 ms vs 52.8 ms runtime.
- Fusing about 200K primitives takes roughly 15 ms in the voxel transformer, and the streaming system runs at 15 fps with 350 ms delay on a single H100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Fuse-and-Refine to GS-LRM | GS-LRM · RealEstate10K | PSNR: 28.10 -> 28.47 | Helped |
| add Fuse-and-Refine to GS-LRM | GS-LRM · DL3DV | PSNR: 28.59 -> 30.61 | Helped |
| remove coarse-to-fine voxel hierarchy (ablation) | Ours on DL3DV · DL3DV | PSNR: 30.61 -> 20.62 | Hurt |
| remove sparse voxel representation (ablation) | Ours on DL3DV · DL3DV | PSNR: 30.61 -> 29.69 | Hurt |
| replace voxel transformer with 3D CNN (ablation) | Ours on DL3DV · DL3DV | PSNR: 30.61 -> 29.44 | Hurt |
| remove splat features (ablation) | Ours on DL3DV · DL3DV | PSNR: 30.61 -> 29.40 | Hurt |
| replace learned fusion with non-learning fusion (ablation) | GS-LRM with fusion · DL3DV | PSNR: 28.59 -> 12.57 | Hurt |
Surprising findings
- Naive non-learning fusion of Gaussians collapses quality (PSNR 12.57 on DL3DV), showing heuristic merging is harmful without learning.
- A coarse-to-fine hierarchy is essential: removing it drops PSNR to 20.62 on DL3DV.
Limitations
- Depends on reasonably good initial primitives; gains are limited under large-baseline setups such as LongVolumetricVideo where GS-LRM also degrades.
- Temporal artifacts persist in dynamic regions because the simple 3D warping relies on error-prone 2D tracking lifted to 3D from sparse views.
Experimental setup
Datasets: RealEstate10K DL3DV Neural3DVideo LongVolumetricVideo
Benchmarks: RealEstate10K DL3DV Neural3DVideo LongVolumetricVideo
Metrics: PSNR SSIM LPIPS Flicker Time (ms)
Baselines: pixelSplat MVSplat TranSplat HiSplat OmniScene DepthSplat GS-LRM 4DGS
Builds on: GS-LRM 3D Gaussian Splatting TAPIR RAFT
Closest papers by meaning
- CompactSplat: Spatially Adaptive Gaussian Distribution for Feedforward Scene Reconstruction (NeurIPS 2026 · similarity 0.92)
- Speedy-Splat: Fast 3D Gaussian Splatting with Sparse Pixels and Sparse Primitives (CVPR 2025 · similarity 0.92)
- HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.91)
- Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene (ICLR 2025 · similarity 0.91)
- Fully Explicit Dynamic Gaussian Splatting (NeurIPS 2024 · similarity 0.91)
- pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.