Research map / 3D vision / Gaussian splatting
SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
SplatFlow jointly generates multi-view image, depth, and Plücker ray latents with a fine-tuned SD3-based rectified flow model and decodes them into 3DGS via a feed-forward GSDecoder, enabling training-free 3DGS editing, pose estimation, and novel view synthesis.
Problem
Text-based generation and editing of real-world 3D scenes with 3D Gaussian Splatting lack a unified framework; existing methods are specialized, task-focused, and often rely on SDS or complex lifting pipelines.
Approach
A multi-view rectified flow model in SD3's latent space jointly models images, depths, and camera rays so that a feed-forward GSDecoder can produce pixel-aligned 3DGS, and standard training-free inversion (SDEdit) and inpainting (RePaint) turn the same generator into an editing and 3D-task tool.
Key results
- Text-to-3DGS on MVImgNet: FID-10K 34.85 and CLIPScore 31.43, versus Director3D's 39.55 and 30.48.
- Camera pose estimation on MVImgNet: 28.8 rotation accuracy@5 and 64.9 camera center accuracy@0.05 without depth, outperforming Ray Diffusion's 17.5 and 24.1.
- 3D object replacement on 100 MVImgNet scenes: CLIPScore 28.47 and CLIP D-sim 0.169 versus DGE's 27.43 and 0.102; with SDS++ 31.30 and 0.224.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SplatFlow instead of Director3D | Text-to-3DGS generation · MVImgNet | FID-10K: 39.55 -> 34.85 | Helped |
| use SplatFlow instead of Director3D | Text-to-3DGS generation · DL3DV | FID-2.4K: 88.44 -> 79.91 | Helped |
| add SDS++ refinement (ablation) | SplatFlow · MVImgNet | CLIPScore: 31.43 -> 32.30 | Helped |
| add SDS++ refinement (ablation) | SplatFlow · MVImgNet | FID-10K: 34.85 -> 35.46 | Hurt |
| use SplatFlow instead of DGE | 3DGS object replacement · MVImgNet object replacement | CLIPScore: 27.43 -> 28.47 | Helped |
| use SplatFlow instead of DGE | 3DGS object replacement · MVImgNet object replacement | CLIP D-sim: 0.102 -> 0.169 | Helped |
| add SDS++ refinement (ablation) | SplatFlow · MVImgNet object replacement | CLIP D-sim: 0.169 -> 0.224 | Helped |
| remove depth latents (ablation) | SplatFlow camera pose estimation · MVImgNet camera pose estimation | Camera Center accuracy@0.05: 62.3 -> 64.9 | Helped |
| increase the number of input views from N=2 to N=6 | SplatFlow novel view synthesis · MVImgNet interpolation | PSNR: 14.73 -> 18.82 | Helped |
| increase the number of input views from N=2 to N=6 | SplatFlow novel view synthesis · MVImgNet extrapolation | PSNR: 15.15 -> 17.96 | Helped |
Surprising findings
- Camera pose estimation was more accurate without depth latents than with them, which the authors attribute to joint depth and pose generation.
Limitations
- Trained on a smaller dataset than Director3D, without full DL3DV-10K or LAION, so comparisons are not like-for-like.
- Text-to-3DGS FID is slightly worse on DL3DV (79.91 vs 88.44 is better; but SDS++ variant raises FID), with refinement trading image quality for text alignment.
Experimental setup
Datasets: MVImgNet DL3DV-7K
Benchmarks: MVImgNet validation set DL3DV validation set
Metrics: FID CLIPScore CLIP D-sim rotation accuracy camera center accuracy PSNR SSIM LPIPS AbsRel delta1 accuracy
Baselines: Director3D DreamScene LucidDreamer DGE MVInpainter RelPose++ Ray Diffusion Ray Regression
Builds on: Stable Diffusion 3 Director3D RePaint SDEdit Ray Diffusion
Closest papers by meaning
- EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting (CVPR 2025 · similarity 0.92)
- SplatFormer: Point Transformer for Robust 3D Gaussian Splatting (ICLR 2025 · similarity 0.92)
- Generative Sparse-View Gaussian Splatting (CVPR 2025 · similarity 0.91)
- GaussianSpa: An "Optimizing-Sparsifying" Simplification Framework for Compact and High-Quality 3D Gaussian Splatting (CVPR 2025 · similarity 0.91)
- 3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors (NeurIPS 2024 · similarity 0.91)
- Speedy-Splat: Fast 3D Gaussian Splatting with Sparse Pixels and Sparse Primitives (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.