Research map / 3D vision / 3D generation
DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DiffSplat fine-tunes pretrained text-to-image diffusion models (SD1.5, SDXL, PixArt, SD3) to generate multi-view Gaussian splat grids ("splat latents"), trained with a diffusion loss plus a 3D rendering loss on splat grids produced by a lightweight reconstruction model.
Problem
Text- or image-conditioned 3D content generation is limited by scarce high-quality 3D data and by multi-view inconsistency in two-stage pipelines that reconstruct 3D from generated multi-view images.
Approach
Treat pixel-aligned Gaussian splat property grids as images in a special style so that pretrained image diffusion models (after fine-tuning a VAE to encode splat grids) can directly generate 3DGS, with an added 3D rendering loss to enforce multi-view consistency.
Key results
- On T3Bench single-object prompts, DiffSplat reaches 30.95% CLIP similarity and 81.00% CLIP R-Precision, outperforming LGM (29.96%, 78.00%).
- On GSO image-conditioned generation, DiffSplat achieves 22.91 PSNR, 0.892 SSIM and 0.107 LPIPS, versus 19.65 PSNR for the best baseline GRM.
- Switching the base model from SD1.5 to SD3-medium improves the splat auto-encoding PSNR from 30.33 to 31.08 and text-conditioned CLIP similarity from 28.66% to 30.15%; the full method generates in about 1-2 seconds on an A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add coordinate maps to RGB inputs (ablation) | Lightweight splat reconstruction model · Splat reconstruction | PSNR: 27.43 -> 29.87 | Helped |
| Add normal maps to RGB inputs (ablation) | Lightweight splat reconstruction model · Splat reconstruction | PSNR: 27.43 -> 28.89 | Helped |
| Add rendering loss to VAE auto-encoding (ablation) | Splat VAE · Splat auto-encoding | PSNR: 26.07 -> 30.33 | Helped |
| Fine-tune the VAE instead of freezing it (ablation) | Splat VAE · Splat auto-encoding | PSNR: 8.64 -> 30.33 | Helped |
| Unfreeze the VAE encoder (ablation) | Splat VAE · Splat auto-encoding | PSNR: 24.73 -> 30.33 | Helped |
| Use view-concat instead of spatial-concat (ablation) | DiffSplat · T3Bench-300 | CLIP R-Prec. (%): 55.92±1.19 -> 58.92±2.30 | Helped |
| Add diffusion loss (ablation) | DiffSplat · T3Bench-300 | CLIP R-Prec. (%): 50.83±1.91 -> 58.92±2.30 | Helped |
| Add rendering loss (ablation) | DiffSplat · T3Bench-300 | CLIP R-Prec. (%): 56.42±2.27 -> 58.92±2.30 | Helped |
| Add rendering loss (ablation) | DiffSplat · GSO-300 | PSNR: 22.30±6.33 -> 22.58±6.05 | Helped |
| Use SD3-medium instead of SD1.5 (ablation) | DiffSplat · T3Bench-300 | CLIP R-Prec. (%): 58.92±2.30 -> 68.08±0.72 | Helped |
| Use SD3 instead of SD1.5 for VAE auto-encoding (ablation) | Splat VAE · Splat auto-encoding | PSNR: 30.33 -> 31.08 | Helped |
| Use DiffSplat instead of the strongest reported baseline | Text-conditioned 3D generation · T3Bench-300 | CLIP R-Prec. (%): 78.00 -> 81.00 | Helped |
Surprising findings
- Training with only the regular diffusion loss (no rendering loss) already performs well given high-quality splat latents, while rendering loss alone is unstable and slow to converge.
Limitations
- Conversion of 3DGS representation to high-quality mesh remains unsolved.
- Training uses only rendered multi-view datasets (G-Objaverse) rather than real multi-view data.
- Experiments are at 256x256 resolution with 4 input views.
Experimental setup
Datasets: G-Objaverse Objaverse Cap3D captions T3Bench prompts GSO
Benchmarks: T3Bench GSO
Metrics: CLIP similarity CLIP R-Precision ImageReward PSNR SSIM LPIPS
Baselines: GVGEN LN3Diff DIRECT-3D 3DTopia LGM GRM 3DTopia-XL LaRa CRM InstantMesh
Builds on: 3D Gaussian Splatting Stable Diffusion DPM-Solver++ ControlNet Large Reconstruction Model
Closest papers by meaning
- Generative Sparse-View Gaussian Splatting (CVPR 2025 · similarity 0.92)
- DiffGS: Functional Gaussian Splatting Diffusion (NeurIPS 2024 · similarity 0.91)
- SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis (CVPR 2025 · similarity 0.90)
- Turbo3D: Ultra-fast Text-to-3D Generation (CVPR 2025 · similarity 0.90)
- DirectTriGS: Triplane-based Gaussian Splatting Field Representation for 3D Generation (CVPR 2025 · similarity 0.90)
- GaussianSpa: An "Optimizing-Sparsifying" Simplification Framework for Compact and High-Quality 3D Gaussian Splatting (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.