Research map / Point clouds / Shape and surface completion
GaussianAnything: Interactive Point Cloud Flow Matching for 3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaussianAnything, a native 3D generative framework that encodes multi-view RGB-D-N renderings into a point-cloud-structured latent via a 3D VAE, decodes to surfel Gaussians, and runs cascaded flow matching for text- and image-conditioned 3D generation and latent point-cloud editing.
Problem
Native 3D diffusion models face challenges in VAE input format, 3D latent space structure (unstructured set latents or tri-planes are hard to edit), and output representation (textureless SDFs or volumetric tri-planes that are memory-hungry at high resolution).
Approach
Encode multi-view posed RGB-D-N renderings with an SRT-style transformer into a set latent, then cross-attend it onto an FPS-sampled sparse point cloud to form a point-cloud-structured latent; decode with a DiT-based upsampler into surfel Gaussians, and generate the point cloud and its features in two cascaded flow-matching stages.
Key results
- Image-to-3D on GSO: CLIP-I 89.06 and P-FID 8.72 versus the best baseline LGM (CLIP-I 87.99, P-FID 40.17); LGM has better FID/KID (19.93 vs 24.21).
- Text-to-3D: ViT-L/14 CLIP score 29.38 versus 28.11 for 3DTopia, and MUSIQ-AVA 4.99 versus 4.16 for LN3Diff.
- Gaussian utilization ratio of 96.84% high-opacity Gaussians, versus 52.63% for LGM and 17.14% for Splatter Image.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace dense point cloud input with multi-view RGB-D renderings (ablation) | 3D VAE · Objaverse 50K subset | LPIPS: 0.174 -> 0.163 | Helped |
| add normal map input (ablation) | 3D VAE · Objaverse 50K subset | LPIPS: 0.163 -> 0.157 | Helped |
| add Gaussian SR module (ablation) | 3D VAE · Objaverse 50K subset | LPIPS: 0.157 -> 0.095 | Helped |
| use 3 Gaussian SR modules (ablation) | 3D VAE · Objaverse 50K subset | LPIPS: 0.095 -> 0.067 | Helped |
| use cascaded two-stage diffusion instead of a single jointly trained DiT (ablation) | GaussianAnything · Text-to-3D | texture and 3D structure quality | Helped |
| edit the point-cloud latent instead of final 3D Gaussians (ablation) | GaussianAnything · 3D-aware editing | editing quality | Helped |
| generate text-conditioned 3D with GaussianAnything instead of LGM | 3D generation · Image-conditioned 3D generation | FID: 19.93 -> 24.21 | Hurt |
| generate text-conditioned 3D with GaussianAnything instead of LGM | 3D generation · Image-conditioned 3D generation | KID (%): 0.55 -> 0.76 | Hurt |
| use GaussianAnything instead of the strongest listed baseline | 3D generation · Image-conditioned 3D generation | CLIP-I: 87.99 -> 89.06 | Helped |
| use GaussianAnything instead of the strongest listed baseline | 3D generation · Image-conditioned 3D generation | P-FID: 19.72 -> 8.72 | Helped |
| use GaussianAnything instead of the strongest listed baseline | 3D generation · Text-to-3D | ViT-B/32 CLIP score: 30.10 -> 31.80 | Helped |
Surprising findings
- LGM, a multi-view-based method, achieves better FID/KID than GaussianAnything despite lower CLIP-I and 3D metrics.
Limitations
- Texture quality is inferior to multi-view-based generators such as LGM.
- Text-conditioned native 3D generation is visually worse than SDS-based alternatives.
- Diffusion is trained only on latent flow matching without rendering loss or 2D priors.
Experimental setup
Datasets: G-Objaverse Objaverse Cap3D 3DTopia captions GSO DF-415
Benchmarks: GSO Objaverse
Metrics: CLIP-I FID KID MUSIQ P-FID P-KID COV MMD CLIP score MUSIQ-AVA
Baselines: OpenLRM Splatter Image One-2-3-45 CRM Lara LGM Shape-E LN3Diff Point-E 3DTopia
Builds on: SiT PixArt 2D Gaussian Splatting Scene Representation Transformer RUST
Closest papers by meaning
- Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation (CVPR 2024 · similarity 0.87)
- Structured 3D Latents for Scalable and Versatile 3D Generation (CVPR 2025 · similarity 0.87)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.86)
- Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation (CVPR 2025 · similarity 0.86)
- Text-to-3D using Gaussian Splatting (CVPR 2024 · similarity 0.86)
- Interactive3D: Create What You Want by Interactive 3D Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.