Research map / Point clouds / Point cloud classification
UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniGS, which replaces point clouds with 3D Gaussian Splatting as the 3D modality in CLIP-aligned language-image-3D pretraining, using a parallel dual-branch ViT with cross-attention guidance from a frozen Uni3D encoder.
Problem
Point clouds are discrete and sparse, leaving a gap to dense 2D images and losing shape and texture detail in multi-modal 3D pre-training for text, image and 3D alignment.
Approach
Represent objects as 3D Gaussians with color and opacity (3DGS) and align them to a frozen CLIP text-image space using a dual-branch ViT encoder: a frozen Uni3D-pretrained fundamental branch (location and color) guides a trainable advanced branch (location, opacity, scale, rotation) via cross-attention (Gaussian-Aware Guidance).
Key results
- Zero-shot classification on ABO reaches 46.97% top-1 vs 37.79% for Uni3D trained from scratch on 3DGS.
- Text-to-3D retrieval on Objaverse achieves 21.00% top-1 average, above CLIP2 (7.40%) and Uni3D (16.70%).
- On SUN RGBD, UniGS reaches 69.64% average accuracy, versus 61.72% for Uni3D trained from scratch on point clouds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the point-cloud representation with 3DGS and use UniGS | 3D text retrieval · Objaverse | Top1 accuracy: +5.2% | Helped |
| Use UniGS rather than CLIP² | 3D text retrieval · Objaverse | Top1 accuracy: +13.6% | Helped |
| Use UniGS rather than CLIP² | zero-shot classification · Objaverse-LVIS and ABO | Top1 accuracy: over 24% | Helped |
| Use UniGS rather than Uni3D | SUN RGBD scene recognition · SUN RGBD | average Top1 accuracy: +7.92% | Helped |
| Add the ViT-based encoder to 3DGS instead of using the convolution-based model (ablation) | UniGS · ABO | Top1 accuracy: 19.35 -> 38.38 | Helped |
| Add pretrained weights without parallel structure or cross-attention (ablation) | UniGS · ABO | Top1 accuracy: 38.38 -> 27.43 | Hurt |
| Add parallel structure to process color and other 3DGS features separately (ablation) | UniGS · ABO | Top1 accuracy: 27.43 -> 37.58 | Helped |
| Add cross-attention between the parallel encoder branches (ablation) | UniGS · ABO | Top1 accuracy: 37.58 -> 46.97 | Helped |
| Remove pretrained weights from the parallel structure with cross-attention (ablation) | UniGS · ABO | Top1 accuracy: 40.57 -> 46.97 | Hurt |
| Increase training data from 10k to 50k to 100k (ablation) | UniGS · ABO | Top1 accuracy: 27.67; 43.06; 46.97 | Helped |
| Increase model scale from UniGS-T to UniGS-S to UniGS-L (ablation) | UniGS · ABO | Top1 accuracy: 42.67; 46.97; 52.30 | Helped |
| Unfreeze the fundamental encoder (ablation) | UniGS · ABO zero-shot classification | Top1 accuracy: 46.97 -> 47.53; 79.38 -> 78.60 | Mixed |
Surprising findings
- Simply swapping point clouds for 3DGS locations in Uni3D lowers zero-shot accuracy on Objaverse (2.30% top-1 vs 16.70% with point clouds), showing naive retraining is not effective.
Limitations
- Lacks validation on outdoor scenes and integration with large language models.
- Requires at least one image with camera pose to optimize 3DGS.
Experimental setup
Datasets: Objaverse ABO MVImgNet SUN RGBD Objaverse-LVIS
Benchmarks: Objaverse-LVIS ABO MVImgNet SUN RGBD
Metrics: top-1 accuracy top-5 accuracy top-10 accuracy
Baselines: Uni3D CLIP2 TAMM ReCon nerf2clip nf2vec
Builds on: Uni3D CLIP 3D Gaussian Splatting CLIP2
Closest papers by meaning
- Uni3D: Exploring Unified 3D Representation at Scale (ICLR 2024 · similarity 0.90)
- Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.89)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.89)
- FHGS: Feature-Homogenized Gaussian Splatting (NeurIPS 2025 · similarity 0.88)
- Chain of Semantics Programming in 3D Gaussian Splatting Representation for 3D Vision Grounding (CVPR 2025 · similarity 0.88)
- UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.