Research map / 3D vision / 3D generation
GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaussTR, a feed-forward Gaussian Transformer that predicts sparse Gaussians for 3D scenes and aligns their splatted features with foundation models (FeatUp/MaskCLIP, Talk2DINO) for self-supervised open-vocabulary occupancy prediction on Occ3D-nuScenes.
Problem
Self-supervised 3D semantic occupancy prediction relies on dense voxel representations and pre-computed 2D segmentation pseudo-labels, which are costly and limit generalization and scalability.
Approach
Represent the scene as a sparse set of feed-forward predicted 3D Gaussians, render them via differentiable splatting into source views, and align rendered features and depth with frozen foundation models, enabling zero-shot open-vocabulary occupancy via text-feature similarity.
Key results
- GaussTR (Talk2DINO) achieves 12.27 mIoU zero-shot on Occ3D-nuScenes, a 2.33 mIoU gain over prior self-supervised methods (e.g., GaussianOcc 10.29 mIoU per Table 1).
- Training takes about 12 hours on 8 A800 GPUs and is reported as ~40% less training time than prior self-supervised methods.
- Using 1800 Gaussians per scene (300 per view), GaussTR reaches 45.19 IoU with FeatUp alignment plus Metric3D depth, versus 43.87 IoU with Metric3D alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GaussTR with Talk2DINO feature alignment | GaussTR · Occ3D-nuScenes | mIoU: 12.27 vs 9.94 | Helped |
| Add Metric3D geometric alignment to feature alignment (ablation) | GaussTR with FeatUp · Occ3D-nuScenes | IoU: 43.87 -> 45.19 | Helped |
| Add feature alignment to Metric3D geometric alignment (ablation) | GaussTR · Occ3D-nuScenes | IoU: 43.87 -> 45.19 | Helped |
| Use feature alignment without Metric3D geometric alignment (ablation) | GaussTR with FeatUp · Occ3D-nuScenes | convergence | Hurt |
| Add FeatUp to MaskCLIP alignment (ablation) | GaussTR · Occ3D-nuScenes | mIoU: +0.93 mIoU | Helped |
| Add auxiliary segmentation supervision to FeatUp alignment (ablation) | GaussTR with FeatUp · Occ3D-nuScenes | mIoU: 10.76 -> 11.34 | Helped |
| Add auxiliary segmentation supervision to Talk2DINO alignment (ablation) | GaussTR with Talk2DINO · Occ3D-nuScenes | mIoU: 12.27 -> 11.34 | Hurt |
| Increase Gaussian queries from 100 to 300 (ablation) | GaussTR with FeatUp, without auxiliary segmentation · Occ3D-nuScenes | mIoU: 10.02 -> 10.76 | Helped |
| Increase Gaussian queries from 300 to 400 (ablation) | GaussTR with FeatUp, without auxiliary segmentation · Occ3D-nuScenes | mIoU: 10.76 -> 10.54 | Hurt |
| Increase Gaussian queries from 100 to 300 (ablation) | GaussTR with FeatUp and auxiliary segmentation · Occ3D-nuScenes | mIoU: 10.53 -> 11.70 | Helped |
| Increase Gaussian queries from 300 to 400 (ablation) | GaussTR with FeatUp and auxiliary segmentation · Occ3D-nuScenes | mIoU: 11.70 -> 11.52 | Hurt |
| Increase Gaussian queries from 100 to 400 (ablation) | GaussTR with Talk2DINO · Occ3D-nuScenes | mIoU: 10.81 -> 12.45 | Helped |
Surprising findings
- Feature alignment alone without geometric (depth) alignment failed to converge, attributed to the optimization dilemma of jointly optimizing Gaussian positions and features.
- Auxiliary segmentation supervision improved FeatUp alignment by 0.94 mIoU but significantly degraded Talk2DINO performance.
Limitations
- Struggles with small objects such as traffic cones and with flat surfaces like roads and sidewalks due to sparsity and occlusion during splatting.
- Performance degrades beyond 300 queries with FeatUp, attributed to attention dilution in 3D self-attention.
Experimental setup
Datasets: Occ3D-nuScenes
Benchmarks: Occ3D-nuScenes
Metrics: IoU mIoU
Baselines: SelfOcc OccNeRF DistillNeRF GaussianOcc
Builds on: 3D Gaussian Splatting RenderOcc Talk2DINO FeatUp MaskCLIP Deformable DETR
Closest papers by meaning
- GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction (CVPR 2025 · similarity 0.92)
- ODG: Occupancy Prediction Using Dual Gaussians (NeurIPS 2025 · similarity 0.90)
- GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction (CVPR 2025 · similarity 0.89)
- VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction (CVPR 2025 · similarity 0.88)
- QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Prediction (NeurIPS 2025 · similarity 0.87)
- Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.