Research map / 3D vision / Gaussian splatting
ReferSplat: Referring Segmentation in 3D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the R3DGS task and the Ref-LERF dataset, and proposes ReferSplat, which attaches referring features to 3D Gaussians and supervises them with confidence-weighted IoU pseudo masks, position-aware cross-modal attention, and Gaussian-text contrastive learning.
Problem
Segmenting objects in a 3D Gaussian Splatting scene from free-form referring expressions with spatial relations, including objects occluded or not visible in novel views, which existing open-vocabulary 3DGS methods handle poorly.
Approach
Assigns each 3D Gaussian a referring feature that directly interacts with word features to render a language-response mask, adds position-aware cross-modal attention to inject spatial cues, and uses contrastive learning on top-scoring Gaussians to separate similar expressions.
Key results
- ReferSplat achieves 29.2 average mIoU on Ref-LERF, versus 20.5 for GOI and 15.8 for Grounded SAM.
- ReferSplat reaches 55.4 mIoU on LERF-OVS and 94.1 on 3D-OVS, the best reported in both tables.
- Ablation on ramen and kitchen: PCMI adds about 5 mIoU and GTCL about 4 mIoU over the baseline; two-stage training reaches 36.9 on ramen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Position-aware Cross-Modal Interaction (PCMI) (ablation) | ReferSplat referring feature fields · Ref-LERF ramen and kitchen | mIoU: 28.4 -> 33.5; 18.5 -> 22.8 | Helped |
| add Gaussian-Text Contrastive Learning (GTCL) (ablation) | ReferSplat referring feature fields · Ref-LERF ramen and kitchen | mIoU: 28.4 -> 32.8; 18.5 -> 21.9 | Helped |
| add two-stage optimization (ablation) | ReferSplat · Ref-LERF ramen and kitchen | mIoU: 35.2 -> 36.9; 24.4 -> 25.2 | Helped |
| use multiplication rather than cosine similarity for Gaussian-text response (ablation) | ReferSplat baseline configuration · Ref-LERF ramen and kitchen | mIoU: 25.4 -> 28.4; 16.8 -> 18.5 | Helped |
| use confidence-weighted IoU rather than IoU without confidence weighting for pseudo-mask selection (ablation) | ReferSplat pseudo-mask generation · Ref-LERF ramen and kitchen | mIoU: 25.3 -> 28.4; 17.1 -> 18.5 | Helped |
| use confidence-weighted IoU rather than top-1 mask selection (ablation) | ReferSplat pseudo-mask generation · Ref-LERF ramen and kitchen | mask quality mIoU: 47.1 -> 52.9; 42.8 -> 49.7 | Helped |
| use confidence-weighted IoU rather than SAM2 mask propagation (ablation) | ReferSplat pseudo-mask generation · Ref-LERF ramen and kitchen | mIoU: 22.6 -> 28.4; 15.5 -> 18.5 | Helped |
| include Gaussian position features and aligned text position features in cross-modal attention (ablation) | ReferSplat PCMI · Ref-LERF ramen and kitchen | mIoU: 27.9 -> 33.5; 18.1 -> 22.8 | Helped |
| include aligned text position features f_p,w,i in addition to Gaussian position features (ablation) | ReferSplat PCMI · Ref-LERF ramen and kitchen | mIoU: 30.3 -> 33.5; 20.5 -> 22.8 | Helped |
| replace attention-based position modeling with direct addition of Gaussian position features to referring features (ablation) | ReferSplat PCMI · Ref-LERF ramen and kitchen | mIoU: 33.5 -> 30.3; 22.8 -> 20.5 | Hurt |
| replace BERT sentence embeddings with CLIP embeddings (ablation) | ReferSplat · Ref-LERF | mIoU: 29.2 -> 23.5 | Hurt |
| increase referring feature dimension from 16 to 32 (ablation) | ReferSplat · Ref-LERF ramen and kitchen | mIoU: 28.4 -> 27.2; 18.5 -> 16.9 | Hurt |
Surprising findings
- Vanilla cross-attention without position features performs below the simple multiplication baseline, indicating the position terms are essential.
- CLIP text embeddings perform far worse than BERT (23.5 vs 29.2 average mIoU) for referring expressions.
Limitations
- Does not handle dynamic scenes (no 4D extension).
- Does not incorporate 3D visual grounding or object size estimation.
- Ref-LERF has a limited number of scenes (four), restricting generalization.
Experimental setup
Datasets: LERF LERF-OVS 3D-OVS Ref-LERF
Benchmarks: Ref-LERF LERF-OVS 3D-OVS
Metrics: mIoU
Baselines: Grounded SAM LangSplat SPIn-NeRF Gaussian Grouping GOI Feature-3DGS LEGaussians
Builds on: 3D Gaussian Splatting LangSplat Grounded SAM SAM2
Closest papers by meaning
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.91)
- Chain of Semantics Programming in 3D Gaussian Splatting Representation for 3D Vision Grounding (CVPR 2025 · similarity 0.90)
- RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation (NeurIPS 2024 · similarity 0.90)
- Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration (CVPR 2025 · similarity 0.90)
- Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting (NeurIPS 2025 · similarity 0.89)
- 3D Vision-Language Gaussian Splatting (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.