Research map / Vision-language and multimodal / Human and embodied modeling
4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 4D language Gaussian splatting field whose time-varying semantics are supervised by LLM sentence embeddings of MLLM-generated object-wise video captions, plus a status deformable network that mixes learned state prototypes.
Problem
Open-vocabulary language queries in dynamic 4D scenes need time-sensitive semantics (object states and actions), which CLIP-based, static 3D language fields cannot capture or supervise at object level.
Approach
Replace vision-feature supervision with object-wise captions from an MLLM (Qwen2-VL-7B) using visual prompts (contour, gray, blur) and motion descriptions, encode captions with e5-mistral-7b into pixel-aligned features, and model per-Gaussian semantics as a convex combination of K learned state prototypes weighted by an MLP over HexPlane features.
Key results
- Time-sensitive querying on HyperNeRF: average Acc 90.83% and vIoU 72.26%, versus 87.58%/68.57% for the Non-Status Field ablation and 54.01%/22.65% for LangSplat.
- Time-agnostic querying: mIoU 82.48% on HyperNeRF and 85.11% on Neu3D, versus 74.92% and 61.49% for LangSplat.
- Ablation on state count K shows K=3 gives the best Acc (97.82%) on chickchicken.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use object-wise MLLM captions and a status deformable network instead of the Non-Status Field | 4D LangSplat · HyperNeRF time-sensitive querying | Acc (%): 87.58 -> 90.83 | Helped |
| Use object-wise MLLM captions and a status deformable network instead of the Non-Status Field | 4D LangSplat · HyperNeRF time-sensitive querying | vIoU (%): 68.57 -> 72.26 | Helped |
| Use 4D LangSplat instead of LangSplat | 4D LangSplat · HyperNeRF time-agnostic querying | mIoU: 74.92 -> 82.48 | Helped |
| Use 4D LangSplat instead of LangSplat | 4D LangSplat · HyperNeRF time-agnostic querying | mAcc: 97.72 -> 98.01 | Helped |
| Use 4D LangSplat instead of LangSplat | 4D LangSplat · Neu3D time-agnostic querying | mIoU: 61.49 -> 85.11 | Helped |
| Use 4D LangSplat instead of LangSplat | 4D LangSplat · Neu3D time-agnostic querying | mAcc: 91.89 -> 98.32 | Helped |
| Add gray and contour visual prompts to blur prompting (ablation) | MLLM object-wise video prompting · Caption prompting | ∆sim: 0.33 -> 2.15 -> 3.32 | Helped |
| Add video-level and image prompts (ablation) | MLLM object-wise video prompting · Caption prompting | ∆sim: 0.14 -> 1.01 -> 3.32 | Helped |
| Vary the number of state prototypes from K=3 (ablation) | Status deformable network · chick chicken | Acc (%): K=2: 94.56; K=3: 97.82; K=4: 95.65; K=5: 94.56; K=6: 94.56 | Mixed |
| Vary the number of state prototypes from K=3 (ablation) | Status deformable network · chick chicken | vIoU (%): K=2: 88.05; K=3: 91.93; K=4: 89.11; K=5: 88.98; K=6: 86.28 | Mixed |
Limitations
- Time-sensitive evaluation relies on manually annotated semantic labels for HyperNeRF and Neu3D, which are small custom test sets.
- Time-sensitive frame selection uses the mean similarity across the video as a threshold, a heuristic choice.
Experimental setup
Datasets: HyperNeRF Neu3D
Benchmarks: HyperNeRF Neu3D
Metrics: mIoU mAcc Acc vIoU
Baselines: LangSplat Feature-3DGS Gaussian Grouping Deformable CLIP Non-Status Field
Builds on: LangSplat 4D Gaussian Splatting SAM DEVA tracking HexPlane
Closest papers by meaning
- LangSplat: 3D Language Gaussian Splatting (CVPR 2024 · similarity 0.91)
- LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS (NeurIPS 2025 · similarity 0.90)
- SOVGaussian: Sparse-View 3D Gaussian Splatting for Open-Vocabulary Scene Understanding (AAAI 2025 · similarity 0.88)
- Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration (CVPR 2025 · similarity 0.88)
- Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding (CVPR 2024 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.