Research map / 3D vision / Gaussian splatting

3D-SPATIAL MULTIMODAL MEMORY

ICLR 2025 · 3D vision › Gaussian splatting · approach: Sparse-view Gaussian splatting for novel views

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes M3, a 3D Gaussian Splatting-based multimodal memory that stores compressed principal scene components from six foundation models and renders their features via Gaussian memory attention.

Problem

Feature splatting methods distilling foundation model features into 3D Gaussians suffer from low-dimensional feature bottlenecks and misalignment between distilled and original foundation model features, and high memory cost for storing high-dimensional features per Gaussian.

Approach

Store original high-dimensional foundation model features in a compact memory bank (principal scene components, selected by similarity reduction), and let each Gaussian carry a low-dimensional learnable principal query that attends over this bank to render aligned features, instead of distilling features directly into Gaussians.

Key results

• M3 (35M params) achieves lower CLIP feature cosine distance than F-3DGS (61M) on Train (0.3140 vs 0.3330) and Garden, using about half the parameters.
• On grounding/retrieval, M3 reaches SigLIP I2T@1 of 55.2 on Train versus 2.6 for F-3DGS.
• Ablation shows 1/4 of the 30k training budget (7k iterations) gives reasonable performance, and 16 degrees per foundation model suffices.

Measured effects reported in the paper

Surprising findings

• F-Splat failed to extract SEEM and LLaMA3 features, which the authors attribute to duplication in ground-truth feature extraction.
• M3 does not beat F-3DGS on every feature-distance metric; e.g., on Garden M3 has worse CLIP cosine than F-3DGS for some models and much worse LLaMAv L2.

Limitations

• Evaluation limited to medium-sized static scenes and few scenes.
• Ground-truth downstream labels are generated by GPT-4o and SoM/Semantic-SAM, not human-verified.
• Stated future work includes a reasoning module operating on the memory bank; the system does not yet reason over memory.

Experimental setup

Datasets: Mip-NeRF360 Garden Tanks & Temples Train Deep Blending PlayRoom Deep Blending DrJohnson M3-Robot (Table-Top, Geisel)

Benchmarks: Mip-NeRF360 Garden Tanks & Temples Train Deep Blending PlayRoom Deep Blending DrJohnson M3-Robot Geisel

Metrics: PSNR SSIM LPIPS cosine distance L2 distance mIoU cIoU AP50 AP60 I2T@1

Baselines: F-3DGS F-Splat

Builds on: 3D Gaussian Splatting Feature 3DGS F-Splat Grendel-GS SEEM

Closest papers by meaning

• Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.88)
• 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning (CVPR 2025 · similarity 0.88)
• Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models (CVPR 2026 · similarity 0.88)
• UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.87)
• LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (ICLR 2025 · similarity 0.87)
• Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.87)