Research map / 3D vision / Gaussian splatting
Memorize What Matters: Emergent Scene Decomposition from Multitraverse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3DGM, a self-supervised, camera-only framework that uses multitraverse driving videos to learn a 3D Gaussian environment map while segmenting ephemeral objects in 2D, and introduces the Mapverse benchmark.
Problem
Building camera-only 3D maps from driving videos where transient objects (pedestrians, vehicles) corrupt multi-view consistency, without human annotations or LiDAR.
Approach
Treats multitraverse mapping as robust differentiable rendering: consensus across traversals defines inliers (permanent environment), while feature-space residuals from distilled robust DINOv2 features identify outliers (transient objects); residuals are mined spatially into masks used to robustly optimize 3D Gaussians.
Key results
- EmerSeg averages roughly 0.45 IoU on Mapverse-Ithaca365, comparable to supervised segmenters trained on ADE20K/Cityscapes, with the best location reaching 56%.
- EnvGS achieves average Chamfer Distance ~0.9 m versus ~1.9 m for DepthAnything on Mapverse-Ithaca365.
- In neural rendering, EnvGS reaches SSIM 0.806 and PSNR 22.78, close to 3DGS+SegFormer (0.806 / 22.81) and well above VanillaNeRF and RobustNeRF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of traversals from 1 to 10 (ablation) | EmerSeg · Mapverse-Ithaca365 segmentation (100 images) | IoU (%): 15.15 -> 56.01 | Helped |
| Increase feature dimension from 4 to 64 (ablation) | EmerSeg · Mapverse-Ithaca365 segmentation | IoU (%): 9.45 -> 44.13 | Helped |
| Increase feature resolution from 25×40 to 110×180 (ablation) | EmerSeg · Mapverse-Ithaca365 segmentation | IoU (%): 28.61% -> 44.13% | Helped |
| Increase feature resolution beyond 110×180 to 140×210 or 160×260 (ablation) | EmerSeg · Mapverse-Ithaca365 segmentation | IoU (%): 44.13 -> 42.48; 41.19 | Hurt |
| Use Denoised DINOv2 features instead of DINO features (ablation) | EmerSeg · Mapverse-Ithaca365 segmentation | IoU (%): 16.51 -> 44.13 | Helped |
| Use Denoised DINOv2 features instead of denoised DINO features (ablation) | EmerSeg · Mapverse-Ithaca365 segmentation | IoU (%): 14.95 -> 44.13 | Helped |
| Add the RobustNeRF loss to 3DGS | 3DGS · Mapverse-Ithaca365 neural rendering | LPIPS: 0.227 -> 0.416 | Hurt |
| Add SegFormer masks to 3DGS | 3DGS · Mapverse-Ithaca365 neural rendering | SSIM: 0.798 -> 0.806 | Helped |
| Use EnvGS instead of 3DGS | 3DGS · Mapverse-Ithaca365 neural rendering | SSIM: 0.798 -> 0.806 | Helped |
| Use EnvGS instead of 3DGS | 3DGS · Mapverse-Ithaca365 neural rendering | PSNR: 22.92 -> 22.78 | Hurt |
Surprising findings
- Adding RobustNeRF's loss to 3DGS does not improve rendering quality in driving scenes.
- Lower feature dimension and resolution sharply degrade segmentation, with IoU dropping to around 10-25% below 32 dimensions.
Limitations
- Struggles with large environmental variations such as nighttime, major seasonal shifts, and adverse weather.
- Noisy segmentation from motion blur or appearance shifts; small or distant objects are missed due to low feature resolution.
- Requires multiple traversals; a single traversal yields minimal ephemeral object detection.
Experimental setup
Datasets: Ithaca365 nuPlan Mapverse-Ithaca365 Mapverse-nuPlan ADE20K Cityscapes
Benchmarks: Mapverse-Ithaca365 Mapverse-nuPlan
Metrics: IoU Chamfer Distance LPIPS SSIM PSNR
Baselines: PSPNet SegViT Mask2Former SegFormer InternImage STEGO CAUSE DepthAnything VanillaNeRF RobustNeRF 3DGS+RobustNeRF 3DGS
Builds on: 3D Gaussian Splatting DINOv2 RobustNeRF COLMAP STEGO
Closest papers by meaning
- VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians (ICML 2025 · similarity 0.86)
- Diffusion Models for Robotics (AAAI 2025 · similarity 0.86)
- Gaussian Splatting SLAM (CVPR 2024 · similarity 0.85)
- SplaTAM: Splat Track & Map 3D Gaussians for Dense RGB-D SLAM (CVPR 2024 · similarity 0.85)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.85)
- 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.