Research map / 3D vision / 3D generation
Memory-based Adapters for Online 3D Scene Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes memory-based adapters (queued 3D voxel memory, channel-shift image adapter, and 3D-to-2D adapter) that let offline 3D scene perception models process streaming RGB-D video online for semantic segmentation, detection and instance segmentation.
Problem
Conventional 3D scene perception models are offline, requiring a fully reconstructed scene, which is unsuitable for robotics that receive streaming RGB-D frames and need perception synchronously with data collection.
Approach
Insert zero-initialized, plug-and-play memory-based adapters after image and point cloud backbones that cache features in a queued voxel memory (point cloud) and a one-frame channel-shift memory (image), plus a 3D-to-2D adapter that injects global 3D context into image features, so existing offline models can be converted to online models by fine-tuning.
Key results
- MkNet-SV+Ours reaches 72.7 mIoU on ScanNet semantic segmentation, above the offline MkNet (71.6) and the online Fs-A (63.5).
- TD3D-SV+Ours achieves 71.3 mAP@25 and 60.5 mAP@50 on ScanNet instance segmentation, versus 53.7/36.8 for TD3D-SV without the adapters.
- FCAF3D-SV+Ours improves object detection mAP@25 from 41.9 to 70.5 on ScanNet, and the adapters still reach state-of-the-art on all three tasks when both backbones are frozen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add memory-based adapters to MkNet-SV | MkNet-SV · ScanNet | mIoU: 71.6 -> 72.7 | Helped |
| Add memory-based adapters to MkNet-SV | MkNet-SV · SceneNN | mIoU: 51.1 -> 56.7 | Helped |
| Add memory-based adapters to FCAF3D-SV | FCAF3D-SV · ScanNet | mAP@50: 20.6 -> 49.9 | Helped |
| Add memory-based adapters to TD3D-SV | TD3D-SV · ScanNet | mAP@50: 36.8 -> 60.5 | Helped |
| Remove residual connection from point cloud module (ablation) | MkNet-SV point cloud module · ScanNet | mIoU: 66.9 -> 64.6 | Hurt |
| Use random initialization for point cloud module (ablation) | MkNet-SV point cloud module · ScanNet | mIoU: 66.9 -> 66.2 | Hurt |
| Remove voxel maxpooling from point cloud module (ablation) | MkNet-SV point cloud module · ScanNet | mIoU: 66.9 -> 64.8 | Hurt |
| Set scaling factor s=1 in point cloud module (ablation) | MkNet-SV point cloud module · ScanNet | mIoU: 66.9 -> 65.3 | Hurt |
| Set scaling factor s=5 in point cloud module (ablation) | MkNet-SV point cloud module · ScanNet | mIoU: 66.9 -> 66.8 | Hurt |
| Insert point cloud module after neck (ablation) | MkNet-SV point cloud module · ScanNet | mIoU: 66.9 -> 66.0 | Hurt |
| Remove residual connection from image module (ablation) | MkNet-SV image module · ScanNet | mIoU: 69.1 -> 67.1 | Hurt |
| Use random initialization for image module (ablation) | MkNet-SV image module · ScanNet | mIoU: 69.1 -> 68.7 | Hurt |
Surprising findings
- The adapted online model outperforms the offline MinkowskiNet on ScanNet semantic segmentation, which the authors attribute to finer local predictions from partial inputs.
- Freezing both image and point cloud backbones and tuning only the adapters still yields state-of-the-art results on all three online tasks.
Limitations
- Image memory stores only one previous frame, so global image context is limited without the 3D-to-2D adapter.
- Online predictions rely on a simple per-task fusion strategy (voxel max-pooling, 3D NMS with score bonus) rather than a learned fusion.
- Depends on posed RGB-D input from visual odometry; not evaluated with reconstruction-free pose estimation errors.
Experimental setup
Datasets: ScanNet ScanNet-25k SceneNN
Benchmarks: ScanNet SceneNN ScanNet online benchmark
Metrics: mIoU mAcc mAP@25 mAP@50
Baselines: MkNet Fs-A AnyView INS-Conv FCAF3D TD3D SoftGroup CAGroup3D
Builds on: Pri3D AnyView TSM MinkowskiNet FCAF3D TD3D
Closest papers by meaning
- 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning (CVPR 2025 · similarity 0.85)
- FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry (NeurIPS 2026 · similarity 0.85)
- Continuous 3D Perception Model with Persistent State (CVPR 2025 · similarity 0.85)
- 3D-SPATIAL MULTIMODAL MEMORY (ICLR 2025 · similarity 0.85)
- EmbodiedSAM: Online Segment Any 3D Thing in Real Time (ICLR 2025 · similarity 0.85)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.