Research map / Detection and segmentation / Open-set and few-shot detection

OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging

CVPR 2025 · Detection and segmentation › Open-set and few-shot detection · approach: Open-vocabulary 3D and world perception · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OnlineAnySeg, an online zero-shot 3D instance segmentation method that merges CropFormer 2D masks using a voxel-hashing scene representation with a mapping table, achieving real-time (15 FPS) results on ScanNet200 and SceneNN.

Problem

Online open-vocabulary 3D instance segmentation of progressively reconstructed scenes requires finding spatial overlaps between 2D VFM masks across views in real time, which prior methods do too slowly or which require training data.

Approach

Maintain a voxel-hash table storing mask IDs per voxel so overlap queries cost O(n) instead of O(n^2), use an append-only hash table with a mapping table for mask merges, and merge masks by combining overlap ratio, CLIP semantic similarity, FCGF geometric similarity, and third-view supporter consensus, without any training.

Key results

• On ScanNet200 the method reaches 18.6 AP, 36.1 AP50, 53.5 AP25 while running at 15 FPS, the highest efficiency among compared methods.
• On SceneNN full sequences it reaches 18.1 AP and 35.3 AP50, versus 9.1 AP for online SAM3D (raw outputs) and 20.1 AP for EmbodiedSAM.
• On intermediate SceneNN sequences (25/50/75%) it gains about +4 AP, +8 AP50 and +10 AP25 over EmbodiedSAM.

Measured effects reported in the paper

Surprising findings

• EmbodiedSAM, trained on ScanNet200, scores highest on ScanNet200 (28.8 AP) but drops sharply on SceneNN, indicating learned merging does not generalize.
• Using only semantic and geometric feature similarity drops AP to 9.7 on ScanNet200, showing spatial overlap cues are the most critical criterion.

Limitations

• Less robust to under-segmentation, reducing accuracy in scenes with significant occlusions.
• Space-time tradeoff makes it hard to scale to very large environments such as floor-level scenes.
• Only evaluated on ScanNet200 and SceneNN indoor RGB-D sequences.

Experimental setup

Datasets: ScanNet200 SceneNN

Benchmarks: ScanNet200 SceneNN

Metrics: AP AP50 AP25 FPS

Baselines: EmbodiedSAM OVIR-3D MaskClustering SAM3D

Builds on: VoxelHashing MaskClustering CropFormer OVIR-3D EmbodiedSAM

Closest papers by meaning

• EmbodiedSAM: Online Segment Any 3D Thing in Real Time (ICLR 2025 · similarity 0.89)
• Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.89)
• SAM2Object: Consolidating View Consistency via SAM2 for Zero-Shot 3D Instance Segmentation (CVPR 2025 · similarity 0.88)
• MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation (CVPR 2024 · similarity 0.88)
• Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.88)
• Segment Any 3D Object with Language (ICLR 2025 · similarity 0.87)