Research map / Point clouds / Segmentation and detection
State Space Model Meets Transformer: A New Paradigm for 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEST, a DETR-style 3D indoor detector whose decoder uses an interactive state space model (ISSM) that treats object queries as SSM states and scene points as inputs, so both are updated jointly with linear complexity.
Problem
In DETR-based 3D indoor object detection, scene point features stay fixed across decoder layers, so later decoder layers contribute little to refining object queries; self-attention to update scene points is quadratic in cost.
Approach
Model queries as SSM system states and scene points as system inputs; make the SSM parameters (Delta, B, C) state-dependent via a spatial correlation module over predicted boxes and a distance-based delay kernel, with Hilbert serialization and bidirectional scanning to handle unordered point clouds, plus inter-state attention and a gated FFN.
Key results
- On ScanNet V2, DEST lifts GroupFree(S) AP50 from 48.9 to 53.2 (highest) and GroupFree(L) AP50 from 52.8 to 58.1.
- On top of VDETR, DEST reaches 78.5 AP25 / 66.6 AP50 on ScanNet V2 and 68.4 AP25 / 51.8 AP50 on SUN RGB-D (highest-run results), setting new SOTA among compared methods.
- Ablation: replacing Mamba2 blocks with the ISSM in the SSM baseline gives a substantial gain, and fixing scene point features in DEST drops performance significantly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the GroupFree(S) transformer decoder with the ISSM-based DEST decoder | GroupFree(S) · ScanNet V2 H | AP50: 48.9 -> 53.2 (+4.3) | Helped |
| Replace the GroupFree(L) transformer decoder with the ISSM-based DEST decoder | GroupFree(L) · ScanNet V2 H | AP50: 52.8 -> 58.1 (+5.3) | Helped |
| Replace the VDETR transformer decoder with the ISSM-based DEST decoder | VDETR · SUN RGB-D H | AP50: 50.4 -> 51.8 (+1.4) | Helped |
| Use multiple Hilbert axis orders across decoder layers instead of one order (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 51.1 -> 52.7 | Helped |
| Use bidirectional rather than unidirectional scanning (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 50.9 -> 52.7 | Helped |
| Add channel flipping to bidirectional scanning (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 52.7 -> 52.4 | Hurt |
| Increase decoder depth from 6 to 12 layers (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 52.7 -> 54.8 | Helped |
| Increase object candidates from 256 to 512 (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 52.7 -> 53.6 | Helped |
| Increase object candidates from 256 to 1024 (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 52.7 -> 53.3 | Hurt |
| Add GLU to both scene points and state points (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 51.9 -> 52.7 | Helped |
| Add depth-wise convolutions to ISSM and GFFN (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 51.8 -> 52.7 | Helped |
| Increase depth-wise convolution kernel size from 8 to 12 (ablation) | GroupFree(S) + DEST · ScanNet V2 | AP50: 52.7 -> 52.6 | No clear effect |
Surprising findings
- A GFFN-only variant of GroupFree (updating scene features without state interaction) gives only a slight gain, indicating limited receptive field matters.
Limitations
- Slight increase in parameters and computation relative to baselines.
- Evaluation limited to two indoor datasets and two DETR baselines.
- Reported SUN RGB-D results on VDETR use the authors' reimplementation without official code for VDETR on that dataset.
Experimental setup
Datasets: ScanNet V2 SUN RGB-D
Benchmarks: ScanNet V2 SUN RGB-D
Metrics: AP25 AP50
Baselines: GroupFree VDETR 3DETR CAGroup3D TR3D FCAF3D BRNet VoteNet
Builds on: Mamba Mamba2 DETR GroupFree VDETR Vision Mamba PTv3
Closest papers by meaning
- 3DET-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection (NeurIPS 2024 · similarity 0.88)
- Cubify Anything: Scaling Indoor 3D Object Detection (CVPR 2025 · similarity 0.87)
- Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping (NeurIPS 2025 · similarity 0.87)
- SPGroup3D: Superpoint Grouping Network for Indoor 3D Object Detection (AAAI 2024 · similarity 0.86)
- V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection (ICLR 2024 · similarity 0.86)
- Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.