Research map / Detection and segmentation / 3D detection and driving
V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes V-DETR, a DETR-based indoor 3D object detector whose 3D Vertex Relative Position Encoding biases cross-attention using offsets from each query's predicted box vertices, reaching new SOTA on ScanNetV2 and SUN RGB-D.
Problem
DETR-style 3D object detectors on point clouds underperform because cross-attention queries attend to points far from target objects, violating locality, likely due to limited 3D training data.
Approach
Compute a relative position bias for each point from its offsets to the eight vertices of the query's currently predicted (rotated) 3D box, refined at every decoder layer, and add it to cross-attention logits; offsets are taken in a canonical object coordinate frame and compressed with a signed-log nonlinearity via a small lookup table with grid_sample.
Key results
- On ScanNetV2 val, V-DETR (TTA) reaches AP25 77.8% and AP50 66.0%, versus 3DETR 65.0%/47.0% and CAGroup3D 75.1%/61.3%.
- On SUN RGB-D, V-DETR (TTA) reaches AP25 68.0% and AP50 51.1%, a new record in the paper's comparison.
- Ablation on ScanNetV2 shows 3DV-RPE (76.7/65.0) beats a 3D box mask (74.0/59.1) and no positional modulation (68.8/44.5) at 360 epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 3DV-RPE attention modulation (ablation) | V-DETR baseline · ScanNetV2, 540 epochs | AP50: 47.6 -> 66.0 | Helped |
| Replace 3DV-RPE with a 3D box mask (ablation) | V-DETR · ScanNetV2, 540 epochs | AP50: 66.0 -> 60.8 | Hurt |
| Add a 3D box mask to 3DV-RPE (ablation) | V-DETR · ScanNetV2, 540 epochs | AP50: 66.0 -> 63.5 | Hurt |
| Use the signed log transform instead of the identity transform in 3DV-RPE (ablation) | V-DETR · ScanNetV2 | AP50: 48.2 -> 65.0 | Helped |
| Use 8 vertices instead of 1 in 3DV-RPE (ablation) | V-DETR · ScanNetV2 | AP50: 54.8 -> 65.0 | Helped |
| Transform relative offsets from world coordinates to object coordinates (ablation) | V-DETR 3DV-RPE · SUN RGB-D | AP50: 46.9 -> 51.1 | Helped |
| Replace PointNet + Transformer encoder with ResNet34 + FPN (ablation) | V-DETR · ScanNetV2 | AP50: 60.1 -> 65.0 | Helped |
| Use object-normalized instead of scene-normalized box parameterization (ablation) | V-DETR · ScanNetV2 | AP50: 61.1 -> 65.0 | Helped |
| Add voxel expansion (ablation) | V-DETR · ScanNetV2 | AP50: 65.0 -> 62.0 | Hurt |
| Add the light-weight FFN for initial 3D boxes (ablation) | V-DETR · ScanNetV2 | AP50: 62.8 -> 65.0 | Helped |
| Increase points from 40K to 100K (ablation) | V-DETR · ScanNetV2 | AP50: 62.4 -> 65.0 | Helped |
| Increase queries and ground-truth repeats in one-to-many matching (ablation) | V-DETR · ScanNetV2, 40K points | AP25: 74.3; 75.6; 75.2 | Mixed |
Surprising findings
- Combining the 3D box mask with 3DV-RPE performs worse than 3DV-RPE alone, and voxel expansion degrades V-DETR while it helps FCAF3D.
Limitations
- Gains evaluated only on indoor benchmarks; outdoor extension is future work.
- Inference is slower than FCAF3D (240 ms vs 128 ms per scene).
Experimental setup
Datasets: ScanNetV2 SUN RGB-D
Benchmarks: ScanNetV2 SUN RGB-D
Metrics: AP25 AP50
Baselines: 3DETR CAGroup3D FCAF3D TR3D Group-Free RBGNet HyperDet3D BRNet H3DNet VoteNet
Builds on: DETR 3DETR Deformable DETR DAB-DETR H-DETR
Closest papers by meaning
- Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.89)
- AS-Det: Active Sampling for Adaptive 3D Object Detection in Point Clouds (AAAI 2025 · similarity 0.88)
- Point2Seq: Detecting 3D Objects As Sequences (CVPR 2022 · similarity 0.88)
- ConQueR: Query Contrast Voxel-DETR for 3D Object Detection (CVPR 2023 · similarity 0.88)
- Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data (NeurIPS 2024 · similarity 0.87)
- State Space Model Meets Transformer: A New Paradigm for 3D Object Detection (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.