Research map / Detection and segmentation / 3D detection and driving
UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UNION, an unsupervised multi-modal (LiDAR+camera+temporal) 3D object detection method that uses DINOv2 appearance embeddings of scene-flow-labeled LiDAR clusters to recover static mobile objects and produce pseudo-class labels, training CenterPoint without self-training.
Problem
Unsupervised 3D object detection from LiDAR typically trains only on dynamic objects, penalizing static instances, and relies on costly multi-round self-training to recover them.
Approach
Use appearance embeddings from a self-supervised image encoder (DINOv2) projected onto LiDAR object proposals to select static proposals that look like dynamic mobile objects, enabling single-round training and appearance-based pseudo-class labels for multi-class detection.
Key results
- Class-agnostic detection on nuScenes val: UNION reaches 39.5 AP and 31.7 NDS, versus 13.8 AP / 15.7 NDS for the best unsupervised baseline HDBSCAN.
- Ablation: adding appearance clustering raises AP from 14.2 (motion estimation only) to 39.5.
- Multi-class: UNION-05pc reaches 25.1 mAP over mobile objects versus 4.9 for HDBSCAN with size prior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UNION's joint LiDAR-camera appearance clustering rather than HDBSCAN | CenterPoint on nuScenes · nuScenes validation, class-agnostic | AP: 13.8 -> 39.5 | Helped |
| Add motion estimation to spatial clustering (ablation) | UNION pseudo-label pipeline · nuScenes validation, class-agnostic | NDS: 15.7 -> 19.8 | Helped |
| Add appearance clustering to select mobile objects and discard background (ablation) | UNION pseudo-label pipeline · nuScenes validation, class-agnostic | AP: 14.2 -> 39.5 | Helped |
| Replace I-JEPA ViT-H/16 with DINOv2 ViT-L/14 with registers (ablation) | UNION · nuScenes validation, class-agnostic | AP: 26.8 -> 39.5; 12.7 | Helped |
| Replace I-JEPA ViT-H/16 with DINOv2 ViT-L/14 with registers (ablation) | UNION · nuScenes validation, class-agnostic | NDS: 25.9 -> 31.7; 5.8 | Helped |
| Train with 5 pseudo-classes instead of 10 (ablation) | UNION · nuScenes validation, multi-class mobile objects | mAP: 20.4 -> 25.1 | Helped |
| Train with 5 pseudo-classes instead of 10 (ablation) | UNION · nuScenes validation, multi-class mobile objects | NDS: 22.1 -> 24.4 | Helped |
| Train with 5 pseudo-classes rather than class-agnostic predictions assigned by size prior (ablation) | UNION · nuScenes validation, vehicle AP | AP: 35.2 -> 31.0 | Hurt |
| Train with 20 pseudo-classes (ablation) | UNION · nuScenes validation, cyclist detection without precision-recall clipping | AP: 1.5 -> 2.2 | Helped |
Surprising findings
- Cyclist AP is essentially zero for most configurations, with the cyclist class failing under clipped PR evaluation.
- UNION with I-JEPA image encoder reaches 26.8 AP, showing sensitivity to the choice of camera encoder.
Limitations
- Rare mobile object classes are likely grouped with background objects in appearance clusters and discarded.
- Relies on implicit assumptions about object frequency when clustering appearance embeddings.
- Unsupervised orientation estimation is hard for classes like pedestrians; still below supervised 10% labels.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes validation
Metrics: AP NDS ATE ASE AOE AVE
Baselines: HDBSCAN OYSTER LISO Supervised 1% Supervised 10% Supervised 100% I-JEPA ViT-H/16
Builds on: ICP-Flow HDBSCAN MODEST CenterPoint DINOv2
Closest papers by meaning
- 4D Unsupervised Object Discovery (NeurIPS 2022 · similarity 0.91)
- Cubify Anything: Scaling Indoor 3D Object Detection (CVPR 2025 · similarity 0.87)
- SeMoLi: What Moves Together Belongs Together (CVPR 2024 · similarity 0.87)
- Commonsense Prototype for Outdoor Unsupervised 3D Object Detection (CVPR 2024 · similarity 0.87)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.87)
- VSRD: Instance-Aware Volumetric Silhouette Rendering for Weakly Supervised 3D Object Detection (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.