Research map / 3D vision / 3D generation
Cross-Modal Distillation for 2D/3D Multi-Object Discovery from 2D Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts motion-guided 2D object discovery to LiDAR point clouds (DIOD-3D, with scene completion pretext) and trains 2D and 3D teacher-student branches that exchange pseudo-labels cross-modally (xMOD), with late fusion at inference.
Problem
Unsupervised multi-object discovery in 3D LiDAR data relies only on 3D motion, which is sparse and hard to generalize; the 2D motion-based object discovery advances have not been exploited for 3D or cross-modal discovery.
Approach
Use 2D optical-flow motion masks as the sole supervision for both 2D and 3D branches, with scene completion as the 3D pretext task, and exchange teacher pseudo-labels across modalities (2D teacher to 3D student and vice versa) so each modality compensates for the other's blind cases; at inference, fuse predictions by retaining overlapping masks across modalities.
Key results
- xMOD (2D+3D) late fusion reaches F1@50 of 42.5 on TRI-PD, 27.4 on KITTI and 42.6 on WOD, improving over the 2D baseline DIOD by roughly 8.7 to 15.1 F1 points.
- Scene completion as the 3D pretext task raises KITTI F1@50 from 19.7 to 27.4 with late fusion.
- Cross-modal distillation improves the 2D branch's KITTI F1@50 by 3.6 points and WOD by 7.6 points over DIOD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply cross-modal training to the 2D branch | DIOD / 2D object discovery · TRI-PD, KITTI, WOD | F1@50: 30.6 -> 35.5 (TRI-PD); 18.7 -> 22.3 (KITTI); 27.5 -> 35.1 (WOD) | Helped |
| Apply cross-modal training to the 3D branch | DIOD-3D / 3D object discovery · KITTI, WOD | F1@50: 15.5 -> 18.9 (KITTI); 25.6 -> 31.0 (WOD) | Helped |
| Apply cross-modal training to the 3D branch | DIOD-3D / 3D object discovery · TRI-PD | F1@50: 39.6 -> 37.5 | Hurt |
| Fuse 2D and 3D predictions at inference using late fusion | xMOD · TRI-PD, KITTI, WOD | F1@50: 39.6 -> 42.5 (TRI-PD); 22.3 -> 27.4 (KITTI); 35.1 -> 42.6 (WOD) | Helped |
| Use late fusion instead of early fusion (ablation) | Multimodal object discovery · KITTI | F1@50: 19.4 -> 27.4 | Helped |
| Add scene completion as the 3D pretext task (ablation) | xMOD · KITTI | F1@50: 19.7 -> 27.4 | Helped |
| Add intra-modal distillation losses (ablation) | xMOD · KITTI | F1@50: 27.4 -> 26.8 | Hurt |
| Use cross-modal training with DINOv2-pretrained features | DIOD / 2D object discovery · TRI-PD test | fg-ARI: 69.7 -> 67.1 | Hurt |
| Use cross-modal training with DINOv2-pretrained features | DIOD / 2D object discovery · KITTI test | fg-ARI: 72.3 -> 76.9 | Helped |
Surprising findings
- Adding intra-modal distillation losses slightly decreased performance (27.4 to 26.8 F1@50 on KITTI), suggesting it is redundant with cross-modal losses.
Limitations
- Segmentation quality degrades for objects within 10 m (cropped) and beyond 30 m where LiDAR is sparse (F1 of 7.2 at 30-70 m).
- Performance on KITTI and WOD remains low in absolute terms (F1@50 of 27.4 and 42.6).
Experimental setup
Datasets: TRI-PD KITTI Waymo Open Dataset
Benchmarks: TRI-PD KITTI Waymo Open Dataset
Metrics: all-ARI F1@50 fg-ARI
Baselines: DIOD DIOD-3D BMOD MoTok Bao et al. PPMP SAVI++ DINOSAUR ClusterNet
Builds on: DIOD motion-guided slot attention DINOSAUR BMOD RAFT
Closest papers by meaning
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.88)
- Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations (NeurIPS 2025 · similarity 0.87)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.87)
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.86)
- PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (NeurIPS 2026 · similarity 0.86)
- Towards 3D Objectness Learning in an Open World (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.