Research map / Vision-language and multimodal / Cross-modal learning
CrossOver: 3D Scene Cross-Modal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CrossOver, a scene-level cross-modal embedding that aligns RGB images, point clouds, CAD meshes, floorplans and text without semantic labels or fully aligned data, evaluated on ScanNet and 3RScan retrieval tasks.
Problem
Cross-modal 3D scene understanding usually assumes complete, instance-aligned modality data and semantic instance labels, which real-world scenes rarely provide; existing object-level alignment methods fail to scale to scene retrieval and localization with missing modalities.
Approach
Align all modalities to the image embedding space at instance level (no pairwise alignment needed), then distill into scene-level and dimensionality-specific (1D/2D/3D) encoders that process raw scene data without semantic instance information, yielding emergent cross-modal behavior for unseen modality pairs.
Key results
- On ScanNet P→R instance matching, CrossOver reaches 96.56% R@50% versus 27.93% for PointBind, despite not training on P→R.
- On 3RScan temporal instance matching, CrossOver achieves 92.31% R@25% same-modal, outperforming LivingScenes (87.50%).
- Cross-modal scene retrieval on ScanNet (I→P) reaches top-1 scene category recall of 64.74% vs 13.78% for PointBind.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add scene-level multimodal interactions to the instance-level baseline | CrossOver · 3RScan I → P instance matching | R@25% recall: 93.63 -> 99.36 | Helped |
| add scene-level multimodal interactions to the instance-level baseline | CrossOver · 3RScan P → R instance matching | R@25% recall: 92.62 -> 100 | Helped |
| use unified dimensionality encoders instead of averaged instance embeddings | CrossOver · ScanNet I → P scene retrieval | top-1 scene matching recall: 8.92 -> 14.01 | Helped |
| use unified dimensionality encoders instead of averaged instance embeddings | CrossOver · ScanNet I → R scene retrieval | top-1 scene matching recall: 8.72 -> 6.04 | Hurt |
| use unified dimensionality encoders instead of averaged instance embeddings | CrossOver · ScanNet P → R scene retrieval | top-1 scene matching recall: 0.76 -> 6.71 | Helped |
| train with non-overlapping image repositories for I → P and I → M (ablation) | CrossOver · ScanNet P → M instance matching | top-1 recall: 59.88 -> 55.46; 59.88 -> 57.49; 59.88 -> 54.99 | Mixed |
| reduce I → P training data to 25% while using 75% for I → M (ablation) | CrossOver · ScanNet P → M instance matching | top-3 recall: 80.81 -> 79.73 | Hurt |
| reduce I → M training data to 25% while using 75% for I → P (ablation) | CrossOver · ScanNet P → M instance matching | top-3 recall: 80.81 -> 79.45 | Hurt |
| reduce the available data for each modality pair to 50% (ablation) | CrossOver · ScanNet P → M instance matching | top-3 recall: 80.81 -> 79.94 | Hurt |
Surprising findings
- Aligning all modalities to a single image reference modality outperforms all-pairwise alignment used by prior work, and the emergent pairs (e.g., P→R) are not trained directly.
- The scene-level encoder needs all modalities at inference to perform reasonably, motivating the separate unified dimensionality encoders.
Limitations
- Scene retrieval generalizability could benefit from training on more diverse indoor and outdoor datasets.
- Assumes a base modality (images) per dataset for alignment, not fully relaxed.
- Text referral encoder is less powerful, causing lower I→R intra-category performance than the instance baseline.
Experimental setup
Datasets: ScanNet 3RScan SceneVerse Scan2CAD ShapeNet
Benchmarks: ScanNet 3RScan
Metrics: instance matching recall scene-level matching recall (R@25/50/75) scene matching recall top-k scene category recall temporal recall intra-category recall
Baselines: ULIP-2 PointBind Instance Baseline (ours) MendNet VN-DGCNN VN-ONet LivingScenes
Builds on: CLIP ImageBind ULIP-2 Point-Bind SceneVerse object referrals
Closest papers by meaning
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.89)
- MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding (AAAI 2025 · similarity 0.88)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.88)
- Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection (AAAI 2025 · similarity 0.88)
- Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations (NeurIPS 2025 · similarity 0.87)
- Cross-Modal Match for Language Conditioned 3D Object Grounding (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.