Research map / Vision-language and multimodal / Cross-modal learning
Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that uni-modal 3D and text encoders are weakly aligned by default, and that projecting features onto a CCA-derived low-dimensional subspace before affine or local-CKA alignment substantially improves 3D-text matching and retrieval on Objaverse-LVIS.
Problem
Uni-modal 3D encoders do not converge to text-like latent structure, and existing post-hoc alignment methods (affine translation, local CKA) give only limited 3D-text matching and retrieval performance.
Approach
Apply CCA to a set of anchor pairs to find a shared low-dimensional subspace of the 3D and text latent spaces, then perform affine or local-CKA alignment inside that subspace instead of the full feature space.
Key results
- Uni-modal PointBERT with CCA subspace (d=50) plus affine alignment reaches 30.8% matching accuracy and 42.2% top-5 retrieval with CLIP text, versus 15.8% and 23.6% for affine alone.
- Local CKA with subspace projection on uni-modal PointBERT reaches 29.4% matching and 60.19% top-5 retrieval with CLIP text, versus 5.8% and 15.2% without projection.
- Default linear CKA between uni-modal 3D and text encoders is at most 0.12, far below the 30-48% reported for vision-text encoders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CCA subspace projection before affine alignment (ablation) | PointBERT and CLIP text encoder · Objaverse-LVIS | matching accuracy: 15.8 -> 30.8 | Helped |
| Add CCA subspace projection before affine alignment (ablation) | PointBERT and CLIP text encoder · Objaverse-LVIS | top-5 retrieval: 23.6 -> 42.2 | Helped |
| Add CCA subspace projection before local CKA matching (ablation) | PointBERT and CLIP text encoder · Objaverse-LVIS | top-5 retrieval: 15.2 -> 60.19 | Helped |
| Add CCA subspace projection before affine alignment (ablation) | SparseConv and CLIP text encoder · Objaverse-LVIS | top-5 retrieval: 34.4 -> 45.6 | Helped |
| Add CCA subspace projection before affine alignment (ablation) | PointNet and CLIP text encoder · Objaverse-LVIS | matching accuracy: 18.4 -> 25.2 | Helped |
| Use affine alignment rather than local CKA without subspace projection (ablation) | PointBERT and CLIP text encoder · Objaverse-LVIS | matching accuracy: 5.8 -> 15.8 | Helped |
| Use local CKA rather than affine alignment without subspace projection (ablation) | PointBERT and CLIP text encoder · Objaverse-LVIS | top-5 retrieval: 23.6 -> 15.2 | Hurt |
| Increase subspace dimension from higher to lower dimensions in CCA + affine alignment (ablation) | PointBERT and CLIP text encoder · Objaverse-LVIS validation set | top-5 accuracy | Mixed |
| Increase the anchor set size (ablation) | CCA subspace projection method · Objaverse-LVIS validation set | top-5 accuracy | Helped |
| Use a more complex SparseConv encoder instead of PointBERT (ablation) | Uni-modal 3D encoder alignment · Objaverse-LVIS | matching accuracy | Hurt |
| Use PointNet instead of a more complex uni-modal 3D encoder (ablation) | Uni-modal 3D encoder alignment · Objaverse-LVIS | matching accuracy | Mixed |
| Use the reduced 3D latent subspace rather than the original 3D latent space (ablation) | PointBERT · Shape retrieval | semantic similarity | Helped |
Surprising findings
- Uni-modal 3D encoders show much weaker alignment with text than vision encoders do, and model complexity (PointNet vs PointBERT vs SparseConv) had little effect on alignment.
- Projected subspaces emphasize semantic content (e.g., retrieving animals for sheep queries) while the full space favors geometric resemblance.
Limitations
- Performance plateaus with more anchors, likely due to the constraint of the fixed low-dimensional subspace.
- Only Objaverse was used; scaling to Objaverse-XL and object vs scene distinctions were not studied.
- Alignment remains well below vision-text alignment levels for uni-modal 3D encoders.
Experimental setup
Datasets: Objaverse Objaverse-LVIS Cap3D captions ShapeNet (background)
Benchmarks: Objaverse-LVIS
Metrics: matching accuracy top-5 retrieval accuracy linear CKA Pearson correlation with Chamfer distance
Baselines: Affine translation (no subspace) Local CKA (no subspace) OpenShape ULIP-2 Uni3D No transformation
Builds on: Platonic Representation Hypothesis latent space translation (affine) local CKA matching CCA
Closest papers by meaning
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.88)
- Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence (NeurIPS 2026 · similarity 0.88)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.88)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.88)
- Uni3D: Exploring Unified 3D Representation at Scale (ICLR 2024 · similarity 0.88)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.