Research map / Vision-language and multimodal / Cross-modal learning
MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MM-Mixing, a two-stage feature-level and input-level mixing alignment framework that trains a 3D point cloud encoder against frozen CLIP-based image and text encoders to improve zero-shot and linear-probe 3D classification.
Problem
Multi-modal (point cloud, image, text) alignment for 3D understanding has not explored mixing-based data augmentation, which could increase diversity and cross-modal consistency of 3D representations.
Approach
Applies feature-level mixing of modality embeddings with contrastive alignment in stage one, then adds input-level point cloud mixing (PointCutMix-style) with feature-level mixed alignment in stage two, using frozen CLIP-style image and text encoders as alignment targets.
Key results
- Zero-shot ScanObjectNN top-1 improved from 51.3% (OpenShape, Point-BERT, ShapeNet) to 61.9% with MM-Mixing.
- Zero-shot Objaverse-LVIS top-1 improved from 46.8% to 51.4% for Point-BERT with Ensembled pre-training.
- Linear probing on ShapeNet-pretrained Point-BERT reaches 90.6% on ModelNet40 and 83.3/85.0/73.2 on ScanObjectNN splits, improving over OpenShape by up to 9.1 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply feature-level and input-level mixing alignment | Point-BERT · ScanObjectNN (zero-shot, ShapeNet) | Top1 accuracy: 51.3 -> 61.9 | Helped |
| Apply feature-level and input-level mixing alignment | Point-BERT · Objaverse-LVIS (zero-shot, Ensembled) | Top1 accuracy: 46.8 -> 51.4 | Helped |
| Apply feature-level and input-level mixing alignment | Point-BERT · ModelNet40 (linear probing, ShapeNet) | accuracy: 90.6 -> 90.6 | No clear effect |
| Apply feature-level and input-level mixing alignment | Point-BERT · ScanObjectNN OBJ-BG (linear probing, ShapeNet) | accuracy: 77.8 -> 83.3 | Helped |
| Use feature-level mixing only (ablation) | MM-Mixing · ModelNet40 (zero-shot, ShapeNet) | Top1 accuracy: 72.9 -> 74.1 | Helped |
| Use input-level mixing only (ablation) | MM-Mixing · ModelNet40 (zero-shot, ShapeNet) | Top1 accuracy: 72.9 -> 73.8 | Helped |
| Combine feature-level and input-level mixing (ablation) | MM-Mixing · ModelNet40 (zero-shot, ShapeNet) | Top1 accuracy: 72.9 -> 75.2 | Helped |
| Use two-stage rather than simultaneous one-stage alignment (ablation) | MM-Mixing · ModelNet40 (zero-shot, ShapeNet) | Top1 accuracy: 73.6 -> 75.2 | Helped |
| Add image loss to text loss (ablation) | MM-Mixing · ModelNet40 (zero-shot, ShapeNet) | Top1 accuracy: 72.6 -> 73.8 | Helped |
| Add point cloud loss to text loss (ablation) | MM-Mixing · ModelNet40 (zero-shot, ShapeNet) | Top1 accuracy: 72.6 -> 73.9 | Helped |
| Combine text, image, and point cloud losses (ablation) | MM-Mixing · ModelNet40 (zero-shot, ShapeNet) | Top1 accuracy: 72.6 -> 75.2 | Helped |
Surprising findings
- Adding more training cost to OpenShape (e.g., more parameters or epochs) does not significantly improve its performance, while MM-Mixing gains persist.
- Performance gains on the larger Ensembled dataset are smaller than on ShapeNet, though still consistent.
Limitations
- Dual-level mixing demands significant computational resources.
- Performance depends on hyperparameter settings, which are hard to tune.
- Integrating multiple datasets can introduce inconsistencies requiring careful preprocessing.
Experimental setup
Datasets: ShapeNetCore 3D-FUTURE ABO Objaverse ModelNet40 ScanObjectNN Objaverse-LVIS
Benchmarks: ModelNet40 ScanObjectNN Objaverse-LVIS
Metrics: top-1 accuracy top-3 accuracy top-5 accuracy
Baselines: PointCLIP PointCLIP v2 ReCon CG3D CLIP2Point OpenShape ULIP
Builds on: OpenShape ULIP PointCutMix CLIP
Closest papers by meaning
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.88)
- TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding (CVPR 2024 · similarity 0.87)
- MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding (AAAI 2024 · similarity 0.87)
- Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training (CVPR 2024 · similarity 0.86)
- A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity (NeurIPS 2025 · similarity 0.86)
- Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.