Research map / Vision-language and multimodal / CLIP and zero-shot
TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAMM, a two-stage 3D shape pre-training method that first adapts CLIP's image encoder with a CLIP Image Adapter and then decouples the 3D feature space into visual and semantic sub-spaces via Image and Text Alignment Adapters.
Problem
Existing multi-modal 3D representation learning methods (ULIP, OpenShape) under-utilize the 2D image modality because rendered images are domain-shifted from CLIP's natural-image pretraining and 3D features cannot align equally well with both visual and semantic image/text features.
Approach
Two-stage pretraining: a lightweight residual CLIP Image Adapter re-aligns rendered-image features with text, then two independent adapters (IAA for visual, TAA for semantic) map 3D features into decoupled sub-spaces aligned with images (multi-view) and text respectively; zero-shot and linear probing use both sub-spaces.
Key results
- Zero-shot Objaverse-LVIS top-1 rises from 46.8% (OpenShape, Point-BERT, Ensembled) to 50.7% with TAMM.
- Linear probing on ModelNet40 with 5-way 10-shot improves from 96.1% (OpenShape) to 99.0% with TAMM on the Ensembled dataset.
- Real-world ScanNet zero-shot average accuracy improves by 3.8% over OpenShape and 10.9% over CLIP2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CLIP Image Adapter (CIA) before pre-training (ablation) | TAMM pre-training on ShapeNet with Point-BERT · Objaverse-LVIS zero-shot | Top-1 accuracy: 12.2 -> 13.7 (single CIA-only ablation row 13.7 vs 12.2 one-stage) | Helped |
| Add CLIP Image Adapter (CIA) to the image-text alignment (ablation) | Image-text contrastive accuracy · ShapeNet pre-training contrastive accuracy | contrastive accuracy: 40.1 -> 60.9 | Helped |
| Add Dual Adapters (IAA and TAA) with CIA (ablation) | TAMM pre-training on ShapeNet · Objaverse-LVIS and ModelNet40 zero-shot | Top-1 accuracy: Objaverse-LVIS 12.9 -> 13.7; ModelNet40 71.4 -> 73.1 (w/o CIA vs full) | Helped |
| Two-stage pre-training (CIA first, then Dual Adapters) instead of one stage (ablation) | TAMM on ShapeNet · Objaverse-LVIS and ModelNet40 zero-shot | Top-1 accuracy: Objaverse-LVIS 12.2 -> 13.7; ModelNet40 71.4 -> 73.1 | Helped |
| Pre-train with both image and text modalities instead of single modality (ablation) | TAMM on ShapeNet · Objaverse-LVIS zero-shot | Top-1 accuracy: image-only 11.9, text-only 10.5, both 13.7 | Helped |
| Use IAA+TAA combined features at inference instead of IAA alone (ablation) | TAMM zero-shot classification · Objaverse-LVIS zero-shot | Top-1 accuracy: IAA only 13.0, TAA only 12.9, combined 13.7 | Helped |
| Use IAA+TAA combined features at inference instead of single adapter (ablation) | TAMM zero-shot classification · ModelNet40 zero-shot | Top-1 accuracy: IAA only 68.1, TAA only 72.4, combined 73.1 | Helped |
| Align 3D shape with 4 rendered views instead of 1, 2, or 8 (ablation) | TAMM multi-view image alignment · Objaverse-LVIS and ModelNet40 zero-shot | Top-1 accuracy: Objaverse-LVIS best at 4 (13.7); ModelNet40 best at 2 and 8 (73.1, 73.0 with 4 at 73.1) | Mixed |
| TAMM pre-training (vs OpenShape) on Ensembled dataset | Point-BERT · Objaverse-LVIS zero-shot | Top-1 accuracy: 46.8 -> 50.7 | Helped |
| TAMM pre-training (vs OpenShape) on Ensembled dataset for linear probing | Point-BERT · Objaverse-LVIS linear probing | Top-1 accuracy: 48.3 -> 59.5 (+11.2) | Helped |
| TAMM pre-training (vs OpenShape) on Ensembled for few-shot linear probing | Point-BERT · ModelNet40 5-way 10-shot | Accuracy: 96.1 -> 99.0 | Helped |
Surprising findings
- Image-based 3D alignment in prior methods (ULIP, OpenShape) contributes much less than text; CLIP image-text contrastive accuracy on rendered images is only 40.1% without the adapter.
Limitations
- Limited computation prevented pre-training on very large 3D backbones with billions of parameters.
- Ablations were run only on small-scale ShapeNet pre-training with Point-BERT.
Experimental setup
Datasets: ShapeNetCore 3D-FUTURE ABO Objaverse Objaverse-LVIS ModelNet40 ScanObjectNN ScanNet
Benchmarks: Objaverse-LVIS ModelNet40 ScanObjectNN ScanNet
Metrics: Top-1 accuracy Top-3 accuracy Top-5 accuracy few-shot accuracy with std contrastive accuracy
Baselines: ULIP OpenShape PointCLIP PointCLIP v2 CLIP2Point ReCon CLIP2
Builds on: ULIP OpenShape CLIP Point-BERT domain adapters
Closest papers by meaning
- ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding (CVPR 2024 · similarity 0.89)
- MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding (AAAI 2025 · similarity 0.87)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.87)
- A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity (NeurIPS 2025 · similarity 0.87)
- 3DCoMPaT200: Language Grounded Large-Scale 3D Vision Dataset for Compositional Recognition (NeurIPS 2024 · similarity 0.87)
- Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spaces (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.