Research map / Vision-language and multimodal / Cross-modal learning
Contrastive Multimodal Fusion With TupleInfoNCE
Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, Li Yi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sample optimization of negative ratios (vs uniform alpha) (ablation) | TupleInfoNCE · NYUv2 semantic segmentation | mIoU: 1.7% mIoU drop without it | Improved |
| add sample optimization of negative ratios (vs uniform alpha) (ablation) | TupleInfoNCE · SUN RGB-D 3D detection | mAP@0.25: 0.5 mAP drop without it | Improved |
| add sample optimization of negative ratios (vs uniform alpha) (ablation) | TupleInfoNCE · CMU-MOSI | Acc-7: 0.6 Acc-7 drop without it | Improved |
| add sample optimization of negative ratios (vs uniform alpha) (ablation) | TupleInfoNCE · CMU-MOSEI | Acc-7: 0.4 Acc-7 drop without it | Improved |
| optimize augmentation (positive samples) rather than hand-designed (ablation) | TupleInfoNCE · NYUv2 semantic segmentation | mIoU: 1.1 mIoU drop without it | Improved |
| optimize augmentation (positive samples) rather than hand-designed (ablation) | TupleInfoNCE · SUN RGB-D 3D detection | mAP@0.25: 0.6 mAP drop without it | Improved |
| add tuple disturbing, sample optimization over plain InfoNCE on tuples | InfoNCE · NYUv2 semantic segmentation | mIoU: 42.1 -> 48.1 | Improved |
| add tuple disturbing, sample optimization over plain InfoNCE on tuples | InfoNCE · SUN RGB-D 3D detection | mAP@0.25: 56.8 -> 58.0 | Improved |
| add depth and normal modalities to RGB (ablation) | TupleInfoNCE · NYUv2 semantic segmentation | mIoU: 40.8 -> 48.1 | Improved |
| add a fourth modality set (L, ab, depth, normal) (ablation) | TupleInfoNCE · NYUv2 semantic segmentation | mIoU: 48.1 -> 48.6 | Improved |
| add an uninformative timestamp modality (ablation) | TupleInfoNCE · CMU-MOSEI | Acc-7: 52.7 -> 52.6 | No clear effect |
| use InfoMin-searched rotation (70 degrees) instead of TupleInfoNCE-optimized (40 degrees) (ablation) | augmentation search · NYUv2 semantic segmentation | mIoU: 47.41 -> 46.91 | Hurt |
Closest papers by meaning
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.88)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.87)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.86)
- QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization (NeurIPS 2024 · similarity 0.86)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
