Research map / Vision-language and multimodal / Cross-modal learning
A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRIANGLE, a triangle-area similarity over three modality embeddings that replaces pairwise cosine in contrastive losses, improving video-text, audio-text retrieval and audio-visual classification over VAST and GRAM.
Problem
Pairwise cosine-based contrastive alignment (anchor-based) cannot jointly align three modalities, leaving non-anchor modalities unaligned and limiting use of the third modality in downstream retrieval and classification.
Approach
Measure three-modality alignment directly as the area of the triangle formed by the three unit embedding vectors (computed from dot products), use it as the similarity inside contrastive losses, and optionally regularize with a pairwise cosine term for the task-relevant pair.
Key results
- Zero-shot video retrieval: TRIANGLE improves over VAST by up to 8.8 R@1 points on MSR-VTT V2T and 8.3 on ActivityNet T2V (up to 9 points cited).
- Zero-shot AudioCaps text-to-audio R@10 reaches 77.1 vs 65.4 for VAST (+11.7); VGGSound 5K R@1 44.8 vs 39.6 for VAST.
- Training from scratch on MSR-VTT: TRIANGLE T2AV R@1 39.4 vs GRAM 38.9 and VAST 36.5; Symile collapses to 0.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace pairwise cosine with triangle-area alignment | TRIANGLE with VAST encoders · Video-text retrieval on MSR-VTT, DiDeMo, ActivityNet, and VATEX | R@1: +5.9; +8.8; +5.4; +4.9; +8.3; +7.3; +3.9; +3.6 | Helped |
| Use triangle-area alignment across text, audio, and video instead of the VAST multimodal approach | TRIANGLE · AudioCaps text-to-audio retrieval | R@1: +0.1 | No clear effect |
| Use triangle-area alignment across text, audio, and video instead of the VAST multimodal approach | TRIANGLE · AudioCaps text-to-audio retrieval | R@10: +11.7 | Helped |
| Use triangle-area alignment across text, audio, and video instead of the VAST multimodal approach | TRIANGLE · VGGSound 5K audio-visual classification | R@1: +5.2 | Helped |
| Use triangle-area alignment across text, audio, and video instead of the VAST multimodal approach | TRIANGLE · VGGSound 5K audio-visual classification | R@10: +5.5 | Helped |
| Add the DTM loss (ablation) | TRIANGLE trained from scratch · MSR-VTT T2AV | R@1: 33.3 -> 39.4 | Helped |
| Add the DTM loss (ablation) | TRIANGLE trained from scratch · MSR-VTT T2AV | R@10: 74.4 -> 81.8 | Helped |
| Add the DTM loss (ablation) | TRIANGLE trained from scratch · MSR-VTT AV2T | R@1: 40.4 -> 41.9 | Helped |
| Add the DTM loss (ablation) | TRIANGLE trained from scratch · MSR-VTT AV2T | R@10: 81.7 -> 80.0 | Hurt |
| Set DTM loss weight lambda to 0.1 rather than 0.0, 0.3, 0.5, or 1.0 (ablation) | TRIANGLE trained from scratch · MSR-VTT T2AV | R@1: 33.3; 39.4; 20.9; 24.0; 38.3 | Helped |
| Set DTM loss weight lambda to 0.1 rather than 0.0, 0.3, 0.5, or 1.0 (ablation) | TRIANGLE trained from scratch · MSR-VTT AV2T | R@10: 81.7; 80.0; 72.4; 71.9; 78.4 | Mixed |
| Add cosine-similarity regularization to triangle-area minimization (ablation) | TRIANGLE · Text-to-visual-audio retrieval versus visual/audio-to-text retrieval | Retrieval performance | Mixed |
Surprising findings
- Symile reaches only ~0.3 R@1 when trained from scratch on MSR-VTT, indicating training instability.
- Further pretraining of VAST on the 150k subset overfit and gave no improvement.
Limitations
- Designed for exactly three modalities; extension to more modalities (convex hull, polygon decomposition) is left to future work.
- Pure area similarity can fail in flat-triangle cases, requiring a hand-tuned cosine regularization weight alpha.
- Gains rely on VAST backbone and pretraining subset; evaluation on a 5k VGGSound subset due to availability.
Experimental setup
Datasets: VAST27M MSR-VTT MNIST AudioMNIST Touch-Vision-Language Dataset
Benchmarks: MSR-VTT DiDeMo ActivityNet VATEX AudioCaps VGGSound 5K TVL
Metrics: R@1 R@10 Acc@1 Acc@5
Baselines: VAST GRAM Symile LanguageBind ImageBind InternVideo-L VideoPrism-b UMT-L ViCLIP VideoCoCa TVL
Builds on: CLIP VAST GRAM Symile CLAP
Closest papers by meaning
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.88)
- Understanding the Emergence of Multimodal Representation Alignment (ICML 2025 · similarity 0.88)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.88)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.88)
- AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.87)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.