Research map / Vision-language and multimodal / Cross-modal learning
Gramian Multimodal Representation Learning and Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRAM, a Gramian-volume similarity measure over n modality embeddings, and a GRAM-based contrastive loss used to further pretrain VAST for video-audio-text retrieval and audio-text classification.
Problem
Pairwise cosine-similarity contrastive alignment of multimodal models (each modality to an anchor) does not ensure alignment among all modalities, hurting tasks needing joint multimodal understanding.
Approach
Measure alignment of k modality embeddings by the Gramian volume (sqrt determinant of Gram matrix) of the parallelotope they span, and use a volume-based contrastive loss so all modalities are aligned jointly; reduces to sine of angle for two modalities.
Key results
- Zero-shot text-to-video R@1 on MSR-VTT improves from 49.3 (VAST T-VA) to 54.2 with GRAM (T-VA), and to 54.8 with audio and subtitles.
- Zero-shot text-to-audio R@10 on AudioCaps reaches 75.3 vs 65.4 for VAST (+9.9).
- Adding modalities on MSR-VTT zero-shot raises T2V R@1 from 52.8 (text+video) to 55.3 with five modalities including depth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace pairwise cosine contrastive alignment with GRAM-based alignment | VAST multimodal model · MSR-VTT zero-shot T2V | R@1: 49.3 -> 54.2 | Helped |
| Replace pairwise cosine contrastive alignment with GRAM-based alignment | VAST multimodal model · DiDeMo zero-shot T2V | R@1: 49.5 -> 54.2 | Helped |
| Replace pairwise cosine contrastive alignment with GRAM-based alignment | VAST multimodal model · ActivityNet zero-shot T2V | R@1: 51.4 -> 59.0 | Helped |
| Replace pairwise cosine contrastive alignment with GRAM-based alignment | VAST multimodal model · MSR-VTT finetuned T2V | R@1: 56.6 -> 64.0 | Helped |
| Replace pairwise cosine contrastive alignment with GRAM-based alignment | VAST multimodal model · AudioCaps zero-shot text-to-audio | R@10: 65.4 -> 75.3 | Helped |
| Add audio to text-video inputs (ablation) | GRAM model · MSR-VTT zero-shot T2V | R@1: 52.8 -> 54.1 | Helped |
| Add subtitles to text-video-audio inputs (ablation) | GRAM model · MSR-VTT zero-shot T2V | R@1: 54.1 -> 54.8 | Helped |
| Add depth to text-video-audio-subtitles inputs (ablation) | GRAM model · MSR-VTT zero-shot T2V | R@1: 54.8 -> 55.3 | Helped |
| Add DAM to GRAM D2A+A2D losses (ablation) | GRAM model · MSR-VTT from scratch T2V | R@1: 38.0 -> 38.9 | Helped |
| Add DAM to GRAM D2A+A2D losses (ablation) | GRAM model · ActivityNet from scratch T2V | R@1: 30.0 -> 30.2 | Helped |
| Use GRAM D2A+A2D+DAM instead of pairwise TV+TA cosine losses (ablation) | GRAM model · MSR-VTT from scratch T2V | R@1: 36.4 -> 38.9 | Helped |
| Use GRAM D2A+A2D+DAM instead of pairwise TV+TA cosine losses (ablation) | GRAM model · ActivityNet from scratch T2V | R@1: 23.6 -> 30.2 | Helped |
Surprising findings
- Modality gap grows under GRAM training, with larger inter-centroid distances than VAST, and the authors state no proof links this gap to performance.
- Cosine baselines fail when one modality lacks enough information, while GRAM uses joint volume to correct such retrievals (qualitative examples).
Limitations
- Volume is only informative when number of modalities k is at most the embedding dimension n; k>n gives zero volume.
- Downstream gains depend on the correlation of modalities in the training data (e.g., weaker on DiDeMo and ActivityNet where audio is semantically dissimilar).
- Pretraining only on a 150k-sample subset of VAST27M; pretrained weights not yet released at review time.
Experimental setup
Datasets: VAST27M MSR-VTT VATEX DiDeMo ActivityNet AudioCaps VGGSound
Benchmarks: MSR-VTT DiDeMo ActivityNet VATEX AudioCaps VGGSound 5K
Metrics: R@1 R@10 Acc@1 Acc@10
Baselines: VAST LanguageBind ImageBind UMT-L InternVideo-L VideoPrism-b mPLUG-2 ViCLIP
Builds on: CLIP VAST ImageBind LanguageBind
Closest papers by meaning
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.90)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.89)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.88)
- DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.88)
- Scaling Language-centric Omnimodal Representation Learning (NeurIPS 2025 · similarity 0.88)
- Mind the Heads: Topological Representation Alignment for Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.