Research map / Vision-language and multimodal / Cross-modal learning
Generative Modeling of Class Probability for Multi-Modal Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CALM, which aligns video and text class-probability distributions over Charades class anchors using a cross-modal probabilistic VAE, improving video-text retrieval and captioning especially out-of-domain.
Problem
Modality discrepancy and information imbalance between video and text make direct contrastive/pairwise matching misaligned, hurting multimodal representation learning and generalization.
Approach
Encode class labels as prompt anchors with CLIP text encoder, compute softmax class-probability distributions for video and text over these anchors, and use a cross-modal VAE to reconstruct the text-anchor distribution from the video-anchor distribution, aligning modalities generatively rather than by direct matching.
Key results
- On MSR-VTT in-domain video retrieval, CALM reaches R@1 50.8%, above the prior best DiffusionRet at 49.0%.
- When trained on MSR-VTT and tested on LSMDC, CALM reaches R@1 21.4% versus 19.6% for T-MASS (out-of-domain).
- On MSVD→MSR-VTT captioning, CALM achieves CIDEr 35.6 versus 30.5 for CLIP4Caption.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CALM instead of DiffusionRet for in-domain retrieval | MSR-VTT video retrieval · MSR-VTT | R@1: 49.0 -> 50.8 | Helped |
| Use CALM instead of T-MASS for cross-dataset retrieval | MSR-VTT-trained retrieval model · DiDeMo | R@1: 37.3 -> 41.2 | Helped |
| Use CALM instead of T-MASS for cross-dataset retrieval | MSR-VTT-trained retrieval model · LSMDC | R@1: 19.6 -> 21.4 | Helped |
| Use CALM instead of EMCL for in-domain retrieval | DiDeMo video retrieval · DiDeMo | R@1: 47.8 -> 51.1 | Helped |
| Use CALM instead of T-MASS for in-domain retrieval | LSMDC video retrieval · LSMDC | R@1: 26.0 -> 27.5 | Helped |
| Use CALM instead of CLIP4Caption for captioning | MSR-VTT captioning · MSR-VTT | CIDEr: 57.0 -> 59.3 | Helped |
| Use CALM instead of CLIP4Caption for cross-dataset captioning | MSVD-trained captioning model · MSR-VTT | CIDEr: 30.5 -> 35.6 | Helped |
| Use CALM instead of CLIP4Caption for in-domain captioning | MSVD captioning · MSVD | CIDEr: 114.6 -> 115.7 | Helped |
| Use CALM instead of CoCAP for cross-dataset captioning | MSR-VTT-trained captioning model · MSVD | CIDEr: 69.8 -> 70.3 | Helped |
| Increase the Charades anchor set from 0 to 50 anchors (ablation) | CALM video retrieval · MSR-VTT | R@1: 48.9 -> 48.1 | Hurt |
| Increase the Charades anchor set from 50 to 100 anchors (ablation) | CALM video retrieval · MSR-VTT | R@1: 48.1 -> 49.7 | Helped |
| Use 157 Charades anchors instead of no anchors (ablation) | CALM video retrieval · MSR-VTT | R@1: 48.9 -> 50.8 | Helped |
Surprising findings
- Class anchors from unrelated COCO object labels still improve over the no-anchor baseline, suggesting the model uses even seemingly irrelevant anchors.
Limitations
- Out-of-domain performance drops from in-domain are still large (13.7% on MSR-VTT R@1), and CLIP4Clip shows smaller drops on some datasets.
- Results rely on re-implemented baselines whose numbers differ from original papers.
Experimental setup
Datasets: MSR-VTT DiDeMo LSMDC MSVD Charades COCO
Benchmarks: MSR-VTT DiDeMo LSMDC MSVD
Metrics: R@1 R@5 R@10 MnR BLEU-4 METEOR CIDEr ROUGE-L
Baselines: CLIP4Clip EMCL UATVR DiffusionRet T-MASS SemSynAN TextKG CoCAP CLIP4Caption
Builds on: CLIP CLIP4Clip T-MASS DiffusionRet VAE
Closest papers by meaning
- Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.87)
- The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities (ICLR 2025 · similarity 0.86)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.86)
- Towards the Causal Complete Cause of Multi-Modal Representation Learning (ICML 2025 · similarity 0.86)
- Generative Multi-modal Models are Good Class Incremental Learners (CVPR 2024 · similarity 0.86)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.