Research map / Vision-language and multimodal / Cross-modal learning
CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CovMatch, a multimodal dataset distillation method that reduces bi-level contrastive distillation to cross-covariance alignment under a linearized contrastive loss, and jointly updates image and text encoders to distill image-text pairs for retrieval.
Problem
Multimodal dataset distillation for image-text contrastive learning is computationally expensive with trajectory matching, and prior methods freeze the text encoder, which limits cross-modal semantic alignment and performance scaling.
Approach
Under a linearized (high-temperature) contrastive loss, the inner problem has a closed-form solution, so the outer objective becomes aligning real and synthetic cross-covariance matrices; this avoids unrolling and lets both encoders be trained online, with a per-modality feature-mean matching regularizer.
Key results
- On Flickr30K with 500 synthetic pairs, CovMatch reaches average retrieval 38.4 versus 31.6 for LoRS, a 6.8-point absolute gain.
- On COCO with 500 synthetic pairs, CovMatch averages 19.6 versus 13.5 for LoRS (6.1-point gain).
- Freezing the text encoder during distillation drops CovMatch from 38.4 to 29.4 average at N=500 in the unfrozen-evaluation setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CovMatch instead of LoRS | Multimodal dataset distillation · Flickr30K, 500 pairs | Avg retrieval: 31.6 -> 38.4 | Helped |
| Use CovMatch instead of LoRS | Multimodal dataset distillation · COCO, 500 pairs | Avg retrieval: 13.5 -> 19.6 | Helped |
| Use CovMatch instead of LoRS | Video-text dataset distillation · WebVid-10M subset, 500 pairs | Avg retrieval: 8.3 -> 11.0 | Helped |
| Unfreeze the text encoder at evaluation (ablation) | LoRS · Flickr30K, 500 pairs | Avg retrieval: 31.6 -> 32.8 | Helped |
| Unfreeze the text encoder during distillation and evaluation (ablation) | CovMatch · Flickr30K, 500 pairs | Avg retrieval: 29.4 -> 38.4 | Helped |
| Unfreeze the text encoder during evaluation (ablation) | CovMatch with text encoder frozen during distillation · Flickr30K, 500 pairs | Avg retrieval: 16.1 -> 29.4 | Helped |
| Increase real-data batch size from 64 to 1024 (ablation) | CovMatch · Flickr30K, 100 pairs | Performance: 29.1 -> 31.1 | Helped |
| Increase projection dimension from 256 to 2304 (ablation) | CovMatch · Flickr30K, 100 pairs | Performance: 16.1 -> 30.5 | Helped |
| Increase projection depth from 2 to 3 layers (ablation) | CovMatch · Flickr30K, 100 pairs | Performance: 30.5 -> 19.9 | Hurt |
| Optimize both image and text inputs rather than only one modality (ablation) | CovMatch · Flickr30K, 100 pairs | Avg retrieval: 24.5 -> 30.5; 20.8 -> 30.5 | Helped |
| Add feature matching to cross-covariance matching (ablation) | CovMatch · Flickr30K | Retrieval performance | Helped |
| Scale the real cross-covariance with an appropriate rho (ablation) | CovMatch · Flickr30K | Retrieval performance | Helped |
Surprising findings
- Unfreezing the text encoder during evaluation gives baselines only marginal gains (LoRS 31.6 to 32.8), indicating they cannot benefit because the text encoder is not involved in their distillation.
- Optimizing the cross-covariance loss alone leads to mean feature mismatches, so a feature matching regularizer is needed.
Limitations
- Assumes pretrained image and text encoders are available; training from scratch is not explored.
- Relies on a linearized (high-temperature) approximation of InfoNCE, justified only heuristically via NTK arguments.
Experimental setup
Datasets: Flickr30K COCO WebVid-10M (subset)
Benchmarks: Flickr30K COCO WebVid-10M subset video-text retrieval
Metrics: IR@1 IR@5 IR@10 TR@1 TR@5 TR@10 average recall
Baselines: Random Herding K-Center MTT-VL LoRS Tesla
Builds on: Matching Training Trajectories (MTT) MTT-VL LoRS FrePo ClipCov
Closest papers by meaning
- BiMAC: Bidirectional Multimodal Alignment in Contrastive Learning (AAAI 2025 · similarity 0.89)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.88)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.88)
- Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation (NeurIPS 2025 · similarity 0.88)
- Efficient Multimodal Dataset Distillation via Generative Models (NeurIPS 2025 · similarity 0.88)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.