Research map / Vision-language and multimodal / Human and embodied modeling
Understanding the Gain from Data Filtering in Multimodal Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Provides a linear-contrastive-learning theory showing teacher-based data filtering improves the clean-fraction dependence of error from 1/η to 1/√(ηn) (large η) and to η-independent (small η) under a stochastic bimodal corruption model.
Problem
How and why teacher-based data filtering helps multimodal contrastive learning when a fraction of image-text pairs is mismatched; prior theory lacked a characterization of the dependence on the clean fraction η.
Approach
Models web pairs as a mixture of matched and randomly mismatched pairs in a linear Gaussian spiked-covariance setting, and analyzes a train-filter-train procedure where a teacher trained on half the data scores the other half by inner product; derives error rates as a function of η with two threshold regimes.
Key results
- Unfiltered contrastive learning error scales as 1/(η√n), matched by a lower bound (Proposition).
- Teacher-based filtering error bounded by 1/√(ηn) for large η and 1/√n for small η (min of T_0.5 and T_0), with η-independent rate in the small-η regime.
- Synthetic experiments (d=10, d̃=8, r=4, γ=10^4, n=10M) confirm the 1/η dependence of unfiltered learning and improved dependence with filtering.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply teacher-based filtering before contrastive learning | Linear contrastive learning on the bimodal corruption model · Bimodal model, large clean fraction | subspace recovery error (ERR): \nicefrac{1}{\eta} -> \nicefrac{1}{\sqrt{\eta}} | Helped |
| Apply teacher-based filtering before contrastive learning | Linear contrastive learning on the bimodal corruption model · Bimodal model, small clean fraction | subspace recovery error (ERR): \nicefrac{1}{\eta} -> independent of \eta | Helped |
| Set the filtering threshold to 0 (ablation) | Teacher-based filtering · Bimodal model | subspace recovery error (ERR): independent of \eta | Helped |
| Set the filtering threshold to r/2 (ablation) | Teacher-based filtering · Bimodal model | subspace recovery error (ERR): \nicefrac{1}{\sqrt{\eta}} | Helped |
| Filter training data before contrastive learning | Downstream model trained with real image-text data · Real image-text data, varying clean fraction η | downstream model performance | Helped |
Surprising findings
- Teacher-based filtering can achieve an error independent of the clean fraction η in the small-η regime, better than the oracle-clean-sample rate 1/√(ηn).
Limitations
- Analysis restricted to linear encoders and linear contrastive loss with Gaussian data and noise.
- Stochastic corruption model with random mismatch may not capture all real web noise.
- Teacher-based filtering is a two-step process; optimal threshold choice left open.
Experimental setup
Datasets: synthetic bimodal Gaussian data real image-text data (from prior work, reproduced)
Benchmarks: synthetic subspace recovery error (ERR)
Metrics: chordal distance (sin-theta Frobenius norm) error error vs clean fraction η
Baselines: unfiltered contrastive learning oracle filtering
Builds on: CLIP spiked covariance model Wedin's theorem Davis-Kahan
Closest papers by meaning
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · similarity 0.86)
- Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift (ICLR 2024 · similarity 0.86)
- CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning (NeurIPS 2024 · similarity 0.85)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.85)
- On the Comparison between Multi-modal and Single-modal Contrastive Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.