Research map / Label-efficient and robust learning / Recognition and long-tail
Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Weighted Point Set Embedding (WPSE), where encoders output sets of (weight, vector) pairs compared by a linear+nonlinear (RFF) kernel, and gives a theory linking symmetric InfoNCE's optimum to pointwise mutual information with downstream excess-risk bounds.
Problem
One-point embeddings in CLIP-style contrastive learning struggle to capture broad, many-to-many concept relationships and the similarity structure of real-world instances; also, theory for CLIP's symmetric InfoNCE has gaps (only upper bounds, or analysis not matching CLIP).
Approach
Replace single-vector embeddings with weighted point sets, scoring similarity by a kernel-weighted sum over point pairs (linear plus random-Fourier-feature nonlinear kernel), and show this class can approximate the pointwise mutual information that minimizes symmetric InfoNCE, with a linear-probe excess-risk bound.
Key results
- On CC12M, WPSE Gaussian reaches 46.12% average zero-shot accuracy vs. 43.78% for CLIP across 13 datasets.
- On CC3M linear probing, WPSE Gaussian averages 69.01% vs. 67.00% for CLIP.
- Ablation on CC3M: WPSE with positive weights collapses to 4.22% zero-shot accuracy, versus 27.04% for WPSE IMQ, indicating negative weights are crucial.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CLIP point embeddings with weighted point sets using the IMQ kernel | CLIP pretrained on CC3M · 13-dataset average zero-shot classification | accuracy (%): 25.03 -> 27.04 | Helped |
| Replace CLIP point embeddings with weighted point sets using the Gaussian kernel | CLIP pretrained on CC12M · 13-dataset average zero-shot classification | accuracy (%): 43.78 -> 46.12 | Helped |
| Replace CLIP point embeddings with weighted point sets using the Gaussian kernel | CLIP pretrained on CC3M · 13-dataset average linear classification | accuracy (%): 67.00 -> 69.01 | Helped |
| Replace CLIP point embeddings with weighted point sets using the Gaussian kernel | CLIP pretrained on CC12M · 13-dataset average linear classification | accuracy (%): 77.89 -> 79.08 | Helped |
| Constrain all weighted-point-set weights to be positive (ablation) | WPSE with IMQ kernel on CC3M · 13-dataset average zero-shot classification | accuracy (%): 27.04 -> 4.22 | Hurt |
| Use only a linear kernel instead of the combined kernel (ablation) | WPSE on CC3M · 13-dataset average zero-shot classification | accuracy (%): 27.04 -> 27.25 | No clear effect |
| Use only a linear kernel instead of the combined kernel (ablation) | WPSE on CC3M · 13-dataset average linear classification | accuracy (%): 68.23 -> 67.40 | Hurt |
| Use only a nonlinear kernel instead of the combined kernel (ablation) | WPSE on CC3M · Pretraining | training status | Hurt |
| Use only a nonlinear kernel instead of the combined kernel (ablation) | WPSE on CC12M · 13-dataset average zero-shot classification | accuracy (%): 46.12 -> 44.05 | Hurt |
| Use only a nonlinear kernel instead of the combined kernel (ablation) | WPSE on CC12M · 13-dataset average linear classification | accuracy (%): 79.08 -> 70.99 | Hurt |
| Use only a linear kernel instead of the combined kernel (ablation) | WPSE on CC12M · 13-dataset average zero-shot classification | accuracy (%): 46.12 -> 45.87 | Hurt |
| Use only a linear kernel instead of the combined kernel (ablation) | WPSE on CC12M · 13-dataset average linear classification | accuracy (%): 79.08 -> 78.61 | Hurt |
Surprising findings
- Restricting weights to be positive causes a severe drop (4.22% zero-shot), showing negative weights are essential.
- Training with only the nonlinear kernel failed (NaN loss) in the authors' experiments, so the linear term is needed for stability.
Limitations
- Increasing the number of points M and RFF dimension adds computation, which the authors acknowledge.
- Theory relies on assumptions (e.g., label subsets proportional to label probabilities, compact support, Lipschitz PMI) that are argued plausible rather than verified.
- Experiments are limited to text-image pretraining on CC3M and CC12M with single-run-style reporting for non-RFF models.
Experimental setup
Datasets: Conceptual Captions 3M Conceptual Captions 12M
Benchmarks: ImageNet CIFAR-10 CIFAR-100 STL-10 Food-101 Caltech-101 Stanford Cars FGVC Aircraft Oxford Flowers EuroSAT DTD Oxford Pets
Metrics: top-1 accuracy mean per-class accuracy
Baselines: CLIP WPSE Linear WPSE with positive weights WPSE Nonlinear
Builds on: CLIP SLIP InfoNCE random Fourier features
Closest papers by meaning
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.86)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.85)
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.85)
- CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning (NeurIPS 2024 · similarity 0.84)
- The Double-Ellipsoid Geometry of CLIP (ICML 2025 · similarity 0.84)
- Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.