Research map / Label-efficient and robust learning / Self-supervised and contrastive
$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-CLR, a contrastive loss that replaces the one-hot positive target with a soft cross-sample similarity distribution derived from caption/class-name text embeddings, improving vision representations across ImageNet, CC3M and CC12M.
Problem
Standard contrastive losses encode a binary similarity graph (only one positive per sample) and ignore similarities across samples, limiting representation quality, especially for disambiguating objects from attributes and backgrounds.
Approach
Reinterpret contrastive losses as modifying a similarity graph, then replace the one-hot positive target in InfoNCE/SimCLR with a soft distribution obtained by softmax over text-encoder similarities between sample captions (or class-name captions), so that related samples are partially treated as positives.
Key results
- On CC12M, X-CLR outperforms CLIP by 0.6% on ImageNet and ImageNet Real.
- On CC3M, X-CLR outperforms CLIP by 17.2% on ImageNet and 18.0% on ImageNet Real.
- On ImageNet, X-CLR reaches 75.6% top-1 vs. 74.3% for SupCon and 63.4% for SimCLR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cross-sample class-label similarities instead of the SimCLR objective | ResNet-50 pretraining · ImageNet | accuracy: 63.4 -> 75.6 | Helped |
| Use cross-sample class-label similarities instead of SupCon | ResNet-50 pretraining · ImageNet Real | accuracy: 79.7 -> 81.5 | Helped |
| Train with X-CLR instead of SimCLR on CC3M | ResNet-50 · ImageNet | accuracy: 57.0 -> 58.2 | Helped |
| Train with X-CLR instead of CLIP on CC3M | ResNet-50 · ImageNet | accuracy: 41.0 -> 58.2 | Helped |
| Train with X-CLR instead of CLIP on CC12M | ResNet-50 · ImageNet Real | accuracy: 66.1 -> 66.7 | Helped |
| Fine-tune pretrained SimCLR with X-CLR for 10 epochs | Pretrained SimCLR ResNet-50 · ImageNet | accuracy: 63.4 -> 66.5 | Helped |
| Replace the Sentence Transformer graph with a WordNet hierarchy-distance graph (ablation) | X-CLR · ImageNet-9 Same Class | accuracy: 66.5 -> 55.7 | Hurt |
| Replace the Sentence Transformer graph with a CLIP text-encoder graph (ablation) | X-CLR · ImageNet-9 Same Class | accuracy: 66.5 -> 67.5 | Helped |
| Replace the Sentence Transformer graph with a random per-sample-pair graph (ablation) | X-CLR · ImageNet | accuracy: 75.6 -> 0.1 | Hurt |
| Increase CLIP training on CC3M from 32 to 100 epochs (ablation) | CLIP · ImageNet | accuracy: 36.8 -> 41.0 | Helped |
| Increase the X-CLR soft-target temperature too much (ablation) | X-CLR · ImageNet | accuracy | Hurt |
| Use higher-quality ImageNet class labels rather than noisy caption labels (ablation) | X-CLR · MIT States Attributes | accuracy: 30.1 -> 30.9 | Helped |
Surprising findings
- Random per-class-pair soft graph performs nearly as well as the sentence-transformer graph on ImageNet (74.5 vs 75.6), suggesting gains partly come from soft targets rather than semantic structure.
- Using a LLaMA2 text encoder for the similarity graph collapses performance (40.9% ImageNet).
Limitations
- Constructing the cross-sample similarity graph requires extra metadata/data and memory to store it.
- Performance depends heavily on graph quality, notably on noisy conceptual captions.
- Only InfoNCE-based objectives are explored; extension to non-contrastive methods is left for future work.
Experimental setup
Datasets: ImageNet-1k CC3M CC12M
Benchmarks: ImageNet ImageNet Real ImageNet-9 ObjectNet MIT States
Metrics: top-1 accuracy KNN accuracy linear probe accuracy label error intra-class connectivity
Baselines: SimCLR SupCon CLIP SCE ReSSL VICReg Barlow Twins
Builds on: SimCLR SupCon CLIP InfoNCE VICReg
Closest papers by meaning
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.90)
- Conditional Negative Sampling for Contrastive Learning of Visual Representations (ICLR 2021 · similarity 0.90)
- Dissecting Representation Misalignment in Contrastive Learning via Influence Function (ICLR 2026 · similarity 0.89)
- A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics (NeurIPS 2025 · similarity 0.88)
- BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.88)
- Variational Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.