Research map / Detection and segmentation / CNN features
Intriguing Properties of Contrastive Losses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use different generalized contrastive losses and prior distributions instead of only NT-Xent (ablation) | ResNet-50 contrastive pretraining · CIFAR-10 and ImageNet | linear evaluation accuracy (top-1): CIFAR-10: 93.9 to 93.8; ImageNet, 3-layer head: 71.3 to 71.3 | No clear effect |
| Increase projection head depth from 2 to 3 layers (ablation) | ResNet-50 contrastive pretraining · ImageNet | linear evaluation accuracy (top-1): 69.3 -> 71.0 | Helped |
| Increase batch size with square-root learning-rate scaling (ablation) | ResNet-50 NT-Xent · ImageNet | linear evaluation accuracy (top-1): 69.3 -> 70.1 | No clear effect |
| Train SimCLR on images containing multiple digits rather than a single digit (ablation) | SimCLR · MultiDigits, random placement | top-1 linear evaluation accuracy: 98.9 -> 96.4 | No clear effect |
| Avoid overlap by placing digits in-grid rather than randomly (ablation) | SimCLR · MultiDigits, 16 digits | top-1 linear evaluation accuracy: 96.4 -> 98.3 | Helped |
| Increase the competing digit size while evaluating the fixed 20×20 digit (ablation) | SimCLR · MultiDigits, 1st digit 20×20 | top-1 linear evaluation accuracy: 97.8 -> 39.9 (τ=0.05); 98.7 -> 20.3 (τ=0.2) | Hurt |
| Increase temperature from τ=0.05 to τ=0.2 (ablation) | SimCLR · MultiDigits, 1st digit 20×20; 2nd digit 80×80 | top-1 linear evaluation accuracy: 39.9 -> 20.3 | Mixed |
| Add unique MNIST digits as shared competing features on ImageNet images (ablation) | SimCLR · DigitOnImageNet | ImageNet classification accuracy | Hurt |
| Add shared random-bit channels to RGB inputs (ablation) | SimCLR · MNIST, CIFAR-10, and ImageNet RandBit | linear evaluation accuracy | Hurt |
| Use a generative VAE rather than contrastive learning with added random bits (ablation) | VAE · MNIST RandBit | linear evaluation accuracy | No clear effect |
| Increase entropy in distribution matching without increasing batch size sufficiently (ablation) | Contrastive distribution-matching loss · RandBit | representation quality | Hurt |
| Use momentum contrast or different batch sizes/losses to address added random bits (ablation) | Contrastive learning · CIFAR-10 RandBit | linear evaluation accuracy | No clear effect |
Closest papers by meaning
- What Should Not Be Contrastive in Contrastive Learning (ICLR 2021 · similarity 0.91)
- Understanding the Behaviour of Contrastive Loss (CVPR 2021 · similarity 0.89)
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (ICML 2020 · similarity 0.89)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.88)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.88)
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.