Research map / Learning theory and algorithms / Clustering
Contrastive Learning is Spectral Clustering on Similarity Graph
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that InfoNCE-based contrastive learning (SimCLR) equals spectral clustering on a similarity graph of augmentations, extends the result to CLIP's pair graph, and proposes Kernel-InfoNCE with mixtures of exponential kernels that beat the Gaussian kernel on CIFAR-10/100 and TinyImageNet.
Problem
Contrastive learning, notably SimCLR and CLIP, lacks a clear theoretical explanation of what objective it optimizes; existing spectral-contrastive analyses use a surrogate loss rather than the standard InfoNCE loss.
Approach
Models the augmentation similarity graph and the embedding Gram matrix as Markov random fields over subgraphs; shows that InfoNCE equals cross-entropy between these MRFs under a unitary out-degree constraint, which with a Gaussian kernel reduces to spectral clustering. The same argument covers CLIP's bipartite pair graph, and a maximum-entropy argument motivates exponential kernels, leading to mix
Key results
- Simple Sum Kernel reaches 91.72% on CIFAR-10 and 68.62% on CIFAR-100 at 400 epochs, versus 90.60% and 66.29% for the SimCLR reproduction.
- On TinyImageNet at 400 epochs, the Simple Sum Kernel reaches 39.38% versus 37.83% for SimCLR.
- With MoCo on CIFAR-100 at 1000 epochs, the Simple Sum Kernel reaches 59.83% versus 57.62% for the MoCo baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Gaussian kernel with a Simple Sum mixture of Gaussian and Laplacian kernels (ablation) | SimCLR · CIFAR-10 | accuracy: 88.11 ± 0.09 -> 89.82 ± 0.09; 90.60 ± 0.15 -> 91.72 ± 0.10 | Helped |
| replace Gaussian kernel with a Simple Sum mixture of Gaussian and Laplacian kernels (ablation) | SimCLR · CIFAR-100 | accuracy: 62.57 ± 0.10 -> 66.67 ± 0.20; 66.29 ± 0.12 -> 68.62 ± 0.15 | Helped |
| replace Gaussian kernel with a Simple Sum mixture of Gaussian and Laplacian kernels (ablation) | SimCLR · TinyImageNet | accuracy: 34.00 ± 0.18 -> 36.61 ± 0.13; 37.83 ± 0.09 -> 39.38 ± 0.16 | Helped |
| replace Gaussian kernel with a Laplacian kernel (ablation) | SimCLR · CIFAR-100, 400 epochs | accuracy: 66.29 ± 0.12 -> 66.09 ± 0.11 | No clear effect |
| replace Gaussian kernel with a gamma=0.5 exponential kernel (ablation) | SimCLR · CIFAR-100 | accuracy: 62.57 ± 0.10 -> 63.48 ± 0.22; 66.29 ± 0.12 -> 65.81 ± 0.19 | Mixed |
| replace Gaussian kernel with a concatenation of Gaussian and Laplacian kernels (ablation) | SimCLR · CIFAR-10 | accuracy: 88.11 ± 0.09 -> 89.89 ± 0.18; 90.60 ± 0.15 -> 91.28 ± 0.07 | Helped |
| replace Gaussian kernel with a concatenation of Gaussian and Laplacian kernels (ablation) | SimCLR · CIFAR-100 | accuracy: 62.57 ± 0.10 -> 66.10 ± 0.21; 66.29 ± 0.12 -> 68.57 ± 0.11 | Helped |
| replace Gaussian kernel with a concatenation of Gaussian and Laplacian kernels (ablation) | SimCLR · TinyImageNet | accuracy: 34.00 ± 0.18 -> 35.98 ± 0.17; 37.83 ± 0.09 -> 38.77 ± 0.22 | Helped |
| replace MoCo's Gaussian kernel with a Laplacian kernel (ablation) | MoCo · CIFAR-100, 200 epochs | accuracy: 47.02 ± 0.11 -> 46.12 ± 0.22 | Hurt |
| replace MoCo's Gaussian kernel with a Simple Sum mixture (ablation) | MoCo · CIFAR-100 | accuracy: 47.02 ± 0.11 -> 46.65 ± 0.06; 52.50 ± 0.07 -> 53.62 ± 0.19; 57.62 ± 0.15 -> 59.83 ± 0.12 | Mixed |
Limitations
- The equivalence assumes all objects are in the softmax denominator, whereas practice uses mini-batches with large batch sizes.
- CLIP analysis uses uniform object sampling rather than CLIP's uniform edge sampling, and the generalized spectral clustering interpretation is not symmetric.
- Kernel-InfoNCE experiments are limited to small vision benchmarks with a single linear-evaluation protocol and modest gains.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet
Metrics: linear evaluation accuracy
Baselines: SimCLR (Gaussian kernel) MoCo Laplacian kernel gamma=0.5 exponential kernel
Builds on: SimCLR InfoNCE spectral contrastive loss (HaoChen et al.) CLIP MoCo
Closest papers by meaning
- Multi-Hierarchical Contrastive Spectral Fusion for Multi-View Clustering (CVPR 2026 · similarity 0.88)
- Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions (ICLR 2023 · similarity 0.88)
- Contrastive Clustering (AAAI 2021 · similarity 0.88)
- Understanding Contrastive Learning via Gaussian Mixture Models (NeurIPS 2025 · similarity 0.88)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.87)
- When Can We Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis? (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.