Research map / Learning theory and algorithms / Kernels and regression theory
When Can We Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes training dynamics of two-layer wide contrastive networks, showing NTK stays nearly constant under cosine-similarity contrastive losses (not dot-product), and that with orthogonality constraints the learned representations are close to PCA of a random-feature matrix.
Problem
When do trained wide contrastive models behave like kernel methods (fixed NTK) or PCA, given that prior work only showed equivalences at the level of optimization objectives rather than training dynamics?
Approach
Derives NTK and gradient-flow dynamics for two-layer non-linear networks under general contrastive losses; proves NTK constancy up to M^{1/6} steps for cosine similarity but divergence for dot-product similarity; under orthogonality constraints shows the trace-maximization matrix C_V(t) stays close to C_V(0), so representations approximate PCA of a random-feature matrix.
Key results
- Under cosine similarity, NTK change decays roughly as 1/sqrt(M) with width and remains nearly constant for about M^{1/6} gradient steps (bound O(M^{-1/6}(log M)^3)).
- Under dot-product similarity, a proposition shows NTK can diverge within log M steps with probability at least 1-25/sqrt(M).
- PCA on C_V(0) gives downstream linear-probe accuracy close to that of a fully trained contrastive model on MNIST, with the gap shrinking as width grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cosine similarity instead of dot-product similarity | Wide contrastive networks · MNIST | NTK change | Helped |
| Increase network width | Orthogonality-constrained cosine-similarity model · MNIST | Frobenius norm of C_V(t) - C_V(0) | Helped |
| Use PCA on the initial matrix C_V(0) rather than the trained matrix C_V(t) | PCA representations · MNIST | Fractional difference between learned representations | Helped |
| Use PCA on C_V(0) or fully train the model | Representations evaluated with a linear classifier · MNIST | Downstream test accuracy | No clear effect |
| Use iterative PCA and gradient updates instead of full training | Trace-maximization optimization · MNIST | Downstream test accuracy | Mixed |
Surprising findings
- Dot-product contrastive losses cause the NTK to diverge within O(log M) steps even for arbitrarily wide networks.
- Certain initializations with identical columns of W cause dimension collapse even under cosine-similarity losses.
Limitations
- Full convergence of NTK and equivalence to PCA at convergence remain open problems.
- Experiments limited to MNIST with 1000 samples and small networks.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: NTK change Frobenius norm of C_V(t)-C_V(0) fractional representation difference downstream linear classifier accuracy
Baselines: PCA on C(0) fully trained contrastive model
Builds on: neural tangent kernel Grassmannian gradient descent InfoNCE SimCLR
Closest papers by meaning
- Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions (ICLR 2023 · similarity 0.90)
- Contrastive Learning is Spectral Clustering on Similarity Graph (ICLR 2024 · similarity 0.86)
- Variational Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- Generalization Analysis for Deep Contrastive Representation Learning (AAAI 2025 · similarity 0.86)
- Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settings (ICML 2025 · similarity 0.86)
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.