Research map / Label-efficient and robust learning / Self-supervised and contrastive
Non-parametric Representation Learning with Kernels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces kernel versions of contrastive self-supervised learning (simple and spectral contrastive losses) and a kernel autoencoder, with new representer theorems and generalisation bounds, evaluated on small datasets against kernel PCA and neural networks.
Problem
Representation learning has been dominated by neural networks; non-parametric kernel methods for self-supervised and reconstruction-based representation learning are largely unexplored, especially in small-data regimes.
Approach
Kernelize a single hidden layer for contrastive (simple and spectral) losses and an autoencoder, prove representer theorems (including under orthonormality constraints) showing learned embeddings can be written via kernel matrices, and derive generalisation bounds for these kernel models.
Key results
- On Circles, Cubes, Iris and Ionosphere, some kernel embedding (k=3 nearest neighbour on 2-dim embedding) outperforms k-NN on original features, as reported in the text.
- Kernel AE outperforms the neural network AE in all considered denoising settings on MNIST, CIFAR-10 and SVHN (mean squared error on test set).
- Kernel contrastive methods perform on par with or outperform the neural network counterparts on CIFAR-10 and MNIST subset classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a learned kernel representation instead of the original features | Kernel representation learning with k-NN · Circles, Cubes, Iris, and Ionosphere | accuracy | Helped |
| Use the proposed kernel representation methods instead of Kernel PCA | Kernel representation learning · Circles, Cubes, Iris, and Ionosphere | classification performance | No clear effect |
| Use kernel representation learning instead of a one-hidden-layer neural network representation | Kernel representation learning · CIFAR-10 and MNIST (first two classes; n=500) | classification performance | No clear effect |
| Use Kernel AE instead of neural network AE | Kernel AE · MNIST, CIFAR-10, and SVHN denoising | mean squared error | Helped |
| Increase the number of unlabelled datapoints | Kernel representation learning · Circles (linear and ReLU kernels) | accuracy | Helped |
Surprising findings
- Kernel AE with a universal kernel and zero regularisation allows any n distinct bottleneck points to achieve perfect reconstruction, so regularisation is necessary.
Limitations
- Exact kernel computations do not scale to large datasets
- Performance depends strongly on kernel choice and no clear trend between embedding methods
- Equivalence to NTK-based infinite-width SSL is left as an open problem
Experimental setup
Datasets: concentric circles cubes Iris Ionosphere CIFAR-10 MNIST SVHN
Benchmarks: concentric circles cubes Iris Ionosphere MNIST CIFAR-10 SVHN
Metrics: k-NN classification accuracy mean squared error on denoising test set
Baselines: Kernel PCA k-nearest neighbours on original features neural network contrastive model neural network autoencoder
Builds on: kernel PCA spectral contrastive loss neural tangent kernel representer theorem
Closest papers by meaning
- Understanding Representation Learnability of Nonlinear Self-Supervised Learning (AAAI 2023 · similarity 0.88)
- I-Con: A Unifying Framework for Representation Learning (ICLR 2025 · similarity 0.85)
- On Statistical Learning Theory for Distributional Inputs (ICML 2024 · similarity 0.84)
- Measuring Self-Supervised Representation Quality for Downstream Classification Using Discriminative Features (AAAI 2024 · similarity 0.84)
- Kernel Representation Learning for Time Sequence: Algorithm, Theory, and Applications (AAAI 2025 · similarity 0.84)
- Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.