Research map / Architectures and efficiency / CNN design
Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves convolutional deep kernel machines on CIFAR-10 by adding Wishart-sampled stochastic kernel regularisation to inducing Gram matrices, a Taylor-approximated KL objective, and TF32 precision, reaching 94.52% accuracy.
Problem
Convolutional deep kernel machines (DKMs), which learn representations directly as Gram matrices without neural network weights, still lag behind neural networks on CIFAR-10 and overfit; they are also numerically unstable in low-precision arithmetic.
Approach
Replace learned inducing Gram matrices with Wishart samples (stochastic kernel regularisation) during training, add jitter, and replace the log-determinant and trace KL terms with second-order Taylor expansions yielding a Frobenius-norm penalty, enabling stable TF32 training for many more epochs.
Key results
- Convolutional DKM with SKR reaches 94.52 ± 0.07% test accuracy on CIFAR-10, matching the Adam-trained neural network (94.55%) and above the prior DKM (92.69%).
- On CIFAR-100 the method reaches 75.31% versus 72.05% for the prior DKM, but below the SGD-trained network (79.42%).
- Removing SKR drops CIFAR-10 accuracy to 93.71% and increases numerical failures from 0/4 to 2/4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add stochastic kernel regularisation (ablation) | Conv. Deep Kernel Machine · CIFAR-10 test | accuracy: 93.71 ± 0.0150 -> 94.52 ± 0.0693 | Helped |
| add stochastic kernel regularisation (ablation) | Conv. Deep Kernel Machine · CIFAR-10 test | Test Log-Likelihood: -0.4512 ± 0.0168 -> -0.3611 ± 0.0073 | Helped |
| use Taylor approximation to the objective (ablation) | Conv. Deep Kernel Machine · CIFAR-10 test | accuracy: 94.46 ± 0.0406 -> 94.52 ± 0.0693 | No clear effect |
| use Taylor approximation to the objective (ablation) | Conv. Deep Kernel Machine · CIFAR-10 test | Test Log-Likelihood: -0.3951 ± 0.0081 -> -0.3611 ± 0.0073 | Helped |
| retain λ=0.1 jitter but remove SKR (ablation) | Conv. Deep Kernel Machine · CIFAR-10 test | accuracy: 93.46 (1 run) -> 94.52 ± 0.0693 | Hurt |
| remove both Taylor approximation and SKR (ablation) | Conv. Deep Kernel Machine · CIFAR-10 test | accuracy: 93.25 (1 run) -> 94.52 ± 0.0693 | Hurt |
| set νℓ to 0 (ablation) | Conv. Deep Kernel Machine · CIFAR-10 test | accuracy | Hurt |
| train for 1200 rather than 200 epochs (ablation) | Conv. Deep Kernel Machine · CIFAR-10 test | accuracy: 93.45 ± 0.0225 -> 94.52 ± 0.0693 | Helped |
| use the proposed Conv. Deep Kernel Machine instead of the previous Conv. Deep Kernel Machine | CIFAR-10 · CIFAR-10 test | accuracy: 92.69 ± 0.0600 -> 94.52 ± 0.0693 | Helped |
| use the proposed Conv. Deep Kernel Machine instead of the previous Conv. Deep Kernel Machine | CIFAR-100 · CIFAR-100 test | accuracy: 72.05 ± 0.2300 -> 75.31 ± 0.0814 | Helped |
| use the proposed Conv. Deep Kernel Machine instead of Neural Network (Adam) | CIFAR-10 · CIFAR-10 test | accuracy: 94.55 ± 0.0361 vs 94.52 ± 0.0693 | No clear effect |
| use Neural Network (SGD + Weight Decay) instead of the proposed Conv. Deep Kernel Machine | CIFAR-10 · CIFAR-10 test | accuracy: 94.52 ± 0.0693 vs 95.36 ± 0.0523 | Hurt |
Surprising findings
- Jitter alone without SKR still degraded accuracy (about 93.46% on one run), suggesting the gain comes from the Wishart sampling rather than the jitter.
- The SGD-trained neural network with the same architecture still outperforms the DKM (95.36% on CIFAR-10), and SGD was less stable than Adam for the DKM.
Limitations
- Remaining gap to the best neural networks in both accuracy and runtime.
- Evaluated only on CIFAR-10 in depth; CIFAR-100 results are preliminary and the paper focuses on a single dataset due to compute limits.
- Some ablation results use single runs, and the Taylor approximation's statistically significant log-likelihood gain is not fully verified.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: test accuracy test log-likelihood failure count Gram matrix condition number
Baselines: Conv. Deep Kernel Machine (previous) Tuned Myrtle10 Kernel DA CG NNGP-LAP-flip Neural Network (Adam) Neural Network (SGD + Weight Decay)
Builds on: convolutional deep kernel machines deep Gaussian processes dropout batch kernel normalisation Myrtle kernels
Closest papers by meaning
- Stochastic Variational Deep Kernel Learning (NeurIPS 2016 · similarity 0.90)
- Convolutional Deep Kernel Machines (ICLR 2024 · similarity 0.89)
- Convolutional Spectral Kernel Learning with Generalization Guarantees (Abstract Reprint) (AAAI 2024 · similarity 0.89)
- End-to-End Kernel Learning with Supervised Convolutional Kernel Networks (NeurIPS 2016 · similarity 0.88)
- To Understand Deep Learning We Need to Understand Kernel Learning (ICML 2018 · similarity 0.88)
- The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods (ICLR 2021 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.