Research map / Architectures and efficiency / CNN design
Convolutional Deep Kernel Machines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces convolutional deep kernel machines (DKMs) with a novel inter-domain inducing point scheme, reaching 92.7% CIFAR-10 accuracy for kernel methods while retaining representation learning.
Problem
Infinite-width NNGP limits lack representation learning and underperform finite neural networks; existing DKMs were limited to small fully connected regression, and naive inducing-point schemes are intractable for convolutional image inputs.
Approach
Define a convolutional kernel over Gram matrices (arccos kernel plus convolutional covariance) and introduce inter-domain inducing points built as learned linear combinations of inducing inputs forming artificial patches, so inducing points need not be images; add batchnorm-like normalisation and rescaling for Gram matrices.
Key results
- Largest convolutional DKM (2048 inducing points in last block) reaches 92.7% on CIFAR-10, SOTA among kernel methods in the paper's comparison.
- Achieves about 99.3% on MNIST and 72% on CIFAR-100.
- Training takes about 77 GPU hours on a single NVIDIA A100, roughly one order of magnitude less than Myrtle kernels (316-508 GPU hours on V100 for full CIFAR-10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add convolutional DKM with learned representations | Kernel methods · CIFAR-10 | accuracy: 91.2% -> 92.69% | Helped |
| Set regularisation strength ν to ∞, removing data-adaptive Gram matrix learning (ablation) | Convolutional DKM · CIFAR-10 | accuracy: 87.86 $\pm$ 0.09 -> 53.74 $\pm$ 0.38 | Hurt |
| Set regularisation strength ν to 10⁻² (ablation) | Convolutional DKM · MNIST, CIFAR-10, and CIFAR-100 | accuracy: 99.20 $\pm$ 0.02 -> 99.16 $\pm$ 0.05; 87.86 $\pm$ 0.09 -> 89.48 $\pm$ 0.15; 64.08 $\pm$ 0.24 -> 65.64 $\pm$ 0.20 | Mixed |
| Remove normalisation from inducing and train-test blocks (ablation) | Convolutional DKM · MNIST, CIFAR-10, and CIFAR-100 | accuracy: 99.20 $\pm$ 0.02 -> 99.16 $\pm$ 0.04; 87.86 $\pm$ 0.09 -> 87.50 $\pm$ 0.19; 64.08 $\pm$ 0.24 -> 64.23 $\pm$ 0.08 | No clear effect |
| Remove rescaling from inducing and train-test blocks (ablation) | Convolutional DKM · MNIST, CIFAR-10, and CIFAR-100 | accuracy: 99.20 $\pm$ 0.02 -> 99.13 $\pm$ 0.04; 87.86 $\pm$ 0.09 -> 87.77 $\pm$ 0.11; 64.08 $\pm$ 0.24 -> 64.05 $\pm$ 0.14 | No clear effect |
| Replace global average pooling with a linear top layer (ablation) | Convolutional DKM · MNIST, CIFAR-10, and CIFAR-100 | accuracy: 99.20 $\pm$ 0.02 -> 99.13 $\pm$ 0.03; 87.86 $\pm$ 0.09 -> 87.89 $\pm$ 0.15; 64.08 $\pm$ 0.24 -> 64.06 $\pm$ 0.19 | No clear effect |
| Add inducing-point mixups (ablation) | Convolutional DKM · CIFAR-10 | accuracy: 87.86 $\pm$ 0.09 -> 87.69 $\pm$ 0.10 | No clear effect |
| Replace categorical likelihood with Gaussian likelihood (ablation) | Convolutional DKM · MNIST, CIFAR-10, and CIFAR-100 | accuracy: 99.20 $\pm$ 0.02 -> 99.05 $\pm$ 0.01; 87.86 $\pm$ 0.09 -> 85.96 $\pm$ 0.29; 64.08 $\pm$ 0.24 -> 26.93 $\pm$ 0.69 | Hurt |
| Increase the number of inducing points (ablation) | Convolutional DKM · CIFAR-10 | accuracy: 83.15 $\pm$ 0.26 -> 92.69 $\pm$ 0.06 | Helped |
Surprising findings
- Categorical likelihood gives dramatic gains on CIFAR-100 over Gaussian likelihood (64.08% vs 26.93%).
- Normalisation and rescaling choices made no meaningful difference in test accuracy, though they affected train performance.
Limitations
- Only low-resolution datasets (MNIST, CIFAR) tested; not ImageNet due to computational tricks not yet developed.
- Double precision floating point used, making training slower than necessary.
- Training is far more expensive than standard CNNs and inducing points scale costs up to 1380 s per epoch for largest model.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100
Benchmarks: MNIST CIFAR-10 CIFAR-100
Metrics: test accuracy test log-likelihood time per epoch
Baselines: NNGP-GAP Myrtle10 NNGP-LAP-flip Conv DGP CKN Sup CKN SOVI Conv GP
Builds on: Deep kernel machines (Yang et al. 2023) sparse DGP inducing point schemes NNGP convolutional DGP inducing patches arccos kernel
Closest papers by meaning
- A theory of representation learning gives a deep generalisation of kernel methods (ICML 2023 · similarity 0.93)
- Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines (NeurIPS 2024 · similarity 0.89)
- Deep kernel processes (ICML 2021 · similarity 0.88)
- Deep Kernel Posterior Learning under Infinite Variance Prior Weights (ICLR 2025 · similarity 0.87)
- Adaptive kernel predictors from feature-learning infinite limits of neural networks (ICML 2025 · similarity 0.86)
- Critical feature learning in deep neural networks (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.