Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Amortized Variational Deep Kernel Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVDKL, a deep kernel learning model whose inducing points and variational parameters are amortized from the shared NN embedding, so the ELBO's fit and capacity terms are coupled and DKL's spurious non-local correlations are reduced.
Problem
Deep kernel learning (DKL) overfits because the neural network can inflate the marginal likelihood by over-correlating all inputs (a "non-local" kernel), since the network is not coupled to the variational terms of the ELBO.
Approach
Generate inducing locations and variational parameters from the same NN embedding used by the kernel (amortized variational inference over inducing points, with a saturating Tanh/Sigmoid output), so the network's projection affects both the model-fit and KL-capacity terms of the ELBO and is regularized by the KL.
Key results
- On CIFAR100 AVDKL reaches 0.7755 MNLL and 80.13% accuracy versus SVDKL's 0.8510 MNLL and 79.92% accuracy.
- AVDKL achieves the best test MNLL on 5 of 6 UCI tabular datasets (Table 1).
- On citation graphs AVDKL gets 85.48% accuracy on Cora, versus SVDKL at 79.83%, though SVDKL has lower MNLL on Cora and CiteSeer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AVDKL instead of GDKL | Tabular classification · HTRU2 | MNLL: 0.067 ± 0.004 -> 0.066 ± 0.004 | Helped |
| Use AVDKL instead of GDKL | Tabular classification · MagicGamma | MNLL: 0.284 ± 0.014 -> 0.286 ± 0.013 | Hurt |
| Use AVDKL instead of GDKL | Tabular classification · Letter | MNLL: 0.101 ± 0.002 -> 0.084 ± 0.003 | Helped |
| Use AVDKL instead of GDKL | Tabular regression · Protein | MNLL: 2.825 ± 0.014 -> 2.735 ± 0.015 | Helped |
| Use AVDKL instead of GDKL | Tabular regression · KeggD | MNLL: -1.979 ± 0.010 -> -2.283 ± 0.024 | Helped |
| Use AVDKL instead of DeepGCN | AVDKL · Cora | accuracy (%): 85.47 ± 0.44 -> 85.48 ± 0.51 | No clear effect |
| Use AVDKL instead of DeepGCN | AVDKL · Cora | MNLL: 0.8091 ± 0.0127 -> 0.7814 ± 0.0174 | Helped |
| Use AVDKL instead of DeepGCN | AVDKL · CiteSeer | accuracy (%): 72.45 ± 0.50 -> 72.54 ± 0.42 | No clear effect |
| Use AVDKL instead of DeepGCN | AVDKL · CiteSeer | MNLL: 0.9670 ± 0.0134 -> 0.9438 ± 0.0089 | Helped |
| Use AVDKL instead of DeepGCN | AVDKL · PubMed | accuracy (%): 79.91 ± 0.30 -> 80.22 ± 0.30 | Helped |
| Use AVDKL instead of SVDKL | AVDKL · CIFAR10 | MNLL: 0.1107 ± 0.0021 -> 0.1068 ± 0.0037 | Helped |
| Use AVDKL instead of SVDKL | AVDKL · CIFAR10 | accuracy (%): 96.82 ± 0.16 -> 96.81 ± 0.11 | No clear effect |
Surprising findings
- On citation-network datasets, increasing AVDKL's number of inducing points improves MNLL but reduces accuracy, a trade-off.
- SVDKL's ELBO was essentially equal to AVDKL's on CIFAR100 while its test MNLL was much worse, showing the ELBO does not capture the generalization gap.
Limitations
- On citation graphs AVDKL trades MNLL against accuracy depending on the number of inducing points.
- SVDKL needed dropout adjustment on CiteSeer to converge, showing sensitivity of training.
- GDKL could not be run on KeggU due to memory overflow, so some comparisons are incomplete.
Experimental setup
Datasets: MagicGamma HTRU2 Letter Protein KeggD KeggU Cora CiteSeer PubMed CIFAR10 CIFAR100
Benchmarks: MagicGamma HTRU2 Letter Protein KeggD KeggU Cora CiteSeer PubMed CIFAR10 CIFAR100
Metrics: MNLL accuracy ECE
Baselines: SVDKL SVGP GDKL DLVKL DeepGCN ResNet-18
Builds on: deep kernel learning stochastic variational deep kernel learning input dependent sparse GP (IDSGP) sparse variational GP
Closest papers by meaning
- Stochastic Variational Deep Kernel Learning (NeurIPS 2016 · similarity 0.88)
- Nearly-tight Bounds for Deep Kernel Learning (ICML 2023 · similarity 0.86)
- A variational approximate posterior for the deep Wishart process (NeurIPS 2021 · similarity 0.85)
- Sliced Kernelized Stein Discrepancy (ICLR 2021 · similarity 0.85)
- Convolutional Deep Kernel Machines (ICLR 2024 · similarity 0.84)
- Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.