Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KEP-SVGP, which places a pair of sparse variational GPs, induced by the left and right singular vectors of the asymmetric attention kernel (via Kernel SVD), on self-attention outputs to get calibrated uncertainty with linear-in-N-times-s² complexity.
Problem
Self-attention in Transformers gives overconfident predictions, and existing GP-based uncertainty methods assume symmetric kernels and have high complexity on large-scale sequence data.
Approach
Model the asymmetric attention kernel with a pair of SVGPs whose inducing features are the adjoint kernel-eigen features from Kernel SVD; the posterior then needs inversion of only a diagonal singular-value matrix, reducing the inversion cost from O(s^3) to O(s).
Key results
- On CIFAR-10, KEP-SVGP reaches 84.70% ACC and 35.15 AURC versus 83.50% and 42.60 for MSP.
- On IMDB, KEP-SVGP reaches 89.01% ACC and 3.00 NLL, improving over SGPA (88.36%, 3.40) with training time 32.30 s/epoch versus 1662.36 for SGPA.
- KEP-SVGP Ensembles beats Deep Ensembles on CIFAR-100 (62.45% vs 59.65% ACC) and on most OOD metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the last self-attention layer with KEP-SVGP | Transformer · CIFAR-10 | ACC: 83.50$\pm$0.43 -> 84.70$\pm$0.61 | Helped |
| Replace the last self-attention layer with KEP-SVGP | Transformer · CIFAR-100 | ACC: 52.82$\pm$0.53 -> 55.02$\pm$0.83 | Helped |
| Replace the last self-attention layer with KEP-SVGP | Transformer · IMDB | ACC: 88.17$\pm$0.52 -> 89.01$\pm$0.14 | Helped |
| Replace the last self-attention layer with KEP-SVGP | Transformer · CoLA | MCC: 26.93$\pm$1.38 -> 30.54$\pm$1.61 | Helped |
| Replace the last self-attention layer with KEP-SVGP | Transformer · CIFAR-100-C | ACC: 39.19 -> 39.69 | Mixed |
| Ensemble five KEP-SVGP models instead of using Deep Ensembles | Transformer ensemble · CIFAR-10 | ACC: 86.43 -> 87.62 | Helped |
| Ensemble five KEP-SVGP models instead of using Deep Ensembles | Transformer ensemble · CIFAR-100 | ACC: 59.65 -> 62.45 | Helped |
| Ensemble five KEP-SVGP models instead of using Deep Ensembles | Transformer ensemble · IMDB | ACC: 89.57 -> 89.68 | Helped |
| Ensemble five KEP-SVGP models instead of using Deep Ensembles | Transformer ensemble · CIFAR-10-C | ACC: 73.93 -> 73.67 | Hurt |
| Use the asymmetric e+r SVGP pair rather than a single-branch pairing (ablation) | 2-layer KEP-SVGP Transformer · CoLA | MCC: 28.71 -> 29.31; 28.10 -> 29.31 | Helped |
| Replace the asymmetric e+r pair with e+e branches (ablation) | 2-layer KEP-SVGP Transformer · CoLA | AURC: 199.48 -> 189.08 | Hurt |
| Replace the asymmetric e+r pair with r+r branches (ablation) | 2-layer KEP-SVGP Transformer · CoLA | NLL: 10.76 -> 9.02 | Hurt |
Surprising findings
- The 2-layer KEP-SVGP performs slightly worse than replacing only the last attention layer, which the authors attribute to shallow layers not being low-rank.
- KFLLLA achieves the best ECE/NLL on CIFAR-10 and CoLA but has poor accuracy and OOD detection.
Limitations
- Approximate posterior relies on a low-rank assumption for the attention matrix, which may not hold for shallow layers.
- Not state-of-the-art on every metric; KFLLLA and SV-DKL are better on some calibration metrics.
- Experiments are limited to small Transformers (7-layer ViT, 1-2 layer text models) on modest benchmarks.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 IMDB CoLA
Benchmarks: CIFAR-10 CIFAR-100 IMDB CoLA CIFAR-10-C CIFAR-100-C SVHN LSUN
Metrics: ACC MCC AURC AUROC FPR95 ECE NLL Brier AUPR
Baselines: MSP Temperature Scaling MC Dropout KFLLLA SV-DKL SGPA Deep Ensembles
Builds on: sparse variational gaussian process SGPA Primal-Attention deep kernel learning
Closest papers by meaning
- Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation (NeurIPS 2023 · similarity 0.87)
- Scalable Variational Bayesian Kernel Selection for Sparse Gaussian Process Regression (AAAI 2020 · similarity 0.87)
- Dual Parameterization of Sparse Variational Gaussian Processes (NeurIPS 2021 · similarity 0.87)
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.86)
- A variational approximate posterior for the deep Wishart process (NeurIPS 2021 · similarity 0.86)
- Sparse Gaussian Processes: Structured Approximations and Power-EP Revisited (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.