Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a hyperspherical-energy repulsion over CKA feature Gram matrices (plus repulsion on synthetic OOD inputs) to deep ensembles and hypernetworks to diversify networks and improve uncertainty/OOD detection.
Problem
Particle-based Bayesian deep learning needs a permutation-invariant, high-dimensional-friendly similarity measure between networks to promote diverse ensembles and hypernetwork posteriors; plain CKA gives vanishing gradients when networks are already similar.
Approach
Treat layerwise CKA-normalized feature Gram matrices as points on a unit hypersphere and minimize their hyperspherical (Riesz s-kernel) energy as a repulsive prior, plus feature repulsion on synthetic outlier inputs with an entropy-maximization term.
Key results
- On Dirty-MNIST vs Fashion-MNIST, Ensemble+OOD HE reaches 99.996% PE AUROC versus 86.10% for the plain ensemble and 94.17% for DDU.
- On TinyImageNet, Ensemble+OOD HE attains 99.31% AUROC on SVHN and 90.94% on DTD versus 89.81% and 68.96% for the ensemble.
- On CIFAR-10 vs SVHN with ResNet32 ensembles, SVGD+HE gives 0.255 NLL and 85.89% accuracy, and Ensemble+OOD HE reaches 96.48% PE AUROC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OOD HE-CKA to an ensemble | LeNet5 ensemble · Dirty-MNIST vs Fashion-MNIST | AUROC PE: 86.095 \pm 1.608 -> 99.996 \pm 0.001 | Helped |
| Add HE-CKA to an ensemble | LeNet5 ensemble · Dirty-MNIST vs Fashion-MNIST | AUROC PE: 86.095 \pm 1.608 -> 94.656 \pm 1.095 | Helped |
| Add OOD HE-CKA to an ensemble | Modified ResNet32 ensemble · CIFAR-10 vs SVHN | AUROC PE: 84.30 \pm 0.400 -> 96.478 \pm 0.413 | Helped |
| Add OOD HE-CKA to an ensemble | ResNet18 ensemble · CIFAR-10 vs SVHN | AUROC PE: 96.18 -> 99.86 | Helped |
| Add OOD HE-CKA to an ensemble | ResNet18 ensemble · TinyImageNet vs SVHN | AUROC PE: 89.81 -> 99.31 | Helped |
| Add OOD HE-CKA to an ensemble | ResNet18 ensemble · CIFAR-100 vs SVHN | AUROC PE: 89.62 -> 99.44 | Helped |
| Use 2 particles rather than 5 (ablation) | ResNet18 HE-CKA ensemble · TinyImageNet ID | ID Accuracy: 63.10 -> 59.21 | Hurt |
| Use 2 particles rather than 5 (ablation) | ResNet18 HE-CKA ensemble · TinyImageNet vs SVHN | AUROC PE: 92.65 -> 89.87 | Hurt |
| Increase particles from 2 to 4 (ablation) | ResNet18 HE-CKA ensemble · TinyImageNet vs SVHN | AUROC PE: 89.87 -> 92.92 | Helped |
| Apply HE-CKA repulsion | ResNet18 ensemble · CIFAR-10 feature diversity | Layer Mean CKA: 0.955 -> 0.479 | Helped |
Surprising findings
- Plain pairwise CKA minimization (SVGD+CKA) yields higher layer-mean CKA (0.870) than the HE variant (0.816), while HE alone reaches 0.479 in the CIFAR-10 ensemble analysis.
- Ensemble+OOD HE on CIFAR-10 vs SVHN has lower accuracy than plain ensemble in some settings and worse ECE than the Hypernet variant, showing trade-offs across metrics.
Limitations
- Many hyperparameters (gamma, beta, layer weights, smoothing terms) need dataset-specific tuning.
- CKA-based kernels are sensitive to number of particles and batch size.
- Higher computational and memory cost than RBF kernels: complexity O(L N^2 n^2).
Experimental setup
Datasets: Dirty-MNIST Fashion-MNIST CIFAR-10 CIFAR-100 SVHN TinyImageNet DTD synthetic 2D four-cluster classification synthetic 1D regression
Benchmarks: Dirty-MNIST vs Fashion-MNIST CIFAR-10 vs SVHN CIFAR-100 vs SVHN TinyImageNet vs SVHN/CIFAR-10/CIFAR-100/DTD
Metrics: NLL accuracy ECE AUROC (predictive entropy) AUROC (mutual information) training batch time CUDA memory
Baselines: Deep Ensemble SVGD+RBF KDE-WGD+RBF SSGE-WGD+RBF DDU Hypernetwork SVGD+CKA (pairwise cosine)
Builds on: Stein Variational Gradient Descent Centered Kernel Alignment Minimum Hyperspherical Energy Deep Deterministic Uncertainty Deep Ensembles
Closest papers by meaning
- Critical feature learning in deep neural networks (ICML 2024 · similarity 0.84)
- Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses (ICML 2024 · similarity 0.83)
- On the Target-kernel Alignment: a Unified Analysis with Kernel Complexity (NeurIPS 2024 · similarity 0.83)
- The Price of Locality: Why Forward-Forward Underperforms Backpropagation? (NeurIPS 2026 · similarity 0.83)
- HYPO: Hyperspherical Out-Of-Distribution Generalization (ICLR 2024 · similarity 0.83)
- Bayesian Low-Rank Learning (Bella): A Practical Approach to Bayesian Neural Networks (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.