Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Input-gradient space particle inference for neural network ensembles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FoRDE, a particle-based variational inference ensemble that repels ensemble members in the space of normalized input gradients with a data-dependent RBF kernel (PCA or tuned lengthscales), improving corruption robustness over deep ensembles.
Problem
Deep ensembles often lack functional diversity, and existing ParVI repulsion in weight space is redundant while function-space repulsion gives little gain over deep ensembles and can underfit training points.
Approach
Perform Wasserstein gradient descent repulsion in the space of normalized first-order input gradients of the true-label logit, using an expected RBF kernel over training data with PCA-based (inverse eigenvalue) lengthscales so that members learn different, high-variance data features.
Key results
- On ResNet18/CIFAR-100, FoRDE-PCA reaches 56.1% corrupted accuracy versus 54.3% for deep ensembles, with cNLL 1.90 vs 1.99.
- On ResNet18/CIFAR-10, FoRDE-PCA gets 80.5% corrupted accuracy vs 78.1% for deep ensembles (about +2.4 points over second-best), though its clean NLL (0.125) is worse than deep ensembles (0.117).
- FoRDE-Identity achieves the best clean NLL on CIFAR-10 (0.113) and best clean accuracy on CIFAR-100 among methods, while FoRDE-Tuned balances clean and corrupted performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PCA rather than identity lengthscales for input-gradient repulsion (ablation) | FoRDE on CIFAR-100 · CIFAR-100-C | accuracy: 54.1 -> 56.1 | Helped |
| Use PCA rather than identity lengthscales for input-gradient repulsion (ablation) | FoRDE on CIFAR-100 · CIFAR-100 | accuracy: 82.1 -> 81.4 | Hurt |
| Use PCA rather than identity lengthscales for input-gradient repulsion (ablation) | FoRDE on CIFAR-10 · CIFAR-10-C | accuracy: 78.0 -> 80.5 | Helped |
| Use PCA rather than identity lengthscales for input-gradient repulsion (ablation) | FoRDE on CIFAR-10 · CIFAR-10 | accuracy: 96.3 -> 96.1 | Hurt |
| Use identity rather than PCA lengthscales for input-gradient repulsion (ablation) | FoRDE on CIFAR-10 · CIFAR-10 | NLL: 0.125 -> 0.113 | Helped |
| Use tuned lengthscales between identity and PCA (ablation) | FoRDE on CIFAR-10 · CIFAR-10-C | accuracy: 78.0 -> 79.1 | Helped |
| Use tuned lengthscales between identity and PCA (ablation) | FoRDE on CIFAR-100 · CIFAR-100 | accuracy: 82.1 -> 82.1 | No clear effect |
| Use tuned lengthscales between identity and PCA (ablation) | FoRDE on CIFAR-10 · CIFAR-10 | accuracy: 96.3 -> 96.4 | Mixed |
Surprising findings
- Function-space repulsion gives little or no improvement over plain deep ensembles, and feature-RDE has notably worse ECE on CIFAR-100 and TinyImageNet.
- FoRDE-Identity is slightly less robust to corruptions than deep ensembles despite higher clean accuracy, suggesting reliance on low-variance features.
Limitations
- Roughly three times the training cost of deep ensembles due to input-gradient and repulsion gradient computation.
- Requires choosing lengthscales; tuned lengthscales (alpha) are needed to balance clean and corrupted performance.
- Clean-data NLL is sometimes worse than deep ensembles for the PCA variant.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet CIFAR-10-C CIFAR-100-C TinyImageNet-C CINIC-10
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet CIFAR-10-C CIFAR-100-C TinyImageNet-C CINIC-10
Metrics: accuracy NLL ECE corrupted accuracy (cA) corrupted NLL (cNLL) corrupted ECE (cECE) epistemic uncertainty on OOD
Baselines: Deep ensembles weight-RDE function-RDE feature-RDE LIT node-BNNs SWAG EmpCov prior
Builds on: Wasserstein gradient descent deep ensembles EmpCov prior local independent training weight-space repulsive deep ensembles
Closest papers by meaning
- Feature Space Particle Inference for Neural Network Ensembles (ICML 2022 · similarity 0.91)
- Repulsive Deep Ensembles are Bayesian (NeurIPS 2021 · similarity 0.88)
- Generative Particle Variational Inference via Estimation of Functional Gradients (ICML 2021 · similarity 0.87)
- A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods (NeurIPS 2023 · similarity 0.86)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.85)
- Functional Gradient Flows for Constrained Sampling (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.