Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Full Bayesian Significance Testing for Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes nFBST, a full Bayesian significance testing framework for neural networks that uses a variational Bayesian neural network, KDE on the posterior of a feature-importance statistic, and Bayesian evidence for H0, enabling global, local and instance-wise significance tests.
Problem
Classical significance tests need distributional derivations of the test statistic under assumed parametric forms, which fail for complex nonlinear relationships; they also only test global, not instance-wise, significance.
Approach
Replace analytic derivation with a Bayesian neural network posterior: sample network parameters from a variational posterior, compute a feature-importance testing statistic (gradient, LRP, DeepLIFT, LIME) per sample, estimate its density with KDE, and report Bayesian evidence for the null as the posterior mass of values less likely than the null point; a quantile-based aggregate (Q-GS) gives globa
Key results
- On simulation Dataset 1 and 2, Grad-, DeepLIFT-, LRP- and LIME-nFBST all reach F1 = 1.0 for global significance, versus 0.70-0.91 for the classical tests (Bootstrap, likelihood ratio, t-test).
- On the small-data Dataset 3 (1,000 samples), nFBST variants reach F1 about 0.82-0.85, while Bootstrap gets 0.68, the likelihood ratio test 0.67 and the t-test 0.68.
- For instance-wise significance on simulated data, each nFBST variant consistently improves AUC over its base feature-importance method across epsilon thresholds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Grad-nFBST instead of the strongest classical baseline | Global significance testing · Dataset 1 | F1-score: 0.91 -> 1 | Helped |
| Use Grad-nFBST instead of the strongest classical baseline | Global significance testing · Dataset 2 | F1-score: 0.89 -> 1 | Helped |
| Use Grad-nFBST instead of the strongest classical baseline | Global significance testing · Dataset 3 | F1-score: 0.68 -> 0.83 | Helped |
| Integrate nFBST with Grad feature importance | Grad feature importance · Simulation datasets, instance-wise significance | AUC | Helped |
| Integrate nFBST with DeepLIFT feature importance | DeepLIFT feature importance · Simulation datasets, instance-wise significance | AUC | Helped |
| Integrate nFBST with LRP feature importance | LRP feature importance · Simulation datasets, instance-wise significance | AUC | Helped |
| Integrate nFBST with LIME feature importance | LIME feature importance · Simulation datasets, instance-wise significance | AUC | Helped |
| Use LIME-based methods rather than other tested importance methods | LIME and LIME-nFBST · Simulation datasets, instance-wise significance | AUC | Hurt |
Surprising findings
- Bootstrap identifies almost all features as significant (recall 0.98-1.0) but with precision near 0.5, and the likelihood ratio test and t-test miss many significant features on the nonlinear toy data.
- LIME-based methods perform worse than other baselines in the instance-wise simulation, which the authors attribute to LIME's sampling efficiency.
Limitations
- Theoretical guarantee that Ev(H0) converges to 1 for insignificant features is stated only under certain constraints and proofs are in an appendix.
- Testing depends on the quality of the Bayesian neural network fit and KDE; approximation error is only indirectly gauged through prediction error.
- Evaluation of real-world data is qualitative (energy efficiency case and MNIST saliency visualization) without quantitative metrics reported in the main text.
Experimental setup
Datasets: synthetic toy regression data synthetic neural-network-generated regression data UCI energy efficiency MNIST
Benchmarks: toy example global significance simulation Datasets 1-3 global significance instance-wise AUC on simulation data UCI energy efficiency MNIST saliency
Metrics: p-value precision recall F1-score AUC Bayesian evidence
Baselines: Bootstrap likelihood ratio test t-test for linear models Grad DeepLIFT LRP LIME
Builds on: Full Bayesian Significance Testing (FBST) variational inference Layer-wise Relevance Propagation DeepLIFT LIME
Closest papers by meaning
- Permutation-Based Hypothesis Testing for Neural Networks (AAAI 2024 · similarity 0.84)
- Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks? (ICML 2024 · similarity 0.83)
- Testing Conditional Mean Independence Using Generative Neural Networks (ICML 2025 · similarity 0.83)
- Neural Conditional Probability for Uncertainty Quantification (NeurIPS 2024 · similarity 0.82)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.82)
- A Metalearned Neural Circuit for Nonparametric Bayesian Inference (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.