Research map / Probabilistic and scientific modeling / Molecules and proteins
Permutation-Based Hypothesis Testing for Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes permutation-based hypothesis tests for neural networks, using partial derivatives of the network output w.r.t. an input to test for nonlinear and general association between that input and the outcome.
Problem
Neural networks are predictive but offer little statistical inference on whether input features are associated with outcomes, and whether such associations are linear or nonlinear.
Approach
Build gradient-based test statistics from network partial derivatives and estimate null distributions by retraining networks on permuted data: a GAM-residual permutation for the nonlinearity test and a feature-permutation for the association test.
Key results
- Nonlinearity test reaches power 1.00 for quadratic and cubic terms and 0.94 for nonsmooth terms at 5% level, with Type-I error 0.05 for linear terms.
- Association test has accurate Type-I error (0.026-0.030 for NN-1 in linear and nonlinear settings) and outperforms LM and GAM in the nonsmooth setting.
- Type-I error inflates to 0.32 under high input correlation, versus 0.05 with independent predictors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the neural-network permutation test rather than LM or GAM for nonsmooth associations | Association testing · Simulation: nonsmooth nonlinear associations | power | Helped |
| Use the neural-network permutation test rather than LM for smooth nonlinear associations | Association testing · Simulation: smooth nonlinear associations | power | Helped |
| Use a neural network permutation test rather than LM or GAM for linear associations | Association testing · Simulation: linear associations | power | Mixed |
| Add a second hidden layer to the association-test network | NN-1 association test · Simulation: nonsmooth nonlinear associations at small signal | power | Helped |
| Use correlated predictors instead of independent predictors | Neural-network permutation association test · Simulation: X1 association test | Type-I error: 0.05 -> 0.09 -> 0.32 | Hurt |
Surprising findings
- Type-I error of the association test rises to 0.32 under high predictor correlation, showing sensitivity to the independence assumption.
Limitations
- Permutation-based procedure is computationally intensive and impractical for very large networks.
- Assumes near-independence among the tested predictor; Type-I error inflates under correlated inputs.
- Requires well-trained networks; no asymptotic null distribution derived.
Experimental setup
Datasets: simulated data CHOP pediatric concussion data AMP PD whole genome sequencing data
Benchmarks: simulation studies (Type-I error and power) CHOP pediatric concussion data AMP PD genetic data
Metrics: Type-I error power p-values
Baselines: linear model generalized additive model NN-1 NN-2
Builds on: Racine (1997) partial derivative test generalized additive models neural additive models
Closest papers by meaning
- Full Bayesian Significance Testing for Neural Networks (AAAI 2024 · similarity 0.84)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.84)
- Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory (ICLR 2024 · similarity 0.83)
- Variable Importance in High-Dimensional Settings Requires Grouping (AAAI 2024 · similarity 0.83)
- On dimensionality of feature vectors in MPNNs (ICML 2024 · similarity 0.83)
- On the Variance of Neural Network Training with respect to Test Sets and Distributions (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.