Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Gaussian Process Neural Additive Models

AAAI 2024 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Infinite-width networks as Gaussian processes · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GP-NAM, a neural additive model whose shape functions are 1-D Gaussian processes approximated by random Fourier features, so only a linear last layer is trained, yielding a convex objective with far fewer parameters.

Problem

Neural additive models for interpretable tabular learning are computationally costly and non-convex, with many parameters and need for regularization.

Approach

Replace each feature's neural shape function with a GP with RBF kernel approximated by a fixed random Fourier feature map (a single cosine layer with unlearnable weights), so only linear weights are learned via least squares/logistic regression or conjugate gradients, giving a convex problem with parameters growing linearly in feature dimension.

Key results

• GP-NAM achieves best AUC on MIMIC-II (0.8508 vs NBM 0.8549, NAM 0.8539 close) and GMSC (0.8674 vs EBM 0.8655), and lowest Click error rate (0.3030 vs 0.3312 NBM).
• GP-NAM uses about 60x fewer parameters than NAM regardless of dimensionality (e.g., 801 vs 54K for Bikeshare, 501 vs 32K for LCD in Table 1 notation).
• Regression results on CA Housing RMSE 0.5586 match EBM and beat NAM (0.5721) and NBM (0.5638), though XGBoost and MLP reach 0.4428 and 0.5041.

Measured effects reported in the paper

Surprising findings

• Despite having far fewer parameters and no hidden-layer learning, GP-NAM matches or beats deeper NAM variants on many datasets.
• A plain linear model is highly competitive on several datasets (Credit, Adult, Churn, FICO, LCD, Microsoft), suggesting many benchmarks are nearly linear.

Limitations

• Only one-dimensional shape functions are modeled; pairwise interaction extension (GP-NA2M) is described but not evaluated.
• Predefined random features (S=100) cannot be tuned, so GP-NAM is not expected to clearly outperform learned NAM or NBM models.
• Multiclass classification is not considered.

Experimental setup

Datasets: CA Housing FICO MIMIC-II MIMIC-III Credit GMSC Click Microsoft Year Yahoo Churn Adult

Benchmarks: MIMIC-II MIMIC-III Credit GMSC Adult Churn FICO LCD Bikeshare Click Microsoft Yahoo

Metrics: AUC error rate MSE RMSE parameter count

Baselines: Linear EBM NAM NODE-GAM NBM MLP XGBoost

Builds on: neural additive models neural basis models random Fourier features explainable boosting machines

Closest papers by meaning

• Additive Gaussian Processes Revisited (ICML 2022 · similarity 0.88)
• Spherical Inducing Features for Orthogonally-Decoupled Gaussian Processes (ICML 2023 · similarity 0.87)
• Randomly Projected Additive Gaussian Processes for Regression (ICML 2020 · similarity 0.86)
• NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning (ICLR 2022 · similarity 0.86)
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation (ICML 2016 · similarity 0.85)
• Improving Neural Additive Models with Bayesian Principles (ICML 2024 · similarity 0.85)