Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Improving Neural Additive Models with Bayesian Principles

ICML 2024 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Approximate inference in Bayesian neural networks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Applies a linearized Laplace approximation with block-diagonal Kronecker-factored GGN to neural additive models, yielding per-feature credible intervals, marginal-likelihood-based feature selection, and MI-based pairwise interaction selection (LA-NAM).

Problem

Neural additive models are interpretable but lack calibrated uncertainty, principled feature selection, and mechanisms for detecting feature interactions.

Approach

Treat each feature network as an independent block in a linearized Laplace posterior so that predictive variance decomposes additively per feature, use the lower bound on the log-marginal likelihood with per-network precisions as ARD for implicit feature selection, and rank feature pairs by posterior mutual information from a last-layer Laplace approximation to add joint feature networks.

Key results

• On MIMIC-III ICU mortality, LA-NAM achieves 79.6 AUROC and 0.264 NLL versus NAM's 77.6 AUROC and 0.274 NLL; with 10 selected interactions LA-NAM reaches 80.2 AUROC, close to LightGBM's 80.6.
• On HiRID, LA-NAM improves NLL to 0.174 from NAM's 0.228 with comparable AUPRC.
• On MIMIC-III, LA-NAM reduces ECE from 0.019 (NAM) to 0.009.

Measured effects reported in the paper

Surprising findings

• Plain ReLU NAM with GELU-activated feature networks performs worse, and the paper notes that NAM ensembles of ReLU/GELU networks collapse in function-space diversity.
• OAK-GP with all pairwise interactions (OAK-GP*) degrades substantially on MIMIC-III (AUROC 71.7 vs 79.9 first-order).

Limitations

• Interaction selection is restricted to second order because the search space grows exponentially for higher orders.
• Laplace approximation assumes block-diagonal (feature-independent) posterior, and the true posterior may have strong dependencies between feature networks, which the method only partly addresses via a last-layer approximation.
• MIMIC-III preprocessing relied on a pre-processed dataset provided by the authors of prior work; medical plausibility checks rely on personal communication with experts.

Experimental setup

Datasets: UCI regression benchmarks UCI classification benchmarks MIMIC-III HiRID synthetic additive regression dataset

Benchmarks: UCI regression UCI classification MIMIC-III ICU mortality HiRID ICU mortality

Metrics: negative log-likelihood AUROC AUPRC ECE root Brier score SKCE quantile calibration error

Baselines: NAM GAM OAK-GP LightGBM EBM deep ensembles MC-Dropout

Builds on: neural additive models linearized Laplace approximation KFAC-GGN automatic relevance determination

Closest papers by meaning

• Neural Additive Models: Interpretable Machine Learning with Neural Nets (NeurIPS 2021 · similarity 0.88)
• Projected Neural Additive Models as Universal Approximators (NeurIPS 2026 · similarity 0.86)
• Gaussian Process Neural Additive Models (AAAI 2024 · similarity 0.85)
• Do Bayesian Neural Networks Actually Behave Like Bayesian Models? (ICML 2025 · similarity 0.82)
• Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks (ICML 2025 · similarity 0.82)
• Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models (CVPR 2024 · similarity 0.82)