Research map / Learning theory and algorithms / Matrix and tensor methods
Tensor Product Neural Networks for Functional ANOVA Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ANOVA-TPNN, a tensor-product neural network with sum-to-zero basis functions that gives identifiable, stable functional ANOVA components trainable by SGD, with a universal approximation proof.
Problem
Neural networks for functional ANOVA models produce unstable component estimates because the decomposition of a function into components is not unique (unidentifiable), which undermines interpretation.
Approach
Replace each tensor-product basis function with a sigmoid-based basis neural network that is constructed to satisfy the sum-to-zero identifiability constraint exactly, so each component is uniquely defined; the parameter count does not grow exponentially with interaction order, and the model is trained by gradient descent with a universal approximation guarantee.
Key results
- ANOVA-T1PNN achieves stability score 0.012 on Calhousing versus 0.045 for NAM1M and 0.039 for NBM1M, and is best on 7 of 9 tabular datasets.
- ANOVA-T2PNN attains the best average prediction rank (3.11) among GA2M-family and black-box models on nine tabular datasets, tied with XGB.
- On synthetic functions, ANOVA-T2PNN reaches component-selection AUROC of 1.000 on f(1) versus 0.330 for NA2M and 0.522 for NB2M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ANOVA-T²PNN instead of NAM or NBM for component selection | ANOVA-T²PNN · Synthetic datasets f¹, f², and f³ | AUROC: 1.000 vs 0.330 and 0.522; 0.943 vs 0.311 and 0.481; 0.956 vs 0.381 and 0.477 | Helped |
| Use ANOVA-T¹PNN instead of NAM and NBM | GA¹M component estimation · Calhousing, Wine, Online, Abalone, FICO, Churn, Credit, Drybean; component stability | stability score: 0.012 vs 0.045 and 0.039; 0.011 vs 0.058 and 0.043; 0.031 vs 0.076 and 0.054; 0.008 vs 0.013 and 0.026; 0.035 vs 0.046 a | Helped |
| Use ANOVA-T²PNN instead of NAM and NBM | GA²M component estimation · Calhousing, Wine, Online, Abalone, FICO, Churn, Credit, Letter, Drybean; component stability | stability score: 0.035 vs 0.071 and 0.075; 0.049 vs 0.087 and 0.065; 0.052 vs 0.072 and 0.072; 0.028 vs 0.047 and 0.038; 0.048 vs 0.089 a | Helped |
| Use ANOVA-T²PNN rather than ANOVA-T¹PNN | ANOVA-TPNN prediction · Calhousing | RMSE: 0.614 -> 0.512 | Helped |
| Use ANOVA-T²PNN rather than ANOVA-T¹PNN | ANOVA-TPNN prediction · Wine | RMSE: 0.725 -> 0.704 | Helped |
| Use ANOVA-T²PNN rather than ANOVA-T¹PNN | ANOVA-TPNN prediction · Abalone | RMSE: 2.135 -> 2.087 | Helped |
| Use ANOVA-T²PNN rather than ANOVA-T¹PNN | ANOVA-TPNN prediction · Letter | AUROC: 0.900 -> 0.984 | Helped |
| Use ANOVA-T²PNN rather than ANOVA-T¹PNN | ANOVA-TPNN prediction · Drybean | AUROC: 0.995 -> 0.998 | Helped |
| Use ANOVA-T¹PNN rather than spline basis expansion | Calhousing prediction · Calhousing | RMSE: 0.636 -> 0.614 | Helped |
| Use ANOVA-T²PNN rather than spline basis expansion | Calhousing prediction · Calhousing | RMSE: 1.326 -> 0.512 | Helped |
| Share basis neural networks across components of the same cardinality | NBM-T¹PNN · Calhousing | stability score: 0.012 -> 0.009 | Helped |
| Share basis neural networks across components of the same cardinality | NBM-T¹PNN · Wine | stability score: 0.011 -> 0.017 | Hurt |
Surprising findings
- Spline-GAM with cubic B-splines underperforms ANOVA-TPNN on Calhousing in both RMSE and stability, attributed to linear extrapolation on outlier inputs.
- On the image experiment, a monotone-unconstrained model gave decreasing-expected concepts an opposite-direction main effect, which the authors argue is an interpretation pitfall.
Limitations
- Higher-order interactions remain computationally demanding for high-dimensional data.
- Interaction screening via NID is needed for high-dimensional datasets, since all second-order interactions are impractical.
Experimental setup
Datasets: Calhousing Wine Online Abalone FICO Churn Credit Letter Drybean Microsoft Yahoo Madelon
Benchmarks: Calhousing Wine Online Abalone FICO Churn Credit Letter Drybean Microsoft Yahoo Madelon
Metrics: RMSE AUROC stability score component selection AUROC
Baselines: NAM NBM NODE-GAM XGBoost DNN Spline-GAM
Builds on: Neural Additive Model Neural Basis Model tensor product basis expansion NODE-GAM Neural Interaction Detection
Closest papers by meaning
- Factor Augmented Tensor-on-Tensor Neural Networks (AAAI 2025 · similarity 0.82)
- Explainable Neural Networks with Guarantee: A Sparse Estimation Approach (AAAI 2025 · similarity 0.82)
- Attention-based Iterative Decomposition for Tensor Product Representation (ICLR 2024 · similarity 0.82)
- Equivariant Polynomial Functional Networks (ICML 2025 · similarity 0.82)
- Can Neural Networks Achieve Optimal Computational-statistical Tradeoff? An Analysis on Single-Index Model (ICLR 2025 · similarity 0.82)
- A Functional Extension of Semi-Structured Networks (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.