Research map / Learning theory and algorithms / Kernels and regression theory
TMPNN: High-Order Polynomial Regression Based on Taylor Map Factorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TMPNN, a Taylor-map-based polynomial neural network that iteratively reapplies a shared-weight order-k polynomial map p times to reach order k^p regression for multi-target tasks, with an ODE interpretation.
Problem
High-order polynomial regression overfits and extrapolates poorly, and its parameter count explodes with order; multi-target regression is usually handled by separate single-output models.
Approach
Extend the input with zero-initialized target dimensions and apply the same Taylor map (shared polynomial weights) p times, so the output is a polynomial of order k^p with far fewer free parameters; the map is interpreted as an Euler discretization of an ODE system, and all targets are fit jointly.
Key results
- On the Feynman benchmark the proposed model (k=3, p=5, no hyperparameter search) is reported as best-performing with the smallest confidence intervals across noise levels.
- On Yacht Hydrodynamics extrapolation splits TMPNN achieves R2 of 0.92 on fn>0.4 and 0.90 on rr>12, the best in each split, versus 0.71 for TabNet on fn>0.4.
- On UCI datasets TMPNN is comparable to CatBoost (e.g., Sarcos MSE 1.40 vs CatBoost 1.71), with CatBoost best overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TMPNN instead of the strongest reported baseline | Regression models · Yacht Hydrodynamics, fn>0.4 extrapolation | R2: 0.71 -> 0.92 | Helped |
| Use TMPNN instead of the strongest reported baseline | Regression models · Yacht Hydrodynamics, rr>12 extrapolation | R2: 0.83 -> 0.90 | Helped |
| Use TMPNN instead of the strongest reported baseline | Regression models · Yacht Hydrodynamics, cp>0.5 and b/d>4.1 extrapolation | R2: 0.99 -> 0.99 | No clear effect |
| Use TMPNN instead of CatBoost | Regression models · 26 UCI datasets with R2>0.5 | average R2: 0.88 ± 0.14 -> 0.88 ± 0.14 | No clear effect |
| Use TMPNN instead of CatBoost | Regression models · Friedman-1, varying sample counts | performance | Helped |
| Use TMPNN instead of CatBoost | Regression models · Friedman-1, added unimportant features | performance | Mixed |
| Use TMPNN instead of other evaluated models | Regression models · Feynman benchmark, three noise levels | accuracy (R2 > 0.999) | Helped |
| Use identity initialization instead of zero initialization (ablation) | TMPNN · Airfoil, k=2, p=5 | R2: 0.66±0.02 -> 0.87±0.01 | Helped |
| Use identity initialization instead of zero initialization (ablation) | TMPNN · Airfoil, k=3, p=5 | R2: -1.07±2.44 -> 0.92±0.01 | Helped |
| Increase steps from p=5 to p=10 with k=3 (ablation) | Identity-initialized TMPNN · Airfoil | R2: 0.92±0.01 -> 0.89±0.05 | Hurt |
| Increase steps from p=5 to p=20 with k=3 (ablation) | Identity-initialized TMPNN · Airfoil | R2: 0.92±0.01 -> 0.87±0.11 | Hurt |
| Increase polynomial-map order from k=3 to k=4 at p=5 (ablation) | Identity-initialized TMPNN · Airfoil | R2: 0.92±0.01 -> 0.80±0.12 | Hurt |
Surprising findings
- Increasing the number of layers p beyond 5 decreased performance, which the authors attribute to the optimizer since weight initialization for higher p was not used.
- TMPNN was the only model whose error decreased when adding a small number of unimportant features to Friedman-1.
Limitations
- Higher p can degrade optimization without proper weight initialization; the initialization approach is not experimentally validated.
- Hyperparameters for the Feynman benchmark were not optimized; CatBoost is best on most UCI datasets.
- Theoretical properties (regularization effect of p, convergence) are not fully examined; limitations for normalization, categorical features and differentiability are open.
Experimental setup
Datasets: UCI regression datasets Sarcos Feynman Symbolic Regression Database Friedman-1 UCI Yacht Hydrodynamics
Benchmarks: UCI regression datasets Feynman Symbolic Regression Database Friedman-1 UCI Yacht Hydrodynamics extrapolation splits
Metrics: MSE R2 accuracy (R2 > 0.999)
Baselines: CatBoost DNNR TabNet XGBoost MLP KNN Random Forest polynomial regression
Builds on: Taylor maps polynomial neural networks Adamax
Closest papers by meaning
- Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operators (NeurIPS 2024 · similarity 0.84)
- Learning Hierarchical Polynomials with Three-Layer Neural Networks (ICLR 2024 · similarity 0.84)
- GINN-LP: A Growing Interpretable Neural Network for Discovering Multivariate Laurent Polynomial Equations (AAAI 2024 · similarity 0.84)
- Towards Learning High-Precision Least Squares Algorithms with Sequence Models (ICLR 2025 · similarity 0.84)
- Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit (NeurIPS 2024 · similarity 0.83)
- Learning Polynomial Problems with $SL(2, \mathbb{R})$-Equivariance (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.