Research map / Learning theory and algorithms / Prediction and decision losses
Smooth Min-Max Monotonic Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes smooth min-max (SMM) monotonic networks, replacing the min/max in min-max networks with monotone LogSumExp functions to avoid silent neurons, and shows competitive or better monotone regression performance than MM, lattices, LMN, XGBoost and isotonic regression.
Problem
Min-max monotonic networks trained by gradient descent often get stuck because min/max operations give zero partial derivatives for silent neurons, leading to underfitting and poor local optima.
Approach
Replace the max and min in the min-max architecture with scaled LogSumExp functions (smooth, strictly increasing, monotone), so every neuron receives gradient; the module keeps MM's asymptotic approximation guarantee and adds only one learnable scale parameter beta.
Key results
- On univariate synthetic functions SMM had the lowest median test MSE in all three settings (e.g., 0.01e-3 vs 0.10e-3 for MM on x^2), with significant differences (Wilcoxon p<0.001).
- MM networks had on average only 2.8 active neurons out of 36 after training on test data, while SMM had 31.6 active on average.
- SMM64 achieved the best results on several comparisons, e.g., 69.5% test accuracy on COMPAS and 7.51 MSE on Auto MPG, outperforming reported LMN, CMNN and COMET results in all listed benchmarks per the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MM min/max with smooth LogSumExp min/max | MM · Univariate functions x², √x, and logistic | test MSE: 0.10 -> 0.01; 0.32 -> 0.02; 0.22 -> 0.01 | Helped |
| Replace MM min/max with smooth LogSumExp min/max | MM · Multivariate monotone functions d=2, 4, and 6 | test MSE | Helped |
| Use early stopping | XGBoost · Univariate and multivariate monotone functions | test MSE | No clear effect |
| Add a 64-unit auxiliary network for unconstrained inputs (ablation) | SMM · Partial-monotone UCI tasks | CV MSE: 0.25 -> 0.14; 0.61 -> 0.24; 1.02 -> 1.03; 1.79 -> 1.78 | Mixed |
| Vary the initial ln β and K (ablation) | SMM · Univariate functions x², √x, and logistic | test MSE: 0.0293; 0.0246; 0.0269; 0.0183; 0.0125 | Mixed |
Surprising findings
- On UCI partial monotone tasks, XGBoost beat SMM on most targets, so SMM was not uniformly best.
- The default beta initialization (ln beta=-1, K=6) was suboptimal across settings, though performance was robust to initial beta overall.
Limitations
- Results on larger real-world benchmarks use only three trials and an oracle-based evaluation protocol from prior work, so differences are not statistically established.
- Isolated SMM with a single layer of adaptive weights may not suffice for complex real-world tasks; an auxiliary network was needed.
- All artificial benchmark functions were smooth, so behavior on non-smooth targets is untested.
Experimental setup
Datasets: synthetic univariate functions x^2, sqrt(x), logistic synthetic multivariate polynomial functions d=2,4,6 UCI Energy Efficiency UCI QSAR UCI Concrete COMPAS BlogFeedback LoanDefaulter ChestXRay Heart Disease Auto MPG
Benchmarks: synthetic univariate functions synthetic multivariate functions UCI Energy UCI QSAR UCI Concrete COMPAS BlogFeedback LoanDefaulter ChestXRay Heart Disease Auto MPG
Metrics: mean squared error test accuracy RMSE training time
Baselines: MM Isotonic regression XGBoost Hierarchical lattice layers Lipschitz monotonic networks COMET Certified monotonic networks CMNN Crystal lattice ensembles
Builds on: min-max networks hierarchical lattice layers Lipschitz monotonic networks constrained monotonic neural networks
Closest papers by meaning
- Scalable Monotonic Neural Networks (ICLR 2024 · similarity 0.89)
- Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations (ICML 2025 · similarity 0.89)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.87)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.86)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.86)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.