Research map / Architectures and efficiency / RNNs and normalization
Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that monotonic MLPs with non-negative or non-positive weights and alternating saturating activations are universal monotonic approximators, and proposes a weight-sign-based activation-switch parametrization that removes weight constraints and works with ReLU.
Problem
Monotonic MLPs built by constraining weights to be non-negative with bounded activations suffer from vanishing gradients and can only represent bounded functions; ReLU (convex) activations with non-negative weights can only represent convex monotone functions.
Approach
Proves that alternating left/right-saturating monotone activations (3 hidden layers, 4 layers overall) give universal monotonic approximation, shows non-positive weight constraints with a single saturating activation (e.g., ReLU) also yield universal approximation, and introduces a switch parametrization that splits W into W+ and W- and applies the activation to each term, removing the need for we
Key results
- On Heart Disease the proposed method reaches 94% ± 1% test accuracy vs 89% for the best baseline (Non-Neg-DNN and Constrained MNN).
- On AutoMPG the proposed method achieves 7.34 ± 0.46 test MSE, lower than Scalable MNN (7.44 ± 1.20) and other baselines.
- On Blog Feedback it achieves 0.149 ± 0.001 test RMSE, the best among the compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the activation-switch formulation instead of the strongest listed baseline | Monotonic MLP · COMPAS test | accuracy: 69.3% ± 0.1% -> 69.5% ± 0.1% | Helped |
| Use the activation-switch formulation instead of the strongest listed baseline | Monotonic MLP · Blog Feedback test | RMSE: 0.150 ± 0.001 -> 0.149 ± 0.001 | Helped |
| Use the activation-switch formulation instead of the strongest listed baseline | Monotonic MLP · Loan Defaulter test | accuracy: 65.4% ± 0.1% vs 65.4% ± 0.1% | No clear effect |
| Use the activation-switch formulation instead of the strongest listed baseline | Monotonic MLP · AutoMPG test | MSE: 7.44 ± 1.20 -> 7.34 ± 0.46 | Helped |
| Use the activation-switch formulation instead of the strongest listed baseline | Monotonic MLP · Heart Disease test | accuracy: 90% ± 2% -> 94% ± 1% | Helped |
| Use non-negative weights with ReLU instead of activation switching (ablation) | Monotonic MLP on f(x)=cos(x)+x · Synthetic function approximation | approximation quality | Hurt |
| Use non-negative weights with sigmoid instead of activation switching (ablation) | Monotonic MLP on f(x)=cos(x)+x · Synthetic function approximation | optimization speed | Hurt |
| Use sigmoid activation with non-negative weights instead of activation switching (ablation) | Monotonic MLP on f(x)=cos(x)+x · Synthetic function approximation, 4 layers | average gradient magnitude: 0.010 vs 0.0019 | Hurt |
| Use sigmoid activation with non-negative weights instead of activation switching (ablation) | Monotonic MLP on f(x)=cos(x)+x · Synthetic function approximation, 10 layers | average gradient magnitude: 1.259 vs 0.00099 | Hurt |
| Use ReLU activation with non-negative weights instead of activation switching (ablation) | Monotonic MLP on f(x)=cos(x)+x · Synthetic function approximation, 4 layers | average gradient magnitude: 3.54 vs 0.010 | Mixed |
| Use ReLU activation with non-negative weights instead of activation switching (ablation) | Monotonic MLP on f(x)=cos(x)+x · Synthetic function approximation, 10 layers | average gradient magnitude: 3311.00 vs 1.259 | Mixed |
| Replace ReLU with ReLU6 (ablation) | Monotonic MLP with non-negative weights · Synthetic function approximation | gradient magnitude | Hurt |
Surprising findings
- Non-positive weight constraints are more expressive than non-negative ones: an MLP with ReLU and non-positive weights is a universal monotonic approximator, whereas non-negative ReLU networks only represent convex functions.
- ReLU6 monotonic MLPs with non-negative weights can become completely dead due to zero gradients in the saturated region.
Limitations
- Theory covers only saturating monotone activations; non-saturating ones such as Leaky-ReLU remain open.
- Hyperparameters for the proposed method were largely taken from prior work, with limited tuning on BlogFeedback; initialization still deviates from standard MLPs.
Experimental setup
Datasets: COMPAS Blog Feedback Loan Defaulter AutoMPG Heart Disease
Benchmarks: COMPAS Blog Feedback Loan Defaulter AutoMPG Heart Disease
Metrics: test accuracy test RMSE test MSE
Baselines: XGBoost Deep Lattice Network Min-Max Net Certified Constrained MNN Scalable MNN Expressive MNN Non-Neg-DNN
Builds on: Deep Lattice Networks Certified Monotonic Networks Constrained Monotonic Neural Networks Min-Max Networks
Closest papers by meaning
- Constrained Monotonic Neural Networks (ICML 2023 · similarity 0.92)
- Unconstrained Monotonic Neural Networks (NeurIPS 2019 · similarity 0.89)
- Smooth Min-Max Monotonic Networks (ICML 2024 · similarity 0.89)
- Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach (NeurIPS 2025 · similarity 0.87)
- Convex Formulations for Training Two-Layer ReLU Neural Networks (ICLR 2025 · similarity 0.85)
- Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.