Research map / Learning theory and algorithms / Kernels and regression theory
Improving Neural Network Generalization on Data-Limited Regression with Doubly-Robust Boosting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRBoost, a post-training method that refits a fully trained neural network's discriminative layer with partial least squares regression and tunes the layer before it with a gradient-free metaheuristic optimizer to reduce generalization error on data-limited time-series regression.
Problem
Improving generalization of trained neural networks in data-limited regression, where the model selection trade-off between training error and generalization gap limits performance and SGD suffers from multicollinearity and surrogate-loss gaps.
Approach
Post-optimize an SGD-pretrained network: replace the discriminant layer with a PLSR statistical learner to shrink capacity and reduce the generalization gap, and optimize the preceding layer with metaheuristic (zero-order) search directly on the target metric to reduce training error without gradients.
Key results
- DRBoost on a pretrained MLP raises R2 on TRAFFIC from 0.835 to 0.853 (CS optimizer) and on ELECTRICITY from 0.828 to 0.852.
- On PEMS04, DRBoost improves GraphWaveNet R2 from -1.241 to 0.868 by calibrating an overfitted network.
- DRBoost yields small or no gains on easy datasets such as PEMSBAY (R2 0.949 to 0.951) and can hurt WaveNet on PEMSBAY.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DRBoost with PSO | MLP 64-64-64-1 · METR-LA | R2: 0.945 -> 0.955 | Helped |
| Apply DRBoost with PSO | MLP 64-64-64-1 · PEMSBAY | R2: 0.949 -> 0.951 | Helped |
| Apply DRBoost with PSO | MLP 64-64-64-1 · PEMS04 | R2: 0.889 -> 0.897 | Helped |
| Apply DRBoost with cuckoo search | MLP 64-64-64-1 · TRAFFIC | R2: 0.835 -> 0.853 | Helped |
| Apply DRBoost with cuckoo search | MLP 64-64-64-1 · ELECTRICITY | R2: 0.828 -> 0.852 | Helped |
| Add the PLSR statistical learner (ablation) | DRBoost with MLP · ELECTRICITY | R2: 0.836±0.009 -> 0.844±0.007 | Helped |
| Add the PLSR statistical learner (ablation) | DRBoost with MLP · METR-LA | R2: 0.953±0.004 -> 0.955±0.001 | Helped |
| Add the zero-order optimizer (ablation) | DRBoost with MLP · METR-LA | R2: 0.944±0.000 -> 0.955±0.001 | Helped |
| Add the zero-order optimizer (ablation) | DRBoost with MLP · PEMS04 | R2: 0.893±0.000 -> 0.897±0.000 | Helped |
Surprising findings
- DRBoost degrades some backbones, e.g., WaveNet on PEMSBAY and TCN on METR-LA, contrary to the expectation of uniform gains.
- Large numbers of latent bases (ds >= 48) hurt performance, so the statistical learner capacity must be tuned to a middle range.
Limitations
- Statistical learner (PLSR) complexity grows with dataset size, limiting large-scale deployment.
- No guarantee that boosted models outperform original networks; some backbones degrade.
- Evaluated only on time-series regression benchmarks.
Experimental setup
Datasets: METR-LA PEMSBAY PEMS04 TRAFFIC ELECTRICITY
Benchmarks: METR-LA PEMSBAY PEMS04 TRAFFIC ELECTRICITY
Metrics: R2 MAE RMSE
Baselines: PSO-free base MLP BAS CS FC-LSTM GRU LSTNET TCN WaveNet GraphWaveNet
Builds on: partial least squares regression particle swarm optimization cuckoo search beetle antennae search
Closest papers by meaning
- Robust Minimax Boosting with Performance Guarantees (NeurIPS 2025 · similarity 0.85)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.85)
- Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data (ICLR 2024 · similarity 0.85)
- Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression (NeurIPS 2024 · similarity 0.85)
- Universal generalization guarantees for Wasserstein distributionally robust models (ICLR 2025 · similarity 0.84)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.