Research map / Learning theory and algorithms / Prediction and decision losses
Minimal Variance Model Aggregation: A principled, non-intrusive, and versatile integration of black box models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Minimal Empirical Variance Aggregation (MEVA), a non-intrusive pointwise linear aggregation of black-box models that learns softmax weights by regressing log-error variances, and proves it converges faster than minimal error aggregation (MEEA).
Problem
Combining predictions from heterogeneous black-box models (ML methods and numerical PDE solvers) for the same quantity of interest, where direct empirical error minimization (MEEA) can fail or generalize poorly in scarce-data regimes.
Approach
Model the aggregation as a best linear unbiased estimate: assume each model is unbiased with error covariance A(x), so weights are the normalized inverse-covariance (softmax of negative log-variances), and learn the log-variance regressors from squared residuals on held-out data, rather than interpolating targets directly.
Key results
- On Boston housing, MEVA with kernel improves on the best aggregated model (Gradient Boosting) by 7% in MSE and the best overall model by 2%.
- On the Laplace equation, the aggregate reaches geometric mean MSE of 10^-6.282 versus 10^-5.523 for the best solver (FDM), roughly one order of magnitude better.
- On Burgers equation, the aggregate reaches geometric mean MSE of 10^-3.106 versus 10^-2.734 for the best solver (Riemann).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| aggregate the PDE solvers with MEVA | Laplace equation solver ensemble · Laplace equation | Geometric mean of MSE (log10): -5.523 -> -6.282 | Helped |
| aggregate the PDE solvers with MEVA | Burger's equation solver ensemble · Burger's equation | Geometric mean of MSE (log10): -2.568 -> -3.106 | Helped |
| aggregate the models with MEVA instead of using the best individual model | Boston housing model ensemble · Boston housing | MSE relative improvement: 7% | Helped |
| aggregate the models with MEVA instead of using the best overall model | Boston housing model ensemble · Boston housing | MSE relative improvement: 2% | Helped |
| use variance minimization rather than empirical error minimization (ablation) | DNN aggregation · Boston housing | MSE | Helped |
| replace the log-error loss with direct MSE loss (ablation) | FNO aggregation on Laplace equation · Laplace equation | MSE | Hurt |
| use an empirical eigenbasis instead of the identity basis for error covariance (ablation) | MEVA · PDE solver aggregation | accuracy | No clear effect |
| apply bias correction (ablation) | MEVA · model aggregation | performance | No clear effect |
| minimize empirical error directly rather than estimate model errors with MEVA (ablation) | model aggregation · Pathological example: a dubious trend | prediction error | Hurt |
| use uniform averaging instead of covariance-informed aggregation | 100 GP-based PDE models · Laplace equation | MSE | Hurt |
| use uniform averaging instead of MEVA | Laplace equation solver ensemble · Laplace equation | MSE | Hurt |
| use MEVA instead of the mean | Laplace equation solver ensemble · Laplace equation | MSE | Helped |
Surprising findings
- Directly training an FNO aggregator with MSE loss collapses to selecting the single best model (alpha approx (1,0,...,0)) instead of combining models.
- On the Boston housing sanity check, the simple mean performs well, and MEEA with a DNN performs worse than the two best individual models aggregated.
Limitations
- Requires held-out unseen data to estimate model errors, reducing training data for models when data is scarce.
- No guarantee the aggregate outperforms the individual models or simple averaging in challenging tasks, at higher computational cost.
- Theory assumes models are unbiased estimators of the target; biased-model extension not found beneficial in practice.
Experimental setup
Datasets: Boston housing Laplace equation synthetic PDE solutions Burgers equation synthetic solutions
Benchmarks: Boston housing Laplace equation Burgers equation
Metrics: MSE geometric mean of MSE (log10) relative improvement r
Baselines: MEEA Gradient Boosting uniform averaging Fourier Neural Operator with direct MSE aggregation individual numerical solvers (FDM, spectral, GP, TVD, Riemann)
Builds on: Mixture of Experts Best linear unbiased estimator Fourier Neural Operator Gradient Boosting nested kriging
Closest papers by meaning
- The Bias-Variance Tradeoff in Data-Driven Optimization: A Local Misspecification Perspective (NeurIPS 2025 · similarity 0.84)
- Tilted Empirical Risk Minimization (ICLR 2021 · similarity 0.84)
- Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation (NeurIPS 2025 · similarity 0.84)
- Learning Latent Variable Models via Jarzynski-adjusted Langevin Algorithm (NeurIPS 2025 · similarity 0.83)
- Robust Distributed Gradient Aggregation Using Projections onto Gradient Manifolds (AAAI 2024 · similarity 0.83)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.