Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Sampling-based Multi-dimensional Recalibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HDR calibration and a sampling-based recalibration method for multi-dimensional probabilistic regression that accounts for dependence across output dimensions.
Problem
Calibrating multi-dimensional probabilistic regression forecasts: existing recalibration and calibration metrics treat each output dimension separately, missing dependence across dimensions.
Approach
Defines HDR calibration using highest density regions of the predictive PDF as the projection, then recalibrates via an isotonic map in projected space and re-samples predictive samples to match the recalibrated projection distribution, optionally adjusting Gaussian predictions for bias, scale, and correlation first.
Key results
- HDR recalibration yields the lowest energy score on all 5 mulan datasets and all 3 fusion models (e.g., scpf -0.79 vs 3.97 pre-hoc).
- On the demand simulation, HDR recalibration reduces total loss to 865.34 from 916.81 pre-hoc and 910.82 with SD recalibration.
- SD recalibration improves SD-ECE but does not improve energy score, arguing SD-ECE is inadequate for multi-dimensional predictions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HDR recalibration instead of the pre-hoc distribution | PNN · scpf, rf1, rf2, scm1d, scm20d | Energy score: scpf: 3.97 -> −0.79; rf1: 0.11 -> 0.08; rf2: 1.06 -> 1.04; scm1d: 1.13 -> 0.36; scm20d: 1.29 -> 0.81 | Helped |
| Apply HDR recalibration instead of the pre-hoc distribution | Fusion dynamics models · Fusion1, Fusion2, Fusion3 | Energy score: Fusion1: 2.48 -> −3.73; Fusion2: 1.95 -> −1.85; Fusion3: 4.79 -> −5.01 | Helped |
| Apply HDR recalibration instead of the pre-hoc distribution | PNN · scpf, rf1, rf2, scm1d, scm20d | HDR-ECE: scpf: 0.30 -> 0.07; rf1: 0.08 -> 0.01; rf2: 0.07 -> 0.09; scm1d: 0.48 -> 0.04; scm20d: 0.48 -> 0.04 | Mixed |
| Apply HDR recalibration instead of the pre-hoc distribution | Fusion dynamics models · Fusion1, Fusion2, Fusion3 | HDR-ECE: Fusion1: 0.34 -> 0.13; Fusion2: 0.45 -> 0.05; Fusion3: 0.35 -> 0.09 | Helped |
| Apply SD recalibration instead of the pre-hoc distribution | PNN · scpf, rf1, rf2, scm1d, scm20d | Energy score: scpf: 3.97 -> 4.00; rf1: 0.11 -> 0.11; rf2: 1.06 -> 1.06; scm1d: 1.13 -> 1.13; scm20d: 1.29 -> 1.31 | No clear effect |
| Apply SD recalibration instead of the pre-hoc distribution | Fusion dynamics models · Fusion1, Fusion2, Fusion3 | Energy score: Fusion1: 2.48 -> 2.48; Fusion2: 1.95 -> 1.93; Fusion3: 4.79 -> 5.03 | Mixed |
| Apply SD recalibration instead of the pre-hoc distribution | PNN · scpf, rf1, rf2, scm1d, scm20d | SD-ECE: scpf: 0.15 -> 0.02; rf1: 0.05 -> 0.01; rf2: 0.05 -> 0.01; scm1d: 0.11 -> 0.02; scm20d: 0.11 -> 0.02 | Helped |
| Apply SD recalibration instead of the pre-hoc distribution | Fusion dynamics models · Fusion1, Fusion2, Fusion3 | SD-ECE: Fusion1: 0.11 -> 0.05; Fusion2: 0.17 -> 0.09; Fusion3: 0.09 -> 0.03 | Mixed |
| Use HDR recalibration instead of no recalibration | Favorita demand forecasting model · Favorita demand simulation (77 test days) | Total loss: 916.81 -> 865.34 | Helped |
| Use SD recalibration instead of no recalibration | Favorita demand forecasting model · Favorita demand simulation (77 test days) | Total loss: 916.81 -> 910.82 | Helped |
Surprising findings
- Per-dimension calibration (SD-ECE) can look good while the joint distribution is miscalibrated, and SD recalibration does not improve the energy score.
Limitations
- Adjustment procedure is specific to Gaussian predictive distributions; complex predictive distributions may need more sophisticated adjustments.
- Projection function is restricted to the predictive PDF; other projections are left for future work.
Experimental setup
Datasets: mulan benchmark (scpf, rf1, rf2, scm1d, scm20d) DIII-D tokamak plasma dynamics data Corporacion Favorita Kaggle sales data
Benchmarks: mulan benchmark regression datasets DIII-D plasma fusion dynamics models Corporacion Favorita demand simulation
Metrics: energy score HDR-ECE SD-ECE accumulated under/over-stock loss
Baselines: Pre-hoc (no recalibration) SD recalibration (Kuleshov et al. 2018) Izbicki et al. 2022 conformal HPD
Builds on: Kuleshov et al. 2018 isotonic recalibration copula calibration (Ziegel & Gneiting) highest predictive density energy score
Closest papers by meaning
- Multivariate Latent Recalibration for Conditional Normalizing Flows (NeurIPS 2025 · similarity 0.85)
- Truthfulness of Calibration Measures (NeurIPS 2024 · similarity 0.83)
- Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implications (ICML 2024 · similarity 0.83)
- Online Bayesian Calibration under Gradual and Abrupt System Changes (NeurIPS 2026 · similarity 0.83)
- Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing (ICLR 2024 · similarity 0.83)
- PAC-Bayes Analysis for Recalibration in Classification (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.