Research map / Probabilistic and scientific modeling / Molecules and proteins
Has the Deep Neural Network learned the Stochastic Process? An Evaluation Viewpoint
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fidelity to Stochastic Process (F2SP), an evaluation criterion targeting the Statistic-GT of stochastic complex systems, and argues from theory and agent-based simulations that ECE, unlike AUC-PR and MSE, can test F2SP using only observed outcomes.
Problem
Standard DNN evaluation metrics for forecasting stochastic complex systems compare predictions to a single observed outcome, so they cannot tell whether the model has learned the underlying stochastic process.
Approach
Defines Statistic-GT as the per-cell probability distribution across Monte Carlo simulations and F2SP as fidelity to it; shows formally that ECE's expected value is zero for a perfect predictor of Statistic-GT, so a low ECE is a necessary condition for F2SP, whereas classification metrics and proper scoring rules are not.
Key results
- ECE shows clear diagonal behavior in train-vs-test S-Level matrices (low ECE when S-Levels match), whereas AUC-PR shows no such trend and MSE only partial trends.
- On NDWS, Precision, Recall and AUC-PR degrade as Dice coefficient decreases, while ECE improves for low-DC (abrupt-change) cases, indicating ECE measures a complementary aspect of performance.
- ECE remains low and stable over long prediction horizons for the DNN trained on the matching S-Level, while AUC-PR and MSE decline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate with ECE rather than AUC-PR | DNNs trained and tested across S-Levels · Forest Fire, Host-Pathogen, and Stock Market | alignment of scores with train-test S-Level match | Helped |
| Evaluate with AUC-PR as train-test S-Level mismatch increases | DNNs trained and tested across S-Levels · Forest Fire, Host-Pathogen, and Stock Market | AUC-PR | Hurt |
| Evaluate with ECE when train and test S-Levels match | DNNs trained and tested across S-Levels · Forest Fire, Host-Pathogen, and Stock Market | ECE | Helped |
| Evaluate with MSE rather than ECE across S-Levels | DNNs trained and tested across S-Levels · Forest Fire, Host-Pathogen, and Stock Market | sensitivity to train-test S-Level match | Mixed |
| Train at S-Level 20 rather than S-Level 10 | ConvLSTM-CA · Forest Fire, test S-Level 20 | ECE over prediction horizon | Helped |
| Train at S-Level 10 rather than S-Level 20 | ConvLSTM-CA · Forest Fire, test S-Level 20 | AUC-PR ability to distinguish stochastic mismatch | No clear effect |
| Evaluate with AUC-PR or MSE over a longer prediction horizon | DNNs trained at S-Levels 10 and 20 · Forest Fire, test S-Level 20 | score over prediction horizon | Hurt |
| Evaluate with ECE rather than classification metrics as Dice Coefficient decreases | Conv-AE · Next Day Wildfire Spread | performance across Dice Coefficient ranges | Mixed |
| Remove the spatial bottleneck (ablation) | ConvLSTM-CA · Deterministic Forest Fire | forecast fidelity to deterministic evolution | Helped |
| Add a spatial bottleneck (ablation) | ConvLSTM-CA · Deterministic Forest Fire | forecast fidelity to deterministic evolution | Hurt |
Surprising findings
- On the NDWS wildfire dataset, ECE improves as the Dice coefficient decreases while classification metrics degrade, contrary to intuition that harder cases yield worse scores across all metrics.
- AUC-PR fails to distinguish the stochastically mismatched DNN from the matched one in the synthetic experiments.
Limitations
- ECE has lower discriminative power than classification metrics and proper scoring rules for ranking DNNs, since it lacks a refinement term.
- ECE requires a sufficient number of samples for reliable convergence (a test batch of 10 grids of 64x64 was used).
- Experiments focus on binary, discrete, multivariate grid prediction; regression tasks and next-day-only NDWS prediction are not covered.
Experimental setup
Datasets: Forest Fire simulations (NetLogo) Host-Pathogen simulations Stock Market simulations Next Day Wildfire Spread (NDWS)
Benchmarks: Forest Fire simulations Host-Pathogen simulations Stock Market simulations Next Day Wildfire Spread
Metrics: AUC-PR MSE ECE Precision Recall F1-score AUC-ROC BCE CRPS Energy Score
Baselines: AUC-PR MSE BCE CRPS Energy Score Precision Recall F1-score AUC-ROC
Builds on: ConvLSTM Conv-AE Expected Calibration Error Brier score decomposition
Closest papers by meaning
- Efficient Training of Neural Stochastic Differential Equations by Matching Finite Dimensional Distributions (ICLR 2025 · similarity 0.85)
- SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series (NeurIPS 2025 · similarity 0.84)
- High Rank Path Development: an approach to learning the filtration of stochastic processes (NeurIPS 2024 · similarity 0.84)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.84)
- In-Context Learning of Stochastic Differential Equations with Foundation Inference Models (NeurIPS 2025 · similarity 0.84)
- Neural Stochastic Flows: Solver-Free Modelling and Inference for SDE Solutions (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.