Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Stability-Informed Initialization of Neural Ordinary Differential Equations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a stability-informed initialization (SII) for neural ODEs that samples linear-system eigenvalues inside the solver's stability region and builds feedforward weights whose linearization has those eigenvalues, improving training and prediction across several benchmarks.
Problem
Neural ODEs trained with fixed-step explicit solvers often have default-initialized linearized poles that lie outside the solver's stability region or near the origin, making training slow and prediction unstable, especially for long horizons.
Approach
Initialize a feedforward network so that its linearization has eigenvalues sampled (by rejection sampling) inside the stability region of the chosen Runge-Kutta solver for the given step size, using block-structured weights factorized as products of matrices with random orthogonal matrices inserted to keep weights dense and to break symmetry.
Key results
- On sequential pixel classification, SII improved accuracy from 97.9% to 99.0% on sMNIST, 90.4% to 94.4% on spMNIST, and 67.6% to 73.3% on sCIFAR.
- On latent dynamics benchmarks, SII reduced normalized MSE on 3D Room from 0.588 to 0.176 and on Mass Spring from 0.102 to 0.066.
- On multivariate time-series forecasting, SII reduced MSE on Air Quality from 0.57 to 0.22 and on Human Activity from 0.29 to 0.11.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use stability-informed initialization instead of default initialization | Neural ODE · sMNIST | accuracy: 97.9 ± 0.20 -> 99.0 ± 0.09 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · spMNIST | accuracy: 90.4 ± 0.41 -> 94.4 ± 0.26 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · sCIFAR | accuracy: 67.6 ± 0.15 -> 73.3 ± 0.32 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · Mass Spring | normalized MSE: 0.102 ± 0.015 -> 0.066 ± 0.008 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · Double Pendulum | normalized MSE: 0.084 ± 0.017 -> 0.061 ± 0.002 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · Molecular Dynamics | normalized MSE: 0.151 ± 0.036 -> 0.106 ± 0.011 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · 3D Room | normalized MSE: 0.588 ± 0.102 -> 0.176 ± 0.055 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · Combustion Engine | MSE: 0.04 ± 0.019 -> 0.01 ± 0.005 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · Human Activity | MSE: 0.29 ± 0.006 -> 0.11 ± 0.003 | Helped |
| use stability-informed initialization instead of default initialization | Neural ODE · Air Quality | MSE: 0.57 ± 0.103 -> 0.22 ± 0.020 | Helped |
Surprising findings
- Default PyTorch initialization places linearized neural ODE poles densely around the origin and sometimes in the right half-plane, and learned models shift poles only slightly left, struggling with fast dynamics.
Limitations
- Method is specific to feedforward networks; extension to other linear-operation models such as CNNs is only suggested.
- Hidden dimension of each layer must be at least the number of dynamic states to preserve eigenvalues.
- Linearized stability does not guarantee nonlinear or time-varying stability; analysis focuses on fixed-step explicit Runge-Kutta solvers.
Experimental setup
Datasets: MNIST CIFAR-10 pMNIST Latent dynamics benchmark datasets (Mass Spring, Double Pendulum, Molecular Dynamics, 3D Room) Combustion engine data Human Activity dataset Air Quality dataset Synthetic teacher-student data
Benchmarks: sMNIST spMNIST sCIFAR Mass Spring Double Pendulum Molecular Dynamics 3D Room Combustion Engine Human Activity Air Quality
Metrics: accuracy normalized MSE MSE test loss
Baselines: Default PyTorch initialization
Builds on: Neural ODE PyTorch default initialization Runge-Kutta methods S4
Closest papers by meaning
- Learning Efficient and Robust Ordinary Differential Equations via Invertible Neural Networks (ICML 2022 · similarity 0.88)
- How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective (ICML 2024 · similarity 0.88)
- Global Convergence in Neural ODEs: Impact of Activation Functions (ICLR 2025 · similarity 0.87)
- Semi-Implicit Neural Ordinary Differential Equations (AAAI 2025 · similarity 0.86)
- Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data (ICLR 2024 · similarity 0.85)
- Improving Neural ODE Training with Temporal Adaptive Batch Normalization (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.