Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Semi-Implicit Neural Ordinary Differential Equations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes semi-implicit neural ODEs (SINODE) that split the right-hand side into nonlinear (explicit) and linear (implicit) parts, solved with IMEX-RK and trained with a discrete adjoint, for stiff neural ODE learning.
Problem
Neural ODEs trained with explicit solvers are limited by stability and become prohibitively expensive on stiff problems (graph diffusion, PDE dynamics), while fully implicit solvers require expensive nonlinear solves.
Approach
Partition the neural ODE right-hand side into a nonlinear NN part treated explicitly and a linear part treated implicitly via IMEX-RK, so each implicit stage reduces to a linear solve reusable across stages and mini-batches, with a reverse-accurate discrete adjoint for gradients.
Key results
- On GRAND graph learning, IMEX-RK2 needs 114 NFEs per epoch on Cora versus 43,908 for RK4 with step size 0.005.
- For KS equation, SINODE with IMEX-RK2 is about 47x faster than Dopri5 and about 33x faster than Crank-Nicolson to reach training loss 1e-3.
- For viscous Burgers at grid 512, IMEX-RK3 is about 6x faster than RK4 per epoch, rising to about 10x at grid 1024.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use semi-implicit IMEX-RK integration instead of explicit or implicit integration | GRAND · Cora, CoauthorCS, and Photo | NFEs per epoch | Helped |
| Use semi-implicit IMEX-RK integration instead of explicit or fully implicit integration | Kuramoto–Sivashinsky equation · KSE, grids 64 and 512 | training time: ∼47 times faster than the classical neural ODE with Dopri5; ∼33 times faster than the neural ODE with Crank--Nicolson | Helped |
| Use semi-implicit IMEX-RK integration instead of explicit or fully implicit integration | Viscous Burgers equation · Burgers, grid 512 | training time per epoch: ∼ 6X faster than RK4 | Helped |
| Use semi-implicit IMEX-RK integration instead of explicit or fully implicit integration | Viscous Burgers equation · Burgers, grid 1024 | training time per epoch: ∼ 10X | Helped |
| Use a time step of 0.2 with semi-implicit integration instead of explicit integration | Kuramoto–Sivashinsky equation · KSE, grid 512 | training feasibility: 0.2; 10^{-7} | Helped |
| Use lower-order IMEX-RK2 instead of higher-order IMEX-RK5 (ablation) | SINODE · KSE, grid 64 and grid 512 | training efficiency: ∼2.5 times; a factor of 5 | Helped |
| Use higher-order IMEX-RK methods instead of lower-order methods (ablation) | SINODE · KSE | training convergence | Helped |
| Use IMEX-RK3 instead of IMEX-RK2 (ablation) | SINODE · Viscous Burgers equation | training performance | Helped |
| Use fully implicit Crank--Nicolson instead of semi-implicit IMEX-RK2 | GRAND · Cora and Photo | training completion | Hurt |
Surprising findings
- Lower-order IMEX-RK methods (IMEX-RK2) train more efficiently than high-order IMEX-RK methods despite slightly slower convergence, because higher-order methods need more stages and linear solves.
- Fully implicit Adams and Crank-Nicolson fail to complete training on Cora because the loss blows up.
Limitations
- Not advantageous for nonstiff problems where explicit neural ODEs are more efficient.
- With direct linear solvers, varying step sizes degrade performance because the Jacobian cannot be reused across time steps.
Experimental setup
Datasets: Cora CoauthorCS Photo Kuramoto-Sivashinsky equation simulations viscous Burgers equation simulations
Benchmarks: Cora CoauthorCS Photo Kuramoto-Sivashinsky equation viscous Burgers equation
Metrics: number of function evaluations (NFE) training time training loss testing accuracy
Baselines: RK4 Dopri5 Explicit Adams Implicit Adams Crank-Nicolson
Builds on: neural ODE GRAND PNODE IMEX-RK methods discrete adjoint
Closest papers by meaning
- Semi-supervised Learning of Dynamical Systems with Neural Ordinary Differential Equations: A Teacher-Student Model Approach (AAAI 2024 · similarity 0.89)
- How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization (ICML 2020 · similarity 0.89)
- How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective (ICML 2024 · similarity 0.89)
- Efficient Certified Training and Robustness Verification of Neural ODEs (ICLR 2023 · similarity 0.89)
- Symbolic Neural Ordinary Differential Equations (AAAI 2025 · similarity 0.87)
- Global Convergence in Neural ODEs: Impact of Activation Functions (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.