Research map / Probabilistic and scientific modeling / Differential equations and PDEs
How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes two control-theory methods (minimum-time Neural ODEs with a learnable terminal time, and Lyapunov-based pre-training followed by forward terminal-time search) to find a shallower effective depth, accelerating Neural ODE training and inference.
Problem
Neural ODEs are slow to train and infer, and the required depth (terminal integration time) is unknown and often larger than needed.
Approach
Treat the terminal time as a learnable parameter under a minimum-time optimal control objective (single stage), and alternatively pre-train with a Lyapunov convergence-rate loss on a short interval then increase the terminal time until the loss threshold is met (two stage), yielding shallower and faster Neural ODEs.
Key results
- On CIFAR-10, minimum-time Neural ODEs with lambda=0.1 reach 81.74% accuracy vs 82.60% for vanilla, with training time 2h22min vs >10h and NFE 146.1 vs 404.9.
- On MNIST, minimum-time NODEs reach 99.53% test accuracy vs 99.57% for vanilla, cutting training time from 4h47min to 20min and inference time from 0.546s to 0.03s.
- On two moons, minimum-time training takes 7.33s vs 46.64s for vanilla and inference 8.4ms vs 71.88ms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize terminal time jointly with network parameters using Minimum-time Neural ODEs | Neural ODE · MNIST | training time: 4h47min23s -> 20min23s | Helped |
| Optimize terminal time jointly with network parameters using Minimum-time Neural ODEs | Neural ODE · MNIST | test accuracy: 99.57% -> 99.53% | No clear effect |
| Optimize terminal time jointly with network parameters using Minimum-time Neural ODEs | Neural ODE · Fashion-MNIST | training time: 4h41min19s -> 22min24s | Helped |
| Optimize terminal time jointly with network parameters using Minimum-time Neural ODEs | Neural ODE · Fashion-MNIST | test accuracy: 93.07% -> 92.68% | No clear effect |
| Optimize terminal time jointly with network parameters using Minimum-time Neural ODEs | Neural ODE · CIFAR-10 | training time: >10h -> 42min37s | Helped |
| Optimize terminal time jointly with network parameters using Minimum-time Neural ODEs | Neural ODE · CIFAR-10 | test accuracy: 82.60% -> 76.38% | Hurt |
| Set temporal regularization weight to λ=0.1 rather than λ=1 (ablation) | Minimum-time Neural ODE · CIFAR-10 | test accuracy: 76.38% -> 81.74% | Helped |
| Set temporal regularization weight to λ=0.1 rather than λ=1 (ablation) | Minimum-time Neural ODE · CIFAR-10 | training time: 42min37s -> 2h22min15s | Hurt |
| Set temporal regularization weight to λ=0.1 rather than λ=1 (ablation) | Minimum-time Neural ODE · CIFAR-10 | inference time: 0.098s -> 0.278s | Hurt |
| Use Lyapunov training with κ=20 on [0,1] instead of κ=10 (ablation) | Convergence-rate-based Neural ODE · Concentric Annuli | dynamics speed | No clear effect |
| Pre-train with the Lyapunov method on a shorter time span (ablation) | Convergence-rate-based Neural ODE · Concentric Annuli | dynamics speed | Helped |
Surprising findings
- Over a longer time span, the Lyapunov-trained model with kappa=20 did not learn faster dynamics, while shorter time spans did, suggesting numerical difficulty of sustaining descent over long horizons.
- Combining minimum-time and Lyapunov approaches gave no significant advantage over the individual methods in preliminary experiments.
Limitations
- Approach 1 depends on choosing the time-regularization weight lambda, which can degrade performance and trades accuracy for speed.
- Approach 2 needs a two-stage process and is harder when Neural ODEs are not at the end of the network.
- No theoretical convergence-rate guarantee for the minimum-time approach.
Experimental setup
Datasets: two moons Concentric Annuli MNIST Fashion-MNIST CIFAR-10
Benchmarks: two moons Concentric Annuli MNIST Fashion-MNIST CIFAR-10
Metrics: test accuracy training time inference time number of function evaluations terminal time
Baselines: Vanilla Neural ODEs LyaNet STEER TayNODE SNOpt
Builds on: Neural ODEs LyaNet STEER SNOpt TayNODE
Closest papers by meaning
- How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization (ICML 2020 · similarity 0.91)
- Go with the flow: Adaptive control for Neural ODEs (ICLR 2021 · similarity 0.91)
- ControlSynth Neural ODEs: Modeling Dynamical Systems with Guaranteed Convergence (NeurIPS 2024 · similarity 0.90)
- Second-Order Neural ODE Optimizer (NeurIPS 2021 · similarity 0.90)
- Global Convergence in Neural ODEs: Impact of Activation Functions (ICLR 2025 · similarity 0.89)
- Semi-Implicit Neural Ordinary Differential Equations (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.