Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Efficient Training of Neural Fractional-Order Differential Equation via Adjoint Backpropagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adjoint backpropagation for neural fractional-order differential equations (FDEs) by solving an augmented reverse-time FDE, reducing training memory versus direct differentiation through the solver while matching baseline accuracy on MNIST, Fashion-MNIST, graph node classification and a Lotka-Volterra parameter-estimation task.
Problem
Training neural fractional-order differential equations by direct differentiation through FDE solver iterations is memory- and compute-intensive, especially for large-scale applications.
Approach
Derives a Caputo-fractional adjoint equation with a right-sided Riemann-Liouville-type reverse-time FDE, solved with a fractional Adams-type predictor and Euler quadrature, so gradients are obtained by a backward FDE pass that reuses the forward trajectory instead of storing all solver iterations.
Key results
- On MNIST with T=20, adjoint backpropagation reduced training GPU memory from 3612 MB to 2432 MB (about 33%) with test error 0.36% vs 0.39% for direct differentiation.
- For large T (up to 200) on MNIST, adjoint training memory fell to about 40% of direct differentiation (e.g., 8.4 GB vs 20.7 GB at T=200).
- On Ogbn-products, adj-F-GRAND-l reached 78.36% accuracy versus 77.25% for F-GRAND-l, enabling batch size 20,000 vs 10,000 on the same GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adjoint backpropagation instead of direct differentiation | Neural FDE on MNIST · MNIST | Train. GPU Mem (MB): 3612 -> 2432 | Helped |
| Use adjoint backpropagation instead of direct differentiation | Neural FDE on MNIST · MNIST test | Test Error: 0.39% -> 0.36% | Helped |
| Use adjoint backpropagation instead of direct differentiation | F-GRAND-l · Ogbn-arxiv | Train. GPU Mem. (MB): 8527 -> 5570 | Helped |
| Use adjoint backpropagation instead of direct differentiation | F-GRAND-nl · Ogbn-arxiv | Train. GPU Mem. (MB): 18180 -> 9086 | Helped |
| Use adjoint backpropagation instead of direct differentiation | F-GRAND-l · Ogbn-products | Train. GPU Mem. (MB): 7950 -> 6602 | Helped |
| Use adjoint backpropagation instead of direct differentiation | F-GRAND-nl · Ogbn-products | Train. GPU Mem. (MB): 17210 -> 11238 | Helped |
| Use adjoint backpropagation instead of direct differentiation | F-GRAND-l · Ogbn-products | Accuracy: 77.25$\pm$0.62 -> \textbf{78.36$\pm$0.32} | Helped |
| Use adjoint backpropagation instead of direct differentiation | Neural FDE on Fashion-MNIST · Fashion-MNIST test, T=1 | test accuracy: 92.35\% -> 92.05\% | Hurt |
Surprising findings
- On Fashion-MNIST at T=1, adjoint training gave slightly lower test accuracy (92.05%) than direct differentiation (92.35%).
Limitations
- Adjoint method may introduce numerical inaccuracies from interpolation errors and sensitivity to initial conditions, especially in chaotic systems.
- Only the basic predictor is used to solve the reverse-mode FDE; more accurate corrector formulas are not considered.
- Hyperparameter-matched gradient validation is limited to a small low-dimensional Lotka-Volterra case; high-dimensional cases were not validated.
Experimental setup
Datasets: MNIST Fashion-MNIST Cora Citeseer PubMed CoauthorCS Computers Photo CoauthorPhy Ogbn-arxiv Ogbn-products synthetic Lotka-Volterra data
Benchmarks: MNIST Fashion-MNIST Cora Citeseer PubMed CoauthorCS Computers Photo CoauthorPhy Ogbn-arxiv Ogbn-products
Metrics: test accuracy GPU memory (MB/GB) training time (s) inference time (s) parameter estimation error
Baselines: GCN GAT HGCN CGNN GDE GRAND-l GRAND-nl F-GRAND-l F-GRAND-nl Node2vec GraphSAGE MLP
Builds on: neural ODE adjoint sensitivity method GRAND F-GRAND fractional Adams-Bashforth-Moulton
Closest papers by meaning
- Neural Variable-Order Fractional Differential Equation Networks (AAAI 2025 · similarity 0.90)
- Coupling Graph Neural Networks with Fractional Order Continuous Dynamics: A Robustness Study (AAAI 2024 · similarity 0.87)
- Integration Matters for Learning PDEs with Backwards SDEs (NeurIPS 2025 · similarity 0.86)
- Operator-Learning-Inspired Modeling of Neural Ordinary Differential Equations (AAAI 2024 · similarity 0.85)
- SHoP: A Deep Learning Framework for Solving High-Order Partial Differential Equations (AAAI 2024 · similarity 0.85)
- Symbolic Neural Ordinary Differential Equations (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.