Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LC-GODE, an architecture-aware latent neural ODE with a GCN graph encoder and a variational seq2seq GRU encoder that extrapolates learning curves for MLPs and CNN cells and ranks training configurations.
Problem
Learning curve extrapolation predicts final neural network performance from early epochs, but existing methods ignore the network architecture, which shapes the loss landscape and training trajectory.
Approach
Conditions a latent neural ODE on a GCN-based architecture embedding (with learnable DiffPool-style pooling) inside a variational seq2seq framework, so one model extrapolates learning curves of unseen architectures and training configurations.
Key results
- LC-GODE reduces test-accuracy extrapolation error versus NODE by 36.13%, 30.72%, 34.97%, and 59.63% across the four datasets.
- On CIFAR-10 accuracy model selection, LC-GODE reduces regret by 96% versus the better of NODE and NSDE and improves predicted-best ranking by 8 positions.
- Model ranking with 10 observed epochs gives about a 20x speedup over full SGD training on the MLP datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add architecture graph information to NODE | Learning-curve extrapolation · car test accuracy, 80 epochs | MAPE: 0.0683 -> 0.0431 | Helped |
| Add architecture graph information to NODE | Learning-curve extrapolation · segment test accuracy, 80 epochs | MAPE: 0.0794 -> 0.0566 | Helped |
| Add architecture graph information to NODE | Learning-curve extrapolation · CIFAR-10 test accuracy, 80 epochs | MAPE: 0.1457 -> 0.1107 | Helped |
| Add architecture graph information to NODE | Learning-curve extrapolation · CIFAR-100 test accuracy, 80 epochs | MAPE: 0.4895 -> 0.2708 | Helped |
| Use graph Transformers instead of GCN message passing (ablation) | LC-GODE · CIFAR-100 | extrapolation accuracy | Mixed |
| Use GAT instead of GCN message passing (ablation) | LC-GODE · CIFAR-10 | extrapolation accuracy | Mixed |
| Use average pooling instead of learnable DiffPool (ablation) | LC-GODE · CIFAR-10 and CIFAR-100 | extrapolation accuracy | Mixed |
| Use max-pooling instead of learnable DiffPool (ablation) | LC-GODE · CIFAR-10 | extrapolation accuracy | Helped |
| Use self-attention instead of GRU for the time-series encoder (ablation) | LC-GODE · CIFAR-10 and CIFAR-100 | extrapolation accuracy | Mixed |
| Use TCN instead of GRU for the time-series encoder (ablation) | LC-GODE · CIFAR-10 | extrapolation accuracy | Helped |
| Increase maximal time T_max (ablation) | LC-GODE · Learning-curve extrapolation | error | Hurt |
| Increase observation length (ablation) | LC-GODE · Learning-curve extrapolation | error | Helped |
Surprising findings
- Test accuracy curves are easier to extrapolate than test loss curves, attributed to accuracy being less sensitive to decision-boundary changes.
- Predicted test loss is less effective than accuracy for selecting the best model, and overall ranking declines for all methods with loss.
Limitations
- Source data is not modeled, so generalization across source tasks is not addressed.
- Ranking by predicted test loss is unreliable, with overall ranking declining for all methods.
- Evaluated on only two MLP tabular tasks and two NAS-Bench-201 CNN tasks.
Experimental setup
Datasets: OpenML car OpenML segment NAS-Bench-201 CIFAR-10 NAS-Bench-201 CIFAR-100
Benchmarks: car segment CIFAR-10 CIFAR-100
Metrics: MAPE RMSE regret ranking speedup
Baselines: LC-BNN LC-PFN VRNN LSTM NODE NSDE
Builds on: Neural ODE Latent ODE (ODE-RNN VAE) GCN DiffPool LC-PFN
Closest papers by meaning
- Difficulty-Aware Learning Curve Extrapolation (AAAI 2026 · similarity 0.89)
- Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks (NeurIPS 2023 · similarity 0.87)
- NAS-Bench-x11 and the Power of Learning Curves (NeurIPS 2021 · similarity 0.87)
- Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations (ICML 2018 · similarity 0.86)
- NeuralSolver: Learning Algorithms For Consistent and Efficient Extrapolation Across General Tasks (NeurIPS 2024 · similarity 0.84)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.