Research map / Probabilistic and scientific modeling / Differential equations and PDEs
AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AttNS, which adds an attention-style additive correction term (inspired by ResNet attention) to AI-hybrid numerical ODE solvers so they generalize and stay robust with limited training data.
Problem
AI-hybrid numerical solvers for ODEs need large amounts of high-resolution training data for their correction term; with limited data they suffer poor generalization and robustness (numerical explosion on chaotic systems).
Approach
Transfer the ResNet attention mechanism to numerical solvers via the dynamical-system view of ResNet, using an additive attention correction Q[S_hat] that takes the integration term S_hat as input (not the state u_n), with a rational-activation MLP that is Lipschitz, avoiding the multiplicative attention's tendency to collapse to the identity.
Key results
- On the k-link pendulum and elastic pendulum benchmarks, AttNS achieves lower MSE than NeurVec, CSM and FSNet under 50% training data (e.g., elastic pendulum MSE 5.41e-7 for the chosen configuration).
- AttNS reaches performance comparable to SOTA AHS with less training data, and shows stable simulations under noise attack where NeurVec explodes numerically.
- Multiplicative attention variant (AttNS-m) performs poorly, and the attention module's Lipschitz construction is claimed to guarantee convergence; the improvement with full data is said to be under 10-15%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase attention depth from h=2 to h=3 or h=4 (ablation) | AttNS · Spring-mass, k-link pendulum, and elastic pendulum | MSE loss: 2.58e-6 -> 2.49e-6; 4.26e-9 -> 4.39e-9; 5.41e-7 -> 5.28e-7 (h=3); 2.58e-6 -> 2.55e-6; 4.26e-9 -> 4.42e-9; 5.41e-7 -> 5.5 | Mixed |
| Reduce attention width from d1=1024 to d1=512 (ablation) | AttNS · Spring-mass, k-link pendulum, and elastic pendulum | MSE loss: 2.58e-6 -> 3.76e-5; 4.26e-9 -> 2.56e-8; 5.41e-7 -> 2.56e-6 | Hurt |
| Increase attention width from d1=1024 to d1=2048 (ablation) | AttNS · Spring-mass, k-link pendulum, and elastic pendulum | MSE loss: 2.58e-6 -> 2.41e-6; 4.26e-9 -> 3.98e-9; 5.41e-7 -> 4.83e-7 | Helped |
| Add a skip connection (ablation) | AttNS · Spring-mass, k-link pendulum, and elastic pendulum | MSE loss: 2.58e-6 -> 4.57e-4; 4.26e-9 -> 6.36e-6; 5.41e-7 -> 1.39e-4 | Hurt |
| Use the full integration term ŜΔtc instead of Ŝ as the attention input (ablation) | AttNS · Spring-mass and elastic pendulum | MSE loss: 2.58e-6 -> 3.67e-6; 5.41e-7 -> 1.65e-6 | Hurt |
| Use the full integration term ŜΔtc instead of Ŝ as the attention input (ablation) | AttNS · k-link pendulum | MSE loss: 4.26e-9 -> 4.18e-9 | No clear effect |
| Replace the AttNS attention module with an LSTM (ablation) | AttNS · elastic pendulum and k-link pendulum | MSE loss: 5.41e-07 -> 7.21e-06; 4.26e-09 -> 1.35e-08 | Hurt |
| Replace the AttNS attention module with a Transformer (ablation) | AttNS · elastic pendulum and k-link pendulum | MSE loss: 5.41e-07 -> 3.64e-04; 4.26e-09 -> 2.65e-06 | Hurt |
Surprising findings
- Skip connections in the attention module substantially hurt simulation accuracy for both linear and chaotic systems.
- Transformer and LSTM attention modules perform worse than the simple MLP-style module, which the authors attribute to lacking Lipschitz continuity.
Limitations
- Only ODEs are studied; PDEs are left to future work.
- AttNS does not fully eliminate the effect of noise inherent in the dataset.
- Improvement is limited when full training data is available (about 10-15%).
Experimental setup
Datasets: spring-mass system simulations k-link pendulum simulations elastic pendulum simulations
Benchmarks: spring-mass system k-link pendulum elastic pendulum
Metrics: MSE loss on test trajectories
Baselines: NeurVec CSM FSNet AttNS-m LSTM Transformer
Builds on: NeurVec ResNet dynamical system perspective SENet
Closest papers by meaning
- Neural Ordinary Differential Equations (NeurIPS 2018 · similarity 0.87)
- Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations (ICML 2018 · similarity 0.86)
- Transolver Is a Linear Transformer: Revisiting Physics-Attention Through the Lens of Linear Attention (AAAI 2026 · similarity 0.85)
- Breakeven complexity: A new perspective on neural partial differential equation solvers (NeurIPS 2026 · similarity 0.84)
- TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision (ICML 2024 · similarity 0.84)
- NeuralSolver: Learning Algorithms For Consistent and Efficient Extrapolation Across General Tasks (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.