Research map / Architectures and efficiency / Attention and transformers
Chaos Meets Attention: Transformers for Large-Scale Dynamical Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a factorized-attention transformer with axial mean-max-min pooling, random Fourier positional encoding, and a Hutchinson-estimated unitary-operator loss to stably forecast long-term statistics of large-scale chaotic flows.
Problem
Autoregressive long-term prediction of high-dimensional dissipative chaotic systems is unstable because positive Lyapunov exponents amplify errors, and existing optimal-transport distribution-matching methods do not scale to high dimensions.
Approach
Combines RFF-based Gaussian-kernel attention with factorized axial attention using mean, max and min pooling (A3M), and enforces a unitary forward operator via a Hutchinson stochastic trace estimate of a von Neumann ergodic-theorem-inspired regularizer, avoiding explicit distribution matching.
Key results
- On KF256, the model reduces long-term ME-APE, ME-LRw and mixing-rate error by 23.1%, 21.0% and 20.0% over the best baseline.
- On TCF, short-term relative L2 at tau=5 improves by 25.0% over the best baseline (0.03 vs 0.04 for MWT), and long-term energy-spectrum and mixing-rate errors improve by 30.8%, 25.0% and 27.3%.
- Ablation on KF128 shows the unitary constraint improves mixing-rate error from 0.10 to 0.08 and A3M pooling improves short-term error (Rel-L2 at tau=5 from 1.02 to 0.98 to 0.92 with both components).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed transformer instead of UNO | KF256 prediction · KF256 | Rel-L2 (τ = 5): 0.92 -> 0.86 | Helped |
| use the proposed transformer instead of MNO | KF256 prediction · KF256 | Rel-L2 (τ = 25): 1.29 -> 1.27 | Helped |
| use the proposed transformer instead of FactFormer | KF256 prediction · KF256 | ME-APE: 0.13 -> 0.10 | Helped |
| use the proposed transformer instead of FactFormer | KF256 prediction · KF256 | KLD: 0.37 -> 0.29 | Helped |
| use the proposed transformer instead of MWT | TCF prediction · TCF | Rel-L2 (τ = 5): 0.04 -> 0.03 | Helped |
| use the proposed transformer instead of MWT | TCF prediction · TCF | Rel-L2 (τ = 25): 0.14 -> 0.13 | Helped |
| use the proposed transformer instead of FactFormer | TCF prediction · TCF | ME-APE: 0.13 -> 0.09 | Helped |
| add Min- and Max-pooling to the base version (ablation) | KF128 model · KF128 | Rel-L2 (τ = 5): 1.02 -> 0.98 | Helped |
| add Min- and Max-pooling to the base version (ablation) | KF128 model · KF128 | ME-APE: 0.17 -> 0.15 | Helped |
| add Min- and Max-pooling to the base version (ablation) | KF128 model · KF128 | ∆λ: 0.11 -> 0.10 | Helped |
| add a unitary-constrained operator to the A3M model (ablation) | KF128 model · KF128 | Rel-L2 (τ = 5): 0.98 -> 0.92 | Helped |
| add a unitary-constrained operator to the A3M model (ablation) | KF128 model · KF128 | ME-LRw: 0.21 -> 0.19 | Helped |
Surprising findings
- Periodic, integer-Fourier-spectrum operator baselines (MNO, UNO) show periodic time-correlation patterns that contradict the monotonic decay expected of chaotic mixing.
Limitations
- Focused on ergodic chaotic systems; does not generalize well to non-ergodic systems where long-term statistics are not well-defined.
- Relies on uniform grid structures, limiting applicability to complex geometries or mesh-based systems.
Experimental setup
Datasets: Kolmogorov Flow (KF256) Turbulent Channel Flow (TCF) Kolmogorov Flow KF128 (ablation)
Benchmarks: Kolmogorov Flow KF256 Turbulent Channel Flow TCF
Metrics: relative L2 error mean energy absolute percentage error mean energy LRw mixing rate difference KL divergence on principal components TKE error
Baselines: MNO UNO MWT FactFormer
Builds on: Factformer Markov neural operator Fourier neural operator random Fourier features von Neumann mean ergodic theorem
Closest papers by meaning
- When are dynamical systems learned from time series data statistically accurate? (NeurIPS 2024 · similarity 0.89)
- DySLIM: Dynamics Stable Learning by Invariant Measure for Chaotic Systems (ICML 2024 · similarity 0.88)
- Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective (NeurIPS 2024 · similarity 0.87)
- The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modeling (NeurIPS 2026 · similarity 0.86)
- Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos (ICML 2025 · similarity 0.86)
- Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.