Research map / Architectures and efficiency / Attention and transformers
AMR-Transformer: Enabling Efficient Long-range Interaction for Complex Neural Fluid Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AMR-Transformer, which tokenizes fluid fields with a physics-constrained adaptive mesh refinement (quadtree) and feeds the tokens to an encoder-only Transformer solver to model long-range interactions efficiently.
Problem
Neural CFD surrogates struggle to capture long-range dependencies (e.g., shockwaves, turbulence) with CNN/GNN receptive-field limits, while Transformers' O(N^2) attention over all grid cells is too costly at high resolution.
Approach
Use a hierarchical multi-way-tree adaptive mesh refinement as a tokenizer, with a Navier-Stokes-aware pruning module (velocity gradient, vorticity, momentum, Kelvin-Helmholtz shear) deciding which regions are subdivided, so a Transformer attends over a small set of multi-scale patches.
Key results
- On the Dam problem, MSE drops to 1.63E-4 versus 1.64E-3 for DeepONet (about 91% relative improvement).
- On the Shock problem, NMSE of 9.32E-4 beats FNO's 5.53E-3 (about 83% improvement).
- AMR tokenizer reduces token count by roughly 60-70% and FLOPs by 70-75% on CFDBench, reaching up to about 60x FLOPs reduction versus ViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AMR-Transformer instead of DeepONet | Dam simulation · Dam | MSE: 1.64E-3 -> 1.63E-4 | Helped |
| Use AMR-Transformer instead of FNO | Cylinder simulation · Cylinder | MSE: 2.74E-5 -> 2.09E-5 | Helped |
| Use AMR-Transformer instead of U-Net | Cavity simulation · Cavity | MSE: 1.58E-2 -> 4.71E-3 | Helped |
| Use AMR-Transformer instead of U-Net | Tube simulation · Tube | MSE: 1.29E-3 -> 1.33E-3 | Hurt |
| Use AMR-Transformer instead of FNO | Shockwave simulation · Shockwave | MSE: 4.83E-3 -> 9.66E-4 | Helped |
| Use AMR-Transformer instead of FNO | Diffusion-Reaction · Diffusion-Reaction | MSE: 1.76E-2 -> 1.75E-2 | Helped |
| Use AMR-Transformer instead of DeepONet | NS-Incom-Inhom simulation · NS-Incom-Inhom | MSE: 5.88E-6 -> 3.76E-6 | Helped |
| Use AMR tokenizer instead of a regular tokenizer | Shockwave Transformer · Shockwave | GFLOPs: 71.37 -> 7.51 | Helped |
| Use only velocity-gradient pruning instead of all physical criteria (ablation) | AMR-Transformer on Dam · Dam | MSE: 1.63E-4 -> 2.80E-4 | Hurt |
| Use only vorticity pruning instead of all physical criteria (ablation) | AMR-Transformer on Dam · Dam | MSE: 1.63E-4 -> 1.93E-3 | Hurt |
| Use only momentum pruning instead of all physical criteria (ablation) | AMR-Transformer on Dam · Dam | MSE: 1.63E-4 -> 1.12E-3 | Hurt |
| Use only Kelvin-Helmholtz instability pruning instead of all physical criteria (ablation) | AMR-Transformer on Dam · Dam | MSE: 1.63E-4 -> 5.40E-4 | Hurt |
Surprising findings
- The Identity baseline outperforms FNO on the Dam problem in MAE, and DeepONet is close to Identity on Cylinder and Dam.
- Individual physical pruning criteria alone give worse accuracy than the combined criteria, and FNO (1.00 NMSE) fails on NS-Incom-Inhom.
Limitations
- Shockwave dataset is small (10 cases) and at 128x128 resolution.
- Pruning thresholds are sampled randomly and tuned post-training; the paper provides limited theoretical justification for the criteria.
- Several reported comparisons rely on baseline configurations taken from CFDBench with no mention of ablating the tokenizer on the other baselines.
Experimental setup
Datasets: CFDBench PDEBench Shockwave (new)
Benchmarks: CFDBench PDEBench Shockwave
Metrics: NMSE MAE MSE GFLOPs token count
Baselines: U-Net FNO DeepONet MeshGraphNet ViT Identity
Builds on: Vision Transformer MeshGraphNet Fourier Neural Operator DeepONet
Closest papers by meaning
- Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries (ICML 2025 · similarity 0.80)
- Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators (NeurIPS 2024 · similarity 0.79)
- A differentiable brain simulator bridging brain simulation and brain-inspired computing (ICLR 2024 · similarity 0.79)
- PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations (ICML 2025 · similarity 0.78)
- Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs (NeurIPS 2024 · similarity 0.78)
- EddyFormer: Accelerated Neural Simulations of Three-Dimensional Turbulence at Scale (NeurIPS 2025 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.