Research map / Architectures and efficiency / Attention and transformers
SEA: State-Exchange Attention for High-Fidelity Physics Based Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a State-Exchange Attention (SEA) module that lets per-field transformer decoders exchange information via cross-attention, combined with a ViT-like mesh autoencoder, for autoregressive CFD field prediction with lower rollout error.
Problem
Autoregressive surrogate modeling of dynamical systems governed by PDEs suffers from rollout error accumulation, and existing models do not explicitly capture coupling between state variables (fields).
Approach
Encode field groups separately with a ViT-like mesh autoencoder, assign one transformer decoder per field, and let fields exchange information through a bottleneck cross-attention module (SEA) at every layer, with TIPI and AdaLN for conditioning on time-invariant parameters.
Key results
- Full ViT-SEA Transformer achieves average rollout error 3.7e-3 on cylinder flow vs 43.96e-3 for GMR-GMUS Transformer and 32.6e-3 for PbGMR-GMUS Transformer-RealNVP (reported 88% and 91% reductions).
- On multiphase flow, SEA reduces mean volume fraction error by about 52% vs basic transformer and 97% vs Field Fusion Encoder transformer (0.12 vs 0.25 and 4.61).
- ViT mesh autoencoder reconstruction error on cylinder flow is 1.7e-3, vs 14.3e-3 for GMR-GMUS and 1.9e-3 for PbGMR-GMUS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the PbGMR-GMUS Transformer-RealNVP baseline with the full ViT-SEA Transformer | Cylinder flow, Re=400 · Cylinder flow, Re=400 | time-average rollout error (×10^-3): 32.6 -> 3.7 | Helped |
| Replace the GMR-GMUS Transformer baseline with the full ViT-SEA Transformer | Cylinder flow, Re=400 · Cylinder flow, Re=400 | time-average rollout error (×10^-3): 43.96 -> 3.7 | Helped |
| Add SEA field exchange instead of using the basic Transformer without exchange | Multiphase flow · Multiphase flow | mean volume fraction error: 0.25 -> 0.12 | Helped |
| Add SEA field exchange instead of using the FFE Transformer | Multiphase flow · Multiphase flow | mean volume fraction error: 4.61 -> 0.12 | Helped |
| Add SEA cross-attention exchange rather than no exchange (ablation) | Vanilla Transformer on cylinder flow · Cylinder flow, CF (v) | RelMSE (×10^3): 626 -> 206.5 | Helped |
| Use SEA rather than addition-based exchange (ablation) | Vanilla Transformer on multiphase flow · Multiphase flow, MP (u) | RelMSE (×10^3): 874.3 -> 904.1 | Hurt |
| Use SEA rather than addition-based exchange (ablation) | Vanilla Transformer on multiphase flow · Multiphase flow, MP (α) | RelMSE (×10^3): 164.5 -> 116.9 | Helped |
| Add conditioning instead of using no conditioning (ablation) | Cylinder-flow temporal model · Cylinder flow | relative MSE (×10^3): 158.63 -> 80 | Helped |
| Add conditioning instead of using no conditioning (ablation) | Multiphase temporal model · Multiphase flow | relative MSE (×10^3): 780 -> 692.9 | Helped |
| Add AdaLN conditioning to TIPI (ablation) | Cylinder-flow temporal model · Cylinder flow | relative MSE (×10^3): 80 -> 38.7 | Helped |
| Add AdaLN conditioning to TIPI (ablation) | Multiphase temporal model · Multiphase flow | relative MSE (×10^3): 692.9 -> 685.9 | Helped |
| Use variational rather than pointwise mesh encoding (ablation) | ViT mesh autoencoder · Studied cases | reconstruction accuracy | No clear effect |
Surprising findings
- On the multiphase case, the SEA module gives only a marginal 40-50% velocity improvement, with most gain concentrated in the volume fraction field.
- Pure addition-based exchange performs nearly as well as cross-attention exchange on the multiphase velocity fields (Table of ablation).
Limitations
- Scalability to many state variables is questionable, since each field needs its own parallel transformer decoder.
- Evaluation limited to two 2D CFD cases with a single-layer temporal transformer.
- Multiphase comparison uses the same autoencoder variants only partially; ablation tables have missing entries.
Experimental setup
Datasets: 2D cylinder flow simulations (icoFoam, OpenFOAM) two-liquid mixing simulations (twoLiquidMixingFoam, OpenFOAM)
Benchmarks: 2D cylinder flow (Re=400) multiphase immiscible two-liquid collapse
Metrics: relative MSE rollout error relative reconstruction error
Baselines: MGN MGN-NI GMR-GMUS Transformer PbGMR-GMUS Transformer-RealNVP Field Fusion Encoder Transformer basic transformer without exchange
Builds on: GMR-GMUS PbGMR-GMUS Transformer-RealNVP Vision Transformer Perceiver adaptive layer norm RoPE
Closest papers by meaning
- PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations (ICML 2025 · similarity 0.88)
- Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators (NeurIPS 2024 · similarity 0.87)
- Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems (ICML 2025 · similarity 0.85)
- Transolver: A Fast Transformer Solver for PDEs on General Geometries (ICML 2024 · similarity 0.85)
- Predictive Geometry of Hidden Trajectories in Transformers (NeurIPS 2026 · similarity 0.85)
- U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.