Research map / Probabilistic and scientific modeling / Dynamical systems
Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Universal Physics Transformers (UPTs), a unified neural operator that encodes meshes or particles into a fixed-size latent space, rolls dynamics forward in latent space via inverse encoding/decoding losses, and decodes at arbitrary query points.
Problem
Neural operator surrogates for PDEs scale poorly to large meshes/particle sets, and architectures differ across Eulerian (grid) and Lagrangian (particle) simulations, preventing a unified approach.
Approach
Compress inputs from any grid or particle set into a fixed-size latent token set via a hierarchical encoder (message passing to supernodes, transformer, perceiver pooling), propagate dynamics purely in latent space with a transformer approximator trained with inverse encoding and decoding reconstruction losses, and decode with a perceiver-style cross-attention at arbitrary query points.
Key results
- On ShapeNet-Car, UPT with 64 latent tokens reaches test MSE 2.31 vs 2.58 for GINO at 48^3 resolution (110K tokens), with UPT far faster per epoch (4 s vs 900 s for best GINO).
- On the transient pipe-flow dataset, UPT-68M outperforms GINO, U-Net and FNO by a large margin in MSE and rollout correlation time across model scales.
- Latent rollout of UPT-68M takes 0.3 s on 1 GPU vs 2.0 s for autoregressive UPT and 1.2 s for GINO, roughly 400x faster than pisoFoam on 16 CPUs (120 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UPT with 64 latent tokens instead of GINO at 48³ resolution | ShapeNet-Car pressure prediction · ShapeNet-Car test | MSE ×100: 2.58 -> 2.31 | Helped |
| Add signed distance function feature engineering and use a 64³ grid (ablation) | UPT · ShapeNet-Car test | MSE ×100: 2.31 -> 2.24; 2.14 | Mixed |
| Replace autoregressive rollout through the physics domain with latent rollout (ablation) | UPT-68M · Transient pipe-flow trajectory | time on 1 GPU: 2.0s -> 0.3s | Helped |
| Replace autoregressive rollout through the physics domain with latent rollout (ablation) | UPT-68M · Transient pipe-flow trajectory | rollout performance | No clear effect |
Surprising findings
- Latent rollout achieves on-par accuracy to autoregressive rollout via the physics domain while being much faster at inference, though it adds training overhead.
- Models trained with a fixed number of supernodes and query points generalize to different numbers of input/output points without retraining.
Limitations
- Latent rollout requires non-negligible training overhead in the current implementation.
- Large-scale Lagrangian datasets are not available, so scaling on Lagrangian simulations is not demonstrated; Lagrangian experiments are small-scale illustrations.
- Generalization beyond fluid dynamics is not evaluated.
Experimental setup
Datasets: ShapeNet-Car TGV3D (LagrangeBench) TGV2D (LagrangeBench)
Benchmarks: ShapeNet-Car pisoFoam pipe flow transient dataset TGV3D TGV2D
Metrics: test MSE rollout correlation time (Pearson < 0.8) inference time velocity error norm
Baselines: GINO U-Net FNO GNS SEGNN
Builds on: Transformer Neural Operator Geometry-Informed Neural Operator (GINO) Perceiver DiT adaLN modulation Fourier Neural Operator
Closest papers by meaning
- PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations (ICML 2025 · similarity 0.89)
- HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutions (NeurIPS 2025 · similarity 0.87)
- SEA: State-Exchange Attention for High-Fidelity Physics Based Transformers (NeurIPS 2024 · similarity 0.87)
- RIGNO: A Graph-based Framework For Robust And Accurate Operator Learning For PDEs On Arbitrary Domains (NeurIPS 2025 · similarity 0.87)
- Space and time continuous physics simulation from partial observations (ICLR 2024 · similarity 0.87)
- Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.