Research map / Architectures and efficiency / RNNs and normalization

Multiple Physics Pretraining for Spatiotemporal Surrogate Models

NeurIPS 2024 · Architectures and efficiency › RNNs and normalization · approach: Scalable approximation and equivariant compute · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Multiple Physics Pretraining (MPP), an autoregressive task-agnostic pretraining of axial-attention transformers on heterogeneous 2D PDE systems with shared normalized field embeddings, showing transfer to unseen physics and 3D inflation.

Problem

Building task-agnostic pretrained surrogate models for spatiotemporal physical systems that can handle multiple heterogeneous PDEs and transfer to low-data or unseen physics regimes.

Approach

Pretrain a single axial ViT to autoregressively predict the next step of multiple physical systems by projecting each system's fields into a shared embedding space via per-sample RevIN normalization and system-specific 1x1 field embeddings, with normalized MSE loss and micro-batch task sampling.

Key results

• A single MPP-pretrained AViT-B (116M) reaches SWE NRMSE 0.00240 and CNS M1.0 0.0281 versus task-specific AViT-B at 0.00047 and 0.0316, matching or beating dedicated FNO-B and UNet baselines without finetuning.
• In the low-data transfer setting to unseen compressible NS 'Far' physics, MPP outperforms VideoMAE and training from scratch by a large margin across data availability levels.
• On 3D inflation, MPP-pretrained models reduce NRMSE on the random-initialized CNS dataset from 0.299 (scratch) to 0.264 (11.7% improvement).

Measured effects reported in the paper

Surprising findings

• Multi-task training sometimes hurts performance on individual tasks relative to task-specific models, which the authors note.
• Removing RevIN normalization or the normalized loss degrades results drastically (e.g., CNS M0.1 NRMSE rises from 0.0172 to 1.5892 when both are removed).

Limitations

• Architecture restricted to uniform grids, limiting non-uniform geometries.
• Training is one-step-ahead only without rollouts, noise, or stability tricks; data is largely fluid-like and clean.
• Multi-task training can degrade individual-task performance; boundary conditions interact poorly with attention for absorbing boundaries.

Experimental setup

Datasets: PDEBench PDEArena Kinetics-400 Something-Something V2

Benchmarks: PDEBench PDEArena

Metrics: NRMSE RMSE

Baselines: UNet FNO ORCA-SWIN-B VideoMAE AViT scratch FNO-B

Builds on: Vision Transformer Reversible Instance Normalization VideoMAE Fourier Neural Operator PDEBench

Closest papers by meaning

• PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations (ICML 2025 · similarity 0.88)
• Space and time continuous physics simulation from partial observations (ICLR 2024 · similarity 0.86)
• Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators (NeurIPS 2024 · similarity 0.86)
• Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation (NeurIPS 2025 · similarity 0.85)
• SCENT: Robust Spatiotemporal Learning for Continuous Scientific Data via Scalable Conditioned Neural Fields (ICML 2025 · similarity 0.85)
• PARCv2: Physics-aware Recurrent Convolutional Neural Networks for Spatiotemporal Dynamics Modeling (ICML 2024 · similarity 0.85)