Research map / Probabilistic and scientific modeling / Differential equations and PDEs
OmniArch: Building Foundation Model for Scientific Computing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OmniArch, a Fourier-encoder plus shared Transformer foundation model for unified 1D-2D-3D PDE pre-training with a PDE-Aligner for physics-informed fine-tuning, evaluated on PDEBench and PDEArena.
Problem
Neural surrogate models for PDEs are typically task-specific and struggle to unify multiple dimensions, grid resolutions, and physical quantities, and to generalize zero-shot or in context.
Approach
Encode multi-dimensional physical fields in the Fourier domain with TopK mode truncation so varying grids map to a common token length, process time-ordered tokens with a shared Transformer using a Temporal Mask (full attention among quantities within a timestep, causal across timesteps), and align predictions with PDE text captions via a frequency-domain PDE-Aligner.
Key results
- OmniArch-L with PDE-Aligner reaches nRMSE 0.0200 on 1D CFD versus 1.4100 for FNO (best prior listed), and 0.0125 on 2D CFD versus 0.0112 for DPOT-L (where DPOT is better per underline).
- Temporal Mask improves over causal mask by roughly 18-20% (e.g., 3D CFD 0.1494 vs 0.1842 nRMSE).
- On zero-shot PDEs (Shock, KH, OTVortex), OmniArch-L obtains 0.2126, 0.2763, 0.1718 nRMSE versus 0.3243 to 1.6667 for baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with PDE-Aligner rather than use pre-training alone (ablation) | OmniArch-L · 1D PDEs | nRMSE: 0.0103 -> 0.0056 | Helped |
| fine-tune with PDE-Aligner rather than use pre-training alone (ablation) | OmniArch-L · 2D PDEs | nRMSE: 0.0440 -> 0.0262 | Helped |
| fine-tune with PDE-Aligner rather than use pre-training alone (ablation) | OmniArch-L · 3D PDEs | nRMSE: 0.3399 -> 0.2697 | Helped |
| add PDE-Aligner during fine-tuning instead of fine-tuning without it (ablation) | OmniArch-L · 1D PDEs | nRMSE: 0.0073 -> 0.0056 | Helped |
| add PDE-Aligner during fine-tuning instead of fine-tuning without it (ablation) | OmniArch-L · 2D PDEs | nRMSE: 0.0345 -> 0.0262 | Helped |
| add PDE-Aligner during fine-tuning instead of fine-tuning without it (ablation) | OmniArch-L · 3D PDEs | nRMSE: 0.3432 -> 0.2697 | Helped |
| use Temporal Mask rather than Causal Mask (ablation) | OmniArch · 2D Incom. | nRMSE: 0.0277 -> 0.0227 | Helped |
| use Temporal Mask rather than No Mask (ablation) | OmniArch · 2D Incom. | nRMSE: 0.0285 -> 0.0227 | Helped |
| use Temporal Mask rather than Causal Mask (ablation) | OmniArch · 2D CFD | nRMSE: 0.0198 -> 0.0148 | Helped |
| use Temporal Mask rather than No Mask (ablation) | OmniArch · 2D CFD | nRMSE: 0.0205 -> 0.0148 | Helped |
| use Temporal Mask rather than Causal Mask (ablation) | OmniArch · 3D CFD | nRMSE: 0.1842 -> 0.1494 | Helped |
| use Temporal Mask rather than No Mask (ablation) | OmniArch · 3D CFD | nRMSE: 0.1923 -> 0.1494 | Helped |
Surprising findings
- The paper reports that DPOT-L achieves a slightly better 2D CFD nRMSE (0.0112) than OmniArch-L with PDE-Aligner (0.0125), though its table bolds OmniArch.
- The authors report that OmniArch's 3D results still trail, and 3D CFD nRMSE remains high (about 0.37) despite improvements.
Limitations
- Model is fundamentally data-driven with limited interpretability even with PDE-Aligner.
- Scalability limited by computation and data, weakening generalization on 3D and shock-wave PDEs.
- Weaker 3D performance acknowledged in the conclusion.
Experimental setup
Datasets: PDEBench PDEArena PDE captions augmented with GPT-4-based physical checking
Benchmarks: PDEBench PDEArena zero-shot PDEs (Shock, KH, OTVortex) inverse problems (Forcing, Buoyancy)
Metrics: nRMSE RMSE classification accuracy
Baselines: PINNs U-Net FNO PDEformer-1 MPP ORCA-SWIN Poseidon DPOT
Builds on: Fourier Neural Operator PDEBench Multiple Physics Pre-training LLaMA
Closest papers by meaning
- SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.85)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.84)
- Large Language-Geometry Model: When LLM meets Equivariance (ICML 2025 · similarity 0.84)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.84)
- SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Models (NeurIPS 2026 · similarity 0.83)
- PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.