Research map / Architectures and efficiency / Attention and transformers
DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPOT, a Fourier-attention transformer pre-trained with auto-regressive denoising (Gaussian noise injection) on 12 mixed PDE datasets (>100k trajectories), scaling to ~1B parameters for zero-shot and fine-tuned PDE operator learning.
Problem
Large-scale pre-training of neural operators for PDEs is hard due to long trajectories, multi-scale dynamics, and varying dimensions, resolutions, and channel counts across PDE datasets.
Approach
Auto-regressive next-frame prediction with small Gaussian noise injected into input frames to reduce error accumulation, combined with a scalable Fourier-attention transformer (shared-MLP frequency mixing, multi-head, temporal aggregation) and balanced dataset sampling.
Key results
- Ours-Ti (7M) reduces FNO-1e-4 zero-shot L2RE from 0.116 (FNO-m) to 0.0606 and sets SOTA on 9 of 12 pre-training subsets among small models.
- The largest DPOT-H (1.03B) zero-shot reaches L2RE 0.00229 on FNO-1e-3 and 0.0379 on PDEArena-NS, versus 0.0798 for DPOT-L (0.5B).
- Fine-tuned DPOT-L (500M) reaches L2RE 0.00322 on CFDBench and 0.00170 on PDEBench SWE; on 3D data fine-tuning cuts error from 0.410 (FNO) to 0.226.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add auto-regressive denoising pre-training (ablation) | DPOT · PDEBench test subsets (average) | L2RE: 0.0178 -> 0.0152 | Helped |
| Use noise level 5E-4 instead of 5E-5 (ablation) | DPOT · PDEBench test subsets (average) | L2RE: 0.0152 -> 0.0156 | Hurt |
| Use noise level 5E-3 or 5E-2 instead of 5E-5 (ablation) | DPOT · PDEBench test subsets (average) | L2RE: 0.0152 -> 0.0672; 0.0152 -> 0.753 | Hurt |
| Use 1, 4, or 16 Fourier attention heads instead of 8 (ablation) | DPOT · PDEBench test subsets (average) | L2RE: 0.0174 -> 0.0214; 0.0174 -> 0.0186; 0.0174 -> 0.0188 | Hurt |
| Use patch size 2, 4, or 16 instead of 8 (ablation) | DPOT · PDEBench test subsets (average) | L2RE: 0.0174 -> 0.369; 0.0174 -> 0.0275; 0.0174 -> 0.0278 | Hurt |
| Apply frequency-domain soft-thresholding (ablation) | DPOT Fourier attention · PDEBench test subsets (average) | Test loss: 0.0174 -> 0.121 | Hurt |
| Use balanced instead of equal dataset sampling (ablation) | DPOT-Tiny · Multiple PDE datasets | L2RE: FNO-1e-5: 0.177 -> 0.0976; PDB-M1: 0.0265 -> 0.0285 | Mixed |
| Replace Fourier mixer with self-attention (ablation) | DPOT · PDEBench test subsets | Test loss: 0.0174 -> 0.0238 | Hurt |
| Scale DPOT from 30M to 500M parameters | DPOT zero-shot · PDEBench (0.1, 0.01) | L2RE: 0.0188 -> 0.0115 | Helped |
| Fine-tune from PDE-pre-trained weights instead of training from scratch | DPOT · PDEBench 3D | L2RE: 0.262 -> 0.226 | Helped |
| Fine-tune from PDE-pre-trained weights instead of training from scratch | DPOT · CNO steady-state | L1-Median: 0.0331 -> 0.0230 | Helped |
| Initialize DPOT with PDE-pre-trained weights | DPOT · FNO-1e-3, trajectory length 500 | L2RE: 0.0912 -> 0.0385 | Helped |
Surprising findings
- Sparsity via soft-thresholding (AFNO-style) substantially increased training loss and hurt test error on PDE data, so DPOT omits it.
- Equal sampling across datasets left some subsets (e.g., FNO-1e-5, PDB-DR) with poor convergence; balanced sampling was needed.
Limitations
- Pre-training uses only 2D (and 3D only in fine-tuning) data; 3D transfer evaluated with limited downstream tasks.
- Some subsets (e.g., PDB-DR, PDEArena-NS-cond) still lag MPP or baselines in zero-shot performance.
- Scaling behavior explored only up to ~1B parameters and a fixed set of datasets.
Experimental setup
Datasets: FNO vorticity datasets (1e-5, 1e-4, 1e-3 viscosity) PDEBench CNS, DR, SWE PDEArena NS CFDBench
Benchmarks: PDEBench PDEArena CFDBench FNO vorticity datasets PDEBench turbulence (downstream) 3D Navier-Stokes (downstream) CNO steady-state (downstream) SuperBench
Metrics: L2RE relative median L1 error RFNE
Baselines: FNO UNet FFNO GK-Transformer GNOT OFormer MPP-AViT FNO-m
Builds on: Fourier neural operator AFNO MPP vision transformer
Closest papers by meaning
- Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training (NeurIPS 2025 · similarity 0.88)
- Efficient Generative Transformer Operators for Million-Point PDEs (NeurIPS 2026 · similarity 0.88)
- Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning (NeurIPS 2024 · similarity 0.87)
- Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs (NeurIPS 2024 · similarity 0.86)
- Active Learning for Neural PDE Solvers (ICLR 2025 · similarity 0.86)
- Geometry Aware Operator Transformer as an efficient and accurate neural surrogate for PDEs on arbitrary domains (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.