Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains neural operators (FNO and VideoMAE-style transformer) on unlabeled PDE inputs without solutions using masked reconstruction and super-resolution proxy tasks, then fine-tunes with fewer simulated solutions and adds similarity-based in-context demos for OOD inference.
Problem
Neural operators for PDEs are data-hungry, and generating labeled PDE solutions is expensive; unsupervised pretraining has been little explored for operator learning, and OOD generalization is poor.
Approach
Pretrain neural operators on cheap unlabeled PDE inputs (coefficients, forcing functions, initial snapshots) with physics-inspired masked-autoencoding and super-resolution proxy tasks, and at test time mine in-context demos by similarity in the model's output space, requiring no extra training.
Key results
- Unsupervised pretraining saves about 5x10^2 to 8x10^5 simulated solutions relative to random initialization at the baseline's best test error across PDEs.
- Vision-pretrained Video-MAE only beats random initialization with large simulated data volumes, while the proposed pretraining saves substantial simulations.
- Adding in-context demos keeps improving FNO's OOD generalization as the number of demos grows; the output-space similarity beats feature-space similarity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on unlabeled PDE data instead of simulating solutions | PDE data generation for operator learning · 2D incompressible Navier-Stokes (PINO) | simulation time: 5499.32 vs 11013.90 sec. | Helped |
| Pretrain on unlabeled PDE data instead of simulating solutions | PDE data generation for operator learning · Reaction-Diffusion (PDE-Bench) | simulation time: 29.65 vs 6657.34 sec. | Helped |
| Increase pretraining samples from 2000 to 57545 (ablation) | FNO · 2D incompressible Navier-Stokes | relative ℓ2 error: 0.3594 -> 0. 3246 | Helped |
| Pretrain on unlabeled PDE data before fine-tuning rather than train from scratch | Neural operators · Poisson, Helmholtz, Reaction-Diffusion, and Navier-Stokes | test error | Helped |
| Use PDE-specific unlabeled pretraining rather than a vision-pretrained checkpoint | Neural operators · Navier-Stokes | relative error across rollout steps | Helped |
| Add similarity-based in-context demos at inference | FNO · OOD PDE benchmarks | OOD generalization | Helped |
| Use backbone features rather than final predictions to identify similar demos | Similarity-based in-context example mining · OOD PDE benchmarks | OOD performance | Hurt |
| Jointly pretrain on unlabeled Poisson and Helmholtz data | FNO · Poisson and Helmholtz | fine-tuning performance | Helped |
| Pretrain on Poisson, then fine-tune on unseen Helmholtz PDE data | FNO · Helmholtz | fine-tuning performance | Hurt |
| Increase perturbation strength for low-volume Navier-Stokes pretraining (ablation) | FNO pretraining · 2D incompressible Navier-Stokes with 8–32 samples | relative error | Helped |
Surprising findings
- Helmholtz is hard: both curves failed to improve error until over 1024 simulated samples, suspected grokking.
- Fine-tuning on unseen PDEs with padded channels performs worse than models pretrained on the same PDE.
Limitations
- Only four PDE systems and two architectures studied.
- Limited set of physics-inspired proxy tasks and augmentations (masking and blur only).
- Fine-tuning on unseen PDEs with padded channels degrades performance.
Experimental setup
Datasets: Poisson Helmholtz Reaction-Diffusion (PDEBench) Navier-Stokes (PINO) Navier-Stokes (PDEBench) ERA5 ScalarFlow Airfoil SSv2 ImageNet
Benchmarks: Poisson Helmholtz Reaction-Diffusion Navier-Stokes ERA5 ScalarFlow Airfoil
Metrics: relative L2 error generalization gap scale and shape decomposition of relative MSE
Baselines: random initialization vision-pretrained VideoMAE (SSv2) vision-pretrained FNO (ImageNet) MoCo v2 feature-based similarity retrieval
Builds on: Fourier Neural Operator VideoMAE MoCo v2 in-context operator learning
Closest papers by meaning
- Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEs (ICML 2024 · similarity 0.89)
- Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs (NeurIPS 2024 · similarity 0.88)
- Physics-Informed Deep Inverse Operator Networks for Solving PDE Inverse Problems (ICLR 2025 · similarity 0.88)
- Improved Operator Learning by Orthogonal Attention (ICML 2024 · similarity 0.88)
- Accelerating Data Generation for Neural Operators via Krylov Subspace Recycling (ICLR 2024 · similarity 0.88)
- Harnessing the Power of Neural Operators with Automatically Encoded Conservation Laws (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.