Research map / Probabilistic and scientific modeling / Differential equations and PDEs

Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs

NeurIPS 2024 · Probabilistic and scientific modeling › Differential equations and PDEs · approach: Neural operators for general geometries · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoDA-NO, a transformer neural operator that tokenizes PDE solutions along the codomain (physical variables) with function-space attention, normalization and positional encoding, pretrained with masked reconstruction for few-shot adaptation to multiphysics PDEs.

Problem

Neural operators struggle to solve coupled multiphysics PDEs with complex geometries, interacting variables, and scarce high-resolution data, and existing PDE foundation models cannot handle varying numbers of physical variables or irregular meshes.

Approach

Treat each physical variable (codomain component) as a token function and extend self-attention, positional encoding, and normalization to function spaces (with FNO/GNO operators for Q/K/V), so a single model can be pretrained by masked reconstruction on one PDE system and fine-tuned to systems with new variables.

Key results

• Pretrained CoDA-NO achieves 0.024 L2 error on NS at Re=400 with 5 few-shot samples, versus 0.200 for GINO and 0.038 for GNN (best baseline).
• On Rayleigh-Benard with Ra=12e3 and 5 shots, pretrained CoDA-NO reaches 0.016 versus 0.049 for U-Net, roughly a factor-of-two or more reduction.
• On PDEBench, CoDA-NO reduces test error by up to 43% over FNO while using only about 2% of its parameters.

Measured effects reported in the paper

Surprising findings

• Removing the normalization layer prevented CoDA-NO from converging.
• Larger baseline models (e.g., GINO at 200M parameters) overfit and perform worse in few-shot settings.

Limitations

• Performance on target PDEs depends on the number of training examples.
• No physics-informed constraints are integrated.
• Higher computational cost and parameter count than several baselines.

Experimental setup

Datasets: NS dataset (TurtleFSI generated) NS+EW fluid-structure interaction dataset (TurtleFSI generated) Rayleigh-Benard convection dataset (generated) PDEBench subset

Benchmarks: fluid-structure interaction NS and NS+EW datasets Rayleigh-Benard convection PDEBench

Metrics: L2 relative error zero-shot super-resolution error parameter count inference and training time

Baselines: GINO DeepONet GNN ViT U-Net FNO

Builds on: Fourier neural operator graph neural operator GINO transformer self-attention instance normalization

Closest papers by meaning

• Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning (NeurIPS 2024 · similarity 0.88)
• Improved Operator Learning by Orthogonal Attention (ICML 2024 · similarity 0.87)
• Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning (ICML 2024 · similarity 0.87)
• Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discovery (ICML 2025 · similarity 0.86)
• RIGNO: A Graph-based Framework For Robust And Accurate Operator Learning For PDEs On Arbitrary Domains (NeurIPS 2025 · similarity 0.86)
• DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training (ICML 2024 · similarity 0.86)