Research map / Graph learning / Graph self-supervised learning
Supercharging Graph Transformers with Advective Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdvDIFFormer, a graph Transformer derived from advective diffusion PDEs that combines global attention (non-local diffusion) with adjacency-based advection, with theory bounding generalization error under topological shifts.
Problem
Generalization of graph learning models under topological distribution shifts (testing graph structures differ from training), which prior work has largely not studied.
Approach
Models node representations as the closed-form solution of an advective diffusion equation, where a learned global attention coupling captures latent non-local interactions and the normalized adjacency acts as an advection velocity for observed structure; the matrix exponential is approximated via Padé-Chebyshev linear solves (AdvDIFFormer-i) or a finite geometric series (AdvDIFFormer-s).
Key results
- On Arxiv temporal splits, AdvDIFFormer-s reaches 53.41% (2018), 51.53% (2019), 49.64% (2020), outperforming GRAND (52.45/50.18/48.01) and other baselines.
- On DPPIN protein interaction networks, AdvDIFFormer-s achieves the best average node regression RMSE of 0.574 versus 0.637 for DIFFormer.
- Theory shows the advective diffusion model's representation change under topology perturbation can be bounded by arbitrary polynomial order, whereas local diffusion has an exponential bound.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add advective diffusion with local message passing and global attention | AdvDIFFormer-s · Arxiv (2018) | accuracy: 52.45 $\pm$ 0.27 -> 53.41 $\pm$ 0.48 | Helped |
| Add advective diffusion with local message passing and global attention | AdvDIFFormer-s · Arxiv (2019) | accuracy: 50.18 $\pm$ 0.18 -> 51.53 $\pm$ 0.60 | Helped |
| Add advective diffusion with local message passing and global attention | AdvDIFFormer-s · Arxiv (2020) | accuracy: 48.01 $\pm$ 0.24 -> 49.64 $\pm$ 0.54 | Helped |
| Add advective diffusion with local message passing and global attention | AdvDIFFormer-s · Twitch (avg) | ROC-AUC: 62.13 $\pm$ 0.34 -> 62.51 $\pm$ 0.07 | Helped |
| Use AdvDIFFormer-s instead of DIFFormer | DPPIN node regression · Test Average | RMSE: 0.637 $\pm$ 0.034 -> 0.574 $\pm$ 0.028 | Helped |
| Use AdvDIFFormer-s instead of DIFFormer | DPPIN node regression · Test Worst | RMSE: 0.710 $\pm$ 0.028 -> 0.644 $\pm$ 0.040 | Helped |
| Use AdvDIFFormer-i instead of GraphGPS | DPPIN edge regression · Test Average | RMSE: 0.182 $\pm$ 0.007 -> 0.166 $\pm$ 0.006 | Helped |
| Use AdvDIFFormer-i instead of DIFFormer | DPPIN edge regression · Test Worst | RMSE: 0.197 $\pm$ 0.003 -> 0.184 $\pm$ 0.011 | Helped |
| Reduce the advection weight beta; beta=0 removes advection (ablation) | AdvDIFFormer-s on DPPIN node regression · DPPIN | RMSE | Helped |
| Reduce the advection weight beta (ablation) | AdvDIFFormer-s on DPPIN edge regression · DPPIN | RMSE | Helped |
| Set beta too small (below 0.5) or too large (above 2.0) (ablation) | AdvDIFFormer-s on Arxiv · Arxiv | accuracy | Hurt |
Surprising findings
- On DPPIN node regression, setting the advection weight beta=0 (dropping observed graph structure) gave the best performance, suggesting pure global attention can generalize better there.
Limitations
- Theoretical guarantees rely on assumptions such as injectivity of g and a stated data-generation hypothesis, and the bound does not guarantee error in practice.
- Only linear diffusion is used; the nonlinear time-dependent variant is left for future work.
- Performance depends on the advection weight beta, which must be tuned per dataset and task.
Experimental setup
Datasets: Arxiv Twitch DPPIN synthetic stochastic block model graphs molecular coarse-grained mapping data
Benchmarks: Arxiv Twitch DPPIN
Metrics: accuracy ROC-AUC RMSE MSE
Baselines: GCN GAT SGC GDC GRAND A-DGNs CDE GraphTrans GraphGPS DIFFormer MLP
Builds on: DIFFormer GRAND graph diffusion equations SGC
Closest papers by meaning
- Feature Transportation Improves Graph Neural Networks (AAAI 2024 · similarity 0.88)
- Graph Generation with Diffusion Mixture (ICML 2024 · similarity 0.86)
- Weisfeiler and Lehman Go Paths: Learning Topological Features via Path Complexes (AAAI 2024 · similarity 0.86)
- Hyperbolic-PDE GNN: Spectral Graph Neural Networks in the Perspective of A System of Hyperbolic Partial Differential Equations (ICML 2025 · similarity 0.86)
- A Generalized Neural Diffusion Framework on Graphs (AAAI 2024 · similarity 0.85)
- Learning Divergence Fields for Shift-Robust Graph Representations (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.