AI Paper Atlas

Research map / Graph learning / Graph self-supervised learning

GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph

Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit S, Guangzhong Sun, Xing Xie

NeurIPS 2021 · Graph learning › Graph self-supervised learning · approach: Semi-supervised node representation learning · 38 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
nest GNN layers inside transformer layers instead of cascading GNN after the PLMUniLM-base text graph encoder · DBLPP@1: 0.6934 -> 0.7267Improved
nest GNN layers inside transformers instead of cascadingUniLM-base text graph encoder · WikiP@1: 0.3712 -> 0.3952Improved
nest GNN layers inside transformers instead of cascadingUniLM-base text graph encoder · ProductP@1: 0.7570 -> 0.7786Improved
add two-stage progressive learning (masked warm-up then original data) (ablation)GraphFormers · DBLPP@1: 0.7096 -> 0.7267Improved
add two-stage progressive learning (ablation)GraphFormers · ProductP@1: 0.7688 -> 0.7786Improved
add two-stage progressive learning (ablation)GraphFormers · WikiP@1: 0.3834 -> 0.3952Improved
use unidirectional graph aggregation (simplified) instead of bidirectional (ablation)GraphFormers · DBLPP@1: 0.7225 -> 0.7267No clear effect
share GNN parameters across layers (ablation)GraphFormers · DBLPP@1: 0.7256 -> 0.7267No clear effect
add learnable position bias in GNN (ablation)GraphFormers · DBLPP@1: 0.7276 -> 0.7267No clear effect
increase number of neighbours from 1 to 5GraphFormers · DBLPP@1: 0.6485 -> 0.7267Improved
use GAT aggregator instead of Max/Mean/Att pooling in cascaded modelPLM+GNN cascaded · WikiP@1: 0.3712 -> 0.3006Hurt
use GraphFormers instead of previous production systemBing ads retrieval · Online A/B testRPM: +1.87%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy