AI Paper Atlas

Research map / Graph learning / GNN architectures

GNN-LM: Language Modeling based on Global Contexts via GNN

Yuxian Meng, Shi Zong, Xiaoya Li, Xiaofei Sun, Tianwei Zhang, Fei Wu, Jiwei Li

ICLR 2022 · Graph learning › GNN architectures · approach: GNN generalization theory · 18 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add GNN reference-context aggregationbase LM · WikiText-103 testperplexity: 18.7 -> 16.8Improved
Add kNN probability interpolation to GNN-LMGNN-LM · WikiText-103 testperplexity: 16.8 -> 14.8Improved
Add kNN probability interpolationbase LM · One Billion Word testperplexity: 23.0 -> 22.8Improved
Add GNN reference-context aggregationbase LM · One Billion Word testperplexity: 23.0 -> 22.7Improved
Add GNN and kNNbase LM · One Billion Word testperplexity: 23.0 -> 22.5Improved
Add GNN reference-context aggregation12L Transformer-XL · Enwik8BPC: 1.06 -> 1.04Improved
Add kNN probability interpolation12L Transformer-XL · Enwik8BPC: 1.06 -> 1.04Improved
Add GNN and kNN12L Transformer-XL · Enwik8BPC: 1.06 -> 1.03Improved
Increase neighbors per source token from 8 to 128 (ablation)GNN-LM · WikiText-103 testperplexityImproved
Use GNN representations for both kNN query and key instead of Transformer representations (ablation)GNN-LM + kNN · WikiText-103 testperplexity: 14.82 -> 14.76Improved
Use GNN representations for kNN key only (ablation)GNN-LM + kNN · WikiText-103 testperplexity: 14.82 -> 15.16Hurt
Use GNN representations for kNN query only (ablation)GNN-LM + kNN · WikiText-103 testperplexity: 14.82 -> 14.97Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy