Research map / Graph learning / GNN architectures
GNN-LM: Language Modeling based on Global Contexts via GNN
Yuxian Meng, Shi Zong, Xiaoya Li, Xiaofei Sun, Tianwei Zhang, Fei Wu, Jiwei Li
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GNN reference-context aggregation | base LM · WikiText-103 test | perplexity: 18.7 -> 16.8 | Improved |
| Add kNN probability interpolation to GNN-LM | GNN-LM · WikiText-103 test | perplexity: 16.8 -> 14.8 | Improved |
| Add kNN probability interpolation | base LM · One Billion Word test | perplexity: 23.0 -> 22.8 | Improved |
| Add GNN reference-context aggregation | base LM · One Billion Word test | perplexity: 23.0 -> 22.7 | Improved |
| Add GNN and kNN | base LM · One Billion Word test | perplexity: 23.0 -> 22.5 | Improved |
| Add GNN reference-context aggregation | 12L Transformer-XL · Enwik8 | BPC: 1.06 -> 1.04 | Improved |
| Add kNN probability interpolation | 12L Transformer-XL · Enwik8 | BPC: 1.06 -> 1.04 | Improved |
| Add GNN and kNN | 12L Transformer-XL · Enwik8 | BPC: 1.06 -> 1.03 | Improved |
| Increase neighbors per source token from 8 to 128 (ablation) | GNN-LM · WikiText-103 test | perplexity | Improved |
| Use GNN representations for both kNN query and key instead of Transformer representations (ablation) | GNN-LM + kNN · WikiText-103 test | perplexity: 14.82 -> 14.76 | Improved |
| Use GNN representations for kNN key only (ablation) | GNN-LM + kNN · WikiText-103 test | perplexity: 14.82 -> 15.16 | Hurt |
| Use GNN representations for kNN query only (ablation) | GNN-LM + kNN · WikiText-103 test | perplexity: 14.82 -> 14.97 | Hurt |
Closest papers by meaning
- Large Language Model Meets Graph Neural Network in Knowledge Distillation (AAAI 2025 · similarity 0.89)
- GLBench: A Comprehensive Benchmark for Graph with Large Language Models (NeurIPS 2024 · similarity 0.88)
- GOFA: A Generative One-For-All Model for Joint Graph Language Modeling (ICLR 2025 · similarity 0.88)
- Graph Neural Prompting with Large Language Models (AAAI 2024 · similarity 0.88)
- LoReC: Rethinking Large Language Models for Graph Data Analysis (NeurIPS 2026 · similarity 0.88)
- LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
