Research map / Architectures and efficiency / Attention and transformers
Pre-Training a Graph Recurrent Network for Text Understanding
Yile Wang, Linyi Yang, Zhiyang Teng, Ming Zhou, Yue Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add position embeddings (ablation) | graph recurrent network · WikiText-103 masked LM | perplexity | Improved |
| add layer normalization to the gates (ablation) | graph recurrent network · WikiText-103 masked LM | perplexity | Improved |
| add dropout 0.1 in each layer (ablation) | graph recurrent network · WikiText-103 masked LM | perplexity | No clear effect |
| increase recurrent steps to 10 or more layers (ablation) | graph recurrent network · WikiText-103 masked LM | perplexity: 1/1.2/1.5/1.8 times slower at 6/8/10/12 layers | Mixed |
| increase hidden size (ablation) | graph recurrent network · WikiText-103 masked LM | perplexity: 1/1.4/1.8/2.2 times training time at 1280/1536/1792/2048 | Improved |
| replace Transformer with graph recurrent network | BERT-base · GLUE dev | average score: 80.18 -> 78.67 | Hurt |
| replace ALBERT-base with graph recurrent network | ALBERT-base · GLUE dev | average score: 79.48 -> 78.67 | No clear effect |
| replace ALBERT-large with graph recurrent network | ALBERT-large · Chinese tasks dev | average accuracy: 77.92 -> 78.78 | Improved |
| replace BERT-base with graph recurrent network | BERT-base · Chinese tasks dev | average accuracy: 78.83 -> 78.78 | No clear effect |
| replace RoBERTa-base with linear-complexity recurrent model | RoBERTa-base · Inference time (length 512) | seconds: 0.051 -> 0.011 | Improved |
| replace attention with graph recurrent model: feature redundancy | BERT-base · principal components for 90% information (n=100) | components: 48.98 -> 52.56 | Improved |
| replace attention with graph recurrent model: feature redundancy | BERT-base · principal components for 90% information (n=300) | components: 103.9 -> 119.3 | Improved |
Closest papers by meaning
- GOFA: A Generative One-For-All Model for Joint Graph Language Modeling (ICLR 2025 · similarity 0.86)
- UniGTE: Unified Graph–Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains (NeurIPS 2025 · similarity 0.86)
- How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias (ICML 2025 · similarity 0.86)
- On the Generalization Ability of Next-Token-Prediction Pretraining (ICML 2025 · similarity 0.86)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.86)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
