AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Pre-Training a Graph Recurrent Network for Text Understanding

Yile Wang, Linyi Yang, Zhiyang Teng, Ming Zhou, Yue Zhang

TPAMI 2025 · Architectures and efficiency › Attention and transformers · approach: Long-context KV and memory attention

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add position embeddings (ablation)graph recurrent network · WikiText-103 masked LMperplexityImproved
add layer normalization to the gates (ablation)graph recurrent network · WikiText-103 masked LMperplexityImproved
add dropout 0.1 in each layer (ablation)graph recurrent network · WikiText-103 masked LMperplexityNo clear effect
increase recurrent steps to 10 or more layers (ablation)graph recurrent network · WikiText-103 masked LMperplexity: 1/1.2/1.5/1.8 times slower at 6/8/10/12 layersMixed
increase hidden size (ablation)graph recurrent network · WikiText-103 masked LMperplexity: 1/1.4/1.8/2.2 times training time at 1280/1536/1792/2048Improved
replace Transformer with graph recurrent networkBERT-base · GLUE devaverage score: 80.18 -> 78.67Hurt
replace ALBERT-base with graph recurrent networkALBERT-base · GLUE devaverage score: 79.48 -> 78.67No clear effect
replace ALBERT-large with graph recurrent networkALBERT-large · Chinese tasks devaverage accuracy: 77.92 -> 78.78Improved
replace BERT-base with graph recurrent networkBERT-base · Chinese tasks devaverage accuracy: 78.83 -> 78.78No clear effect
replace RoBERTa-base with linear-complexity recurrent modelRoBERTa-base · Inference time (length 512)seconds: 0.051 -> 0.011Improved
replace attention with graph recurrent model: feature redundancyBERT-base · principal components for 90% information (n=100)components: 48.98 -> 52.56Improved
replace attention with graph recurrent model: feature redundancyBERT-base · principal components for 90% information (n=300)components: 103.9 -> 119.3Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy