Research map / Architectures and efficiency / Attention and transformers
Language Modeling with Gated Convolutional Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace recurrent connections with gated temporal convolutions | Language model · Google Billion Word test | perplexity: 39.8 -> 38.1 | Helped |
| replace recurrent connections with gated temporal convolutions | Language model · WikiText-103 test | perplexity: 48.7 -> 44.9 | Helped |
| replace recurrent connections with gated temporal convolutions | Language model · Penn Treebank test | perplexity: 109.3 -> 108.7 | No clear effect |
| add GLU gating instead of ReLU activation (ablation) | Gated convolutional network · Google Billion Word | perplexity: about 5 perplexity points | Helped |
| add GLU gating instead of Tanh activation (ablation) | Gated convolutional network · WikiText-103 | perplexity | Helped |
| add gating units (ablation) | Tanh convolutional network · WikiText-103 | perplexity | Helped |
| replace GLU with bilinear layers (ablation) | Gated convolutional network · Google Billion Word | perplexity: GLU improves another 20 perplexity points over the bilinear model | Hurt |
| replace bilinear layers with linear convolutional layers (ablation) | Bilinear convolutional network · Google Billion Word | perplexity: more than 40 perplexity points | Hurt |
| add weight normalization (ablation) | Gated convolutional network training · Google Billion Word | convergence speed: over two times | Helped |
| add gradient clipping (ablation) | Gated convolutional network training · Google Billion Word | convergence speed | Helped |
| use bottleneck blocks (ablation) | GCNN · Google Billion Word | responsiveness (tokens/s, GPU): 2,282 -> 45,878 | Helped |
Closest papers by meaning
- Memory Architectures in Recurrent Neural Network Language Models (ICLR 2018 · similarity 0.90)
- Kernel-Based Approaches for Sequence Modeling: Connections to Neural Methods (NeurIPS 2019 · similarity 0.89)
- Recurrent Stacking of Layers for Compact Neural Machine Translation Models (AAAI 2019 · similarity 0.88)
- Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures (ICLR 2025 · similarity 0.86)
- Liger: Linearizing Large Language Models to Gated Recurrent Structures (ICML 2025 · similarity 0.86)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.