Research map / Architectures and efficiency / RNNs and normalization
Language Modeling with Recurrent Highway Hypernetworks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine RHN with hypernetwork scaling (HyperRHN) vs plain RHN and LSTM baselines | recurrent highway hypernetwork · Penn Treebank character-level | BPC test: 1.19 vs 1.31 (LSTM) | Helped |
| add grid (horizontal) connections to RHN instead of depth-wise highway stacking (ablation) | RHN · Penn Treebank character-level | BPC: 1.40 bpc vs HyperRHN 1.19 | Hurt |
| skip layer normalization and use only recurrent dropout | RHN-based hypernetwork · Penn Treebank character-level | BPC test: 1.19 without layer norm vs 1.25 for Layer Norm HyperLSTM | No clear effect |
| plain RHN vs 2-layer LSTM at comparable size | RHN · Penn Treebank character-level | BPC val: 1.24 vs 1.31 val (LSTM) | Helped |
Closest papers by meaning
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.88)
- Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems (NeurIPS 2025 · similarity 0.88)
- Liger: Linearizing Large Language Models to Gated Recurrent Structures (ICML 2025 · similarity 0.87)
- Recursive Language Models (NeurIPS 2026 · similarity 0.87)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.87)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.