Research map / Architectures and efficiency / RNNs and normalization
Character-Aware Neural Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace word input embeddings with character-CNN representations | LSTM language model · Penn Treebank test | perplexity: 97.6 -> 92.3 | Helped |
| Replace word input embeddings with character-CNN representations | LSTM language model · Penn Treebank test | perplexity: 85.4 -> 78.9 | Helped |
| Use character inputs instead of morpheme inputs | Large LSTM · Data-s: Czech, German, Spanish, French, Russian | perplexity: 398 -> 371; 263 -> 239; 177 -> 165; 196 -> 184; 271 -> 261 | Helped |
| Use character inputs instead of morpheme inputs | Large LSTM · Data-s: Arabic | perplexity: 148 -> 148 | No clear effect |
| Use character inputs instead of morpheme inputs | Small LSTM · Data-l: Czech, German, Russian, English | perplexity: 615 -> 578; 331 -> 305; 331 -> 313; 233 -> 216 | Helped |
| Use character inputs instead of word inputs | Small LSTM · Data-l: Spanish and French | perplexity: 186 -> 169; 202 -> 190 | Helped |
| Add highway layers after the CharCNN (ablation) | LSTM-Char-Small · Penn Treebank test | perplexity: 100.3 -> 90.1 | Helped |
| Add highway layers after the CharCNN (ablation) | LSTM-Char-Large · Penn Treebank test | perplexity: 84.6 -> 78.9 | Helped |
| Replace two highway layers with one (ablation) | LSTM-Char-Small · Penn Treebank test | perplexity: 90.1 -> 92.3 | Hurt |
| Replace two highway layers with one (ablation) | LSTM-Char-Large · Penn Treebank test | perplexity: 78.9 -> 79.7 | Hurt |
| Replace highway layers with an MLP layer (ablation) | LSTM-Char-Small · Penn Treebank test | perplexity: 90.1 -> 111.2 | Hurt |
| Replace highway layers with an MLP layer (ablation) | LSTM-Char-Large · Penn Treebank test | perplexity: 78.9 -> 92.6 | Hurt |
Closest papers by meaning
- Character-Level Language Modeling with Deeper Self-Attention (AAAI 2019 · similarity 0.91)
- Adaptive Input Representations for Neural Language Modeling (ICLR 2019 · similarity 0.89)
- Training Neural Networks as Recognizers of Formal Languages (ICLR 2025 · similarity 0.88)
- Memory Architectures in Recurrent Neural Network Language Models (ICLR 2018 · similarity 0.88)
- On the State of the Art of Evaluation in Neural Language Models (ICLR 2018 · similarity 0.88)
- Mogrifier LSTM (ICLR 2020 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.