Research map / Large language models / Language, safety and interpretability
Character n-Gram Embeddings to Improve RNN Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add char3-MS-vec to AWD-LSTM | AWD-LSTM · PTB | test perplexity: 57.3 -> 55.56 | Helped |
| Add char3-MS-vec to AWD-LSTM-DOC | AWD-LSTM-DOC · WT2 | test perplexity: 58.03 -> 57.30 | Helped |
| Add char3-MS-vec to QRNN | QRNN · WT103 | test perplexity: 33.0 -> 31.81 | Helped |
| Add char3-MS-vec to encoder and decoder | EncDec · IWSLT16 En-Fr, En-De, Fr-En, De-En | BLEU: 34.37 -> 35.48; 23.05 -> 23.27; 34.07 -> 34.43; 28.18 -> 28.86 | Helped |
| Add char3-MS-vec to encoder and decoder | EncDec · Gigaword headline generation | ROUGE: 46.77 -> 47.04; 24.87 -> 25.09; 43.58 -> 43.80 | Helped |
| Use character n-gram embeddings with multi-dimensional self-attention | AWD-LSTM · PTB infrequent-word inputs (frequency lower than 2,000) | test perplexity: 41.14 -> 40.50 | Helped |
| Replace multi-dimensional self-attention with unweighted summation (ablation) | char3-MS-vec · PTB | test perplexity: 55.56 -> 57.39 | Hurt |
| Replace multi-dimensional self-attention with standard self-attention (ablation) | char3-MS-vec · PTB | test perplexity: 55.56 -> 57.30 | Hurt |
| Exclude char n-gram embeddings C from word tying (ablation) | char3-MS-vec · PTB | test perplexity: 55.56 -> 56.19 | Hurt |
| Remove ordinary word embeddings E (ablation) | char3-MS-vec · PTB | test perplexity: 55.56 -> 58.88 | Hurt |
| Use character 4-grams instead of character 3-grams (ablation) | char n-MS-vec · WT103 | test perplexity: 31.81 -> 32.21 | Hurt |
| Use character 4-grams instead of character 3-grams (ablation) | char n-MS-vec · PTB | test perplexity: 55.56 -> 55.64 | No clear effect |
Closest papers by meaning
- LightRNN: Memory and Computation-Efficient Recurrent Neural Networks (NeurIPS 2016 · similarity 0.88)
- Character-Aware Neural Language Models (AAAI 2016 · similarity 0.87)
- Fast-Slow Recurrent Neural Networks (NeurIPS 2017 · similarity 0.86)
- Neural Bag-of-Ngrams (AAAI 2017 · similarity 0.86)
- Character-Level Language Modeling with Deeper Self-Attention (AAAI 2019 · similarity 0.86)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.