Research map / Architectures and efficiency / RNNs and normalization
Slim Embedding Layers for Recurrent Neural Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress input embeddings to 20% and use dropout 0.1 | LSTM on PTB, 300 hidden nodes · PTB test | perplexity: 88.56 -> 88.19 | Helped |
| Compress input embeddings to 5% and use dropout 0.1 | LSTM on PTB, 650 hidden nodes · PTB test | perplexity: 82.59 -> 81.14 | Helped |
| Reduce input embedding compression ratio from 1 to 1/8 (ablation) | LSTM on 44M · 44M test | perplexity: minimum of 96.30 | Helped |
| Increase input embedding compression to 1/512 (ablation) | LSTM on 44M · 44M test | perplexity: 103.61 | Hurt |
| Use only one sub-vector per word (K=1), causing multiple words to share identical embeddings (ablation) | SE on 44M · 44M test | perplexity | Hurt |
| Use xxhash instead of the paper's random mapping (ablation) | SE on 44M · 44M test | perplexity: 95.73 -> 97.35 | Hurt |
| Compress both input and output embeddings more aggressively (ablation) | SE on 44M · 44M test | perplexity | Hurt |
| Use K=4 sub-vectors rather than K=1 at fixed 1/8 compression (ablation) | SE on 44M with both embedding layers compressed · 44M test | perplexity | Helped |
| Use one sub-vector per word (K=1) at fixed 1/8 compression (ablation) | SE on 44M with both embedding layers compressed · 44M test | perplexity | Hurt |
| Replace NCE with importance sampling (ablation) | 3-layer 2048 LSTM SE on BillionW · BillionW | perplexity: 39.5 -> 38.3 | Helped |
| Add a normal LSTM language model as a reranking feature | Moses reranking · WMT12 newstest2011 | BLEU: 25.69 -> 26.11 | Helped |
| Add a compressed LSTM language model as a reranking feature | Moses reranking · WMT12 newstest2011 | BLEU: 25.69 -> 26.25 | Helped |
Closest papers by meaning
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.87)
- Scaling Embedding Layers in Language Models (NeurIPS 2025 · similarity 0.87)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.87)
- Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning (NeurIPS 2025 · similarity 0.87)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.86)
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.