Research map / Architectures and efficiency / Attention and transformers
Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation
Xiang Kong, Qizhe Xie, Zihang Dai, Eduard Hovy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MoS with BPE word coding | LSTM seq2seq · IWSLT 2014 DE-EN | BLEU: 27.41 ± 0.15 -> 28.91 ± 0.06 | Improved |
| Add MoS with BPE word coding | NIC · MSCOCO | CIDEr: 88.50 ± 0.47 -> 89.26 ± 0.11 | Improved |
| Add MoS with BPE word coding | Transformer Big · WMT 2014 EN-DE | BLEU: 28.7 -> 29.6 | Improved |
| Add MoS with BPE word coding | Transformer Big · WMT 2014 EN-FR | BLEU: 41.7 -> 42.1 | Improved |
| Use a learned Hybrid-LightRNN table of size 10k rather than 5k (ablation) | Hybrid-LightRNN-MoS · IWSLT validation | BLEU: 29.69 -> 30.07 | Improved |
| Use a learned Hybrid-LightRNN table of size 5k rather than 1k (ablation) | Hybrid-LightRNN-MoS · IWSLT validation | BLEU: 28.73 -> 29.69 | Improved |
| Separately model frequent words (ablation) | LightRNN · IWSLT validation | BLEU: 27.39 -> 30.07 | Improved |
| Learn the encoding table rather than use a frequency-sorted table (ablation) | Hybrid-LightRNN-MoS · IWSLT validation | BLEU: 25.84 -> 27.39 | Improved |
| Learn the encoding table rather than use a random table (ablation) | Hybrid-LightRNN-MoS · IWSLT validation | BLEU: 24.98 -> 27.39 | Improved |
| Disable BPE OOV translation and replace generated OOVs with UNK (ablation) | BPE-MoS · IWSLT validation | BLEU: 30.19 -> 30.14 | Hurt |
Closest papers by meaning
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.85)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.85)
- LoLCATs: On Low-Rank Linearizing of Large Language Models (ICLR 2025 · similarity 0.85)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.84)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.84)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
