Research map / Architectures and efficiency / RNNs and normalization
Accelerating Neural Machine Translation with Partial Word Embedding Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply P-VQ to the target embedding and softmax | Transformer NMT · WMT17, WMT18, WMT19 | BLEU: -0.04 | No clear effect |
| Apply P-VQ to the target embedding and softmax | Transformer NMT · Chinese-to-English target embedding | parameters: 74.35% reduction | Helped |
| Use lookup-based shared-part logits with P-VQ | Transformer NMT softmax · Chinese-to-English | FLOPs: 74.42% reduction | Helped |
| Apply P-VQ | Transformer NMT · WMT14 | BLEU: 39.87 -> 39.58 | Hurt |
| Add gradual curriculum compression (ablation) | P-VQ · WMT17 | BLEU: 23.82 -> 24.26 | Helped |
| Increase the window size from 128 to 384 (ablation) | P-VQ · WMT17 | BLEU: 24.36 -> 24.20 | No clear effect |
| Increase window size to 448 (ablation) | P-VQ · WMT17 | BLEU: 24.20 -> 23.47 | Hurt |
| Reduce group count from 512 to 32 (ablation) | P-VQ · WMT17 | BLEU: 24.32 -> 23.28 | Hurt |
| Use a partial shared embedding while retaining word-specific dimensions | Vector-quantized NMT softmax · Chinese-to-English WMT | BLEU: P-VQ 24.26; DC 23.64; SE 23.65 on WMT17 | Helped |
| Use LightRNN embedding compression | Transformer NMT · WMT17, WMT18, WMT19 | BLEU: 19.53; 18.15; 19.68 | Hurt |
Closest papers by meaning
- From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables (AAAI 2022 · similarity 0.91)
- Compressing Word Embeddings via Deep Compositional Code Learning (ICLR 2018 · similarity 0.88)
- Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs (ICLR 2019 · similarity 0.88)
- All Word Embeddings from One Embedding (NeurIPS 2020 · similarity 0.88)
- Online Embedding Compression for Text Classification Using Low Rank Matrix Factorization (AAAI 2019 · similarity 0.87)
- Multilingual Neural Machine Translation With Soft Decoupled Encoding (ICLR 2019 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.