Research map / Architectures and efficiency / RNNs and normalization
From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace trained embeddings with random embeddings | Transformer with RWE+linear, 6 layers · WMT En-Fr | BLEU: 38.41 -> 35.76 | Hurt |
| Replace trained embeddings with random embeddings | Transformer with RWE+linear, 6 layers · WMT En-De | BLEU: 27.03 -> 23.04 | Hurt |
| Increase RWE model depth from 6 to 8 layers (ablation) | Transformer with random embeddings · WMT En-Fr | BLEU: 35.76 -> 37.11 | Helped |
| Increase RWE model depth from 6 to 8 layers (ablation) | Transformer with random embeddings · WMT En-De | BLEU: 23.04 -> 25.83 | Helped |
| Use GPQ instead of PQ, incorporating cluster variance (ablation) | Transformer with structured partitioning · WMT En-Fr | BLEU: 39.95 -> 40.04 | Helped |
| Use GPQ instead of PQ, incorporating cluster variance (ablation) | Transformer with unified partitioning · WMT En-Fr | BLEU: 40.02 -> 40.51 | Helped |
| Use GPQ instead of PQ, incorporating cluster variance (ablation) | Transformer with unified partitioning · WMT En-De | BLEU: 26.87 -> 27.35 | Helped |
| Use GPQ instead of PQ, incorporating cluster variance (ablation) | Transformer with unified partitioning · IWSLT Pt-En | BLEU: 39.34 -> 40.36 | Helped |
| Use unified rather than structured partitioning with GPQ (ablation) | Transformer · WMT En-Fr | BLEU: 40.04 -> 40.51 | Helped |
| Use unified rather than structured partitioning with GPQ (ablation) | Transformer · WMT En-De | BLEU: 27.11 -> 27.35 | Helped |
| Use unified rather than structured partitioning with GPQ (ablation) | Transformer · IWSLT Pt-En | BLEU: 40.16 -> 40.36 | Helped |
| Use GPQ rather than PQ to add cluster variance (ablation) | LSTM with unified partitioning · WMT En-Fr | BLEU: 32.05 -> 33.82 | Helped |
Closest papers by meaning
- Unsupervised Neural Machine Translation (ICLR 2018 · similarity 0.91)
- All Word Embeddings from One Embedding (NeurIPS 2020 · similarity 0.91)
- Accelerating Neural Machine Translation with Partial Word Embedding Compression (AAAI 2021 · similarity 0.91)
- Neural Machine Translation with Joint Representation (AAAI 2020 · similarity 0.90)
- Rethinking Embedding Coupling in Pre-trained Language Models (ICLR 2021 · similarity 0.89)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.