Research map / Architectures and efficiency / RNNs and normalization
Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax with continuous word-embedding outputs trained with NLLvMF | sequence-to-sequence machine translation · IWSLT16 fr--en | training time: 4h -> 1.9h | Helped |
| replace softmax with continuous word-embedding outputs trained with NLLvMF | sequence-to-sequence machine translation · IWSLT16 fr--en | BLEU: 31.4 -> 32.1 | Mixed |
| replace softmax with continuous word-embedding outputs trained with NLLvMF | sequence-to-sequence machine translation · IWSLT16 de--en | BLEU: 25.8 -> 25.1 | Hurt |
| replace softmax with continuous word-embedding outputs trained with NLLvMF | sequence-to-sequence machine translation · IWSLT16 en--fr | BLEU: 31.0 -> 31.7 | Helped |
| use NLLvMF rather than L2 regression (ablation) | word2vec continuous-output model · IWSLT16 fr--en | BLEU: 27.2 -> 29.5 | Helped |
| use cosine loss rather than L2 regression (ablation) | word2vec continuous-output model · IWSLT16 fr--en | BLEU: 27.2 -> 29.1 | Helped |
| use fasttext embeddings rather than word2vec embeddings (ablation) | MaxMargin continuous-output model · IWSLT16 fr--en | BLEU: 29.6 -> 31.0 | Helped |
| tie decoder input embeddings to pretrained target embeddings (ablation) | fasttext NLLvMF model · IWSLT16 fr--en | BLEU: 30.4 -> 32.1 | Helped |
| add reg2 alongside reg1 (ablation) | fasttext NLLvMF model · IWSLT16 fr--en | BLEU: 30.4 -> 32.1 | Helped |
| use NLLvMF continuous outputs rather than BPE softmax | machine translation · WMT16 de--en | BLEU: 30.1 -> 28.8 | Hurt |
Closest papers by meaning
- MLE-Guided Parameter Search for Task Loss Minimization in Neural Sequence Modeling (AAAI 2021 · similarity 0.88)
- From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables (AAAI 2022 · similarity 0.88)
- Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (ICLR 2018 · similarity 0.88)
- Pointer Sentinel Mixture Models (ICLR 2017 · similarity 0.88)
- Minimum Divergence vs. Maximum Margin: an Empirical Comparison on Seq2Seq Models (ICLR 2019 · similarity 0.88)
- Accelerating Neural Machine Translation with Partial Word Embedding Compression (AAAI 2021 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.