Research map / Architectures and efficiency / RNNs and normalization
Incrementally Learning the Hierarchical Softmax Function for Neural Language Models
Hao Peng, Jianxin Li, Yangqiu Song, Yaopeng Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| handle Huffman tree change with prefix matching instead of reserving old tree (ablation) | incremental CBOW · vector MSE vs global training | MSE: 7.84 × 10−3 -> 6.66 × 10−3 | Improved |
| handle Huffman tree change with prefix matching instead of reserving old tree (ablation) | incremental Skip-gram · vector MSE vs global training | MSE: 7.83 × 10−3 -> 6.68 × 10−3 | Improved |
| incremental update instead of full retraining | Skip-gram · training time | speedup: up to 30 times | Improved |
| incremental update instead of full retraining | CBOW · training time | speedup: up to 10 times | Improved |
| incremental update vs global retraining | word vectors · word similarity benchmarks | Spearman correlation | No clear effect |
| incremental update vs global retraining | word vectors · dependency parsing | UAS/LAS | No clear effect |
| reserve old tree instead of updating it (ablation) | incremental training · dependency parsing | UAS/LAS | Hurt |
| increase update corpus size | incremental training · training speedup | speedup | Hurt |
Closest papers by meaning
- COPAL: Continual Pruning in Large Language Generative Models (ICML 2024 · similarity 0.85)
- Evolving Subnetwork Training for Large Language Models (ICML 2024 · similarity 0.85)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.85)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.85)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.84)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
