Research map / Architectures and efficiency / RNNs and normalization
Learning Intrinsic Sparse Structures within Long Short-Term Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn smaller hidden structures with group-Lasso ISS | Stacked LSTM · Penn TreeBank | test perplexity: 78.57 -> 78.65 | No clear effect |
| Learn compact structures instead of directly designing the same small hidden sizes (ablation) | Stacked LSTM · Penn TreeBank | test perplexity: 85.66 vs 78.65 | Helped |
| Use a stronger ISS sparsity setting, yielding hidden sizes 381 and 535 (ablation) | Stacked LSTM · Penn TreeBank | test perplexity: 78.57 -> 76.03 | Helped |
| Apply independent L1 weight regularization (ablation) | Stacked LSTM · Penn TreeBank | test perplexity: 78.57 -> 78.60 | No clear effect |
| Reduce RHN width to 726 using ISS (ablation) | RHN · Penn TreeBank | test perplexity: 65.4 -> 65.0 | Helped |
| Reduce RHN width to 517 using ISS (ablation) | RHN · Penn TreeBank | test perplexity: 65.4 -> 65.4 | No clear effect |
| Reduce RHN width to 403 using ISS (ablation) | RHN · Penn TreeBank | test perplexity: 65.4 -> 67.7 | Hurt |
| Reduce RHN width to 328 using ISS (ablation) | RHN · Penn TreeBank | test perplexity: 65.4 -> 71.2 | Hurt |
| Fine-tune BiDAF with ISS regularization to 2.08M weights | BiDAF · SQuAD | F1: 77.85 -> 76.71 | Hurt |
| Increase ISS regularization to reach 1.03M weights (ablation) | BiDAF · SQuAD | F1: 77.85 -> 75.47 | Hurt |
| Train BiDAF from scratch with ISS regularization to 1.17M weights | BiDAF · SQuAD | F1: 77.85 -> 76.22 | Hurt |
| Threshold small weights at 0.02 after training (ablation) | BiDAF · SQuAD | ExactMatch and F1 | No clear effect |
Closest papers by meaning
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.87)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.86)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.86)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.86)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.86)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.