Research map / Architectures and efficiency / Attention and transformers
Hash Layers For Large Sparse Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Switch routing with fixed balanced token hashing | single-layer 64-expert Transformer · Pushshift.io Reddit | Valid PPL: 23.65 -> 23.16 | Helped |
| replace Switch routing with fixed balanced token hashing | single-layer 64-expert Transformer · RoBERTa+cc100en | Valid PPL: 27.41 -> 26.99 | Helped |
| increase expert count from 64 to 128 (ablation) | Hash Layer · Pushshift.io Reddit | Valid PPL: 23.16 -> 22.89 | Helped |
| use 5 sparse layers with 16 experts instead of one layer with 64 experts (ablation) | Hash Layer · Pushshift.io Reddit | Valid PPL: 23.16 -> 23.21 | Mixed |
| replace standard dictionary with BB BPE dictionary (ablation) | Hash Layer 1x16 · Wikitext-103 | Valid PPL: 32.32 (Std. Dict); 11.58 (BB Dict) | Mixed |
| use multiple independently indexed hashes instead of one (ablation) | Hash Layer 1x32 · Pushshift.io Reddit | Valid PPL: 23.58 -> 23.48 (2 hashes); 23.38 (4 hashes); 23.28 (8 hashes) | Helped |
| use fixed random assignment instead of balanced assignment (ablation) | Hash Layer 1x64 · Pushshift.io Reddit | Valid PPL: 23.16 -> 23.22 | No clear effect |
| cluster similar tokens into the same expert (ablation) | Hash Layer 1x64 · Pushshift.io Reddit | Valid PPL: 23.22 -> 23.90 | Hurt |
| disperse tokens within clusters across experts (ablation) | cluster-based Hash Layer 1x64 · Pushshift.io Reddit | Valid PPL: 23.90 -> 23.17 | Helped |
| hash by sequence position instead of input content (ablation) | Hash Layer 1x64 · Pushshift.io Reddit | Valid PPL: 24.90 -> 25.07 | Hurt |
| hash using current and previous tokens instead of only current token (ablation) | Hash Layer 1x64 · Pushshift.io Reddit | Valid PPL: 23.16 -> 24.19 | Hurt |
| hash using the previous token instead of the current token (ablation) | Hash Layer 1x64 · Pushshift.io Reddit | Valid PPL: 23.16 -> 24.16 | Hurt |
Closest papers by meaning
- LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing (NeurIPS 2024 · similarity 0.86)
- Learning and Transferring Sparse Contextual Bigrams with Linear Transformers (NeurIPS 2024 · similarity 0.85)
- KALAHash: Knowledge-Anchored Low-Resource Adaptation for Deep Hashing (AAAI 2025 · similarity 0.85)
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.85)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.85)
- Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.