Research map / Architectures and efficiency / Mixture of experts
$L^3$: Large Lookup Layers
Albert Tseng, Christopher De Sa
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add L3 layers to a dense transformer | 800M, 1.5B, 2.6B active dense models · Wikitext2 / zero-shot downstream | perplexity and average accuracy: 800M: 22.02 -> 20.23 (2 L3); 2.6B: 15.43 -> 14.51 | Improved |
| use L3 instead of an iso-FLOP iso-sparse MoE | 800M and 1.5B models · FineWeb-Edu perplexity gap over dense | perplexity | Improved |
| MoE equivalent of one L3 layer at 1.5B scale | MoE · 1.5B FineWeb-Edu | perplexity | Hurt |
| use LZW-based embedding allocation instead of uniform allocation (ablation) | L3 · 800M FineWeb-Edu | perplexity improvement over dense | Improved |
| remove per-token embedding cap (k = infinity) in LZW allocation (ablation) | L3 · 800M | perplexity | Improved |
| cap embeddings per token at 512 or 256 (ablation) | L3 LZW allocation · 800M | perplexity | Mixed |
| tie W_K and W_V weights (ablation) | L3 · 800M, 2 L3 | perplexity | No clear effect |
| vary number of L3 layers vs embeddings per layer (2x710K, 4x355K, 1x1420K) (ablation) | L3 · 800M | perplexity improvement over dense | No clear effect |
| place L3 layer too early or too late in the model (ablation) | L3 single layer 355K embeddings · 800M, 1B tokens | final training perplexity | Mixed |
| offload L3 parameters to CPU with first L3 layer after decoder layer 1 (ablation) | 2.6B L3 model · inference BS=1 on one GPU | tokens/s per sequence: 776 -> 692 | Hurt |
| place first offloaded L3 layer after decoder layer 5 (ablation) | 2.6B L3 model · inference BS=1 | tokens/s per sequence: 776 -> 772 | No clear effect |
| train L3 model vs dense | 800M model on 8xA100 · training throughput | tokens/s: 155K -> 135K | Hurt |
Closest papers by meaning
- MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup (NeurIPS 2026 · similarity 0.89)
- Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders (NeurIPS 2025 · similarity 0.89)
- Scaling Embedding Layers in Language Models (NeurIPS 2025 · similarity 0.89)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.88)
- Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing (ICML 2026 · similarity 0.88)
- Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
