AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

$L^3$: Large Lookup Layers

Albert Tseng, Christopher De Sa

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add L3 layers to a dense transformer800M, 1.5B, 2.6B active dense models · Wikitext2 / zero-shot downstreamperplexity and average accuracy: 800M: 22.02 -> 20.23 (2 L3); 2.6B: 15.43 -> 14.51Improved
use L3 instead of an iso-FLOP iso-sparse MoE800M and 1.5B models · FineWeb-Edu perplexity gap over denseperplexityImproved
MoE equivalent of one L3 layer at 1.5B scaleMoE · 1.5B FineWeb-EduperplexityHurt
use LZW-based embedding allocation instead of uniform allocation (ablation)L3 · 800M FineWeb-Eduperplexity improvement over denseImproved
remove per-token embedding cap (k = infinity) in LZW allocation (ablation)L3 · 800MperplexityImproved
cap embeddings per token at 512 or 256 (ablation)L3 LZW allocation · 800MperplexityMixed
tie W_K and W_V weights (ablation)L3 · 800M, 2 L3perplexityNo clear effect
vary number of L3 layers vs embeddings per layer (2x710K, 4x355K, 1x1420K) (ablation)L3 · 800Mperplexity improvement over denseNo clear effect
place L3 layer too early or too late in the model (ablation)L3 single layer 355K embeddings · 800M, 1B tokensfinal training perplexityMixed
offload L3 parameters to CPU with first L3 layer after decoder layer 1 (ablation)2.6B L3 model · inference BS=1 on one GPUtokens/s per sequence: 776 -> 692Hurt
place first offloaded L3 layer after decoder layer 5 (ablation)2.6B L3 model · inference BS=1tokens/s per sequence: 776 -> 772No clear effect
train L3 model vs dense800M model on 8xA100 · training throughputtokens/s: 155K -> 135KHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy