Research map / Architectures and efficiency / Attention and transformers
Block Transformer: Global-to-Local Language Modeling for Fast Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Block Transformer, a global-to-local subword-level architecture (block decoder for coarse global attention, token decoder for local attention within blocks) that cuts KV cache overheads and yields 10-20x inference throughput at equal perplexity.
Problem
Autoregressive transformer inference is bottlenecked by prefill latency and by KV cache memory I/O that grows with sequence length, causing quadratic memory reads during batch decoding.
Approach
Aggregate subword tokens into fixed-size blocks so global self-attention runs over block embeddings at lower layers, then decode tokens within each block with local attention only, conditioned on a single context embedding from the block decoder; the token decoder thus needs no global KV cache or prefill, and a 1:1 parameter split between decoders is found to be best.
Key results
- Block Transformers reach 10-20x (up to 25x in some settings) higher inference throughput than vanilla Pythia-style transformers at comparable loss/zero-shot performance.
- A 1:1 parameter split between block and token decoders gives a U-shaped loss curve with best performance at 1:1 for block length 4.
- Uptraining from a pretrained vanilla model recovers near-full performance with about 10% of the original training steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace vanilla attention with global-to-local block attention | Pythia · Prefill throughput | 1K tokens per second: 2.3 -> 21.0 | Helped |
| replace vanilla attention with global-to-local block attention | Pythia · Decode throughput | 1K tokens per second: 6.2 -> 44.1 | Helped |
| replace vanilla attention with global-to-local block attention | Pythia · WikiText | perplexity: 20.1 -> 19.1 | Helped |
| replace vanilla attention with global-to-local block attention | Pythia · HellaSwag | accuracy: 35.05 -> 31.13 | Hurt |
| replace vanilla attention with global-to-local block attention | Pythia · PIQA | accuracy: 68.10 -> 64.35 | Hurt |
| use a lookup-table embedder instead of a RoBERTa-based encoder embedder (ablation) | Block Transformer · language modeling | training loss | Helped |
| use a lookup-table embedder instead of a CLS-token embedder (ablation) | Block Transformer · language modeling | training loss | Helped |
| use prefix decoding instead of cross-attention for context embeddings (ablation) | Block Transformer · language modeling | perplexity | Helped |
| use prefix decoding instead of summation for context embeddings (ablation) | Block Transformer · language modeling | perplexity | Helped |
| increase prefix length beyond four tokens (ablation) | Block Transformer · language modeling | perplexity | Helped |
| allocate parameters equally between block and token decoders (ablation) | Block Transformer with block length 4 · language modeling | training loss | Helped |
| increase block length (ablation) | Block Transformer · inference | throughput | Helped |
Surprising findings
- A complex RoBERTa-style encoder embedder does not outperform a simple lookup-table embedder, and it lowers throughput.
- MSE or contrastive losses on the block decoder degrade performance when trying to predict next block embeddings for parallel decoding.
Limitations
- Block Transformers need more parameters and FLOPs than perplexity-equivalent vanilla models, raising training cost and hindering on-device use.
- Experiments are limited to models up to 1.4B parameters, so scaling beyond 1B is unverified.
- The token decoder depends on a single context embedding, so it has limited access to global context, causing higher loss at later positions in a block.
Experimental setup
Datasets: The Pile PG19 LAMBADA WikiText HellaSwag PIQA ARC-easy
Benchmarks: LAMBADA WikiText HellaSwag PIQA ARC-easy PG19 Needle-In-a-Haystack
Metrics: perplexity accuracy throughput (1K tokens/s) memory per sample (MB) training loss
Baselines: Vanilla Transformer (Pythia-style) MEGABYTE
Builds on: Pythia MEGABYTE GPT-NeoX FlashAttention
Closest papers by meaning
- Efficient Large Language Model Inference with Neural Block Linearization (NeurIPS 2025 · similarity 0.88)
- MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM (NeurIPS 2026 · similarity 0.88)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.88)
- Star Attention: Efficient LLM Inference over Long Sequences (ICML 2025 · similarity 0.88)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.88)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.