Research map / Architectures and efficiency / Attention and transformers
Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOMER, a training-free hierarchical context-merging scheme that splits long inputs into chunks, merges them across transformer layers with attention-based token pruning, and uses DFS ordering for logarithmic memory, extending Llama-2 context to 64k.
Problem
LLMs are limited by a fixed context window and the quadratic cost of self-attention; existing extension methods either need costly training or only modify positional encodings without reducing computation.
Approach
Divide long input into chunks that are processed independently in early layers, then progressively merge adjacent chunks across layers after attention-based token pruning with position-bias calibration, and propagate pruning decisions to lower layers so the cache has fixed length; a DFS computation order makes peak memory logarithmic in input length. No finetuning is required.
Key results
- On passkey retrieval at 32k, Llama-2-13b-chat with HOMER reaches 0.804 accuracy versus 0.224 for the best baseline (YaRN 64k).
- On QuALITY, Llama-2 with HOMER improves accuracy from 0.327 (plain) to 0.358, and NTK + HOMER reaches 0.388.
- Peak GPU memory at 64k inputs is reduced by at least 73.4% relative to baselines (HOMER 21.3 GB vs. >80 GB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HOMER to plain Llama-2-7b-chat | Llama-2-7b-chat · Passkey retrieval, 32K | accuracy: 0.026 -> 0.776 | Helped |
| Apply HOMER on top of YaRN | Llama-2-13b-chat · Passkey retrieval, 32K | accuracy: 0.224 -> 0.860 | Helped |
| Apply HOMER to plain Llama-2 | Llama-2-7b-chat · QuALITY validation | accuracy: 0.327 -> 0.358 | Helped |
| Apply HOMER on top of NTK | Llama-2-7b-chat · QuALITY validation | accuracy: 0.379 -> 0.388 | Helped |
| Apply HOMER on top of YaRN | Llama-2-7b · PG-19, 64K | perplexity: $>10^{\phantom{1}}$ -> 8.83 | Helped |
| Use HOMER instead of the baseline setup | Llama-2 · Peak GPU memory, 64k input | memory (GB): $>80$ -> 21.3 (at least -73.4%) | Helped |
| Use HOMER instead of the baseline setup | Llama-2 · Inference, 100 tokens from 32k context | run time (seconds): 18.828 -> 7.169 (162.6% speedup) | Helped |
| Use calibrated attention-based top-K pruning instead of random token pruning (ablation) | HOMER · Passkey retrieval, 16k | accuracy: 0.006 -> 0.890 | Helped |
| Add calibration to attention-based top-K pruning (ablation) | HOMER · Passkey retrieval, 16k | accuracy: 0.056 -> 0.890 | Helped |
| Add propagative refinement of lower-layer embeddings (ablation) | HOMER · Passkey retrieval, 16k | accuracy: 0.002 -> 0.890 | Helped |
| Use propagative refinement instead of random token gathering (ablation) | HOMER · Passkey retrieval, 16k | accuracy: 0.116 -> 0.890 | Helped |
| Use propagative refinement instead of layer-wise top-K token gathering (ablation) | HOMER · Passkey retrieval, 16k | accuracy: 0.040 -> 0.890 | Helped |
Surprising findings
- Plain Llama-2-7b-chat with HOMER gets 0.776 passkey accuracy at 32k while several baselines collapse to near zero beyond 8k-16k.
- Attention-based top-K pruning without calibration performs nearly as poorly as random pruning (0.056 vs 0.006), while calibration raises it to 0.890.
Limitations
- Evaluated only on Llama-2 models (7B and 13B), so generality to other architectures is untested.
- Paper states the method could be combined with finetuning but does not explore it; the approach relies on hand-tuned merging schedule and chunk-length choices.
Experimental setup
Datasets: WikiText-103 (calibration segments)
Benchmarks: passkey retrieval QuALITY PG-19
Metrics: retrieval accuracy accuracy perplexity peak GPU memory (GB) inference time (s)
Baselines: Plain Llama-2 Position Interpolation NTK-aware scaling YaRN
Builds on: Position Interpolation NTK-aware scaling YaRN Flash Attention 2
Closest papers by meaning
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.89)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.89)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.89)
- Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.89)
- An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.