Research map / Large language models / Reasoning and chain of thought
Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mesa-Extrapolation, a training-free weave-PE method that chunks the input into a triangular attention layout and applies Stair PE to the final chunk to extend LLM context beyond the max training length with lower memory and linear decoding cost.
Problem
LLMs' inference ability sharply declines beyond their maximum training length (the length extrapolation problem), and existing extrapolation fixes either need fine-tuning or incur quadratic memory and time cost or discard tokens.
Approach
Introduces Stair PE, a weave function that keeps relative positions exact up to N and then increases them in fixed-width steps, combined with a chunk-based triangular attention matrix whose last chunk uses Stair PE; the authors give theory that weave PE lets hidden states stay above a threshold beyond the effective window.
Key results
- Mesa-Extrapolation keeps passkey retrieval accuracy stable beyond the training length on LLaMA-family models, with weave-PE methods (ReRoPE, Leaky-ReRoPE, Mesa) extrapolating while Origin fails.
- On GovReport with LLaMA2-7B-Chat, Mesa-Extrapolation maintains ROUGE scores roughly flat from 3k to 11k where Origin drops to 0 after 6k-7k.
- Mesa-Extrapolation shows approximately linear memory and decoding time growth, versus quadratic growth for ReRoPE and Leaky-ReRoPE, which the authors report as the lowest memory and fastest decoding among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mesa-Extrapolation instead of the unmodified model for inputs beyond the training window | LLaMA2-7B-Chat · GovReport, 5k–11k | ROUGE-1/ROUGE-2/ROUGE-L | Helped |
| Use Mesa-Extrapolation instead of Dynamic-NTK | LLaMA2-7B-Chat · GovReport, 3k–11k | ROUGE-1/ROUGE-2/ROUGE-L | Mixed |
| Use Mesa-Extrapolation instead of ReRoPE or Leaky-ReRoPE | LLaMA2-7B-Chat · GovReport, 1k–11k | BLEU | Mixed |
| Use chunk-based triangular attention with Stair PE | Mesa-Extrapolation · Long-context inference | memory usage | Helped |
| Use chunk-based triangular attention with Stair PE | Mesa-Extrapolation · Long-context inference | decoding time | Helped |
| Apply Mesa-Extrapolation instead of Origin | LLaMA models · Passkey retrieval beyond maximum training length | accuracy | Helped |
| Apply Mesa-Extrapolation instead of ReRoPE or Leaky-ReRoPE | LLaMA models · Passkey retrieval | maximum extrapolation length | Helped |
| Apply Mesa-Extrapolation to MPT-7B | MPT-7B · Passkey retrieval beyond maximum training length | accuracy | Hurt |
| Apply Mesa-Extrapolation instead of other methods | MPT-7B · LongEval Lines | accuracy | Helped |
| Apply Mesa-Extrapolation | PyThia-6.9B · LongEval Lines | accuracy | Hurt |
Surprising findings
- MPT-7B with ALiBi extrapolates well on perplexity to about 3.5k but fails on passkey and LongEval, which the authors attribute to ALiBi's decaying bias suppressing distant tokens and to an approximate ALiBi implementation in MPT.
Limitations
- Not validated at lengths much longer than tested due to limited resources.
- Performance is unstable on MPT-7B (ALiBi approximation) and shows variability in mid-length ranges on summarization.
- Only plug-in evaluation without fine-tuning; combination with fine-tuning left for future work.
Experimental setup
Datasets: Pile GovReport LongBench LongEval generated passkey dataset
Benchmarks: passkey retrieval Pile perplexity GovReport ROUGE/BLEU LongEval lines
Metrics: accuracy NLL/perplexity ROUGE-1/2/L BLEU memory usage decoding latency
Baselines: Origin ReRoPE Leaky-ReRoPE Dynamic-NTK LM-Infinite Streaming-LLM
Builds on: RoPE ALiBi ReRoPE Leaky-ReRoPE Self-Extend
Closest papers by meaning
- Scaling Laws of RoPE-based Extrapolation (ICLR 2024 · similarity 0.91)
- PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.90)
- InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory (NeurIPS 2024 · similarity 0.89)
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.88)
- YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.88)
- An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.