Research map / Architectures and efficiency / Attention and transformers
Wavelet-based Positional Representation for Long Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a relative positional representation built from multi-scale Ricker wavelets (replacing learned RPE embeddings) that extrapolates beyond the training length without clipping the attention field, shown to beat RoPE, ALiBi and Trans-XL in WikiText-103 perplexity and a Llama-2-7B long-context test.
Problem
Transformer language models fail to extrapolate beyond the maximum training sequence length; RoPE relies on absolute positions and ALiBi restricts the effective receptive field, hurting long-range dependencies.
Approach
Interprets RoPE as a fixed-scale Haar-like wavelet transform over head dimensions and ALiBi as multi-window attention, then builds a relative position representation from d wavelet functions with multiple scales and shifts (Ricker base wavelet, s=8 scales, d/s=16 shifts), with no clipping and no receptive-field limit.
Key results
- On WikiText-103 (L_train=512), Wavelet (Ricker) reaches 17.99 perplexity at 2512 tokens without non-overlapping inference vs 18.41 for ALiBi and 19.05 for Trans-XL... (with RoPE rising to 77.90 at 2512).
- On the Llama-2-7B long-context setup (L_train=4096), the wavelet method reduces CodeParrot perplexity versus RoPE at 4k (9.00 vs 9.45) and 32k (8.60 vs 8.90).
- Restricting the scale to a single value (Haar fixed scale) increases perplexity markedly at long lengths (299.26 at 2512), supporting the need for multiple scales.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RoPE with wavelet positional representation | Transformer · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 93.94 -> 17.99 | Helped |
| replace ALiBi with wavelet positional representation | Transformer · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 18.41 -> 17.99 | Helped |
| replace RoPE with wavelet positional representation | Llama-2-7B · CodeParrot, 32 k tokens, non-overlapping inference (L_train=4096) | perplexity: 8.90 -> 8.60 | Helped |
| use Gaussian instead of Ricker wavelet (ablation) | Wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 17.99 -> 17.88 | Helped |
| use Haar instead of Ricker wavelet (ablation) | Wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 17.99 -> 18.17 | Hurt |
| use Morlet instead of Ricker wavelet (ablation) | Wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 17.99 -> 26.56 | Hurt |
| restrict the Haar wavelet to a fixed scale (ablation) | Wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 18.17 -> 299.26 | Hurt |
| increase the scale range to {2^1,2^2,...,2^8} with shifts {0,1,2,...,15} (ablation) | Ricker wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 17.99 -> 18.02 | No clear effect |
| increase the scale range to {2^2,2^3,...,2^9} with shifts {0,1,2,...,15} (ablation) | Ricker wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 17.99 -> 18.07 | No clear effect |
| reduce scale patterns and increase shift patterns to {0,1,2,...,31} (ablation) | Ricker wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 17.99 -> 21.73 | Hurt |
| increase scale patterns and reduce shift patterns to {0,1,2,...,7} (ablation) | Ricker wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 17.99 -> 17.84 | Helped |
| remove shift-pattern diversity (ablation) | Ricker wavelet positional representation · WikiText-103 validation, 2512 tokens, without non-overlapping inference (L_train=512) | perplexity: 17.99 -> 18.29 | Hurt |
Surprising findings
- The authors found that a RoPE-based wavelet variant took over five times longer to train than expected and was halted.
Limitations
- Higher computational and memory overhead from relative position computation.
- Long-context experiments only report non-overlapping perplexity due to memory limits.
Experimental setup
Baselines: RoPE ALiBi Trans-XL NoPE T5 Bias
Builds on: RoPE ALiBi relative position representation (RPE)
Closest papers by meaning
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.89)
- HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization (ICML 2025 · similarity 0.89)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.89)
- Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.88)
- Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.