Research map / Architectures and efficiency / Attention and transformers
Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fourier Position Embedding (FoPE), which replaces each RoPE dimension with a Fourier series of frequencies and zeroes out under-trained low frequencies, improving length generalization of pre-trained and fine-tuned LMs.
Problem
RoPE-based language models fail to length-generalize because the periodic extension of attention is damaged by spectral distortion from linear/activation layers and by under-trained low-frequency components from finite-length truncation.
Approach
Model RoPE as a non-uniform discrete Fourier transform, then give each attention dimension a Fourier series (dominant frequency plus learned harmonics with random small coefficients) and set frequencies below the floor 2π/N to zero, with the harmonic weights implemented as fixed (non-trained) matrices.
Key results
- On 1.2B models, FoPE keeps Passkey retrieval accuracy stable beyond the training length, while RoPE drops to zero at twice the training length.
- On SmolLM-1.7B fine-tuned to 4k, FoPE improves TREC by 15.0 points in the 8k+ bucket and SAMSum by 8.87 points over RoPE.
- FoPE is slightly worse than ALiBi on pre-training perplexity, but better than RoPE in both C4 and Books settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RoPE with FoPE | SmolLM-1.7B · GovReport, MultiNews, TREC and SAMSum across 0-4k, 4-8k and 8k+; TriviaQA 8k+ | score: GovReport: +0.25, +1.15, +0.36; MultiNews: +0.21, +1.87, +1.38; TREC: +4.00, +14.0, +15.0; SAMSum: +9.50, +9.48, +8.87; | Helped |
| Replace RoPE with FoPE | SmolLM-1.7B · TriviaQA 0-4k and 4-8k | score: TriviaQA 0-4k: ↓3.24; 4-8k: ↓2.49 | Hurt |
| Add Fourier Series (FS) to FoPE (ablation) | 60M FoPE model · Gutenberg Books; length generalization | perplexity and Passkey Retrieval accuracy | Helped |
| Clip the floor frequency to zero (CF) (ablation) | 60M FoPE model · Gutenberg Books; current dataset and sequence length | perplexity | Helped |
| Combine Fourier Series and Clip Floor to Zero (ablation) | 60M FoPE model · Gutenberg Books; length generalization | perplexity and Passkey Retrieval accuracy | Helped |
| Increase attention-head dimension (ablation) | FoPE · Gutenberg Books; length generalization | perplexity and Passkey Retrieval accuracy | Helped |
| Increase the number of attention heads or layers (ablation) | FoPE · Gutenberg Books; length generalization | perplexity and Passkey Retrieval accuracy | Hurt |
| Set the variance σ of frequency coefficients to 0.3 (ablation) | 60M FoPE model with D=16 · Gutenberg Books; longer contexts | perplexity | Helped |
| Set the number D of sampled frequencies to 64 (ablation) | 60M FoPE model with σ=0.3 · Gutenberg Books | Passkey Retrieval accuracy | Helped |
| Vary D from 16 to 128 (ablation) | FoPE · Gutenberg Books | perplexity | No clear effect |
| Apply QK_Norm (ablation) | RoPE · Toy models; sequence lengths 512, 1024, 2048, 4096 and 8192 | loss: 5.50 -> 5.46; 6.01 -> 5.56; 6.58 -> 5.89; 6.99 -> 6.32; 7.16 -> 6.66 | Helped |
| Apply QK_Norm (ablation) | RoPE-A · Toy models; sequence lengths 512, 1024, 2048, 4096 and 8192 | loss: 5.72 -> 5.69; 5.86 -> 5.89; 6.18 -> 6.27; 6.46 -> 6.59; 6.67 -> 6.81 | Mixed |
Surprising findings
- FoPE underperforms RoPE on TriviaQA for shorter contexts (0-4k and 4-8k), despite gains at 8k+.
- Adding more attention heads or layers was found to aggravate spectrum damage more than increasing head dimension.
Limitations
- Pre-training experiments use small models (60M to 1.2B) and small token budgets, so scaling behavior is not established.
- FoPE is slightly worse than ALiBi on pre-training perplexity and worse than RoPE on TriviaQA at short lengths.
- Hyperparameters (sigma, D) must be tuned per model scale, and the best values are estimated by grid search.
Experimental setup
Datasets: C4 Gutenberg Books GovReport MultiNews TREC TriviaQA SAMSum Passkey Retrieval
Benchmarks: Passkey Retrieval GovReport MultiNews TREC TriviaQA SAMSum MMLU PIQA HellaSwag ARC Easy
Metrics: perplexity accuracy cross-entropy loss ROUGE
Baselines: RoPE ALiBi NoPE KERPLE FIRE YaRN RoPE-A
Builds on: RoPE ALiBi YaRN OLMo SmolLM
Closest papers by meaning
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.89)
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.88)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.88)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.87)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.87)
- Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.