Research map / Large language models / Language, safety and interpretability
How Do Position Encodings Affect Length Generalization? Case Studies On In-Context Function Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains small from-scratch decoder-only Transformers via meta-learning on regression and Boolean in-context function tasks to study how position encodings (NoPE, ALiBi, FIRE, dynamic YaRN) affect length generalization.
Problem
Length generalization of in-context learning in Transformers: ICL performance degrades on inputs longer than those seen in training, and the role of position encodings in this failure is unclear.
Approach
Uses in-context function learning tasks with position-bias-free data (regression and Boolean functions) trained from scratch so that length extrapolation is affected only by the position encoding choice; compares PEs and analyzes recency bias, induction heads, and serial-position effects.
Key results
- ALiBi and FIRE show the most stable length extrapolation, with the best PE extending the effective maximum input length to over 8x the training length.
- NoPE mitigates length generalization failure but errors begin at about twice the training length, and NoPE often reduces in-distribution performance.
- Parity is hard for all PEs, yielding near random-guess performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use positional encodings instead of NoPE | Transformer for in-context function learning · OOD sequence lengths | length generalization | Helped |
| Use ALiBi or FIRE instead of NoPE | Transformer for in-context function learning · Linear Regression | squared error | Helped |
| Use ALiBi or FIRE instead of NoPE | Transformer for in-context function learning · Sparse Majority | accuracy | Helped |
| Use Dynamic YaRN | Transformer for in-context function learning · Sparse Majority | accuracy | Hurt |
| Use FIRE at OOD lengths | Transformer for in-context function learning · 0-1 Threshold Functions | accuracy | Hurt |
| Use positional encodings instead of NoPE | Transformer for in-context function learning · Conjunction and Disjunction | accuracy | No clear effect |
| Use positional encodings instead of NoPE | Transformer for in-context function learning · Parity | accuracy | No clear effect |
| Increase model size (ablation) | NoPE Transformer · Linear Regression | squared error | Hurt |
| Use ALiBi instead of NoPE | Transformer for in-context function learning · Linear Regression and Sparse Majority | attention to recent tokens | Mixed |
| Add positional encodings | Transformer for in-context function learning · Repeated in-context examples | induction-head effectiveness | Hurt |
| Repeat non-informative data at inference | ALiBi Transformer · Duplicated data attack | error rate | Hurt |
| Repeat non-informative data at inference | FIRE Transformer · Duplicated data attack | error rate | Hurt |
Surprising findings
- Position encodings improve OOD length generalization but appear to weaken induction-head behavior, indicating a trade-off with data generalization.
- ALiBi shows a recency bias yet performs better on length generalization, contradicting the recency-bias hypothesis.
Limitations
- Scope limited to self-attention auto-regressive models trained from scratch on synthetic function tasks.
- Several function classes (e.g., sparse linear regression, decision trees, CNFs/DNFs) were excluded from experiments.
- Parity remained unsolved; no additional training techniques such as teaching sequences were used.
Experimental setup
Datasets: synthetic in-context regression functions (linear regression, sparse linear regression, decision tree, ReLU networks) synthetic Boolean functions (conjunction, disjunction, sparse majority, 0-1 threshold, integer halfspace, parity)
Benchmarks: in-context linear regression Boolean function tasks (conjunction, disjunction, sparse majority, 0-1 threshold, integer halfspace, parity)
Metrics: squared error accuracy
Baselines: NoPE ALiBi FIRE Dynamic YaRN Null Classifier least squares
Builds on: Garg et al. 2022 in-context function learning Bhattamishra et al. 2024 Boolean function ICL Llama 2 architecture ALiBi FIRE YaRN
Closest papers by meaning
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.91)
- Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs (NeurIPS 2025 · similarity 0.88)
- Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.88)
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.87)
- Understanding and Improving Length Generalization in Recurrent Models (ICML 2025 · similarity 0.87)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.