Research map / Architectures and efficiency / Attention and transformers
RePo: Language Models with Context Re-Positioning
Huayang Li, Tianyu Zhao, Deng Cai, Richard Sproat
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace linear RoPE positions with learned re-positioning (RePo) | OLMo-2 1B · RULER NIAH | average accuracy: 85.9 -> 91.3 | Improved |
| replace linear RoPE positions with RePo | OLMo-2 7B · RULER NIAH | average accuracy: 97.3 -> 97.9 | Improved |
| replace linear RoPE positions with RePo | OLMo-2 1B · HybridQA table | exact match: 24.43 -> 26.70 | Improved |
| replace linear RoPE positions with RePo | OLMo-2 7B · HybridQA table | exact match: 33.52 -> 37.61 | Improved |
| replace linear RoPE positions with RePo | OLMo-2 1B · LongBench (<16K, YaRN) | average score: 20.93 -> 27.86 | Improved |
| replace linear RoPE positions with RePo | OLMo-2 7B · LongBench (<16K, YaRN) | average score: 25.65 -> 32.03 | Improved |
| replace linear RoPE positions with RePo | OLMo-2 1B · general short-context benchmarks | average: 53.70 -> 53.73 | No clear effect |
| replace linear RoPE positions with RePo | OLMo-2 7B · general short-context benchmarks | average: 70.92 -> 70.30 | Hurt |
| remove positional encoding entirely (NoPE) instead of RoPE (ablation) | OLMo-2 1B continual pretraining · RULER NIAH | average accuracy: 85.9 -> 80.2 | Hurt |
| NoPE baseline on long-context (ablation) | OLMo-2 1B continual pretraining · LongBench (<16K, YaRN) | average score: 20.93 -> 6.98 | Hurt |
| interleave 2 RoPE and 1 NoPE layers (R2N1) instead of RePo (ablation) | OLMo-2 1B · LongBench (<16K, YaRN) | average score: 20.93 -> 22.95 | Mixed |
| interleave 1 RoPE and 2 NoPE layers (N2R1) (ablation) | OLMo-2 1B · LongBench (<16K, YaRN) | average score: 20.93 -> 11.29 | Hurt |
Closest papers by meaning
- ReAttention: Training-Free Infinite Context with Finite Attention Scope (ICLR 2025 · similarity 0.88)
- RePO: Understanding Preference Learning Through ReLU-Based Optimization (NeurIPS 2025 · similarity 0.87)
- Where does In-context Learning Happen in Large Language Models? (NeurIPS 2024 · similarity 0.87)
- Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism (ICML 2025 · similarity 0.87)
- Mixture of In-Context Experts Enhance LLMs' Long Context Awareness (NeurIPS 2024 · similarity 0.87)
- Understanding In-Context Learning from Repetitions (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
