Research map / Large language models / Instruction and fine-tuning
MrRoPE: Mixed-radix Rotary Position Embedding
Qingyuan Tian, Wenhong Zhu, Xiaoran Liu, Xiaofeng Wang, Rui Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MrRoPE-Pro instead of YaRN | LLaMA3-8B-Instruct · Proofpile PPL 128K | perplexity: 2.38 -> 2.34 | Improved |
| use MrRoPE-Pro instead of MrRoPE-Uni | LLaMA3-8B-Instruct · Proofpile PPL 128K | perplexity: 2.41 -> 2.34 | Improved |
| use MrRoPE-Uni instead of YaRN at the longest context | LLaMA3-8B-Instruct · Proofpile PPL 128K | perplexity: 2.38 -> 2.41 | Hurt |
| use MrRoPE-Pro instead of YaRN at short context | LLaMA2-7B-chat · Proofpile PPL 4K | perplexity: 6.02 -> 5.72 | Improved |
| use MrRoPE-Pro instead of YaRN | LLaMA3-8B-Instruct · RULER 128K | score: 79.9 -> 86.6 | Improved |
| use MrRoPE-Pro instead of YaRN | Qwen2.5-3B-Instruct · RULER 128K | score: 50.1 -> 53.2 | Improved |
| use MrRoPE-Pro instead of YaRN | LLaMA3-8B · Infinite-Bench KV Retrieval | accuracy: 9% -> 27% | Improved |
| use MrRoPE-Pro instead of YaRN | LLaMA3-8B · Infinite-Bench QA Dialogue | accuracy: 10% -> 22% | Improved |
| use MrRoPE-Pro instead of YaRN | LLaMA3-8B · Infinite-Bench average | accuracy: 44% -> 49.8% | Improved |
| use MrRoPE-Pro instead of YaRN | LLaMA3-8B · Infinite-Bench Code Debug | accuracy: 3% -> 3% | No clear effect |
| use MrRoPE-Pro instead of YaRN | LLaMA2-7B-Instruct · LongBench-v2 LSD | accuracy: 30.0 -> 40.0 | Improved |
| use MrRoPE-Pro instead of YaRN | LLaMA2-7B-Instruct · LongBench-v2 SD | accuracy: 16.7 -> 16.7 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- YaRN — YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024) · better on 16 of 16 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.88)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.87)
- 3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding (AAAI 2025 · similarity 0.86)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.86)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.86)
- LieRE: Lie Rotational Positional Encodings (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
