Research map / Large language models / Language, safety and interpretability
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-RPE, a rotary position encoding on a 3D (Bloch) sphere with separate within-chunk and across-chunk angles, to improve long-context modeling of LLaMA models under position interpolation.
Problem
RoPE suffers from long-term decay and loses positional resolution when position interpolation is used to extend LLM context windows beyond pretraining length.
Approach
Extends 2D RoPE to a 3D rotation inspired by the Bloch sphere: the chunk-internal angle theta encodes within-chunk relative position and the angle phi encodes across-chunk position, giving controllable long-term decay and better resolution under linear interpolation.
Key results
- On LongBench, 3D-RPE-LLaMA2-7B-Chat scores 47.40 single-doc QA and 60.10 multi-doc QA versus 40.30 and 46.60 for ChatGLM3-6B-32k.
- Training-free 3D-RPE with linear interpolation on LLaMA2-7B-Base reaches perplexity 7.71 at 16k on PG19 versus 15.11 for PI.
- On LEval GSM, 3D-RPE-LLaMA2-7B-Chat gets 21.01 EM versus 19.00 for LLaMA2-7B-Chat and Vicuna1.5-7B-16k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard RoPE with 3D-RPE | LLaMA-2-7B-chat · LongBench Multi-Doc QA | score: 22.60 -> 60.10 | Helped |
| Replace standard RoPE with 3D-RPE | LLaMA-2-7B-chat · LongBench Summarization | score: 24.70 -> 28.99 | Hurt |
| Use 3D-RPE instead of the original positional encoding | LLaMA-2-7B-chat · LEval CodeU | EM: 1.11 -> 2.22 | Hurt |
| Use 3D-RPE instead of the original positional encoding | LLaMA-3-8B-Instruct · LEval CodeU | EM: 4.44 -> 4.44 | No clear effect |
| Use 3D-RPE instead of the original positional encoding | LLaMA-3-8B-Instruct · LEval TOEFL | EM: 82.89 -> 82.89 | No clear effect |
| Use 3D-RPE instead of the original positional encoding | LLaMA-3-8B-Instruct · LEval QuALiTY | EM: 64.34 -> 61.38 | Hurt |
| Use 3D-RPE instead of the original positional encoding | LLaMA-3-8B-Instruct · LEval GSM | EM: 76.00 -> 80.00 | Helped |
| Use 3D-RPE instead of the original positional encoding (ablation) | LLaMA-2-7B-Base · PG19 validation, 32k | perplexity: 15.11 -> 9.34 | Helped |
| Combine 3D-RPE with linear position interpolation (ablation) | LLaMA-2-7B-Base · PG19 validation, 16k | perplexity: 15.11 -> 7.71 | Helped |
| Combine 3D-RPE with linear position interpolation (ablation) | LLaMA-2-7B-Base · PG19 validation, 32k | perplexity: >10^2 -> 9.34 | Helped |
| Replace standard RoPE with 3D-RPE | LLaMA2-7B-Base · PG19 test, 32k | perplexity: 7.04 -> 8.09 | Hurt |
| Replace standard RoPE with 3D-RPE | LLaMA2-7B-Base · Proof-Pile test, 32k | perplexity: 2.50 -> 2.89 | Hurt |
Surprising findings
- 3D-RPE underperforms on several LEval tasks (QuALiTY, CodeU) and on LongBench summarization it is roughly comparable to baselines.
- LongLoRA-32k achieves lower perplexity than 3D-RPE at 32k and 100k on PG19 and Proof-pile.
Limitations
- Gains are mixed: 3D-RPE does not improve several LEval tasks and LongBench summarization, and LongLoRA is better in some perplexity settings.
- Theoretical resolution advantage holds only under assumptions (chunk size >= 3) and linear interpolation; the chunk size and hyperparameters are heuristic.
- Experiments are limited to LLaMA-2-7B and LLaMA-3-8B; no evaluation on other architectures or larger scales.
Experimental setup
Datasets: LongAlpaca-12k LongAlpaca-16k RedPajama-Data PG19 Arxiv Math Proof-pile
Benchmarks: LongBench LEval PG19 Arxiv Math Proof-pile
Metrics: F1 Rouge exact match perplexity accuracy
Baselines: LongAlpaca-16k LongChat-32k LongLLaMA ChatGLM3-6B-32k StreamingLLM ChunkLLaMA-16k LongLoRA Activation Beacon PI NTK YaRN
Builds on: RoPE Position Interpolation LongLoRA PoSE LongChat
Closest papers by meaning
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.88)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.87)
- HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.86)
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.86)
- 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.