Research map / Architectures and efficiency / Attention and transformers
What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes RoPE query/key weight-vector pairs via their angles to show non-orthogonal pairs relate to syntax and orthogonal pairs to semantics, then proposes Angle-based Weight Masking (AWM) that freezes non-orthogonal pairs during LoRA fine-tuning.
Problem
How LLMs using rotary position embedding utilize positional information across heads and layers, and how to reduce trainable parameters during fine-tuning of query and key weights.
Approach
The angle between the two weight vectors forming a RoPE pair determines how input-sensitive the pair's rotated direction is: non-orthogonal pairs (large |cos α|) attend to fixed relative positions and capture basic syntax, while near-orthogonal pairs capture high-level semantics and are the ones changed by fine-tuning. AWM freezes pairs with |cos α| above a threshold and fine-tunes only the rest,
Key results
- On Llama-2-7b with LoRA(r=8), AWM with threshold 0.001 raised TruthfulQA mc2 from 49.48 to 50.97 while freezing 92.12% of query weight vector pairs.
- On Phi-2 with LoftQ, AWM (threshold 0.001, 40.72% fixed pairs) raised GSM8K accuracy from 48.75 to 52.92.
- Fine-tuning changes concentrate on near-orthogonal query/key pairs; average absolute cosine similarity drops after the first ~3 layers and stays low.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AWM with τ=0.001 to LoRA(r=8) (ablation) | Llama-2-7b · TruthfulQA | mc1: 33.41 -> 35.00 | Helped |
| add AWM with τ=0.001 to LoRA(r=8) (ablation) | Llama-2-7b · TruthfulQA | mc2: 49.48 -> 50.97 | Helped |
| add AWM with τ=0.005 to LoRA(r=8) (ablation) | Llama-2-7b · GSM8K | accuracy: 10.92 -> 11.90 | Helped |
| add AWM with τ=0.001 to LoRA(r=8) (ablation) | Llama-2-7b · HellaSwag | accuracy: 58.81 -> 58.91 | Helped |
| add AWM with τ=0.001 to LoRA(r=8) (ablation) | Llama-2-13b · TruthfulQA | mc1: 30.72 -> 32.07 | Helped |
| add AWM with τ=0.001 to LoRA(r=8) (ablation) | Llama-2-13b · TruthfulQA | mc2: 44.74 -> 45.88 | Helped |
| add AWM with τ=0.001 to LoRA(r=8) (ablation) | Llama-2-13b · GSM8K | accuracy: 18.42 -> 19.48 | Helped |
| add AWM with τ=0.01 to LoRA(r=8) (ablation) | Llama-2-13b · HellaSwag | accuracy: 61.31 -> 61.45 | Helped |
| add AWM with τ=0.01 (ablation) | Llama-2-7b + LoftQ · WikiText-2 | perplexity: 5.518 -> 5.485 | Helped |
| add AWM with τ=0.01 (ablation) | Llama-2-7b + LoftQ · GSM8K | accuracy: 39.20 -> 40.86 | Helped |
| add AWM with τ=0.001 (ablation) | Llama-2-7b + LoftQ · WikiText-2 | perplexity: 5.518 -> 5.480 | Helped |
| add AWM with τ=0.005 (ablation) | Llama-2-7b + LoftQ · GSM8K | accuracy: 39.20 -> 38.44 | Hurt |
Surprising findings
- Freezing non-orthogonal query/key pairs during fine-tuning sometimes improves downstream performance rather than hurting it.
- On Phi-2 with LoftQ, AWM lowered WikiText-2 perplexity at threshold 0.01 (9.553 to 9.766) while improving GSM8K.
Limitations
- AWM only applies to query or key weights, limiting the reduction in trainable parameters.
- Only the query (and value) is fine-tuned in experiments; statistical significance not established because runs were not repeated.
- Syntax vs. semantics interpretation is based on qualitative attention visualizations of a few inputs.
Experimental setup
Datasets: Alpaca WikiText-2 GSM8K
Benchmarks: TruthfulQA GSM8K HellaSwag WikiText-2
Metrics: mc1 mc2 accuracy perplexity
Baselines: LoRA LoftQ
Builds on: RoPE LoRA LoftQ
Closest papers by meaning
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.91)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.90)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.90)
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.88)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.87)
- Scaling Laws of RoPE-based Extrapolation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.