AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

How Base Frequency Shapes RoPE: An Analytical Study of Frequency-Band Formation

Yui Oka, Itsumi Saito, Kyosuke Nishida, Kuniko Saito

ICLR 2026 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace lowest-frequency RoPE dimensions with NoPE (r=0.50) (ablation)Llama-3 · Wikitext-103 L=4096perplexity: 2.29 -> 84.50Hurt
replace lowest 10% frequency dimensions with NoPE (r=0.9) (ablation)Llama-3 · Wikitext-103 L=4096perplexity: 2.29 -> 2.29No clear effect
replace low-frequency dimensions with NoPE (r=0.9) (ablation)Phi-3 · Wikitext-103 L=4096perplexity: 2.84 -> 46.11Hurt
replace low-frequency dimensions with NoPE (r=0.9) (ablation)Gemma · Wikitext-103 L=4096perplexity: 2.52 -> 2.70Hurt
apply YaRN position interpolationLlama-2 · Wikitext-103 L=4096band index: 53.53 -> 51.93No clear effect
increase base theta from 10000 to 1000000RoPE from scratch Ltrain=512 · WikiText-103band index: 30.12 -> 15.37Improved
increase training length from 512 to 2048RoPE theta=10000 · WikiText-103band index: 30.12 -> 52.12Improved
set theta equal to training length (FMRoPE) instead of 10000RoPE from scratch Ltrain=512 · WikiText-103 inference L=3512perplexity: > 100 -> 23.00Improved
set theta equal to training length (FMRoPE) instead of 10000RoPE from scratch Ltrain=512 · WikiText-103 inference L=512perplexity: 19.39 -> 19.58Hurt
add FMRoPE theta=1512 during YaRN fine-tuning instead of theta=10000YaRN · WikiText-103 inference L=15512perplexity: 52.59 -> 20.51Improved
add FMRoPE theta=1512 during YaRN fine-tuning instead of theta=10000YaRN · WikiText-103 inference L=512perplexity: 19.10 -> 19.62Hurt
enlarge inference base to 2048 while training at theta=1024FMRoPE 1B-scale LM · L=2048perplexity: 42.36 -> 19.33Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy