AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

Shaowen Wang, Yuke Zheng, Tansheng Zhu, Shuang Chen, Shaofan Liu, Suncong Zheng, Jian Li

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Attention head mechanisms · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add AdaRoPE instead of RoPELLaMA 430M · NLU avgmean accuracy: 50.65 -> 51.23Improved
add AdaRoPE instead of RoPELLaMA 430M · pretrainingloss: 2.692 -> 2.663Improved
add AdaRoPE instead of RoPEQwen3 Gated Attention 430M · WikiText-103perplexity: 26.20 -> 20.08Improved
add AdaRoPE instead of RoPELLaMA 1.3B · NLU avgmean accuracy: 56.71 -> 57.94Improved
add head-wise frequencies (vs shared frequency) (ablation)AdaRoPE LLaMA 430M · NLU avgmean accuracy: 50.90 -> 51.23Improved
add AdaScale (ablation)AdaRoPE LLaMA 430M · NLU avgmean accuracy: 50.77 -> 51.23Improved
add learned AdaScale instead of fixed log scaling (ablation)AdaRoPE LLaMA 430M · NLU avgmean accuracy: 50.58 -> 51.23Improved
add head-wise frequency parameterization instead of learned base (ablation)AdaRoPE LLaMA 430M · NLU avgmean accuracy: 50.77 -> 51.23Improved
add AdaRoPE instead of YaRN scaling (extrapolation)LLaMA 8B · RULER 64kaccuracy: 12.80 -> 51.87Improved
add AdaFreq (ablation)AdaRoPE extrapolation · RULER 64kaccuracy: 13.13 -> 15.83Improved
add AdaScale (ablation)AdaRoPE extrapolation · RULER 64kaccuracy: 9.97 -> 15.83Improved
add AdaScale (ablation)AdaRoPE extrapolation SmolLM 1.7B · NLUaccuracy: 64.01 -> 62.73Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy