AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis

Qingyue Yang, Jie Wang, Xing Li, Yinqi Bai, Tong Xialiang, Hui-Ling Zhen, Jianye HAO, Mingxuan Yuan, Bin Li

ICLR 2026 · Large language models › Language, safety and interpretability · approach: Reasoning mechanisms and failure modes · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
allocate per-layer KV budget using q-similarity (more budget to low-similarity layers) instead of CAKE allocationKV cache compression · LongBench avg, Llama-3.1-8B, budget 1024average score: 48.13 -> 48.43Improved
q-similarity budget allocation vs CAKE on Qwen2.5-7B at budget 1024KV cache compression · LongBench avg, Qwen2.5-7B, budget 1024average score: 47.70 -> 47.72No clear effect
q-similarity vs CAKE on Qwen2.5-7B at budget 512KV cache compression · LongBench avg, Qwen2.5-7B, budget 512average score: 45.56 -> 45.78No clear effect
combine Block Influence with q-similarity for layer pruningShortGPT on Llama-3.1-8B (28% pruned) · Avg of PIQA, HellaSwag, Winogrande, ARC-Easyaverage accuracy (%): 53.51 -> 59.11Improved
combine Block Influence with q-similarity for layer pruningShortGPT on Qwen-2.5-7B (43% pruned) · Avg of PIQA, HellaSwag, Winogrande, ARC-Easyaverage accuracy (%): 46.04 -> 48.60Improved
combine Block Influence with q-similarity for layer pruningShortGPT on Qwen-2.5-7B (39% pruned) · Avg of PIQA, HellaSwag, Winogrande, ARC-Easyaverage accuracy (%): 49.98 -> 50.42No clear effect
combine Block Influence with q-similarity for layer pruningShortGPT on Llama-2-7B (31% pruned) · Avg of PIQA, HellaSwag, Winogrande, ARC-Easyaverage accuracy (%): 54.48 -> 55.82Improved
disable RoPE on a high q-similarity sequential head (ablation)sequential attention head · attention map visualizationqualitative diagonal qualityHurt
perturb query temporal order (q cosine similarity 0.99 -> 0.97) with RoPE on (ablation)sequential attention head · attention map visualizationqualitative predictability: 0.99 -> 0.97Hurt
move key massive channel to a higher-frequency (lower index) RoPE channel (ablation)sequential attention head · attention map visualizationperiodic diagonals observedImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy