Research map / Large language models / Language, safety and interpretability
Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis
Qingyue Yang, Jie Wang, Xing Li, Yinqi Bai, Tong Xialiang, Hui-Ling Zhen, Jianye HAO, Mingxuan Yuan, Bin Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allocate per-layer KV budget using q-similarity (more budget to low-similarity layers) instead of CAKE allocation | KV cache compression · LongBench avg, Llama-3.1-8B, budget 1024 | average score: 48.13 -> 48.43 | Improved |
| q-similarity budget allocation vs CAKE on Qwen2.5-7B at budget 1024 | KV cache compression · LongBench avg, Qwen2.5-7B, budget 1024 | average score: 47.70 -> 47.72 | No clear effect |
| q-similarity vs CAKE on Qwen2.5-7B at budget 512 | KV cache compression · LongBench avg, Qwen2.5-7B, budget 512 | average score: 45.56 -> 45.78 | No clear effect |
| combine Block Influence with q-similarity for layer pruning | ShortGPT on Llama-3.1-8B (28% pruned) · Avg of PIQA, HellaSwag, Winogrande, ARC-Easy | average accuracy (%): 53.51 -> 59.11 | Improved |
| combine Block Influence with q-similarity for layer pruning | ShortGPT on Qwen-2.5-7B (43% pruned) · Avg of PIQA, HellaSwag, Winogrande, ARC-Easy | average accuracy (%): 46.04 -> 48.60 | Improved |
| combine Block Influence with q-similarity for layer pruning | ShortGPT on Qwen-2.5-7B (39% pruned) · Avg of PIQA, HellaSwag, Winogrande, ARC-Easy | average accuracy (%): 49.98 -> 50.42 | No clear effect |
| combine Block Influence with q-similarity for layer pruning | ShortGPT on Llama-2-7B (31% pruned) · Avg of PIQA, HellaSwag, Winogrande, ARC-Easy | average accuracy (%): 54.48 -> 55.82 | Improved |
| disable RoPE on a high q-similarity sequential head (ablation) | sequential attention head · attention map visualization | qualitative diagonal quality | Hurt |
| perturb query temporal order (q cosine similarity 0.99 -> 0.97) with RoPE on (ablation) | sequential attention head · attention map visualization | qualitative predictability: 0.99 -> 0.97 | Hurt |
| move key massive channel to a higher-frequency (lower index) RoPE channel (ablation) | sequential attention head · attention map visualization | periodic diagonals observed | Improved |
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.90)
- AttentionPredictor: Temporal Patterns Matter for KV Cache Compression (NeurIPS 2025 · similarity 0.89)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.89)
- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.88)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.88)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
