Research map / Architectures and efficiency / Attention and transformers
Revealing Long-context Potential of Attention Heads via Frequency Kernels
Senyu Han, Yilu Cao, Kai Yu, Lu Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| mask 10 top-LPS heads (ablation) | Llama-3.2-3B-Instruct · RULER 128K | accuracy: 65.78 -> 11.13 | Hurt |
| mask 10 random heads (ablation) | Llama-3.2-3B-Instruct · RULER 128K | accuracy: 65.78 -> 66.49 | No clear effect |
| mask 20 top-LPS heads (ablation) | Llama-3.1-8B-Instruct · RULER 128K | accuracy: 75.81 -> 7.93 | Hurt |
| mask 20 random heads (ablation) | Llama-3.1-8B-Instruct · RULER 128K | accuracy: 75.81 -> 74.70 | Hurt |
| mask 20 top-LPS heads (ablation) | Qwen3-4B · RULER 4K | accuracy: 95.08 -> 28.5 | Hurt |
| mask 20 random heads (ablation) | Qwen3-4B · RULER 128K | accuracy: 66.3 -> 61.54 | Hurt |
| mask 10 top-LPS heads (ablation) | Llama-3.2-3B-Instruct · GSM8K | accuracy: 76.95 -> 69.83 | Hurt |
| mask 20 top-LPS heads (ablation) | Llama-3.1-8B-Instruct · GSM8K | accuracy: 84.15 -> 76.95 | Hurt |
| mask 20 top-LPS heads (ablation) | Qwen3-4B · GSM8K | accuracy: 42.15 -> 40.18 | No clear effect |
| mask top-LPS heads (ablation) | Llama and Qwen models · MMLU | accuracy: 66.69 -> 64.97 | No clear effect |
| amplify low-frequency kernels of 20 top RPS heads | Qwen3-4B · RULER 128K | accuracy: 66.3 -> 68.41 | Improved |
| amplify low-frequency kernels of 20 top RPS heads | Qwen3-4B · RULER 64K | accuracy: 75.66 -> 77.07 | Improved |
Closest papers by meaning
- Scale-invariant attention (NeurIPS 2025 · similarity 0.88)
- HyperAttention: Long-context Attention in Near-Linear Time (ICLR 2024 · similarity 0.88)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.88)
- Virtual Head Attention (NeurIPS 2026 · similarity 0.87)
- Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization (ICML 2025 · similarity 0.87)
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
