Research map / Architectures and efficiency / Attention and transformers
Attention with Routed-Memory for Learnable Sparse Control
QIUHAO Zeng, Jerry Huang, Peng Lu, Ruiyi Fang, Gezheng Xu, Zihao Jing, Yufei Cui, Charles Ling, Gang Niu, Boyu Wang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gated soft write (fixed 1/2 sparsity) instead of FIFO eviction (ablation) | ARM · Recall-intensive tasks | average accuracy: 17.41 -> 24.42 | Improved |
| add learnable sparsity instead of fixed 1/2 retrieval (ablation) | ARM · Recall-intensive tasks | average accuracy: 24.42 -> 25.40 | Improved |
| add learnable sparsity instead of fixed 1/4 retrieval (ablation) | ARM · Recall-intensive tasks | average accuracy: 24.94 -> 25.40 | Improved |
| ARM instead of sliding window (256) | Llama-3.1-8B · LongBench | average score: 2.88 -> 18.02 | Improved |
| ARM instead of full attention | Llama-3.1-8B · LongBench | average score: 17.09 -> 18.02 | Improved |
| ARM instead of full attention | Llama-3.1-8B · RULER | average accuracy: 8.01 -> 15.12 | Improved |
| ARM instead of StreamingLLM | Llama-3.1-8B · RULER | average accuracy: 4.88 -> 15.12 | Improved |
| ARM instead of full attention | Llama-3.1-8B · LM-eval commonsense | average accuracy: 72.27 -> 71.89 | No clear effect |
| ARM instead of full attention | Llama-3.1-8B · WikiText | perplexity: 7.54 -> 7.86 | Hurt |
| ARM instead of full attention | Llama-3.1-8B · Recall-intensive tasks | average accuracy: 25.31 -> 25.40 | No clear effect |
| ARM instead of full attention | Llama-3.1-8B · LongBench LCC | edit similarity: 20.04 -> 11.21 | Hurt |
| ARM instead of full attention | Llama-3.1-8B · LongBench TREC | accuracy: 38.00 -> 62.50 | Improved |
Closest papers by meaning
- ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning (ICML 2026 · similarity 0.90)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.90)
- $\text{D}_{2}\text{O}$: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models (ICLR 2025 · similarity 0.90)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.90)
- CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.90)
- Training Free Exponential Context Extension via Cascading KV Cache (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
