Research map / Architectures and efficiency / Attention and transformers
Curse of High Dimensionality Issue in Transformer for Long Context Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates autoregressive sequence modeling as supervised learning, analyzes attention sparsity, and proposes Dynamic Group Attention (DGA), which groups low-importance tokens' keys/values to cut attention cost in long-context LLMs.
Problem
Long-context Transformer LLMs waste computation because all tokens consume equal attention resources even though attention weights are sparse, and token-dropping methods can disrupt needed token interactions.
Approach
Treats non-focal tokens as groups whose keys and values are aggregated via query-weighted pooling, keeps top-importance focal tokens individual, and adds complementary KV pairs so grouped tokens remain accessible under the causal mask; theory argues group coding reduces noise sensitivity and condition number.
Key results
- On LongBench-E, DGA-LLM reaches 21.14 average vs 21.69 for vanilla LLaMA2-7B, while cutting 16K inter-token latency to 28.80 ms from 69.70 ms (2.42x faster).
- On the EM score benchmark averaged over 4K-32K, DGA-LLM scores 27.7 versus 33.6 for vanilla LLaMA2-7B and 26.1 for LM-Infinite.
- At 16K, DGA-LLM's inter-token latency is 28.79 ms, 3.28x faster than MInference (94.34 ms) on a single A800 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Dynamic Group Attention instead of vanilla self-attention | LLaMA2-7B · LongBench-E | ITL / ms: 69.70 -> 28.80 | Helped |
| Use Dynamic Group Attention instead of vanilla self-attention | LLaMA2-7B · LongBench-E | EM score (Avg.): 33.6 -> 27.7 | Hurt |
| Use Dynamic Group Attention instead of vanilla self-attention | LLaMA2-7B · LongBench-E | Avg.: 21.69 -> 21.14 | Hurt |
| Use Dynamic Group Attention instead of vanilla self-attention | LLaMA2-7B · LongBench-E across context lengths | EM score: 4K: 37.2 -> 35.0; 8K: 36.4 -> 27.4; 16K: 33.8 -> 25.6; 32K: 26.8 -> 22.6 | Mixed |
| Use Dynamic Group Attention instead of vanilla self-attention | LLaMA2-7B · LongBench-E 4K | ITL / ms: 36.87 -> 26.26 | Helped |
| Use Dynamic Group Attention instead of vanilla self-attention | LLaMA2-7B · LongBench-E 8K | ITL / ms: 42.32 -> 26.87 | Helped |
| Use Dynamic Group Attention instead of vanilla self-attention | LLaMA2-7B · LongBench-E 16K | ITL / ms: 69.70 -> 28.79 | Helped |
| Use group coding in attention | DGA · OPT-125M validation training on SlimPajama | validation loss | Helped |
| Use group coding in attention | DGA · LLaMA2-7B question answering on SlimPajama | average KL-divergence under Gaussian noise | Helped |
Surprising findings
- Vanilla LLaMA2-7B outperforms DGA-LLM on the EM score average (33.6 vs 27.7), so the efficiency gain comes with a notable accuracy drop on retrieval-style evaluation.
- MInference and LM-Infinite have higher inter-token latency than vanilla self-attention in the reported decoding measurements.
Limitations
- Average LongBench-E performance is roughly on par with vanilla attention rather than improved, and EM score drops substantially versus vanilla.
- Pretraining experiments are short (1000 steps) and restricted to small models (GPT2-S, OPT-125M), so convergence claims are limited.
- Theoretical results rely on simplifying assumptions (single-layer, single-head, linear coding approximation, Gaussian noise), and the sparsity bound's dependence on the model is not fully characterized.
Experimental setup
Datasets: SlimPajama WikiText2 Proof-pile LongBench-E
Benchmarks: LongBench-E EM score on multi-document QA (4K-32K) inter-token latency
Metrics: average task score EM score inter-token latency (ms) validation loss KL divergence under Gaussian noise
Baselines: StreamingLLM LM-Infinite MInference Vanilla self-attention
Builds on: flash attention StreamingLLM MInference LongLoRA Position Interpolation
Closest papers by meaning
- Core Context Aware Transformers for Long Context Language Modeling (ICML 2025 · similarity 0.91)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.90)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.90)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.90)
- Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions (ICLR 2025 · similarity 0.89)
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.