Research map / Architectures and efficiency / Attention and transformers

Dual Dimensionality for Local and Global Attention

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Distance-Adaptive Representation (DAR), which keeps full-dimensional keys/values for a local window and projects tokens beyond it to a lower-dimensional bottleneck, and shows via Pythia pretraining and OLMo-2-1B SFT that this matches full-dimensional attention while uniform reduction hurts.

Problem

Decoder-only Transformers use the same key/value dimensionality for all preceding tokens regardless of distance, even though nearby tokens may matter more for next-token prediction; it is unclear whether distant tokens need full representational capacity.

Approach

Allocate representational capacity by token distance: full-dimensional keys/values inside a local window and a learned low-dimensional bottleneck (down-projection then up-projection) for tokens outside it, trained end-to-end with the standard next-token loss, contrasted with uniform dimensionality reduction.

Key results

• At Pythia-70M, DAR with d_down=128 (w=128) reaches Rel. 99.61% perplexity vs Vanilla, while uniform reduction at d_down=128 reaches 105.49%.
• At Pythia-410M, DAR with d_down=d/4 achieves Rel. 97.98% perplexity relative to Vanilla.
• On OLMo-2-1B SFT downstream tasks, d_down=512 (d/4) gets Rel. 102.19% and d_down=256 gets 98.68%, while d_down=128 drops to 88.29%.

Surprising findings

• DAR with reduced distant-token dimensionality slightly outperforms the full-dimensional baseline in perplexity at several scales.
• Window-size sweep shows performance is stable for w>=4, suggesting only a handful of nearby tokens need full dimensionality.

Limitations

• Current implementation stores two sets of KV states per token, doubling KV cache memory rather than reducing it.
• Experiments limited to 70M-410M pretraining and a 1B SFT setting; no larger scales or other architectures tested.
• Global path up-projects back to full dimension rather than computing attention in the reduced space.

Experimental setup

Datasets: The Pile Tulu 3 SFT mixture FineWeb-Edu WikiText-103 C4

Benchmarks: FineWeb-Edu perplexity C4 perplexity WikiText-103 perplexity MMLU HellaSwag CommonsenseQA GSM8K MBPP Multi-News (LongBench)

Metrics: perplexity relative perplexity (Rel.%) accuracy normalized accuracy exact match pass@1 ROUGE

Baselines: Vanilla full-dimensional attention Uniform dimensionality reduction Multi-head Latent Attention StreamingLLM H2O

Builds on: Multi-head Latent Attention Pythia sliding window attention RoPE

Closest papers by meaning

• Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.87)
• $\text{D}_{2}\text{O}$: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models (ICLR 2025 · similarity 0.87)
• Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.87)
• Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.87)
• Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.87)
• SALS: Sparse Attention in Latent Space for KV Cache Compression (NeurIPS 2025 · similarity 0.87)