Research map / Architectures and efficiency / Attention and transformers

Core Context Aware Transformers for Long Context Language Modeling

ICML 2025 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Core Context Aware (CCA) Attention, a plug-and-play attention replacement that pools token groups into weighted core tokens plus a local window, and fine-tunes pretrained LLMs for linear-cost long-context modeling.

Problem

Full self-attention over very long contexts (e.g., 128K) is computationally and memory expensive, and redundant context dilutes representations; existing sparse/fixed-pattern methods drop cross-token information.

Approach

Partition tokens into groups and compress each group into one core token via importance-weighted pooling (scored by the group's last token), attend to core tokens for distant context and raw neighboring tokens for local context, and fuse both in a single softmax; this reuses pretrained projections and needs only brief fine-tuning.

Key results

• On multi-document EM at 128K, CCA-LLM scores 32.0 vs 30.3 for LLaMA2-7B-80K with 7.9x lower first-token latency (15.89s vs 124.85s).
• On LongBench-E with LLaMA2-7B-80K, CCA-LLM averages 22.24 vs 22.42 for full attention while cutting first-token latency to 6.42s (5.7x) and memory to 33.86GB (44% lower).
• On LongBench-E with LLaMA3.1-8B-128K, CCA-LLM averages 37.81 vs 37.93 for the full-attention baseline, with 3.1x faster first-token latency and 49% less memory.

Measured effects reported in the paper

Surprising findings

• Full-attention LLaMA2-7B-80K's EM score drops at 128K relative to shorter contexts, while CCA-LLM's rises at 64K and 128K.
• Partial fine-tuning (only W^Q, W^K, W^V) reaches 21.96 average on LongBench-E versus 22.24 for full fine-tuning.

Limitations

• Gains over full attention are small on several benchmarks (e.g., LongBench-E average slightly below full attention for LLaMA2-7B-80K and LLaMA3.1-8B), and on RULER CCA-LLM trails full attention at 8K and 32K.
• Evaluation focuses on LLaMA2/LLaMA3.1/Qwen2.5 7-8B-class models with continued fine-tuning; no training-from-scratch results are reported.

Experimental setup

Datasets: SlimPajama

Benchmarks: LongBench-E LongBench RULER multi-document EM score (Lost in the Middle) MMLU Proof-pile

Metrics: average score exact match first token latency memory footprint perplexity accuracy

Baselines: LLaMA2-7B-32K LLaMA2-7B-80K StreamingLLM LM-Infinite MInference LongLoRA

Builds on: LongLoRA Position Interpolation FlashAttention

Closest papers by meaning

• Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.91)
• Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.90)
• A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.90)
• Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.89)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
• Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.89)