Research map / Architectures and efficiency / Attention and transformers

Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ComprExIT, a soft context compression method that treats a frozen LLM as a feature extractor, aggregating multi-layer hidden states into token anchors and allocating them to compression slots via entropy-regularized optimal transport (Sinkhorn).

Problem

LLM-as-a-compressor soft context compression methods lose contextual information relative to using the full context, due to poor coordination among compression tokens (lack of allocation) and dilution of useful signals across layers.

Approach

Decouple compression from self-attention: use a frozen LLM's hidden states, gate-mix them across layers into per-token anchors (depthwise), then allocate anchors to K compression slots through a globally coordinated optimal transport plan with marginal constraints (widthwise), instead of independent per-slot attention.

Key results

• On six in-domain QA datasets with Llama-3.2-1B, ComprExIT reaches 66.55 average F1 vs 56.16 for ICAE (about 18.5% gain over the strongest compression baseline).
• With Llama-3.2-3B, ComprExIT averages 72.88 F1, within 0.64 of the uncompressed prompt-tuning baseline (73.35) and 10.56% above the strongest compression baseline.
• Compression stage is more than 2x faster than ICAE (0.18s vs 0.43s at context length 512).

Surprising findings

• Final-layer hidden states perform worst among single layers for compression, while intermediate layers are more informative.
• Naive ICAE gist-token attention is highly inter-correlated with a steep singular-value decay, suggesting redundant compression slots.

Limitations

• Not tested on larger backbones (32B or 70B) due to limited compute.
• Not tested on substantially longer contexts (32k or 128k tokens).
• Only text-only language models evaluated; vision-language extension left to future work.

Experimental setup

Datasets: SlimPajama SQuAD NewsQA TriviaQA SearchQA HotpotQA Natural Questions MRQA out-of-domain sets XSum FEVER SST-2 QuALITY

Benchmarks: SQuAD NewsQA TriviaQA SearchQA HotpotQA NQ BioASQ DROP DuoRC RACE RelationExtraction TextbookQA

Metrics: EM F1 ROUGE-1 ROUGE-L latency (s) reconstruction NLL

Baselines: ICAE 500x Activation Beacon Prompt tuning Zero-shot [w/ context] Zero-shot [w/o context]

Builds on: ICAE Denseformer Sinkhorn algorithm Activation Beacon 500x compressor

Closest papers by meaning

• When Less is More: The LLM Scaling Paradox in Context Compression (NeurIPS 2026 · similarity 0.91)
• Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.90)
• EvoPress: Accurate Dynamic Model Compression via Evolutionary Search (ICML 2025 · similarity 0.89)
• Long Context Compression with Activation Beacon (ICLR 2025 · similarity 0.89)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.89)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.88)