Research map / Large language models / Language, safety and interpretability

ParallelComp: Parallel Long-Context Compressor for Length Extrapolation

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ParallelComp, a training-free parallel chunk-based long-context compression method with KV cache eviction and attention-bias calibration that lets 8B LLMs trained on 8K extrapolate to 128K on one A100 80GB.

Problem

Length extrapolation to ultra-long contexts (>128K) in LLMs is limited by memory bottlenecks and attention sink in training-free methods, especially chunk-based parallel attention.

Approach

Split input into chunks encoded in parallel with reused position encodings, rank chunks by query self-information and evict low-attention KV tokens per chunk, and calibrate attention bias by evicting tokens with abnormally high attention scores.

Key results

• On InfiniteBench beyond 128K, the 8B ParallelComp-based model reaches 91.17% of GPT-4's average performance and is the only model above 90% on the needle-in-a-haystack tasks (R.PK, R.Num, R.KV) as reported in the paper.
• On LongBench, the ParallelComp-Llama3.1 average is 38.15 versus 36.94 for the Llama3.1 baseline under a 24K KV budget.
• Compressing each chunk's KV cache to half size gives a 1.76x chunk throughput improvement and a reported 23.50x prefill speedup, per the paper's throughput table.

Measured effects reported in the paper

Surprising findings

• Parallel attention itself partially mitigates recency and sink bias relative to local attention, and KV cache eviction alone tends to exacerbate attention bias.
• Evicting high-attention 'bias' tokens in specific layer ranges can improve performance, with ablations showing layer-dependent effects of sink, recency, and middle token eviction.

Limitations

• Calibration threshold and retained-chunk hyperparameters are manually set and dataset-dependent.
• Some tasks show degradation in the compressed variants and the calibration effect on task performance is uncertain per the authors.
• Theoretical analysis relies on heuristic assumptions (e.g., isotropic sub-Gaussian dot products), with a proof sketch that is loosely argued.

Experimental setup

Datasets: LongBench InfiniteBench NarrativeQA

Benchmarks: LongBench InfiniteBench NarrativeQA perplexity

Metrics: accuracy F1 rouge perplexity throughput peak memory

Baselines: Dynamic-PI NTK-Aware ChunkLlama InfLLM AttenCalibration-NTK APE StarAttention GPT-4 Claude-2 Kimi-Chat

Builds on: ChunkLlama InfLLM NTK-Aware scaled RoPE Flash Attention AttenCalibration

Closest papers by meaning

• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.89)
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression (NeurIPS 2025 · similarity 0.89)
• Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.89)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.89)
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory (NeurIPS 2024 · similarity 0.88)