Research map / Architectures and efficiency / Quantization

When Less is More: The LLM Scaling Paradox in Context Compression

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: KV cache quantization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that larger memory-token compressors (0.6B–90B, Qwen3 and LLaMA-3.2) reconstruct text with higher surface scores yet produce less faithful reconstructions on knowledge-overwriting and semantic-drift QA, with mid-sized (3–4B) compressors most faithful.

Problem

Scaling compressor size in lossy context compression (compressor–decoder with memory tokens) is assumed to improve quality, but standard reconstruction metrics may hide fidelity failures where source facts are overwritten by parametric priors or where structural relations drift.

Approach

Identifies a Size-Fidelity Paradox and isolates two failure modes with diagnostic QA tasks (knowledge overwriting via counterfactual FaithEval/ConflictQA; semantic drift via DeepSeek-R1-generated structural QA over FineWeb/FaithEval), then links them to effective rank of memory embeddings and teacher-forced conditional entropy.

Key results

• At 64x compression, LLaMA-3.2-90B has higher BLEU than LLaMA-3.2-3B (0.33 vs 0.29) but lower knowledge-overwriting accuracy (0.53 vs 0.62 average).
• At 4x compression, Qwen3-32B's semantic-drift average is 0.69 versus 0.83 for Qwen3-4B, a 16.87% relative drop from group best despite BLEU of 0.97.
• Qwen3 decoder ablation at 16x shows the 8B compressor has higher overwriting and drift error than the 4B compressor under both Qwen3-0.6B and Qwen3-4B decoders.

Surprising findings

• Larger compressors achieve lower training loss and higher reconstruction scores yet produce less faithful reconstructions on diagnostic QA.
• Fidelity-oriented QA accuracy peaks around 3–4B compressors rather than at the largest scale.

Limitations

• Conclusions are tied to memory-token compressor-decoder architectures and training objectives; other compression paradigms may scale differently.
• Effective rank and conditional entropy are diagnostic associations rather than a causal account of the paradox.
• Diagnostic QA tasks stress source-specific fidelity and may not reflect all downstream applications.

Experimental setup

Datasets: Fineweb FaithEval ConflictQA ArxivCorpus MuSiQue QASPER BBH

Benchmarks: FaithEval ConflictQA FineWeb semantic drift QA MuSiQue QASPER BBH

Metrics: BLEU ROUGE-L chrF++ BERTScore QA accuracy EM F1 effective rank conditional entropy

Baselines: Full ICL Zero-shot w/ Context w/o Context

Builds on: FaithEval ConflictQA Fineweb

Closest papers by meaning

• Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission (NeurIPS 2026 · similarity 0.91)
• Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.89)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.89)
• Accuracy is Not All You Need (NeurIPS 2024 · similarity 0.88)
• Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.88)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.88)