Research map / Large language models / Reasoning and chain of thought

ARC-Encoder: learning compressed text representations for large language models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ARC-Encoder, a trainable encoder with pooled hidden states and a small MLP that compresses context into continuous embeddings consumed by frozen decoder LLMs, shared across multiple decoders.

Problem

Long contexts from RAG and chain-of-thought raise inference cost, but effective soft context compression usually requires fine-tuning or modifying the decoder, which can degrade general abilities.

Approach

An encoder built from a truncated Llama3.2 3B (no causal mask) averages hidden-state queries in the last self-attention layer to produce x-times fewer continuous representations fed directly after the decoder's embedding layer; the decoder stays frozen, and a shared encoder is adapted to several decoders via small per-decoder MLPs and special tokens.

Key results

• With Mistral 7B decoder, ARC-Encoder at 4x pooling (multi-decoder, shared encoder) reaches 45.6 average vs 49.2 open-book and 39.5 for the best PISCO-like baseline at 4x.
• With Llama3.1 8B decoder, the Llama-specific ARC-Encoder4 reaches 48.0 average, slightly above open-book (47.4), with about 3.0B encoder parameters.
• On ZeroSCROLLS-style long-context tasks, ARC-Encoder8 with Llama2 7B Chat gets NQA 27.5 vs 16.1 for the Llama2 Chat 4k baseline, and Qspr 28.3 vs 17.2.

Surprising findings

• Pretraining is essential: fine-tuning without pretraining performs far worse, with pretraining adding roughly +16 to +19 average points.
• Truncating the encoder to 0 layers (3.2B) gives similar or better average than the default 3.0B encoder, suggesting the removed layers add little.

Limitations

• Depends heavily on task formatting in the fine-tuning data, limiting long-context summarization performance.
• Performance degrades at higher compression ratios (e.g., 32x).
• Long-context extension only tested on models with small native context windows (4k).

Experimental setup

Datasets: Common Crawl (filtered via dactory) HotpotQA Natural Questions TriviaQA SQuAD FLORES CNN-DailyMail PG-19 RedPajama ArXiv Wikipedia

Benchmarks: Natural Questions TriviaQA HotpotQA SQuAD FLORES CNN-DailyMail NarrativeQA Qasper GovReport QMSum PubMedQA BioASQ

Metrics: exact match BLEU ROUGE-L F1

Baselines: LLMLingua2 ICAE-like xRAG-like PISCO-like CEPE Llama2-32k Instruct

Builds on: ICAE PISCO memory tokens xRAG

Closest papers by meaning

• Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference (AAAI 2025 · similarity 0.88)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.88)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.88)
• In-context Autoencoder for Context Compression in a Large Language Model (ICLR 2024 · similarity 0.88)
• Long Context Compression with Activation Beacon (ICLR 2025 · similarity 0.88)
• Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.88)