Research map / Large language models / Reasoning and chain of thought

ICaRus: Identical Cache Reuse for Efficient Multi-Model Inference

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ICaRus, which freezes the base model's logical encoder (KV-cache generator) and fine-tunes only LoRA-adapted logical decoders so multiple task-specialized models share one identical KV cache, cutting memory and recomputation in multi-model agent serving.

Problem

In multi-model agentic inference each model keeps its own KV cache for the same prompt, causing memory explosion, cache eviction and recomputation, and prefix caching cannot be shared across different models.

Approach

Decompose a decoder-only Transformer into a logical encoder (produces KV cache) and logical decoder (predicts next token from the cache); freeze the encoder (base model) during training and fine-tune only the decoder with LoRA-style adapters, so all task models reuse an identical KV cache across all layers. At inference, the base and adapted paths run in parallel in the decode phase, sharing weigh

Key results

• ICaRus with LLaMA-3.1-8B matches multi-model accuracy (avg 46.9 vs 47.0 for three independent models) while sharing one KV cache across three task models.
• In multi-agent ReAct serving on LLaMA-3.1-8B, ICaRus improves maximum throughput by 1.4x, 2.3x, and 3.8x with 2, 4, and 8 models, and reduces P95 latency up to 11.1x at the baseline's peak throughput QPS.
• On Qwen3-8B-Base ICaRus beats the multi-model baseline on GSM8K (87.3 vs 85.4) and HumanEval (86.6 vs 81.7); on Qwen3-14B-Base it improves GSM8K from 85.6 to 88.8.

Surprising findings

• ICaRus sometimes exceeds independently fine-tuned task-specific models in accuracy, which the authors attribute to implicit regularization from freezing the encoder.
• Under swap-based KV management the latency gains persist (up to 12.1x P95), indicating the benefit is orthogonal to the eviction policy.

Limitations

• Accuracy is slightly below independent task models on some benchmarks (e.g., GSM8K 67.9 vs 69.7 and GPQA 28.8 vs 27.2 mixed), and evaluation covers only a few tasks and model families.
• Requires training each specialized decoder with a shared frozen encoder; models trained independently cannot share caches without retraining.
• Latency and throughput evaluations use a synthetic workload derived from HotPotQA and sampled agent statistics rather than real deployments.

Experimental setup

Datasets: MetaMathQA-40k Evol-Instruct-Code-80k OASST1 ToolAce HotPotQA

Benchmarks: GSM8K GSM-Plus HumanEval HumanEval+ GPQA-Diamond BFCL HotPotQA

Metrics: zero-shot accuracy P95 latency throughput (QPS)

Baselines: Multi Model (independent fine-tuned models) Base Model multi-LoRA serving baseline DroidSpeak KVFlow

Builds on: LoRA vLLM DroidSpeak KVFlow ReAct

Closest papers by meaning

• RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse (ICML 2026 · similarity 0.90)
• KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems (NeurIPS 2025 · similarity 0.87)
• AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systems (NeurIPS 2026 · similarity 0.87)
• KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.86)
• CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.86)
• Beyond Speedup - Utilizing KV Cache for Sampling and Reasoning (ICLR 2026 · similarity 0.86)