Research map / Large language models / Reasoning and chain of thought

RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

ICML 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RelayCaching, a training-free method that reuses decoding-phase KV caches from previous agents in downstream prefill and selectively recomputes deviating tokens in middle layers, cutting TTFT up to 4.7x in multi-agent LLM pipelines.

Problem

Multi-agent LLM pipelines redundantly re-prefill shared content generated by upstream agents, inflating KV cache memory and time-to-first-token, while existing prefix caching and pre-computed KV reuse methods either lose accuracy on agent outputs or have low reuse rates.

Approach

Directly reuse decoding KV caches in later agents' prefills despite prefix variation, using a layer-range profiler (U-shaped value-cosine similarity profile and inter-layer correlation of token deviations) to restrict rectification to middle layers, and a token selector combining deviation-based and attention-based influence criteria to recompute only a sparse set of tokens.

Key results

• RelayCaching matches full-prefill accuracy (e.g., GSM8K 84.84% vs 85.97% for 2 agents on Llama-3.1-8B) while maintaining over 80% KV reuse in most multi-agent settings.
• On Qwen3-0.6B with AIME prompts, RelayCaching reduces per-agent TTFT by up to 4.71x at Agent 5 (104.8 ms vs 493.9 ms for full prefill).
• At 12k cumulative tokens, RelayCaching achieves 0.42 s TTFT versus 3.89 s for full prefill (9.2x) and 1.08 s for CacheBlend.

Surprising findings

• Direct reuse without rectification (Zero Reuse) drops GSM8K accuracy to about 40% on Llama-3.1-8B, showing decoding KV caches are not safely reusable as-is.
• Pre-computed caching methods EPIC and CacheBlend lose substantial accuracy on decode-to-prefill reuse (EPIC 58.73% on GSM8K).

Limitations

• Only homogeneous multi-agent systems are supported; heterogeneous models cannot directly share KV caches.
• Hyperparameters (layer range, thresholds) are determined by offline profiling and fixed thresholds may not transfer to tasks with very different distributions.
• Several results rely on a single model for the main analysis (Mistral-7B) and on a small set of agent workflows.

Experimental setup

Datasets: 2WikiMQA GSM8K HumanEval MMLU AIME 2024

Benchmarks: GSM8K MMLU HumanEval AIME 2024

Metrics: accuracy pass@1 reuse rate TTFT

Baselines: Full Prefill Zero Reuse CacheBlend EPIC KVCOMM

Builds on: KVCOMM CacheBlend EPIC

Closest papers by meaning

• KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems (NeurIPS 2025 · similarity 0.92)
• ICaRus: Identical Cache Reuse for Efficient Multi-Model Inference (ICLR 2026 · similarity 0.90)
• LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents (ICML 2026 · similarity 0.90)
• AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systems (NeurIPS 2026 · similarity 0.90)
• Cache-to-Cache: Direct Semantic Communication Between Large Language Models (ICLR 2026 · similarity 0.90)
• Hogwild! Inference: Parallel LLM Generation via Concurrent Attention (NeurIPS 2025 · similarity 0.89)