Research map / Large language models / Reasoning and chain of thought

MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation

ICML 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MVR-cache, a semantic caching method that learns a lightweight pointer-network prompt segmenter (trained with REINFORCE on a theory-derived BCE reward) so that multi-vector MaxSim matching improves cache hit rates under vCache's error guarantees.

Problem

Semantic caches for LLM inference rely on single-vector cosine similarity, which misses subtle semantic differences and yields low cache hit rates while needing to respect error guarantees.

Approach

Learn a variable-length prompt segmentation policy (BERT + MLP + LSTM pointer network) that jointly segments cached and new prompts, scored by a symmetric segmentation-aware MaxSim, and train it with REINFORCE using a reward derived from the vCache MLE objective.

Key results

• Up to 37% higher cache hit rate than vCache under the always-cache protocol on SemCacheSearchQueries (abstract says up to 37%; introduction says up to 25%).
• Reduces end-to-end inference time by up to 6% under the same error bound on cache-on-miss protocol.
• On SemCacheClassification, MVR-cache increases cache hits by 9% over vCache.

Surprising findings

• Two-stage retrieval (HNSW top-20 then MVR rerank) achieves nearly the same recall as full MVR scan (0.179 vs 0.183), so single-vector retrieval serves as an effective coarse stage.
• Symmetric MaxSim gives only a slight gain over vanilla unidirectional MaxSim; larger candidate split sets do not improve over punctuation-level splitting.

Limitations

• Requires offline labeled training data (ground-truth LLM responses for 3K prompts per dataset) and a per-dataset segmentation model.
• Theoretical guarantees rely on Gaussian class-conditional and balanced-class assumptions that the paper only partly validates empirically.
• Only text inputs are handled; multimodal extension left for future work.

Experimental setup

Datasets: SemCacheSearchQueries (ORCAS subset) SemCacheClassification PromptBench (SQuAD-v2 perturbations) QNLI (perturbed)

Benchmarks: SemCacheSearchQueries SemCacheClassification PromptBench QNLI

Metrics: cache hit rate error rate latency

Baselines: vCache ColBERT POQD

Builds on: vCache ColBERT POQD pointer network REINFORCE

Closest papers by meaning

• vCache: Verified Semantic Prompt Caching (ICLR 2026 · similarity 0.91)
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.87)
• SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.87)
• Generative Caching for Structurally Similar Prompts and Responses (NeurIPS 2025 · similarity 0.87)
• EPIC: Efficient Position-Independent Caching for Serving Large Language Models (ICML 2025 · similarity 0.86)
• Training Free Exponential Context Extension via Cascading KV Cache (ICLR 2025 · similarity 0.86)