Research map / Large language models / Reasoning and chain of thought

vCache: Verified Semantic Prompt Caching

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes vCache, a semantic prompt cache that learns a per-embedding similarity threshold online via sigmoid logistic fitting, giving user-defined error-rate guarantees while maximizing cache hits.

Problem

Semantic caches for LLMs use a single static similarity threshold, which gives no formal error-rate guarantees, yields unexpected error rates, and suboptimal cache hit rates.

Approach

Learn an embedding-specific decision threshold online by fitting a sigmoid model of correctness versus similarity to observed cache outcomes, using confidence bounds and randomized exploration to enforce a user-specified error bound delta without offline training.

Key results

• vCache meets user-defined error bounds delta while steadily increasing cache hit rate, whereas GPTCache's error rate grows with sample size.
• On SemCacheLMArena, vCache achieves up to 26x lower error and 12.5x higher cache hit rate than static-threshold baselines.
• vCache reaches a 57% cache hit rate while keeping error below 0.5% on SemCacheLMArena.

Surprising findings

• No static threshold below 1.0 may keep error bounded as prompt diversity grows, since GPTCache error rises with sample size.
• GPTCache reported results likely overstate its performance because its error rates have not converged and trend upward.

Limitations

• Relies on i.i.d. data and the sigmoid family to model correctness probability; violations may invalidate the guarantee.
• LLM-as-a-judge for response equivalence on long responses adds an extra LLM inference, though run asynchronously.

Experimental setup

Datasets: LM-Arena human preference dataset three classification datasets web-search queries

Benchmarks: SemCacheLMArena SemCacheClassification SemCacheSearchQueries SemCacheCombo

Metrics: error rate cache hit rate latency ROC curve

Baselines: GPTCache GPTCache + fine-tuned embedding vCache + fine-tuned embedding

Builds on: GPTCache HNSW Extreme Value Machine

Closest papers by meaning

• MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation (ICML 2026 · similarity 0.91)
• Generative Caching for Structurally Similar Prompts and Responses (NeurIPS 2025 · similarity 0.90)
• Tail-Optimized Caching for LLM Inference (NeurIPS 2025 · similarity 0.88)
• LaCache: Robust Semantic Caching for LLM Serving (NeurIPS 2026 · similarity 0.87)
• SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.87)
• Cache-to-Cache: Direct Semantic Communication Between Large Language Models (ICLR 2026 · similarity 0.87)