Research map / Large language models / Language, safety and interpretability

Generative Caching for Structurally Similar Prompts and Responses

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GenCache, a generative client-side cache that clusters structurally similar prompt-response pairs, synthesizes a validated regex-based Python program per cluster, and executes it on cache hits to produce variation-aware responses.

Problem

Exact prompt matching misses structurally similar prompts with minor variations, while semantic caching returns incorrect cached responses that ignore critical differences, which is risky for agentic actions.

Approach

Instead of storing LLM responses verbatim, cache a program generated by an LLM from in-context exemplars of a prompt-response cluster; the program (with a stored regex for prompt structure) synthesizes customized responses for new prompts locally, validated by an LLM before caching and with feedback-driven regeneration.

Key results

• On Param-Only synthetic prompts, GenCache reaches 97.81% hit rate with 1.97% negative hits vs. GPTCache 90.92% hit rate with 100% negative hits.
• On Param-w-Synonym, GenCache hit rate is 83.66% with 7.84% negative hits; GenCache-feedback lowers negative hits to 4.4%.
• In agentic workflows, GenCache raises cache hit rate for the FLASH cloud-ops agent to 54.7% from 34.4% with exact matching and cuts execution time from 53.45s to 39.91s; on Laser it reaches 37.2% vs 5.7%.

Measured effects reported in the paper

Surprising findings

• GPTCache returns incorrect cached responses for 100% of its hits on the non-repeating synthetic prompts, showing semantic caching is unsafe without prompt repetition.
• On structurally diverse prompts, GenCache's hit rate collapses to 4.23% with 27.6% negative hits, so it is ineffective outside its target regime.

Limitations

• Fails on structurally diverse prompts where cached regexes do not generalize; only 10.4% of cache-creation LLM calls yielded reusable caches in FLASH.
• Targets only reversible actions; incorrect cache hits can cause erroneous non-reversible actions.
• Evaluated mostly on synthetic prompts derived from WebShop and a small proprietary incident dataset (298 incidents).

Experimental setup

Datasets: WebShop Microsoft incident diagnosis data (298 incidents)

Benchmarks: WebShop-derived synthetic prompt sets (Param-Only, Param-w-Synonym, Structural) Laser web agent on WebShop (200 requests) Cloud-operations agent FLASH incidents

Metrics: cache hit rate positive hit rate negative hit rate LLM calls token usage execution time

Baselines: exact prompt matching GPTCache GenCache-feedback

Builds on: GPTCache ReAct SentenceTransformers in-context learning

Closest papers by meaning

• vCache: Verified Semantic Prompt Caching (ICLR 2026 · similarity 0.90)
• MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation (ICML 2026 · similarity 0.87)
• CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.86)
• EPIC: Efficient Position-Independent Caching for Serving Large Language Models (ICML 2025 · similarity 0.86)
• Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents (NeurIPS 2025 · similarity 0.86)
• Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMs (ICML 2025 · similarity 0.86)