Research map / Large language models / Language, safety and interpretability
Auditing Prompt Caching in Language Model APIs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops statistical timing audits (two-sample KS test with valid p-values) to detect prompt caching and its sharing level in 17 real-world LLM APIs, finding global cache sharing in seven and inferring OpenAI's text-embedding-3-small is decoder-only.
Problem
Prompt caching in LLM APIs causes data-dependent timing differences that enable side-channel timing attacks and privacy leakage across users, and providers rarely disclose their caching policies or cache-sharing level. It also can leak architecture information such as whether a model is decoder-only
Approach
Compare time-to-first-token distributions from a cache-hit procedure (victim sends prompt, attacker sends prefix-sharing prompt) against cache-miss random prompts using a one-sided two-sample Kolmogorov-Smirnov test, with the victim/attacker user placement set to test per-user, per-organization, or global sharing.
Key results
- Caching detected in 8 of 17 API providers, with global cache sharing detected in 7 providers.
- Average precision for distinguishing cache hits from misses mostly around 0.8 across detected APIs; Replicate reached 1.00 server-side.
- Evidence that OpenAI text-embedding-3-small is a decoder-only Transformer, inferred from prefix caching with different suffixes; Claude 3 Haiku and GPT-4o mini showed only per-organization sharing, matching documentation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use longer prompts (ablation) | Prompt-caching audit on globally caching Fireworks, Perplexity, and Replicate APIs · Cache-hit versus cache-miss classification | average precision | Helped |
| Use shorter prompts (ablation) | Prompt-caching audit · Cache-hit versus cache-miss classification | average precision | Hurt |
| Increase the matching-prefix length (ablation) | Prompt-caching audit with same-prefix, different-suffix prompts · Cache-hit versus cache-miss classification | average precision | Helped |
| Decrease the matching-prefix length (ablation) | Prompt-caching audit with same-prefix, different-suffix prompts · Cache-hit versus cache-miss classification | average precision | Hurt |
| Vary model size (ablation) | Fireworks Llama 3.1 and 3.2 models · Cache-hit versus cache-miss classification | average precision | No clear effect |
| Share the cache globally across users | Seven API providers where global cache sharing was detected · Cross-user prompt privacy | global cache sharing detected | Hurt |
| Use same-prefix prompts with different suffixes | OpenAI text-embedding-3-small · Model architecture inference | decoder-only architecture evidence | Helped |
| Use prompt caching with response-time classification | DeepSeek Chat · DeepSeek Chat cache-hit detection | statistically significant difference between cache-hit and cache-miss times | No clear effect |
Surprising findings
- OpenAI text-embedding-3-small global cache sharing was a vulnerability that was patched after disclosure; cache hits returned slightly different embeddings (around 1e-4 to 1e-5 per coordinate) than cache misses.
- Practical prompt extraction attacks were not achieved despite prefix-caching leakage.
Limitations
- Prompt extraction token-by-token attacks were not made practical; difficulty detecting a single additional cached token.
- Audits depend on API routing and multi-server caches, requiring many victim requests (up to 25) in some APIs.
- Cannot identify implementation details of caching; DeepSeek caching was not detected by timing despite cache-hit tokens reported.
Experimental setup
Datasets: randomly generated letter prompts (PromptLength 5000)
Benchmarks: audit p-values across 17 LLM API providers precision-recall curves for cache-hit vs cache-miss timing
Metrics: p-value average precision
Baselines: Anthropic Claude 3 Haiku caching OpenAI GPT-4o mini caching
Builds on: Kolmogorov-Smirnov test KV cache reuse prompt caching
Closest papers by meaning
- RepeatLeakage: Leak Prompts from Repeating as Large Language Model Is a Good Repeater (AAAI 2025 · similarity 0.86)
- vCache: Verified Semantic Prompt Caching (ICLR 2026 · similarity 0.86)
- LaCache: Robust Semantic Caching for LLM Serving (NeurIPS 2026 · similarity 0.85)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.85)
- Quantifying and Analyzing Entity-Level Memorization in Large Language Models (AAAI 2024 · similarity 0.85)
- Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.