Auditing Prompt Caching
Free while signed in. Answers cite the passages they came from.
Researchers from Stanford investigate how timing differences in LLM APIs can leak private user information through global prompt caching. They propose statistical audits to detect caching and reveal potentially significant security risks. Key insights include:
Side-channel timing attacks – When an LLM API caches prompts globally, repeat or prefix-matching prompts complete faster. Attackers can exploit these timing differences to infer what others have prompted, posing serious privacy concerns.
Statistical audit for detection – The paper introduces a hypothesis-testing method to systematically detect caching, distinguishing cache hits from misses using carefully constructed prompts. Empirically, the authors found multiple major API providers using global caches.
Architecture leakage – Timing differences for partial-prefix cache hits indicate a decoder-only Transformer backbone. The authors demonstrated that embedding models like OpenAI’s text-embedding-3-small are also susceptible, inadvertently leaking proprietary architectural details.
Responsible disclosure & mitigations – The authors notified affected API providers, many of whom updated documentation or disabled global caching. The recommended fix is per-user caching and transparent disclosures of caching policies to avoid privacy leakages.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack