Auditing Prompt Caching
Researchers from Stanford investigate how timing differences in LLM APIs can leak private user information through global prompt caching. They propose statistical audits to detect caching and reveal potentially significant security risks. Key insights include:
Ask this paper
Side-channel timing attacks – When an LLM API caches prompts globally, repeat or prefix-matching prompts complete faster. Attackers can exploit these timing differences to infer what others have prompted, posing serious privacy concerns.
Statistical audit for detection – The paper introduces a hypothesis-testing method to systematically detect caching, distinguishing cache hits from misses using carefully constructed prompts. Empirically, the authors found multiple major API providers using global caches.
Architecture leakage – Timing differences for partial-prefix cache hits indicate a decoder-only Transformer backbone. The authors demonstrated that embedding models like OpenAI’s text-embedding-3-small are also susceptible, inadvertently leaking proprietary architectural details.
Responsible disclosure & mitigations – The authors notified affected API providers, many of whom updated documentation or disabled global caching. The recommended fix is per-user caching and transparent disclosures of caching policies to avoid privacy leakages.