Logits of API-Protected LLMs Leak Proprietary Information
Free while signed in. Answers cite the passages they came from.
The paper shows that the softmax bottleneck in modern LLMs means even logit-level APIs leak enough information to reconstruct hidden architectural details.
Softmax bottleneck exploit: Because the output distribution is a linear projection of a lower-dimensional embedding, clever API queries can expose the embedding rank and related structural parameters.
Cheap recovery: With fewer than $1,000 in API calls, the authors estimate the embedding dimension of OpenAI's gpt-3.5-turbo at ~4,096 and recover related non-public quantities.
Auditing use-case: The same technique can detect silent model updates, identify shared ancestry between different provider models, and discover hidden layer widths - making it a potential auditing tool as much as an attack.
Mitigations: The paper outlines defenses providers can deploy (logit-bias restrictions, output calibration) while arguing that some transparency gains from this capability are net positive.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack