Watermark Basins: Black-Box Detection of Distilled LLMs
Hongyan Chang, Hady Elsahar, Tom Sander, Pierre Fernandez and colleagues at FAIR (Meta Superintelligence Labs) show that watermarked teacher outputs leave a key-specific direction in a distilled student's parameters, and use it to build WM-Basin, a black-box detector that finds distilled models far more often than token-level tests.
Ask this paper
The signal. Fine-tuning on watermarked teacher traces writes a key-specific geometric signal into parameter space. That signal lies orthogonal to the ordinary task fine-tuning subspace, so it persists when token statistics no longer show it.
Detector. WM-Basin queries the suspect model through its API only. It maps the generated text to gradients of a fixed clean probe model, projects out the task directions, and tests whether the residual aligns with the basin for the watermark key.
Results. At a 0.1% false-positive rate it detects 79% of distilled students at n-gram=1 and 45% at n-gram=3, against 29% and 1.3% for the best token-level baseline. Students range from 135M to 8.2B parameters, distilled on mathematical reasoning.
Robustness. The signal survives quantization, pruning and reinforcement-learning post-training after distillation, which are the usual ways to obscure a distilled model.
Abstract
Proprietary language models and their reasoning traces are vulnerable to API-based distillation, where an adversary trains a student model on the teacher model's outputs. To obtain evidence of distillation, model providers can watermark the teacher's outputs, aiming to leave a radioactive signal in the student's outputs. Existing black-box detectors search for such radioactivity using token-level statistics of the suspect model's outputs. However, token-level signals are not always detectable, for instance when the distillation traces are sampled at a low temperature. In this work, we discover the watermark basin, a persistent signal that enables more reliable watermark detection: during fine-tuning, watermarked teacher traces embed a distinct, key-specific geometric signal in parameter space that lies orthogonal to the ordinary task-specific fine-tuning subspace. Building on this, we introduce Watermark Basin Alignment (WM-Basin), a black-box watermark detector that queries the suspect model through its API, without access to its weights or logits. WM-Basin maps generated text to gradient representations using a fixed clean probe model, projects out the ordinary fine-tuning task directions, and tests whether the residual gradient aligns with the watermark basin induced by the watermark key. We evaluate WM-Basin extensively across diverse teacher models and student architectures (from 135M to 8.2B parameters) distilled on mathematical reasoning. At a strict 0.1% false-positive rate, WM-Basin detects 79% of distilled models at n-gram=1 and 45% at n-gram=3, substantially improving over the state-of-the-art token-level baseline (29% and 1.3%, respectively). Because the watermark signal resides in a persistent basin, it survives post-distillation modifications including quantization, pruning, and reinforcement-learning post-training.