Verbalizing Subliminal Learning Effects Using Text Optimization
Nathan Hu, Sanmi Koyejo and Christopher Potts (Stanford) detect subliminal learning, where distillation data carries a teacher trait that is not legible in the data, by recovering the trait as a readable prompt.
Ask this paper
Theory: Prompted subliminal learning is a case of context distillation, so the dataset in principle identifies the teacher's prompt.
SALVE: The method optimizes a soft prompt on the data, asks the same model to verbalize it as text, and uses beam search to make the verbalization reliable; common text-optimization baselines fail at this task.
Data versus student: SALVE recovers the trait even from datasets where the student did not pick it up, and changing student training to improve context distillation can create the effect.
Wider coverage: It also detects traits in mixtures with unrelated data, in data from activation-steered teachers, and in preference data selected by Logit-Linear Selection for sycophancy and misalignment.
Abstract
Subliminal learning is a phenomenon in which a distillation dataset transmits traits from the teacher model that are not legibly encoded in the dataset itself. This introduces a new challenge for model development and creates new risks from data poisoning. In this work, we use text optimization to detect subliminal learning effects and describe them as legible prompts. Subliminal learning from a prompted teacher motivates our approach. We observe that this is a special case of context distillation and leverage this observation to show that, in theory, the prompted subliminal learning dataset identifies the teacher's prompt. We reduce recovering this prompt to a text optimization problem and present a method to approximately solve it. Our method, SALVE (Search-Aided Latent Verbalization), optimizes a soft prompt, queries the same model to verbalize it as text, and uses beam search to make the verbalization reliable. In the standard subliminal learning setting, SALVE reliably recovers legible prompts that name the teacher's trait, while common text optimization methods fail to do so. In addition, we find that there are settings in which SALVE recovers the teacher's trait from a dataset even when subliminal learning fails, but that modifying student training to improve context distillation can create subliminal learning effects. We lastly show that SALVE detects subliminal learning effects in three additional settings: (1) mixtures of subliminal learning data and unrelated data, (2) data generated when the teacher is biased via activation steering, and (3) subsets of real preference data selected via Logit-Linear Selection. Overall, our results deepen our understanding of subliminal learning and present SALVE as a method to proactively detect subliminal learning effects.