SAFE-Teen: Evaluating Chatbot Safety Across Adolescent Mental Health Conversations
Andrea Mock, Diyi Yang, Caroline Figueroa and colleagues at Vals AI and Stanford run a clinician-informed safety evaluation of 18 chatbot APIs across 32,555 simulated conversations with adolescents aged 13-17.
Ask this paper
Design. 983 variants of 106 seed scenarios vary how much the teen discloses, whether they accept help and whether they push back. 22 safety criteria come from previously identified harms.
Overall. The criterion failure rate is 43.3% without a teen-aware system prompt and 24.9% with it.
Indirect risk. Failures to respond to indirect suicide risk remain at 52%.
Pushback. When the teen pushes back, failures on diagnosis, medication and substance-misuse boundaries rise 10-12 points without the prompt, for more than 80% of chatbots.
Implication. Safety evaluation has to follow the conversation past the first response and test different teen reactions.
Abstract
Adolescents are increasingly turning to AI chatbots for mental health support, advice, and companionship, raising safety concerns. While recent evaluations have started to examine safety across multiple conversational turns, few compare how slight changes in a teen's responses can affect a chatbot's safety in the same scenario. We conduct a clinician-informed evaluation of chatbot safety in simulated conversations with adolescents aged 13-17, examining how safety failures vary with adolescents' responses and teen-aware system prompting. We evaluated 32,555 simulated conversations across 18 chatbot APIs and two system-prompt conditions using 22 safety criteria grounded in previously identified harms. Conversations spanned 983 variants of 106 seed scenarios covering urgent safety issues, health advice, and identity and relationships. The overall criterion failure rate was 43.3% without the teen-aware prompt and 24.9% with it. The teen-aware prompt reduced failures, but it didn't eliminate them. When a teenager pushed back on a chatbot's answer, failures on mental health diagnosis, medication, and substance misuse boundaries rose by 10-12 percentage points without the prompt, for more than 80% of chatbots. These findings reveal persistent safety failures in simulated adolescent-chatbot interactions and show that safety depends on how a conversation unfolds.