Forecasting Rare Language Model Behaviors

Elicitation probabilities - By sampling multiple outputs from a query and measuring how often a target (undesired) behavior occurs, they estimate how "at-risk" each query is. Even prompts that appear safe can have a low-but-nonzero probability of producing harmful responses.

Power-law scaling of risks - The authors show that the largest elicitation probabilities (the worst-case queries) grow predictably with the number of queries sampled. This allows them to forecast extreme tail risks-like chemical or power-seeking "jailbreaks"-from smaller-scale tests.

Multiple safety metrics - They formalize metrics such as worst-query risk (the maximum single probability of a bad behavior), behavior frequency (fraction of queries likely to succeed in eliciting it), and aggregate risk (chance any query draws out the failure). All can be extrapolated to larger deployment volumes.

Improved red-teaming - By identifying which model (or how much sampling) best uncovers failures, they can allocate limited red-teaming budget more efficiently. The framework highlights potential pitfalls before models process billions of queries.