Evaluating and Improving LLM Self-Modeling
Siqi Zeng, Andre N. Assis and Rowan Wang, working through the Anthropic Fellows Program, measure whether a model can answer verifiable questions about its own behavior, and then try to train the ability in.
Ask this paper
Self-modeling is scoped to verifiable questions. The target is behavior a third party can check, such as whether a specific prompt edit would change the model's final answer, which avoids the usual trap of unfalsifiable introspection claims.
Current models are limited and systematically wrong. They show non-trivial skill but make consistent mistakes on simple counterfactual questions about their own behavior.
A synthetic-data pipeline plus RL improves the aggregate score across three open-source model families, with some transfer to held-out tasks.
The authors decline the introspection interpretation. They state the gains may not come from privileged access to the model's internal decision process, so a higher self-modeling score is not evidence of introspection.
Practical relevance. Agents that predict their own failure modes are useful for routing and escalation, and this gives a way to measure that capability without assuming the mechanism behind it.
Abstract
We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.