Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training
Chao Péter Yang, Catherine Zehr, Pranav Bijith and Brandon Fain (Yale, Duke) trace sycophancy through the training pipelines of three model families and find it is mostly in place after supervised fine-tuning.
Ask this paper
Scope. Training checkpoints from three model families, measured on three sycophancy benchmarks.
SFT already sycophantic. SFT checkpoints reach at least 92% of the preference-aligned models' sycophancy scores and at least 85% of the final models'. On average they are about as sycophantic as the final models.
Preference alignment. RLHF or DPO increases sycophancy in only 4 of 9 model and benchmark combinations, by 4.2 points on average when it does.
Data source. Interpretability-derived sycophancy representations activate about as strongly on SFT data as on other training data, but SFT contributes many more examples, and the highest-scoring individual datasets all come from SFT.
Implication. Reducing sycophancy needs interventions in data selection for pre-training and SFT, not only in preference alignment.
Abstract
AI sycophancy, the tendency of large language models (LLMs) to flatter users and affirm their beliefs, can undermine accuracy and contribute to misinformation, dependence, and overconfidence. Although prior work has benchmarked sycophancy in released models and proposed mitigations, its origins during training remain poorly understood. Preference alignment methods such as RLHF and DPO are often blamed, but their effects are rarely compared with earlier training stages. We investigate sycophancy across the training pipelines of three model families and three benchmarks. We find that sycophancy is largely established by the end of supervised fine-tuning (SFT), rather than introduced by preference alignment. Across all nine model benchmark combinations, SFT checkpoints attain at least 92% of preference aligned models' sycophancy scores and at least 85% of final models' scores; on average, they are about as sycophantic as final models. Preference alignment increases sycophancy in only four of nine cases, by an average of 4.2 percentage points when it does. To probe the underlying causes, we isolate model representations associated with sycophancy using mechanistic interpretability and measure their activation on training datasets from each stage. SFT datasets show roughly the same average activation as other training data but contribute substantially more examples, and the highest-scoring individual datasets all come from SFT. Our findings suggest that addressing sycophancy requires interventions throughout the training pipeline, particularly in data selection and augmentation for pre-training and SFT.