Where to show demos in your prompt?
Free while signed in. Answers cite the passages they came from.
Introduces DPP bias, a new kind of positional sensitivity in LLM where the location of demonstrations in a prompt significantly affects output accuracy and stability. While prior work focused on demo content and order, this study reveals that moving an identical demo block across different sections of a prompt, e.g., before vs. after the user query, can change accuracy by up to 20 points and flip a large percentage of model predictions.
Four canonical demo positions were evaluated: start or end of the system prompt (ssp, esp) and start or end of the user message (sum, eum). Placing demos at the start of the system prompt (ssp) consistently delivered the best performance across most tasks and models, while placing them after the query (eum) degraded accuracy and induced high volatility.
Two new metrics, Accuracy-Change and Prediction-Change, were introduced to quantify how performance and decision stability are impacted purely by demo placement.
Smaller models (e.g., Qwen-1.5B, LLAMA3-3B) are highly sensitive to demo position. For instance, on the AG News dataset, accuracy dropped from 76% (ssp) to 56% (eum) for Qwen-1.5B. In contrast, larger models like LLAMA3-70B show more stability but still exhibit shifts in optimal positioning depending on the task.
Scaling trends show that as model size increases, both accuracy differences and prediction flips caused by positional changes decrease. However, in generation tasks like summarization (e.g., XSUM, CNN/DM), even the largest models remain fragile, with prediction flip rates near 100% for late-positioned demos.
No universal best position: While ssp dominates in classification and reasoning tasks, sum or even eum occasionally performs better in generative or arithmetic settings, especially for larger models like Qwen-72B or LLAMA3-70B.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack