Evaluating Language Models in Realistic Conversational Contexts
Ilija Subasic, Andrew Rabinovich, and Zhao Chen (Upwork) release UPHELD, a reference-full benchmark of professionally scripted human-to-human dialogues with 36,000 per-turn human annotations, then show standard automatic metrics and LLM judges correlate poorly with expert judgment.
Ask this paper
Human-authored, not synthetic: Hundreds of complete dialogues written by professional script writers with realistic turn densities, over 30,000 expert-generated turns. Most conversational benchmarks derive and validate their metrics on synthetic data.
Existing metrics fail the correlation test: Both classical automatic metrics and reference-free LLM-as-a-judge approaches are unreliable when correlated against expert human judgment at this scale.
Mixture-of-Judges recovers about 30 percent: Combining multiple evaluative signals improves correlation with human assessment by roughly 30 percent over single-judge approaches.
Beyond factual correctness: The target is human-scale conversational consistency across long multi-turn interactions, which summarization and short-form QA frameworks were never built to measure.
Why it matters: If your agent evaluation rests on a single LLM judge scoring long conversations, this is the paper that quantifies how much you should trust it.
Abstract
As Large Language Models (LLMs) are increasingly deployed to serve open-ended, multi-turn interactions, evaluating conversational quality at human scale has become a central challenge. Existing evaluation frameworks built for summarization, translation, or short-form QA tasks fall short of adequately measuring the consistency of human-scale dialogue, especially when derivation and validation of these metrics themselves often rely on synthetic rather than human sources. We fill the gap by introducing UPHELD (UPwork Human-Scale Evaluated Long Dialogues), a large, reference-full benchmark for evaluating human-scale conversational ability beyond factual correctness. UPHELD consists of hundreds of complete human-to-human dialogues authored by professional script writers, with realistic turn densities and 36,000+ per-turn human annotations across 30,000+ expert-generated dialogue turns. Using UPHELD, we systematically evaluate classical automatic metrics and reference-free LLM-as-a-judge approaches, and find them unreliable when correlated with expert human judgment. Building off this analysis, we use UPHELD to develop a Mixture-of-Judges framework that combines multiple evaluative signals and improves correlation with human assessments by approximately 30%. Overall, UPHELD provides a robust, human-grounded foundation for evaluating human-scale conversational intelligence that fills a crucial gap in the pre-existing LLM dataset landscape.