VoiceLongMemEval: Do Assistants Remember How You Sounded?
Ramit Pahwa, Parivesh Priye, and Apoorva Beedu build VoiceLongMemEval, a long-horizon conversational memory benchmark where every answer depends on how something was said rather than on what was said.
Ask this paper
The gap it fills. Existing long-conversation benchmarks test retrieval, temporal reasoning, or knowledge updates over transcripts, and therefore cannot see the paralinguistic channel that human-agent interaction actually carries.
Adversarially gated construction. Every item passes a three-stage gate ensuring a strong language model fails when given only the transcript, so the benchmark cannot be solved by text memory alone.
A pervasive affect gap. Supplying text-track paralinguistic metadata boosts accuracy by 0.09 to 0.38, reaching 0.61 to 0.69 with evidence hints, while standard ASR pipelines systematically discard exactly this signal.
Audio-native models help but do not close it. Models reading speech directly reach 0.354 to 0.412 against 0.325 blind, which is real extraction of prosody and voice events but far from the metadata ceiling.
Abstract
With the growing scale of multi-agent architectures and large language models, deployed AI assistants are increasingly tasked with reasoning over long, continuous, multi-session conversation histories. Current benchmarks evaluate this dialogue history as information retrieval over long horizon, temporal reasoning, or knowledge updates, while crucially ignoring the fundamental dynamics of human-agent interaction, i.e. how they said it. To address this gap, we present VoiceLongMemEval (VLME) benchmark, where every answer depends on paralinguistic metadata (emotion labels, prosody descriptors, and voice events) attached to conversational turns, which is otherwise unrecoverable from the words alone. Every item passes a three-stage adversarial gate, ensuring that a strong language model fails when given only the transcript. Evaluating leading frontier and open-weight models reveals a pervasive affect gap; providing text-track paralinguistic metadata yields a 0.09 to 0.38 accuracy boost (0.61 to 0.69 when prompted with evidence hints), while standard ASR pipelines systematically discard this signal. Additionally, audio-native models successfully extract these cues directly from speech (0.354 to 0.412 vs. 0.325 blind). Code and dataset will be made available upon acceptance.