PACE: Towards Surfacing Hidden Conflicts in User Requests
Yoojin Kim, Jihyoung Jang and Hyounghun Kim introduce PACE, a dataset for whether an assistant can notice that a perfectly reasonable request is wrong for this user right now, given facts it must retrieve rather than facts it is handed.
Ask this paper
Conflict-based refusal is the missing behavior: prior work optimizes faithful execution of requests. PACE scores whether the model should have pushed back given the user's circumstances.
Implicit retrieval is the hard part: the conflict-inducing fact sits in an egocentric knowledge base with no lexical bridge to the request, so the usual request-to-evidence association fails.
PaceMaker as the proposed system: specialized agents coordinate query reformulation, multi-hop graph traversal and conflict-aware filtering to pull the decisive evidence.
Two separate scores: evidence retrieval quality and conflict decision accuracy are reported independently, which keeps a lucky refusal from looking like understanding.
Why it matters: every personal-agent product is about to hit this. Compliance is easy to measure and appropriate refusal is the thing users will actually judge.
Abstract
Personalized assistants should not only comply with user requests but also assess whether those requests are appropriate given the user's current circumstances. However, prior work has primarily focused on accurately executing requests, overlooking the need for assistants to account for context and engage in conflict-based refusal. Furthermore, while existing work on conflict or safety detection relies on explicitly provided factors, real-world scenarios often involve implicit factors that must be retrieved from a knowledge base (KB). To this end, we introduce Personalized Assistants for Conflict Evaluation (PACE), a dataset for evaluating whether models can identify latent constraints, expressed as egocentric knowledge or events, that render seemingly reasonable user requests inappropriate. PACE pairs user requests grounded in well-defined personas with egocentric KB facts, requiring models to integrate contextual evidence to determine whether a request is conflicting. This implicit retrieval setting hinders the direct association between user requests and conflict-inducing knowledge, making it difficult for existing models to identify relevant user-specific facts. To address this challenge, we further propose PaceMaker, a multi-agent framework in which specialized agents coordinate across query reformulation, multi-hop graph traversal, and conflict-aware filtering to retrieve contextually decisive evidence. Experiments on PACE evaluate both evidence retrieval quality and conflict decision accuracy, showing that PaceMaker consistently outperforms existing approaches.