KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents
Yaxing Lyu and colleagues build KC-Bench to measure whether a tool-using model can reconcile user instructions, its own parametric knowledge and live environmental observations before it acts on any of them.
Ask this paper
Three conflict types: world-knowledge conflicts, input inconsistencies and multi-source temporal conflicts, which is a finer taxonomy than the usual hallucination framing.
Heavily filtered and instrumented: 238 tasks manually screened from over 1,000 candidates, with a user simulator, stateful tools, deterministic environment assertions, an open-source natural-language evaluator and human trajectory verification.
No model is reliable across all three: evaluation of nine models including DeepSeek-V4-Flash, GLM-5.2 and MiniMax-M3 shows substantial cross-domain variation on factual correction, identity consistency and temporal conflict resolution.
Conflicts propagate into actions: in the simulated environments a missed conflict reaches tool calls and synthetic protected-data flows, which is the concrete harm path.
Deliberately model-level: the benchmark isolates model behavior rather than ranking complete agent frameworks, which makes it useful as a diagnostic when picking a backbone.
Abstract
As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before taking actions. We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability across world-knowledge conflicts, input inconsistencies, and multi-source temporal conflicts. Its 238 tasks are manually screened from more than 1,000 generated candidates and combine a user simulator, stateful tools, deterministic environment assertions, an open-source natural-language evaluator, and human trajectory verification. Evaluation of nine models, including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3, shows substantial cross-domain variation: no model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings. In the simulated environments, missed conflicts can propagate to tool calls or synthetic protected-data flows. KC-Bench isolates this model-level behavior rather than ranking complete agent frameworks, and provides a reproducible diagnostic for developing conflict-aware reasoning and execution safeguards.