KC-Bench Evaluates LLM Agent Conflicts
KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents
KC-Bench provides developers with a reproducible diagnostic tool to improve conflict-aware reasoning in AI agents.
KC-Bench is a controlled multi-turn benchmark with 239 tasks for measuring LLM agents' ability to reconcile knowledge conflicts. It evaluates nine models including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3 across factual correction, identity consistency, and temporal conflict resolution. The benchmark shows substantial cross-domain variation with no model handling all conflict types reliably. Missed conflicts can lead to tool call errors or synthetic protected-data flows.
KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents
As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before taking actions. We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability across world-knowledge conflicts, input inconsistencies, and multi-source temporal conflicts. Its 238 tasks are manually screened from more than 1,000 generated candidates and combine a user simulator, stateful tools, deterministic environment assertions, an open-source natural-language evaluator, and human trajectory verification. Evaluation of nine models, including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3, shows substantial cross-domain variation: no model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings. In the simulated environments, missed conflicts can propagate to tool calls or synthetic protected-data flows. KC-Bench isolates this model-level behavior rather than ranking complete agent frameworks, and provides a reproducible diagnostic for developing conflict-aware reasoning and execution safeguards.
