KierstonBench：用 Allentown 全城模拟评测大模型刑侦与监控能力
Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks
给 Allentown 12.5 万居民造了一套假档案来考 AI 办案，16 个模型里大多数不及格，连前沿模型也会被骗。
arXiv 论文发布 Kierston，一个城市级模拟引擎，为宾州 Allentown 约 12.5 万名居民生成电信、金融、行政和车牌记录。配套的 KierstonBench 包含 32 个任务族，覆盖记录关联、日常活动重建、案件调查和虚假账户识别，模型只能通过受限的记录接口进行调查。对 16 个语言模型的评测显示，大多数模型无法完成记录调查，提交了记录不支持的指认；少数前沿模型能解决大部分任务并在证据不足时放弃作答，但仍会落入常见欺骗陷阱。
Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks
Large language models are candidates for collecting and analyzing records in criminal investigations and mass surveillance. Their capability and accuracy in this work cannot currently be measured. Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark evaluates these capabilities in a simulated environment. We present Kierston, a city-scale simulation engine, and KierstonBench, a benchmark that tests these capabilities against known simulated events. Kierston models approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities and the telecommunications, financial, administrative and license-plate records that surveillance and investigative systems collect about them. These records derive from a common behavioral timeline and are distributed across sources representing separate providers. Models investigate through a restricted records interface, linking observations across sources without access to the underlying simulation. The benchmark comprises 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts. In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. A small number of frontier models solved most of the tasks and abstained when evidence was insufficient. In some cases these models still fell for common deceptions and assumptions, and none replaced an investigator.