想看看AI能不能识破谎言?这篇论文用4个模型做了近4万次测试,结论是准确率跟人差不多,但不同理论引导下AI的倾向差异很大。
该研究构建了7个基于主流欺骗理论的RAG模型,在5个公开数据集、700条陈述上测试,涉及gpt-4o、claude-sonnet-4-6、ollama/llama3、deepseek-v4-flash四个大模型,共生成39,200次欺骗判断。RAG模型准确率54.5%,基线模型54.6%,两者无显著差异,均与人类水平相当。RAG模型(57.0%)比基线模型(59.7%)更少出现真实性偏差,但效应量很小。不同理论对准确率影响不大,但对响应偏差影响显著,从可验证性方法的32.2%到真相默认理论的88.1%。
Theory-Guided Deception Detection: A RAG-Based Artificial Intelligence Exploration
The current work developed seven Retrieval-Augmented Generation (RAG) models based on leading deception theories and compared how deception judgments were made relative to baseline models. Across 700 statements drawn from five published deception datasets, four large language models (gpt-4o, claude-sonnet-4-6, ollama/llama3, deepseek-v4-flash), and two run-types (RAG vs. baseline), a total of 39,200 deception judgments were rendered. Detection accuracies were consistent with typical human accuracies and not statistically different across RAG (54.5%) and baseline models (54.6%). RAG-based models (57.0%) were less truth-biased than baseline models (59.7%), but the effect size was quite small. Theoretical perspective mattered little for accuracy yet mattered substantially for response bias, which ranged from highly lie-biased (the verifiability approach, 32.2%) to highly truth-biased (truth-default theory, 88.1%). Content effects and model effects further moderated the results. Theory-guided AI judgments are unreliable with current parameters, yet they might show promise with additional datasets, model testing, and theory-to-data matching.