这项研究揭示了LLM作为攻击者的行为规律和可靠性差异，做AI安全评估或红队测试的团队值得关注——它告诉你不同模型在真实攻击场景下的稳定性和失败模式，直接指导模型选型和防御策略。
这是首个大规模实证研究，测量了LLM在重复渗透测试中的行为一致性。研究对4个模型（Claude Sonnet 4、Gemini 2.5 Flash-Lite、GPT-4o-mini、qwen2.5-coder:14b）各进行100次攻击，目标为固定蜜罐（含OWASP Juice Shop等脆弱服务）。结果显示，Gemini 2.5 Flash-Lite成功率最高（85%），Claude因API故障中断39次但仍达61%，qwen仅25%且主要因过早完成失败。模型失败模式各异：Claude受API截断影响，qwen过早终止，GPT-4o-mini耗尽迭代预算。跨模型成功率差异显著（p<0.001），且首次利用时间集中在15-30秒内。
How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency
Large language models (LLMs) can autonomously conduct multi-stage cyber attacks, but the consistency of their offensive behavior under repeated trials remains unstudied. This work presents the first large-scale empirical measurement of LLM attack consistency: 400 autonomous penetration testing runs (4 models, 100 each) against an identical honeypot hosting OWASP Juice Shop and two additional vulnerable services, holding prompt, orchestrator, and target constant. No model emitted a content refusal that survived the orchestrator's one-shot authorization re-prompt at iterations 0-1. Claude Sonnet 4's API calls did encounter upstream service unavailability - 91 of 1,135 calls returned HTTP 529 overloaded_error during a documented Anthropic capacity event, truncating 39 of 100 Claude runs. An earlier draft catalogued these as safety refusals; on full-log audit they are upstream API failures, not model-level refusals. Despite this, Claude achieved full exploitation in 61 of 100 runs; Gemini 2.5 Flash-Lite in 85; GPT-4o-mini in 56 while deploying 98 unique attack strategies; qwen2.5-coder:14b in 25. Failure modes are model-distinctive: Claude through API truncation (39 runs), qwen through premature completion (52), GPT-4o-mini through iteration-budget exhaustion (23). Cross-service credential reuse appeared only in configurations retaining the most conversation history (qwen 57%, GPT-4o-mini 49%, cloud models 0% on 5-exchange windows). Cross-model exploitation rate differences are statistically significant (p < 0.001) with large effect sizes; qwen vs. Gemini SQL injection rates differ at Cohen's h = 1.12. First-exploit timing fell within a 15-30 second wall-clock range. To our knowledge, this is the first study to measure autonomous LLM attack behavior at N=100 per model across a multi-service target.