这篇论文戳破了 Transformer 在入侵检测中“近乎完美”的假象——做网络安全 AI 研究的团队，尤其是依赖 CIC-IDS2017 基准的，建议仔细看 padding 和分割协议的影响，否则你的模型评估可能虚高 0.24 macro-F1。
该研究重新评估了 Transformer 等时序架构在网络入侵检测中的真实效果，发现其性能提升主要来自 padding 方式而非架构本身。在无 padding 的真实序列上，Transformer 的 macro-F1 达 0.89，但零填充掩码评估下骤降 0.24，而 LSTM、GRU 和 1D-CNN 保持稳定。在无泄漏分组评估中，随机森林最稳健，Transformer 的误报率从 0.04% 升至 2.7%，增加 67 倍。研究呼吁采用无泄漏分割、明确 padding 披露和序列感知基准测试作为标准实践。
Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017
Recent deep learning approaches for network intrusion detection increasingly incorporate temporal architectures such as recurrent networks and Transformers, often reporting near-perfect performance on CIC-IDS2017. However, many existing studies neither supply their temporal modules with genuine sequence inputs nor evaluate under realistic, leakage-free conditions, making it unclear whether reported gains arise from true sequence-modeling capability. In this work, we reformulate CIC-IDS2017 as a temporal intrusion-detection task by constructing ordered flow sequences from network conversations and benchmarking nine classical and deep learning architectures under a random split, two leakage-free splits, and a padding-scheme ablation. The central finding is that padding convention, not architecture, determines the Transformer's performance: on genuinely sequential (non-padded) windows the Transformer achieves the highest macro-F1 of any model in the experiment (0.89); under zero-pad+mask evaluation it drops markedly (-0.24 macro-F1), while LSTM, GRU, and 1D-CNN remain stable. Under leakage-free group evaluation the Random Forest is the most robust model (+0.009), while the Transformer's false-alarm rate grows from 0.04% to 2.7%, a 67-fold increase invisible under conventional protocols. These findings demonstrate that evaluation methodology -- specifically padding convention and split protocol -- has a larger effect on reported performance than architectural choice, and that widely used random splits with repeat-last padding can overestimate model robustness by up to 0.24 macro-F1. We advocate leakage-free splits, explicit padding disclosure, and sequence-aware benchmarking as standard practice in future IDS research. Code and implementation details are available at https://github.com/zachmocz/temporal-ids-bench.