射频无人机基准测试中的数据泄露:理论分析与受控实验

How Much Do RF Drone Benchmarks Overstate? A Controlled Study and Theory of Data Leakage in UAV Signal Identification

精选理由

这篇论文用严格的数学证明和实验告诉你,为什么很多射频无人机识别论文的99%准确率不可信——数据泄露让测试集形同虚设。

AI 摘要

本文揭示射频信号识别基准测试中数据泄露导致精度高估的问题。通过Cover函数计数定理证明,当独立录音数R小于等于特征维度d/2时,分类器可记忆录音-标签映射,使朴素准确率趋近1。在合成实验中,10次种子测试显示,朴素平衡准确率从贝叶斯水平升至1.0,而分组评估降至随机水平,最大差距约0.5。在公开DroneRF数据集上,无人机型号识别(AR vs Bebop)的宏F1从0.74骤降至0.46(二类随机水平)。消融实验证实几乎所有通胀源自片段级数据泄露。

原文 · arXiv cs.LG

How Much Do RF Drone Benchmarks Overstate? A Controlled Study and Theory of Data Leakage in UAV Signal Identification

Radio-frequency (RF) sensing is a central modality for counter-unmanned-aerial-system (counter-UAS) defence because it exploits the control, telemetry, and video links between a drone and its operator. Reported accuracies for RF-based drone detection and identification are often very high, but many are obtained using cross-validation that splits a small number of continuous recordings into short segments. This can place near-duplicate slices of the same recording in both training and test partitions, creating data leakage. We study this leakage problem through theory and measurement. We formalise the optimism of segment-level cross-validation and show, using Cover's function-counting theorem, that a classifier can exactly memorise the recording-to-label map when the number of independent recordings, R, is small relative to the feature dimension, d. In particular, this can occur when 2R is less than or approximately equal to d. Under these conditions, naive accuracy approaches 1, and the inflation gap approaches 1 - ACC*, where ACC* is the Bayes accuracy. The inflation eases only once R grows beyond this separability threshold. A controlled synthetic experiment with 10 seeds confirms the predicted curves: naive balanced accuracy rises from the Bayes level toward 1.0 as recording-specific nuisance variation grows, while honest recording-grouped evaluation declines to chance, with a gap reaching about 0.5. On the public DroneRF dataset, pooled leave-one-recording-out cross-validation shows drone type identification, AR versus Bebop, collapsing from a naive macro-F1 of 0.74 to 0.46, the two-class chance level. A leakage-pathway ablation attributes essentially all of the inflation to segment-level leakage.