做文本相似度、抄袭检测或代码比对的开发者,可以关注这种不依赖语言的新统计特征——它比传统方法更鲁棒,且直接可用。
该论文将视觉计算领域的共现矩阵(COM)和游程长度矩阵(RLM)特征改编用于通用字符串(单词、短语、代码、文本)的相似度计算。这些特征纯统计且不依赖语言信息,适用于任何语言或语法结构。实验表明,在合成数据集上,COM和RLM特征优于最长公共子序列、编辑距离等传统统计方法;在3/4的案例中,其统计显著性显著高于基于距离的第二佳方法(P值<0.001)。在真实文本抄袭数据集上,RLM特征取得了最佳结果。
Proposal and study of statistical features for string similarity computation and classification
Adaptations of features commonly applied in the field of visual computing, co-occurrence matrix (COM) and run-length matrix (RLM), are proposed for the similarity computation of strings in general (words, phrases, codes and texts). The proposed features are not sensitive to language related information. These are purely statistical and can be used in any context with any language or grammatical structure. Other statistical measures that are commonly employed in the field such as longest common subsequence, maximal consecutive longest common subsequence, mutual information and edit distances are evaluated and compared. In the first synthetic set of experiments, the COM and RLM features outperform the remaining state-of-the-art statistical features. In 3 out of 4 cases, the RLM and COM features were statistically more significant than the second best group based on distances (P-value < 0.001). When it comes to a real text plagiarism dataset, the RLM features obtained the best results.