这篇论文在开源数据剖析工具 Desbordante 里加了概率函数依赖发现,对比了传统近似依赖,做数据清洗的话可以看看具体差异。
该论文研究了概率函数依赖 (pFD) 在数据剖析中的应用,并在开源工具 Desbordante 中实现了 pFD 发现算法。通过对比 pFD 与近似函数依赖 (AFD) 在脏数据场景下的表现,发现 pFD 在某些情况下比 AFD 更有优势。论文还评估了 pFD 发现算法的运行时间和内存消耗,并与 AFD 发现算法进行了比较。实验结果显示 pFD 和 AFD 的输出不能互相替代。
Extending Desbordante with Probabilistic Functional Dependency Discovery Support
Data profiling aims to extract complex patterns from data for further analysis and use that data in domains such as data cleaning, data deduplication, anomaly detection, and many more. Functional dependencies (FDs) are one of the most well-known patterns. However, they are poorly suited for these tasks, as real data is usually dirty, and the rigid definition of FDs does not allow algorithms to locate them. For this reason, there are several formulations aimed at relaxing FDs to support dirty data, with approximate functional dependency (AFD) being the most popular one. Another formulation is the Probabilistic Functional Dependency (pFD), which we aim to support inside Desbordante - a science-intensive, high-performance and open-source data profiling tool implemented in C++. However, pFDs are relatively poorly studied, compared to AFDs. In this paper we study pFDs, both analytically and empirically. We start by assessing how different pFDs and AFDs are by studying cases in which pFDs have an edge over AFDs. Then, we implement the algorithm for pFD discovery, as well as study its run time and memory consumption. We also compare it with an AFD discovery algorithm. Lastly, we study the output of both algorithms to learn whether or not it is possible to use AFD discovery algorithm to get pFDs and vice versa.