统计嵌入:数值表格数据的相似性检索与可解释对齐方法

Statistical Embeddings for Similarity, Retrieval, and Interpretable Alignment of Numeric Tabular Datasets

精选理由

做数据科学或材料信息学的团队终于有了一个无需统一变量名就能对齐异构表格数据的方法,检索准确率高达0.9还支持隐私保护,做RAG或算法选择的开发者可以直接参考。

AI 摘要

该论文提出一种针对数值表格数据集的统计嵌入方法,通过结构化探索性数据分析描述符、预训练句子变换器和典型相关分析(CCA)实现跨数据集相似性检索与可解释对齐。方法无需共享变量名或特征约定,能自动识别驱动对齐的关键统计描述符,并支持差分隐私保护。在15个数据集(涵盖通用基准、材料信息学和核级石墨表征)上评估,P@1分数达0.9,检索和聚类结构鲁棒。该框架为异构数值数据集成到检索增强生成(RAG)流水线提供了统计上下文保留的路径,适用于数据驱动算法选择和模拟模型初始化。

原文 · arXiv cs.LG

Statistical Embeddings for Similarity, Retrieval, and Interpretable Alignment of Numeric Tabular Datasets

Numeric tabular datasets are the dominant data format in scientific practice, yet large language models lack native mechanisms for representing numeric datasets in a meaningful way across heterogeneous feature spaces. Existing approaches either target predictive modeling over individual datasets, which requires a shared set of variable definitions, or lack mechanisms for interpretable cross-dataset alignment. The proposed methodology characterizes numeric tabular datasets through structured exploratory data analysis descriptors, embeds those descriptors into a shared vector space using a pretrained sentence transformer, and quantifies cross-dataset similarity via Canonical Correlation Analysis (CCA). Furthermore, a penalized formulation of CCA is applied to recover sparse, interpretable variable-level correspondences between datasets, identifying which statistical descriptors or variable-level quantities drive cross-dataset alignment without requiring shared variable names or feature conventions. Differential privacy is optionally applied to the descriptor set prior to embedding, supporting deployment in sensitive data contexts without requiring access to raw observations at time of comparison. The methodology is evaluated across 15 datasets spanning general-purpose benchmarks, materials informatics, and nuclear-grade graphite characterization. Results demonstrate a total P@1 score of 0.9, with known nearest-neighbor retrieval and cluster structure remaining robust across embedding ablations and differential privacy budgets. The proposed framework provides a principled pathway for integrating heterogeneous numeric data into retrieval-augmented generation pipelines while preserving statistical context, with direct applications to data-driven algorithm selection and simulation model initialization for unknown datasets.