基于知识图谱的企业数据助手：模拟评估应对不完美提示词
Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs
研究者在真实企业里测试了 LLM 数据助手怎么应对用户写得乱的提示词，用知识图谱做中介比单纯靠 schema 效果好，做企业数据工具的可以参考。
一项设计科学研究在企业 CFO 部门的数据湖中部署了 LLM 数据助手，发现业务用户的提示词常包含领域词汇且缺少 schema 信息。研究设计并评估了两种方案：基于语义相似度的 schema 中介，以及从数据湖自动构建知识图谱的中介层。使用真实数据的模拟评估结合 LLM-as-a-Judge 和人工验证显示，知识图谱方案在 Precision、Recall、F1 和用户意图保留上优于 schema 方案。
Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs
Enterprise data assistants using large language models (LLMs) promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit's data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information. This Design Science Research study examines automatic resolution using a semantic mediation layer. We design and evaluate two design variants: schema-based mediation via semantic similarity, and knowledge graph-based mediation leveraging an automatically constructed knowledge graph from the data lake. Our simulation-based evaluation using real-world data, supported by LLM-as-a-Judge assessments and human validation, shows that knowledge graph-based mediation outperforms the schema-based approach in Precision, Recall, F1, and preserving user intent. We propose a design principle for knowledge graph-based mediation, demonstrating that structured knowledge representation bridges user intent and database structure and guides the design of enterprise data assistants across skill levels.
