大语言模型上下文理解评估研究
Evaluation of Contextual Understanding in Large Language Models
研究人员提出了S3KG评估方法，能更准确衡量大模型在问答任务中的上下文理解能力。
该研究提出了一种基于知识图的评估框架S3KG，结合结构相似性和语义相似性。研究团队在问答基准测试中验证了S3KG的有效性，能够测量LLM生成响应的正确性、忠实度和可解释性。该框架特别关注模型如何提取、整合和推理上下文信息，而非仅依赖表面准确性。
Evaluation of Contextual Understanding in Large Language Models
Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.