Knowledge Graph Evaluation Framework for LLMs
Evaluation of Contextual Understanding in Large Language Models
S3KG provides a more accurate way to evaluate how well LLMs understand context compared to traditional metrics like perplexity or BLEU.
Researchers developed S3KG, a knowledge graph-based evaluation framework for Large Language Models.
The framework introduces Semantic Structural Similarity for KGs to measure structural and semantic similarity.
It was validated against established metrics on a question-answer benchmark, assessing correctness, faithfulness, and interpretability in LLM responses.
Evaluation of Contextual Understanding in Large Language Models
Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include S3KG, context understanding, model evaluation, large language models.
Source attribution: curated from arXiv cs.LG (@Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala, Pragatheeswaran Vipulanandan, Uthayasanker Thayasivam, Kamal Premaratne).
