专门测希腊语变格的基准,500道题验证模型是不是真懂语法规则,Sophea-Genesis-1拿了头名,权重还开源了。
MORFES是一个面向现代希腊语屈折能力评估的基准,包含500个专家验证的条目,侧重低频词元以考察规则应用而非记忆。论文对LLaMA到Qwen3、DeepSeek-R1、Magistral、Kimi K2等开放权重模型进行了评测。研究者自研的Sophea-Genesis-1模型在屈折形态任务上领先,同时通用能力与同等规模模型相当。数据与模型权重均在Hugging Face公开。
MORFES: A Benchmark for Productive Inflectional Competence in Modern Greek
Modern Greek is a richly inflected language, yet the language models built for it are evaluated mainly on factual knowledge, and no benchmark is dedicated to their inflectional competence. We introduce MORFES (Morphological Open-class Recognition-and-Formation Evaluation Suite), a benchmark of 500 expert-verified items that tests the recognition and production of Greek inflected forms, favoring lower-frequency lemmas so that a correct answer reflects the rule rather than a memorized form. We make it publicly available at https://huggingface.co/datasets/KIEFERSA/MORFES. We evaluate a range of open language models on MORFES, situating them within the rapidly scaling open-weight ecosystem from LLaMA to Qwen3, DeepSeek-R1, Magistral, and Kimi K2, where multilingual coverage grows but grammatical competence in morphologically rich languages remains under-measured. Among them, Sophea-Genesis-1, a model we developed and release as open weights at https://huggingface.co/KIEFERSA/Sophea-Genesis-1, leads on inflectional morphology while matching similarly sized models in general capability.