Papers

dhAIBench:48 个模型在九项语言学标注任务上的基准测试

Benchmarking Large Language Models for Linguistic Annotation

Why we picked it

做语言学标注、文本规范化的研究者可以直接用这个开源工具跑自己模型,不用自己搭评估流程。它连置信度校准和成本都记录,比通用排行榜更贴近实际标注场景。数字人文团队值得上手试试。

arXiv 发布论文介绍 dhAIBench,一个面向语言学标注的 Python 基准测试与标注工具,配有可浏览和共享结果的网页界面。

Body

该框架记录每次运行的提示词、配置和预测,可比较标注质量、置信度校准、可重复性和估算成本。研究团队通过 293 次运行、48 个模型、九项语言学标注任务(涵盖形态学、句法、语义学、语篇-语用、拼写规范化和学习者语言纠正)展示了该框架。

结果显示,部分模型与参考标注一致性较高,包括非标准化的历史语言变体和需要语境解释的任务。结论支持按具体语言学任务评估模型,而非依赖通用排行榜。

基准已在 dhaibench.silent3.

ff.cuni.

cz 公开。

Original · arXiv cs.AI

Benchmarking Large Language Models for Linguistic Annotation

Assessing large language models for linguistic annotation requires task-specific evaluation under documented conditions. We present dhAIBench, a Python benchmarking and annotation tool with a web interface for exploring and sharing results. It records prompts, configurations and predictions, enabling comparisons of annotation quality, confidence calibration, repeatability and estimated cost. We demonstrate the framework and the associated methodology through 293 runs of 48 models on nine linguistic annotation tasks spanning morphology, syntax, semantics, discourse-pragmatics, spelling normalization and learner-language correction. Models achieve high agreement with reference annotations on several tasks, including those on non-standardized, historical varieties as well as requiring contextual interpretation. The results support evaluating models against specific linguistic tasks, rather than relying on general rankings. To this end, we make the current and future benchmarks publicly available at dhaibench.silent3.ff.cuni.cz.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on Europe's AI governance and technology ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include dhAIBench, benchmarks, LLM evaluation, digital humanities.

Source attribution: curated from arXiv cs.AI (@Ondřej Tichý, Anna Marklová, Barbora Bulantová, Magdalena Titlbachová, Jiří Milička).