Apple Paper Tests LLMs for Machine Translation Quality Annotation
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
MT quality evaluation is often treated as a solved problem once an LLM is in the loop, but this study shows agreement between models and humans is shaky on both sides. Teams building localisation pipelines or multilingual products should benchmark their chosen LLM per language pair and scheme before trusting automated MQM or ESA scores. The comparison with established evaluation frameworks like MQM gives practitioners a concrete reference point.
Apple researchers evaluated how well large language models perform two machine translation quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). They compared model judgments with human annotators across multiple language pairs, domains and levels of annotation granularity. The findings indicate that both humans and models struggle with reliable quality annotation, challenging the assumption that LLMs can simply replace human evaluation. The paper argues LLMs must be thoroughly evaluated across these dimensions before use as human substitutes.
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human…
