这篇论文研究了语言模型水印在跨语言场景下的公平性问题,发现差异主要源于语言类型学家族,而非特定语言。它提出的评估框架很实用。
研究评估了六种水印方案在三种开源模型、十一门语言上的表现。研究发现,水印检测和质量的差异主要存在于语言类型学家族之间,而非特定语言。该研究提出了一个包含四部分的评估框架,包括基于部署上下文校准的检测阈值、三种独立的质量测量范式,以及一种广义熵分解方法。
Auditing Cross-Lingual Fairness in Language Model Watermarking
Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements. Multilingual deployment exposes evaluation-design choices that are inconsequential on English but determine conclusions cross-lingually. We propose an evaluation framework with four components: detection thresholds calibrated empirically per deployment context, a threshold-independent companion measurement that distinguishes calibration failures from detection failures, three disjoint quality measurement paradigms (distributional, paired-semantic, and reference-perplexity), and a generalized-entropy decomposition of cross-language disparity over a typological family partition. Applied to six watermarking schemes, three open-weight generators, eleven languages spanning four scripts and eight typological families, and both base and instruction-tuned regimes, the framework reveals failure modes that single-language single-paradigm evaluation cannot surface. Across detection and quality, observed disparity is predominantly between-family on the typological partition, indicating that cross-lingual fairness gaps in watermarking are structural to language properties rather than idiosyncratic to particular languages.