TokEval帮你更科学地评估分词器，实验显示不同特性影响不同能力，比盲目预训练更高效
TokEval是一个分词器评估框架，超越了标准的生育率和压缩率指标，捕捉语言和结构特性。研究人员通过控制预训练实验，仅改变分词器的训练数据混合、预分词策略和训练算法。他们在bits-per-byte和多个基准上评估模型，涵盖语言理解、数学推理和代码生成。实验表明不同内在属性对模型能力有不同影响：信息论指标预测语言建模能力（Spearman rho高达0.80），而结构敏感指标与任务准确率相关。
TokEval: A Tokenizer Evaluation Suite
Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers' training data mixture, pretokenization strategy, and training algorithm. We evaluate the resulting models on bits-per-byte (a tokenizer-agnostic version of perplexity) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information-theoretic metrics predict language modeling abilities (Spearman rho up to 0.80), while structure-sensitive metrics, such as those measuring digit and line-break handling, correlate with task accuracy. We hope TokEval enables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree.