这篇论文用数据证明印度语言在LLM中平均被吃掉8倍token，马来语更高达13倍，还分析了原因和修复方向，做多语言应用的朋友值得看看。
该研究使用FLORES-200平行语料库，量化了六种分词器下十四种印度语言的Tokenizer Tax。在GPT-3.5和GPT-4使用的cl100k_base分词器中，印度语言平均token化代价是英语的8.0倍，马拉雅拉姆语高达13.0倍，有效上下文窗口缩减至英语用户的12%。主要机制是失败的字节对合并导致单字节token碎片化，merge失败率与代价高度相关（Pearson r=0.89）。多语言分词器如XLM-R和o200k_base可将平均代价降低73%，表明该问题可通过设计修复。固定上下文预算下，印度语言文档保留的原始内容显著少于英语，Belebele基准上的阅读性能差异主要由语言资源丰富度而非分词器行为解释。
The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages
Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages. In this work, we quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus, measuring tokenization fertility across six widely used tokenizers and fourteen languages. Under cl100k_base (used by GPT-3.5 and GPT-4), Indian languages experience an average 8.0x tokenization tax relative to English, reaching 13.0x for Malayalam, reducing the effective context window to as little as 12% of that available to English users for equivalent semantic content. We identify the primary mechanism behind this disparity: failed byte-pair merges that leave text fragmented into single-byte tokens, with merge failure strongly correlating with tokenizer tax (Pearson r = 0.89). We further show that this phenomenon is not an inherent property of Indic scripts but a consequence of tokenizer design. Multilingual tokenizers such as XLM-R and OpenAI's o200k_base reduce the average Indic tokenizer tax by 73%, demonstrating that the disparity is largely remediable. Beyond token statistics, we quantify a practical consequence by showing that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents. Finally, we examine the relationship between tokenizer fertility and reading comprehension performance on the Belebele benchmark, finding that the apparent correlation is largely explained by language resource availability rather than tokenizer behavior alone.